Friday, 2014-01-10

*** banix has quit IRC00:01
*** mrodden has quit IRC00:03
*** mrodden has joined #openstack-infra00:04
*** mrodden has quit IRC00:04
fungijeblair: that's what i'm wondering. in 2.4 self-owned is when owner_group_id=group_id00:04
*** rwsu has joined #openstack-infra00:07
*** dcramer_ has joined #openstack-infra00:07
*** oubiwann has quit IRC00:08
*** sarob has joined #openstack-infra00:09
*** gokrokve has quit IRC00:13
*** mdenny has quit IRC00:13
*** gokrokve has joined #openstack-infra00:13
*** fallenpegasus2 has joined #openstack-infra00:16
*** fallenpegasus has quit IRC00:18
*** gokrokve has quit IRC00:18
*** yamahata has joined #openstack-infra00:20
*** dcramer_ has quit IRC00:20
bknudsonis "No distributions at all found for netifaces>=0.5" ? I was having this problem a few days ago with netaddr.00:21
bknudsonis ... a known problem00:21
bknudsonI'm running into it with keystone now, but gate doesn't seem to have it.00:22
bknudsondoesn't seem to have a problem with it00:22
fungibknudson: i have patches up to fix that in swift... don't recall if they've merged yet00:24
bknudsonfungi: is merged in swift00:24
fungithough i guess my patches just fix it for running tests under tox00:24
*** AaronGr is now known as AaronGr_afk00:25
bknudsonis the gate using older virtualenv or whatever the prob was?00:25
fungibknudson: yes, we have the gate set to use virtualenv 1.10.1 which bundles pip 1.4.100:25
bknudsonok, that makes sense.00:26
fungibknudson: what you're seeing crops up with virtualenv 1.11 which bundles pip 1.500:26
fungibknudson: the *fix* for that is to badger the netifaces devs to start uploading releases to pypi00:26
fungi(or, more work probably, to stop using netifaces and replace it with something else)00:26
bknudsonI don't know how keystone picked this up.00:27
bknudsonpicked up the requirement on netifaces00:27
bknudsonok, test-requirements.txt has it.00:27
fungibknudson: i saw someone updated the shared bug and etherpad with a note about proposed patches to keystone for it00:27
fungithere may already be an outstanding change which wants your review ;)00:28
*** fallenpegasus2 has quit IRC00:28
*** dkliban is now known as dkliban_afk00:28
bknudsonmorganfainberg: that was only 2 years ago.00:29
morganfainbergwell, jenkins uploaded a fix because global requirements changed irt00:30
bknudsonmaybe I should run tests more often than every 2 years.00:30
morganfainberg is the last time that line changed00:30
morganfainberg~2 months ago00:30
morganfainbergpypi seems to claim 0.8 is "state of the art"00:30
* morganfainberg is confused.00:31
morganfainbergfwiw, test-requirements.txt:netifaces>=0.5 is the only place i see netifaces in the code00:31
morganfainbergmight be safe to remove it?00:31
bknudsonmorganfainberg: that would be nice!00:31
*** rfolco has joined #openstack-infra00:32
*** vipuls is now known as vipuls-away00:33
*** vipuls-away is now known as vipuls00:33
*** dstanek has joined #openstack-infra00:33
fungiyes, so sounds like the "stop using it" solution might actually be the easiest path for keystone ;)00:33
* morganfainberg ducks back over to -dev.00:33
morganfainbergfungi, hopefully!00:33
fungibknudson: propose a change which just removes that line, and see what breaks00:33
bknudsonfungi: I'll try it.00:33
*** dpyzhov has quit IRC00:34
*** dpyzhov has joined #openstack-infra00:36
*** ^d is now known as ^demon|away00:37
*** ^demon|away has quit IRC00:39
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Capture libvirtd logs
*** praneshp has quit IRC00:48
*** yamahata has quit IRC00:50
*** praneshp has joined #openstack-infra00:51
*** fallenpegasus has joined #openstack-infra00:54
*** rnirmal has quit IRC00:59
*** dcramer_ has joined #openstack-infra01:03
*** dpyzhov has quit IRC01:04
*** sarob_ has joined #openstack-infra01:06
*** sarob has quit IRC01:09
*** sarob_ has quit IRC01:11
*** jasondotstar has joined #openstack-infra01:16
openstackgerritA change was merged to openstack-infra/config: Move devstack hooks from infra config to solum repo
*** rfolco has quit IRC01:20
*** jasondotstar has quit IRC01:22
*** jasondotstar has joined #openstack-infra01:23
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for bug 1261182
*** dstanek has quit IRC01:23
*** fallenpegasus has quit IRC01:24
*** mozawa has joined #openstack-infra01:24
*** fifieldt has joined #openstack-infra01:24
*** fallenpegasus has joined #openstack-infra01:25
*** thuc has quit IRC01:26
*** thuc has joined #openstack-infra01:26
*** thuc has quit IRC01:30
*** mriedem has joined #openstack-infra01:32
*** nosnos has joined #openstack-infra01:33
*** yaguang has joined #openstack-infra01:33
jog02014-01-10 01:32:52.982 | /opt/stack/new/devstack-gate/ line 349: [[: stable/grizzly: division by 0 (error token is "grizzly")01:36
*** alexpilotti has quit IRC01:38
*** alexpilotti has joined #openstack-infra01:39
jog0looks like a bash error01:39
lifeless[[ is eval mode01:39
lifelessprobably wants [ stable/grizzly ]01:39
lifelessor better [ -z "stable/grizzly" ]01:40
jog0    if [[ "$LOCALRC_BRANCH" -ne "stable/grizzly" ]] && \01:41
jog0is the current logic01:41
fungijog0: sdague had a change to fix it, but the fix was incorrect. i need to see if there was a new patchset01:43
dimsgoogle search says -ne is for arithmetic only....01:43
lifelessdims: correct01:43
jog0fungi: ahh01:43
lifeless != for strings01:43
fungijog0: should still be in review01:43
fungiturns out there is more there than just the trivial -ne vs != operator error01:43
sdagueyeh so we should push that through01:45
fungisdague: how can i tell whether it's working? with -x off and not echoing when it runs, i'm having trouble confirming from the console logs01:48
sdagueso probably ping dkranz tomorrow01:50
sdaguehonestly, my brain is sort of liquid at the moment01:50
fungioh, wait, i should have been looking at grenade logs anyway i think01:50
jog0so the gate is moving again btw01:50
fungisdague: makes sense01:50
sdaguejog0: it's still at 71 changes01:51
jog0sdague: yup01:51
jog0but we just merged a bunch01:51
jog0sdague: hopefully tomorrow AM things will be better01:53
sdaguejog0: we can always hope01:53
jog0I'me hopeful because the gate failure rates graph  on looks good01:53
jog0err better01:53
morganfainberglifeless, if you have a handy link, is there a way to know the worker-id in-side the test framework ... or a way to identify ... say a worker-specific temp dir?01:54
morganfainberglifeless, if so, i think i can fix keystone parallel testing in a couple lines of code01:54
lifelessmorganfainberg: you could use your pid01:54
morganfainberglifeless, ah, valid idea01:54
morganfainberglifeless, cool.01:54
lifelesswouldn't scale to multiple machines sharing e.g. one ldap server, but single machine parallelism should be sufficient for01:55
*** reed has quit IRC01:55
sdaguelifeless: is there a more correct way to handle this -
sdaguebecause the problem with that declaration is that I can't run tox -epy27 one_test01:57
sdaguebecause it runs them all regardless01:57
lifelessright, because it's unionin it with tests.unit as a filter02:00
lifelessjust run testr by hand ?02:00
lifeless.tox/py27/bin/testr run one_test02:00
lifelessor .tox/py27/bin/activate && testr run one_test ?02:01
sdaguehmmm... ok, no way to fix it so the passthrough works as expected?02:03
*** fallenpegasus has quit IRC02:03
*** sandywalsh has quit IRC02:03
*** fallenpegasus has joined #openstack-infra02:04
*** fallenpegasus2 has joined #openstack-infra02:07
*** rockyg has quit IRC02:07
*** fallenpegasus has quit IRC02:07
sdaguewell, it's a new year, so I'll harass you once more for moving testrepository to git somewhere :)02:10
sdagueok, time to really call it a night02:10
*** coolsvap has quit IRC02:14
*** sarob has joined #openstack-infra02:17
*** fallenpegasus2 has quit IRC02:18
*** melwitt has quit IRC02:18
*** fallenpegasus has joined #openstack-infra02:18
openstackgerritA change was merged to openstack-infra/config: Move nova/savanna to only use bare-precise nodes
*** sandywalsh has joined #openstack-infra02:19
*** fallenpegasus has quit IRC02:21
*** fallenpegasus2 has joined #openstack-infra02:21
*** sarob has quit IRC02:21
*** julim has joined #openstack-infra02:23
*** fallenpegasus2 has quit IRC02:23
openstackgerritDevananda van der Veen proposed a change to openstack-infra/config: Enable tempest/ironic gate tests
*** fallenpegasus has joined #openstack-infra02:27
*** gokrokve has joined #openstack-infra02:27
*** praneshp has quit IRC02:28
morganfainbergis there a way to run a final cleanup when a testr worker exits?02:31
*** dstanek has joined #openstack-infra02:31
morganfainbergi guess i could just do the cleanup ahead of time.02:32
morganfainberge.g. at the start of a test run02:32
*** dstanek has quit IRC02:36
*** dcramer_ has quit IRC02:42
*** jasondotstar has quit IRC02:49
*** jerryz has quit IRC02:51
*** dkranz has joined #openstack-infra02:53
*** ^d has joined #openstack-infra02:55
*** fallenpegasus has quit IRC02:56
*** dcramer_ has joined #openstack-infra02:56
*** banix has joined #openstack-infra02:57
*** fallenpegasus has joined #openstack-infra02:57
lifelessjeblair: so - thought - calculate the expected % chance of the next job passing along each branch, then schedule the next highest % job.03:02
lifelessjeblair: would need a chance metric that isn't just constant, or you fill the tree equally03:03
lifelessjeblair: stats from different project success rates might do that03:04
*** sarob has joined #openstack-infra03:04
*** gokrokve has quit IRC03:04
lifelessjeblair: or a bias towards success; or an offset to probability based on branch forkign03:04
*** gokrokve has joined #openstack-infra03:05
*** gokrokve has quit IRC03:09
*** mestery has quit IRC03:16
*** nati_ueno has quit IRC03:17
*** ^d has quit IRC03:18
*** miqui has quit IRC03:22
*** CaptTofu has quit IRC03:23
*** CaptTofu has joined #openstack-infra03:24
*** fallenpegasus has quit IRC03:26
*** pcrews has quit IRC03:27
*** CaptTofu has quit IRC03:28
*** harlowja is now known as harlowja_away03:31
*** Shrews has quit IRC03:39
openstackgerritJames E. Blair proposed a change to openstack-infra/publications: Update periodic pipeline slide and url
*** jishaom has joined #openstack-infra03:40
clarkbmy laptop doesnt want to charge now :/ gonna need to sort that out during luncj03:42
jeblairclarkb: you can use mine if needed.  also, it's possible the computer at the lectern may be usable since yours is on the net.03:43
clarkbcool thanks03:44
jeblairclarkb: fungi: i think we should go ahead and revoke mordred's gerrit access too, just in case03:44
*** gokrokve has joined #openstack-infra03:45
*** chandankumar has joined #openstack-infra03:45
fungijeblair: you mean in addition to what i mentioned in the e-mail thread? (i already removed his ssh keys) remove him from the administrators group too?03:45
fungiand core infra groups, et cetera i guess03:46
clarkbuntil he can invalidate lp auth stuff03:46
jeblairfungi: yes; in case his lp creds are compromised03:46
fungiin case of cached lp openid03:46
fungidoing now03:46
jeblairfungi: i have a copy of his group membership list; i can remove it with a sql command03:47
jeblairfungi: shall i?03:48
fungijeblair: oh, i guess do that. it'll be faster03:48
fungithat'll cover corner cases03:49
*** ryanpetrello has joined #openstack-infra03:49
fungishould we worry about his access to lp resources? jenkins master webuis? wiki?03:50
StevenKfungi: If you're concerned, his LP account can be suspended (IE: no login for him) until he's back and fine03:51
*** nicedice has joined #openstack-infra03:51
fungigithub organization controls also spring to mind, and pypi03:52
jeblairfungi: good point.  without knowing the details, we should probably hit those too.03:53
funginot authenticated through lp, but similar cached login credential risks03:53
jeblairfungi: i'll remove him from openstack-ci-admins and openstack-administrators on lp, which should take care of jenkins and lp itself03:54
jeblairfungi: do you want to take care of github?03:54
jeblairpypi going to be hard since no groups03:54
fungijeblair: doing github now03:55
*** CaptTofu has joined #openstack-infra03:55
jeblairugh: openstack-ci-admins is owned by mordred and not openstack-administrators...03:57
jeblairi guess we should ask an lp question to get that changed03:57
*** CaptTofu has quit IRC03:58
*** CaptTofu has joined #openstack-infra03:58
*** dcramer_ has quit IRC04:01
fungii've removed emonty from openstack-infra, openstack, openstack-dev, openstack-ci and stackforge organization owner teams on github04:04
fungii need a once-a-month refresher on why their interface annoys me. that was my booster shot04:04
jeblairoh good grief04:05
StevenKjeblair: You can probably ask a question, and then ping wgrant in #launchpad and point him at it04:05
jeblairthere are pypi packages owned _only_ by him04:05
jeblairStevenK: thx04:05
clarkbjeblair: ping dstufft?04:05
clarkbdstufft: please to rescue us from emonty04:06
*** ryanpetrello has quit IRC04:06
jeblairclarkb: sdirectory on pypi04:06
jeblairgah sdirector even04:06
fungi sdirector04:07
jeblairdstufft: mordred had his laptop stolen, and we're not sure about the extend of possible compromise...04:10
jeblairdstufft: would it be possible for you to give us a list of pypi packages owned by 'sdirector'04:10
jeblairdstufft: and then we ask you to add 'openstackci' as an owner and remove sdirector for a subset of that list?04:11
jeblairdstufft: it would be much appreciated04:11
clarkbjeblair person sitting next to lana behind us may help on pypi front too04:15
clarkbforget his name04:15
*** mriedem has quit IRC04:17
fifieldtpoor monty04:22
*** AaronGr_afk is now known as AaronGr_Zzz04:22
*** praneshp has joined #openstack-infra04:27
*** yamahata has joined #openstack-infra04:39
*** CaptTofu has quit IRC04:47
*** CaptTofu_ has joined #openstack-infra04:47
openstackgerritKhai Do proposed a change to openstack-infra/config: Add new javamelody-plugin project
*** sarob has quit IRC04:50
*** sarob has joined #openstack-infra04:51
*** jishaom has quit IRC04:51
*** banix has quit IRC04:52
*** sarob has quit IRC04:56
openstackgerritAlexei Kornienko proposed a change to openstack/requirements: Added argcomplete requirement
*** jroovers has joined #openstack-infra04:59
*** amotoki has joined #openstack-infra05:00
*** coolsvap has joined #openstack-infra05:02
*** praneshp_ has joined #openstack-infra05:04
*** praneshp has quit IRC05:05
*** praneshp_ is now known as praneshp05:05
*** masayukig has quit IRC05:14
*** afazekas has joined #openstack-infra05:17
*** rcarrillocruz has quit IRC05:27
*** nicedice has quit IRC05:27
*** nati_ueno has joined #openstack-infra05:28
*** SergeyLukjanov has joined #openstack-infra05:34
*** ^d has joined #openstack-infra05:35
*** kraman1 has joined #openstack-infra05:36
devanandajeblair: re: ironic check/gate patch, can we enable it (move it out of experimental pipe) in a non-voting way, ,in all three projects?05:36
devanandajeblair: my concern is if we only add it to ironic gate, then ironic's gate may be unknowingly broken bychanges in devstack or tempest05:37
jeblairdevananda: right, but then if keystone breaks ironic, tempest can't merge patches05:38
devanandaand we'll have no way to unblock our own gate05:38
jeblairdevananda: yes, you'd have to propose a change to tempest or devstack in that case05:39
devanandajeblair: it seems like a terrible idea for this to remain in experimental pipie for devstack/tempest but be gating for ironic05:40
devanandadevs in those projects wouldn't know when they break us05:40
jeblairdevananda: let's discuss this another time05:40
*** SergeyLukjanov has quit IRC05:41
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Switch horizon to use bare-precise nodes
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Switch ceilometer to use bare-precise nodes
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Switch swift to use bare-precise nodes
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Switch cinder to use bare-precise nodes
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Switch glance to use bare-precise nodes
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Switch keystone to use bare-precise nodes
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Switch heat to use bare-precise nodes
*** ^d has quit IRC06:06
fifieldtnice presentation06:06
*** nicedice has joined #openstack-infra06:15
anteayafifieldt: where are you watching from?06:16
*** dstanek has joined #openstack-infra06:16
anteayafifieldt: nice06:17
anteayahow is the stream quality?06:17
openstackgerritSteve Kowalik proposed a change to openstack-infra/config: Add a Loading paragraph onto zuul's status page
*** HenryG has quit IRC06:22
fifieldtvery good06:24
anteayaglad to have you along in spirit06:26
*** gyee_ has quit IRC06:34
clarkbjeblair: I think I neglected to mention you did the crm114 work :/ sorry06:36
jeblairclarkb: and the bug graphs.  ;) it's ok; that's the nature of backstage work.  :)06:41
*** HenryG has joined #openstack-infra06:41
fifieldtcan I interest anyone in reviewing a one line change to jeepyb? :)06:42
*** pblaho has joined #openstack-infra06:44
anteayafifieldt: +106:46
fifieldtcheers mate :)06:46
anteayaa comment, I don't usually see folks voting on their own patches06:47
anteayaneutron keeps getting a new patchset from Jenkins before we can get this in:
*** jcoufal has joined #openstack-infra06:54
openstackgerritTom Fifield proposed a change to openstack-infra/jeepyb: Add entrypoint for welcome_message
*** yolanda has joined #openstack-infra07:13
*** jamielennox is now known as jamielennox|away07:13
*** yamahata has quit IRC07:14
*** gokrokve has quit IRC07:16
*** gokrokve has joined #openstack-infra07:17
*** gokrokve has quit IRC07:21
dstufftclarkb: jeblair sorry I was asleep07:41
*** dpyzhov has joined #openstack-infra07:41
*** fbo_away is now known as fbo07:46
*** jasondotstar has joined #openstack-infra07:47
dstufftclarkb: jeblair guess y'all are asleep now  yourslef :(07:47
*** gokrokve has joined #openstack-infra07:47
*** gokrokve_ has joined #openstack-infra07:49
*** flaper87|afk is now known as flaper8707:51
*** gokrokve has quit IRC07:52
*** katyafervent has quit IRC07:52
*** gokrokve_ has quit IRC07:53
*** dstanek has quit IRC07:54
openstackgerritMichael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter.
clarkbdstufft we are awake07:58
dstufftclarkb: oh cool07:59
clarkbhelp us dstufft you are our only hope07:59
dstufftclarkb: do you still need help?07:59
dstufftwhat do you need from me?07:59
clarkbcan you get us a list of pypi packages owned by sdirector then we can give you a subset to take ownership of using the openstackci account08:00
clarkbmordreds laptop was stolen and we are taking precautions08:00
dstufftIs sdirector monty on pypi?08:00
clarkbcurrently on perth cell service so may be slow to respond08:01
clarkbyes sdirector is monty08:01
clarkbdstufft thanks we are finding wifi now so that we can edit that list08:02
openstackgerritMichael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter.
*** denis_makogon has joined #openstack-infra08:04
*** rcarrillocruz has joined #openstack-infra08:08
*** ilyashakhat has joined #openstack-infra08:09
jeblairdstufft: if you could add 'openstackci' as an owner for all of those and remove 'sdirector' that would be swell.  thanks.08:12
dstufftclarkb jeblair I'm going to trust you on this given the nature of the problem. If I get yelled at or sdirector ends up not being monty I'm gonna be not a happy camper :D but yea I can do that08:13
*** katyafervent has joined #openstack-infra08:15
jeblairdstufft: thanks.  i could gpg sign something if you want, but i don't know if i'm in your web of trust.08:15
*** sergmelikyan has joined #openstack-infra08:16
*** luqas has joined #openstack-infra08:17
* katyafervent is now away: See you later!08:17
*** katyafervent is now known as katyafervent_awa08:17
*** alexpilotti has quit IRC08:17
openstackgerritMichael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter.
jeblairkatyafervent_awa: please disable your away message;  thanks!08:22
*** mancdaz_away is now known as mancdaz08:23
*** dizquierdo has joined #openstack-infra08:24
*** dstanek has joined #openstack-infra08:25
*** katyafervent_awa is now known as katyafervent08:27
dstufftjeblair: clarkb Ok, here's all that's left
*** yamahata has joined #openstack-infra08:29
*** persia__ is now known as persia08:30
clarkbdstufft looks good thanks.08:31
*** persia has quit IRC08:31
*** persia has joined #openstack-infra08:31
dstufftclarkb: smack monty for not having his laptop encrypted :D08:31
clarkbwe will08:32
*** hashar has joined #openstack-infra08:33
dstufftclarkb: let me know if there's anything else I can help with08:33
* dstufft sheds his admin privs08:33
clarkbI think thats it for now. big help owe you beer08:34
*** mayu has joined #openstack-infra08:35
mayuping anteaya08:35
*** nati_ueno has quit IRC08:39
mayuis there a guide for donating test resource08:39
*** hashar has quit IRC08:39
mayuanyboby know how to donate test resource08:40
*** dizquierdo has quit IRC08:43
*** hashar has joined #openstack-infra08:43
*** chandankumar_ has joined #openstack-infra08:48
*** CaptTofu_ has quit IRC08:48
*** CaptTofu has joined #openstack-infra08:49
*** rcarrillocruz1 has joined #openstack-infra08:52
*** mayu has quit IRC08:52
*** rcarrillocruz has quit IRC08:54
openstackgerritA change was merged to openstack-infra/devstack-gate: Gather horizon/apache2 logs
*** chandankumar_ has quit IRC08:59
kruskaklianteaya: u there ?08:59
*** luqas has quit IRC09:00
*** jpich has joined #openstack-infra09:05
openstackgerritMichael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter.
*** derekh has joined #openstack-infra09:06
*** mozawa has quit IRC09:15
*** gilliard has quit IRC09:19
*** bauzas has joined #openstack-infra09:21
*** yassine has joined #openstack-infra09:22
*** luqas has joined #openstack-infra09:23
*** yassine has quit IRC09:29
*** markmc has joined #openstack-infra09:32
*** _ruhe is now known as ruhe09:33
*** chandankumar_ has joined #openstack-infra09:40
*** gokrokve has joined #openstack-infra09:48
*** denis_makogon has quit IRC09:48
*** nosnos has quit IRC09:49
*** hashar has quit IRC09:51
*** chandankumar_ has quit IRC09:52
*** gokrokve has quit IRC09:53
*** yaguang has quit IRC09:54
*** dizquierdo has joined #openstack-infra09:56
*** CaptTofu has quit IRC09:57
*** nicedice has quit IRC10:01
*** nicedice has joined #openstack-infra10:02
*** yamahata has quit IRC10:05
*** nicedice has quit IRC10:09
*** SergeyLukjanov has joined #openstack-infra10:10
*** nicedice has joined #openstack-infra10:10
EmilienMHello, I wonder why has not been merged, and what could I do ?10:22
*** ruhe is now known as _ruhe10:26
*** jorisroovers has joined #openstack-infra10:28
*** jroovers has quit IRC10:30
*** michchap_ has joined #openstack-infra10:31
*** michchap has quit IRC10:32
*** jorisroovers has quit IRC10:32
SergeyLukjanovEmilienM, it looks like zuul was restarted in this time10:33
SergeyLukjanovwhen it was merged10:33
EmilienMSergeyLukjanov: yeah, what I thought. What could we do then?10:33
SergeyLukjanovEmilienM, you need to reverify it10:34
SergeyLukjanovbut due to the fact that it's not a bug but just a zuul restart, I think that you can use "reverify bug 1" this time10:34
SergeyLukjanovor probably ask someone to approve it again, AFAIK it should help too10:35
EmilienMSergeyLukjanov: ok, thank you for your help10:35
SergeyLukjanovEmilienM, np10:35
*** chandankumar_ has joined #openstack-infra10:37
*** lcestari has joined #openstack-infra10:38
*** dpyzhov has quit IRC10:44
*** bogdando has joined #openstack-infra10:44
EmilienMSergeyLukjanov: it worked, thanks :-)10:45
SergeyLukjanovEmilienM, great ;)10:45
*** gokrokve has joined #openstack-infra10:48
sergmelikyanGuys, why jobs on Zuul are restarting?  Verify job on already around 1 day is stuck in queue10:50
*** nicedice has quit IRC10:50
*** pblaho has quit IRC10:51
*** gokrokve has quit IRC10:53
*** _ruhe is now known as ruhe11:00
SergeyLukjanovsergmelikyan, it was reverified about 5 hoursago11:01
SergeyLukjanovsergmelikyan, dsvm jobs are queued currently11:02
SergeyLukjanovsergmelikyan, you find it here -
*** Bada has joined #openstack-infra11:03
*** ruhe is now known as ruhe_away11:04
*** ruhe_away is now known as ruhe11:04
*** dstanek has quit IRC11:05
sergmelikyanSergeyLukjanov, thx! missed reverify comment (thought it was previous one, sent by me yesterday).11:06
SergeyLukjanovsergmelikyan, np11:07
*** fbo is now known as fbo_away11:07
*** johnthetubaguy has joined #openstack-infra11:14
*** Ajaeger has joined #openstack-infra11:15
*** chandankumar_ has quit IRC11:18
*** luqas has quit IRC11:18
*** rfolco has joined #openstack-infra11:19
*** luqas has joined #openstack-infra11:21
*** andreaf has quit IRC11:26
*** andreaf has joined #openstack-infra11:26
*** dpyzhov has joined #openstack-infra11:29
*** ruhe is now known as _ruhe11:33
*** yamahata has joined #openstack-infra11:34
*** _ruhe is now known as ruhe11:45
*** jasondotstar has quit IRC11:45
*** ymr1 has joined #openstack-infra11:46
*** ymr1 has quit IRC11:48
*** gokrokve has joined #openstack-infra11:48
*** ymr1 has joined #openstack-infra11:50
*** gokrokve_ has joined #openstack-infra11:50
kruskakliwhen I'm using tox based on master it want to install a requirement named 'psutil' which fails, thus I can't run my testcase, any ideas on what to do about it?11:53
*** gokrokve has quit IRC11:53
kruskakliin stable/havana this problem doesn't occur11:53
kruskaklibased on neutron/master that is11:54
*** gokrokve_ has quit IRC11:55
Ajaegerkruskakli: is psutil on pypi?11:55
AjaegerSeems to be - so it's not that bug I had in mind11:55
kruskakliAjaeger: sorry I don't know anything about python and its tools11:57
*** yamahata has quit IRC12:00
kruskakliI run: tox -e py27 -- neutron.tests.unit.ml2.test_mecha nism_ncs12:00
kruskakliand it ends with: Could not find a version that satisfies the requirement psutil>=0.6.1,<1.0 (from -r /home/vagrant/neutron/requirements.txt (line 19))12:01
*** BobBallAway is now known as BobBall12:01
*** chandankumar_ has joined #openstack-infra12:07
zigomordred: There's now Debian official images in HP cloud! :)12:09
zigomordred: Do you know who I should get in touch at Rackspace to make it happen there as well?12:09
*** dstanek has joined #openstack-infra12:11
*** chandankumar_ has quit IRC12:12
kruskaklinot knowing anything, if I do: pip install psutil I seem to get psutil-1.2.1 which seem to contradict the 'psutil>=0.6.1,<1.0' requirement12:22
kruskakliso how can anyone get this to work?12:23
kruskakliperhaps I should ask it in the neutron channel?12:23
* kruskakli sigh...12:23
*** dstanek has quit IRC12:33
*** praneshp has quit IRC12:35
*** bauzas has quit IRC12:43
*** coolsvap has quit IRC12:46
*** CaptTofu has joined #openstack-infra12:46
*** gokrokve has joined #openstack-infra12:48
*** jroovers has joined #openstack-infra12:49
*** GheRivero is now known as GheRivero_afk12:49
*** hashar has joined #openstack-infra12:49
*** gokrokve has quit IRC12:53
*** mozawa has joined #openstack-infra12:53
*** plomakin has quit IRC12:53
*** plomakin has joined #openstack-infra12:53
*** fbo_away is now known as fbo12:57
*** SergeyLukjanov is now known as _SergeyLukjanov13:05
*** matel has joined #openstack-infra13:08
*** SergeyLukjanov has joined #openstack-infra13:08
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: parse the failed jobs in stream
*** smarcet has joined #openstack-infra13:09
kruskakliYES, an ugly workaround made my day: sed -i -e '/psutil/ s/,.*//' requirements.txt13:09
*** fbo is now known as fbo_away13:15
*** dpyzhov has quit IRC13:17
*** sandywalsh has quit IRC13:18
sergmelikyanNow I am sure that is restarting. Around 20 minutes ago I have seen job for this change with ETA 24 minutes and almost finished gate jobs13:19
*** weshay has joined #openstack-infra13:20
portanteclayg: this almost merged, made it to 2nd in line, only to fail. :(
portanteI'll reverify it, back in line it goes ...13:20
portanteacoles: ^^^13:21
*** dpyzhov has joined #openstack-infra13:22
*** mestery has joined #openstack-infra13:28
*** nprivalova has joined #openstack-infra13:28
*** mestery has quit IRC13:29
*** sparkycollier has joined #openstack-infra13:31
*** sandywalsh has joined #openstack-infra13:32
*** mfink has joined #openstack-infra13:32
*** ryanpetrello has joined #openstack-infra13:32
*** eharney has quit IRC13:32
*** dizquierdo has quit IRC13:39
*** fifieldt has quit IRC13:39
*** sparkycollier has quit IRC13:39
*** thomasem has joined #openstack-infra13:42
*** nicedice has joined #openstack-infra13:43
matelHello, Does anyone know if we have a swap partition in the devstack nodes?13:43
*** CaptTofu has quit IRC13:44
*** Shrews has joined #openstack-infra13:45
*** CaptTofu has joined #openstack-infra13:45
*** che-arne has joined #openstack-infra13:47
*** wenlock has joined #openstack-infra13:48
*** gokrokve has joined #openstack-infra13:48
*** dprince has joined #openstack-infra13:50
*** Ajaeger has quit IRC13:52
*** gokrokve has quit IRC13:53
clarkbmatel: we do there is a setup swap function in devstack-gate13:54
*** jasondotstar has joined #openstack-infra13:54
matelclarkb: thanks!13:54
*** dkliban_afk is now known as dkliban13:54
fungianteaya: on i think the main challenge is that translators are working steadily on neutron strings and the gate has been backed up deeper than it could clear between scheduled translation update proposal jobs. maybe just wait for translators or the gate volume to settle down slightly13:56
fungimayu left :( i wanted to discuss the test resource offer! oh well13:56
fungiEmilienM: on 62733, as a core reviewer on that project, you can just (re)add your approval vote if there's no real bug to reverify against13:59
fungiSergeyLukjanov: ^13:59
EmilienMfungi: thank you, I'll know for next time :-)13:59
SergeyLukjanovfungi, morning13:59
SergeyLukjanovfungi, readd means that you could just post +A commend again?14:00
fungikruskakli: actually, i think that *is* the bug Ajaeger was thinking of, he just didn't know where to confirm... if you look at there are no tarballs for <1.0 (only external links)14:02
*** mestery has joined #openstack-infra14:03
fungiSergeyLukjanov: yes, and good afternoon to you sir14:03
kruskaklifungi: ok, should I log it as bug somewhere, or?14:03
hasharSergeyLukjanov: your Zuul patch to show the reconfigured timestamp is almost good to go i think, just missing an update of the status HTML. :D14:04
*** rockyg has joined #openstack-infra14:04
*** banix has joined #openstack-infra14:04
fungikruskakli: anyway, your options are to use pip 1.4.1 instead of 1.5 (which probably means using virtualenv 1.10.1 instead of 1.11), or adjust tox.ini to pass --allow-external psutil --allow-insecure psutil to the pip install command. bug is
SergeyLukjanovhashar, yup, thanks for review, it's on my backlog, hope to complete it today14:05
hasharSergeyLukjanov: also commented out an idea which would be to detect the Zuul layout.yaml  is in a git repository and thus get the git sha1 from there.14:05
hasharbut that is a different feature14:05
SergeyLukjanovhashar, that's the next I was planned to do14:05
fungikruskakli: sorry, linked wrong bug. clearly i'm tracking too many open bugs in my browser right now.
SergeyLukjanovnext step*14:05
hasharSergeyLukjanov: awesome :)14:06
SergeyLukjanovhashar, it's not really helpful due to the fact that in infra layout.yaml file is installed by puppet AFAIK and isn't in the repo I think14:06
SergeyLukjanovhashar, but probably we could add some git/puppet magic to store sha to the file14:07
hasharSergeyLukjanov: on my install the layout is in a git repository which is then used to deploy.14:07
SergeyLukjanovhashar, oh, cool, then I'll try to make such patch too14:07
SergeyLukjanovhashar, I'm only able to read puppet scripts, no good exp on running them14:08
*** dcramer_ has joined #openstack-infra14:08
kruskaklifungi: perhaps the 'psutil>=0.6,<1.0' is too restrictive? seems awkward to have to revert to an older version of pip (but what do I know... :-)14:09
hasharSergeyLukjanov: don't waste your time on it though :D14:09
SergeyLukjanovhashar :)14:09
*** rockyg has quit IRC14:09
fungikruskakli: i agree. presumably something in openstack is broken with >=1 (you could propose a change to openstack/requirements to lift that cap and see if integration tests pass)14:10
BobBallDoes anyone have a nodepool setup / instructions for setting up a nodepool easily that we can use to test and ?14:10
openstackgerritA change was merged to openstack-infra/jeepyb: Add entrypoint for welcome_message
fungiBobBall: jeblair is still drafting the official documentation for it, so right now is't pretty much code comments, docstrings and whatever might be in the readme14:11
fungiBobBall: i know he was working on it this week while he was travelling for lca, so hopefully it will be clearer soon14:12
BobBalland I don't suppose there is a test nodepool somewhere?14:12
*** chandankumar has quit IRC14:12
*** thuc has joined #openstack-infra14:12
fungiBobBall: we have a nodepool instance more or less running on our server if that's what you're asking. or if you mean a way to test code changes, i believe it has some unit tests. i'd have to double check that though14:13
*** thuc_ has joined #openstack-infra14:13
*** bauzas has joined #openstack-infra14:13
BobBallI meant a way to test code changes ;)14:13
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: wip: moving readiness checks into stream
fungizigo: pvo might know who to contact about getting official debian images added to rackspace cloud... i assume these i see in the rackspace image-list are unofficial?
SergeyLukjanovBobBall, we're using nodepool for building slaves for savanna-ci, I don't know about any problems with installing it14:17
*** thuc has quit IRC14:17
zigofungi: Correct.14:17
zigoI have no idea how they have been made.14:18
zigofungi: pvo == Paul Voccio, right?14:19
* fungi checks /whois14:19
ruheBobBall: that should be easy. just tweak all prepare_node* files to reduce number of installed packages. then configure nodepool to use your openstack cloud (it could be even devstack installation)14:20
BobBallthat is correct zigo14:20
BobBallthe other person who might be able to help is anthonym (Ant Messerli)14:20
BobBallthanks ruhe14:20
BobBallsorry! antonym... Erroneous h there.14:21
SergeyLukjanovBobBall, nodepool will work only with nova-net out of the box14:21
anteayafungi: perhaps the weekend is a good time for 6478614:22
fungiBobBall: looking in the nodepool source, we seem to have the beginnings of fakes in the tools dir, but tox isn't configured for running any unit tests and i don't see any written yet14:22
*** oubiwann has joined #openstack-infra14:23
fungianteaya: perhaps... the gate's flowing much faster now that some fixes are in, but the head is still ~15 hours old14:23
BobBallok, thanks to all.  I'll dig deeper.14:23
fungiBobBall: if you end up getting time to write tests though, we'd love the contribution! i'm quite sure it's on the to-do list once people are free'd up from other urgency14:24
*** pliszka has joined #openstack-infra14:24
fungitwo things you can never have enough of... documentation and tests14:25
notmynamewhere are the files that config what channels are logged?14:26
anteayaSergeyLukjanov: please don't suggest people game the reverify bug number syntax14:26
anteayacore can reverify no bug14:26
anteayaor ask in here for a bug number14:26
ruhefungi: btw, is there a way to find prioritized backlog for infra so i could pick up a task? or should i contact infra team?14:27
*** dims has quit IRC14:27
anteayafungi: thanks14:27
* anteaya admires fungi's juggling skills14:28
kruskaklianteaya: Hi, I've managed to get my test.env going again and would like to get back my voting rights (I'll settle with manual voting hereafter :-)14:28
anteayakruskakli: let's start with you offering logs as a patch for a week14:29
SergeyLukjanovanteaya, does you mean bug about slave agent failures?14:29
anteayathat enables people to review your logs and offer your comments for improvement14:29
kruskaklianteaya: ?14:29
SergeyLukjanovanteaya, and I don't think that cores could reverify no bug (
SergeyLukjanovanteaya, only be resending approve as fungi mentioned before14:29
anteayakruskakli: you have to prove your system is stable and that you can offer logs for developers14:29
kruskaklianteaya: but if it's not doing any voting, does it matter?14:30
anteayaSergeyLukjanov: my biggest concern is the suggestion that people game the syntax14:30
*** DinaBelova has joined #openstack-infra14:30
*** dims has joined #openstack-infra14:31
anteayawe are trying to get folks in neutron to use the current syntax as suggested, telling people to game that undoes all my effort14:31
anteayakruskakli: how many patches have you offered to openstack as a contributor?14:31
kruskaklianteaya: none14:31
SergeyLukjanovanteaya, I'm probably misunderstanding you :(14:32
*** thuc_ has quit IRC14:32
*** thuc has joined #openstack-infra14:32
*** cyeoh_ has joined #openstack-infra14:32
*** cyeoh_ is now known as cyeoh14:32
notmynamefungi: do you know where the configs are that control what channels are logged and available in eavesdrop?14:32
SergeyLukjanovanteaya, are you talking about recheck syntax for savanna-ci or something else?14:33
anteayakruskakli: offer at least 3 patches to any project of your choosing and then get back to me on the importance of test logs for developers14:33
anteayaSergeyLukjanov: you suggested EmilienM do recheck bug 1 for his patch14:33
anteayaplease don't tell people to do that14:33
* kruskakli scratching head...14:33
SergeyLukjanovanteaya, oh, yup, the reason was that I forgot firstly about the approve readding14:34
funginotmyname: yeah, what ruhe linked14:34
*** mfink has quit IRC14:35
kruskaklianteaya: hm...I don't understand, I'm not doingn any openstack dev. at the moment, I just want to maintain our NCS driver14:35
*** prad has joined #openstack-infra14:35
SergeyLukjanovanteaya, I don't like bug 1 too and currently know about the correct way to reverify14:35
notmynamefungi: ruhe: thanks14:35
*** mfink has joined #openstack-infra14:35
anteayaSergeyLukjanov: okay fine, please encourage people to follow the intended use of reverify14:35
SergeyLukjanovanteaya, sure14:36
anteayaSergeyLukjanov: the correct way, if core on the project, is reverify no bug14:36
openstackgerritJohn Dickinson proposed a change to openstack-infra/config: start logging openstack-swift
anteayaSergeyLukjanov: if not core, to offer reverify bug <bug-number>14:36
fungianteaya: no, reverify no bug hasn't worked for a couple months. we disabled that, remember?14:36
SergeyLukjanovanteaya, reverify no bug was removed14:36
anteayafungi: I thought core could still use it14:36
SergeyLukjanovanteaya, the only way is to resend approve comment14:36
anteayain the case of infra issues14:37
fungianteaya: you (remove if necessary and re-)add your approval vote14:37
anteayaoh okay14:37
*** thuc has quit IRC14:37
anteayaso again, core can do this14:37
SergeyLukjanovanteaya, fungi, but on the other hand, it looks like reverify no bug :(14:37
fungianteaya: and for infra issues, when the infra team is doing this en masse, we have a 'zuul enqueue' utility we can use to add changes directly to a pipeline14:37
SergeyLukjanovbut little more uncomfortable14:37
anteayakruskakli: how can you fully hope to understand what developers need from your system, if you have no perception of what a developer experiences?14:38
*** rossella_s has joined #openstack-infra14:38
fungiSergeyLukjanov: the big difference from the old "reverify no bug" is that it's now limited to core reviewers on projects, who are hopefully somewhat more accountable for and aware of the problem at hand14:38
anteayafungi: so core can still use revierify no bug?14:39
fungianteaya: no, i just got done explaining that14:39
fungianteaya: core can use approval votes14:39
*** dpyzhov has quit IRC14:39
anteayak, they remove and readd approval14:39
SergeyLukjanovfungi, oh, correct, it's in fact limited to the core now14:39
SergeyLukjanovand that's ok14:39
fungianteaya: the "approved" column you see in the review screen14:39
*** yamahata has joined #openstack-infra14:39
anteayaso non-core needs a bug number14:40
anteayanot a fake one14:40
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Add support to specify GitLab version.
fungianteaya: i was just pointing out that by removing zuul's match on the "reverify no bug" comment syntax, effectively the only ways to reenqueue a change are for a core reviewer to readd an approval vote, for a contributor to leave a "recheck bug xxxxx" comment (which i've also seen going horribly wrong lately, but that's another issue), or for an infra root admin to directly add the change to the gate14:41
SergeyLukjanovanteaya, the problem is that the fake one could be used ;)14:43
fungiruhe: prioritized infra task backlog... (bugs tagged low-hanging-fruit are hopefully things which any contributor can pitch in on without needing root access to infra's servers)14:43
*** jorisroovers has joined #openstack-infra14:44
anteayaSergeyLukjanov: encouraging folks to game the system, which is a big problem in Neutron which I and others are trying to address, please don't add to the problem14:44
anteayafungi: thank you14:44
fungiruhe: we also try to have bug triage days once per milestone or so, to make sure those are properly tagged for most efficient outside contribution14:44
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Add support for parameters in pipeline publisher.
*** eharney has joined #openstack-infra14:45
kruskaklianteaya: perhaps I misunderstand what you are saying but if we need to extend our driver we will do that, if someone changes something that require us to change our driver, we will do that, if something changes that breaks our driver we will "raise a flag"/vote so that the problem can be resolved, any other openstack dev. is not in our plans at the moment14:45
ruhefungi: thank you14:45
*** jroovers has quit IRC14:46
anteayakruskakli: your use of 3rd party testing tools and others have alerted us to a big problem, you are creating feedback with no knowledge of how that feedback is used and consumed by the openstack developers14:48
sdagueoh look, someone poorly configured zuul -
anteayathis is a big problem14:48
fungikruskakli: i have to agree with anteaya... if a vendor's main contribution is to write a driver which supports their proprietary hardware/systems, that's somewhat anti-social in a free software project. gaining familiarity with the ongoing development of the project is encouraged for effective integration with the community14:48
*** gokrokve has joined #openstack-infra14:48
anteayawhile not a requirement yet, we are heading in the direction of requiring two active contributores per 3rd party test system for voting to be enabled14:49
SergeyLukjanovanteaya, +1 for such requirement14:49
anteayaotherwise the messs you can make with your system because you don't know how the results are consumed are just too difficult for the workflow of the developers affected14:49
fungiit helps to have enough familiarity with openstack internals to be able to point out *why* a change is breaking your black box, and to suggest possible alternative solutions14:50
*** jorisroovers has quit IRC14:50
fungiand that is best accomplished through being an active contributor to the core components with which your device or software is interacting in openstack14:50
kruskaklianteaya: I'm sorry to hear that, I've been contributing to open source as long as the concept has existed, but always on a voluntary basis according to what effort I've been able to muster, I need to ponder over this...14:51
*** burt has joined #openstack-infra14:52
anteayakruskakli: we aren't targeting you, please no that14:52
anteayathere is a large influx of people asking for 3rd party testing permission who also have no prior experience contributed to openstack and we are trying to figure out how to address it14:53
*** gokrokve has quit IRC14:53
kruskaklianteaya: sure, I'm taking it personal, but 2 person means 20% of our dev. capacity14:53
anteayahonestly we didn't expect people who have never contributed to openstack would be operating these systems14:54
anteayawe have to be consistent with all the 3rd party test accounts and there are a large number of them now14:54
kruskaklisorry I meant I'm not taking it personal14:54
anteayaI'm glad to hear that14:54
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: Add SQLalchemy database model
anteayayou need to understand what a disruption this is for our devs14:55
fungiit has placed a surprising additional amount of workload on the infra team and, to a great extent, other subprojects that we're trying to teach the third-party ci operators about the openstack development process and workflow, particularly challenging if they're not expecting to contribute directly to the internals of the project14:55
anteayawe have quite a few tests systems run by people with no appreciation of how their logs or lack of them are interupted development14:56
kruskakliok, I'll bring it up with my bozz and we'll see how to proceed14:56
anteayakruskakli: also know we are still figuring out how to approach this from our end14:57
kruskaklianyway, have a good weeken to you all I'm going home :-)14:57
anteayawe didn't expect what we are having to deal with so we didn't have any prior plans as to how to approach it14:57
*** pcrews has joined #openstack-infra14:58
anteayakruskakli: have a good weekend as well14:58
fungikruskakli: good weekend to you too!14:58
*** CaptTofu has quit IRC14:59
*** dcramer_ has quit IRC15:00
*** mriedem has joined #openstack-infra15:01
*** Ajaeger1 has joined #openstack-infra15:02
*** mtreinish has quit IRC15:04
*** jergerber has joined #openstack-infra15:04
*** jergerber has quit IRC15:04
*** mtreinish has joined #openstack-infra15:06
*** coolsvap has joined #openstack-infra15:09
*** dizquierdo has joined #openstack-infra15:14
*** rwsu has quit IRC15:15
*** mrodden has joined #openstack-infra15:17
*** CaptTofu has joined #openstack-infra15:20
*** jgrimm has joined #openstack-infra15:24
*** rwsu has joined #openstack-infra15:26
*** pblaho has joined #openstack-infra15:26
*** rcleere has joined #openstack-infra15:27
sdaguejeblair: as I'm looking over the ever increasing complexity of some of our javascript tools, thoughts on doing something like this instead -
*** jgrimm has quit IRC15:28
*** jasondotstar has quit IRC15:29
bknudsondojo javascript library also has templates --  -- haven't used it other than in widgets.15:31
*** jgrimm has joined #openstack-infra15:31
*** ^d has joined #openstack-infra15:32
*** markmcclain has joined #openstack-infra15:36
*** pliszka has left #openstack-infra15:36
ruhefungi: i'm working on CI/CD for storyboard. puppet module is already on review ( the next step is to add to site.pp15:38
ruhefungi: should i file a ticket for a new server vm for storyboard?15:38
ruheticket in
*** russellb is now known as rustlebee15:39
*** gokrokve has joined #openstack-infra15:39
*** mfink has quit IRC15:39
*** kraman1 has quit IRC15:40
openstackgerritAntoine Musso proposed a change to openstack-infra/zuul: Use venv to build documentation
*** mfink has joined #openstack-infra15:40
*** mrodden1 has joined #openstack-infra15:42
*** wenlock has quit IRC15:42
*** alaski has quit IRC15:43
*** mrodden has quit IRC15:43
openstackgerritMichael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter.
*** dansmith is now known as damnsmith15:44
*** GheRivero_afk is now known as GheRivero15:46
*** ymr1 has quit IRC15:47
*** dizquierdo has quit IRC15:47
*** jasondotstar has joined #openstack-infra15:48
fungiruhe: nah, i can just launch a server and either apply the change once merged or in a separate development puppet env to check it out15:55
*** herndon_ has joined #openstack-infra15:56
ruhefungi: that's great. i'll get back to you once storyboard is ready to be deployed :)15:57
openstackgerritSean Dague proposed a change to openstack-infra/config: provide time in queue in zuul ui
sdagueok, I'm pretty happy with that one :)15:58
fungiruhe: let me make sure i find some time to review it, and then i'll test it out15:58
fungiruhe: it's on my list for later today, after i get more done on the server config management for our upcoming openid provider service15:58
*** dangers_away is now known as dangers15:59
*** flaper87 is now known as flaper87|afk15:59
fungiruhe: i'd love for you to have a demo storyboard already up in our infra by the time the sprint happens15:59
*** thedodd has joined #openstack-infra15:59
ruhefungi: yeah, that's my (and everybody else's) to have it running before the sprint15:59
SergeyLukjanovsdague, oh, it looks interesting ;)15:59
ruhe*my goal15:59
fungiruhe: it's a laudable one16:00
fungii won't be at the sprint, but i gather some other infra folk may be popping over from fosdem for it16:00
ruhefungi: afaik mordred and jeblair will be there16:01
*** herndon_ has quit IRC16:01
fungiruhe: perfect16:02
SergeyLukjanovsdague, thx16:02
SergeyLukjanovsdague, I'll take a look on your change today, it looks very useful16:03
sdagueyeh, then I stop having to do math to see how bad our queue is16:03
sdagueand, right now, top of queue entered 15+ hrs ago16:03
fungii love that we have so many people contributing usability improvements to the zuul status screen now. that tells me there are *tons* of people who find it useful already too16:03
SergeyLukjanovfungi, I'd like to rewrite the js code to the more readable/patchable style, but I'm afraid that my knowledge of js isn't good enough now16:05
fungithough the test nodes graph has me worried that we have a lot of our aggregate quota full of stale entries stuck in a building or deleting state... checking up on that now16:05
fungiSergeyLukjanov: nor mine either, but sounds like an excellent opportunity to get better at it ;)16:05
SergeyLukjanovfungi, I'll probable return back to this idea after a pack of minor improvements to the server side zuul16:06
sdagueSergeyLukjanov: actually, look at
openstackgerritMichael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter.
sdagueI think that would go a huge way16:10
SergeyLukjanovsdague, I think that templating will not be enough to make it more readable16:11
*** mdenny has joined #openstack-infra16:11
SergeyLukjanovsdague, I'm looking on something like angular.js or knockout.js16:11
SergeyLukjanovboth of them supports templating16:11
SergeyLukjanovand will provide us with ability to not rewrite the whole html for all changes16:12
SergeyLukjanovonly changed elements16:12
sdagueSergeyLukjanov: yeh, that would be a far more extensive undertaking16:13
sdagueI think you could get rid of a ton of debt just getting to templates16:13
sdagueand then do a further conversion in the future16:13
SergeyLukjanovsdague, yup, that's not so destructive as a fully rewrite ;)16:13
sdagueSergeyLukjanov: and could be done in bite sized chunks16:13
sdaguehonestly, if jeblair isn't against it, I was going to start trickling in templated stuff over time16:14
*** gokrokve has quit IRC16:18
*** gokrokve has joined #openstack-infra16:18
*** pblaho has quit IRC16:20
*** UtahDave has joined #openstack-infra16:21
*** gokrokve has quit IRC16:23
fungiyeah, so roughly half of the nodes nodepool knew about were >3 hours old and not picking up any jobs. i've got 5 parallel delete loops going over the list of stale nodes now, to get us back to capacity, then i'll restart nodepool gracefully and start watching for any new signs of a persistent leak16:25
fungisome of the nodes were >100 hours old16:25
*** alaski has joined #openstack-infra16:26
fungiwhich takes us back to the jenkins restarts early in the week when we cleared everything16:26
*** reed has joined #openstack-infra16:26
*** beagles is now known as beagles_brb16:27
markmcclainso I've got a question about list.o.o16:28
fungimarkmcclain: ask away16:29
markmcclainI updated my subscriber address yesterday and it seems to take effect immediately16:29
markmcclainand then this morning mail started showing up on the old address16:29
*** SergeyLukjanov has quit IRC16:30
fungithat's weird. old dates on the headers?16:30
fungiwondering if those might have been stuck in a deferred queue somewhere16:30
markmcclainall current messages16:30
fungidoes the subscriber address in mailman show your old one again?16:31
*** kraman1 has joined #openstack-infra16:32
jog0 is showing no gate hits16:32
*** gyee_ has joined #openstack-infra16:32
jog0which I find hard to believe16:32
fungijog0: do i need to check the listener daemon?16:33
jog0fungi: that is graphite data16:33
markmcclainfungi: yes16:33
fungijog0: how long has it been?16:33
fungimarkmcclain: i'll check for any strange errors on the list server, or signs anyone might have been doing maintenance16:34
fungimaybe someone restored an earlier subscriber database or something16:34
jog0a few hours16:34
markmcclainfungi: thanks.. I'll just go ahead and update the address again16:34
fungimarkmcclain: sorry about that--i agree that's not how it should have to work ;)16:35
markmcclainno worries16:35
*** derekh has quit IRC16:36
fungimarkmcclain: yeah, nobody's logged into a shell on that server for nearly 4 weeks16:36
*** AaronGr_Zzz is now known as AaronGr16:36
portantejog0 that logstash appears cut off ...16:37
jog0portante: logstash? that was a graphite link16:37
*** CaptTofu has quit IRC16:38
*** jcoufal has quit IRC16:38
portantethat link above ... :)16:38
fungihowever, portante, i agree it's truncated16:38
fungiand it ends abruptly there16:39
jog0anyway see graphs at
openstackgerritMate Lakat proposed a change to openstack-infra/nodepool: Readme enhancements
*** jog0 is now known as flashgordon16:44
*** changbl has quit IRC16:46
*** markmcclain has quit IRC16:49
fungi[savior of the universe]16:53
* fungi seems to get queen's flash gordon theme stuck in his head every friday, like clockwork16:53
funginode deletes are still going, but the test nodes graph is already starting to look a lot more sane16:54
fungiand we've got waaay more changes being tested at a time now as a result16:55
openstackgerritRuslan Kamaldinov proposed a change to openstack-infra/storyboard: Get rid of Django
*** ruhe is now known as _ruhe16:59
guitarzanfungi! king of the impossible!16:59
*** rwsu has quit IRC17:00
*** Ajaeger1 has quit IRC17:02
*** dcramer_ has joined #openstack-infra17:04
*** wenlock has joined #openstack-infra17:06
*** bnemec is now known as beekneemech17:07
*** _ruhe is now known as ruhe17:07
*** pcrews has quit IRC17:07
*** gokrokve has joined #openstack-infra17:08
*** dcramer_ has quit IRC17:09
funginope, that's jog0's job title17:10
openstackgerritDirk Mueller proposed a change to openstack/requirements: Allow newer versions of psutil as well
*** rnirmal has joined #openstack-infra17:11
zarofungi: _david_ requested repro steps for this..
zarofungi: i'm not sure how we setup the CLA, would you be able to provide info?17:13
fungizaro: that runs
*** SergeyLukjanov has joined #openstack-infra17:19
zarofungi: i'll take a look. thanks.17:19
*** mancdaz is now known as mancdaz_away17:23
*** beagles_brb is now known as beagles17:24
*** hashar has quit IRC17:25
*** madmike has joined #openstack-infra17:25
*** CaptTofu has joined #openstack-infra17:26
*** jpich has quit IRC17:27
*** mfink has quit IRC17:27
fungizaro: at the time we set it up, the extent of the instructions was the "auth.contributorAgreements" entry in
fungizaro: "the admin must also insert one or more rows into contributor_agreements and create agreement files under '$site_path'/static, so users can actually complete one or more agreements."17:29
*** dstanek has joined #openstack-infra17:29
fungizaro: so that's basically what that script is doing17:29
zarofungi: seems like we all this is done thru db.  is there an equivalent way to do this stuff from gerrit ui?17:30
fungizaro: nope, or at least there was none at the time17:30
fungias it was, i had to contribute patches upstream (which landed in 2.5) and backport them to 2.4 because the feature was still half-baked17:30
sdakerecently I changed permissions on the #heat channel to add logging support for the channel, where are the irc logs stored?17:31
zarofungi: do you remember if this was our hack or was it something recommended by gerrit docs?17:31
sdakethanks fungi17:31
fungizaro: it was our automation of the (very minimal) gerrit docs17:31
*** mriedem has quit IRC17:32
fungizaro: the bugs i fixed were unrelated, the cla signing interface in 2.4 was simply broken when we started trying to use it17:32
*** markmcclain has joined #openstack-infra17:32
fungizaro: specifically bugs with the non-group based self-verifying cla functionality. so it's possible gerrit devs might consider a broken feature to not have been an actual released feature, since you couldn't use it without patching and rebuilding gerrit17:34
fungii think one of the bugs was there, and one of the bugs was actually related to the contact information form being broken/unusable instead17:34
zarofungi: wow, seems like it would be difficult for me to repro on 2.417:35
zarofungi: and it might not even be a bug against upstream at all because we are upgrading from our forked gerrit.17:36
fungiit may not be worth our time to reproduce, since they fixed it in 2.5 but did not backport fixes to 2.417:36
fungizaro: well, it's a failure to support upgrading from a broken feature which nobody could have used without backporting the fixes themselves, as we did17:36
fungigerrit fixed them upstream, but they fixed them in 2.517:37
openstackgerritSushil Kumar proposed a change to openstack-dev/hacking: Add -U to pip install command in tox.ini
fungiand i think they may also have dropped support for non-group-backed contributor agreements in 2.5 as well17:37
*** dstanek has quit IRC17:37
*** blamar has joined #openstack-infra17:40
*** dpyzhov has joined #openstack-infra17:41
zaronot sure what to do about this.  i don't think this is an upstream bug at all now.17:42
*** dcramer_ has joined #openstack-infra17:42
*** ruhe is now known as _ruhe17:43
fungizaro: well, it's an upstream bug, i think, but i can understand them not caring about it17:43
zarofungi: do we have to automate the upgrade from openstack gerrit 2.4 to upstream gerrit 2.8?17:44
fungizaro: as i said, the feature was broken in 2.4, and they opted not to support it, but rather replaced bits of it in 2.5. i think that's more or less the resolution17:44
zarofungi: my thought was that we do manual upgrade from 2.4 to 2.8, then automated from 2.8 and beyond.  since 1st upgrade is such a big spread.17:45
fungizaro: i'm still in favor of just working around it, so long as the situation's documented upstream for the benefit of others who may have also hacked the broken cla support in 2.4 into a working state and then later decide to upgrade17:45
*** che-arne has quit IRC17:45
*** che-arne has joined #openstack-infra17:45
zarofungi: yeah, i agree.  there was another problem as well where we needed to truncate a table before the upgrade.17:46
fungizaro: we can discuss upgrade automation with the group. we'll want to script it regardless, whether it's in the form of scripts in our puppet repo or prewritten scripts in pastes or an etherpad17:46
fungibecause we want it to be exactly repeatable in case we end up having to roll back, revise and try again17:46
*** dcramer_ has quit IRC17:47
zarofungi: cool.  i'll update the bug with more accurate info, like the fact that we are upgrading from our forked gerrit 2.4.17:47
*** BobBall is now known as BobBallAway17:47
*** harlowja_away is now known as harlowja17:47
fungispecifically that we hand-patched 2.4 with fixes backported from 2.5 to get it working17:47
fungibecause it had been released in a broken state in 2.417:48
zarook. i have a better picture of what happened now.  i'll update the bug to see if _david_ wants to pursue.  i'm thinking it's unlikely that he would.17:50
fungiright, he seemed to be pretty insistent that it should get fixed upstream if it's a bug, but in this case i think it's a design flaw which upstream knew about and decided not to fix back in 2.517:51
fungiinstead they redid how some of it worked, and seemed to have decided not to provide an upgrade path from the nonworking implementation in 2.417:52
fungiwhich makes sense. it's a pragmatic choice17:52
fungito have actually used it in 2.4 you would have had to be running a fork17:53
fungijust to get it usable17:53
*** dstanek has joined #openstack-infra17:56
*** hogepodge has quit IRC17:57
*** nati_ueno has joined #openstack-infra18:01
*** nati_uen_ has joined #openstack-infra18:02
reedwhen will people be able to create new projects on stackforge?18:02
*** nati_uen_ has quit IRC18:03
fungireed: i've been reviewing the new project proposals and looking for one with no further issues i can test with. i didn't get a chance to look back at them again since wednesday when i went through them last, but hopefully i can try to put one through today18:03
*** mriedem has joined #openstack-infra18:04
*** w_ has joined #openstack-infra18:04
fungiright now trying to get the gate back into a happy state (nodepool is taking far too long to gracefully stop)18:04
*** nati_uen_ has joined #openstack-infra18:04
reedfungi, check this one :)18:04
*** nati_ueno has quit IRC18:05
fungireed: i'd missed that one since it got auto-abandoned18:05
fungiadding to my review18:05
*** GheRivero is now known as GheRivero_afk18:05
reedfungi, thanks18:05
*** mriedem has quit IRC18:06
*** olaph has quit IRC18:06
fungii've restored it. i'll give it a quick rebase18:06
fungiactually, it needs editing too. i'll leave comments18:07
*** luqas has quit IRC18:07
*** jerryz has joined #openstack-infra18:08
*** markmc has quit IRC18:09
*** pcrews has joined #openstack-infra18:12
flashgordonfungi: heh18:12
*** vipuls is now known as vipuls-away18:15
*** vipuls-away is now known as vipuls18:15
*** Bada has quit IRC18:16
*** mgagne1 has joined #openstack-infra18:16
openstackgerritGuido Günther proposed a change to openstack-infra/jenkins-job-builder: maven-deploy: don't require id
openstackgerritGuido Günther proposed a change to openstack-infra/jenkins-job-builder: project_maven: Don't require artifact-id and group-id
*** mgagne2 has joined #openstack-infra18:18
*** thuc has joined #openstack-infra18:18
openstackgerritGuido Günther proposed a change to openstack-infra/jenkins-job-builder:  Explain howto use maven's pre/postbuilders
*** mgagne has quit IRC18:19
*** thuc has quit IRC18:19
*** thuc has joined #openstack-infra18:19
*** thuc has quit IRC18:20
*** thuc has joined #openstack-infra18:21
*** mgagne1 has quit IRC18:21
*** thuc has quit IRC18:21
openstackgerritGuido Günther proposed a change to openstack-infra/jenkins-job-builder: project_maven: Don't require artifact-id and group-id
*** thuc has joined #openstack-infra18:22
*** gyee_ has quit IRC18:23
*** hogepodge has joined #openstack-infra18:27
*** sarob has joined #openstack-infra18:28
*** sarob has quit IRC18:28
*** sarob has joined #openstack-infra18:28
*** mgagne has joined #openstack-infra18:30
*** mgagne is now known as mgagne118:30
*** mgagne2 has quit IRC18:30
harlowjahmmm, qq, it seems like most of the stackforge jobs (and alot of others) are sitting at 100% done (but not going past 100%), something is up i guess right?18:34
fungiharlowja: yes, that's me18:34
fungii'm trying to gracefully restart nodepool18:34
harlowjathx fungi18:34
fungishould pick back up here momentarilyu18:34
funginothing's broken, its just we had a lot of cruft i'm trying to clean out to get our capacity back up18:35
harlowjacool, for those that use the czuul tool that i made (trunk has the following that i added, czuul --project "stackforge*")18:35
harlowjato watch certain projects only :-P18:35
*** SergeyLukjanov has quit IRC18:35
fungiharlowja: basically, we're out of available nodepool-managed nodes for a bit while it's restarting18:36
*** dizquierdo has joined #openstack-infra18:36
fungizuul gobbled up the remaining supply while nodepool was busy trying to work through provider nova api timeouts18:36
harlowjaneat stuff, i saw that jeblair i think was doing some conference about zuul,openstackinfra, is that video online anywhere?18:36
harlowja (maybe its happening right now?)18:37
fungiharlowja: if you're talking about lca this week, not sure if they've got any vids up. the presentation slides are probably mostly reflected in
harlowjaah, thx18:38
*** SergeyLukjanov has joined #openstack-infra18:38
flashgordonfungi: having fun as the infra skeleton crew18:43
*** Ryan_Lane2 has joined #openstack-infra18:45
fungiflashgordon: of course18:45
fungialso, nodepool is back up again18:45
fungiwithin the next 10 minutes we should have nodes running more tests18:45
Ryan_Lane2on a scale of 10 - not interested, if I added github support to zuul would my changes be accepted upstream?18:45
fungiRyan_Lane2: curious what you mean by "support" (github triggers or some such?)18:46
Ryan_Lane2github triggers and reporting back into github18:46
Ryan_Lane2basically gating support for it.18:46
fungiRyan_Lane2: i suspect it would need to come with a fake backend and a fair battery of tests since we'd not want to connect to github when running zuul unit tests18:47
fungiRyan_Lane2: but i'll defer to jeblair on it18:47
fungias to whether or not we'd accept patches for support in-tree18:48
*** DinaBelova has quit IRC18:48
Ryan_Lane2I'm not working somewhere not using gerrit, but am quickly approaching the limits of what I can do in jenkins18:48
fungiseems like a reasonable request to me as long as we can be assured we don't break it and don't need to touch github ourselves, but i'd seek additional consensus when people aren't in post-lca travel18:48
Ryan_Lane2sounds good. it may be that I don't end up using zuul, but I wanted to check before I went down that rabbit hole18:49
fungiRyan_Lane2: looked at turbo-hipster? (non-jenkins job runner written in python by some of our contributors)18:49
zarofungi: about discussion yesterday regarding Change Owner owner_group_id is set to 0 in the gerrit DB..18:50
fungizaro: yeah, did you ever get additional details on that?18:50
zarofungi: did some testing and found out that gerrit 2.8 doesn't even have that column anymore.18:50
fungizaro: i'm going to guess 0 means "no owner"18:50
zarofungi: in fact for 2.9 they moved all users from the DB.18:51
Ryan_Lane2I think we're fine using jenkins, but wanted to handle gating better than jenkins itself can handle18:51
zarofungi: So the migrated tables don't even get used anymore, i think everthing got moved from db to git repo.18:51
fungizaro: any idea when they're targeting for a 2.9 release? it sounds like they're doing such heavy refactoring we might be better trying to work on an upgrade to 2.9 if it's soon, and save ourselves extra work18:52
Ryan_Lane2fungi: do you know if zuul could have a merge event in master trigger events in open change requests?18:52
zarofungi: which means i don't have to patch that at all :)18:52
fungiRyan_Lane2: i'd need to know what "trigger events in open change requests" meant in this case18:52
fungizaro: heh18:52
Ryan_Lane2well, I'd like to do something like rebase and recheck open change requests, so they are always tested against master18:53
*** mfer has joined #openstack-infra18:54
Ryan_Lane2my goal is continuous deployment, where all change requests are fully tested against the master and are always ready to merge and deploy18:55
fungiRyan_Lane2: jeblair was talking about possibly adding a timed trigger to retest open changes in zuul and report on them if they're stale18:56
Ryan_Lane2that would be awesome18:56
fungibasically as a low-priority pipeline18:56
fungiso it gets available workers if there's nothing else taht needs them18:56
Ryan_Lane2in my case I'd have it as a trigger that occurs on master merges and retests. I just need to add a trigger for that, I'd imagine?18:57
fungiRyan_Lane2: if you're interested in supporting that (i know it was on sdague's wishlist too) you should talk to jeblair about implementation details18:57
*** dpyzhov has quit IRC18:57
fungiRyan_Lane2: that sounds sane... if master moves and there are changes which are still open, retest and report18:57
Ryan_Lane2well, let me ensure I'm going to use this before I make any promises :)18:58
fungimaybe with a throttle so it won't do it unless it's been for a configurable period of time18:58
Ryan_Lane2yeah, that's a good idea18:58
fungiand maybe only have it leave a comment if it's failing18:58
fungior rather, only comment if it would be a state/vote change18:59
* Ryan_Lane2 nods18:59
fungito catch open changes which go from broken to working without a new patchset because master grew a feature they needed or something18:59
*** rwsu has joined #openstack-infra18:59
Ryan_Lane2I can already tell this is going to be a pain in github since it doesn't have gating support18:59
fungiRyan_Lane2: you know you love travis-ci. don't deny it ;)19:00
Ryan_Lane2we're not using it19:00
*** herndon_ has joined #openstack-infra19:01
*** Ryan_Lane2 has quit IRC19:02
*** herndon_ has quit IRC19:02
*** andreaf has quit IRC19:03
mferfungi the other day you provided feedback on i'm looking for a bit of direction because what you said appears to differ from what's documented. i'm trying to understand it before i make any changes19:04
mferfungi can you take a look?19:04
*** Ryan_Lane2 has joined #openstack-infra19:04
fungimfer: sure! i'm behind on circling back to check feedback on reviews where i commented19:04
fungithe week's been a bit crazy19:05
mferfungi i understand that. i've gone long periods just hoping to catch up.19:05
*** rossella_s has quit IRC19:05
*** johnthetubaguy has quit IRC19:06
*** mrodden1 has quit IRC19:06
*** Ryan_Lane2 has quit IRC19:06
*** Ryan_Lane2 has joined #openstack-infra19:07
flashgordonfungi: I assume you already knwo but no movement in the gate queue19:07
*** dizquierdo has quit IRC19:07
fungiflashgordon: it's picking back up now. i was restarting nodepool to clear out cruft nodes which had us only using half our capacity19:07
*** mrodden has joined #openstack-infra19:08
flashgordonfungi: ack19:08
*** dizquierdo has joined #openstack-infra19:08
fungiflashgordon: it needs a few more minutes to spin up additional nodes so it can catch back up19:08
*** mriedem has joined #openstack-infra19:09
flashgordoncool wasn't sure how long ramp up was supposed to take19:09
sdagueflashgordon: btw -
fungiflashgordon: spinning up new nodes takes on the order of 15 minutes, and it gets throttled a little so we don't invoke the wrath of our generous cloud providers19:10
sdaguethat will move the overall gate stats to a dedicated page19:10
flashgordonsdague: woot19:10
*** Ryan_Lane2 has quit IRC19:10
fungiflashgordon: due to a gate reset when it was winding down, zuul's allocating a bunch of the initial pool to changes in the check pipeline19:10
*** herndon_ has joined #openstack-infra19:11
flashgordonfungi: huh?19:11
sdagueoh man - I so *want* in  :)19:11
sdagueI'm using it locally now, so much more handy to figure out the state of things19:11
flashgordonso whats the time in queue for gate?19:12
fungiflashgordon: changes get tested first-come-first-served, so a gate reset means that all the restacked changes in the gate pipeline get to wait their turn after changes which were already in the check pipeline waiting on nodes19:12
flashgordonfungi: O_o oh19:13
*** dstanek has quit IRC19:13
flashgordonfungi: can we weigh gate somehow19:13
flashgordonactually scratch that19:13
flashgordonthats a can of worms19:14
sdagueflashgordon: we did that once, then we starved out check19:14
*** ivar-lazzaro has joined #openstack-infra19:14
fungiflashgordon: we tried priorities, but it meant that when the gate got deep enough to use all available nodes, check pipeline changes sat indefinitely waiting for resources and devs never got any feedback19:14
sdagueflashgordon: top of gate is 16hrs19:14
flashgordonsdague: yeah can o worms19:14
flashgordonsdague: 16 hours nice!19:14
flashgordonso the answer would be prioritize the top x in gate or something like that19:14
openstackgerritDavanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Capture libvirtd logs
sdagueflashgordon: pull - and gvfs-open status.html19:15
fungiflashgordon: it might be possible to do a weighted distribution algorithm, like you said, instead of straight prioritization, but that would need new code in zuul19:15
flashgordonfungi: right, maybe just prioritize top 1 of gate19:15
*** w_ is now known as olaph19:15
flashgordonsdague: who nice!19:16
flashgordonfungi: but this is an unsual case and meh19:17
fungiflashgordon: yeah, i suspect we still have a node leak, probably manifests around jenkins master restarts, but i want to watch it closely now that i've cleaned it up and see what starts to pile up when19:18
*** _ruhe is now known as ruhe19:18
flashgordoncool, thanks for running the infra all by your self btw19:18
fungibut you can see on the graph now we've got far more nodes "in use" than we did a few hours ago19:18
fungiflashgordon: well, i'm not all by myself, just happen to be by myself for a good chunk of the day in this timezone19:19
*** ryanpetrello has quit IRC19:19
flashgordonfungi: I think the in-use graph is wrong19:19
fungithe other root admins have been pitching in from au as they got time (admirably missing meals and recreation i expect)19:19
flashgordonbecause those in-use numbers were like that when no new jobs were running19:20
flashgordonfungi: right, thank you au folks19:20
fungiflashgordon: the in use graph will be more right in a bit. used nodes get switched to deleting as nodepool gets a moment to catch its breath19:20
*** ryanpetrello has joined #openstack-infra19:20
fungiso those were left "in use" while nodepool was gracefully winding down, and only how has a chance to start deleting them19:21
fungier, only now19:21
*** dpyzhov has joined #openstack-infra19:23
*** sarob_ has joined #openstack-infra19:23
*** ryanpetrello has quit IRC19:25
fungithere's several hundred previously used nodes which are in the process of being deleted and replaced by building/ready ones19:27
*** sarob has quit IRC19:27
fungi(though those quickly switch from ready to in use as they get assigned jobs)19:27
*** sarob_ has quit IRC19:28
*** herndon_ has quit IRC19:28
*** reed has quit IRC19:28
flashgordonfungi: I am amazed this stuff ever works. at this scale things are hard. its pretty amazing19:29
* fungi couldn't agree more19:29
fungii'm keeping tabs on the nodes which were in used state prior to the nodepool restart and making sure they're dropping steadily, so it seems to be working out as intended19:30
*** sarob has joined #openstack-infra19:32
fungihopefully with twice as many workers back into the pool, we'll burn through what's in the gate a bit faster19:32
sdaguefungi: yeh, we all seriously owe you beers for holding down the fort by yourself this week19:32
fungisdague: nah, it's always someone's turn to shore up the timbers when everyone else needs to travel19:32
openstackgerritSergey Lukjanov proposed a change to openstack-infra/zuul: Collect and report last reconfigured timestamp
fungii'm banking up karma for when i'm going to be on a two-week wedding travel hiatus during the release. everyone will hate me then, so this works on balance19:33
*** dizquierdo has quit IRC19:33
openstackgerritSergey Lukjanov proposed a change to openstack-infra/zuul: Add zuul_version to status.json samples
*** dizquierdo has joined #openstack-infra19:35
*** mrodden has quit IRC19:38
*** mrodden has joined #openstack-infra19:39
*** praneshp has joined #openstack-infra19:41
*** denis_makogon_ has joined #openstack-infra19:43
*** denis_makogon_ is now known as denis_makogon19:45
*** dims is now known as dimsum19:45
openstackgerritSamuel Merritt proposed a change to openstack-dev/hacking: Fix typo in package summary
*** thedodd has quit IRC19:49
*** dpyzhov_ has joined #openstack-infra19:49
*** dpyzhov has quit IRC19:49
*** dpyzhov_ is now known as dpyzhov19:49
*** yolanda has quit IRC19:49
openstackgerritSergey Lukjanov proposed a change to openstack-infra/zuul: Add zuul_version to status.json samples
*** ruhe is now known as _ruhe19:51
*** thuc has quit IRC19:53
*** thuc has joined #openstack-infra19:53
*** ryanpetrello has joined #openstack-infra19:56
*** ryanpetrello has quit IRC19:57
sdaguefungi: how long after a pypi land will we get things in our mirror?19:57
*** thuc has quit IRC19:58
*** rfolco has quit IRC19:59
*** thuc has joined #openstack-infra19:59
*** _david_ has joined #openstack-infra19:59
*** vipuls is now known as vipuls-away20:00
*** vipuls-away is now known as vipuls20:00
fungisdague: by pypi land, you mean we uploaded a release to pypi or one of our requirements uploaded a new release?20:00
*** rockyg has joined #openstack-infra20:00
*** dizquierdo has quit IRC20:01
openstackgerritSean Dague proposed a change to openstack-infra/config: make a dedicated page for gate status
sdaguefungi: as in taskflow just hit pypi20:03
*** ^d has quit IRC20:03
sdaguehow long before a requirements recheck will pull that version?20:03
*** ^d has joined #openstack-infra20:03
fungii'll have to see if they're configured to trigger a mirror refresh on release... checking20:04
fungisdague: note that they don't have their project configured to run mirror jobs after a pypi upload in the release pipeline20:05
sdagueharlowja: can you look into doing that ^^20:06
*** mrmartin has joined #openstack-infra20:06
fungisdague: so it will be the daily (which i think hits at 0500 utc?) or the next time someone else tags a release or merges an openstack/requirements change, whichever comes first20:06
harlowjafungi just 'post-mirror-python26' and such?20:06
fungiharlowja: yup, there are adjacent examples in that file20:07
*** mozawa has quit IRC20:07
openstackgerritSean Dague proposed a change to openstack-infra/config: add in the optional ; everywhere
*** ryanpetrello has joined #openstack-infra20:07
*** lcestari has quit IRC20:07
*** dpyzhov has quit IRC20:09
sdaguehaving that time in gate right on the UI exposes some really interesting things20:11
openstackgerritJoshua Harlow proposed a change to openstack-infra/config: Ensure taskflow upload affects the mirrors
sdaguelike the heat change that's top of the check queue that entered almost 4 hrs ago, and is still waiting on getting a devstack node for pg20:11
*** Ryan_Lane2 has joined #openstack-infra20:11
sdaguebasically it looks like the top 5 check queue items are lost20:12
fungisdague: yeah, there are times when jenkins fails to run a job for whatever reason and zuul readds that to the work queue, but it's waiting in line for things which came in after zuul realized that20:12
openstackgerritA change was merged to openstack-dev/hacking: Fix typo in package summary
fungier, which came in before zuul realized that20:13
openstackgerritAndreas Jaeger proposed a change to openstack/requirements: Add openstack-doc-tools
*** wenlock_ has joined #openstack-infra20:13
openstackgerritSergey Lukjanov proposed a change to openstack-infra/zuul: Display zuul_version in zuul status page
openstackgerritSergey Lukjanov proposed a change to openstack-infra/zuul: Display last_reconfigured in zuul status page
fungibut the first ~ 50% of the gate pipeline has workers assigned now20:14
*** dpyzhov has joined #openstack-infra20:14
sdagueyeh, it looks like under current load it's ~2hrs until you get a dsvm node20:14
sdaguefungi: are we still bursting?20:14
*** vipuls is now known as vipuls-away20:15
fungiwe're maxing out the ~500 vm aggregate quota we've set across the various providers/zones20:15
*** herndon_ has joined #openstack-infra20:15
sdagueok, so now it's straight up load20:15
fungiactually, looks like the aggregate quota is closer to 45020:16
*** wenlock_ has quit IRC20:17
dimsumguess we'll need the weekend to clear the queue?20:19
flashgordondimsum: like the nick20:19
dimsumflashgordon, thanks! flashgordon brings back memories too...20:20
fungii've got one hpcloud vm which is gumming up the works. nodepool delete times out on it, nova list says it's in deleting state and has been for ages, i was able to ssh into it so i did a sudo poweroff and it shut down, but nova still things it's trying to delete. weird20:20
*** Ajaeger has joined #openstack-infra20:20
*** praneshp has quit IRC20:21
*** praneshp has joined #openstack-infra20:22
flashgordonlooks like we found a nova bug :)20:22
*** Ryan_Lane2 has quit IRC20:22
*** sarob has quit IRC20:23
*** sarob has joined #openstack-infra20:23
*** gokrokve has quit IRC20:25
Ajaegerflashgordon: found - or fixed? ;)20:25
flashgordonAjaeger: found a bug in hpclouds nova20:26
AjaegerOoops ;(20:26
*** hashar has joined #openstack-infra20:26
*** sarob has quit IRC20:28
SergeyLukjanovsdague, here is your change about adding time in queue live -
openstackgerritSean Dague proposed a change to openstack-infra/config: make merge conflict changes black
fungiflashgordon: yeah, it's been in a deleting state for 38 days according to the uptime command i ran before shutting it down to see whether that would help20:30
openstackgerritSean Dague proposed a change to openstack-infra/config: clean up possible js incompatibilities
sdagueSergeyLukjanov: cool20:31
*** prad has quit IRC20:32
flashgordonfungi: if only we new people at HP20:32
fungisdague: btw, that heat change finally got a worker for its one missing job run20:32
sdaguefungi: cool20:32
fungiflashgordon: well, people at hp who weren't about to head out for friday afternoon beer20:32
*** dkliban is now known as dkliban_afk20:33
flashgordonwe are all on the west coast now I think20:33
fungiyeah, it's after noon there, right? ;)20:33
sdagueflashgordon: so I don't think I'll be able to get enough mental energy to deal with the grenade patches today. Lets just set asside a couple hours in montreal and run through them side by side20:33
sdagueflashgordon: I thought HP just moved all your offices to delta terminals?20:34
fungisdague: budget cuts... greyhound terminals20:34
flashgordonsdague: works for me20:35
flashgordonbtw the grapite server seems to be failing to serve images20:36
*** changbl has joined #openstack-infra20:37
flashgordonnot to hammer the fungi too much20:37
fungiflashgordon: sdague:
fungilooks like we give it quite the workout20:38
marundoes testr run tests in parallel using multiple processes, or...?20:39
fungimight be i/o-bound...
flashgordonmarun: multi process20:40
sdagueSergeyLukjanov: the change I stuck on top of that also color codes out the merge conflicts as black20:40
flashgordonbut if lifeless is around he will correct me20:40
marunflashgordon: danke20:40
sdagueso you can pick them out easier from straight fails20:41
sdaguemarun: it's multi process20:41
*** thedodd has joined #openstack-infra20:41
marunsdague: i figured as much, but wanted to be sure.  someone was worried about runtime configuration changes affecting other tests but that's impossible afaik20:41
marun(so long as teardown calls reset())20:41
SergeyLukjanovsdague, I've already +1'ed it :) live
flashgordonmarun: as long as those are running in seperate threads20:42
sdaguemarun: as long as you aren't using filesystem shared state20:42
sdagueor ports or something20:42
flashgordonso concurrency=4 is 4 threads20:42
flashgordonerr processes20:42
sdagueflashgordon: 4 processes20:42
sdaguemarun: in the cases where we know we might have a global resource in tempest, we use the oslo lockutils20:43
sdaguewith external=True20:43
sdaguelike doing agregates manipulation20:43
marunsdague: external=True?  presumably that uses file-based locking to ensure cross-process concurrency, but what does external=False do?20:43
flashgordonmarun: correct file-based20:44
marunconcurrency => synchronization20:44
sdaguemarun: external=False (which is the default) will do a lock which is safe for multi threaded environments, but not multi process20:44
*** Ryan_Lane2 has joined #openstack-infra20:44
marunsdague: ah, good to know.20:44
fungiso graphite... 4x vcpu, one of which is entirely pegged on iowait20:45
*** Ryan_Lane2 has quit IRC20:47
*** sarob has joined #openstack-infra20:49
harlowjamordred yt (question about testr +
*** sarob_ has joined #openstack-infra20:51
harlowjamaybe someone else knows also20:51
*** dprince has quit IRC20:52
fungiharlowja: quite probably. may not be me, but you never know unless you ask20:52
harlowjaok, probably simple, so is it possible to get the colorized output via running python with a testr enabled project20:53
harlowjalike python --verbose test --no-parallel doesn't seem to  do it20:53
harlowja*maybe its a feature which doesn't exist yet20:53
fungiharlowja: i think nova has a color filter which you can feed the subunit stream through20:53
harlowjaright, any idea if that can just be integrated into testr directly?20:54
*** dpyzhov has quit IRC20:54
harlowjaso it could be like python --verbose test --no-parallel --colorized20:54
fungiharlowja: for the most part, sitting and staring at the process of test running is not part of the intended design of testr20:54
harlowjadurn it, haha20:54
*** sarob has quit IRC20:54
harlowjathats why it could be an option :-P20:54
fungiit's geared toward running all or specific sets of tests, evaluating the results, and then has a robust set of options to rerun tests based on the results of previous runs20:54
fungiso you can, for example, tell it to just re-run failing tests20:55
*** sandywalsh has quit IRC20:55
fungiuseful for tight iteration when debugging your code20:55
*** ^d is now known as ^demon|away20:56
sdaguefungi: well given that testr regex defaults to or, you can't do that via tox in a lot of projects20:56
fungisdague: true20:57
fungiyou'd have to activate your venv and call testr directly20:57
sdaguefungi: right20:57
sdagueat which point I want to start stabbing people20:57
sdagueand bring back run_tests.sh20:57
sdagueharlowja: there are some wrappers in tempest as well20:58
sdagueharlowja: so I think they should be fixed in testr, however testr is currently in a bzr tree, and I don't have patience for that. So I'll probably start a stackforge project with a common set of testr wrappers21:00
flashgordonfungi: so libvirt21:01
flashgordonwe have a patch to make nova new libvirt friendly21:01
sdagueflashgordon: did you test it on a cloud archive enabled box yet?21:02
flashgordonsdague: not yet, trying to get someone who has one set up to do that21:02
fungii was about to ask exactly that21:02
sdaguebecause while we *know* that was a fail21:02
sdaguewe don't know it's the only one21:02
sdagueflashgordon: don't you work for a company with a public cloud... ;)21:02
flashgordondimsum: ^21:03
*** prad has joined #openstack-infra21:03
fungisdague: well, both nova unit tests and tempest fail due to what seems like the new libvirt in uca21:03
flashgordonsdague: I don't know what your talking abuot ;)21:03
fungiso either would be a good test21:03
fungiboth would be better21:03
sdagueactually, I did set up a cloud archive rax guest21:03
sdaguelast night21:03
flashgordonsdague: awesome thanks21:03
fungii have scripts to autobuild one and a stub which will add uca before running tempest, if it becomes necessary21:04
sdagueyeh, let me at least run the unit tests21:04
fungibut my bandwidth is pretty low right now, so i probably wouldn't get to it right away21:04
*** reed has joined #openstack-infra21:05
fungiso, on graphite, the iowait was almost entirely the kjournald thread according to iotop. on a whim i remounted / with relatime and now the iowait is gone, and i can see that same cpu is pegged with user cycles instead, running carbon-cache.py21:06
flashgordonsdague: what version of cloud archive?21:06
*** smarcet has left #openstack-infra21:06
flashgordonsudo add-apt-repository cloud-archive:havana21:06
sdagueflashgordon: havana21:06
sdagueon 12.0421:06
*** sandywalsh has joined #openstack-infra21:06
fungiflashgordon: i was doing this before installing other packages on the vm:
fungithat should be all you need to activate it21:08
flashgordonfungi: cool thanks21:08
*** vipuls-away is now known as vipuls21:08
* flashgordon uses his free cloud resources 21:08
fungiadding that would cause my tempest runs to consistently fail, and i attached nova logs to the bug too if they're of interest21:08
funginova and libvirtd logs attached to
fungiit very well may be that the libvirt hangs were masking an incompatibility in nova, and once fixed that's where we ended up21:11
*** Ryan_Lane2 has joined #openstack-infra21:12
_david_zaro, fixed :
flashgordonyeah there are two nova patches up to for new libvirt21:12
fungialso, the change at the head of the gate just timed out on a tempest-postgres job, resetting all ~100 changes... *sigh*21:13
*** sarob_ has quit IRC21:15
flashgordonwas that a rax or hp server21:15
fungiflashgordon: i believe i reproduced the issue on both21:15
*** sarob has joined #openstack-infra21:15
fungifor tempest runs anyway21:15
flashgordonfungi: neato21:16
flashgordonso the question is21:16
flashgordondo those two fix it21:17
fungiin combination, or is this an either/or fix?21:17
*** DennyZhang has joined #openstack-infra21:18
flashgordonin combination21:18
*** sarob has quit IRC21:19
flashgordonfungi: I can run with the libvirt stuff if you want21:19
*** sarob_ has joined #openstack-infra21:19
*** Ryan_Lane2 has quit IRC21:19
*** Ryan_Lane2 has joined #openstack-infra21:20
fungiflashgordon: please do. i only originally weighed in on this bug because someone wanted to re-add uca to our slaves after it broke everything last time, and i wanted to make sure it didn't merge until we had confidence that wasn't going to happen again21:20
*** Ajaeger has quit IRC21:20
flashgordonyup thats why I am manaully checking21:20
fungibasically i asked on the review whether it had been tested, and there was silence, but it kept accumulating positive votes, and i was worried21:21
flashgordonfungi: which review?21:21
fungiit's probably been abandoned since november, don't exactly remember21:23
flashgordonanyway i'll run with this, you have enough on your plate21:23
fungibasically i'm blocking any proposals to add uca to our slaves again until someone confirms nova unit tests and tempest work21:23
*** CaptTofu has quit IRC21:24
flashgordonfungi: that is exactly what I intend to do21:24
*** CaptTofu has joined #openstack-infra21:24
fungifor the tempest test, i did but with that snipped i pasted you earlier before installing git21:25
arosenHi, I'm running into this error today trying to install nova's test-requirements.txt i was wondering if anyone has hit this yet:
fungiarosen: looks like maybe pylint released a broken package?21:27
fungi"unknown encoding: IBO-8859-1"21:27
*** mgagne1 has quit IRC21:27
fungiyeah, i don't know that encoding either21:27
fungii assume someone fat-fingered ISO-8859-121:28
arosenyea, i saw that.. I guess the gate isn't hitting it because it hasn't pulled in the latest pylint and it's using a cached copy?21:28
*** SergeyLukjanov has quit IRC21:29
fungiquite possible--we may want to emergency cap it. looking to see if 0.25.0 is shiny-new21:29
arosenHrm nova specifies pylint==0.25.221:29
fungiyeah, current pylint is weeks old 1.1.021:29
fungimaybe a new release of something else specified a broken pylint as a requirement21:30
fungisitepackages=true there21:30
fungi"Found existing installation: pylint 0.25.0"21:30
fungisdague: example of what i was talking about the other day ^21:31
sdagueit is another pip frustration21:31
funginova says it requires pylint==0.25.2 but the local 0.25.0 on the system gets used instead21:31
fungiarosen: try uninstalling your system package of pylint21:32
arosenIf i remove pylint 0.25.0 i get this error:
sdagueit completely sucks that pip under tox can't give you the right thing21:32
*** starmer has joined #openstack-infra21:33
zaro_david_: cool. thanks.  i'll give a try21:33
fungiarosen: i'd suggest `fing /usr -name "*pylint*"` and see if there's bits left behind21:34
fungier, find21:34
_david_zaro, Fixing gerrit-javamelody plugin now:
*** starmer has quit IRC21:34
fungiarosen: because you're installing pylint into a virtualenv and it's complaining about pylint imports it found in /usr/local21:34
fungiarosen: oh, wait, you're not21:35
fungiyou're installing them system-wide with pip instead21:35
*** vipuls is now known as vipuls-away21:36
*** vipuls-away is now known as vipuls21:36
fungiarosen: so i guess pip uninstall pylint and then use find to go on a hunting expedition for pieces which might have been left behind21:36
*** Ryan_Lane2 has left #openstack-infra21:36
fungithough i think the "package init file './test/' not found (or not a regular file)" is probably the real problem21:38
arosenfungi:  yup same thing: see line 1 shows i have no more pylint21:38
fungioh, or maybe that's just a stray, benign message21:39
arosenfungi:  the problem is it causes testtools not to be installed.21:40
fungiit's complaining about syntax errors in /usr/local/lib/python2.7/dist-packages/pylint/test/input/ which was not there before you started to install according to your find comment21:40
*** ryanpetrello has quit IRC21:40
fungiso either that file is from a corrupt archive, or your interpreter is wrong21:41
*** ryanpetrello has joined #openstack-infra21:41
fungiwhen it's done it claims "Successfully installed pylint testrepository testtools" though21:42
fungiso maybe the syntax errors in the pylint test are really there upstream in that version, and they just get ignored (but complain loudly at you anyway)?21:42
arosenfungi:  thanks i fixed it. There was actually a bug in the code //usr/local/lib/python2.7/dist-packages/pylint/test/input/ once i fixed the that it worked.... Very odd.21:45
fungiyeah, that's a little strange21:46
*** dstanek has joined #openstack-infra21:46
*** DennyZhang has quit IRC21:46
*** DennyZhang has joined #openstack-infra21:47
fungimaybe you want to check the tarball against the checksum on pypi to make sure it's not been deviled with21:47
fungi says pylint-0.25.2.tar.gz should have an md5sum of d878d7688a4f5290dc5b53a83687240021:49
fungiyou should probably check your pip cache to see if you get the same21:49
* fungi gets weirdly paranoid about things like that21:50
*** vipuls is now known as vipuls-away21:51
arosenfungi: Hrm i'm not finding the pip cache where is that?21:51
fungiarosen: i'm guessing /root/.pip/cache/ but if you don't have one, then pip's probably not caching downloads21:53
fungicould also be some artifacts in /tmp/pip-build-root21:54
arosenroot@arosen-desktop:~/.pip_download_cache# md5sum
arosenYea it matches the hash21:54
fungiaha. and okay, good21:55
fungiso it's probably just a bug in 0.25.2 which is spewing nonsense during install but gets ignored by pip21:55
arosenHad to remove it and set export PIP_DOWNLOAD_CACHE . Gets me into the same situation where i need to edit /usr/local/lib/python2.7/dist-packages/pylint/test/input/func_w0705.py21:55
fungithe errors all seemed to be in unit tests for pylint, not in actual run-time code21:55
fungiso i'm guessing pip is running the pylint's provided test suite, which is broken, but pip could care less anyway21:56
fungior rather the for pylint is running the test suite, after being invoked by pip, but you get the idea21:57
fungiand probably spews test errors on stdout/stderr but still exits 021:57
*** CaptTofu has quit IRC21:58
*** gokrokve has joined #openstack-infra21:58
*** CaptTofu has joined #openstack-infra21:58
*** che-arne has quit IRC21:59
*** jasondotstar has quit IRC21:59
*** eharney has quit IRC22:03
*** mrmartin has quit IRC22:07
fungiwe're using 83% of the 315gb root partition on graphite. checking to see why/where22:08
*** DennyZhang has quit IRC22:10
*** iccha has joined #openstack-infra22:12
*** sarob_ is now known as sarob22:12
*** pballand has joined #openstack-infra22:14
*** dimsum has quit IRC22:15
pballandafter starting from a cookie cutter project, how does one configure the project to copy appropriate source to .tox (running unit tests via 'tox -epy27' ?22:15
*** jgrimm has quit IRC22:16
sdaguefungi: so there are a ton of pg jobs that are getting killed because of timeouts22:16
sdaguewhen they were about 2minutes from success22:16
sdaguethat's what reset the whole queue just now22:16
sdaguebut you can see a bunch of it in the check queue as well22:17
pballandnever mind - it must have been something in my environment; please disregard my previous question22:17
fungipballand: disregarded!22:17
fungimost of them happening on rax? we probably just need to bite the bullet and up the timeouts there unless someone can make the job run faster?22:17
fungisdague: ^22:18
flashgordoncan't we yell at rax a bit for fun too?22:18
sdagueno it's all hp nodes22:18
flashgordonsdague: interesting22:19
sdaguewe need to stop blaming rax, hp nodes are taking 50 minutes to run a tempest run today22:19
* flashgordon looks at the slowest tempest test list22:19
*** burt has quit IRC22:19
fungisdague: any particular az?22:20
sdaguefungi: no, it's be on a bunch of them22:21
*** mfer has quit IRC22:21
fungihuh. that's... stellar22:21
flashgordon2014-01-10 22:13:36.408 | tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario[compute,image,network,volume]  265.92522:21
fungistill, my question stands. do we up the timeout, or is this a legit issue?22:21
fungi(legit issue with the tests/job)22:22
fungiguess that can also be all of the above22:22
*** alexpilotti has joined #openstack-infra22:22
flashgordonmake tempest faster list
fungiit's possible the runtime on tempest has just been slowly creeping up with new jobs, and we're finally bumping up against the ceiling there?22:23
flashgordonfungi: jenkins plots that right?22:23
*** hashar has left #openstack-infra22:23
fungiflashgordon: not long-term (we run too many jobs, so i think we only keep a day history)22:24
fungii'm double-checking though22:24
flashgordonjenkins data is too dirty22:25
flashgordondo we graphite this?22:25
flashgordonor logstash22:25
* flashgordon checks logstash22:25
fungioh, also jenkins trending is worthless because of the way we're constantly aborting/cancelling jobs in the gate22:25
openstackgerritAaron Greengrass proposed a change to openstack-infra/config: Remove hardcoded config assumptions, cleanup variables
fungigraphite's probably a slow angle of investigation at the moment given the load/iowait there, but i also think we maybe haven't been trending job duration yet (i could be wrong)22:26
flashgordonmessage:"tests in" AND message:"Ran" AND build_name:"check-tempest-dsvm-postgres-full"22:26
flashgordonspot check is showing things are worse22:28
sdaguefungi: if you filter jenkins by only success runs, it's ok22:28
sdaguethat's what my scraper used to do22:28
sdaguebut with this many aborts in there, it's a little tougher to get good data22:29
flashgordonmessage:"tests in" AND message:"Ran" AND build_name:"check-tempest-dsvm-postgres-full" AND build_status:"SUCCESS"22:29
flashgordonworks pretty well22:29
flashgordona lot of jobs around 2000 seconds22:29
fungiwhich is still well under the timeout22:30
Alex_Gaynormordred: Where do I file bug reports against pbr? Specifically if try ot install a package that depends on pbr, and you don't have git installed you get a horrendous error message "file does not exist"22:30
flashgordonfungi: thats just tempest not setup22:30
fungiyeah then it's probably very close22:30
*** katyafervent has quit IRC22:30
*** katyafervent has joined #openstack-infra22:31
fungiAlex_Gaynor: that sounds familiar--i thought it was fixed, maybe just not yet released. anyway
flashgordon2231 tests today22:31
flashgordon2193 around new years22:31
*** jerryz has quit IRC22:32
fungiAlex_Gaynor: anyway, what are you doing with a machine with no git installed? don't they just embed it in the kernel these days ;)22:32
*** dimsum has joined #openstack-infra22:32
Alex_Gaynorfungi: Totally empty docker image22:33
flashgordonnot a big bump in tests22:33
fungiflashgordon: so maybe openstack got slower22:34
fungiflashgordon: or maybe openstack-as-run-by-one-of-our-providers got slower22:34
flashgordonor something in devstack got slower22:35
flashgordonwhy is this seen more in postgres?22:35
flashgordonsdague: whats your take on bumping the timeout?22:36
sdagueflashgordon: +1 on bumping timeouts22:36
flashgordonit may be a decent short term solution22:36
flashgordonsdague: by how much 5 minutes?22:37
sdagueremember though that you need to handle it in the inner script as well22:37
sdagueif you are going to bump them, add 30 minutes22:37
* flashgordon looks at fungi22:37
flashgordonwe should also look into getting these tests faster
fungiflashgordon: i can bump the timeout(s both places)22:38
fungii'll set them to 95 outer, 90 inner i guess22:38
fungijust postgres or are we flying into the danger zone anywhere else obvious?22:39
flashgordonsdague: oh watcher of zuul22:40
*** vipuls-away is now known as vipuls22:40
*** dangers is now known as dangers_away22:41
sdagueflashgordon: well, some of those take real time22:42
flashgordonsdague: agreed, but how much time22:42
sdaguethat amount of time22:42
*** thomasem has quit IRC22:43
sdaguegrrr... I've been waiting around for 15 minutes now to get a 10th jenkins gerrit event22:43
sdagueflashgordon: after a week of frustration, I'm getting all the fundamentals addressed in ER, like actually having frameworks for spitting up fake, but real looking, gerrit events, so we can actually test everything as unit tests in ER22:45
sdagueso I can get this passing -
flashgordonsdague: sweet22:46
flashgordonwe can have a review party next week and get that all in22:46
sdaguesounds good22:47
*** oubiwann has quit IRC22:47
sdaguethat will let us really easily add more jobtypes22:47
sdagueactually, it defaults to working on all the job types, we can add the readiness files to make them less likely to miss22:48
fungiokay, so bump postgres-full to 95(90) and... do we care about postgres (non-full) timeout?22:48
sdagueI think it's functionally correct, but our unit tests are such a wreck that they require uber hackiness22:48
sdaguefungi: I don't think so22:49
fungik, patch inbound22:49
*** prad has quit IRC22:49
sdaguealso, glanceclient is failing unit tests a lot22:49
flashgordonsdague: I noticed that lolz22:49
flashgordonthats embarrassing for them22:50
sdagueflashgordon: I don't think they know22:50
flashgordonthats even worse22:52
pballandare 3rd party packages cached somewhere?  (How do the build system/developers deal with 3rd-party packages in pypi where the server is down?)22:53
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Increase gate tempest-dsvm-postgres-full timeout
*** prad has joined #openstack-infra22:53
fungisdague: flashgordon: if you'll both +1 ^ i'll self-approve it in22:53
*** markmcclain has quit IRC22:53
lifelessflashgordon: hi?22:54
fungipballand: it depends on the project. for openstack official project jobs, we force them to use a limited-content pypi mirror we maintain with a set of automated jobs. for stackforge projects and some leaf infra/dev projects we just hit pypi directly and deal with the false negative results when it goes down22:54
*** alexpilotti has quit IRC22:55
flashgordonlifeless: o/ I used your name in vane with regard to a question about testr22:55
sdaguefungi: done22:55
fungivein too ;)22:55
flashgordonfungi: done22:55
openstackgerritDevananda van der Veen proposed a change to openstack/requirements: Bump version of sphinxcontrib-pecanwsme
sdaguefungi: it would be nice if we could automate that inner / outer part22:56
fungicomplicity confirmed22:56
pballandfungi: that makes sense.  Any recommendations for a stackforge project that relies on a 3rd party package from a flaky server?  (is it acceptable to cache the package in the project?)22:56
sdagueany chance the yaml values could be sent down automatically?22:56
fungisdague: agreed, it could probably be done with a parameter and some shell math or pipe through bc if you want to get old-school22:56
flashgordonso gate delay is 16 hours not too bad22:57
anteayaflashgordon: you were correct the first time, vane22:57
anteayaI'm wrong, vain22:57
flashgordonI just spent the last 5 minutes googling that too22:58
openstackgerritDevananda van der Veen proposed a change to openstack/requirements: Bump version of sphinxcontrib-pecanwsme
*** madmike has quit IRC22:58
fungipballand: well, pypi has gotten much more reliable now that they've put their service behind a cdn. what package? convince them to start uploading directly to pypi if they don't (otherwise pip 1.5 will also need scary-looking options to install them anyway)22:59
pballandfungi: the package is antlr3:
*** CaptTofu has quit IRC23:01
fungisdague: for bonus points, in devstack-gate we could find a way to check the job start time against when the timeout command gets invoked and take the difference into account, so we only need fudge time for file uploads at the end and and whatnot23:01
fungipballand: yep...
fungipballand: talk them into uploading to pypi23:02
anteayaflashgordon: :D23:02
fungipballand: right now, with latest pip, you'd have to add --allow-external antlr_python_runtime --allow-unverified antlr_python_runtime23:02
fungipballand: basically, packages which can only be installed through external pypi linking need to get publicly shamed into uploading their tarballs. if some of the pip/pypi maintainers had their way, that would already be disallowed23:04
*** thuc has quit IRC23:04
*** thuc has joined #openstack-infra23:05
*** CaptTofu has joined #openstack-infra23:06
*** banix has quit IRC23:08
*** thuc has quit IRC23:09
fungipballand: and it's not just for the sake of stability. most of those externally-linked packages are just served up via plain http, so no endpoint validation to even confirm you're not the victim of a man-in-the-middle attack feeding you a compromised piece of software23:10
pballandfungi: okay, there are already concerns regarding the stability of that package - there is a history of introducing bugs in new versions, so perhaps caching the exact version of the code is the best way forward for now23:11
fungipballand: or switching to an alternative implementation if there is one23:11
*** jerryz has joined #openstack-infra23:12
*** _david_ has quit IRC23:12
fungipballand: or going rogue and uploading a slightly renamed fork of it to pypi, then depending on that, if they're stubborn and you're so inclined23:12
pballandfungi: :-)23:13
fungipballand: SpamapS recently (a couple days ago) had luck talking the netaddr devs into uploading their releases to pypi, so it's not necessarily a futile endeavor. some upstreams are just unaware of what's going on in the present python packaging revolution23:14
*** pcrews has quit IRC23:15
pballandfungi: I'll give it a shot; nice to know the alternatives if it doesn't work out23:16
*** GheRivero_afk is now known as GheRivero23:16
*** rcleere has quit IRC23:16
lifelessflashgordon: ah, so yes you are correct23:16
pballandfungi: first, I need to wait for the site to come back online to find the author :-/23:18
fungipballand: yeah, that sucks. maybe if you mention to them that had they uploaded copies directly to pypi like most projects, you could have continued to install their software while they were busy dealing with (hard drive crash|power outage|law enforcement)23:22
*** herndon_ has quit IRC23:22
*** rnirmal has quit IRC23:22
flashgordonfungi: so with libvirt 1.1.1 I couldn't get nova-cpu to crash23:23
*** thedodd has quit IRC23:24
openstackgerritA change was merged to openstack-infra/config: Increase gate tempest-dsvm-postgres-full timeout
fungiflashgordon: was that with or without those patches?23:28
david-lyleI'm trying to upload a new version of django_openstack_auth to pypi, can someone add me as a maintainer?  I've pushed a tag on the gerrit side, now I need to publish.  current maintainers are gabrielhurley and openstackci23:28
fungidavid-lyle: jenkins should have uploaded it for you automatically. i'll look into why it didn't23:29
*** ryanpetrello has quit IRC23:30
david-lylefungi: thanks, didn't realize that was automated23:31
fungidavid-lyle: 1.1.4 right?23:31
fungilooks properly signed23:31
fungii don't see the tarball at
fungichecking into the tarball job23:32
david-lyleI may have confirmed my key after pushing the tag, don't know if that would cause issues23:32
*** GheRivero is now known as GheRivero_afk23:33
*** CaptTofu has quit IRC23:34
*** pballand has quit IRC23:34
fungidavid-lyle: it's probably just zuul being backed up, since its in the throes of a fairly large gate reset right now and i don't see any signs the job got triggered on any of our jenkins masters yet. i'll check the zuul log to make sure it's getting enqueued23:36
david-lylefungi: ok, I thought I would have to push, so was looking for a way to do that, but I understand zuul is loaded right now, wasn't trying to be impatient23:37
*** bauzas has quit IRC23:37
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: wip: moving readiness checks into stream
fungidavid-lyle: no problem at all, i want to make sure the automation is actually working as designed (but it seems to have successfully uploaded the previous releases, so i expect it's probably okay)23:38
sdaguewell the check queue is basically empty, that's good to no longer have things there for hours23:39
*** rockyg has quit IRC23:41
*** vipuls is now known as vipuls-away23:44
*** UtahDave has quit IRC23:47
jeblairfungi: what happened to nodepool?23:48
*** banix has joined #openstack-infra23:49
*** mrodden has quit IRC23:49
sdaguefungi: btw - I updated the indentation on
fungijeblair: roughly 50% of the aggregate quota were between 8 hours and 4 days old23:52
jeblairfungi: ready or used?23:52
fungijeblair: one vm was undeletable, in hpcloud, stuck in deleting state according to novaclient (for over a month since i logged into it and uptime said 38 days before i shut it down)23:52
fungijeblair: a mix23:53
fungisome claimed to be in ready, some delete, some used23:53
*** sarob has quit IRC23:53
*** sarob has joined #openstack-infra23:54
fungii did a graceful shutdown and deleted all the nodes which were more than a few hours old since they clearly weren't picking up any jobs. that put the gate behind a couple hours all total23:54
fungithough the graceful shutdown never completed. i nodepool deleted all the non-used nodes at the end, except for the one which would never delete, then killed nodepoold and cleared the leftover pidfile23:55
fungithe undeletable hpcloud slave is still there... devstack-precise-hpcloud-az2-77737123:56
jeblairfungi: you cane run 'nodepool delete' on old nodes without stopping nodepool in the future23:56
fungijeblair: right, i'm aware. i was worried though since it wasn't deleting them on its own, dating back to around the jenkins master restarts23:57
fungiit was deleting some, but had a bunch which the cleanup thread should have taken care of23:57
fungiso i didn't want them piling right back up again23:57
*** sarob has quit IRC23:57
*** wenlock has quit IRC23:58
fungithough the graph leading up to when i started working on it shows a marked decline in capacity (slow trending increase in both building and deleting count, leaving a narrower and narrower envelope of in use nodes)23:59

Generated by 2.14.0 by Marius Gedminas - find it at!