Thursday, 2013-08-01

bodepdmordred: did you know there is a jjb module on the forge?00:08
bodepdcrazy :)00:08
*** gyee has quit IRC00:09
mordredoh! that's pabelanger00:09
bodepdwho is that?00:10
bodepdthe name os familiar00:10
bodepdalso, the plugin code I used is blatently stolen from here:00:11
mordredthere is a puppet-jenkins module run by the jenkinsci people?00:12
bodepdduh :)00:12
pabelangermordred, bodepd yup00:13
bodepdI borrowed some bits from that, and some from your guys for my franken-fork00:13
mordredbodepd: nice00:13
mordredso -00:13
*** bnemec has quit IRC00:15
*** bnemec has joined #openstack-infra00:15
*** michchap has quit IRC00:15
*** michchap has joined #openstack-infra00:16
bodepdlooks like I got everything installed, but I'm not convined it works :)00:20
*** ladquin has joined #openstack-infra00:21
melwittis this the right channel to talk about devstack?00:23
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Further tuning of lists
jeblairi'm going to stop puppet on lists and apply those changes manually00:24
*** sdake_ has quit IRC00:25
clarkbmelwitt: depends00:27
clarkbmelwitt: if you are interested in how we use devstack to test and gate the projects then yes00:28
clarkbmelwitt: but if you are wondering why $feature isn't included or something along those lines you will probably have better luck asking dtroyer directly00:28
clarkbmelwitt: and if you aren't sure feel free to ask. we tend to not bite00:29
jeblairlists cpu usage increase with the openstack list is dramatic.00:30
clarkbthat is cool00:30
*** andrew_plunk has joined #openstack-infra00:30
pabelangerjeblair, is password protected?00:31
openstackgerritA change was merged to openstack-dev/pbr: Add skip_pip_install to setup.cfg
jeblairpabelanger: not really; we just need to get around to adding a redirect to the root00:31
jeblairpabelanger: try that00:32
melwittclarkb: thanks, makes sense. I bumped into something that didn't work on rhel and commenting out one line in made it work. I'll find their launchpad or list where I can see if it can help others00:32
pabelangerjeblair, cool, that works00:32
*** cybertron_ has joined #openstack-infra00:33
*** bnemec has quit IRC00:33
clarkbjeblair: the tuning lgtm, I did have one question inline though00:33
*** adalbas has joined #openstack-infra00:33
jeblairclarkb: the answer is yes (i mentioned that in the commit msg)00:34
clarkbmelwitt: I would definitely submit a bug, should be the devstack project on launchpad00:34
clarkbjeblair: bah, you know that is one behavior of gerrit that drives me nuts. It always scrolls down to the files so you skip the commit message00:35
openstackgerritDan Bode proposed a change to openstack-infra/config: Ensure python pip is installed before zuul
melwittclarkb: k, will do. thanks00:35
clarkbjeblair: I just noticed recently that it does that every time I open a change00:35
*** UtahDave has quit IRC00:36
jeblairthe queue is currently 130313 and falling at a rate of about 500/min (which gives us a bit more than 4 hours to git it to a reasonable level)00:36
jeblair(though it's only up to 43 out of the 50 runners so far)00:36
openstackgerritDan Bode proposed a change to openstack-infra/config: Ensure /var/lib/zuul is owned by zuul
openstackgerritDan Bode proposed a change to openstack-infra/config: Update zuul to be compatible with the latest apache module
bodepdI'm not sure what to do about the last chnage00:42
bodepdI updated zuul to work with the latest version of apache00:42
bodepdbut that is just a small part of the change that needs to happen00:42
jeblairbumped the the number of queue runners to 7000:42
openstackgerritDan Bode proposed a change to openstack-infra/config: Add jenkins::plugin defined resource type
jeblairok, that was probably too much; load average is 4.5 now (on a 2vcpu machine)00:50
*** dina_belova has joined #openstack-infra00:50
bodepdthose are the changed I had to make today. feel free to ask questions00:51
jeblairotoh, it's now falling at 1000/min, so i'll leave it for now.00:55
*** dina_belova has quit IRC00:55
jeblairthe load avg is high, but the cpu stats are good00:56
*** changbl has joined #openstack-infra00:59
*** dkranz has joined #openstack-infra01:00
fungii've had responsive exim servers with load >50 and only 2x cpu. can actually be okay with sufficient ram and i/o bandwidth as long as that's the only thing the server needs to handle01:01
clarkbdoes each queue run in its own process? the new config stuff implies that is the case01:03
fungieach delivery gets farmed out to an available runner01:04
fungisimilar to how apache handles requests01:05
*** reed has quit IRC01:13
pabelangerjeblair, When you have time, can you do you want to register with, so we can do ISN dialing:
pabelangerjeblair, I can setup the asterisk side, but need valid ITAD info01:21
openstackgerritRussell Bryant proposed a change to openstack-infra/config: pbx: update SIP config to help deal with NAT issues
*** vipul is now known as vipul-away01:26
*** sandywalsh has quit IRC01:33
clarkbmordred: I noticed earlier today that you don't use gerrit email. I am going to give it a shot and see if it makes reading list mail easier01:38
mordredclarkb: going to give no gerrit mail a shot?01:39
mordredyeah - I pretty much just poll the UI all day long01:39
mordredthe biggest downside is the ui is bad at telling me that someone has made comments I should look at01:39
mordredor that someone has voted -1 on a change if someone else has voted positively01:39
mordredbut - I believe I still keep up with it better than I would with list mail01:40
mordredI also disabled all bug email01:40
mordredthat one I don't think is the right choice01:40
mordredand I need to undo01:40
clarkbya I tend to poll gerrit pretty frequently especially when I am in a review binge01:40
*** nibalizer has quit IRC01:43
*** pcrews has quit IRC01:43
*** bnemec has joined #openstack-infra01:45
*** michchap has quit IRC01:45
*** cybertron_ has quit IRC01:46
*** sandywalsh has joined #openstack-infra01:49
clarkbI think we broke the zuul jobs launched per hour01:50
clarkb*per hour graph on the zuul status page01:50
*** dina_belova has joined #openstack-infra01:52
fungiclarkb: the change to track job results via statsd requires an update to the zuul status page01:53
fungis/./_/ mostly, i think?01:54
clarkbfungi: it added the pipelines to the path too01:56
clarkbfungi: but I thought that required a zuul restart. is that why jeblair was rechecking jobs?01:56
*** dina_belova has quit IRC01:56
fungii probably flew under my radar while i was out at dinner, if so01:57
*** thomasm has joined #openstack-infra01:59
fungilast zuul restart was almost 20 hours ago01:59
*** melwitt has quit IRC02:03
*** mriedem has joined #openstack-infra02:05
openstackgerritA change was merged to openstack-infra/config: Add branch tarball job for tempest.
*** jrex_laptop has quit IRC02:11
*** michchap has joined #openstack-infra02:23
*** anteaya has quit IRC02:26
*** andrew_plunk has quit IRC02:26
*** HenryG has joined #openstack-infra02:34
*** dguitarbite has joined #openstack-infra02:34
*** vipul-away is now known as vipul02:35
*** HenryG has quit IRC02:42
*** erfanian has joined #openstack-infra02:43
lifelessjeblair: when you are around, can we talk about ? perhaps a voice call is needed to get more bandwidth on it?02:45
*** michchap has quit IRC02:46
*** SergeyLukjanov has joined #openstack-infra02:50
*** dina_belova has joined #openstack-infra02:51
SpamapSThe authenticity of host '[]:29418 ([]:29418)' can't be established.02:56
SpamapSRSA key fingerprint is ee:2f:ac:1b:f8:25:d0:39:be:55:02:c7:76:5e:39:53.02:56
SpamapSoh.. n/m .. my chroot doesn't have my ~/.ssh/known_hosts02:57
*** michchap has joined #openstack-infra02:57
mordredSpamapS: :)02:57
SpamapSNeed to figure out how to just do a git review -d in the local cached version of a repo02:58
*** yaguang has joined #openstack-infra02:58
SpamapS(guessing that is high on the list of things to make di-b useful for CI/CD anyway)02:58
*** michchap has quit IRC02:59
*** michchap has joined #openstack-infra03:00
*** mriedem has quit IRC03:00
mordredwell, not for ci03:03
mordredit's probably useful for dev workflow though03:03
*** jjmb has quit IRC03:04
lifelessSpamapS: uhm, WAT ?03:04
lifelessSpamapS: I think you're pushing on the wrong knob.03:04
*** SergeyLukjanov has quit IRC03:05
*** yaguang has quit IRC03:05
*** dina_belova has quit IRC03:15
*** yaguang_ has joined #openstack-infra03:19
*** UtahDave has joined #openstack-infra03:20
*** michchap has quit IRC03:23
*** michchap has joined #openstack-infra03:24
zaromordred, jeblair, clarkb, fungi: WIP vote patches have been submitted to upstream.03:28
zaro   &
fungizaro: awesome!03:30
fungii recall submitting patches to google's gerrit... their workflow is interesting, or at least was at the time03:31
zarofungi: how so?03:32
* fungi looks back at his notes03:32
*** michchap has quit IRC03:33
fungieh, mostly that they do push via https into gerrit, even though released gerrit didn't support that at the time (maybe it does now?)03:36
zaroi think so, at least the doc indicates so..
*** bnemec has quit IRC03:40
*** bnemec has joined #openstack-infra03:40
* fungi looks forward to yet another new option there!03:41
*** erfanian has quit IRC04:02
*** pcrews has joined #openstack-infra04:03
*** vogxn has joined #openstack-infra04:06
*** nati_ueno has quit IRC04:14
*** bnemec has quit IRC04:16
*** bnemec has joined #openstack-infra04:16
*** amotoki_ has joined #openstack-infra04:18
*** UtahDave has quit IRC04:19
*** amotoki has quit IRC04:19
*** nicedice has quit IRC04:23
*** dkliban_afk has quit IRC04:23
*** emagana has joined #openstack-infra04:28
*** afazekas has joined #openstack-infra04:34
*** markmcclain has quit IRC04:36
*** nayward has joined #openstack-infra04:41
*** nayward has quit IRC04:46
*** toddmorey has quit IRC04:52
*** rcleere has quit IRC04:54
*** yaguang_ has quit IRC05:00
*** toddmorey has joined #openstack-infra05:01
*** nayward has joined #openstack-infra05:12
*** dina_belova has joined #openstack-infra05:15
*** dina_belova has quit IRC05:16
*** toddmorey has quit IRC05:17
*** dina_belova has joined #openstack-infra05:25
*** rcleere has joined #openstack-infra05:33
*** koolhead11|away has joined #openstack-infra05:33
*** koolhead11|away is now known as koolhead1105:34
*** nayward has quit IRC05:37
*** nayward has joined #openstack-infra05:40
*** jhesketh has quit IRC05:41
*** jhesketh has joined #openstack-infra05:41
*** toddmorey has joined #openstack-infra05:48
*** nayward has quit IRC05:49
*** dina_belova has quit IRC05:52
*** toddmorey1 has joined #openstack-infra05:55
*** toddmorey has quit IRC05:55
*** dguitarbite has quit IRC05:55
*** michchap has joined #openstack-infra05:56
*** michchap has quit IRC05:59
*** michchap has joined #openstack-infra06:00
*** toddmorey1 has quit IRC06:11
*** pabelanger has quit IRC06:15
*** odyssey4me has joined #openstack-infra06:15
*** dina_belova has joined #openstack-infra06:20
*** michchap has quit IRC06:21
*** yaguang has joined #openstack-infra06:21
*** nayward has joined #openstack-infra06:22
*** dina_belova has quit IRC06:25
*** nayward has quit IRC06:27
*** pabelanger has joined #openstack-infra06:32
*** emagana has quit IRC06:34
*** mestery_ has joined #openstack-infra06:34
*** mestery has quit IRC06:36
*** amotoki_ has quit IRC06:37
marunanybody around?06:38
*** jjmb has joined #openstack-infra06:43
*** dina_belova has joined #openstack-infra06:44
*** bnemec has quit IRC06:49
*** dina_belova has quit IRC06:50
*** bnemec has joined #openstack-infra06:50
*** dina_belova has joined #openstack-infra06:55
*** jpich has joined #openstack-infra06:58
*** bnemec has quit IRC06:59
*** bnemec has joined #openstack-infra07:00
*** vogxn has quit IRC07:06
*** dina_belova has quit IRC07:08
*** jhesketh has quit IRC07:09
*** jhesketh has joined #openstack-infra07:09
*** vogxn has joined #openstack-infra07:11
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added post-task publisher
jheskethDo any of the jenkins workers run tests against databases other than sqlite?07:12
*** Ryan_Lane has quit IRC07:16
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added some more scm options
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added more options to the Build Timeout plugin
*** rcleere has quit IRC07:26
*** vogxn has quit IRC07:30
*** amotoki has joined #openstack-infra07:32
*** nayward has joined #openstack-infra07:35
*** toddmorey has joined #openstack-infra07:41
*** yaguang has quit IRC07:46
*** sdake has quit IRC07:46
*** toddmorey has quit IRC07:58
*** dina_belova has joined #openstack-infra07:59
*** SergeyLukjanov has joined #openstack-infra07:59
*** yaguang has joined #openstack-infra08:06
*** jjmb has quit IRC08:07
*** hashar has joined #openstack-infra08:08
*** dina_belova has quit IRC08:10
*** derekh has joined #openstack-infra08:27
*** fifieldt has quit IRC08:27
*** dina_belova has joined #openstack-infra08:28
*** vogxn has joined #openstack-infra08:28
*** nayward has quit IRC08:49
*** nayward has joined #openstack-infra08:51
*** toddmorey has joined #openstack-infra08:58
*** nayward has quit IRC08:58
*** toddmorey has quit IRC09:02
*** nayward has joined #openstack-infra09:03
*** toddmorey has joined #openstack-infra09:03
*** bnemec has quit IRC09:05
*** bnemec has joined #openstack-infra09:06
*** nayward has quit IRC09:13
*** toddmorey has quit IRC09:19
*** nayward has joined #openstack-infra09:24
*** nayward has quit IRC09:28
*** carlp-away is now known as carlp09:30
openstackgerritEmilien Macchi proposed a change to openstack-infra/jeepyb: Add puppet-heat in launchpad project list
*** nayward has joined #openstack-infra09:43
*** dina_belova has quit IRC09:57
*** vandermerwe has joined #openstack-infra10:02
*** toddmorey has joined #openstack-infra10:11
*** toddmorey has quit IRC10:27
*** nayward has quit IRC10:29
*** vandermerwe has quit IRC10:34
*** vandermerwe has joined #openstack-infra10:38
*** SergeyLukjanov has quit IRC10:41
*** yaguang has quit IRC10:53
*** bnemec has quit IRC10:57
*** bnemec has joined #openstack-infra10:57
*** weshay has joined #openstack-infra10:57
*** dina_belova has joined #openstack-infra10:58
*** sdake_ has joined #openstack-infra10:59
*** sdake_ has joined #openstack-infra10:59
*** dina_belova has quit IRC11:02
*** dina_belova has joined #openstack-infra11:11
*** SergeyLukjanov has joined #openstack-infra11:11
openstackgerritSean Dague proposed a change to openstack/requirements: update jsonschema requirements to match reality
*** jjmb has joined #openstack-infra11:26
*** toddmorey has joined #openstack-infra11:27
*** vogxn has quit IRC11:27
*** odyssey4me has quit IRC11:28
*** vogxn has joined #openstack-infra11:29
*** hashar has quit IRC11:30
*** boris-42 has joined #openstack-infra11:31
*** toddmorey has quit IRC11:31
*** odyssey4me has joined #openstack-infra11:36
*** toddmorey has joined #openstack-infra11:39
sdaguemordred: so global requirements repo tests are pretty fubar11:41
sdague and
*** rfolco has joined #openstack-infra11:41
openstackgerritSean Dague proposed a change to openstack/requirements: make sqla requirements safe for nova
*** mriedem has joined #openstack-infra11:42
openstackgerritSean Dague proposed a change to openstack/requirements: update jsonschema requirements to match reality
lifelessmordred: <- no feedback from you on those suggestions11:43
lifelessmordred: you actioned one, the other not?11:43
*** vogxn has quit IRC11:45
*** vsergeyev has joined #openstack-infra11:48
*** hashar has joined #openstack-infra11:49
*** zul has quit IRC11:54
*** vandermerwe has quit IRC11:55
*** toddmorey has quit IRC11:55
*** zul has joined #openstack-infra11:57
*** HenryG has joined #openstack-infra11:58
*** sandywalsh has quit IRC11:58
*** dkranz has quit IRC12:00
*** bnemec has quit IRC12:02
*** lcestari has joined #openstack-infra12:06
*** dkliban_afk has joined #openstack-infra12:08
*** sandywalsh has joined #openstack-infra12:11
*** vandermerwe has joined #openstack-infra12:12
*** carlp is now known as carlp-away12:15
*** weshay has quit IRC12:15
*** weshay has joined #openstack-infra12:15
*** vandermerwe has quit IRC12:20
*** vandermerwe has joined #openstack-infra12:20
*** CaptTofu has joined #openstack-infra12:20
*** vandermerwe is now known as cthulhup12:25
*** zul has quit IRC12:27
*** zul has joined #openstack-infra12:27
*** woodspa has joined #openstack-infra12:35
*** toddmorey has joined #openstack-infra12:35
*** zul has quit IRC12:50
*** zul has joined #openstack-infra12:51
*** cthulhup has quit IRC13:02
*** zul has quit IRC13:02
*** zul has joined #openstack-infra13:03
*** yaguang has joined #openstack-infra13:03
*** cthulhup has joined #openstack-infra13:05
*** vogxn has joined #openstack-infra13:13
*** dkliban_afk has quit IRC13:18
*** jjmb has quit IRC13:20
*** fbo_away is now known as fbo13:29
*** changbl has quit IRC13:29
*** zul has quit IRC13:30
*** CaptTofu has quit IRC13:32
*** krtaylor has quit IRC13:32
*** amotoki has quit IRC13:32
*** CaptTofu has joined #openstack-infra13:32
*** CaptTofu has quit IRC13:36
mordredlifeless: purely for api symmetry. having a setter and then attribute access for the getter just feel weird13:40
sdaguemordred: so... the requirements check job, any idea what's up with it?13:45
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Put the needed -r in
mordredsdague: ^^13:46
*** dkranz has joined #openstack-infra13:47
*** pentameter has joined #openstack-infra13:47
openstackgerritA change was merged to openstack-infra/config: Put the needed -r in
mordredsdague: k. when puppet kicks, that job should be fixed. sorry13:49
*** shardy is now known as shardy_afk13:49
mordredsdague: also, we discussed yesterday that that job can go away once the pbr/jeepyb gate on requirements lands - and once the devstack requirements gate lands13:50
*** cthulhup has quit IRC13:51
*** anteaya has joined #openstack-infra13:52
*** prad_ has joined #openstack-infra13:52
koolhead11mordred, morning13:56
mordredmorning koolhead1113:57
*** dhellmann_ is now known as dhellmann13:57
koolhead11mordred, dm13:59
*** SergeyLukjanov has quit IRC14:01
*** amotoki has joined #openstack-infra14:03
*** vijendar has joined #openstack-infra14:03
sdaguemordred: cool, let me know when you think rechecks are safe14:04
sdaguethe night's sleep helped a lot in understanding why we were failing. I was completely lost yesterday afternoon, and this morning after 2 sips of coffee I saw it14:05
mordredsdague: sleep and coffee are often the bug14:06
*** hashar_ has joined #openstack-infra14:06
*** mrodden has joined #openstack-infra14:07
*** hashar has quit IRC14:08
*** hashar_ is now known as hashar14:08
mordredsdague: I rechecked that one alreadyu14:08
*** dkliban_afk has joined #openstack-infra14:09
*** hashar has left #openstack-infra14:09
*** mrodden has quit IRC14:10
*** pabelanger has quit IRC14:11
*** mrodden has joined #openstack-infra14:11
*** toddmorey has quit IRC14:11
*** markmcclain has joined #openstack-infra14:11
*** CaptTofu has joined #openstack-infra14:11
BobBallsdague - any thoughts on how you'd like devstack to handle the badness in removal-of-python-lxml?14:11
BobBallDo you think we need to use venv?14:12
sdaguewe really can't use a venv in the gate14:12
*** amotoki has quit IRC14:12
BobBalloh - why? just for my understanding14:12
*** markmcclain has quit IRC14:12
*** mrodden has quit IRC14:12
*** markmcclain has joined #openstack-infra14:12
*** mrodden has joined #openstack-infra14:13
mordredBobBall: also, could you remind me what the removal-of-lxml problem is?14:14
BobBallit force removes Xen14:14
mordredah yes. that one14:15
BobBallI hit my head a few times for that14:16
BobBallcouldn't understand it...14:16
BobBallxen was installed14:16
BobBallI ran devstack14:16
BobBalland then things started breaking14:16
BobBallbut I didn't tie it back to running devstack because I didn't imagine that it would be removing xen!14:16
mordredBobBall, sdague: so, what if we wrap the remove lxml in an ifndef virt=xen?14:17
mordredor ...14:17
sdagueyeh, can we just not do it in the xen case?14:17
BobBallbut the problem with wanting to remove lxml exists for xen as well as non-xen14:17
BobBallso if we did that there might be random breakages when running devstack with xen14:18
mordredoh right - this is the thing that caused me to start beating my head against my desk the other day14:18
BobBallSorry... and I hope your desk is padded :D14:18
mordrednope. but my head is not veryuseful14:19
mordredok - here's a suggestion... (it's bad)14:19
BobBallFair enough!14:19
mordredwhat if...14:19
*** krtaylor has joined #openstack-infra14:19
mordredwe changed this line:14:19
mordredif [[ $TRACK_DEPENDS = True ]]; then14:19
*** avtar has joined #openstack-infra14:20
mordredif [[ $TRACK_DEPENDS = True -o "$VIRT_DRIVER" = 'xenserver' ]]; then14:20
BobBallso if we're using XS then we always use a venv?14:20
BobBalland we then also prevent removal of python-lxml in that case14:20
mordredwe'd also have to wrap the lxml thing14:20
BobBallThat might work14:20
mordredother option...14:21
BobBallWhat's the purpose of TRACK_DEPENDS? is it roughly FORCE_VENV or could I add that as a param?14:21
mordredfigure out how to get an lxml version new enough onto the system via rpm14:21
*** cthulhup has joined #openstack-infra14:22
mordredso that pip install -U lxml is a no-op14:22
BobBallit should be a no-op ATM14:22
BobBallI think the version in CentOS is new enough14:22
BobBallhang on14:22
mordredthen are we protecting against a problem that's only theoretical?14:22
BobBallmaybe I'm thinking of a diff one14:22
*** cthulhup has quit IRC14:22
BobBalltheoretical in this CentOS case - let me check14:22
*** cthulhup has joined #openstack-infra14:22
*** yaguang has quit IRC14:24
BobBallBoo :( I'm wrong.  pip freeze shows lxml=3.2.3 but only python-lxml 2.2.3 is available14:24
*** zul has joined #openstack-infra14:25
dtroyerBobBall: I think I just realized something…you're working on a xen-on-RHEL6 here right?  Not a XenServer (or whatever the self-contained product is)?14:26
BobBallthis seems to be a CentOS issue then - things like F19 have 3.2.114:26
BobBallxenserver-core on CentOS 6, right.  Not XenServer itself - but the XAPI packages etc14:26
BobBallI'm trying to make devstack run in dom0 (otherwise devstack is running in a VM which is always fine because it's seperate to xen and XAPI so python-lxml isn't needed for us)14:27
dtroyerBobBall: ok, thanks…I've been imagining XenServer all this time…not sure what that changes except my assumptions...14:27
*** andrew_plunk has joined #openstack-infra14:27
dtroyerBobBall: IIRC this is one reason its in domU there14:28
BobBalldevstack in dom0 on XenServer is much harder because XS is stripped down - loads of packages would be missing...14:28
BobBallLet me try adding a config "FORCE_VENV" to devstack which would pick parts of the TRACK_DEPENDS code (and TRACK_DEPENDS would add FORCE_VENV) - that might be good enough14:29
BobBallWhy can't we use venv in the gate though?14:29
*** datsun180b has joined #openstack-infra14:29
*** luis_ has joined #openstack-infra14:30
dtroyerBobBall: thinking out loud: if we 'fix' the python path issue (or work around it) on RHEL this all goes away because pip-installed packages co-exist and take priority over rpm-installed packages14:30
*** _TheDodd_ has joined #openstack-infra14:30
luis_hi i have some changes that didn't get picked up by jenkins here is one:
luis_any help is appreciated14:31
BobBallWhat is the generic python path in RHEL problem?14:31
dtroyer(from memory) pip installs packages to the same paths as rpm, overwriting the rpm-managed files.14:32
BobBallwhy is that RHEL specific though14:33
dtroyerI forget which one is wrong, but IIRc it is a RHEL-specific config issue14:33
BobBallah haha14:33
dtroyerdunno why, fedora works14:33
mordredBobBall: mainly because we want to _catch_ these issues and not gloss over them in the normal case14:33
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added pre-scm-buildstep wrapper
*** adalbas has quit IRC14:34
sdagueTRACK_DEPENDS doesn't actually work, btw14:35
BobBallthanks sdague14:35
sdagueI tried it yesterday14:36
sdagueit's probably fixable14:36
sdagueit just doesn't work as is in the tree14:36
openstackgerritDavid Caro proposed a change to openstack-infra/jenkins-job-builder: Added pre-scm-buildstep wrapper
sdagueBobBall: the reason we can't use venv in the gate is because we need to be able to have a system that works without them14:37
sdaguebecause ubuntu and redhat and others need to be able to package this all at the end of the day14:37
sdagueso our gate needs to be able to produce a system, with python libs installed globally, that can run14:37
mordredalso - ubuntu and redhat have both promised to provide backport system packages if needed14:37
BobBallBut it might be OK for some jobs to run outside of venv and some to run inside? e.g. perhaps xenserver-core could gate using a venv as long as the existing devstack runs don't?14:38
*** dina_belova has quit IRC14:39
*** changbl has joined #openstack-infra14:40
clarkbluis_ It looks like that got lost in the zuul + jenkins log shuffle yesterday. you can leave a comment on the change with 'recheck no bug' to rerun the tests14:43
*** psedlak has joined #openstack-infra14:43
luis_clarkb, ah ic, great thx14:43
clarkbjhesketh: yes all of the unittest slaves have mysql, postgresql, and mongodb available to them14:44
*** cthulhup has quit IRC14:44
sdagueBobBall: for live testing we really can't use a venv14:45
*** adalbas has joined #openstack-infra14:46
sdaguemordred: hmmm.... so puppet run yet?14:46
sdagueoh there it is14:46
BobBallokay - then I don't think that TRACK_DEPENDS or FORCE_VENV would work - I don't want a solution that would prevent us from adding a gate job in the future14:48
fungijhesketh: if you're still wondering about databases tested in our ci, we do have for example opportunistic tests against mysql, postgres and mongodb in some projects, though the scope of those tests is fairly limited they do interact with actual databases14:49
clarkbwe remove python-lxml? I thought projects depend on it?14:50
BobBallwe remove it then pip reinstalls it14:50
sdagueclarkb: just in RHEL, because, rhel is *fun*14:51
*** burt has joined #openstack-infra14:51
fungiluis_: was 39589 originally uploaded as a draft? if so, tests don't run until a new (non-draft) patchset is uploaded or the published draft is rechecked with a review comment. also your change needs to be rebased onto the tip of master to eliminate existing merge conflicts between them before it can be tested14:51
*** adalbas has quit IRC14:51
clarkbis this related to the discussion mordred had in #openstack-dev? we need a different local pip root on rhel and friends14:51
BobBallI thought we always removed python-lxml - due to the RHEL issue - but it's removed everywhere?14:51
clarkbBobBall on debian dervatives system and pip packages install side by side14:52
*** andrew_p_ has joined #openstack-infra14:52
fungiluis_: nevermind, clarkb seems to have replied to you already14:52
clarkbnot sure if we need to forcefully remove things there14:52
mordredsdague: I just wrote this:
BobBallagree we probably don't _need_ to - but we do it anyway :)14:52
mordredsdague: because the world is an evil place14:52
*** michchap has joined #openstack-infra14:52
sdagueok, I'll look14:52
*** michchap has quit IRC14:52
* fungi needs to not respond to people while reading scrollback14:52
mordredit's related to ... wait for it ... wait for it ...14:53
sdaguemordred: can you land sqla now that it passes requirements14:53
clarkbfungi its ok, that is what I did.14:53
mordredredhat instaling local packages into /usr14:53
clarkbfungi I just happened to do it first :)14:53
*** michchap has joined #openstack-infra14:53
*** hashar has joined #openstack-infra14:53
mordredthereby completely removing all logic and reason around package installed vs. source installed thigns14:53
clarkbmordred the lxml thing?14:53
*** andrew_plunk has quit IRC14:54
mordredpython install installs into /usr14:54
luis_fungi, actually i think that may have been the problem, i had tried to push the patch out as a draft but it failed, so i ended up pushing it as non-draft, and marking it as wip, but maybe that is why it wasnt picked up14:54
mordredseriously, who at redhat can I horsewhip14:54
luis_thanks though, the recheck worked14:54
BobBalldprince? ;)14:54
mordredthis is a completely unworkable and daft thing14:54
mordredhe's not in channel14:54
mordredrussellb: can you find me someone at redhat to yell at and throw things please?14:55
BobBallIt was an opportune time to go on holiday then14:55
mordredwell, I'm betting this is old and deep and I'm going to get nowhere14:55
mordredsdague: on it14:55
sdagueso very close on global requirements.... so very close14:56
fungisdague: reapproved that one now14:56
mordredfungi: could you do the jsonschema one too?14:56
sdaguedtroyer: can you take a look at this - ?14:57
fungimordred: yup was just looking at it14:57
sdaguebecause my bash-fu starts to glaze over on that14:57
mordredsdague, dtroyer: that's not for the requirements thing - it's for the other14:57
sdaguemordred: right, I know14:57
dtroyeram testing fedora now…not sure it has the same problem RHEL does14:57
russellbmordred: sorry.  i accept your horsewhip on behalf of red hat.14:58
mordreddtroyer: it does - I spun up a fedora node and verified the /usr install behavior14:58
* mordred horsewhips russellb14:58
mordredmm. that was fun :)14:58
uvirtbotmordred: Error: Could not parse XML returned by HTTP Error 404: Not Found14:58
sdagueman, the comments in devstack keep getting more and more fun :)14:58
*** SergeyLukjanov has joined #openstack-infra14:59
fungiwe need some kernel-grade snarkisms in there14:59
dtroyerfungi: on it14:59
funginothing brightens my day like slogging through dull code review only to stumble across a humorous/sarcastic comment15:00
uvirtbotmordred: Error: Could not parse XML returned by HTTP Error 404: Not Found15:00
sdagueI was pretty close to putting the whole of who's on first in the unfubar_setuptools function15:00
mordredomg uvirtbot. those are not launchpad urls15:00
*** andrew_p_ has quit IRC15:00
*** mrodden has quit IRC15:01
fungiuvirtbot is hoping rh might tack an lp api onto their bz15:01
uvirtbotfungi: Error: "is" is not a valid command.15:01
*** andrew_plunk has joined #openstack-infra15:01
fungialso, hoping that i might say something actionable, apparently15:01
*** cp16net is now known as cp16net|away15:02
*** krtaylor has quit IRC15:03
openstackgerritA change was merged to openstack/requirements: make sqla requirements safe for nova
mordredrussellb: how do I find the source for the python spec file for the official rpms for fedora? and how do I submit a patch?15:07
*** cp16net|away is now known as cp16net15:07
openstackgerritA change was merged to openstack/requirements: update jsonschema requirements to match reality
russellbmordred: patches submitted to bugzilla.redhat.com15:08
*** adalbas has joined #openstack-infra15:08
russellbmordred: so master is rawhide, and then a branch for each fedora release15:09
*** pabelanger has joined #openstack-infra15:10
mordredrussellb: thanks!15:11
russellbsure np15:11
mordredrussellb: I'm going to attempt to just submit the patch to unbreak python15:11
mordredrussellb: rather than asking other people to15:11
russellbnice moves15:11
*** boris-42 has quit IRC15:11
*** mrodden has joined #openstack-infra15:12
russellbmordred: or you want to try a test build of a package with your changes, you should be able to do it with the 'mock' utility15:13
russellbor there's a 'fedpkg' thing that wraps a bunch of stuff  and can invoke mock correctly for ou15:13
*** vogxn has quit IRC15:14
russellbs/or you/if you/15:14
*** shardy_afk is now known as shardy15:14
mordredrussellb: neat!15:14
russellbfedpkg mockbuild15:14
russellbbuilds the package in a chroot populated with the deps from the spec file15:15
BobBallThat's great for when it comes down/if it gets accepted - but to work around in devstack could we use pip install --install-option="--prefix=..."?15:15
*** toddmorey has joined #openstack-infra15:15
russellbmordred: gif of the week, seems applicable here:
russellbright sdague?  :-)15:16
*** toddmorey has quit IRC15:17
*** dina_belova has joined #openstack-infra15:19
*** jjmb has joined #openstack-infra15:19
clarkbpoor Hal^H^H^Hmordred15:19
*** toddmorey has joined #openstack-infra15:21
*** toddmorey has quit IRC15:23
*** sparkycollier has joined #openstack-infra15:23
fungii feel like that *all* the time15:25
fungii think there's something i started to fix about 20 years ago and what i'm doing now is just the other prerequisite work which has cropped up15:25
sdaguerussellb: right on man :)15:26
*** cppcabrera has joined #openstack-infra15:26
*** cppcabrera has left #openstack-infra15:27
*** toddmorey has joined #openstack-infra15:27
*** vogxn has joined #openstack-infra15:29
openstackgerritThierry Carrez proposed a change to openstack-infra/odsreg: Fix module paths in recent additions
*** vijendar has quit IRC15:31
mroddenso the in jeepyb is designed to be run on the Gerrit server locally?15:32
mroddeni might hack that up to be run as remotely as a jenkins job or something...15:33
*** UtahDave has joined #openstack-infra15:35
openstackgerritA change was merged to openstack-infra/odsreg: Fix module paths in recent additions
clarkbmrodden: it is but the only local thing it does is create the local repo replica. should be possible to run remotely without that15:36
clarkbI believe mordred may have done this before?15:36
mroddenclarkb: yep, thats what i concluded as well... not sure why the local project needs to be made by manage_projects, i think Gerrit does that anyways?15:37
mroddensorry local git repo15:37
*** rnirmal has joined #openstack-infra15:37
clarkbgerrit does not do it15:37
mroddeni always end up with a git repo in Gerrit when i used the web ui to make a project15:38
clarkbreplication will fail and you get a lot of stacktraces. gerrit does create the non replica repos under its review site dir though15:38
mordredwhat did I do?15:38
mroddenoh a replica15:38
clarkbyes, we use a local replica that apache serves to reduce load on the gerrit werver15:39
notmynamethe pep8 spec has been updated (first time since 2001). I hope this doesn't cause a lot of stupid patches in openstack projects
clarkbit shouldn't if we continue to pin pep8 each release15:41
mroddenclarkb: that makes sense, i thought that project_dir was pointing to the git repo under gerrit_home/git/15:41
clarkbnotmyname can we vote and gate on the swift functional tests?15:41
notmynameclarkb: hmm...have they been causing any problems?15:42
clarkbnotmyname: not on our end. I was hoping you would know :)15:42
notmynameclarkb: actually, I've been somewhat disconnected recently (oscon + another one day conf this week). today, I'm WFH to get caught up on the openstack stuff15:42
notmynameclarkb: I hope to be able to give you a better answer tomorrow15:43
*** hashar has quit IRC15:43
clarkbthat works, thanks15:43
notmynameclarkb: in general, I don't see it as a pressing issue, so figuring that out hasn't been at the top of my priority list. but it's good to have (IMO), so sooner is better than later :-)15:44
pleia2good morning15:44
clarkbnotmyname: definitely not pressing, but trying to avoid tests that only run in the check queue that eventually always fail...15:45
clarkbpleia2 good morning15:46
*** dina_belova has quit IRC15:47
EmilienMI'm not sure to  be at the right place, but I have a question. I've done a bunch of monitoring scripts for Nagios (working with other tools like Sensu). They are here : and also on Nagios Exchange. I was wondering if it could make sense to create a dedicated repository on StackForge ?15:47
mordrednotmyname: we will not update the pep8 pin until the next release cycle15:48
*** SergeyLukjanov has quit IRC15:48
notmynamemordred: sure sure. but even so, updates like "line length of 79 is preferred, but up to 99 is ok" could lead to all sorts of silly patch sets whenever the pin is moved :-/15:49
mordrednotmyname: well, I doubt anyone will lengthen the lines15:50
mordrednotmyname: but I know that line will make jeblair quite happy15:50
notmynamenot that I think updating pep8 for the first time in 11 years is bad :-)15:50
mordredreading through the updates right now, so far they all seem great15:50
mordredah - the explicit relative imports rule has been relaxed15:52
anteayaI don't know where else to ask this question, so feel free to redirect me if there is a better place, the topic is storyboard15:53
anteayaif I invoke the django shell with pythonn shell and load up the story and task models from:
*** krtaylor has joined #openstack-infra15:54
jeblairmordred, notmyname: "i look forward to reducing the number of silly line cont"15:54
jeblair                    "inuations."15:54
mordredjeblair: :)15:54
mordred"Always use a def statement instead of assigning a lambda expression to a name."15:54
mordredthey had to put that in?15:54
anteayaI should get all the stories if I execute Story.objects.all() - which I do - and I also should get all the tasks if I execute Task.objects.all() should I not?15:55
clarkbmordred: that was my favorite change :)15:55
clarkbanteaya: do you not get all tasks?15:56
clarkbanteaya if you connect to the db directly do you see tasks?15:57
anteayaclarkb: no I get an error15:57
marunmordred: have you seen this?:
uvirtbotLaunchpad bug 1207162 in devstack ""Unbreaking setuptools" fails under Precise" [Undecided,New]15:57
anteayaclarkb: I would only see the schema, but I can look at the schema, I will paste both15:57
mordredmarun: how could it fail under precise?15:57
marunmordred: It's failing for me, virgin install15:58
mordredmarun: we run all of the devstack-gate under precise?15:58
burtI reported that bug15:58
* mordred is so confused15:58
mordredhow does it break?15:58
* mordred goes to read bug15:58
marunVersionConflict: (netaddr 0.7.5 (/usr/lib/python2.7/dist-packages), Requirement.parse('netaddr>=0.7.6'))15:58
marunmordred ^15:58
marunnova can't load any plugins due to this conflict15:58
*** zul has quit IRC15:58
burtIn short /usr/local/lib/python2.7/dist-packages/easy-install.pth gets /usr/lib/python2.7/dist-packages instead of /usr/local/lib/python2.7/dist-packages due to that change15:58
*** jjmb has quit IRC15:59
* mordred sobs15:59
burt(and I think the gate isn't updated yet, isn't it a clone from ?)16:00
anteayaclarkb I am tracking down an error, a newly created task is not viewed, I am trying to find out it if is in the db16:00
mordreddevstack is run from trunk ever time16:00
burtthat's a wtf moment for me then too16:00
mordredon a pristine precise machine16:00
jeblairmordred: nice stuff in there about exception chaining.16:01
mordredjeblair: yah16:01
*** sarob has joined #openstack-infra16:01
mordredburt, marun: well, I'm working on the unbreak-setuptools for fedora16:01
mordredwhere redhat has decided to do CRAZY things with python16:01
mordredI'll stick this on the list for after taht16:01
burtI think this is higher priority IMHO16:01
anteayaclarkb: so I can see that the stories_task table is created, but I can't see the tasks, I am not sure if I should be able to see them in the shell independently of the story model or not, not enough django foo yet16:01
clarkbanteaya: looks like the source and pyc are out of sync maybe?16:01
anteayaokay let me look at that16:02
anteayahow do I fix it?16:02
anteayanuke the pyc?16:02
clarkbya I would try that16:02
anteayaokay thanks16:02
marunmordred: I would vote with burt.  The fact that this is happening despite the gate passing is a serious concern.16:02
anteayaclarkb: so maybe when I git pull changes I have to nuke the storyboard.db file and the pyc files16:03
mordredI think it's actually the same block of code- so it's likely we can fix both with the same patch16:03
clarkbanteaya: ya I would do that16:03
anteayaokay thanks16:03
marunmordred: I'm trying to create a precise deployment to work on the bug that is preventing neutron from being in the gate, and I can't even get devstack to complete on precise.16:03
clarkbmordred you know we have image building issues because openvz16:04
mordredthat is true16:04
mordredanyway - I'm looing in to it right now16:04
marunmordred: awesome, thank you!16:04
*** michchap has quit IRC16:11
*** gyee has joined #openstack-infra16:15
*** andrew_plunk has left #openstack-infra16:16
*** david-lyle has joined #openstack-infra16:17
openstackgerritA change was merged to openstack-infra/config: Further tuning of lists
*** vijendar has joined #openstack-infra16:18
anteayaclarkb: w00t, just learned about pyclean16:18
anteayapyclean storyboard worked16:19
*** dina_belova has joined #openstack-infra16:20
*** zul has joined #openstack-infra16:24
*** SergeyLukjanov has joined #openstack-infra16:25
*** avtar has quit IRC16:28
mordredmarun: ok. I just ran by hand in a node that is what a current devstack-gate node would look like16:36
mordredso _something_ is happening in devstack-gate node prep that is making the node be in a different state than a pristine precise node16:37
mordredI wiil now track down what that is16:37
*** CaptTofu has quit IRC16:37
*** BobBall is now known as BobBall_Away16:38
*** jjmb has joined #openstack-infra16:38
fungimordred: looking at the devstack image build failures, do we need to blow away the current cached snapshot and let it rebuild from a vanilla provider image?16:40
jeblairfungi: if we can't build the image, blowing away the image would be a bad idea16:40
fungisomething is persisting in trying to grab the openvz tarball for 11.10 even though that's fixed in devstack since16:40
jeblairfungi: is it fixed in all branches?16:40
fungioh! i bet that's it16:40
jeblairfungi: i believe mordred knew that and was doing a patch16:41
fungiahh, okay. i was willing to backport his patch from yesterday if nobody was working on that16:41
jeblairfungi: if you remove the cached images, no nodes will be able to be built at all and everything will stop in about 5 minutes.16:41
jeblair(it would likely succeed in raising the priority of the fix)16:41
jeblairfungi: but it's designed to handle this case gracefully -- it tries to build a new image, and only if that succeeds, does it become the 'current' image for new nodes16:43
clarkbfungi I almost prefer removing the openvz bits at this point16:43
clarkbor hosting the image ourselves16:43
jeblairfungi: and any images that are not the 'current' image still stick around for 24 hours16:43
clarkbotherwisw we go through this again in 9 months16:43
anteayaclarkb: thanks, that worked, I can now see that creating a new storyboard task does not create an object in the database. Now to find out why.16:44
jeblairfungi: so if the new image breaks in an exciting way, you have 24 hours to manually remove it and the previous one will be used.16:44
*** andrew_plunk has joined #openstack-infra16:44
*** cp16net is now known as cp16net|away16:44
fungijeblair: right it will keep using the old image. just wondering about cases where he old image is no longer capable of being used as a base for the new image (or does it always build from a fresh provider image rather than iterating from the previous one?)16:44
jeblairfungi: always builds from scratch16:44
* fungi rescinds his misconceptions16:45
fungiso yes, almost certainly just blocked on missing devstack stable backports now16:45
*** fbo is now known as fbo_away16:47
mordredburt: the easy-install.pth files that you pasted into the bug16:49
mordredburt: are those from /usr/local/lib/python2.7/dist-packages/easy-install.pth ?16:49
*** sarob has quit IRC16:49
*** emagana has joined #openstack-infra16:49
burtmordred: yep16:49
mordredburt: any chance you can show me what they look like _before_ a devstack run?16:49
*** sarob has joined #openstack-infra16:49
burtgive me 2 minutes to rebuild the boxes16:49
mordredburt: because here's what I get:16:50
mordredwhich does not match either of yours16:50
burt.. hm.16:51
burtthere's no /usr/local/lib/python2.7/dist-packages/easy-install.pth16:51
mordredok. that was the state my system was in before I ran devstack16:52
mordredso that's good16:52
*** pcrews has quit IRC16:52
mordredI mean- in that we started from the same state16:52
burtyou are starting from a clean precise 12.04 ?16:52
mordredburt: I'm starting from a devstack-gate node which has things precached (so the state of the nodes from the gate)16:54
*** sarob has quit IRC16:54
mordredburt: which starts as clean precise, but then some things do happen to prep it16:54
mordredoh - actually16:55
mordredburt: when you said doesn't work - in what way does it fail?16:55
mordrednevermind. I see the error in the bug16:55
burtmordred: lunch, brb in an hour16:56
mordredburt, marun: this is with you guys running with neutron enabled?16:57
burtafaik yes -- I don't disable/enable anything in the localrc but take the default16:57
mordredok. no, then you don't get neutron16:57
burter no then16:57
*** burt is now known as burt-afk16:57
mordredI don't see any attempts to stevedore netaddr in my logs16:58
*** vogxn has quit IRC16:58
marunmordred: I was trying to run with neutron16:59
*** vogxn has joined #openstack-infra16:59
marunmordred: does neutron have some os dependencies that are breaking things?16:59
mordredmarun: unclear - but it's breaking for burt without neutron17:00
marunmordred: ah, ok17:00
marunmordred: how long has neutron been out of gate17:00
mordredmarun: several weeks at least17:00
marunmordred: oy17:01
*** koolhead11 has quit IRC17:01
mordredmarun: it was flaky/breaking everything for a WHILE before that17:01
marunmordred: :(17:02
mordredomg. my new test node is the best thing in the world17:02
mordredmarun: I'm going to start clean and just run that code and see what it does to the system17:03
marunmordred: ok17:04
*** nati_ueno has joined #openstack-infra17:04
*** boris-42 has joined #openstack-infra17:05
*** ^d has joined #openstack-infra17:06
*** ladquin has quit IRC17:07
mordredjeblair, clarkb: devstack-gate nodes have setuptools and pip pre-installed on them because I believe we're doing that in our base template now17:09
jeblairmordred: i believe that is the case17:10
mordredjeblair: I think we need to move it up from outside of that - because it's pre-doing something outside of devstack that devstack is also trying to do17:10
mordredso we're not catching errors in devstack's attempt17:10
jeblairmordred: okay; this is starting to feel icky; it is apparently required for absolutely everything we do...17:11
mordredyeah. agree17:11
jeblairmordred: because pbr doesn't work without it17:11
mordredit might be time to step back and re-look at the whole problem17:11
jeblairmordred: yeah; keep in mind that devstack nodes aren't just used for devstack now17:11
mordredand it's not just pbr - it's anything in the world that tries to install distribute as part of its depends via pip17:11
jeblairmordred: so if you fix it for devstack, it may break for something else17:12
*** koolhead11 has joined #openstack-infra17:13
mordredjeblair: so - there is another option here...17:14
mordredthe big reason to attempt to upgrade setuptools out of band is to prevent anything that depends on distribute from breaking the system17:15
*** jpich has quit IRC17:15
mordredpbr used to depend on d2to1 which depended on distribute17:15
dstufftpip 1.4 internally translates distribute to setuptools IIRC17:16
*** derekh has quit IRC17:16
dstufftand distribute <= 0.7 translates setuptools to distribute17:16
mordreddstufft: are you saying that if I upgrade to pip 1.4 _first_ that the distribute translation might be caught?17:16
mordredby pip?17:16
dstufftI didn't write or test that17:17
dstufftbut that's what i'm told17:17
*** thomasbiege has joined #openstack-infra17:18
mordreddstufft: nope. doesn't work17:20
*** sarob has joined #openstack-infra17:20
dstufftMarcus sits on a throne of lies17:20
*** jjmb has quit IRC17:20
*** vipul is now known as vipul-away17:21
*** jjmb has joined #openstack-infra17:21
fungisounds like a comfortable seat, all things considered17:21
dstufftmordred: oh, maybe I just have details of the fix wrong17:21
jeblairstarting puppet on lists.o.o17:22
clarkbjeblair: ok17:22
jeblairthat will take exim back to 50 queue runners.17:22
clarkbmordred: when you are next in seattle you must show me the kexec setup, because I am really interested but probably won't have time to sort it out any time soon17:22
jeblair70 worked, but let's see if we can do 50.17:22
clarkbmordred: and for removing the setuptools fix stuff from the base node... ugh17:23
clarkbmordred: I don't think that fix is on the d-g images yet17:23
clarkbmordred: because we haven't been able to build a new image since my fix went in17:23
*** vogxn has quit IRC17:23
jeblairi'm thinking we really need to get back to a state where our starting point is just os-packages (possibly with add-on archives)17:24
mordredjeblair: I probably agree - but can you define 'our' there?17:24
*** nicedice has joined #openstack-infra17:24
jeblairall of our nodes and all of our test processes17:24
*** rcleere has joined #openstack-infra17:24
jeblairso that step 1 isn't "pip install something globally"17:25
mordredyou mean test all of openstack from distro packages and not using pip virtualenvs?17:25
*** reed has joined #openstack-infra17:25
jeblairi do not mean that17:25
*** sarob has quit IRC17:25
luis_hi, does anybody know what could be the problem here:
jeblairwe have somehow conspired to move the starting point for all of our test processes to _before_ the start of our test processes.17:25
luis_setup seems to be failing due to: error in setup command: Error parsing /home/jenkins/workspace/gate-neutron-pep8/setup.cfg: ImportError: neutron.hooks.setup_hook17:26
jeblairmordred: did you propose the openvz change to the other devstack branches?17:27
mordredjeblair: not the second one, no17:27
jeblairclarkb, fungi: maybe you want to take that on then?17:27
mordredluis_: processing setup.cfg is trying to import neutron.hooks.setup_hook17:27
clarkbya, I can do that17:27
mordredluis_: this causes neutron/ to be imported17:27
mordredwhich I see you are changing in this pach17:27
fungiclarkb: i can take it if you're not at a good stopping point on something17:28
mordredso I'm guessing that something about this causes it to be impossible to import that file anymore outside of a venv with all the deps17:28
*** andrew_plunk has quit IRC17:28
clarkbfungi: I just got to my desk so perfect time to start something. I am good17:28
mordredI would strongly suggest not doing the gettext install in neutron/__init__ as it will cause issues17:28
luis_mrodden, thanks for the pointers, i will try to see if doing it somewhere else would work17:29
jeblairclarkb, fungi: graphite is about to run out of disk space (though i can recover some by deleting the old trees)...17:30
jeblairbut it also could potentially use a cpu/memory bump.17:30
jeblairit looks _okay_, but i think one more doubling of resources wouldn't hurt it.17:30
clarkbjeblair: wfm17:31
jeblair(let me check and see where the cpu bump thresholds are)17:31
clarkbjeblair: there are times I have asked it to graph a thing and I have had to wait a while17:31
*** Ryan_Lane has joined #openstack-infra17:31
jeblairclarkb: yeah17:31
*** vipul-away is now known as vipul17:31
jeblairwe're at 2g/2cpu; we'd have to go to 8g/4cpu to get a cpu increase.17:32
jeblairclarkb, fungi: does that sound reasonable, or should i just get a nova volume for the data?17:32
jeblairi believe the load is low now because of the broken graph; i expect it to increase again when we fix it17:33
jeblair(though perhaps we should make an optimized statsd entry just for that graph)17:34
*** avtar has joined #openstack-infra17:36
clarkbthe history cpu usage is pretty consistent17:37
clarkbis graphite capable of using more cpu time?17:38
clarkbmordred: jeblair fungi and backport the devstack openvz image fix17:39
fungijeblair: it looks like most of the cpu usage on graphite.o.o is iowait, so i'm not sure additonal cpus will help? does it need faster storage? or more cache? (the memory graph suggests it very well may benefit there)17:40
fungijeblair: maybe just doubling the ram and disk will actually help cpu impact if so17:41
fungiespecially if rackspace also scales i/o bandwidth with the flavor17:41
*** dina_belova has quit IRC17:43
fungisometimes interpreting resource utilization trends feels a bit like reading tea leaves17:43
clarkbfungi: indeed17:43
clarkbI think your comment about more cache is probably valid though17:44
*** fbo_away is now known as fbo17:45
*** fbo is now known as fbo_away17:45
*** dkliban_afk is now known as dkliban17:46
*** jjmb has quit IRC17:47
pleia2Ryan_Lane: available for coffee/lunch later to chat openstack workshop?17:48
openstackgerritSergey Lukjanov proposed a change to openstack-dev/hacking: Extract configs-related code to the configs helper
openstackgerritSergey Lukjanov proposed a change to openstack-dev/hacking: Import exceptions list is now configurable
*** pcrews has joined #openstack-infra17:51
Ryan_Lanepleia2: yep. I was hoping you'd want to meet today17:51
clarkbits worth noting on my devstack backports I used the first commit's change id and commit message but I applied both the first commit and the commit that fixes the first commit17:51
clarkbthey are squashed together17:51
pleia2Ryan_Lane: cool, any time is good for me really, so just let me know and I'll pop downstairs17:51
*** sarob has joined #openstack-infra17:51
*** UtahDave has quit IRC17:52
Ryan_LaneI have a couple meetings, but will be available afterwards17:52
Ryan_Lanemaybe 1:00?17:52
pleia2Ryan_Lane: sounds good17:52
pleia2Ryan_Lane: just ping me then17:52
*** sparkycollier has quit IRC17:53
Ryan_Lanewill do17:53
*** fbo_away is now known as fbo17:54
*** ladquin has joined #openstack-infra17:54
fungiclarkb: yep, i actually checked to make sure17:54
fungiclarkb: it's what i would have done as well17:54
reeddid you get your hotels for New Orleans? W or Sheraton?17:54
reedor Westin, and which of the two W? :)17:55
*** koolhead11 has quit IRC17:55
reeddon't want to be by myself17:55
clarkbjeblair: russellb pabelanger is safe to apply without the CID functionality? eg the logger won't die or otherwise disagree with the lack of info?17:56
*** sarob has quit IRC17:56
russellbclarkb: it will just be blank17:57
russellbthose fields in the CSV file that is17:57
russellbno big deal17:57
clarkbrussellb: ok17:58
*** vipul is now known as vipul-away17:58
russellbclarkb: and it will be populated in the case that someone calls in via SIP directly17:58
russellb(with whatever they want it to say, heh)17:58
*** Ryan_Lane has quit IRC18:01
*** toddmorey has quit IRC18:01
*** ^d has quit IRC18:02
openstackgerritA change was merged to openstack-infra/config: Add a CDR log file
*** vipul-away is now known as vipul18:03
*** Ryan_Lane has joined #openstack-infra18:04
*** odyssey4me has quit IRC18:05
*** jjmb has joined #openstack-infra18:07
*** changbl has quit IRC18:08
*** burt-afk is now known as burt18:08
*** koolhead11 has joined #openstack-infra18:09
*** lcheng has quit IRC18:09
mgagneWith jenkins scp and console timestamps plugins: when copying console log to a scp site, how do you make it include console timestamps too?18:09
clarkbmgagne: the console timestamps are part of the console log so it just happens18:10
*** lcheng has joined #openstack-infra18:10
mgagneclarkb: unfortunately, it's not the case for me =(18:10
clarkbmgagne: you do have to enable the timestamps on a per job basis is that missing?18:11
*** cp16net|away is now known as cp16net18:12
mgagneclarkb: I enabled timestamps for my jobs and added a config to copy console log to a scp site. So I'm wondering if I'm missing a config18:12
*** andrew_plunk has joined #openstack-infra18:14
*** jjmb has quit IRC18:16
*** andrew_plunk has quit IRC18:16
*** andrew_plunk has joined #openstack-infra18:16
clarkbmgagne: do the timestamps show up in the jenkins served console logs?18:18
clarkbmordred: I did not approve that or +2 because I wasn't a part of the conversation taht went on around that18:19
mgagneclarkb: yes, I have an option box in the sidebar to show/hide console timestamps. Could it be I installed the wrong plugin or you guys have an older one? I don't see this option on jenkins.o.o.18:19
clarkbmordred: but you do have the needed votes if you want to approve now18:19
*** hashar has joined #openstack-infra18:19
clarkbmgagne: yeah that doesn't sound the same18:19
clarkbmgagne: is what we use18:20
mgagneclarkb: I use the same, v1.5.4. :-/18:20
*** mestery_ is now known as mestery18:20
clarkblet me see what version we are running18:20
mgagneclarkb: don't tell me 1.0beta =)18:21
*** gyee has quit IRC18:21
clarkbmgagne: 1.3.118:21
mgagneclarkb: ok, I'll start from there and see what I can do to get the same behaviour. Could be a change in the format used to store the console log.18:22
*** sarob has joined #openstack-infra18:22
bodepdI would like to split the pip module off into it's own repo18:23
bodepd(probably not a bad place to start :) )18:23
clarkbbodepd: like a forge module?18:23
bodepdis there by-chance a doc that just says how to do it :)18:23
bodepdclarkb: could be eventually. I just want a repo puppet-pip to exist at18:24
bodepdclarkb: git://
bodepdclarkb: and then for config to use that repo18:24
bodepdthen eventually, the same needs to happen for zuul, jenkins, etc.18:25
bodepdfor now, those are the only ones I need18:26
clarkbthis is something we want to do, but isn't something we have really done. so blazing new territory18:26
*** CaptTofu has joined #openstack-infra18:26
bodepdoh good!18:26
mgagnebodepd: there's some modules available on
clarkbmgagne: ya those are ones that mordred forked for reasons...18:26
mgagnebodepd: not sure how releases are managed.18:26
bodepdif pip was released up there, I could live with that18:27
clarkbbodepd: making modules that are available on the forge would be the easiest way for us to reconsume them18:27
*** sarob has quit IRC18:27
mgagneclarkb: forked or not, I'm more interested in the workflow to manage/release such modules =)18:27
bodepdme too. I just need to be able to consume them18:27
clarkbit is manual :(18:27
bodepdand I want to use the same code infra does18:28
bodepdthere are auto-release tools the forge team is looking at18:28
bodepdI haven't had a chance to look at them18:28
bodepdbut I am sure they assume: 1 module, 1 repo18:28
bodepd(well, it seems likely they would)18:29
*** melwitt has joined #openstack-infra18:29
*** sarob has joined #openstack-infra18:29
mgagnebodepd: having multiple modules in the same repo would be a challenge from a release pov as IMO, releases should be tagged =)18:29
*** zul has quit IRC18:29
clarkbya repo per module is sane18:30
bodepdwhich goes back to the original question, would you be open to a patch to decouple pip18:30
*** sarob has quit IRC18:30
bodepdand where do I start :)18:30
fungibodepd: familiar with git filter-branch?18:30
clarkbbodepd: I would be open to it. The where to start is tough. we could filter branch out the pip module18:30
*** sarob has joined #openstack-infra18:31
bodepdthat preserves the history :)18:31
clarkbimport it into gerrit, publish to forge then remove it from the config repo18:31
clarkbmordred: jeblair ^18:31
fungithat's the order i'd do it in. also probably would need to add it to the modules list we pull in, presumably18:31
bodepddoes it need to be initially published to forge?18:31
bodepdyeah. I can figure out how to patch the config repo18:32
clarkbbodepd: yes so that we can use it18:32
mgagnebodepd: check install_modules.sh18:32
bodepdI'm a little less certain about how to create it as a new project18:32
clarkbor we figure out submodules or puppet librarian if it stopped sucking18:32
bodepdpuppet-librarian-simple :)18:32
mgagneclarkb: haha :D18:32
fungii would say we need to publish it to the forge before we rip it out of our main repo, but that can happen in multiple stages and we can just put changes to that module on hold while it's being worked through18:32
jeblairclarkb: i agree with what you have been saying18:32
mgagnebodepd: you are biased :P18:33
jeblairclarkb: submodules --18:33
bodepdmgagne: as soon as it support forge installs, it *will* conquer the world18:33
bodepdI'm such a submodule hater18:33
* fungi ph33rs git submodules18:33
mordredah. good. I was saying no to submodules - extra complex, no need18:33
fungimordred: nice save18:34
bodepduse a Puppetfile before you use submodules18:34
mgagnebodepd: and then we will have the same problem again and puppet-librarian-dummy will be created ;)18:34
mordredbodepd: I still don't know how to do that18:34
mordredalthogh you showed me one18:34
mordredis that the librarian thing?18:34
* mordred actually would not mind using a Puppetfile instead of install_modules btw18:34
mordredyeahyeah. I've _seen_ a Puppetfile18:35
bodepdthe problem is that librarian-puppet really does suck18:35
mordredI don't know how to use it18:35
mordredso that is librarian-puppet18:35
bodepdlibrarian-puppet install18:35
clarkbbodepd: I know :)18:35
mgagnemordred: I like the way works as it does not require an external tool but it requires modules to be available on forge =)18:35
mordredyeah. I believe it needs versions we don't use or something crazy18:35
bodepdthat is where it gets painful18:35
bodepdthe format is good, but the tool is so bad I wrote my own18:35
mordredoh - well...18:36
mordredwell, clearly we're not going to put github references in there18:36
bodepdanyways, focus b/c we don't have to solve that today :)18:36
mordreddude, I'm just taking a quick break from setuptools hell18:36
bodepdsound fun :)18:36
clarkbbodepd: I think we can start by splitting it out at least18:37
bodepdhow do I create a new project under openstack-infra ?18:37
bodepdand pip is actually a pretty easy one to start with18:37
clarkbbodepd: but s/stackforge/openstack-infra/18:37
mgagnemordred: upstream needs discipline and tag their releases as they should have been doing since the start.18:37
jeblairclarkb: i think graphite will be able to use more cpus because much of it is apache mod wsgi (which are sep procs) there's a single carbon-cache which will not scale and be a bottleneck :(, and then there's node.js for statsd.  so in general, some reasonable ability to use a few processors.18:37
jeblairfungi: ^18:37
clarkbjeblair: I think we should bump mem and cpu then18:38
mordredmgagne: discipline to tag your releases becomes much easier when tagging your releases creates your releases... :)18:38
mgagnemordred: I have nothing to add to that comment =)18:39
*** cthulhup has joined #openstack-infra18:39
fungijeblair: clarkb: yeah, given that, i agree that turning the flavor knob up two clicks instead of the one makes sense18:39
jeblairresize button clicked18:40
annegentlehey pleia2 anteaya, did you work out who's going to present to Codechix?18:40
anteayanot yet18:41
pleia2annegentle: Ryan_Lane and I are, also looking for someone else (there are lots of people around, shouldn't be an issue, but I would prefer another woman just because of the nature of the thing)18:41
anteayaI was going to talk to pleia2 about it next week18:41
*** NobodyCam has joined #openstack-infra18:42
anteayaannegentle: did you have any thoughts or suggestions?18:42
pleia2if anteaya can't make it, thinking of putting feelers out on -women linkedin group if that's ok18:43
anteayapleia2: I should have a better sense of whether I can make it or not next week18:43
openstackgerritSergey Lukjanov proposed a change to openstack-dev/hacking: Extract configs-related code to the configs helper
openstackgerritSergey Lukjanov proposed a change to openstack-dev/hacking: Import exceptions list is now configurable
NobodyCammay the new zuul page looks very nice! congrats -infra18:43
pleia2anteaya: great, thanks :)18:43
annegentleanteaya: pleia2: would it make sense for Rackspace to sponsor CodeChix for those types of workshops? Not to fly people in necessarily, but for Rupa to use as she wishes.18:43
*** dina_belova has joined #openstack-infra18:43
NobodyCam* may I say ...18:44
Ryan_Lanepleia2: I'm happy to just be a TA and have you and another woman present18:44
anteayaannegentle: I can't comment, I have just been asked to present18:44
pleia2annegentle: perhaps! I know she's always looking for sponsors for various things (also planning on doing some codechix events in SF soon, not sure if she's sorted out space - rackspace has an office here now if I recall?)18:44
anteayaI don't know anything about the direction and mandate of the group18:44
annegentlepleia2: that's right, we have an office in SF18:45
anteayaRyan_Lane: it would be lovely to meet you, I hope I can go18:45
jeblairNobodyCam: thanks :)18:45
Ryan_LaneI've presented at codechix before. there's no requirement for the presenter to be a woman, but I'm sure it's not a bad goal :)18:45
annegentleRyan_Lane: you're awesome, you know that. Don't feel like you have to take a back seat at all18:45
pleia2annegentle: +118:45
annegentleRyan_Lane: :) it's always nice to show that OpenStack has women presenters!18:45
*** vipul is now known as vipul-away18:45
Ryan_Laneannegentle: ageeed18:45
annegentleok I'll see what I can do to help, just wanted to hear your latest-n-greatest18:46
pleia2thanks for checking in :)18:46
mordredpleia2: anteaya should be able to go if we can figure out all of her travel accounts first18:46
pleia2mordred :D18:46
annegentlemordred: Awesome.18:46
anteayaI couldn't say anything yet18:46
anteayabut mordred can18:46
clarkbfungi: jeblair mordred
jeblairclarkb: fungi:
jeblairclarkb: fungi: can you look that over, even though it's not passing tests?18:48
clarkbpreviously I think we asked ptls to delegate by adding members to the ptl group. that still the case?18:48
clarkbjeblair: sure18:48
*** dina_belova has quit IRC18:49
jeblairclarkb: fungi: i'd like to merge the breaking change, and then retest that, and then merge it and restart zuul to pick it all up (in a relatively short period)18:49
mordredclarkb: for the client libs, I think we have other projects using core18:49
jeblairmordred: i'm not sure we should...18:49
fungiclarkb: yes, the .*-ptl groups were meant to be "project lead plus delegates for git repo permission purposes"18:49
mordredah. I'm with all of you now18:50
jeblairi'm really troubled by the idea that lots of people would automatically get the ability to irrecovably screw up the release process18:50
mordredI'd be most onboard with them just adding people to the -plt18:50
clarkbI will update my review with info about using the ptl group18:50
*** vijendar has quit IRC18:51
jeblairmaybe we should rename them at some point, but it doesn't seem terribly important now.18:51
*** sdake has joined #openstack-infra18:52
*** sdake has quit IRC18:52
*** sdake has joined #openstack-infra18:52
* fungi will be mostly afk for a few hours starting around 19:30z today. time again for the monthly osug here in the triangle18:52
mgagneclarkb: It seems -Dtimestamper-consolenotes=true is required for console log to be stored in the old format and for console log copied by scp to include timestamps. Will be able to test and confirm in 1h. Might worth noting if you are planning on upgrading this plugin.18:53
*** dina_belova has joined #openstack-infra18:54
*** vijendar has joined #openstack-infra18:56
reedjeblair, fungi: we're kicking off the community/user group portal project today18:57
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Support multiple triggers
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Add a timer trigger
mriedemmtreinish: launchpad doesn't seem to have changed this bug status after the fix was merged?
uvirtbotLaunchpad bug 1207066 in tempest "Tempest is dumping debug-level messages to console output" [High,Triaged]18:57
fungireed: i will make sure to mention it at our user group meeting today, in that case!18:57
mordredclarkb, sdague, jeblair: <--- that's a job where I made it go back to just doing the upgrade of setuptols18:57
reedwe'll need two machines, one for staging and one later for production18:57
reedand a couple of hostnames, too18:58
reednot sure how we'll call this thing18:58
jeblairreed: django app, right?18:58
reedjeblair, drupal18:58
fungireed: oh, by kicking off you mean starting development. i'll hold off plugging it to the ug just yet in that case ;)18:59
reedfungi, yes, it's just a start of development18:59
* mordred goes afk18:59
jeblairreed: do the devs have an existing puppet setup or expertise to contribute to that?18:59
*** cthulhup has quit IRC19:00
reedjeblair, I think they do, and if not they're willing to learn19:00
mtreinishmriedem: yes it merged that's weird that it didn't update the status though19:00
mriedemmtreinish: i have another one in nova like that19:00
mriedemmerged 3 weeks ago19:00
reedI'll ask them to integrate in our workflow19:01
mriedemmtreinish: maybe it's only bugs i open?19:01
jeblairreed: cool, then let's try to start with a basic puppet install that we can help them iterate on, so it's not overwhelming19:01
jeblairreed: just let us know the hostname you want.  :)19:02
reedi'll file a ticket after the meeting today19:02
*** andrew_plunk has quit IRC19:04
sdaguereed: who's driving the drupal thing? I've actually do a bit of that for some local non profits, so might have some helpful input19:04
jeblairsdague: do you use puppet with that by any chance? :)19:04
reedsdague, mrmarton will be the main dev on the team with colleague19:05
sdagueI don't19:05
sdaguewell, not yet. I've been thinking about it for one of the sites, just haven't gotten there yet19:05
jeblairreed: when's the meeting?19:05
reedmrmarton and his pal have done lots of drupal development too, that's what made them win19:05
reedjeblair, today, 2pm PDT19:05
reedon skype though19:05
jeblairreed: and are they going to be there and talking about19:06
jeblairwell, um, have fun i guess.19:06
*** changbl has joined #openstack-infra19:06
jeblairyou know where we are if you want help from anyone working on the project.  :)19:06
reedyeah ... alternative that would not cost € to the guys in Hungary?19:06
openstackgerritA change was merged to openstack-infra/zuul: Support multiple triggers
jeblairreed: i thought there was an irc meeting19:06
reedirc is too slow19:07
reedwe'll have irc meetings later19:07
openstackgerritA change was merged to openstack-infra/zuul: Add a timer trigger
*** sarob has quit IRC19:09
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add queue_smtp_domains as a parameter
*** toddmorey has joined #openstack-infra19:14
jeblairi'm going to approve that now, it's a brown-bag syntax error ^19:14
clarkbgo for it19:16
jeblairso actually, i'm going to do more than that19:17
fungino valid value found for dictionary value @ data['pipelines'][0]['trigger']19:18
clarkbpleia2: re were you planning on updating that change or would you like me to?19:18
*** pabelanger has quit IRC19:18
jeblairstopping puppet on zuul19:18
clarkbpleia2: either way is fine, I just want to avoid stepping on toes and duplicating work19:18
*** vipul-away is now known as vipul19:18
jeblairi'm going to merge the layout change, then the puppet fix, then run puppet manually on zuul19:19
*** boris-42 has quit IRC19:19
jeblairthen restart zuul19:19
clarkbjeblair: ok, that sounds safe19:19
pleia2clarkb: I can take care of it, just been wrapped up in other things these past couple days19:19
*** boris-42 has joined #openstack-infra19:20
clarkbjeblair: is there value in removing the race there just to be extra cautious or should I abandon that change?19:20
clarkbpleia2: no problem. I know you have been busy lately19:20
dtroyermordred, sdague:  this will be helpful for the pip stuffs on fedora/centos:
jeblairclarkb: i'd like to avoid merging that change in favor of continuing the major overhaul.19:21
openstackgerritA change was merged to openstack-infra/config: Specify gerrit triggers
sdaguemordred., clarkb: does that solve the thing you were talking about in -qa ?19:21
sdaguedtroyer: looking...19:21
sdaguedtroyer: oh.... right19:22
sdaguedtroyer: you tested on fedora?19:22
clarkbjeblair: ok19:22
sdaguedtroyer: to the gate it goes19:23
openstackgerritA change was merged to openstack-infra/config: Add queue_smtp_domains as a parameter
openstackgerritA change was merged to openstack-dev/pbr: Hierarchical sphinx API documentation generation
jeblairrestarting zuul19:25
jeblairdone, and rechecks left19:26
*** Ryan_Lane has quit IRC19:27
jeblairthat should be it for the invasive zuul changes for a bit.19:27
*** jjmb has joined #openstack-infra19:27
*** dina_belova has quit IRC19:29
harlowjathx jeblair19:30
*** jjmb has quit IRC19:32
openstackgerritA change was merged to openstack-infra/config: Handle html log annotations.
*** sandywalsh has quit IRC19:33
sdaguejeblair: I noticed earlier today that stable/grizzly sits in the same pipeline as master19:33
openstackgerritA change was merged to openstack-infra/config: Handle log paths in logstash gearman client better
sdagueis that intentional?19:34
jeblairsdague: yes; it'll become particularly important once we have bidirectional grenade testing, and is also useful for compat testing for client libs, etc.19:34
sdagueok, fair19:35
hasharjeblair: hi!  I finally completed our little status page :]19:35
sdaguenow I just need to figure out where mordred ran off to :) Because the land of magical ponies is close19:35
hasharjeblair: thank you very much for all the inspiration!19:35
*** rnirmal has quit IRC19:36
sdaguejeblair: also did you notice the zuul jobs launched per hour stopped updating yesterday19:36
*** pabelanger has joined #openstack-infra19:36
jeblairhashar: great!  i added progress bars to ours, have you seen?
hasharjeblair: yeah I have been following your zuul commits19:37
harlowjaqq for ya'll, is there anyway to force a dependency Y for a review X, without having to make code changes to said review X19:37
hasharwill hopefully migrate to gear in october.19:37
jeblairhashar: (i still need to add those to the page in the repo that's based on yours)19:37
hasharthat looks very nice19:37
jeblairhashar: cool, i think some other people are setting up a new system with gear now (bodepd), so hopefully it'll be pretty smooth by then19:38
jeblairsdague: yes, i'll fix that soon (i moved some statsd keys around)19:38
hasharjeblair: will have a look at gear, and might end up writing a bunch of tiny workers for the most basic tasks (such as running pep8)19:39
jeblairhashar: great! i'd like to move in that direction too!19:39
mroddenclarkb: its a bit more involved than i thought to be able to run manage-projects remotely. when updating the ACLs it has to build the groups file, which then needs to look up the group UUIDs in Gerrit which currently requires parsing the gerrit.config file for the database connection information19:40
jeblairhashar: i was brainstorming with mordred, and we came up with the idea of trying to use linux containers to separate the gear worker from where the job is actually running19:40
jeblairhashar: we haven't tried that, but it sounds like it could be useful for privilege separation19:40
clarkbmrodden: hmm, we could probably split out the gerrit DB config info into a manage projects config file19:40
clarkbmrodden: but ya that would break you if you need the other file to be local19:40
clarkbharlowja: are you asking if you can force a dependency during the check and gate tests?19:41
mroddenclarkb: yeah that was my thought too, although, it is possible to do SQL queries on gerrit through the SSH api19:41
mroddenoddly enough19:41
clarkbmrodden: it is, but we have decided that using direct DB connections is better beacuse you get sanitized db inputs when you use python-mysql or psycopg2 for example19:42
*** dkliban has quit IRC19:42
mroddenyeah. plus its nasty to have to parse query results from an SSH channel...19:42
harlowjaclarkb mainly putting up, have active review which is solved by the previous bug, but tried to add dependency between them (after the second one is commited) without changing the code in the second one seems like its not possible, if that makes sense19:42
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Add projects filter to zuul dashboard
clarkbharlowja: ya, this has to do with the way git works. The sha1 of your parent in git is included in the calculation of the current commit's sha119:44
*** dina_belova has joined #openstack-infra19:44
clarkbharlowja: this makes fetches really fast but means changing a parent changes the child (makes sense, but does force an update in gerrit)19:44
harlowjak, i can find something to change, checkout first review, then apply second review (with small modification) ontop and be all set19:45
clarkbyou don't need to change anything19:45
*** UtahDave has joined #openstack-infra19:46
clarkbunless I am mistaken. Simply creating the dependency in the git DAG is sufficient19:46
*** sandywalsh has joined #openstack-infra19:46
harlowjak, let me try that again19:46
fungiyeah, gerrit will note that you've made no changes to the parent and won't bother to update it, but will update the child to depend on it19:48
fungii do that all the time19:48
openstackgerritSean Dague proposed a change to openstack-infra/config: add devstack gate to requirements changes
openstackgerritSean Dague proposed a change to openstack-infra/config: add missing neutron-full job to neutron gate
openstackgerritSean Dague proposed a change to openstack-infra/config: make oslo.config gating on devstack/tempest runs
* fungi disappears for a few hours... bbl19:49
clarkbjeblair: sdague,n,z can we get those in so that d-g image updates can be run in one AZ before hitting the others?19:49
clarkbonce that is happy I think we can consider
clarkbfungi: if you haven't left yet19:50
jeblairclarkb: i +2'd those19:50
clarkbfungi: is subunit2html still causing problems?19:50
clarkbjeblair: ty19:50
sdagueclarkb: sure, though man zuul is pretty backed up right now19:50
clarkbsdague: yeah, that is what happens when jeblair restarts all the things19:51
jeblairmy fault19:51
fungiclarkb: subunit2html is not directly. but testr is (i have a change up to fix that)19:51
sdagueok, well while I wait for the world to clear, clarkb do you understand the crux of the thing?19:51
fungiclarkb: specifically, we were invoking testr and overriding the interpreter in ways it ignored19:51
clarkbfungi: awesome I will look for it, currently trying hard to do code review19:51
*** mgagne has quit IRC19:51
fungii figure i will be 100% code review when i get home this evening19:52
* fungi really leaves now19:52
* jeblair gets lunch19:52
clarkbsdague: <- that what you are talking about?19:52
sdaguehmmm... gate is also bouncing on things like this -
sdaguethat just because of jenkins load?19:53
sdagueclarkb: you think that's actually our solution?19:53
clarkbsdague: yes jenkins load19:54
clarkbsdague: the indicator is the inprogress job failed because we don't have enough executors to run all of the inprogress jobs that start when we restart all the things :(19:54
*** ladquin has quit IRC19:55
clarkbso back to I feel like I am missing history somewhere19:55
jeblairclarkb: it should never fail -- it's failing because it's taking jenkins more than 10 minutes to execute an api call to modify a node, which is ridiculous.19:55
jeblairwhich is why i'm adding more jenkinses19:55
clarkbI read that last sentence in Gollum's voice19:56
jeblairi said it in Gollum's voice19:56
jeblairactually, that one wasn't a timeout.  i'm going to look into it after lunch19:57
anteayaso I can create a new task in the django shell and it shows up in the gui19:57
clarkbsdague: it says restore merging the two files, but it doesn't actually merge the two files19:57
clarkbsdague: what am I missing19:57
anteayaI can't successfully query all tasks from the django shell19:57
clarkbsdague: oh I get it19:58
sdagueI think it allows content from either file19:58
anteayaand I can't create a task in the django gui19:58
clarkbsdague: ya, which is wrong19:58
sdagueyou know, let me actually test it19:58
*** mgagne has joined #openstack-infra19:58
sdagueclarkb: well, only sort of19:58
clarkbsdague: instead, we should just have one file19:58
sdagueclarkb: ok19:59
clarkbwe shouldn't be so split brained that we allow duplicate and potentially conflicting requirements in different places (not sure if that script allows conflicts)19:59
clarkbinstead we should have the one list to rule them all and in its power bind them19:59
sdaguethe new version of doesn't allow it19:59
sdaguebut I get the point19:59
sdagueso if we do that, then we could actually do the following20:00
sdaguemv requirements.txt -> global-requirements.txt20:00
sdagueput test-requirements.txt => global-requirements.txt20:00
sdaguethen have actual requirements.txt be the requirements for that repo20:00
sdaguewhich means the unit testing could land without a pbr change20:01
clarkbsdague: I like that20:02
openstackgerritlin-hua-cheng proposed a change to openstack/requirements: Add support for Keystone V3 Auth in Horizon.
clarkband unittesting wasn't something I had considered would be enabled but ya that sounds good20:02
sdagueyeh... let me tell you, seeing how unit testing exploded on this was *fun* :)20:02
*** Ryan_Lane has joined #openstack-infra20:03
*** cp16net is now known as cp16net|away20:06
*** dina_belova has quit IRC20:06
Ryan_Lanepleia2: ok, ready20:08
*** ladquin has joined #openstack-infra20:08
Ryan_Lanepleia2: any where good to meet?20:08
*** dina_belova has joined #openstack-infra20:08
sdagueclarkb: actually.... I should just merge this into the unit test patch20:09
pleia2Ryan_Lane: hey, can meet at wikimedia street entrance, I'll be down in a minute20:09
*** koolhead11 has quit IRC20:09
Ryan_Lanecool. heading down there.20:09
*** Ryan_Lane has quit IRC20:09
clarkbsdague: logstash should be ready for your annotation stuff now, any idea why it 500s instead of 404ing when you request non existant files?20:10
*** cp16net|away is now known as cp16net20:12
*** afazekas has quit IRC20:13
lifelessmordred: module level getters are weird20:16
lifelessmordred: perhaps the setter shouldn't be a setter but a helper20:16
*** Ryan_Lane has joined #openstack-infra20:17
*** nati_ueno has quit IRC20:17
*** psedlak has quit IRC20:17
*** nati_ueno has joined #openstack-infra20:18
*** dina_belova has quit IRC20:18
*** rnirmal has joined #openstack-infra20:19
*** nati_ueno has quit IRC20:22
*** dkranz has quit IRC20:24
*** prad_ has quit IRC20:25
anteayaI am getting closer in figuring out what is happening with storyboard tasks, there seems to be a dependency on milestones/branches/projects and stories but I am uncertain which way to proceed because I don't have the vision of what relationships are supposed to be dependencies20:27
anteayaguess I am waiting for ttx20:27
openstackgerritA change was merged to openstack-infra/config: Additional path patterns for log readme files.
*** nati_ueno has joined #openstack-infra20:29
clarkbanteaya: after reading that sentence I would wait for ttx too :)20:29
anteayaclarkb: ha ha ha20:29
*** HenryG has quit IRC20:30
openstackgerritAlejandro Cabrera proposed a change to openstack-infra/config: Add Python 3.3 gate for Marconi + Client
clarkbjeblair: mordred
*** cppcabrera has joined #openstack-infra20:32
openstackgerritSean Dague proposed a change to openstack/requirements: (WIP) add testing to requirements
anteayain other news it appears that if I run `testr run --parallel tempest.api.identity` on a brand new devstack install (or a refresh with ./ && ./ the tests complete very quickly but if the vm instance has been running away (running tempest tests) that same command takes far longer to complete20:32
anteayaafter this run is finished I was going to unstack and stack again to be sure I am finding the same thing again20:33
clarkbanteaya: interesting20:34
cppcabreraHey. I heard that py33 gates were added recently to our CI backend! I'm happy to hear it. Who reviews/approves patches for project looking to start using the py33 gate? :)20:34
clarkbcppcabrera: mordred jeblair fungi and myself20:34
anteayaclarkb: I find it to be so20:34
clarkbcppcabrera: I will take a look in a bit20:34
cppcabreraI submitted one a moment ago for the marconi project:
cppcabreraThanks, clarkb!20:34
anteayanot that it really matters since in most use cases the devstack/tempest set-up is single use disposable20:35
clarkbanteaya: yeah, but understanding the behavior is probably a good thing20:36
mordredlifeless: it should _probably be a proper object20:36
anteayaclarkb: I concur20:36
*** whoops has joined #openstack-infra20:36
whoopsdrop something?20:37
anteayaha ha ha20:37
anteayagood to see you20:37
clarkbit is times like this I feel like I did poorly when choosing a nick20:37
*** Ryan_Lane has quit IRC20:38
*** boris-42 has quit IRC20:38
openstackgerritAlejandro Cabrera proposed a change to openstack-infra/config: Add Python 3.3 gate for Marconi + Client
anteayaclarkb: I like ghost tooth20:39
anteayayou could go back to ghost tooth and I would talk to you20:39
lifelessmordred: so I don't want to polish an irrelevancy ;) - I will check for any other issues and +1 shortly20:39
anteayato be fair his last name is hoops20:39
whoopsyeah this nick sorta fell into my lap20:40
clarkbanteaya: so you are saying that my real problem is my parents provided me with a poor name for IRC?20:40
mordredlifeless: thanks - it's a good note in general - but I do think I'd like to probably make that an object in a later refactor20:40
anteayaclarkb: heck no20:40
clarkbits ok, I complain to them that they gave me a boring name20:41
anteayathey gave you access to a great language with stuff like ghost tooth in it20:41
*** Ryan_Lane has joined #openstack-infra20:41
anteayaokay that is your prerogative20:41
anteayanot piling on that fight20:41
anteayamine came about because my nephews wanted to play computer games at my place20:41
* mordred always thinks of clarkb as 'ghost tooth'20:42
anteayaso I said okay if they showed me how20:42
anteayaha ha ha20:42
*** cp16net is now known as cp16net|away20:43
clarkbcppcabrera: the change looks good to me, but quick question. Are you expecting those tests to pass soon? For many projects I think we expect to spend a lot of time dealing with dependencies before we even worry about testing20:44
*** prad_ has joined #openstack-infra20:44
sdagueclarkb: so that's not right -, but my brain is breaking on why the test-requirements.txt part doesn't20:45
clarkbcppcabrera: in any case happy to make you a guinea pig20:45
clarkbsdague: looking20:45
sdagueyou can actually run tox on that tree20:46
sdagueand it does a thing20:46
cppcabreraI expect them to pass once our deps pass, as well, so hopefully soon. I decided to make those gates non-voting after checking with the rest of team marconi, but yeah, we're happy to start gating on that, clarkb. :)20:47
clarkbsdague: but not the expected thing?20:47
sdaguewell, doesn't seem to work on the test-requirements.txt file20:48
*** cp16net|away is now known as cp16net20:48
*** thomasbiege has quit IRC20:48
sdaguehmmm... actually it works outside of the repo20:49
sdaguelifeless: in testr, setUp is called in the children?20:50
*** vipul is now known as vipul-away20:52
*** dkliban has joined #openstack-infra20:52
lifelesssdague: huh? testr doesn't have any impact on what is called in the backends20:52
lifelesssdague: the only coupling is on test selection (for parallelism) and the output from the children (subunit)20:52
*** woodspa has quit IRC20:53
mordredsdague: setUp is called in each test runner for each test case20:53
sdaguemordred: ok, like I thought20:53
lifelessif the test runner does setUp :) - testr doesn't care20:53
sdaguewell I've gone code blind on why that link above doesn't work20:53
mordredsdague: it says success...20:54
sdaguemordred: pull the repo20:54
sdagueand run tox20:54
*** hashar has quit IRC20:54
sdagueI'm doing the merge of the requirements files into the patch where I added unit tests20:55
mordredsdague: btw - why the use-site-packages and the -U ?20:55
mordredAttributeError: 'NoneType' object has no attribute 'utf_8_decode'20:55
mordredIOError: [Errno 2] No such file or directory: 'tests/files/gr-base.txt'20:55
mordredis that more the error you're seeing?20:55
*** rcleere has quit IRC20:56
mordredor this MismatchError: 'testtools>=0.9.32' not in ['# Imported by ldapdns so required to generate', '# the sample configuration file', 'setuptools_git>=0.4']20:56
sdaguethat one20:56
*** rfolco has quit IRC20:57
*** pcrews has quit IRC20:58
*** SergeyLukjanov has quit IRC21:00
clarkbfungi: I left a comment on that and would be curious to know what you think21:01
openstackgerritA change was merged to openstack-infra/config: Add gerritbot for openstack-glance channel
* fungi is mostly on silent running, but will take a look21:02
openstackgerritA change was merged to openstack-infra/config: Add Python 3.3 gate for Marconi + Client
fungimichael dehaan from ansibleworks is presenting on openstack automation at the moment21:02
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Update jobs launched graph with new paths
marunsdague: ping21:03
clarkbfungi: no rush21:03
clarkbfungi: I don't think we want to merge that change when stuff is busy (because paranoia)21:03
fungiclarkb: commented21:05
*** vipul-away is now known as vipul21:05
marunsdague: I'm seeing a problem with this change:
marunsdague: I was hoping you could explain why it was necessary so I could figure out the best way to approach a solution.21:06
mordredmarun: I'm still working on that one :)21:07
mordredmarun: this is what happens:
marunmordred: is that the same issue?21:07
clarkblifeless: is fungi's comment there a bug with testr?21:07
mordredmarun: yup21:07
marunah, ffs21:08
marunmordred: it breaks rhel, too.21:08
marunmordred: for different reasons21:08
mordredmarun: yah21:08
mordredso, my current approach21:08
mordredis find a different way to fix that21:08
mordredsince that one is doing the'right' thing21:08
mordredand the other thing is a workaround21:08
mordredbut since the workaround is breaking other things21:08
notmynamemordred: I found your picture on the internets!
mordredalso - once sdague and I start running in things - it's possible that this will sort itself21:10
mordredmarun: so it's possible that it's all part of one giant breakage21:10
mordrednotmyname: nice!21:10
mordrednotmyname: wait - which one is me, and which one is jeblair?21:10
notmynamemordred: I think the answer to anything regarding that pictures is "wat?"21:10
mordrednotmyname: fair21:11
clarkbI'm not sure how to feel about that21:11
clarkbEmilienM: are you trying to say we are just one weird party?21:11
marunmordred: so as a short-term solution to keep working in the meantime, is it safe to just comment out unfubar_setuptools?21:11
fungii hope i'm the one with the headband21:12
mordredmarun: on rhel, sure, go for it21:12
mordredmarun: if that fixes things for you21:13
mordredfor the gate in general, nope, that would break EVERYTHING (sadly)21:13
marunmordred: what breakage is unfubar intended to fix?21:13
mordredthe link I just pasted you21:13
mordredthat's what happens without that function21:13
mordredthis is all related to the distribute/setuptools remerge21:13
marunah, so this is crap raining from above?21:14
clarkbbecause thing depend on distribute21:14
clarkbwhich breaks other things21:14
mordredonce we get past it, the world will be better21:14
mordredbut for now, egads21:14
marunif it's not one thing, it's another.21:15
mordredmarun: believe me. I could not possibly agree with you more21:15
marunok, well at least i understand the enormity of the situation now21:15
mordredmarun: sdague and I have been inside of this rabbit hole for a _while_ now21:15
mordredand I'm starting to get stockholm syndrome21:15
mordredsdague: the file does not seem to be being copied well21:15
mordredsdague: I added some addDetails calls:21:16
clarkbjeblair: mordred: zaro: any chance I can get more reviews on that? I would like to flush out the JJB change queue so that new changes coming in can be required to have tests21:16
marunmordred: i don't get it - how on earth is this supposed to be upgradeable?21:18
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add a test periodic job
marunmordred: it seems so incompatible they should have just changed the name of the project21:19
mordredwell - there's a few different things21:19
mordredif you upgrade it by itself, it works21:19
jeblairclarkb: can you approve ?21:19
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Added post-task publisher
mordredunless there are system versions in some cases21:19
jeblairsdague: should fix the graph21:19
marunthe system version screws everything up21:19
mordredmarun: but part of the problem is that what happened was a few years ago, setuptools was forked and called distribute21:20
mordredbut distribute installed itself as setuptools21:20
clarkbjeblair: reviewing and can approve if all looks well21:20
mordredand in fact, monkeypatched any existing setuptools you happened to have21:20
maruntoo clever by half21:20
mordredthen all the distros decided to ship distribute as setuptools21:20
mordredbecause it was more maintained and setuptools was 'dead'21:20
mordredthen, a few months ago21:20
mordredthey reconciled21:20
mordredand merged distribute back in to setuptools21:21
mordredand made distribute go away21:21
jeblairclarkb: (i plan on replacing that almost immediately with actual the jobs and setting the timer to midnight utc if it all checks out)21:21
mordreddstufft: marun isn't too pleased with the setuptools/distribute merge21:21
clarkbjeblair: ok, is that why the name isn't very verbose?21:21
*** thomasm has quit IRC21:21
mordredmarun: actually, most of this would work fine21:21
clarkbI can live with it if that is part of the plan21:21
*** pabelanger has quit IRC21:21
jeblairclarkb: yep.  i'm writing the patch that removes the job now.21:22
mordredmarun: for projects that have explicit dependencies in their install_requires on distribute21:22
zaroi'm not sure what to do about this..
jeblairclarkb: i just want to make sure that things more or less work before i start spamming stable maint21:22
zarodo we need to update the bug or something?21:22
marunmordred: it is a disaster for anything installed from os packages21:22
clarkbjeblair: it failed layout testing21:22
marunmordred: which is an awful lot of the world21:22
clarkbjeblair: looking at that now21:22
mordredmarun: well - only for things installed with os packages that are then overlaying pip packages on top21:22
mordredand it's _worse_ in redhat21:22
marunmordred: very much worse :(21:23
mordredbecause of the way redhat has broken python21:23
mordredby making install install into /usr by default21:23
clarkbjeblair: I think you need to quote the cron spec string?21:23
mordredoverlaying 3rd party and system things21:23
marunmordred: I don't understand that one at all.21:23
mordredso there are about 5 bad thigns all forming a perfect storm here21:23
mordredor, to be fair, 2 bad things21:23
jeblairclarkb: not a bad idea21:23
mordredbut 2 bad thigns are enough21:23
jeblairzaro: i'll leave another comment in that review21:23
mordredmarun: I'm going to fix redhat's python or die trying21:23
burtmordred: when's the wake?21:24
marunmordred: i'd be happy to participate in that effort21:24
*** prad_ has quit IRC21:24
marunmordred: it's unlikely to change in 6.x, but maybe in 721:24
dtroyermordred, marun:  I extracted the unfubar_setuptools() stuff to a stand-alone script to try to get a handle on it away from devstack…it ugly but does basically the same thing on precise, f18 and rhel6:
mordreddtroyer: awesome!21:25
*** openstackgerrit has quit IRC21:25
*** openstackgerrit has joined #openstack-infra21:25
marunmordred: so as a stopgap, the only way I can see rhel working with devstack after this change is having a way to install distribute from scratch, no os packages involved.21:25
dtroyerit's designed to go into devstack/tools21:25
marundtroyer: ah, install from source.  brilliant21:26
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Add a test periodic job
dtroyermarun: that's basically the approach I took, except using the vendor packages to bootstrap it.  that has the side-effect of also working on systems with stuff already installed21:26
jeblairclarkb: ^21:26
dtroyermarun: it's mordred's bits unrolled a little21:26
clarkbjeblair: do we know why the graphite graph on status isn't updating?21:26
mordreddtroyer: you are a magical pony!21:27
dtroyerI'm doing a bit more sanity-checking on fresh installs then I'll propose it...21:27
clarkbjeblair: and I will approve the periodic test if the layout test apsses21:27
jeblairclarkb: this channel is busy today; you can't keep up with gerritbot!21:27
dtroyeronce we're happy with how it works, can call it like install_prereqs works21:27
jeblairclarkb: thx21:27
mordreddtroyer: I thnk you want to make just be an fi21:28
clarkbjeblair: I have had a hard time keeping up all week21:28
mordredor even skip it21:28
jeblairclarkb: (though you can approve it before it passes :)21:28
mordredhrm. no. maybe you're right21:28
mordrednevermind. ignore me - you're totally right21:28
clarkbjeblair: I approved the graphite fix as its already broken and seems low impact21:28
dtroyermordred: we don't have the setuptools tarball there yet…only gets downloaded21:28
clarkbjeblair: ok will approve periodic test21:28
*** dina_belova has joined #openstack-infra21:29
dtroyerthat's why we do it again at line 9221:29
mordreddtroyer: this is truly amazing isn't it?21:29
dtroyerof course, if leaving distribute there is ok, the bits at 92 can go away21:29
openstackgerritA change was merged to openstack-infra/config: Update jobs launched graph with new paths
dtroyermordred: I'll get shot around here for saying this, but python packaging makes CPAN look like a dream21:30
mordreddtroyer: no you won't21:30
mordreddtroyer: dstufft, who runs pypi, might even agree with you21:30
mordredalthough, fwiw, we _are_ working on trying to fix it21:30
dtroyerfortuantely I didn't know he ran that when I met him ;)21:30
openstackgerritA change was merged to openstack-infra/config: Add a test periodic job
dstufftI didn't run it when we met :)21:31
*** dkliban has quit IRC21:31
dstufftI've become an admin since then21:31
dstufftAlso what's PyPI doing wrong now21:31
dtroyerwell then, you've taken on quite the challenge!21:31
marundtroyer: have you tested with centos/rhel?21:31
* mordred goes to test on fedora21:31
dtroyerI'm getting ready to on fresh installs.  I did run it on already-devstack-ed vms21:31
marundtroyer: does 'python' have any dependency on having setuptools installed?21:32
marun(sorry if that's a dumb question)21:33
mordredmarun: it does21:33
dstufftWheels don't21:33
dstufftmarun: We're trying to make it better? ;/21:33
mordreddstufft: nothing - we're battling setuptools right now21:33
*** dina_belova has quit IRC21:33
mordredmarun: oh - wait21:33
mordredmarun: not the for setuptools21:34
mordredmarun: it just uses distribute/itself21:34
dstufftit uses itself21:34
mordreddstufft: we've found GLORIOUS issues21:34
mordreddstufft: also - because I'm yelling at everyone ...21:34
dstufftJason is on vacation in scandanavia21:34
mordreddstufft: redhat has changed python so that it installs into /usr by default21:34
sdaguemordred: ok.... sorry had phone things21:35
marunmordred: …because installing things not from packages is bad, mmkay?  and not just because we've made it dangerous!21:35
sdaguemordred: so I'm doing this -         shutil.copy("tests/files/test-project.txt", self.proj_test_file)21:35
burtdtroyer: whoops, tried to source it.21:35
dtroyerscript fails spectacularly on fresh installs21:35
sdaguehow can that not be working?21:35
dstufftmordred: hrm?21:35
dstufftyou mean /usr/lib/python2.7/site-packages21:36
dstufftor w/e the fuck the url is21:36
dstufftThat's actually where Python installs by default21:36
dstufftDebian patches their python21:36
mordredI've been yelling the wrong things then21:36
dstufftso it installs to /usr/local/lib/python2.7/whatever21:36
mordredso - the problem this uncovered today21:36
mordredis that "yum install python-pip ; pip install -U pip ; yum  remove python-pip" breaks things21:37
openstackgerritMatthew Treinish proposed a change to openstack-dev/pbr: Add option to run testr serially
dtroyerI've been thinking that the redhat issue was mixing site-packages and dist-packages, or maybe not using dist-packages when we expect them to21:37
mordredsince pip is overlaying what yum is doing21:37
*** cppcabrera has left #openstack-infra21:37
dstufftdist-packages is a strictly debian thing21:37
mordreddist-packages is another debian thing - which I _really_ don't understand21:37
dstufftwell ubuntu and shit too, but that's because of debian21:37
dtroyeriirc there was a long argument about that somewhere I foudn a while back and the result was "this is the way it is"21:37
fungiclarkb: jeblair: i had a couple comments to that effect in the timer triggers change... basically recap cron's field order and also document that sepcial sixth "seconds" field21:37
fungier, special21:37
dstufftmordred: I think plan was apt-get would be dist-packages, and pip install would be site-packages21:38
dstufftbut it got fucked up21:38
dstufftor something21:38
mordredso - that would work and help21:38
dstufftDo you need to pip install things at the syste level?21:38
jeblairfungi: yeah, i'm not sure i want to document seconds; i just put it in there so the unit tests don't take >1 min; though i suppose we could.21:38
mordredlike, globally?21:38
mordredthat's how all of the devstack gate works21:38
dstufftI should probably know this stuff by now21:39
marundstufft: hard to run things as root if they're not installed at the system level21:39
mordredthere are a LOT of moving pieces21:39
mordreddstufft: basically, openstack gave up on distro packages as a basis for deployment 2 years ago ish21:39
dstufftmordred: so why not just uninstall python-pip and install from get-pip.py21:40
mordreddstufft: that's essentially what we're working on now21:40
dstufftand the setuptools equivilant21:40
clarkbjeblair: mordred fungi I am running the image update job in az3 now. The devstack changes appear to have merged21:40
mordreddstufft: but, I mean, holy crap this is all broke as hell21:40
reedjeblair, can you join the discussion on #openstack-community?21:40
mordredclarkb: excellent21:40
*** mrmartin has joined #openstack-infra21:40
*** fifieldt has joined #openstack-infra21:40
dstufftmordred: Well Linux will just say you shouldn't be using pip to install things globally21:41
marunwait, why isn't being used?21:41
marundstufft: linux who?21:41
dstufftmarun: any of the distros21:41
marundstufft: it works on ubuntu and debian (except for this mess)21:41
dstufftThe people who work on them yell at me all the time saying pip shouldn't fuck with system packages21:41
sdaguedstufft: the reason we need to in the gate, is to ensure that the resultant could be packaged21:41
dstufftto which I tell them idc21:41
marundstufft: that's the point of isolating pip install to /usr/local21:41
dstufftbut that's what they say :)21:42
marundon't mess with system packages21:42
* fungi disappears again for a bit21:42
sdaguemordred: ok, can I steal you back for a minute?21:42
mordredsdague: YES!21:42
sdagueso... those unit tests21:42
sdagueis there a better way to copy the files over?21:42
dstufftmarun: yea, Debian did that to prevent these kind of issues afaik, not because they like the fact people are doing it21:42
sdaguebecause I just don't understand why that would fail21:43
mordreddstufft: well, sure - but they all have a distro view21:43
mordreddstufft: having an upstream view is much different, it turns out :)21:43
marundstufft: yes, the distro view.  but not everything gets os packages21:43
marundstufft: at least they gave the rest of us an out21:43
marundstufft: rh, not so much21:43
dstufftmarun: Yea I bitch at Nick from time to time about it21:43
dstufftI think that's compiled in :[21:44
burtdtroyer: should I just replace the unfubar line with in to test this?21:44
*** pentameter has quit IRC21:44
mordredsdague: sorry - still poking trying to figure out why it's not copying21:44
dtroyerburt: not yet.  it doesn't work on fresh installs21:44
burtokey dokey.21:44
mordreddstufft: at the start of openstack, we made distro packages for everything and did our installs using them21:44
mordreddstufft: then ubuntu got mad at us, saying that was their job21:44
mordreddstufft: and redhat agreed with them, saying it was their job21:44
mordredso we quit21:44
mordredand adopted pip21:44
mordredI'm 3 seconds away from telling everyone to shove it, we're going to start making packages for all the distros and hosting our own apt/yum repo...21:45
sdaguemordred: you should make ebuilds21:45
dtroyerok, so I've got a process to bootstrap pip/distribute on win***s, we want setuptools instead right?21:45
* sdague ducks21:45
marunso why can't be used?21:45
mordredsdague: any time someone can show me how to run a stable gentoo system ...21:45
sdaguemordred: now you're hunting unicorns21:46
dtroyermarun: I think that's what I'm looking for21:46
dstufftdtroyer: you pretty much always want setuptool now21:46
dstufftdstribiute is merged back in and dead21:46
dtroyerok, thanks21:46
mordreddtroyer, marun: if we use git-pip or direct downloading...21:46
mordredwe'll wnat to make sure we can re-direct it at our mirror21:46
mordredand/or pre-downloaded things21:46
dstufftget-pip uses a bundled copy of pip21:46
dstufftin order to install from pypi21:47
mordreddoes it honor ~/.pip/pip.conf ?21:47
dstufftdon't think it supports command line args, probably still supports the config file and env vars21:47
mordredwe can have the image build job do get-pip21:47
mordredand stash the results locally21:47
mordredso that devstack can just consume them21:47
dstufftoh maybe it does21:48
dstufftthat's what it runs21:48
dstufftjust unpacked a local pip and runs pip.bootstrap()21:48
dtroyerdstufft: is that the place we should get it from?21:48
dtroyerI mean, in scripts and all21:48
dstufftyou'll want to make sure any system package is uninstalled before running it21:48
marun…and make the system cough if someone tries to install os packages on top...21:49
dtroyermordred: we need to handle that for devs too, that's why I wanted the stand-alone script21:49
*** burt has quit IRC21:49
marunit may require setuptools already installed21:49
dstufftyea it does21:49
clarkbmgagne: you about? does puppet default to using root as the owoner?21:50
mordreddtroyer: totally. I just want to make sure that we always have the opportunity to pre-fetch anything that devstack wants to grab from the internets21:50
mordredand/or direct things to our mirror21:50
maruninstall to custom locations!!
mgagneclarkb: puppet defaults to the user running puppet, most of the time it's root.21:51
mordredsdague: I'm not sure it's a file copy problem21:51
sdaguemordred: ok21:51
mordredsdague: I think it might actually be a bug in update.py21:51
mordredlike, I think the test might be telling us something :)21:51
sdagueok, I'm happy with that as the answer21:52
sdaguebut like I said, I think I went code blind :)21:52
mordredsdague: can you explain to me what you did with global-requirements and test-requirements ?21:52
sdagueso I needed second eyes21:52
dstufftI don't know if it helps at all21:52
dstufftare you aware of --user?21:52
sdaguecopy requirements.txt global-requirements21:52
sdaguecat test-requirements >> global-requirements21:52
sdagueplus a couple of comment lines seperating21:52
mordreddstufft: yeah - but we're installing clouds here- we're trying to test that a system level install of a cloud works21:53
dstufftmordred: ok21:53
mordredwe _explicitly_ want to install system wide21:53
*** bnemec has joined #openstack-infra21:53
dstufftalterantively install your own Python in /opt :V21:53
dstufftand tell RedHat's to go fuck off21:53
sdaguedstufft: then we need to own python updates21:53
sdaguewhich is not desirable21:54
mordredright. we're trying to test that people can install this in a general way21:54
mordrednot that we can instal it knowing special things21:54
sdagueand that the distros can actually build a packaged version21:54
mordredwell, I'm caring less and less about that ...21:54
mordredbut yeah21:54
sdagueI still think that's *very* important from an ecosystem perspective21:55
sdaguebut that's beer talk21:55
sdaguelets get back to bug21:55
sdaguebecause I think it's all that stands between us and magical ponies21:55
mordredit may not look like it - but I am actually trying to step through it21:55
mordredyes. I agree21:55
mordredso many ponies21:55
mordredmillions of dancing ponies21:55
mordred(now imagine millions - like, actually _millions_ of ponies dancing all in a mob)21:55
dstufftmordred: if you have contacts at RH tell them to go adopt the ddebian patch :D (not it fixes the problem right this second but)21:56
EmilienMHi, I need some review to push puppet-heat into StackForge > &
mordreddstufft: I'm going to21:56
mordreddstufft: I'll be calling ... people21:56
clarkbmgagne: but it looks like it won't change it from the existing value if the file exists?21:56
clarkbmgagne: I did a quick puppet agent --noop test and it didn't seem to want to change those permissions21:56
mordreddstufft: I'd try to convince python to change its behavior...21:56
dstufftmordred: in the future this will be.. better even if they refuse to.21:56
mordreddstufft: but something tells me that even if I were successful...21:57
mordreddstufft: it would still onlybe 2.521:57
dstufftmordred: It would only land in 3.4 even if you were successful *today*21:57
clarkbjeblair: can we reenable puppet on jenkins.o.o?21:57
mordreddstufft: what if pip just magically started installing things into /usr/local ...21:57
mordreddstufft: you have the power to change the world... abuse it!!!21:57
dstufftmordred: the problem is we just ask python where it should install things at, this is how the debian patch works (it patches Python), and it's how virtualenv works, and how installing python into /opt works etc21:58
dstufftWe'd have to detect if we're using a global python, and reason about if /usr/local made sense, and that sounds like it has massive breakage potentional21:58
* marun is boggled that zip and unzip are separate commands/packages in rhel21:59
clarkbmordred: safe to approve ?21:59
clarkbmordred: I am not up to speed on todays "The State of PBR"21:59
dstufftI'll send an email to Nick too whine again22:00
openstackgerritA change was merged to openstack-infra/config: Set ownership of JJB config file to root
mgagneclarkb: could be that puppet won't change the ownership of a file if it exists already but would default to root if it has to create it. Want me to enforce root ownership?22:00
clarkbmgagne: no, I think the non enforcement is fine22:00
clarkbmgagne: as it will grandfather in people using jenkins22:00
clarkbmordred: I am going to approve it, you can -2 if it will break the world22:02
mordredclarkb: totally +2/aprv22:02
mordredtotally safe22:02
dstufftmordred: It's really not a Python problem either, Python expects to own it's tree (which it does if you install from anywhere other than a package manager). Linux Distros takes away control over part of it's tree and RH doesn't provide a non packager owned place. (ACtually discovering this was a big revalation as to why the RH people I knew got so angry about people using pip at the global level)22:02
mordreddstufft: that's an excellent point22:03
*** bnemec has quit IRC22:03
jeblairmrmartin: meet clarkb, mordred, and fungi, the other core infrastructure people22:03
clarkbmrmartin: hello22:03
*** bnemec has joined #openstack-infra22:03
mrmartinhi guys22:04
jeblairmrmartin: will be working on the drupal user group system with reed22:04
jeblairin #openstack-community, i was explaining how to get started with our system22:04
jeblairit sounds very promising22:04
jeblairmrmartin: sdague is also interested in this project; he knows a lot about our infrastructure and does some things with drupal in his spare time22:04
datsun180binstalling to /usr/local/lib wouldn't be a problem if sys.path in python didn't somehow kick /usr/local/lib to below /opt/stack/* and /usr/lib/, from what i'm seeing in my own environment22:05
mrmartinthe first sprint of the work will involve the ci system, we want to integrate drupal and use the same processes as python projects.22:05
jeblairmrmartin: we love running tests, and lots of them. :)22:05
datsun180bcontext: i work in trove and i'm very silly in that i'm building things with vagrant too, so what i'm saying may be at right angles to the rest of you22:05
marundstufft: translation: rh people angry at themselves for making a mess of things.22:06
mtreinishjeblair, fungi, mordred: can I get a quick review on: so we can save tempest logs from runs22:06
dstufftdatsun180b: on Debian /usr/local/ is before /usr/22:07
dstufft['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL']22:07
mrmartinok, I start to prepare the project, and I'll spend the next two weeks with you guys finding out how to integrate this platform.22:07
*** reed has quit IRC22:07
mrmartinthanks for your support.22:07
jeblairmtreinish: done22:08
dstufftdatsun180b: hrm, well mine's Ubuntu specifically22:08
mtreinishjeblair: cool thanks22:08
clarkbEmilienM: the puppet-heat change lgtm22:09
EmilienMclarkb: thx22:09
EmilienMclarkb: if you could also review
datsun180bhrm indeed22:10
clarkbEmilienM: done22:10
EmilienMclarkb: cool22:11
*** beagles has quit IRC22:11
EmilienMclarkb: I have another question. Does it make sense to submit a new repo in StackForge with monitoring Scripts for OpenStack, which could be use by many monitoring system like Nagios or other tools ?22:12
clarkbmordred: sdague I thought oslo.config was consumed through releases and not git22:12
*** _TheDodd_ has quit IRC22:12
clarkbmordred: sdague was that changed during the great requirements upheaval?22:12
sdagueclarkb: yes22:12
sdaguethink of it like the python-*clients22:12
*** whoops has quit IRC22:12
clarkbsdague: ok22:12
sdaguealso see my long email on list :)22:13
clarkbEmilienM: if you feel that the infrastructure behind stackforge would be useful then yes22:13
dstufftmordred: so I emailed nick asking what it would take to get RH to do what debian did22:13
clarkbEmilienM: stackforge has been called infrastructure as a service... If that is useful to you then yes, If you end up with a singel reviewer and use the gate-noop jobs then probably not22:13
dstufftThat's about my only RH contact I think :[22:14
mordreddstufft: well, he's not a bad one in this world22:14
dstufftspecially since he's involved in packaging on the python side too :D22:14
clarkbsdague: in that change is there a reason we included neutron but not postgres?22:14
*** changbl has quit IRC22:14
mordreddstufft: next pycon, we should really sit down and try to work through some of the uptream vs. distro stuff from a python pov22:15
*** rnirmal has quit IRC22:15
dstufftMaybe CPython can make some sort of policy or document for people packaging it that they should do /usr/local22:15
mordredif we had a policy that made sense taking both sides in to account22:15
* clarkb jumps into the gray. Didn't arch start unifying /usr22:15
dstufftdoesn't stop the pain today, but might make it better in the future22:15
sdagueclarkb: because there shouldnt' be any difference between mysql & postgresql jobs with that22:16
clarkbsdague: I see22:16
*** mrmartin has quit IRC22:17
*** lcestari has quit IRC22:19
clarkbmordred: any idea why unittests failed there?22:19
clarkbdoesn't seem related to the change22:19
jeblairclarkb: puppet restarted on zuul and jenkins22:19
clarkb so that is happening22:20
clarkbjeblair: thank you22:20
clarkbAlex_Gaynor: ^ does that look familiar to you?22:20
mordredclarkb: it's an oops in the dependent package22:21
mordredclarkb: it's something in the unittests - the functionality works22:22
clarkbmordred: in cffi22:22
clarkboh you are referrng to pbr22:22
clarkbI am context switched into broken xattr beacuse the gate is going boom!22:22
clarkbmordred: look at the gate-swift-docs thing above22:24
mordredclarkb: it's a distutils/setuptools thing22:24
mordredhow did that happen on one of our nodes?22:24
clarkbthey just pushed a new release22:24
mordredare we not using up to date tox everywhere?22:24
clarkbits not setuptools its cffi22:25
clarkbor at least new cffi released just now is tickling a problem22:25
mordredoh. nop22:25
mordredI'm sorry, you're right22:25
clarkbso uh ya22:25
clarkbwe should find a way for maybe the mirror to let us know when this happens22:25
clarkbshould we have it email the infra list when it fails to install a thing?22:26
mordredassert backend.__version__ == __version__22:26
* clarkb tries to reproduce locally to see if debugging can be done22:26
mordredclarkb: probably22:26
jeblairclarkb, mordred: er what?22:27
jeblairi'm pretty sure emailing someone is not the right kind of automation22:27
clarkbmordred: hmm is that complaining that libffi version != cffi's version22:27
clarkbjeblair: well we need something that tells us it may have broken22:27
mordredclarkb: you've got to be kidding me22:27
clarkbjeblair: but we want it to update the mirror anyways22:27
clarkbmordred: going to test22:27
*** mrodden has quit IRC22:27
lifelessclarkb: hi, looking22:27
lifelessclarkb: (I had to drop C off to kindy then get my monthly allergy shot)22:27
*** mriedem has quit IRC22:28
dhellmannclarkb: we don't have an existing repo with pycadf code; we're waiting for this one to be approved so we can avoid adding the code to the ceilometer code base22:28
mtreinishclarkb: yeah arch moved things from /bin and /sbin into /usr:
clarkbpip install cffi works22:29
mtreinishbut /usr/local still exists22:29
*** plomakin has quit IRC22:29
jeblairclarkb: i'm not up to speed enough to say anything helpful, but i would like to avoid having "email someone" be a design goal of our automation.  what kind of breakage are we talking about?22:29
mordredsdague: found it22:29
clarkbdhellmann: mtreinish gotcha22:29
*** emagana has quit IRC22:29
dhellmannclarkb: after the repo is created, I'll set up the packaging stuff as the first commit22:29
*** plomakin has joined #openstack-infra22:29
lifelessclarkb: no, it's not a testr bug22:29
clarkbjeblair: is breaking because the mirror scripts pushed cffi 0.7.122:29
mordredsdague: gr-base.txt is a file from before you added test-requirements to requirements.txt22:30
jeblairclarkb: did the mirror scripts to the wrong thing?22:30
sdaguemordred: oh... ffs22:30
clarkbjeblair: I think so, I haven't confirmed but it did the last time cffi broke22:30
sdagueok, I'll fix it22:30
lifelessclarkb: the .testr.conf specifies ${PYTHON:-python} as the thing to run.22:30
clarkblifeless: aha22:30
clarkbjeblair: yes the error occured in the mirror script22:31
jeblairclarkb: what do you want the mirror script to do instead?22:31
lifelessclarkb: so either having the venv activated (so python will be on the path) or specifying the actual binary to run, is needed.22:31
clarkbjeblair: make it known that something potentially broke22:32
*** vijendar has quit IRC22:32
clarkbI cannot reproduce locally22:32
lifelessjeblair: so22:32
mordredsdague: I've got a fix in tree...22:32
lifelessjeblair: is now an ok time to talk about the mirror list thingy?22:33
*** krtaylor has quit IRC22:33
jeblairclarkb: so you don't want to alter the behavior of the mirror script?22:33
lifelessjeblair: and/or rsync or something access to the mirror ?22:33
*** ogelbukh has quit IRC22:33
jeblairclarkb: as in, you don't want to either upload or not upload something differently than what's happening now?22:33
clarkbjeblair: if possible I would like to not upload the one package that failed22:33
clarkbjeblair: but pip makes that very hard aiui22:33
jeblairclarkb: so istr that we wanted not to gate the world22:34
mordredjeblair: right, but we do want to make our mirror a good thing22:34
jeblairclarkb: and that if upstream packagers broke our expectations, we wanted that to propogate to the projects22:34
jeblairmordred: is it bad?22:34
mordredjeblair: and we're hitting problems now where a package will update22:34
mordredwe'll get _half_ of it22:34
clarkbjeblair: it is bad to not upgrade everything else because xattr and cffi are derping22:34
mordredand one of its deps will not build on our mirror builder22:35
mordredbut we'll still upload part of the upgraded thing22:35
mordredeven though things failed22:35
clarkboh, I can reproduce, you have to install xattr not cffi22:35
clarkbxattr has not updated recently, so I think cffi update may have broken xattr22:35
jeblairmordred, clarkb: if we were not using a mirror, would this be a problem?22:35
clarkbjeblair: yes22:36
mordredah - my bad22:36
mordredI thought the other22:36
mordredin that cas,e I agree with jeblair22:36
clarkboh you mean the badness and not xattr, in that case no22:36
dstufftmordred: ok, got the path forward from Nick, I'm gonna have to join a fedora mailing list (exactly what I was hoping for!)22:36
jeblairit doesn't sound like a problem with the mirror.22:36
mordredI thought this was a mirror builder issue that was producing a state in the mirror that was not what was happening in the world22:36
mordreddstufft: yay!22:36
*** weshay has quit IRC22:36
mordredjeblair: agree22:37
clarkbmordred: jeblair: what is happening is the mirror is letting through brokeness22:37
clarkbit should not do that22:37
mordredit should22:37
mordredit should mirror the world22:37
mordredif xattr is now broken22:37
clarkbI thought the whole point of the mirror was to protect against stuff like this?22:37
mordredwe need for swift to know to pin it22:37
mordredit's to protect us against flaky network connections22:37
mordredand stuff22:37
clarkbok, so back to the email suggestion. We want to know when the mirror lets this stuff through22:37
mordredour mirror having content different than the world at large is bad22:38
clarkbwe don't need to wait for swift, xattr does not install something should set off a klaxon22:38
jeblairclarkb: the thing that's like what you're suggestion that the mirror should do is protect us from putting _ourselves_ in this situation by having bad deps22:38
jeblairbut if someone else makes them bad, we need to know about it.22:38
mordredthat I agree with22:38
jeblairmordred: what do you agree with?22:38
jeblairmordred: ping22:39
mordredjeblair: that we want the mirror to proect us from putting _ourselves_ in this situation22:39
mordredlifeless: cinder has run in to this:
mordredlifeless: which I'm pretty sure is due to the test having an id attribute22:40
clarkbjeblair: should the mirror try to be more proactive in some way when it knows the world is broken?22:40
jeblairclarkb: i think the current notification mechanism is very effective.  i'm not interested in getting emails when something fails to install.22:41
clarkbjeblair: or will we just let it fall through?22:41
*** bnemec has quit IRC22:41
lifelessmordred: all tests have an id function, it's part of the contract of being 'a test'22:41
clarkbAlex_Gaynor: xattr is broken with new cffi on precise22:41
mordredlifeless: but I've been asked by harlowja why it's bombing there22:41
mordredlifeless: no, I mean they have a attribute called id22:41
Alex_Gaynorclarkb: grumble, thanks for the heads up, investigating22:41
mordredthat is not the function22:41
lifelessmordred: oh, you can't do that. It will break things.22:41
lifelessits reserved.22:41
jeblairclarkb: most of the time fixing something like this involves a requirements change.  i really don't want to be seen (or actually be) responsible for fixing things every time some pypi upstream package breaks.22:41
jgriffithmordred: lifeless I think the point is why did it trigger suddenly here?22:41
mordredyes. that's what I said22:41
jgriffithmordred: haha22:41
harlowjaya, seems odd22:42
mordredlifeless: the question is what jgriffith asked22:42
jeblairclarkb: we have empowered everyone in the project to fix this situation.22:42
jgriffithmordred: sorry.. not to cut in, but I'm extremely curious22:42
lifelessoh, well if you were using a runner that happened to not call id() it would not show up22:42
mordredjgriffith: take it away... I'm just channel you22:42
*** bnemec has joined #openstack-infra22:42
clarkbAlex_Gaynor: if I pin cffi to 0.7.0 before installing xattr that works22:42
mordredlifeless: I believe they've been on testr/testtools for a while22:42
harlowjawho am i channeling?22:42
jeblairclarkb: if it should email anyone, it should probably email openstack-dev.22:42
Alex_Gaynorclarkb: go ahead and do that for now, i tlooks like the whole 0.7.1 release is broken22:42
* harlowja must channel someone to22:42
lifelesssame as if you broke but didn't happened to call it22:42
mordredAlex_Gaynor: isn't this the second time in like 2 weeks that cffi broke something?22:43
jgriffithlifeless: not an explanation22:43
dtroyermordred, dstufft, marun: dinnertime, but before I do here's a vastly simplified    It needs to get the downloaded scripts locally first if available so they can be pre-cached and I'm not happy with the version detection.  but it works a hell of a lot better.22:43
clarkbjeblair: I don't want to be responsible either. I am on board with this. I really thought other people thought differently and was one of the reasons I didn't want to use the mirror for strong enforcement of requirements22:43
clarkbbut I am fully grokking now (some months later apparently)22:44
jgriffithlifeless: without this patch, tests run fine22:44
lifelessjgriffith: so before the test is run the runner asks for the id(); I don't think it asks for it again unless you trigger a failure.22:44
jgriffithlifeless: with this patch... barf22:44
jgriffithlifeless: ahhh22:44
jgriffithlifeless: now that helps22:44
lifelessjgriffith: the patch is broken, and it's triggering a cascade error which shows up the fact that you'd been violating the contract.22:44
Alex_Gaynormordred: I don't believe so, no.22:44
*** ogelbukh has joined #openstack-infra22:44
mordredor was it ffi22:44
harlowjalifeless just odd that the cascade would show up in a different test22:45
jgriffithlifeless: thanks22:45
lifelessnow - its ugly that the Test contract impinges on your objects, but it is what it is.22:45
mordredclarkb: didn't we have a broken release a few weeks ago with something?22:45
jgriffithharlowja: since we're trying to be fancy and paralell etc it's not so odd I don't think22:45
*** datsun180b has quit IRC22:45
harlowjathread locals for id?22:45
lifelessharlowja: hmmm, it doesn't does it ?22:45
clarkbmordred: yes xattr switched to cffi and there were two problems. One cffi requires libffi-dev and two it didn't build on our linuxes22:46
dstufftmordred: 5 minutes in and Nick already told me Openstack shouldn't use global pip and should sue a software collection instead! lol22:46
harlowjalifeless ok, let me see what i can do to fix that, interesting to know22:47
*** mrodden has joined #openstack-infra22:47
marundstufft: Nick who?22:49
lifelessharlowja: we can't easiy see what test it broke on, because we're in a try:except: block in the framework22:49
dstufftmarun: Nick Coghlan22:49
lifelessharlowja: there is no reason to think it is failing and reporting in different tests - that would be super odd22:50
dstufftmarun: He's a friend who works @ RH, and is agaisnt using pip at the global level, needless to say I don't really agree with him :)22:50
dstufftbut thems the breaks between linux distros and language specific packagers ;P22:51
marundstufft: I work at RH, and I think railing against global installation with pip is conflating a symptom with the disease22:51
dstufftmarun: well then !22:51
dstufftmarun: I'm going to be emailing uh, about why they should patch their python to install to /usr/local22:51
dstufftlater on tonight probably22:52
locke105where are the scheduled downtimes for -infra systems posted? i can't seem to find them...22:52
clarkbharlowja: I believe mordred has a quick one liner that will run each test in its own test runner one at a time22:52
clarkbharlowja: may help in finding the problem test22:52
clarkblocke105: right here22:52
harlowjaoh, good idea22:52
dstufftfor now i'm going to get some dinner22:52
clarkblocke105: usually in this channel, and #openstack-dev and #openstack. Planned outages will be annoucned to the openstack-dev and possibly openstack mailing lists depending on impact22:53
clarkblocke105: and I suppose a prolonged unplanned outage would get mail list treatment as well22:53
locke105clarkb: ok, was looking for like a schedule for future downtimes22:53
locke105unless there isn't one22:53
harlowjamordred do u have said script memorized? :-P22:54
clarkblocke105: we typically don't need to plan that far in advance as long as we avoid milestone and release candidate weeks :)22:54
clarkblocke105: you need an outage for the pypi-mirror rename right?22:54
locke105yeah was wondering when that was happening22:54
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Support periodic jobs in log custom function
jeblairclarkb: can you aprv that ^ ?22:55
*** pcrews has joined #openstack-infra22:55
clarkbjeblair: yup22:55
clarkbjeblair: thoughts on outage time for locke105 ?22:55
jeblairclarkb: that's the cause of the test jobs not running (and backing up, and throwing exceptions in the zuul log)22:55
jeblairclarkb, locke105: i think mordred was going to think on that, but i believe we've all been overwhelmed this week so far.22:56
locke105on an unrelated note, i can't seem to reach
mordredlocke105: yeah - sorry man - it's been a week in setuptools hell22:56
clarkbjeblair: should line 6 be an elif?22:56
locke105lol np22:56
jeblairlocke105: it has no services, but you can see the dashboard at
jeblairclarkb: yes, thanks.  :)22:56
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Support periodic jobs in log custom function
jeblairclarkb: i'm clearly about burned out for the day myself.22:57
clarkbjeblair: me too22:57
locke105jeblair: oh nice, was just wondering if were aware that URL wasn't working or not22:57
clarkbthis week has been weird22:57
clarkbI think getting up at 0530 yesterday didn't help22:57
clarkbjeblair: approved22:58
locke105who would have thought setuptools could cause so much chaos...22:59
mordredjeblair: +100022:59
openstackgerritA change was merged to openstack-infra/config: Support periodic jobs in log custom function
*** zaro0508 has joined #openstack-infra23:02
clarkbAlex_Gaynor: I have confirmed that import cffi ; cffi.FFI() is sufficient to trigger that assertion. cffi seems broken23:02
jeblairperiodic jobs are running now23:02
clarkbAlex_Gaynor: will you guys be pinning upstream?23:02
clarkbjeblair: \o/23:02
clarkbmordred: image built23:03
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Set periodic trigger to midnight UTC
clarkbmordred: does that look like a broken image to you?23:04
Alex_Gaynorclarkb: a new release has been issued which resolves it23:04
jeblairclarkb: one more speedy aprv for today please, and then i'm going to defer non-fixup changes till tomorrow:
Alex_Gaynorclarkb: purging 0.7.1 from any chcaces should be sufficient23:05
clarkbjeblair: ko23:05
*** avtar has quit IRC23:05
clarkbAlex_Gaynor: good to know I will rerun the mirror updates now to get new xattr then I can kill new cffi23:05
clarkbafter I approve jeblair's thing23:05
clarkbjeblair when you have a moment can you take a look at that tempest job23:05
jeblairclarkb:  seems to work btw23:06
jeblairclarkb: looking23:06
clarkbjeblair: can we shift the periodic jobs to alter in the day?23:06
clarkbjeblair: our quiet time is from about 11pm to 2 or 3am PST23:06
clarkbmaybe run it at 6UTC?23:06
jeblairclarkb: sure, let's do in the change where we actually add the jobs though23:07
jeblairclarkb: i mostly want to just stop running the zuul tests every 5 mins now.23:07
*** emagana has joined #openstack-infra23:08
*** bnemec has quit IRC23:08
clarkbmirror jobs running23:08
jeblairclarkb: have any other jobs run on az3?23:08
*** bnemec has joined #openstack-infra23:08
openstackgerritDan Bode proposed a change to openstack-infra/config: Add puppet-pip
clarkbjeblair: I am not sure23:09
jeblair2013-08-01 22:39:29.076 | 2013-08-01 22:39:29 error: invalid command 'egg_info'23:09
jeblairclarkb: i mean, that looks fishy.23:09
bodepdlet me know if you  guys have questions about that commit23:09
openstackgerritA change was merged to openstack-infra/config: Set periodic trigger to midnight UTC
clarkb^ ignore that one23:10
uvirtbotclarkb: Error: "ignore" is not a valid command.23:10
bodepdI don't have permission to release things to the forge under openstackci23:10
bodepdso I will need a little help to get this done23:10
clarkbjeblair: however that may be on an old image23:10
*** dims_ has joined #openstack-infra23:11
*** bnemec has quit IRC23:12
clarkbcffi has released a newer version...23:12
clarkbAlex_Gaynor: new version appears to fix the bug23:14
jeblairclarkb: when was the new image in place?23:14
* clarkb goes back to sorting out az323:14
clarkbjeblair: ~215523:14
*** bnemec has joined #openstack-infra23:14
jeblairclarkb: we should have d-g log which image a machine was build from23:14
jeblairclarkb: that was launched at 22:20:1623:17
jeblairclarkb: and seems to be working23:17
clarkbI see it in syslog23:18
clarkband the timestamps indicate we are good23:18
clarkbI am checking the node with the failure now23:18
jeblairbodepd: i think we will be happy to help when we are not slammed :)23:18
bodepdjeblair: no worries23:18
bodepdjeblair: let me know23:18
clarkb177734 was also launched after the new image so potentially related23:19
jeblairclarkb: oh, the failure you pointed out is on stable/grizzly23:19
clarkbjeblair: the id numbers would indicate that hosts created after the new image by and large work23:20
jeblairclarkb: perhaps the new image doesn't play well with that branch?23:20
clarkbjeblair: oh interesting.23:20
clarkbthat is possible23:20
*** bnemec has quit IRC23:22
jeblairclarkb: there's a 1:1 correlation with az3 nodes and that failure for that change, and the converse for !az3 and not having that failure23:22
*** bnemec has joined #openstack-infra23:22
jeblairclarkb: i think it breaks stable/grizzly, and we should revert the change to puppet23:22
clarkb may give us more info23:23
clarkbjeblair: ok I will whip that up. I think we should approach this by refactoring the d-g base out of the same base of the rest of our servers which is what you were talking about this morning23:23
*** changbl has joined #openstack-infra23:24
jeblairclarkb: yeah, we should let devstack handle this kind of thing and keep the bare slave more bare.  :)23:24
openstackgerritClark Boylan proposed a change to openstack-infra/config: Revert "Ensure latest setuptools after installing pip."
jeblairclarkb: i think we can remove this pip stuff from our base class, and put it just where it's needed for our servers for now (jeepyb and zuul)23:25
jeblairclarkb: then put it in the jenkins slave manifest for bare==false23:26
jeblairso the unit test slaves get it23:26
jeblairthen count on devstack to do this for d-g runs23:26
jeblairclarkb: i also rechecked 39511 (even tho it's in the gate queue) because i expect that to finish before the gate jobs.23:27
clarkbjeblair: sounds good23:28
clarkbjeblair: we can do a quick grep for provider => pip and add it there23:28
jeblairit has 3/5 runs in az323:28
*** pabelanger has joined #openstack-infra23:30
jeblairclarkb: all the az3 runs have failed with that error, non az3 are continuing23:30
ianwdtroyer: i'm going to investigate making a dummy rpm dependency, that's about the best thing i can think of?23:30
jeblairclarkb:  i approved the change; you want to kick off a new image build when it merges?23:31
*** toddmorey has quit IRC23:31
openstackgerritA change was merged to openstack-infra/config: Revert "Ensure latest setuptools after installing pip."
jeblairclarkb: ^23:32
jeblair#status alert testing for stable/grizzly branch is broken on some devstack-gate nodes, fix in progress23:33
dtroyerianw: I just looked at my really-really-bare centos image and removing python-setuptools even took cloud-ini with it.  Ugh …23:35
*** bnemec has quit IRC23:35
jeblairclarkb: i kicked off an image build for az323:35
*** bnemec has joined #openstack-infra23:36
clarkbjeblair: thank you, I got distracted23:37
dstufftdtroyer: lol23:37
dstufftdtroyer: I promise this is going to get better, atleast on the python side :[23:38
clarkbwould a single giant venv manage to do what the ditros need as far as packages goes?23:38
clarkbeverything is known to work with the system C libs, all the packages installed together work, but you avoid the global root?23:39
*** pcrews has quit IRC23:39
dstufftwon't tell you if versions clash with system packages, but IMO that's the distro's problem to solve23:40
ianwclarkb: how do you get all the daemons to run in the virtualenv?23:40
dstufftianw: /path/to/venv/bin/daemon-name23:40
clarkbianw: at the beginning of devstack's run it creates a virtualenv and source the activate script23:40
dstufftor that23:41
dstufftall you need to do is use the things installed in the venv's bin dir23:41
dstufftwhich activating just messes with $PATH23:42
*** pcrews has joined #openstack-infra23:42
dtroyerthe downside is that creates an even larger departure from our testing to the packaged environment23:43
ianwi guess i mean do things like init scripts respect that23:43
dstufftlooks like that's basically what a software collection is lol23:43
dtroyercan we do a hybrid and install pip 1.4 and setuptools to a non-system location and still use it to install packages as usual?23:43
dstufftwith --prefix or --root or whatever the option is23:44
clarkbianw: devstack shouldn't care about that23:44
dstufftdtroyer: FWIW the typical answer to packaged environments in python world is "let the packagers figure it out", not the nicest answer but dealing with system level deps is painful and alot of the distros historically haven't been willing to make using pip less painful23:45
*** jrex_laptop has joined #openstack-infra23:46
ianwclarkb: but does it?  a separate virtualenv certainly seems right, but i just wonder how many  issues there will be around things that don't respect the virtualenv, or need help figuring it out23:47
dtroyerdstufft: right…and when they diverge it makes our life even better.  whether or not it is the default, rhel letting anything install over packaged files out of the box is broken.  I'm getting tired of working around that brokenness.  and that's hard to admit because I've been a RH homie since the red disk beta23:47
clarkbianw: devstack is for development, it doesn't care about init scripts for the services it is installing from source and so on23:48
clarkband as long as everything is started through devstack and it has activated the virtualenv all children should pick up that environment too23:50
dstufftdtroyer: yea23:50
dstufftdtroyer: I'm going to try and get fedora-python to adopt a solution like Debian does23:51
dstufftI'm not particularly hopeful23:51
dstufftBecuase arguing on distutils-sig isn't enough for me! I need new and exciting places to argue23:51
clarkbdstufft: does rhel do the same thing with gems?23:52
dtroyerIIRC they've been over that more than once, I think it has become a sore point for them23:52
clarkbor is gems sufficiently different that it is a non issue for taht community23:52
dtroyerfwiw, perl has the distinction between system libs and local libs baked in.  it seemed to work well enough23:53
dstufftclarkb: unsure about for ``gem``, for bundler the gems don't get installed into a sys.path like location, they get installed outside of that and then bundler adds them to the sys.path23:53
dstufftdtroyer: that's essentially what the debian patch does23:53
ianwclarkb: "everything started via devstack it works" <- i think that's a departure from status quo.  maybe an acceptable one, however23:53
clarkbianw: that is how the gate works23:54
*** bnemec has quit IRC23:54
clarkbthen in theory the rhel packagers make native packages for you when each release comes out23:54
clarkbif you are doing continuous deployment type things then the disk image builder do your own distro type thing is probably prudent23:55
*** bnemec has joined #openstack-infra23:55
dstufftyou can make init scripts start things inside of a venv pretty easily if you're doing the CD thing and want to use pip or a virtualenv23:55
*** morganfainberg has joined #openstack-infra23:59

Generated by 2.14.0 by Marius Gedminas - find it at!