Saturday, 2013-09-28

*** tjones has joined #openstack-infra00:03
*** hemna_ is now known as hemnafk00:11
*** rockyg has quit IRC00:11
*** rockyg has joined #openstack-infra00:14
*** melwitt has quit IRC00:22
*** dkliban has quit IRC00:25
*** blamar has quit IRC00:28
*** julim has joined #openstack-infra00:41
*** CaptTofu has joined #openstack-infra00:42
*** CaptTofu has quit IRC00:47
*** reed has quit IRC00:47
*** tjones has quit IRC00:54
*** CaptTofu has joined #openstack-infra00:56
*** dims has quit IRC01:08
*** nati_ueno has quit IRC01:10
*** rcleere has joined #openstack-infra01:12
*** rcleere has quit IRC01:15
*** senk has joined #openstack-infra01:16
*** ericw has quit IRC01:22
*** ericw has joined #openstack-infra01:22
jog0wow with tempest being so flaky its amazing we can merge anything at all01:22
*** dims has joined #openstack-infra01:23
clarkbjog0: the gif of the toddler trying to get the block on the bucket and eventually giving up and taking the cover off comes to mind for some reason01:36
clarkbAlex_Gaynor: mordred I have realized that you guys twitter a lot01:36
mordredclarkb: we do01:36
clarkbalso, its like IRC where people don't need to subscribe to the channel to hear what you have to say. Seems dangerous :)01:37
*** ericw has quit IRC01:39
clarkbpleia2: you too01:41
clarkbjeblair: fungi: we need a support group or something01:41
pleia2clarkb: yeah, twitter is multicast irc01:43
lifelessunlike regular irc01:43
pleia2and I use bitlbee to read twitter feed and often tweet, so it's in irssi too ;)01:43
clarkbIRC has the benefit of allowing ~500 characters >_>01:45
lifelessand hyperlinks01:45
lifelessmordred: btw
mordredlifeless: bonghits01:48
clarkbis someone in the market for a dsl modem?01:48
*** senk has quit IRC01:48
lifelessmordred: I know right01:48
lifelessclarkb: I upgraded to vDSL recently01:48
lifelessclarkb: 10Mbps up 50Mbps down01:49
lifelessclarkb: but the ISP shipped a technicolor modem01:49
lifelessclarkb: Technicolor, known for dsl modems? ... not really.01:49
*** davidhadas_ has quit IRC01:49
clarkb is what I have. seems to mostly work01:50
*** davidhadas_ has joined #openstack-infra01:50
* mordred has fios01:50
lifelessthe technicolor stuff is shite01:50
clarkbmordred: I bet my vdsl has better bandwidth than your fios :)01:51
lifelessmordred: have you seen """  File "/opt/stack/venvs/python-glanceclient/local/lib/python2.7/site-packages/pip/backwardcompat/", line 90, in fwrite01:51
clarkbI get about 70/25mbps out of it01:51
lifeless    f.write(s)01:51
lifelessValueError: I/O operation on closed file01:51
lifelessfrom pip01:51
lifelessclarkb: I'm not talking speed, rather quality.01:51
lifelessclarkb: not linux, has to reboot to disable dhcp on an interface01:51
mordredlifeless: yes. but I do not remember what causes it01:51
lifelesshaphazard UI01:51
mordreddstufft: do you remember ^^ ?01:51
mordreddstufft: I remember we both looked at that liek a month ago or so...01:52
lifelesshmm 0.2% packet loss out of the freecloud dc01:52
lifelessthe first modem they gave me ran for 24 hours then started rebooting in a loop01:53
dstufftmordred: I don't remember01:55
dstufftclarkb: mordred I also have fios, I get like 60/25 and if I felt like paying like $70 more a month I could get 150/3501:56
dstufftor something like that01:56
dstufftI think I can get up to 300 :V01:56
*** schwicht has quit IRC01:57
clarkbyeah I had fios in portlandish01:57
clarkbit was good too. I have also had comcast which was not fun01:57
dstufftwe used to have comcast01:58
dstufftit was terribad01:58
pleia2I had fios in PA, but back to comcast (but it's ok, I have a backup connection too)01:58
clarkbmy favorite thing was that they knew when I needed it and would always have an outage. Once I did a 5am remote course thing and of course outage01:58
clarkbnot in any way prepared for debugging connectivity issues at 5am01:59
clarkbpleia2: your condo is multihomed?01:59
*** lifeless_ has joined #openstack-infra02:00
*** lifeless has quit IRC02:00
clarkbI think pleia2 wins this competition02:00
pleia2clarkb: yeah, we have p2p microwave that goes to the building02:00
pleia2100M for the whole building, so not spectacular but it's not bad02:01
pleia2especially good during the day when no one else is home :)02:01
*** lifeless has joined #openstack-infra02:06
*** lifeless_ has quit IRC02:07
clarkbdstufft: haven't gotten twine into place yet, but should soon. very happy twine is a thing02:07
dstufftclarkb: :D02:08
dstufftclarkb: lemme know how it goes02:08
dstufftI plan to make a new upload API for pypi evnetually02:08
dstufftand will update twine for it too02:08
clarkbthe change to use it conflicts with another change we want to get in, so fungi and I need to sort out how we want to get those in02:08
dstufftit conflicts?02:08
clarkbdstufft: git conflicts02:09
clarkbbut nevermind fungi's change merged so I get to rebase \o/02:09
clarkbwhich is good because now I know which direction we want to order them02:09
*** notmyname_ has joined #openstack-infra02:15
*** cyeoh_ has joined #openstack-infra02:16
*** notmyname has quit IRC02:17
*** cyeoh has quit IRC02:17
openstackgerritClark Boylan proposed a change to openstack-infra/config: Use twine to upload python package to pypi.
*** HenryG has quit IRC02:17
*** notmyname_ is now known as notmyname02:17
clarkbdstufft: thanks02:18
dstufftclarkb: np!02:19
*** tizzo has joined #openstack-infra02:21
clarkbjust found the thing on raw_input in and people wonder why pbr exists :)02:26
justinabrahmsDo you guys use a fork of gerrit?02:31
justinabrahmsor just configure it differently?02:31
mordredclarkb: :)02:31
mordreddstufft: which is the pip that no-one should ever run earlier than due to security?02:31
mordredjustinabrahms: both02:32
clarkbjustinabrahms: it is a slight fork unfortunately. But we are very close to being off of the fork02:32
mordredjustinabrahms: we're working on re-aligning with upstream though02:32
mordredwhat clarkb said02:32
justinabrahmsWhat's the main reason for the fork?02:32
clarkbjustinabrahms: the only major things in the fork that you would probably notice are the work in progress feature and a second default dashboard02:32
dstufftmordred: anything older than 1.3 oesn't verify SSL02:32
dstufftmordred: anything older than 1.4 doesn't give you the option to not download shit from random hosts02:33
clarkbjustinabrahms: the main reason is that gerrit didn't want our work in progress feature. But now gerrit has plugins that can do that for us and _david_ is writing one so win02:33
dstufftmordred: anything older than 1.5 (not relesased) doesn't not download random shit by defailt02:33
dstufftmordred: so take your pick :)02:33
justinabrahmsI'm going to be setting up our gerrit instance with chef (currently have a mostly vanilla setup at this point) and looking to mimic some of the things you guys have, just trying to get a sense of what is different. :)02:33
clarkbjustinabrahms: upstream adopted our default css stuff so look and feel should be similar from upstream02:34
justinabrahmsmostly trying to ape the: CI must approve, push result to github   aspects.02:34
clarkbjustinabrahms: we also add at least one extra review category (approval) but that is configuration not code02:34
mordredjustinabrahms: fwiw (not trying to get into a chef v. puppet debate or anything- they're both fine) but we do have a completely puppet repo with all of our puppet02:35
clarkbjustinabrahms: we use three review categories. verified which is where CI systems vote and has ranges -2 to +2, code review which is where humans can vote, we allow anyone to +/-1 and core reviewers to +/-2. And lastly approved which is +1 only02:35
justinabrahmsmordred: understood. If I didn't already have a chef instance, I'd just use what you have.02:35
mordredas well as docs - so please feel free to steal/crib/borrow whatever is useful!02:35
mordred in case you haven't found it02:35
mordredwe've also got a bunch of management scripts to do things like manage gerrit projects via yaml files - and of course there is zuul :)02:36
morganfainbergmordred, i so want to crib / use more of it! but not yet :(02:36
clarkbtypically workflow is as soon as you submit code Jenkins et al test it and vote +/-1 based on the results. Humans then review it and when you have two +2 votes from humans reviewers can hit the +1 approved vote which puts things in the gate queue02:36
mordredhappy to have you as part of the gerrit family!02:36
dstufftfwiw python-dev is discussing PEP453 (pep bootstrap) a few folks have expressed concern over the 2.7 backport, so if you feel strongly about it one way or the other you might want to weigh in02:36
clarkbthe gate queue tests that change as it would merge if it passes tests and if tests pass you get merged02:36
dstufftpip bootstrap*02:36
clarkboh yes manage_projects02:37
morganfainbergmordred, any idea when/if we'll be upgrading to a later version of gerrit?  i want some of the OS niceityies (trivial rebase, WIP) but i made the mistake of going with a later version early on02:37
morganfainbergand i'm concerned about trying to downgrade.02:37
mordredmorganfainberg: soonish. we're actively working on it02:37
clarkbmordred: and make manage_projects better02:38
morganfainbergmordred, awesome.  that will make me quite happy.02:38
mordredmorganfainberg: between _david_'s new plugin and zaro's patches I think we'l l get somewhere soon02:38
morganfainberg(and likely my boss too)02:38
mordredclarkb: neat!02:38
justinabrahmsclarkb: is it the case that one of those +2s has to be from jenkins? Or can 2 core humans +2 and get it into zuul?02:38
clarkbmorganfainberg: _david_ (works on gerrit and libreoffice iirc) has a WIP gerrit plugin in the works and zaro has a gerrit patch to extend the gerrit ACL system to make it useable for us02:38
morganfainbergand when i have time… new zuul for internal work..but thats currently a beast.02:38
justinabrahmsalso, if zuul fails, does it reopen the review?02:38
morganfainberg(for me)02:38
morganfainbergclarkb, awesome. i'll go look for it02:39
mordredjustinabrahms: jenkins only votes in the verified column02:39
morganfainbergthe ACL patch that is02:39
clarkbjustinabrahms: jenkins must vote +2 in the verified column so you need one +2 in verified and two +2s in code review (though only one +2 in code review is enforced)02:39
mordredwhat clarkb said02:39
clarkbjustinabrahms: also if you just want to fiddle with gerrit sans the super fancy CI stuff you can push to review-dev.openstack.org02:39
mordredjustinabrahms: and the code reviews don't close until jenkins/zuul close them - so, yes, if the test fail, a negative verified vote is posted02:39
justinabrahmsgot it.02:40
clarkbalso you want to enable ssh so that git-review will work (it doesn't support pushing via https yet)02:41
justinabrahmsTo you guys support a "To Be Reviewed" status?02:42
justinabrahmsie: I need to fix this prod bug right now, please submit a changeset for later review?02:42
justinabrahms(Google had that, but wasn't sure if you guys did)02:42
clarkbmorganfainberg: I think usptream new gerrit can do trivial rebase or cherry picking02:42
morganfainbergclarkb, i am fair certain cherry pick yes, trvial rebase detection wasn't there.02:43
morganfainbergor well.. trvial rebase issues02:43
morganfainbergi might be one rev behind though at this point.02:43
clarkbmorganfainberg: oh trivial rebase detection. that is just a hook iirc you should be able to do it with your gerrit02:43
morganfainbergso, might have progressed02:43
clarkbjustinabrahms: is that something that gets merged now and reviewed later? (trying to understand the workflow)02:43
morganfainbergclarkb, yeah, i've loathed to try and patch gerrit though (I'm not a java dev and each time i try i get … lets just say bad results).  I figured i'd wait for OS to catch up to where I am and move over and then stay in lock-step02:44
justinabrahmsclarkb: correct.02:44
justinabrahmsit was for things like: "I need to fix this at 3am. We can review it in the morning."02:44
mordredjustinabrahms: we do not support that concept here02:44
mordredI have no idea if gerrit itself can do it02:45
justinabrahmsI think that's fine. If it were massively important, presumably you could circumvent the review state all together and manually go over changes.02:45
morganfainbergjustinabrahms, I am unsure how well that would work.  you could get into a strange bind if things got based on that patch.. or piled on.02:45
mordredalthough we don't give those acls to any of our devs02:45
mordredbut yeah02:46
clarkbmordred: run it as a patchset created hook02:46
clarkbgah morganfainberg ^02:46
morganfainbergclarkb, heheeh took me a while to get the keystone people to type 4 characters :P02:46
clarkbjustinabrahms: also check out jeepyb, it is a collection of tools we built around gerrit. Gerrit Project Builder -> GPB -> jeepyb02:47
clarkbjustinabrahms: it isn't perfect though, it unfortunately encodes a bunch of what we do, but patches are welcome (and mordred has been trying to genericize it too)02:47
dstufftmordred: clarkb btw we found the problem that was causing thrift to timeout02:48
morganfainbergclarkb, wish i could be 100% dedicated to writing openstack code (and contribute to infra/qa more).  maybe when i have more staff here i can do that :)02:48
dstufftthe thrift install02:48
dstufftit'll take awhile for a deploy to fix it to be fully deployed though :[02:49
clarkbmorganfainberg: I wish I had more time to do the random housekeeping nice to have stuff02:50
justinabrahmsYea,  I think it makes less sense for infrastructural things like openstack, but might make more sense for an application.02:50
justinabrahmsLooking over this stuff, there are a few things I think I'll need. generate project on disk from chef-like location, shuffle commits to github (I think jeepyb has this?), notify jenkins that a thing needs to be tested, wire jenkins to let gerrit know how to deal with things.. that may be it?02:52
clarkbshuffling commits to github is done with gerrit's replication feature (now in a plugin), jeepyb creates projects in gerrit based on config and an optional location. wiring jenkins is good and can be done with the gerrit jenkins plugin or zuul. And git-review. I really think general opinion of gerrit would be better if git review were used more :)02:54
justinabrahms(as background, this will be for a closed-source project that will be used by ~6 engineers)02:54
lifeless clarkb mordred02:55
lifelesslooks suspiciously like it, in that it's coming out of ''02:55
justinabrahmsI think I might punt on a lot of the ACL stuff for now (at least until we migrate our open source tools over)02:55
clarkbjustinabrahms: I think it is common to go about it in steps. Don't need everything all at once for it to be useful02:56
clarkblifeless: yes looks like the thing. The last stacktrace especially (I didn't look closely at the others)02:57
clarkbI really need to make time to go give the infra overview talk in portland02:57
clarkbjustinabrahms: eventually you end up with
*** markmcclain1 has quit IRC02:59
*** basha has joined #openstack-infra02:59
mordreddstufft: any chance you know when pip 1.4.2 is coming out?03:00
dstufftmordred: no set date, whenever one of us proposes it with no arguments from the rest I gues03:01
mordreddstufft: I think someone from tripleo is desperately waiting for it03:01
mordredbecause of the fix to the on-disk permissions on install03:01
mordredand also there's a patch that landed 8 days ago that might make lifeless happy03:01
*** basha has quit IRC03:02
dstufftmordred: is it in the 1.4.2 milestone?03:02
mordreddstufft: one is this:
mordredthat's the lifeless one03:02
mordreddstufft: he's hitting that bug right now03:02
mordredthe other was merged a WHILE ago03:03
mordredthis one:
dstufftmordred: the one you linked looks like it's only on the develop branch so it'd need backported but that's not a big deal03:04
mordredis the one that will make tripleo guys happy03:04
mordredwhich seems to be in 1.4.203:04
dstufftmordred: can you send me an email ( with any tickets you want in 1.4.2 and i'll see about getting them backported to 1.4.x if they aren't and propose making a release03:05
dstufftdone or not03:05
dstufftjust so I don't forget^03:05
mordreddstufft: yah. thanks!03:05
lifelessdstufft: mordred: it *might* be different03:05
lifelessbut it's super repeatable03:05
dstufftlifeless: test it with develop and see if it happens?03:06
lifelessdstufft: it's deep in dib03:06
lifelessdstufft: on a remote datacentre machine with packet loss03:06
lifelessdstufft: not an idea develop environment03:06
mordredlifeless: sudo pip install -e git:// ?? :)03:10
*** basha has joined #openstack-infra03:11
clarkbmordred: what heppens when pip breaks itsefl? pipception?03:11
*** basha has quit IRC03:11
dstufftclarkb: I've done that03:12
dstufftif it's not in a venv it's terrible03:13
dstufftthere's a reason why a goal of mine is to make pip bundle or not depend on setuptool03:13
mordredclarkb: what happens is you learn a lot more than you knew before about installing and uninstalling python packages03:14
*** mrodden1 has joined #openstack-infra03:14
lifelessI added debug output to source-repositories.03:14
lifelessah no, it did not fix it :)03:14
*** jerryz has quit IRC03:14
lifelessnow why, with the pypi mirror element in place03:15
lifelessis Downloading/unpacking jsonschema>=0.7,<3 (from warlock>=1.0.1,<2->-r requirements.txt (line 7))03:15
*** mestery_ has joined #openstack-infra03:15
lifeless  Error <urlopen error [Errno 101] Network is unreachable> while getting (from
dstuffthow old is your pip03:16
* mordred hangs head03:16
*** mrodden has quit IRC03:16
*** mestery has quit IRC03:16
lifelessdstufft: I'm expecting file:// here.03:16
mordredwe still don't have https on our mirror03:16
lifelessdstufft: there is a local mirror.03:16
dstufftoh dur03:16
lifelessdstufft: so tis is astonishing me.03:16
dstuffti didn't read the whole thing03:16
mordredlifeless: how do you have the local mirror configured?03:16
dstufftI just say http://pypi03:16
dstufftand was like wat03:16
dstuffttyping is hard03:17
dstufftgood thing I don't type for a living03:17
mordredlifeless: can you point me at source code?03:17
lifelessmordred: sure, tripleo-image-elements/elements/pypi03:18
lifelessahhhh found it03:18
lifelessPIP_INDEX_URL= <- naughty script03:18
mordredooh. that is a naughty script03:19
lifelessI missed that in elements/openstack-clients/install.d/51-openstack-clients03:19
lifelessmordred: but this should hint to you how shoddy the networking I'm getting is03:19
lifelessmordred: was failing consistently...03:19
mordredlifeless: that's impressive03:21
mordredwe should CDN pypi.openstack.org03:22
clarkbdstufft: the mirror was born prior to https so we figured there wasn't much reason for https. We should probably rethink that03:22
dstufftclarkb: +103:23
dstufftin pip 1.5 it'll start printing messages03:23
dstuffttelling you it's insecure03:23
*** CaptTofu has quit IRC03:24
clarkbit will help fix several other issues too03:24
clarkbwithin our CI systems03:24
lifelessother fun stuff today03:24
lifelesswas finding out that you can't change dhcp allocation pools in neutron03:25
lifelessit's not 'in the spec'03:25
uvirtbotLaunchpad bug 1111572 in neutron "quantum subnet-update can't update allocation-pool " [High,Invalid]03:25
clarkblifeless: can you delete create?03:27
clarkbthe other ip address management tool I have used in the past required that you delete create, was annoying03:27
clarkbbut worked03:27
mordredclarkb: I think we don't want https until we figure out a way to get lifeless rsync mirroring03:29
mordredclarkb: because otherwise we'll break his squid03:29
clarkbmordred: we can do both03:30
mordredI'm just saying, we should do both or neither03:30
mordrednot just one03:30
lifelessclarkb: not when it's in use03:30
*** ericw has joined #openstack-infra03:34
*** rcleere has joined #openstack-infra03:34
*** basha has joined #openstack-infra03:43
*** pabelanger has quit IRC03:44
*** yonglihe_ has joined #openstack-infra03:45
*** yongli has quit IRC03:45
*** tian has quit IRC03:45
*** tian has joined #openstack-infra03:45
*** ryanpetrello has joined #openstack-infra03:48
*** pabelanger has joined #openstack-infra03:49
mordredok. jenkins and jenkins01 restarted03:59
*** ryanpetrello has quit IRC04:10
*** tizzo has quit IRC04:15
*** ryanpetrello has joined #openstack-infra04:16
jog0wow I go away for a few hours and gate queue goes down by 204:17
clarkbprogress >_>04:17
jog0clarkb: this is what it reminds me of
*** nati_ueno has joined #openstack-infra04:22
*** rcleere has quit IRC04:23
*** ericw has quit IRC04:30
*** sarob has joined #openstack-infra04:49
*** Ryan_Lane has quit IRC05:00
*** HenryG has joined #openstack-infra05:09
*** cody-somerville has quit IRC05:13
*** AlexF has joined #openstack-infra05:15
*** SlickNik_ has joined #openstack-infra05:20
*** Ryan_Lane has joined #openstack-infra05:31
*** sarob has quit IRC05:34
*** sarob has joined #openstack-infra05:34
*** Ryan_Lane has quit IRC05:39
*** sarob has quit IRC05:39
*** SlickNik_ has quit IRC05:42
*** AlexF has quit IRC05:44
*** tjones has joined #openstack-infra05:45
*** ryanpetrello has quit IRC05:45
*** tjones has quit IRC05:49
*** HenryG has quit IRC06:02
*** Ryan_Lane has joined #openstack-infra06:07
*** Ryan_Lane has quit IRC06:11
*** SergeyLukjanov has joined #openstack-infra06:16
*** Ryan_Lane has joined #openstack-infra06:37
*** Ryan_Lane has quit IRC06:41
*** fbo is now known as fbo_away06:45
*** sarob has joined #openstack-infra06:45
*** sarob has quit IRC06:50
*** dmakogon_ has joined #openstack-infra06:54
*** Ryan_Lane has joined #openstack-infra07:07
*** SpamapS has quit IRC07:08
*** Ryan_Lane has quit IRC07:11
mordredjog0: woot!07:24
*** fbo_away is now known as fbo07:30
*** SergeyLukjanov has quit IRC07:31
*** alexpilotti has joined #openstack-infra07:34
*** Ryan_Lane has joined #openstack-infra07:38
*** nati_ueno has quit IRC07:38
*** Ryan_Lane has quit IRC07:43
*** nati_ueno has joined #openstack-infra07:51
*** alexpilotti has quit IRC07:52
*** Ryan_Lane has joined #openstack-infra08:09
*** dguitarbite has joined #openstack-infra08:11
*** Ryan_Lane has quit IRC08:13
*** nati_ueno has quit IRC08:18
*** Ryan_Lane has joined #openstack-infra08:39
*** Ryan_Lane has quit IRC08:43
*** Ryan_Lane has joined #openstack-infra09:10
*** Ryan_Lane has quit IRC09:14
openstackgerritDirk Mueller proposed a change to openstack/requirements: Revert "Describe SQLAlchemy constraint more clearly."
*** tizzo has joined #openstack-infra09:20
*** tizzo has quit IRC09:25
*** AlexF has joined #openstack-infra09:27
*** tvb has joined #openstack-infra09:32
*** tizzo has joined #openstack-infra09:33
*** Ryan_Lane has joined #openstack-infra09:40
*** Ryan_Lane has quit IRC09:44
*** AlexF has quit IRC09:45
*** dguitarbite has quit IRC09:58
*** boris-42 has joined #openstack-infra09:59
*** SlickNik_ has joined #openstack-infra10:01
*** dmakogon_ has quit IRC10:03
*** SlickNik_ has quit IRC10:05
*** Ryan_Lane has joined #openstack-infra10:11
*** Ryan_Lane has quit IRC10:15
*** jtzl has quit IRC10:17
*** jtzl has joined #openstack-infra10:17
*** SpamapS has joined #openstack-infra10:25
*** mkirk_ has quit IRC10:39
*** mkirk has joined #openstack-infra10:40
*** Ryan_Lane has joined #openstack-infra10:41
openstackgerritDarragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use yaml local tags to support including files
*** tvb has quit IRC10:48
*** Ryan_Lane has quit IRC10:50
*** tvb has joined #openstack-infra10:56
openstackgerritSean Dague proposed a change to openstack-infra/config: start voting on neutron-pg job
openstackgerritSean Dague proposed a change to openstack-infra/config: add neutron-pg testing to neutronclient
openstackgerritSean Dague proposed a change to openstack-infra/config: double jobs on neutron to prevent races from landing
*** CaptTofu has joined #openstack-infra11:06
*** tizzo has quit IRC11:06
*** Ryan_Lane has joined #openstack-infra11:16
*** Ryan_Lane has quit IRC11:20
*** michchap has quit IRC11:40
*** michchap_ has joined #openstack-infra11:41
*** michchap has joined #openstack-infra11:41
openstackgerritSean Dague proposed a change to openstack/requirements: add an explanation to the sqla requirement
openstackgerritDarragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Provide default ConfigParser object
openstackgerritDarragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use yaml local tags to support including files
*** michchap_ has quit IRC11:45
*** Ryan_Lane has joined #openstack-infra11:47
dimsso, jgriffith has a review that bumps up num_iscsi_retries to try to get to root cause of - BUT the problem is check-tempest-devstack-vm-neutron does not run the test we need tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern - Any ideas on how to reproduce the problem? Please leave hints on the review11:49
uvirtbotLaunchpad bug 1226337 in tempest "tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern flake failure" [High,Triaged]11:49
*** Ryan_Lane has quit IRC11:51
*** CaptTofu has quit IRC11:51
sdaguedims: the fails are triggered on the non neutron jobs11:52
*** jhesketh has quit IRC11:55
dimssdague, thx! i see it now11:58
*** schwicht has joined #openstack-infra12:03
*** CaptTofu has joined #openstack-infra12:12
*** Ryan_Lane has joined #openstack-infra12:17
*** Ryan_Lane has quit IRC12:21
guohliuhello, has that pypersing or cliff issue solved on stable/grizzly?12:41
*** CaptTofu has quit IRC12:42
*** Ryan_Lane has joined #openstack-infra12:48
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: Revert "Enable q-vpn service"
*** Ryan_Lane has quit IRC12:52
sdagueguohliu: I don't believe so12:52
sdaguefungi, jeblair, mordred, clarkb: if any of you guys end up on today - might get neutron back under control12:53
fungisdague: thanks for the heads up. i'll try to have a look after breakfast12:54
*** tvb|afk has joined #openstack-infra12:57
sdaguefungi: cool, thanks12:59
*** tvb has quit IRC12:59
sdagueok, time to get ready for some bike riding and good tea shop, I may check back in late in the day13:04
fungihave fun13:07
guohliusdague: ok, thx13:10
guohliusdague: and we really can't bypass this issue by update that quantumclient requirments temporarily?13:12
*** Ryan_Lane has joined #openstack-infra13:18
sorenfungi: fwiw:
sorenfungi: That'll let us all enable csrf protection without breaking jenkins-job-builder.13:20
*** Ryan_Lane has quit IRC13:23
*** SergeyLukjanov has joined #openstack-infra13:32
*** michchap has quit IRC13:45
*** michchap has joined #openstack-infra13:45
*** Protux has quit IRC13:49
*** Ryan_Lane has joined #openstack-infra13:49
*** adalbas has quit IRC13:49
*** michchap_ has joined #openstack-infra13:50
*** Protux has joined #openstack-infra13:51
*** mestery has joined #openstack-infra13:53
*** Ryan_Lane has quit IRC13:53
*** michchap has quit IRC13:54
*** mestery_ has quit IRC13:54
*** dmakogon_ has joined #openstack-infra13:58
*** adalbas has joined #openstack-infra14:01
*** dmakogon_ has quit IRC14:02
*** dguitarbite has joined #openstack-infra14:04
*** adalbas has quit IRC14:13
*** adalbas has joined #openstack-infra14:13
jeblairsdague: fyi
sdaguejeblair: ah, oops. Guess that what happens when it doesn't like to launchpad14:17
sdaguejeblair: actually, any idea why Related-Bug didn't link that one?14:18
*** Ryan_Lane has joined #openstack-infra14:19
jeblairsdague: maybe it's still looking at the project->lp mapping and refusing to link to something other than openstack-ci14:19
jeblairsdague: what do you think of the results from salvatore's change?14:21
jeblairi'm confused by the disagreement between jog0's bot and salvatore14:22
*** Ryan_Lane has quit IRC14:23
jeblairclarkb, jog0: i registered the 'openstackrecheck' nick and added the password to hiera14:46
jeblairclarkb: (email address info, etc, in the usual place)14:47
jeblairfungi, mordred: fyi ^14:47
jeblairfungi, mordred, clarkb: i also restructured our automated system related addresses to all be an alias for one account (info is in the usual place).14:48
jeblairsoren: do you need a crumb for every request?  or can you get a crumb and reuse it for several requests?14:49
jeblairsoren: (i'm wondering about doubling the number of http requests overall)14:49
*** Ryan_Lane has joined #openstack-infra14:49
sdaguejeblair: so I think the issue is there are a few different neutron races going on14:51
sdagueand as far as I can tell they are all basically related to bad db management. I actually think they are doing what nova used to do which is only send ids on the wire then have the otherside go look things up in the db.14:52
*** dguitarbite has quit IRC14:53
*** Ryan_Lane has quit IRC14:54
sdagueyeh, honestly, there are a few different things happening, and I think that the elastic-recheck checks tend to be broader than they really should be14:54
jeblairok, so the distable qvpn route still seems promising?14:54
jeblair(btw, that explanation does make sense of the logstash observation it happened a few times in the past and then greatly increased in frequency several days ago)14:55
sdagueso this is the deadlock history -
*** dizquierdo has joined #openstack-infra14:59
sdagueman, you know what I really want... corrolating the specific fails we get in the gate against all the fails we get in the gate, to know if we've got classifications for all of them14:59
sdaguedammit, I need more hours in the day to write code :)14:59
sdaguejeblair: also, when you get a chance - is a really important narrowing of that fail15:00
sdagueand points us back to a keystoneclient issue15:00
sdagueso it would be good to +2 it into elastic-recheck15:01
*** AlexF has joined #openstack-infra15:02
sdagueok... biking... later all15:04
jeblairwe need to make that a yaml file so we can end the quoting madness15:04
*** tvb|afk has quit IRC15:05
*** HenryG has joined #openstack-infra15:05
*** alexpilotti has joined #openstack-infra15:09
jeblairrestarting zuul15:09
*** fbo is now known as fbo_away15:10
*** Ryan_Lane has joined #openstack-infra15:20
*** Ryan_Lane has quit IRC15:24
Alex_Gaynorjeblair, mordred, clarkb: FYI, fastly found the issue that was causing some timeouts on installing thrift,  they're in the processing of rolling out a fix15:33
*** SergeyLukjanov has quit IRC15:36
*** Ryan_Lane has joined #openstack-infra15:50
*** Ryan_Lane has quit IRC15:55
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: doc: fix up strict errors
*** thomasm has joined #openstack-infra16:15
*** tvb has joined #openstack-infra16:16
*** tvb has joined #openstack-infra16:16
*** changbl has quit IRC16:16
*** tvb has quit IRC16:20
*** Ryan_Lane has joined #openstack-infra16:21
*** Ryan_Lane has joined #openstack-infra16:21
*** AlexF has quit IRC16:24
*** Ryan_Lane has quit IRC16:25
*** dizquierdo has left #openstack-infra16:27
openstackgerritA change was merged to openstack-infra/gerritlib: Add method to list Gerrit Groups.
*** SnowDust has joined #openstack-infra16:38
*** ericw has joined #openstack-infra16:41
*** salv-orlando_ has joined #openstack-infra16:46
*** ericw has quit IRC16:46
*** ericw has joined #openstack-infra16:47
*** salv-orlando has quit IRC16:49
*** salv-orlando_ is now known as salv-orlando16:49
*** Ryan_Lane has joined #openstack-infra16:52
fungisoren: thanks for the link!16:53
*** Ryan_Lane has quit IRC16:56
fungijeblair: what was the upshot of the discussion with sdague on vs ? i had approved 48793 before noticing 48767 so went back and -2'd it, but i'll switch back to a +2 there if it's one we still want to go forward with16:56
*** ericw has quit IRC16:57
funginevermind, looks like you +2'd it after discussing with sdague rather than before. switching my vote back so it can hopefully merge17:11
* fungi disappears to run errands... bbl17:14
SnowDust anyone ?17:14
*** dguitarbite has joined #openstack-infra17:17
*** alexpilotti has quit IRC17:24
*** thomasm has quit IRC17:24
*** thomasm has joined #openstack-infra17:36
*** dguitarbite has quit IRC17:37
mordredsoren: neat re: crsf17:42
mordredSnowDust: we don't really do anything with redstack17:46
*** DennyZhang has joined #openstack-infra17:46
dstufftmordred: looks like you'll be getting a 1.4.2 in the near future17:50
mordreddstufft: woot!17:50
mordredI know some humans who will be happy17:51
dstufftthe pip dev channel is talking about it, we're gonna see about knocking out the issues17:51
dstufftthen see if the RM wants me to make a release or wants to (and is able) to do it himself again17:51
* mordred gets prepared to force zul to get 1.4.2 rammed in to the next ubuntu17:53
ekarlso1.4.2 of ?17:54
mordredekarlso: pip18:02
openstackgerritA change was merged to openstack-infra/pypi-mirror: Rename jeepyb to pypi-mirror
*** cody-somerville has joined #openstack-infra18:06
*** salv-orlando has quit IRC18:06
mordreddstufft: do I need to explicitly depend on wheel to get the pip wheel commands? or did you guys vendor it?18:09
dstufftit's not vendored18:09
openstackgerritA change was merged to openstack-infra/pypi-mirror: Start building wheels in the mirror
openstackgerritA change was merged to openstack-infra/pypi-mirror: Strip out the things we don't need from jeepyb
*** DennyZhang has quit IRC18:11
openstackgerritA change was merged to openstack-infra/pypi-mirror: Remove spurious uses of output return values
*** dims has quit IRC18:12
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Rework run_shell_command
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Do not pass unicode where byte strings are wanted
*** schwicht has quit IRC18:22
*** sarob has joined #openstack-infra18:28
*** basha has joined #openstack-infra18:28
*** adalbas has quit IRC18:29
justinabrahmsquestion on gerrit replication: how do you guys have github replication setup? I'm looking through the docs and I'm not seeing anything in the replication config that is per-project where you'd indicate the other repo.18:30
*** Ryan_Lane has joined #openstack-infra18:30
justinabrahmsDo you guys not use gerrit replication and instead use zuul?18:30
clarkbjustinabrahms: we use gerrit replication. we have a 1:1 mapping in gerrit project name to github project so openstack/nova in our gerrit maps to
justinabrahmsThat makes sense.18:31
clarkband the gerrit user can push to ssh://
justinabrahmsHow is that setup? Is it via something like github deploy keys?18:31
mordredwe have a service account at github with an ssh key18:32
mordred is the file - although probably completely looks nonsensicle there18:32
mordredjustinabrahms: here's what it looks like expanded:
justinabrahmsactually looks reasonable there.18:34
justinabrahmsI think chef has broken my brain.18:34
*** sarob has quit IRC18:35
mordredauthGroup means "replicate all repos that can be seen by this group" - mirror means "force push everything all the time"18:35
*** sarob has joined #openstack-infra18:35
justinabrahmsactually, looking at the docs, mirror also means to delete any refs which aren't present in the gerrit config18:35
clarkbyeah it is destructive if you use the remote end for anything else18:36
*** tvb has joined #openstack-infra18:38
*** tvb has quit IRC18:38
*** tvb has joined #openstack-infra18:38
*** sarob has quit IRC18:40
mordredwe don't use the remote end for anyting else - we really only touch github because if we don't mirror it there, someone else will18:41
mordred(true story, btw, that happened before we did)18:41
clarkbmordred: they still own the favicon account :)18:42
clarkbmordred: which is on my list to fix but wordpress ugh18:42
mordredthey do? wait - what?18:42
mordredwhat favicon account?18:42
clarkbmordred: for openstack/18:42
mordredwhat's a favicon account?18:42
clarkber not favicon18:42
clarkbwhatever the damn thing is that makes the pictuere show up and requires wordpress18:43
clarkbclearly I am not awesome enough to remember18:43
clarkbthat thing18:43
mordredoh. and yeah. gravatar18:43
mordredand wordpress18:43
*** basha has quit IRC18:45
mordredclarkb: ok. I'm taking care of that right now while I'm thinking about it18:46
clarkbmordred: the reason we want to change it (well one reason) is to get a higher res logo18:47
mordredclarkb: do you think we need to make a specific email alias for the gravatar accout? or use one of the current ones?18:47
clarkbmordred: so make sure you don't copy pasta the old one18:47
clarkbmordred: I think review.o.o or similar is fine18:47
clarkber review at o.o18:47
mordredclarkb: any idea where where's a good high res I should use?18:50
clarkbmordred: I think even the small png for the first logo is better res18:50
clarkbnot sure how gravatar will scale if you use the large png18:51
openstackgerritA change was merged to openstack/requirements: Revert "Describe SQLAlchemy constraint more clearly."
mordredclarkb: done18:54
clarkblooks much better18:55
clarkbhow horrible was the wordpress stuff?18:55
openstackgerritMonty Taylor proposed a change to openstack-dev/pbr: Use wheels for installation
mordredclarkb: not bad18:57
mordredclarkb: I added an account and put it in the file18:57
mordredclarkb: the above pbr change may be interesting :)18:58
clarkbare we building wheel mirrors yet?18:59
mordrednope. but the new pypi-mirror command is ready for it18:59
mordredthat change above should test that it will work18:59
mordredif it works, then I'll propose a change to config to use pypi-mirror instead of jeepyb18:59
mordredwhich will mean we'll start building them18:59
mordredthen we can roll out consumption of them18:59
mordred(the mirror building should continue to build old tarball mirrors too)19:00
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: log info on job deletion
*** cody-somerville has quit IRC19:02
mordredclarkb: the neat thing is, I believe that we will not have to make any changes to devstack to take advantage19:06
clarkbdoes pip expect wheels to live under those dirs on the mirror?19:06
openstackgerritA change was merged to openstack-dev/pbr: Update tox config to use latest upgrades
mordredclarkb: it does when we change the pip.conf to tell it to look there19:07
mordredAlex_Gaynor: uhm. around?19:08
Alex_Gaynormordred: yeah19:08
mordredI'm confuseled by that error19:08
Alex_Gaynormordred: waaa, never seen anything like this before; does it reproduce?19:09
mordredAlex_Gaynor: dunno. let me re-run19:09
Alex_Gaynormordred: I could theoretically imagine how if this patch changed when a stream was closed it could affect it?19:10
mordredok. the gate is still pretty flaky19:10
mordredAlex_Gaynor: it really shouldn't have19:11
Alex_Gaynorelastic recheck is pretty great19:13
openstackgerritMonty Taylor proposed a change to openstack-infra/devstack-gate: Add pypi-mirror to the list of pre-cloned repos
mordredAlex_Gaynor: yeah it is19:15
*** SnowDust has quit IRC19:15
mordredclarkb: ^^ the above is needed for further work/testing on wheel patches19:15
clarkbyes super flaky, one of the changes to help make it less flaky flaked out... I am trying to find a bug to reverify against19:15
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: KeepLongStdio argument for JUnit publisher
openstackgerritAntoine Musso proposed a change to openstack-infra/jenkins-job-builder: parameters self expansion
clarkband reverified, hopefully it merges this time19:23
justinabrahmswoo! first part of replication setup!
uvirtbotjustinabrahms: Error: "_^" is not a valid command.19:26
mordredjustinabrahms: woot!19:27
*** Ryan_Lane has quit IRC19:28
sorenjeblair: I don't know, to be honest. Let me try.19:45
*** sarob has joined #openstack-infra19:46
*** sarob has quit IRC19:51
sorenjeblair: That totally seems to work. I thought it was supposed to be a nonce, but I guess not.19:52
*** weshay has quit IRC19:58
justinabrahmshow do you guys manage the SSH key of your gerrit box? Is that managed via puppet? or do you just add a new one on each deploy, then upload it to github?20:00
justinabrahmssecond option seems more secure & more work (which I suppose is a truism of "more secure")20:00
sorenjeblair: Patch updated to only fetch crumb once. Neat. Thanks for the suggestion.20:01
pabelangerjustinabrahms, did the same thing you are, mine are stored in hiera (puppet)20:02
justinabrahmschef has encrypted cookbooks, iirc, which seem like they would be a reasonable fit here.20:02
*** salv-orlando has joined #openstack-infra20:03
mordredjustinabrahms: yeah. we put them in heira for puppet20:07
mordredwhich serves a similar role20:07
*** basha has joined #openstack-infra20:08
mordredso we have a thing that holds our passwords and keys, and then our puppet manifests reference those by name20:08
*** AlexF has joined #openstack-infra20:14
*** rickerc has quit IRC20:18
*** DennyZhang has joined #openstack-infra20:22
*** basha has quit IRC20:37
*** jhesketh_ has quit IRC20:39
*** rickerc has joined #openstack-infra20:40
clarkbmordred: you may find useful20:41
*** thomasm has quit IRC20:45
*** changbl has joined #openstack-infra20:50
mordredclarkb: fascinating20:51
*** tvb has quit IRC20:55
*** AlexF has quit IRC21:09
*** SergeyLukjanov has joined #openstack-infra21:12
*** davidhadas_ has quit IRC21:14
*** SergeyLukjanov has quit IRC21:25
*** tvb has joined #openstack-infra21:26
*** SergeyLukjanov has joined #openstack-infra21:26
jog0sdague: its hard to find an accurate query to use for elstic-recheck until we know the root cause. so the bugs being to high level makes sense.  Hopefully we will get better at this over time as the number of bugs drops and we reign in transient bugs more21:30
jog0sdague: just pulled down the change for so it will be running now.21:33
jog0also sdague to tell if there are unknown failures in the gate we dump out warnings to openstack-qa saying unknown bug21:34
*** tvb has quit IRC21:34
jog0jeblair: let me know when you want to start running the bot in infra under the new user21:34
jog0so I can kill my bot21:34
*** SergeyLukjanov has quit IRC21:44
openstackgerritA change was merged to openstack-infra/devstack-gate: Revert "Enable q-vpn service"
*** tjones has joined #openstack-infra22:00
*** tvb has joined #openstack-infra22:01
*** tjones has quit IRC22:04
*** DennyZhang has quit IRC22:31
*** alexpilotti has joined #openstack-infra22:51
sdaguejeblair / mordred - - SSL error on pypi... making requirements tests "fun"22:58
Alex_Gaynorsdague: this is a known bug in fastly (PyPi's CDN), they've lidentified it and are in the process of deploying a fix22:58
sdagueAlex_Gaynor: ok, any idea on ETA?22:59
sdagueit's bad enough to be fighting our own bugs, then to be taken down by external ones :)22:59
Alex_Gaynorsdague: They're in the process of landing and deploying it, it'll take some time to get it rolled out to their full fleet of PoPs, but a) I imagine US pops will come first so it'll be effectively resolved for us soon, b) it's intermitent so it's still possble to proceed, just really annoying :(, it's not totally impossible to install thrift :/23:00
sdagueAlex_Gaynor: well it bounced me 2 out of 3 times today on requirements, so it's pretty bad23:01
Alex_GaynorSorry, I realize it sucks :/23:01
Alex_Gaynorsdague: agreed :/23:01
sdagueyep, no worries, it's a weekend so it's not too bad. Just hope it's fixed by monday, otherwise we're going to have lots of challenges23:02
sdagueI've gotten used to the fact that openstack infrastructure is basically the best monitoring framework for the health of the internet. :) All these services that you think are always up, turn out to be down more than you realize23:02
notmynamesdague: is really a bot that just asks the -infra team, isn't it?23:04
sdaguethat my understanding. It just texts mordred23:04
sdagueAlex_Gaynor: well at least one of the fails was something other than thrift -
Alex_Gaynordstufft: FYI
uvirtbotLaunchpad bug 1232592 in openstack-ci "pypi CDN bugs cause SSL errors when trying to install packages" [Undecided,New]23:19
dstufftAlex_Gaynor: :[23:20
dstufftAlex_Gaynor: Not sure I have a better answer then what you have already said23:20
Alex_Gaynordstufft: sure23:20
dstufftFastly always does phased roll outs so they don't take down their entire network at once with a bad deploy23:21
dstufftsdague: Alex_Gaynor I can give you a temporary bypass the CDN if that'd help23:22
dstufftwell no I lie I can't23:22
dstufftI don't have DNS control23:22
sdaguedstufft: if you think it's solved within 24 hrs, it's fine to wait, we'll just recheck23:23
sdagueif you think it's going to bleed into the work week, we might want to figure out a bypass23:23
dstufftsdague:  a full rollout takes a week or two, Not sure at what rate their rollout actually occurs though (e.g. is 50% of the hosts got the fix within 24 hours? I have no idea)23:24
dstufftsdague: this is running PR tests?23:24
dstufftpull requests23:25
dstufftI don't know the gerrit terms23:25
dstufftdoes this operate in a trusted context23:25
sdagueit's testing proposed patches23:25
dstufftsdague: I ask because I do have an url that bypasses the CDN, but it also does not have TLS23:26
dstufftso you're open for MITM23:26
sdaguethese are devstack nodes, so they are single use23:26
dstufftif this is during a job that is testing random crap, that's not a big deal because you already have to deal with untrusted code in that context23:26
dstufftif this runs on a trusted node, that is a bad thing23:26
sdaguethe jobs that have bounced are all untrusted23:27
dstufftsdague: bypasses the CDN but there is zero SLA on that url so if we decide to move things around or something that url may stop working (but it probably won't in the next week or so)23:27
sdagueI'll let mordred or jeblair comment on whether an untrusted url for this would be a good solution come the work week23:27
dstufftit also bypasses the LB23:27
dstufftso I don't suggest using it unless you need to23:28
sdagueyeh, so I think mostly we only hit it on jobs which bypass our mirror23:28
dstufftsdague: I'm sorry we're affecting you though23:28
sdagueyeh, no worries, stuff happens23:28
dstufftVarnish isn't really designed to serve big files23:28
dstufftif you have jobs that are hitting pypi directly (besides the obvious mirror jobs) you might want to setup a global mirror23:30
sdagueso this only actually seems to be affecting us on our global requirements repo, that's small enough of a surface that it's easy enough to work around23:30
dstufftone that just tracks PyPI in entirity23:30
dstufftsdague: ok23:30
sdaguedstufft: we actually use the fact that we have a limitted mirror to prevent packages from sneaking in new dependencies23:31
sdagueit's a second line of defense23:31
dstufftsdague: ya23:31
sdaguethanks for the options though, if it becomes a bigger issue might want to sort out switching the mirror23:32
sdaguetime for me to pumkin for the night, later folks23:33
*** nati_ueno has joined #openstack-infra23:37
*** ericw has joined #openstack-infra23:39
*** reed has joined #openstack-infra23:40
*** nati_ueno has quit IRC23:41
fungiright, we figured hitting pypi to test changes to the set of packages from which we build our mirror was an infrequent enough job that having to *occasionally* recheck because of network screwiness was acceptable. though it wasn't until the thrift download problems that i realized just how often we do change the requirements list23:48
*** reed has quit IRC23:54
boris-42fungi hi23:54
boris-42fungi does gate work properly?)23:54
boris-42fungi I mean it's empty!=023:55
fungiboris-42: it works best when empty ;)23:55
boris-42fungi =))23:55
openstackgerritA change was merged to openstack/requirements: Removed a reference to quantum and removed a dead file
*** ianw has quit IRC23:58
*** ericw has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!