Tuesday, 2013-09-24

clarkbreed: you can fetch the git ref and look there for committer vs author00:00
reedyeah, that makes the whole shebang even more complicated00:00
* reed grows less and less happy as time passes00:00
jheskethreed: I think that's called getting older ;-)00:01
reedjhesketh, by this time I'll be a very grumpy old man by the age 4300:01
jheskethwhat are you working on?00:02
*** sarob has quit IRC00:02
reedjhesketh, https://bugs.launchpad.net/openstack-community/+bug/119564800:02
uvirtbotLaunchpad bug 1195648 in openstack-community "Report separately first activities" [Wishlist,Fix committed]00:03
reedActivity Board00:03
jheskethah cool00:06
openstackgerritA change was merged to openstack-infra/jeepyb: Revert "Remove tempest from list of projects that direct release."  https://review.openstack.org/4792100:08
openstackgerritJay Pipes proposed a change to openstack-infra/config: Enable gate on OpenStack Chef Repository  https://review.openstack.org/4788000:09
*** michchap has quit IRC00:10
*** michchap has joined #openstack-infra00:11
Alex_Gaynorclarkb: yes they pass (I never send patches like that without testing myself) I can leave a comment with shell output if it'd help00:12
*** michchap has joined #openstack-infra00:14
*** dcramer_ has joined #openstack-infra00:14
clarkbAlex_Gaynor: no, just confirmation from you that they work is good enough00:14
*** dkehn_away is now known as dkehn00:15
openstackgerritA change was merged to openstack-infra/config: Check and gate requirements against PyPy  https://review.openstack.org/4775600:17
clarkbpip folks https://review.openstack.org/#/c/45555/7 <0.8 picks up 0.8b2 for example is that because of the silliness in tox and old pip?00:17
clarkbdo we have a preferred way to deal with that?00:17
lifelessmordred: there is a little, from time to time. But there should not as a rule be any.00:17
clarkbmordred: ^ I think you would know00:18
*** sarob_ has quit IRC00:19
Alex_Gaynorclarkb: yes. That's because of old pip00:22
mordredclarkb: what?00:24
mordredclarkb: yes00:24
mordredclarkb: we need to update the tox.ini00:24
clarkbmordred: to specify the install command?00:24
mordredclarkb: the patch I posed to the mailing list this morning00:25
mordredclarkb: want me to make one real quick?00:25
*** mrodden has joined #openstack-infra00:25
*** mrodden1 has quit IRC00:26
clarkbmordred: no rush, I am just trying to flush my review queue and would like to do so properly (with an understanding of things)00:26
clarkbI left a +1 on that keystone change as <0.8 is correct but it does require new pip00:26
mordredclarkb: it's not the <0.8 that requires the new pip00:27
mordredclarkb: https://review.openstack.org/4794200:27
clarkbmordred: it is though anyone with old pip will get hit by this issue00:28
mordredclarkb: it's possible we should have opened a project-wide bug on this00:28
mordredyes. but it's not due to sqla<0.800:28
mordredit's due to sphinx00:28
mordredand they should upgrade to pip>=1.400:28
mordredor else evil00:28
clarkbmordred: sqla<0.8 ionstalls 0.8b200:28
clarkbso you need ti for sqla too00:28
clarkbyes everyone should upgrade pip00:28
mordredyes. you are corect00:29
mordredI'm 3 seconds from decreeing by fiat that openstack does not support pip < 1.4 in any way00:29
mordredI'm not sure I have that power00:29
mordredbut I think I might do it anyway00:29
clarkbjog0: mtreinish I intend on cleaning up the puppet for elastic-recheck running on logstash.o.o tomorrow00:33
clarkbjog0: mtreinish hopefully we can get that running rsn00:34
clarkband with that time to head home and find dinner00:34
jog0clarkb: sounds good, still a bit more to do to make it pip able00:35
jog0can we get this onto openstack-infra today?00:35
jog0so I can send out an email00:35
clarkbI dont thibk so I need to afk for a bit then code review00:37
clarkbtomorrow may be possible.00:38
jog0clarkb: no problem00:40
mordredclarkb: how's your review queue00:42
openstackgerritA change was merged to openstack-infra/config: Enable gate on OpenStack Chef Repository  https://review.openstack.org/4788000:45
*** pcm_ has quit IRC00:48
clarkbmordred ~2300:48
jog0clarkb: haha try 14100:48
mordredclarkb: is pypi-mirror in your review list?00:50
mordredfungi: ^^ how about you?00:50
jog0mrodden: who was user #100:51
mordredjog0: jeblair00:51
clarkbmordred it isnt00:51
mordredjust for the record - jenkins is #3 and Dolph is #4 :)00:51
jog0is anyone here good at writting wiki pages that are short : https://wiki.openstack.org/wiki/GerritJenkinsGit#Test_Failures00:52
clarkbno :)00:53
jog0not trying to blame actually wondering00:53
mordredwe tend to like complete thoughts around here00:54
*** anteaya has joined #openstack-infra00:54
jog0very complete00:54
*** amotoki has joined #openstack-infra00:55
jog0I like the link to jenkins which is now wrong00:55
*** UtahDave has quit IRC00:55
* anteaya having arrived home, and currently armed with a purring cat, serves notice to whatever virus has been dogging her movements00:55
mordredanteaya: kitteh!00:57
anteayaI'm happy now, so is kitteh00:58
anteayamordred, did pleia2 tell you we got plaques?01:02
anteayanow I need to get a mantle piece to put it on01:02
anteayaRyan_Lane got one too01:03
*** nosnos has joined #openstack-infra01:03
mordredanteaya: nice!01:04
*** dprince has joined #openstack-infra01:04
anteayaI was very surprised01:04
openstackgerritA change was merged to openstack-infra/config: Fix dependencies for activity-dev  https://review.openstack.org/4770801:04
*** jerryz has quit IRC01:08
*** jhesketh has quit IRC01:09
*** morganfainberg has quit IRC01:11
*** morganfainberg has joined #openstack-infra01:12
*** Ryan_Lane has quit IRC01:14
jeblairanteaya: better than plagues which is what i thought you said you got :)01:23
*** melwitt has quit IRC01:24
jeblairrestarting zuul to pick up this morning's bugfix01:25
*** yaguang has joined #openstack-infra01:25
jeblair(eta 28 mins)01:25
clarkbhmm too late to merge Alex_Gaynor's change I guess01:26
clarkbthats ok it can go in next round01:26
clarkbjeblair: https://review.openstack.org/#/c/46869/ is the change I am referring to01:27
clarkbjeblair: I didn't approve as lifeless had a -1 and I tend to defer zuul approvals to you01:27
lifelessjeblair: did you have any feedback on the baremetal test plan ?01:28
jeblairlifeless: i have not had a chance to read it.  i hope to before tomorrow's meeting.01:28
jeblairlifeless: thank you for sending it01:28
lifelessjeblair: ack, np.01:29
openstackgerritAlex Gaynor proposed a change to openstack-infra/config: Handle when `id` is null.  https://review.openstack.org/4795301:29
*** schwicht has quit IRC01:37
*** dcramer_ has quit IRC01:39
*** russellb has quit IRC01:43
*** russellb has joined #openstack-infra01:44
*** dprince has quit IRC01:53
*** rfolco has joined #openstack-infra01:57
*** jgriffith has joined #openstack-infra01:58
jgriffithclarkb: ping01:58
clarkbjgriffith: pong (I am cooking dinner so won't be super responsive)02:02
jgriffithclarkb: go cook :)02:02
jgriffithclarkb: I'll post to the ML02:02
jgriffithLooking for a copyright expert02:02
clarkboh that isn't me. mordred and jeblair ^02:02
jgriffithclarkb: ahh... thanks :)02:03
jgriffithenjoy your dinner02:03
jgriffithjeblair: mordred My question is WRT this: https://bugs.launchpad.net/keystone/+bug/121417602:03
uvirtbotLaunchpad bug 1214176 in swift "Fix copyright headers to be compliant with Foundation policies" [Low,In progress]02:03
jgriffithjeblair: mordred going backand modifying all of the copyright headers doesn't seem kosher to me02:04
*** morganfainberg has quit IRC02:08
*** morganfainberg has joined #openstack-infra02:08
clarkbjgriffith: oh so in this particular case, the openstack LLC gave the openstack foundation copyright so this particular change is kosher02:09
jgriffithclarkb: hmm... okie dokie02:09
jgriffithI stand corrected02:09
clarkbadding openstack foundation copyright to files without original LLC copyright or files that haven't been written by foudantion employees is wrong02:09
clarkbnow whether or not we need to change them wholesale in the fiels is questionable02:10
jgriffithpersonally my udnerstanding was that regardless of "giving" you don't modify the headers02:10
mordredwell, we've been instructed by legal people in this case that we shoudl change the headers from LLC to Foundation02:10
mordred_I_ think that's crazy02:10
mordredbecause the files were (c) LLC back when that copyright was correct02:10
jgriffithmordred: I agree, but I typically feel that way when lawyers give directions :)02:10
jgriffithmordred: exactly!!02:11
*** svarnau has quit IRC02:11
mordredand then, a copyright of (c) $year Foundation should be _added_02:11
mordredapparently that's not what happened or something02:11
mordredand I dont know02:11
mordredand lawyers lawyers02:11
* mordred throws a shoe at the law student on the couch02:11
jgriffithmordred: careful, you'll be slapped with a law suit02:12
mordredwouldn't be the first time ...02:12
anteayajeblair, agreed02:12
jgriffithSo we believe that what should happen here is to "add" not search/replace then?02:13
*** jerryz has joined #openstack-infra02:13
jgriffithI'm fine with that02:13
jgriffiththat makes sense02:13
jgriffithrewriting existing items makes no sense02:13
* anteaya wonders if they even sell mantle pieces big enough for plagues02:14
mordredjgriffith: no - lawyer people and the foundation have told us to replace them02:14
mordredmy ranting earlier does not constitute legal opinion02:15
mordredas much as I'd love it to02:15
mordredanteaya: I think plagues are actually usually quite small02:15
mordreda tiny vial of some powder02:15
*** emagana has quit IRC02:15
jgriffithmordred: oh... ok02:15
jgriffithmordred: well if that's the case I suppose I'll +2/A02:16
jgriffithmordred: thanks02:16
*** zyluo has joined #openstack-infra02:17
anteayamordred, oh okay guess I'm covered either way with a standard size mantle piece02:18
*** rfolco has quit IRC02:30
*** jhesketh has joined #openstack-infra02:33
*** morganfainberg has quit IRC02:37
*** morganfainberg has joined #openstack-infra02:37
*** ryanpetrello has quit IRC02:45
*** dims has quit IRC02:48
*** Ryan_Lane has joined #openstack-infra02:50
*** dguitarbite has joined #openstack-infra03:18
*** jerryz has quit IRC03:20
*** blamar has quit IRC03:20
*** pcrews_ has quit IRC03:21
*** emagana has joined #openstack-infra03:26
*** jhesketh has quit IRC03:30
*** emagana has quit IRC03:31
*** pcrews_ has joined #openstack-infra03:32
*** anteaya has quit IRC03:33
*** morganfainberg has quit IRC03:39
*** morganfainberg has joined #openstack-infra03:40
*** jhesketh has joined #openstack-infra03:42
*** pcrews_ has quit IRC03:45
*** enikanorov-w has quit IRC03:48
*** enikanorov-w has joined #openstack-infra03:49
*** zehicle has joined #openstack-infra03:51
*** SergeyLukjanov has joined #openstack-infra04:00
openstackgerritKhai Do proposed a change to openstack-infra/config: Setup a private gerrit instance for security reviews  https://review.openstack.org/4793704:02
*** dguitarbite has quit IRC04:09
*** DennyZhang has joined #openstack-infra04:13
*** morganfainberg has quit IRC04:16
*** morganfainberg has joined #openstack-infra04:19
*** hashar has quit IRC04:23
*** SergeyLukjanov has quit IRC04:24
*** vipul is now known as vipul-away04:27
*** morganfainberg has quit IRC04:35
*** morganfainberg has joined #openstack-infra04:35
*** DennyZhang has quit IRC04:36
*** DennyZhang has joined #openstack-infra04:36
*** reed has quit IRC04:43
*** boris-42 has joined #openstack-infra04:58
*** DennyZhang has quit IRC05:12
*** DennyZhang has joined #openstack-infra05:13
SpamapSgah... gate fails eating my brain05:14
*** DennyZhang has quit IRC05:15
*** DennyZhang has joined #openstack-infra05:17
*** DennyZhang has quit IRC05:21
*** DennyZhang has joined #openstack-infra05:21
*** nicedice has quit IRC05:30
*** blamar has joined #openstack-infra05:35
*** yaguang has quit IRC05:42
*** pblaho has joined #openstack-infra05:43
*** yaguang has joined #openstack-infra05:56
*** yolanda has joined #openstack-infra06:00
*** DennyZhang has quit IRC06:02
*** vipul-away is now known as vipul06:02
*** emagana has joined #openstack-infra06:22
jog0mtreinish: I updated the queries file06:33
*** afazekas has joined #openstack-infra06:37
*** emagana has quit IRC06:44
*** fbo_away is now known as fbo06:46
*** yaguang has quit IRC06:54
*** yaguang has joined #openstack-infra06:54
*** dkehn_ has joined #openstack-infra06:57
*** flaper87|afk is now known as flaper8706:58
*** dkehn has quit IRC07:01
*** jcoufal has joined #openstack-infra07:02
*** qba73 has joined #openstack-infra07:07
*** Ryan_Lane has quit IRC07:18
mikaljog0: you around enough for a quick question?07:19
*** hashar has joined #openstack-infra07:27
*** markmc has joined #openstack-infra07:28
*** rpodolyaka has joined #openstack-infra07:29
yaguangneed help , I push a patch half an hour ago,  all Jenkins gate fails  https://review.openstack.org/#/c/47965/07:34
*** SergeyLukjanov has joined #openstack-infra07:35
ttxclarkb: looks like we have intended behavior from update-bug as of 4 hours ago. Haven't seen a "related bug" comment yet, though07:36
*** tvb has joined #openstack-infra07:38
*** tvb is now known as Guest1663807:39
*** Guest16638 is now known as tvb07:43
*** tvb has joined #openstack-infra07:43
*** annegentle has quit IRC07:44
*** annegentle has joined #openstack-infra07:45
jd__mordred: arf, happy to learn cloud archive was reverted :-/07:45
jd__mordred: that seems like a critical problem, is there a bug number about that nova issue?07:45
*** Ryan_Lane has joined #openstack-infra07:49
*** yolanda has quit IRC07:51
*** dizquierdo has joined #openstack-infra07:51
*** SergeyLukjanov has quit IRC07:53
*** mkerrin has joined #openstack-infra07:53
*** kpavel has joined #openstack-infra07:56
*** Ryan_Lane has quit IRC07:56
*** SergeyLukjanov has joined #openstack-infra08:01
rpodolyakahmm... all tempest test seem to be constantly failing for new changesets08:14
rpodolyakaI don't see errors in console.log except "2013-09-24 08:06:52 ERROR: Module nbd does not exist in /proc/modules"08:15
rpodolyakadoes anyone see something similar too?08:15
*** qba73 has quit IRC08:16
*** Ryan_Lane has joined #openstack-infra08:24
*** Ryan_Lane has quit IRC08:28
*** zyluo has quit IRC08:34
afazekashttp://status.openstack.org/zuul/ all job failing08:39
SpamapS2013-09-24 07:53:44.566 | Downloading/unpacking pyparsing>=2.0.1 (from cliff>=1.4.3->python-neutronclient>=2.3.0,<3->-r /home/jenkins/workspace/gate-heat-pep8/requirements.txt (line 20))08:40
afazekaspep8: No distributions matching the version for pyparsing>=2.0.1 (from cliff>=1.4->-r /home/jenkins/workspace/gate-neutron-pep8/test-requirements.txt (line 3))08:40
SpamapS2013-09-24 07:53:44.566 |   Could not find a version that satisfies the requirement pyparsing>=2.0.1 (from cliff>=1.4.3->python-neutronclient>=2.3.0,<3->-r /home/jenkins/workspace/gate-heat-pep8/requirements.txt (line 20)) (from versions: 1.5.6, 1.5.6, 1.5.7, 1.5.7)08:40
SpamapSfungi: here?08:41
SpamapSclarkb: ?08:41
SpamapSjeblair: ?08:41
SpamapSmordred: ?08:41
* SpamapS looks for the big red panic button08:41
SpamapSlooks like a new version of cliff is pulling in an as-yet-un-synced pyparsing08:42
*** therve has joined #openstack-infra08:43
*** raies_ has joined #openstack-infra08:45
ttxSpamapS: we are on the dark side of the moon right now08:45
raies_anyone can help me on build failure of https://review.openstack.org/#/c/47339/08:45
afazekasraies_: all job failing http://status.openstack.org/zuul/08:46
raies_afazekas: then what I should do ?08:47
ttxraies_: at this point, wait a bit08:47
SpamapSyeah I think the mirror sync may fix it08:47
* ttx tries to update infra status08:48
raies_I think I need not to upload further patches ??08:48
SpamapSraies_: you can do a recheck on them later if you just want them up for review08:49
afazekaswait until the pep8 checks start working on the http://status.openstack.org/zuul/ , after that probably you can do a 'recheck no bug'08:49
raies_thanks to all of you. :)08:50
SpamapSlooks like the mirror up 'n failed08:50
ttxwhere is te statusbot when you need it08:52
SpamapSttx: any chance you have privileges on jenkins.o.o ?08:52
ttxSpamapS: I don't08:54
SpamapSOk, because I think that job just has to be re-run08:55
ttxit also looks like I'm not whitelisted to openstackstatus bot08:55
*** BobBall_Away is now known as BobBall08:58
SpamapSttx: for some reason I thought we had an EMEA infra person09:00
ttxSpamapS: we don't09:03
SpamapSSo, IIRC, fungi and mordred are UTC-409:04
SpamapS(assuming mordred is actually at home.. big assumption)09:04
SpamapSjeblair and clarkb are UTC-709:04
SpamapSttx: did you log in to jenkins and actually check that you don't have rights to start jobs?09:12
SpamapS../manifests/site.pp:    statusbot_auth_nicks    => 'jeblair, ttx, fungi, mordred, clarkb, sdague',09:12
chmouelin the url http://logs.openstack.org/47/45447/9/check/gate-tempest-devstack-vm-large-ops/0cc80b5/09:19
chmouelwhere is the 0cc80b5 come from?09:19
chmoueljeblair: ^09:20
chmouelthey seems to be random between all jobs (so not tight to commit)09:20
chmoueland have multiple of them09:20
openstackgerritJulien Danjou proposed a change to openstack/requirements: Bump ceilometerclient dependency  https://review.openstack.org/4799409:22
*** boris-42 has quit IRC09:27
*** dizquierdo has left #openstack-infra09:30
*** tvb has quit IRC09:35
*** tvb has joined #openstack-infra09:36
*** tvb is now known as Guest2285409:36
*** yolanda has joined #openstack-infra09:36
*** Guest22854 is now known as tvb09:37
*** tvb has joined #openstack-infra09:37
*** jhesketh_ has joined #openstack-infra09:53
*** thomasbiege1 has joined #openstack-infra09:58
*** dizquierdo has joined #openstack-infra10:16
*** ndipanov has joined #openstack-infra10:19
ndipanovhey infra folks - nova gate seems to be failing badly due to pip struggling with pyparsing versions10:22
*** thomasbiege1 has quit IRC10:22
*** fbo is now known as fbo_away10:25
*** thomasbiege has joined #openstack-infra10:25
*** mordred has quit IRC10:29
*** jesusaurus has quit IRC10:30
*** mordred has joined #openstack-infra10:31
jd__ndipanov: ah there's not nova only actually10:33
jd__I just hit it on ceilometerclient10:34
ndipanovjd__, is infra using some mirrors/caching for pypi?10:34
ndipanovjd__, since pyparsing seems to install fine on my laptop10:34
ndipanovwith thw same pip args10:34
*** schwicht has joined #openstack-infra10:39
*** thomasbiege has quit IRC10:40
*** thomasbiege has joined #openstack-infra10:40
*** thomasbiege has quit IRC10:47
*** alexpilotti has joined #openstack-infra10:48
*** jesusaurus has joined #openstack-infra10:49
*** yaguang has quit IRC10:55
*** boris-42 has joined #openstack-infra11:03
*** zhiyan has joined #openstack-infra11:12
zhiyanadarazs_: ping11:12
adarazs_zhiyan: pong11:13
*** adarazs_ is now known as adarazs11:13
zhiyanadarazs_: sorry to interrupt you, i have a quick ask11:13
sdaguedhellmann seems to have released a new cliff which bumped the min version of pyparsing beyond what's in global requirements11:13
sdagueso it's created a wedge11:13
adarazszhiyan: okay, though I'm not sure you really want to ask me :)11:13
*** pcm_ has joined #openstack-infra11:14
ttxSpamapS: sorry, got sidetracked, French IRS surprise investigation11:14
ttxA great way to lose 3 hours of work.11:14
zhiyanadarazs: :) we have a change at https://review.openstack.org/#/c/47161/ , event it not get approve but it's a change on the right way, but as Jenkins said, the change break the test.11:15
adarazszhiyan: check my name, I don't think I'm the nickname you're looking for :)11:15
ttxI /should/ be able to update statusbot but it's not really working11:15
openstackgerritSean Dague proposed a change to openstack/requirements: uncap pyparsing requirement  https://review.openstack.org/4801411:16
zhiyanadarazs: so i have a question is that, how can we landing this type change? seems it need disable some test case. and after this change landing we will change devstack to work with it (actually we did it at https://review.openstack.org/#/c/47552/ )11:16
adarazszhiyan: I don't know what you're talking about, I don't think you meant to talk to me.11:16
zhiyanadarazs: oh, sorry.11:17
zhiyanok folks who can give some light to me?11:17
*** dkranz has quit IRC11:17
zhiyanfungi ^^11:18
sdaguezhiyan: you have to have the tool support deprecation, you can't just remove one set of commands and change them to something else11:18
zhiyansdague: make sense...11:20
zhiyansdague: thanks, will think about it.11:20
zhiyansdague: btw, for more common scenario, if a change break the tempset test, but it's a right change, how can i handling it? just ask disable some test case to ask it land down? and then to change the test? is it a correct approach?11:26
sdaguezhiyan: in the general case, it needs a bug, and the core team on the project to have at least one +2 on the patch you want to change. Then we can do a skip, land, and unskip with the new behavior. Once we get past release I'll write more of this down11:28
sdagueanyone know how often the mirrors are build to infra?11:30
sdaguefungi you up yet?11:30
ttxsdague: not everyone gets up at 7am like you do :)11:31
sdagueyep, but fungi is at least in my tz :)11:31
sdagueso I think we actually have to do that cap instead11:32
sdaguebecause the infra mirror takes time11:32
ttxlet's all blame dhellmann!11:34
* ttx fetches torches and pikes11:34
openstackgerritSean Dague proposed a change to openstack/requirements: cap cliff to < 1.4.5  https://review.openstack.org/4801911:35
sdagueok, there is the alternate patch11:35
ttxsdague: at this point in release it might actually make more sense.11:35
sdagueI was torn on what to propose first11:36
sdaguejd__: how do you feel about - https://review.openstack.org/#/c/48019/ ?11:37
sdagueman, we need a .eu infra person :)11:37
*** fbo_away is now known as fbo11:43
sdagueug, there is another issue, even on the cliff pin11:44
openstackgerritMark McLoughlin proposed a change to openstack/requirements: Use oslo.config 1.2.0  https://review.openstack.org/4802211:44
sdaguehttps://jenkins02.openstack.org/job/gate-grenade-devstack-vm/10635/console - cliff isn't working with grizzly versions of python distutils11:45
sdagueok, this is going to be unwinding for hours I expect11:46
*** ArxCruz has joined #openstack-infra11:47
boris-42sdague hi11:48
boris-42sdague do you know something about http://logs.openstack.org/13/48013/2/check/gate-rally-pep8/5661b67/console.html why I am not able to install gevent ?)11:49
*** thomasbiege has joined #openstack-infra11:51
rpodolyakaboris-42: apparently, libevent1-dev package is not installed11:52
*** thomasbiege has quit IRC11:52
boris-42rpodolyaka do you know it is possible to install it?)11:52
rpodolyakaboris-42: not sure, whether we have policy what libs can be installed on Infra VMs11:52
boris-42yeah yeah..11:53
rpodolyakaboris-42: so Infra guys should know better :)11:53
boris-42fungi hi11:54
sdaguewhy would be installing gevent in a pep8 run?11:54
boris-42sdague actually I don't know why OpenStack CI does this=)11:54
boris-42sdugue https://review.openstack.org/#/c/48013/11:55
boris-42here is the problem patch sdague ^11:55
*** yaguang has joined #openstack-infra11:57
sdagueyour out of tree test suite is doing it11:57
sdague2013-09-24 11:45:24.123 | Downloading/unpacking gevent==0.13.8 (from fuel-ostf-tests->-r /home/jenkins/workspace/gate-rally-pep8/requirements.txt (line 15))11:57
sdague2013-09-24 11:45:24.123 |   Running setup.py egg_info for package gevent11:57
sdaguethat's one of the reasons you really shouldn't pip install random external urls11:57
*** mriedem has joined #openstack-infra11:57
*** mriedem has quit IRC11:58
boris-42sdague it is not random url=)11:58
sdagueit's not in our git... so I consider it random :)11:58
boris-42sdague it is the future part of this system..11:58
*** amotoki has quit IRC11:59
boris-42sdague it will be soon in stackforge..11:59
sdagueyeh, then you should put it into a stackforge git tree11:59
*** sandywalsh has quit IRC11:59
sdagueright, well that's the crux of your problem. The requirements in the out of tree tarball11:59
jd__sdague: I think we can kick in the mirroring earlier, no?12:00
* jd__ didn't read the backlog yet12:00
jd__ttx: sorry about IRS :]12:00
boris-42sdague by the way how we will fix this problem?12:01
boris-42sdague when we will be on stackforge..12:01
sdagueboris-42: don't know, and working on gate break now, so it will have to wait12:01
boris-42sdague but it will be ok to add gevent in global requirements?12:02
sdaguejd__: so.... even with the mirror update - http://logs.openstack.org/14/48014/1/check/gate-grenade-devstack-vm/386eb51/console.html seems not solvable12:02
jd__sdague: what what what is happening! /o\12:02
*** afazekas has quit IRC12:02
jd__sdague: ok you got me, +2'ed12:03
sdaguejd__: so grizzly devstack version of python tools apparently don't work with cliff 1.4.512:03
*** dims has joined #openstack-infra12:08
*** thomasm has joined #openstack-infra12:10
zhiyansdague: got it, thanks.12:10
*** sandywalsh has joined #openstack-infra12:12
*** mriedem has joined #openstack-infra12:17
*** dkranz has joined #openstack-infra12:18
sdagueso I think this - https://review.openstack.org/#/c/48032/12:19
sdagueand then cliff pin, will get the spice flowing again12:19
*** afazekas has joined #openstack-infra12:20
openstackgerritDolph Mathews proposed a change to openstack/requirements: remove oslo.config from dev-requirements exceptions  https://review.openstack.org/4803312:20
openstackgerritDolph Mathews proposed a change to openstack/requirements: remove oslo.config from dev-requirements exceptions  https://review.openstack.org/4803312:22
*** weshay has joined #openstack-infra12:25
*** cppcabrera has joined #openstack-infra12:28
*** dims has quit IRC12:29
*** dims has joined #openstack-infra12:30
*** zhiyan has left #openstack-infra12:30
sdagueman... I hope dhellmann isn't on a plane today12:30
*** dolphm has joined #openstack-infra12:31
*** dkliban has joined #openstack-infra12:33
*** adalbas has joined #openstack-infra12:34
*** EmilienM has quit IRC12:35
sdaguejd__: once this merges - https://review.openstack.org/#/c/48032/ - can you recheck the cliff capping review?12:36
sdagueI'll probably be in transit12:36
*** EmilienM has joined #openstack-infra12:36
jd__sdague: yes12:37
dhellmanngood morning12:41
dhellmannjd__, sdague : that cliff release was supposed to fix the debian packaging issue with pyparsing12:42
dhellmannjd__, sdague : what's going on now?12:42
jd__dhellmann: it can be installed with the Python env of Grizzly for whatever reason12:42
jd__dhellmann: and good morning! ;-)12:42
dhellmannjd__: log?12:42
jd__dhellmann: http://logs.openstack.org/14/48014/1/check/gate-grenade-devstack-vm/386eb51/console.html12:43
dhellmannjd__: that's the "setuptools tries to upgrade itself with distribute" problem12:44
jd__ah, the grand classic12:44
dhellmannwell, I don't see distribute in the log so maybe it's just setuptools trying to upgrade itself12:45
jd__dhellmann: what's the solution to that?12:45
dhellmannfind the thing that's trying to update setuptools as a dependency, and remove the dependency; then ensure the latest setuptools is installed at the start of the job12:45
*** weshay has quit IRC12:46
*** mriedem has quit IRC12:46
*** tvb has quit IRC12:46
*** ndipanov has quit IRC12:46
*** jhesketh has quit IRC12:46
*** zul has quit IRC12:46
*** Reapster has quit IRC12:46
*** clayg has quit IRC12:46
*** locke105 has quit IRC12:46
*** jcooley has quit IRC12:46
*** jeremyb has quit IRC12:46
*** rockstar has quit IRC12:46
*** chmouel has quit IRC12:46
*** jcooley has joined #openstack-infra12:46
dhellmannor just pin cliff in grizzly12:46
*** rockstar has joined #openstack-infra12:46
*** clayg has joined #openstack-infra12:46
*** Reapster has joined #openstack-infra12:46
*** jeremyb has joined #openstack-infra12:46
dhellmannthat's probably the simplest solution12:46
*** weshay has joined #openstack-infra12:46
*** tvb has joined #openstack-infra12:46
*** rockstar has quit IRC12:46
*** rockstar has joined #openstack-infra12:46
jd__that's what sdague did I think with https://review.openstack.org/#/c/48032/12:46
*** mriedem has joined #openstack-infra12:46
*** tvb has quit IRC12:46
*** tvb has joined #openstack-infra12:46
*** DennyZhang has joined #openstack-infra12:47
dhellmannI guess I should have called that cliff 1.5 :-(12:47
*** ndipanov has joined #openstack-infra12:47
*** zul has joined #openstack-infra12:47
*** ndipanov has quit IRC12:47
*** ndipanov has joined #openstack-infra12:47
*** chmouel has joined #openstack-infra12:47
dhellmannthat change was in master, though12:47
zulugh new cliff?12:47
* jd__ investigates12:48
*** jhesketh has joined #openstack-infra12:48
dhellmannzul: just trying to fix a dependency issue on pyparsing, no code changes12:48
*** locke105 has joined #openstack-infra12:48
*** jhesketh_ has quit IRC12:49
dhellmannat least no important changes12:49
zuldhellmann:  ah the pyparsing 2.0 thing?12:49
dhellmannzul: yeah, pyparsing 2.0.1 works with python 2 and 3, and cliff 1.4.5 takes advantage of that12:49
*** jhesketh_ has joined #openstack-infra12:49
jd__pyparsing, sourceforge hmmm12:49
zuldhellmann:  cool so i can drop a patch we carry :)12:49
dhellmannzul: patch?12:50
zuldhellmann:  we are shipping pyparsing 2.0 so we had to force cliff to use pyparsing 2.012:50
dhellmannzul: how did you make that work on python 2?12:50
sdaguedhellmann: so the issue is actually that new clients have to work with old servers, so clients pull in new cliff12:51
*** rfolco has joined #openstack-infra12:51
sdaguewhich is actually what grizzly devstack tripped over12:51
zuldhellmann: patched the setup.py i think12:51
dhellmannsdague: ah, gotcha12:51
dhellmannsdague: should I do a 1.4.6 that undoes 1.4.5 and then a 1.5 that redoes it?12:51
sdaguedhellmann: I think that might work12:52
zuldhellmann: like https://github.com/dreamhost/cliff/commit/6a35ba1b0d7b40eaa184769400ab7445427925dc12:52
dhellmannzul: maybe the issue was just with getting pyparsing 2 to install, then, but I thought it had python 3 code in the lib12:52
sdaguethough did you see the log in grenade with setuputils ?12:52
zuldhellmann:  ah i dont necessarily care about python3 yet for cliff since i havent gotten to clients that use it yet12:52
dhellmannzul: I meant how did you get pyparsing 2 to work with python 212:52
zuldhellmann:  nothing special tests pass with it12:53
sdaguedhellmann: https://jenkins02.openstack.org/job/gate-grenade-devstack-vm/10644/console - as long as you think you can get a 1.4.6 to work where 1.4.5 failed there, I think we'll be good12:53
jd__I can't seem to find what triggers that problem12:53
sdaguethen we can start unwinding the work arounds12:53
dhellmannsdague: that error usually means something tried to upgrade setuptools as a dependency12:54
sdaguedhellmann: ok12:54
*** kgriffs_afk is now known as kgriffs12:54
*** mkoderer_ is now known as mkoderer12:54
sdagueanyway, I need to move into the real office. Back on in 30 or so12:54
jd__2013-09-24 12:31:18.147 | 2013-09-24 12:31:18   File "setup.py", line 22, in <module>12:54
jd__2013-09-24 12:31:18.149 | 2013-09-24 12:31:18     pbr=True)12:54
jd__so it's something using pbr, not cliff12:55
jd__but the log talk about cliff just before that12:55
jd__so I don't get how what's happening12:55
dhellmannsdague: ok, I'm going in this morning, too, so it will be a couple of hours before I can really work on this12:55
*** tkammer has joined #openstack-infra12:56
dimsHow do i annotate reviews busted by the cliff bug? is this summary ok? "Caused by bug 1229647. need to wait for https://review.openstack.org/#/c/48019/ to land."12:56
uvirtbotLaunchpad bug 1229647 in grenade "TypeError" [Undecided,New] https://launchpad.net/bugs/122964712:56
*** pcm_ has quit IRC12:56
*** dprince has joined #openstack-infra12:56
*** pblaho has quit IRC12:57
*** pblaho has joined #openstack-infra12:57
*** dolphm has quit IRC13:00
dhellmannjd__: neutron client has cliff>=1.4.3, but no upper cap, so it wouldn't have mattered what version number I used for this release. :-/13:01
jd__dhellmann: I don't think it matters indeed13:02
*** afazekas has quit IRC13:03
openstackgerritDavid Shrewsbury proposed a change to openstack-infra/config: Add #stackforge-libra to gerritbot  https://review.openstack.org/4804013:03
*** tkammer has quit IRC13:03
dhellmannjd__, sdague : I'm spinning up a grizzly dev env to try to reproduce this13:03
*** yaguang has quit IRC13:04
*** tkammer has joined #openstack-infra13:04
*** dkliban has quit IRC13:06
*** yolanda has quit IRC13:07
*** yolanda has joined #openstack-infra13:08
*** julim has joined #openstack-infra13:12
fungiwow, so much scrollback13:12
*** pblaho has quit IRC13:13
mordredfungi: I was just thinking the same thing13:14
tkammerhey guys, is there a known issue with zuul at the moment?13:15
jd__gate's broken13:15
*** jhesketh has quit IRC13:16
dimsstaring at 48019,1 on http://status.openstack.org/zuul/ - that's probably the first one since https://review.openstack.org/48032 landed13:16
*** jhesketh has joined #openstack-infra13:17
jd__mordred, fungi: if you missed my earlier message in this brouhaha, I wanna know more precisely why cloud archive has been reverted :(13:18
* fungi looks to see if there's still new pyparsing badness in the gate13:18
*** dkehn_ is now known as dkehn13:19
mordredjd__: I _think_ zul is looking at the nova/libvirt thing. I believe sdague is too13:20
*** ryanpetrello has joined #openstack-infra13:21
mordredSpamapS: even if ttx had perms to start jobs in jenkins, it doesn't work that way, you can't just run a single job in jenkins - it needs to be run by zuul13:21
*** ryanpetrello has quit IRC13:21
fungisdague: and yeah, while i'm usually up earlier, i got talked into a few rounds of scrabble last night which ran unusually late13:21
mordredndipanov: yes. we use a mirror13:21
*** thomasbiege has joined #openstack-infra13:22
*** cthulhup has joined #openstack-infra13:22
dhellmannjd__: it looks like the global requirements for stable/grizzly have pyparsing>=1.5.6,<2.0 -- is that what you see? devstack refused to install nova for me at all because of the dependency conflict13:23
dhellmannjd__: so I didn't even get to the point where the error in the logs occurred13:23
mordredok. I gave up on reading all of the logs -13:23
mordreddhellmann: how can I be helpful re: the cliff stuff13:24
dhellmannmordred: http://logs.openstack.org/14/48014/1/check/gate-grenade-devstack-vm/386eb51/console.html13:24
jd__dhellmann: weird, maybe grenade is not using it?13:24
dhellmannmordred: the error there looks like the "setuptools upgrading itself as a dependency" issue13:24
mordredyes. that looks exactly like that error13:24
*** dizquierdo has quit IRC13:24
mordreddhellmann: is that grenade run trying to upgrade to latest cliff?13:24
dhellmannbut we can't figure out *what* caused the upgrade13:24
openstackgerritJulien Danjou proposed a change to openstack-infra/devstack-gate: Enable Ceilometer alarm services  https://review.openstack.org/4804613:25
dhellmannI don't know exactly what grenade does. is that the upgrade tester thing?13:25
mordredit is13:25
dhellmannok, I'm not sure how to reproduce that locally13:25
mordreddhellmann: it looks like it's still doing setup.py develop - and I think we've changed devstack to do pip install -e .13:25
fungiit's probably been said at this point, but problems encountered like that where a pip install of something tries to upgrade setuptools usually means something is incorrectly specifying a dependency on setuptools13:25
dhellmannwhat I've tried is running stable/grizzly devstack and that fails on a version conflict when it tries to install nova13:26
dhellmannfungi: yeah, I looked through the dependency chain for cliff and couldn't find anything like that13:26
ndipanovmordred, I assumed so, but couldn't see it specified in the pip arguments so I gave up and sent an email :)13:27
dhellmannfungi: and I don't see that happening in the logs, either, so I'm stumped13:27
*** cthulhup has quit IRC13:27
dhellmannmordred: is that substantially different?13:27
ttxmordred: in related news, I tried to /msg openstackstatus to warn everyone but it ignored me13:28
mordreddhellmann: yes13:28
ttxmordred: I'm in the whitelist though, dunno why it fails13:28
* dhellmann sighs13:28
fungiokay, i'm now caught up on scrollback (in this channel at least)13:28
mordredI'm about to propose a patch to devstack stable/grizzly13:28
sdaguedhellmann: so 1.5 wouldn't have prevented this break, however it would have at least been signaling that something larger was comming13:28
sdaguemordred: which piece of the fix are you running at stable/grizzly?13:28
dhellmannsdague: yeah. The way I read the semantic versioning instructions implied that a point release was ok.13:29
*** changbl has quit IRC13:29
fungittx: i'll dig up the statusbot docs for you in a sec, but it doesn't support /msg i don't think. we usually just do...13:30
fungi#status alert dependency problems in gating, currently under investigation... more news as it unfolds13:30
openstackstatusNOTICE: dependency problems in gating, currently under investigation... more news as it unfolds13:30
*** ChanServ changes topic to "dependency problems in gating, currently under investigation... more news as it unfolds"13:30
dhellmannsdague: this was a no-code, bug fix, change :-/13:30
*** markmcclain has quit IRC13:30
sdagueyeh, the dependency graph though really means like a merge commit, the impact is the entire dependency tree below you13:31
ttxfungi: oh!13:31
ttxfungi: I guess i'm spoiled by chanserv/nickserv13:32
sdaguedhellmann: I guess we need to get on that automatically running after out dependencies thing13:32
fungittx: http://ci.openstack.org/irc.html#statusbot13:32
fungi(for future reference)13:32
ttxfungi: yes, I read that but assumed it required /msg13:32
dhellmannsdague: ok, I think I see what you mean there.13:33
fungittx: aha, got it. we should probably clarify in the doc. i agree it's vague on that point13:33
dimssdague, 48019,1 seems to have gotten through (looking at http://status.openstack.org/zuul/)13:33
dhellmannsdague: it seems like neutronclient should not be specifying the version of pyparsing they want, since they don't use it directly13:34
dimsone thing still running (https://jenkins01.openstack.org/job/gate-requirements-install/284/console)13:34
sdaguedhellmann: sorry, language not all there, still caffinating :)13:34
dhellmannsdague: me, too :-)13:34
sdaguedims: yep, that's also a devstack run, so we need to make sure that works as well13:34
jd__https://review.openstack.org/#/c/48019/ is almost ready to got but has timeout issue13:34
dhellmannsdague: let's just say I'll try to remember to bump the 2nd digit next time13:35
dimssdague, ouch, that one kicked the bucket13:35
sdaguewhich it didn't....13:35
sdaguemordred: http://logs.openstack.org/19/48019/1/check/gate-requirements-install/9562d98/console.html this bug known to you?13:35
sdaguethat's a fail I don't recognize13:36
sdaguedhellmann: works for me :)13:36
fungiwow, that job doesn't usually take so long13:36
fungisdague: "Build timed out (after 30 minutes). Marking the build as failed."13:36
sdagueyeh, exactly13:37
fungisdague: the gate-requirements-install job is currently limited in jenkins/jjb to 30 minutes13:37
dims"timed out (after 30 minutes)"13:37
fungiso that's jenkins killing it at that point13:37
fungibigger question is why it took so long--doesn't usually go so slowly13:37
sdaguefungi: right, exactly13:38
sdaguefungi: what does that job actually do?13:38
*** matsuhashi has quit IRC13:38
sdaguewe timing out on a network connection or something?13:38
fungiit tests installing everything listed in the global-requirements and dev requirements to make sure they specify valid existing package versions on pypi.python.org and that they're sanely coinstallable13:39
fungiso that we don't land conflicting requirements13:39
*** matty_dubs|gone is now known as matty_dubs13:40
mordredsdague: https://review.openstack.org/4804913:40
mordredsdague: for one thing13:41
dimsfungi, seems to have 4-5 min pauses in between13:41
sdaguemordred: ok, cool13:41
fungidims: it's possible there's something wrong with the slave where that job ran--i'm checking it out now13:41
jd__sdague: fungi: this is not the first time that timeout issue happens today13:42
sdaguejd__: interesting13:42
jd__it just seems damn slow for an unknown reason13:43
mordredsdague: I don't know that that will get us out of the hole, but it certainly needs the backporting :)13:43
sdaguediskimage-builder is sneaking in pypi urls directly? - https://jenkins02.openstack.org/job/gate-requirements-install/200/console13:44
dimsfungi, watching this log now. saw one 5 min pause already - https://jenkins02.openstack.org/job/gate-requirements-install/200/console13:45
fungii don't see anything concerning about the slave where that job ran, though i don't have a great way to check i/o performance (since i don't have a good baseline to compare against), but other system resources are basically clear and no noticeable packet loss/latency for it either13:46
fungii'll hop on that one and see what it's doing exactly13:46
fungisdague: gate-requirements-install hits pypi.python.org directly because it needs to test combinations which aren't on our mirror yet13:47
sdaguemordred: yeh, the backport would be a good thing. If that passes check I'll merge it13:47
mordredsdague: actually, I just amended it to add the pip install/upgrade stuff13:47
sdaguefungi: ok, which means network burps will kill us, right?13:48
sdaguewhich seems to be what's happening13:48
fungisdague: absolutely13:48
sdagueso we can't unfuxor ourselves....13:48
mordredsdague: what's the unfuxor we need?13:48
fungisdague: well, i don't see network problems with the slave, but possibly the fastly cdn in front of pypi has issues, or something on the route to it13:48
* mordred trying to catch up on everything13:49
sdaguemordred: we have to cap cliff in global requirements13:49
*** dkliban has joined #openstack-infra13:49
sdagueno code has merged on any project since cliff 1.4.5 hit the mirrors13:49
mordredis that just because of the grenade errors?13:50
jd__sad day13:50
mordredor are there others?13:50
sdaguealso because cliff 1.4.5 requires pyparsing >= 2.0.113:50
sdaguewhich isn't in requirements13:50
mordredoh for the love of13:50
mordredthe pyparsing version thing is goingto be the death of me13:50
dhellmannthis was supposed to be the fix for that13:51
mordredyeah, I know13:51
dhellmannhowever, I'm seriously considering cliff 2.0 not using cmd2 or pyparsing13:51
mordredso, we can't get the mirror to update with new pyparsing because of pypi fail13:51
mordredand we can't get the requirements cap in because of, well, pypi fail13:51
sdagueright, exactly13:51
dhellmannthe mirror update failure isn't causing the grenade failure, is it?13:52
mordredno. that's setuptools upgrade love13:52
sdaguedhellmann: no, the grenade failure was the old setuptools issue13:52
sdagueand I've landed a grenade workaround for that13:52
jd__dhellmann: nor Python :]13:52
sdaguewhich just installs cliff upfront13:52
dhellmannoh, is that the pip change?13:52
sdaguedhellmann: https://review.openstack.org/#/c/48032/13:53
sdaguethat is a very tactical work around13:53
dimsdhellmann, latest run  with pyparsing>=2.0.1 problem - https://jenkins02.openstack.org/job/gate-nova-python26/4127/console13:53
sdagueto be removed as soon as the rest of the system comes together13:53
mordredI think we need a smarter mirror script thing13:54
dhellmanndims: ok, that's the mirror update failure breaking devstack13:54
*** nosnos has quit IRC13:54
dhellmannwhen I run stable/grizzly devstack I get an error installing nova because of a conflict on requirements for pyparsing -- how does grenade get past *that*?13:55
sdaguedhellmann: actually, that's the wedge13:55
sdaguedhellmann: because we install cliff upfront at 1.4.413:55
dhellmannduh, ok13:55
*** nosnos has joined #openstack-infra13:55
sdaguethe cliff requirement is actually >= 1.4.3 so nothing else installs newer cliff13:55
* dhellmann drinks more caffeine13:56
dimsdhellmann, :)13:56
dhellmannmordred: smarter how?13:56
dhellmannmordred: it might be easier to just set up a full pypi mirror, and have the restricted mirror script pull from that13:57
mordreddhellmann: one that can respond to such events as "we need package X in the mirror please" or "this is the _change_ in reqs"13:57
mordreddhellmann: oy13:57
mordreddhellmann: full pypi mirror is hard and doesn't really work because none of the full mirroring software is worth anything13:57
mordreddhellmann: because it all respects the external links bs13:57
fungiyeah, stracing the pip install process, it does actually seem to get what i think are 404 errors and then tries several x.pypi.p.o mirrors on downloads to no avail, before ending up in a futex_wait_private/egain (resource temporarily unavailable) loop for a while13:58
*** nosnos has quit IRC13:58
dhellmannmordred: ok, well, that can be fixed. I thought the issue was with timeouts, not the logic.13:58
mordredit's both13:58
fungiso i think those are the pauses we're seeing13:58
fungino child processes that i can see either13:58
mordredwhat I'm saying about smarter mirror script13:58
mordredis that it would e nice to be able to put in better retry loops13:58
*** dolphm has joined #openstack-infra13:58
mordredfungi: wait - we're trying the mirrors?13:59
fungimordred: pip seemed to be13:59
mordredin our mirror build? we should turn those off13:59
fungii saw references to a.pypi.python.org and b.pypi.python.org in the strace13:59
mordredok. well, that's at least one thing we can do13:59
fungimordred: in the gate-requirements-install job13:59
mordredfungi: where is that job running?14:01
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Tweak gate-requirements-install  https://review.openstack.org/4805514:02
*** thomasbiege has quit IRC14:03
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Tweak gate-requirements-install  https://review.openstack.org/4805514:03
mordredfungi: ^^14:03
dimsmordred, there's one running here now - https://jenkins01.openstack.org/job/gate-requirements-install/285/console14:03
fungimordred: http://paste.openstack.org/show/47426/14:03
mordredbtw - I'm not sure we should even keep gate-requirements-install14:03
sdaguemordred: that's basically covered now that we have the global requirements sync in devstack, right?14:04
mordredsince we actually have the pbr integration test running a more complete version of this14:04
fungididn't we discuss that it's basically redundant now that devstack uses the requirements list?14:04
fungioh, and pbr yes14:04
*** EmilienM has quit IRC14:04
fungiand you and sdague also just said those things14:04
*** hashar has quit IRC14:04
mordredso it's just more places for pypi to bork on us14:04
mordredwhy is openstack/requirements not triggering the other requirements tests?14:05
fungithat seems like an oversight14:06
sdagueso what triggers the mirror update?14:07
mordredoh - nevermind. looking at wrong file14:07
*** EmilienM has joined #openstack-infra14:07
*** yaguang has joined #openstack-infra14:08
*** thomasbiege has joined #openstack-infra14:08
sdaguemordred: so... the devstack stable/grizzly backport is hitting the issue that pyparsing 2.0.1 isn't in our mirror14:09
fungisdague: three things: a daily timer, changes merging to the requirements project, and pypi uploads of some of our library projects which other projects of ours depend upon14:09
openstackgerritMonty Taylor proposed a change to openstack-infra/config: Remove gate-requirements-install  https://review.openstack.org/4805814:09
sdaguefungi: so how do you handle the propose case?14:10
mordredsdague: we might have to ninja approve the requirements change then14:10
fungisdague: proposed changes to the requirements project don't use our mirror14:10
mordredif the requirements change to cap it is hitting issues14:10
sdaguefungi: ... the devstack jobs seem to be14:10
fungior at least aren't supposed to14:10
*** rcleere has joined #openstack-infra14:11
fungii believe the mirror selection script is supposed to set it to use pypi.python.org when it's testing a requirements project change14:11
sdaguemordred: yeh, before ninja approving things, I think we should figure out how the jenga blocks are currently stacked, because I'm afraid of us wedging things later14:11
* fungi looks14:11
mordredsdague: TOTALLY AGREE14:11
sdaguefungi: ok, I can tell you that's definitely not happening14:12
mordredthis seems like an overly borked pile of jenga14:12
sdaguefungi: https://review.openstack.org/#/c/48014/ being the example14:12
mrodden^ phrase of the day14:12
uvirtbotmrodden: Error: "phrase" is not a valid command.14:12
mroddeni have many "overly borked piles of jenga" to fix today...14:13
sdaguearen't emergent systems fun....14:13
fungisdague: the idea, at least, is encoded at http://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/select-mirror.sh#n3514:13
mordredok. quick question. I think we've got too many workarounds for me to remember what our ultimate end-state is14:13
funginow to figure out why it's not doing that14:13
mordredsdague: in a perfect world, do we _want_ to be at pyparsing >= 2.0.114:13
mordredand/or, would force-adding pyparsing 2.0.1 to our mirror by hand unstick us14:14
sdaguemordred: sure, though I guess is there a reason we can't be at >= 1.5.7?14:14
*** thedodd has joined #openstack-infra14:14
mordredwell, I more mean - adding 2.0.1 by hand to our mirror would not be hard14:14
fungiin fact, it would be too easy, and would feel like cheating14:14
mordredif we think that would get us past the log jam in a way that doesn't involve landing interrelated patches14:14
sdagueso.... the only issue is stable14:15
mordredok. what's that issue14:15
sdagueso on stable, where we don't have a single req list, if we force the mirror to 2.0.1, is that going to break us?14:16
sdaguegiven that stable is using master clients14:16
mordredis stable capped?14:16
sdaguewell, it's scattered14:16
dhellmannI'm 75% to the point of not needing pyparsing in cliff, if that helps.14:16
sdagueI don't think we have a cap14:16
*** vipul has quit IRC14:16
*** Ghe_Holidays has quit IRC14:17
mordredI want to die14:17
sdaguemordred: but I want to understand one other thing first14:17
sdaguewhy when I uncapped pyparsing, didn't that trigger mirror updates in the check side/14:17
*** dcramer_ has joined #openstack-infra14:17
sdagueand if I bump it to >= 2.0.1 will it?14:17
mriedemhey guys, i'm seeing flake8 failures getting through the gate in nova14:18
uvirtbotLaunchpad bug 1229753 in nova "Several flake8 errors in new xenserver code" [Undecided,New]14:18
sdaguemriedem: it's going to need to wait until we get the rest of the gate fixed14:18
mordredsdague: mirror updates only happen in the post queue14:18
mriedemsdague: ok, didn't know there were current issues, just joined. i'll just work on fixing the nova bug for now (whack a mole i guess)14:19
mordredmriedem: if it's xenserver code, is it in files that might need requirements that aren't normally instlaled?14:19
mriedemmordred: nope, pure flake8 failures14:19
mriedemfails locally for me14:19
mordredmriedem: k. will come back to it later14:19
fungiokay, so it doesn't look like any of our devstack jobs use select_mirror.sh, unless i'm missing it somewhere14:19
mordredfungi: facepalm14:20
fungier, seelct-mirror.sh14:20
mordredfungi: wait - don't we do that in d-b?14:20
openstackgerritSean Dague proposed a change to openstack/requirements: uncap pyparsing requirement  https://review.openstack.org/4801414:20
fungiyeah, looks like devstack-vm-gate-wrap.sh hard-sets mirrors14:20
mordredfungi: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh#L18814:21
mordredfungi: select_mirror.sh should be installed on d-g nodes, because they're slaves, yeah?14:21
mordredfungi: how about just having d-g call select-mirror.sh14:21
mordredso that we don't have duplicate logic14:22
fungimordred: probably would make sense, yes14:22
sdagueso, we should probably figure out how to get that status bot to update folks on the issue14:22
mordredoh - fungi and I can do that14:22
fungisdague: nothing to figure out, but what's the new status since i last updated it?14:22
* mordred always screws it up though14:22
mordredsdague: fwiw, I've got a command locked and loaded to force in pyparsing 2.0.1 if we decide we want that14:23
fungi"changes being applied to unstick pyparsing dependency conflict in the gate, eta 1600 utc"?14:24
sdaguemaybe, we don't really have an eta yet, as we've not quite managed to figure out a version that gets us through14:24
sdaguemordred: can you just jam that into the mirror?14:24
mordredsdague: yes14:24
sdagueok, lets do that14:25
mordredyeah - we've kind of hitting up against the systemic issue which is reuqirements in stable branches14:25
sdagueyeh, well, I get voted down every time I want to cap things :)14:25
mordredsdague: this will fix that14:25
fungisdague: we're going to cap stable branches rsn14:25
mordredok. mirror is updated14:25
*** markmcclain has joined #openstack-infra14:25
*** adalbas has quit IRC14:25
*** ryanpetrello has joined #openstack-infra14:26
mordredsdague: retrying things that want pyparsing 2.0.1 should be better now14:26
sdagueAlex_Gaynor: hey, +Aing changes that don't pass check don't actually help :P14:26
*** markmcclain has quit IRC14:26
Alex_Gaynorsdague: Ugh. Apparently I shouldn't review before breakfast, I misread that as passing check :/ Sorry 'bout that.14:26
*** dolphm has quit IRC14:26
sdagueyeh, no worries14:27
*** markmcclain has joined #openstack-infra14:27
*** julim has quit IRC14:27
openstackgerritSean Dague proposed a change to openstack/requirements: change pyparsing to >= 2.0.1  https://review.openstack.org/4801414:28
*** markmc has quit IRC14:28
*** changbl has joined #openstack-infra14:28
*** yaguang has quit IRC14:28
openstackgerritSean Dague proposed a change to openstack/requirements: cap cliff to < 1.4.5  https://review.openstack.org/4801914:29
sdaguethis next change is gratuitous to kick it from the gate14:29
sdagueor, last change I should say14:29
*** markmcclain has quit IRC14:29
sdagueok, so https://review.openstack.org/#/c/48014/ is where we are betting our money on14:30
*** markmcclain has joined #openstack-infra14:31
openstackgerritMonty Taylor proposed a change to openstack-infra/devstack-gate: Use select-mirror in devstack-gate  https://review.openstack.org/4806414:32
mordredfungi: ^^14:32
dims2013-09-24 14:31:46.065 | 2013-09-24 14:31:46 Downloading/unpacking pyparsing>=2.0.1 (from cliff>=1.4.3->python-neutronclient==
dims2013-09-24 14:31:46.065 | 2013-09-24 14:31:46   Storing download in cache at /var/cache/pip/http%3A%2F%2Fpypi.openstack.org%2Fopenstack%2Fpyparsing%2Fpyparsing-2.0.1.tar.gz14:32
mordredwoot. the force add worked :)14:32
*** dizquierdo has joined #openstack-infra14:35
*** datsun180b has joined #openstack-infra14:35
*** DennyZha` has joined #openstack-infra14:36
*** DennyZhang has quit IRC14:36
dimslooks like other runs in 48014,3 are past the pyparsing/cliff problem, watching https://jenkins01.openstack.org/job/gate-requirements-install/288/console14:38
*** thomasbiege has quit IRC14:38
*** vipul has joined #openstack-infra14:39
*** Ghe_Holidays has joined #openstack-infra14:40
fungimordred: hah, i was just in the process of figuring out how to write that14:40
sdaguemordred: so we should create a bug to hang the rechecks off of14:40
portantefolks, there are a lot of pent up jobs that at least the swift team would like to recheck, what is the best way to approach this?14:40
sdagueportante: they aren't going to pass14:40
sdagueplease wait until we get the gate stable again14:41
portantecertainly, just trying to think ahead to when they will be stable and how we should consider this14:41
portantesdague: thanks, and we are not taking any action until you give the go-ahead14:41
sdagueyep, actually, I have the beginnings of a script for that, was just trying to figure out where it should live14:42
*** dkliban has quit IRC14:42
*** salv-orlando has joined #openstack-infra14:42
dimssdague, fungi - couple of 2 min pauses so far...14:42
*** MoXxXoM has joined #openstack-infra14:43
*** dolphm has joined #openstack-infra14:44
*** adalbas has joined #openstack-infra14:46
sdaguemordred: so while we wait for that requirements change to pass, gerrtlib release?14:46
jeblairmordred, fungi: good morning14:46
mordredmorning jeblair !14:46
mordredjeblair: welcome to a lovely morning of chaos14:46
fungijeblair: morning14:47
sdaguebecause then I could find a place to put my recheck script for gerrit14:47
sdaguewhich people might appreciate today14:47
* mordred is in favor of both a release and your script14:47
sdaguebut it needs fixes in unreleased gerritlib14:47
* mordred defers to jeblair - jeblair uses gerritlib in more things14:47
jeblairmordred: while i do that, can you give me a quick summary?14:48
fungijeblair: new cliff pinned a minimum version of pyparsing greater than our specified maximum version in requirements lists14:48
mordredjeblair: the mirror hit some intractable requirements issues exacerbated by pypi upstream going batty14:48
*** davidhadas has joined #openstack-infra14:48
* dhellmann hangs head in shame14:48
mordredjeblair: so the mirror update scripts couldn't get the mirror updated to contain the new version14:49
fungiand yes, slow pypi.p.o performance14:49
sdaguealso, clients on master, and servers on stable for grizzly devstack testing, created a fun grenade paradox14:49
mordredalso - we discovered that d-g was not setting mirror == upstream for requirements changes14:49
sdaguewith the same issue14:49
sdagueburned down, fell over, then sank into a swamp14:49
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Do not build Network Admin Guide anymore  https://review.openstack.org/4807014:49
mordredAND - lack of stable/grizzly requirements pinning/coordination14:49
sdaguebut the 4th castle, that one stayed up laddy14:49
dhellmannso we're working again?14:50
fungienough of these events and we'll have the strongest castle in these parts14:50
dhellmannor at least the shallowest swamp14:50
fungidhellmann: requirements change needs to finish landing first, i believe14:50
sdaguedhellmann: so... requirements is in check14:50
sdaguebut looks promissing14:50
sdaguethen that goes to gate14:50
dhellmannok, I'll hang out a little longer14:51
mordredthings we did so far - forced pyparsing 2.0.1 into the mirror by copying it there by hand, proposed changes to backport pip changes from master devstack to stable/grizzly devstack, change to devstack-gate for using select-mirror.sh14:51
sdaguethen the spice will flow14:51
fungiour castle is built on a submerged foundation of previous crumbled castles14:51
dims48014,3 - 2 devstack and 1 grenade run still going on14:51
dhellmannI ran into an issue with tab completion when I tried to remove cmd2/pyparsing from cliff, but I'll keep working on that.14:51
dhellmannand, sdague, that will be cliff 2.0 :-)14:51
* portante but the third castle ...14:51
jeblairsdague: wow, i didn't see https://review.openstack.org/#/c/47821/14:52
jeblairsdague: i'm really curious about that, because your commit message basically reduces to "zuul does not work".14:53
sdaguejeblair: so.... what were you using for ids?14:54
*** reed has joined #openstack-infra14:54
sdagueso the gerrit -h says the id needs to come first14:54
*** dkliban has joined #openstack-infra14:55
sdagueand I was having some issues with idempotent ids, and weird errors14:55
sdaguewhich I may have worked around a different way14:55
sdagueso the patch was basically just making it be what the gerrit help expected14:56
fungiwhen i reviewed that, i mainly +2'd it based on the parameter order matching other places where we were manually constructing gerrit review calls via the api, and also the order shown in the api docs14:56
*** thomasbiege has joined #openstack-infra14:56
sdaguelet me generate the fail case once the rest of this calmed down14:56
jeblairsdague: you've tested it, with supplying --message, and other params?14:57
* mordred has to pop onto a call - but jeblair is awake now14:57
dimssdague, 48014,3 - all green14:57
*** julim has joined #openstack-infra14:57
mordredjeblair, sdague: is sdague using gerrit 2.6?14:58
sdagueno, this was against upstream14:58
sdaguesorry, agains r.o.o14:58
*** SergeyLukjanov has quit IRC14:58
sdagueok... back to the gate wedge14:58
mordredok. just  making sure14:58
sdaguerequirements reviewers, please +2 / +A - https://review.openstack.org/#/c/48014/14:59
*** SergeyLukjanov has joined #openstack-infra14:59
*** dolphm has quit IRC15:00
mordredwow. that was quick15:00
*** yportnova has quit IRC15:00
*** yportnova has joined #openstack-infra15:00
jeblairsdague: ok, i mostly wanted to point out that i disagreed with the commit message's suggestion that the current implementation does not work.  if that changes any of your thinking, i'll stand by.  if you're sure that the new version works and are still ready for me to tag, i will do so.15:01
*** markmc has joined #openstack-infra15:02
*** pcrews has joined #openstack-infra15:02
sdaguejeblair: so this - http://paste.openstack.org/show/47428/ (which is the first rev of my recheck script)15:03
sdagueworked fine with those changes yesterday on a local set of changes15:03
*** datsun180b has quit IRC15:05
*** datsun180b has joined #openstack-infra15:06
*** ruhe has joined #openstack-infra15:06
*** pcrews has left #openstack-infra15:07
* SpamapS reads the backscroll in fascination15:08
SpamapSmordred: indeed my view of "that is what broke, so run it again" is entirely simplistic. Though I don't understand why zuul has to run a periodic job. :-P15:08
jeblairsdague: so you're happy? :)15:08
*** senk has joined #openstack-infra15:08
jeblairSpamapS: who else would run it?15:08
dhellmannsdague: the gate jobs look happy, are we unstuck?15:10
jeblairsdague: tag pushed15:10
*** prad has joined #openstack-infra15:11
SpamapSjeblair: some moron who can run things in jenkins15:11
mordredSpamapS: we don't use jenkins to initiate jobs15:12
mordredSpamapS: it's too stupid15:12
jeblairSpamapS: are you volunteering? :)15:12
mordredSpamapS: we also have 3 jenkinses15:12
*** julim has quit IRC15:12
*** morganfainberg has quit IRC15:13
SpamapSI was more thinking that there's a need for a big red panic button than there is a need to let people run jenkins jobs.15:15
mordredSpamapS: I'm not sure what you mean? (not being snarky, want more thoughts)15:16
*** morganfainberg has joined #openstack-infra15:16
sdaguedhellmann: yeh, gate should be good15:18
*** wchrisj has joined #openstack-infra15:18
dhellmannsdague: cool, I'm going to head to the office then15:18
*** UtahDave has joined #openstack-infra15:18
*** ruhe has quit IRC15:19
jeblairSpamapS: feel free to work on statusbot.  i'm swamped, and updating it is not my highest priority.15:19
mordredSpamapS: ah - you mean a way for not-infra to hit the button and let people know that there may be an issue?15:20
jeblairSpamapS: statusbot updates a wiki page; and we can add banners to status.o.o and gerrit, etc, using some javascript.  that bit just needs to be written.15:21
jeblairfungi, mordred: speaking of which, should we update status?15:22
mordredjeblair: yeah. I think we've back out of the woods, no?15:22
*** senk has quit IRC15:25
fungidid we have a bug number for rechecks/reverifies?15:27
*** ruhe has joined #openstack-infra15:27
SpamapSok back15:28
SpamapSok so "the big red button" is not a mechanical thing15:28
SpamapSIt is a wiki page specifically designed to aid in dealing with any crisis.15:29
SpamapSit is in the topic of this channel, maybe others, linked to in gerrit, jenkins, etc. etc.15:29
fungisdague: did we have (or need) a bug number for rechecks/reverifies?15:30
sdagueno, not yet15:30
sdaguesorry, otp, so slow to respond15:31
*** rnirmal has joined #openstack-infra15:31
fungijust didn't know if i should say recheck/reverify no bug in the status ok notice15:31
*** jeremyb has quit IRC15:31
*** jeremyb has joined #openstack-infra15:31
sdagueyeh, we should have a bug15:31
sdagueto track it15:31
sdaguejeblair: you have opinions on where recheck tooling should good?15:32
sdagueis that in config, or we have another repo that makes sense? I'd rather not spin up another repo if I can help it15:32
*** mrodden has quit IRC15:33
*** markmcclain has quit IRC15:33
*** fbo is now known as fbo_away15:33
jeblairsdague: this is for rechecking old changes?15:34
sdaguejeblair: yes15:34
jeblairsdague: i think zuul should do that, actually15:35
sdagueok, so a set of clis in the zuul repo?15:35
jeblairsdague: i thought we had a bug about that, but i can't find it15:35
sdagueI can do that15:35
jeblairsdague: er15:35
jeblairsdague: i mean, zuul should have a trigger that can do that; eg, it shouldn't be external15:36
sdagueok, well there are times when having external tooling for this is useful15:36
jeblair(we shouldn't need to leave comment spam for something like that)15:36
sdaguelike selecting on criteria that you want to drive rechecks, because you know it will be an issue15:36
jeblairsdague: yes, that i agree with15:36
jeblairsdague: like "we need to recheck all the keystone changes because ...." or "we need to recheck all the approved changes because..."15:37
jeblairsdague: but the "auto-recheck all changes older than X days" should be built into zuul15:37
jeblairsdague: i'd put it in config/tools for starters15:38
jeblair("recheck no bug" is something specific to our configuration, so it should be close to that)15:39
*** psedlak is now known as psedlak|afk15:39
*** dcramer_ has quit IRC15:39
sdaguejeblair: yeh, well we could take logic back into zuul15:39
sdagueanyway, I wanted a prooving ground15:40
jeblairsdague: yeah, i think the nice way to do it would be to add it to the gerrit trigger in zuul as a pseudo event, so that we can add it as a trigger to the existing check pipeline15:41
*** senk has joined #openstack-infra15:42
*** datsun180b has quit IRC15:43
*** datsun180b has joined #openstack-infra15:44
jeblairprobably also copying some of the logic from the timer trigger15:44
jeblairit could also be its own trigger; and maybe if it _used_ the gerrit trigger (in the way that the gerrit reporter uses it), it wouldn't need to much code.  actually, i'm starting to like that idea.15:45
sdagueok, so I'm still not sure where we land with where I should start hacking on cli tools for this :)15:45
*** senk has quit IRC15:45
jeblairneed to _copy_ much code15:45
jeblairsdague: infra/config/tools15:45
*** thomasbiege has quit IRC15:45
jeblairsdague: because "recheck no bug" is an openstack-ism, not a zuul-ism.15:46
*** zehicle_at_dell has joined #openstack-infra15:46
fungi#status ok changes seem to be making it through the gate once more, and so it should be safe to "recheck bug 1229797" or "reverify bug 1229797" on affected changes as needed15:48
openstackstatusNOTICE: changes seem to be making it through the gate once more, and so it should be safe to "recheck bug 1229797" or "reverify bug 1229797" on affected changes as needed15:48
*** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/"15:48
uvirtbotLaunchpad bug 1229797 in openstack-ci "cliff 1.4.5 & pyparsing 2.0.1 cause sadness in the gate" [Undecided,New] https://launchpad.net/bugs/122979715:48
fungiduelling bots15:48
jeblairwham bam spam15:48
*** mrodden has joined #openstack-infra15:48
*** senk has joined #openstack-infra15:48
* fungi has maps to the stars, but no maps to the spam15:49
*** senk has quit IRC15:50
*** atiwari has joined #openstack-infra15:52
SpamapSfungi: follow the maps to the stars, then follow the scent of salted meat15:54
SpamapSEncino is just over the hill from Bel Aire.. you surely can smell the spam from there. :)15:54
*** dcramer_ has joined #openstack-infra15:55
*** boris-42 has quit IRC15:55
jeblairclarkb, mordred, lifeless: reviewing the triple-o test plan; overall i believe i understand it and agree in principle.  when i try to think about the gritty details, i see things like one part says "Jenkins spawns a gate node instance" while another part says "make new nodepool base image definition"...15:57
jeblairclarkb, mordred, lifeless: is it worth my getting into that level of detail at this point, or should i just think in broad strokes?15:58
mordredjeblair: I think broad strokes - I believe that some of those things are things where we could go either way and might want to audible on it as we start moving forward15:58
openstackgerritSean Dague proposed a change to openstack-infra/config: add first pass recheck script  https://review.openstack.org/4808715:59
sdagueportante: so you guys should be good to recheck stuff now, sorry for not pinging you once the gate cleared15:59
mordred(like, we talked about re-using d-g nodes in seattle, then I realized we _could_ have an additional nodepool config script for these nodes)15:59
*** markmc has quit IRC15:59
jeblairok.  i think i understand then.16:00
portanteno problem, thanks16:01
dkranzjeblair: Is the "triple-o test plan" somewhere I can look at it?16:02
sdagueportante: actually we hit a reset on neutron16:02
portantewhat bug number should we use?16:02
sdagueso the rechecks won't work yet16:02
sdagueanother 40 minutes probably16:02
jeblairdkranz: see email to openstack-infra list16:02
uvirtbotLaunchpad bug 1229797 in openstack-ci "cliff 1.4.5 & pyparsing 2.0.1 cause sadness in the gate" [Undecided,New]16:02
*** senk has joined #openstack-infra16:02
sdaguethat's the recheck bug for this16:02
dkranzjeblair: OK, thanks. Behind on email :)16:02
*** matty_dubs is now known as matty_dubs|lunch16:02
portantenotmyname: see above for recheck bug number to use16:02
jeblairdkranz: (it's at the level of "how do we actually run tests on something that needs bare metal" rather than "exactly what tests do we run")16:03
*** gyee has joined #openstack-infra16:03
jeblairdkranz: Subject: [OpenStack-Infra] Plan for testing nova baremetal and TripleO16:03
jeblairdkranz: i believe it will also be an infra meeting topic today16:03
dkranzjeblair: Great, thanks.16:03
sdagueok, off to lunch, but as soon as 48014,3 merges, rechecks should work again16:04
*** anteaya has joined #openstack-infra16:04
*** rpodolyaka has left #openstack-infra16:05
*** rpodolyaka has quit IRC16:05
*** thomasbiege has joined #openstack-infra16:06
*** thomasbiege has quit IRC16:07
*** cody-somerville has quit IRC16:08
*** Ryan_Lane has joined #openstack-infra16:08
*** cody-somerville has joined #openstack-infra16:09
fungimordred: as an upshot of the debian tarball repacking conversation yesterday, what do you think about a configurable/cli option to pbr which makes changelogs like this? (reduces the nova changelog size by 75%) http://paste.openstack.org/show/4743816:10
*** nijaba_ is now known as nijaba16:11
funginot sure if anyone would find it useful, but i'm planning to do something similar for one of my non-openstack projects16:12
*** boris-42 has joined #openstack-infra16:12
*** senk has quit IRC16:13
fungiclearly it loses the full commit messages from every commit, but maybe it still captures enough to make it viable for a changelog16:13
*** ryanpetrello has quit IRC16:13
*** kpavel has quit IRC16:14
*** zehicle has quit IRC16:14
jeblairfungi: pretty16:14
fungiand also, i have to agree that a >5mib changelog in nova tarballs is kinda crazy16:14
*** nijaba has quit IRC16:14
SpamapSIf I wanted to gather my thoughts on a "dealing with crisis" suggestion for infra, what ML shoudl I send that too? -dev?16:14
mordredfungi: I like it16:15
*** tvb has quit IRC16:15
fungiSpamapS: possibly either -dev or -infra depending on the kind of input you're looking for16:15
mordredfungi: and we're realy just using --gnu for no good reason at all16:15
mordredfungi: so I will accept that patch16:15
fungimordred: well, the decorations (rst style) are mostly irrelevant to my point, which is wondering whether git shortlog grouped by reverse-ordering tags matching ^[0-9\.]\+$ are a sane approach16:16
*** nijaba has joined #openstack-infra16:16
*** nijaba has quit IRC16:16
*** nijaba has joined #openstack-infra16:16
fungiit could be re-massaged into something which looks like a traditional gnu changelog format without much effort16:17
jd__fungi: that changelog looks pretty good16:18
jd__I've been afk for a couple of hours and don't have the full backlog; what's the status on gate?16:18
jeblairjd__: https://wiki.openstack.org/wiki/Infrastructure_Status16:19
fungimordred: also curious whether the shortshas and patch counts are wanted/useless, if it should not batch up the prerelease tags together into the releases covering them but instead break them out independently, et cetera16:19
jd__jeblair: thanks :)16:19
*** senk has joined #openstack-infra16:20
fungicould also be made to treat unsigned tags the same way as prerelease tags and roll them into the next signed tag16:20
mordredfungi: I think it'sa great idea - I'm not sure it's _important_ to do it16:21
mordredfungi: as I really don't give a crap at all about 5M of changelog in a tarball - it's 2014. HOWEVER, I do think it's an improvement, so if you (or someone else) does want to make that prettier/more compact/better, I'm 100% all in for it16:22
fungiyup, agreed. was mainly just interesting to consider what could be done with the git log features available16:22
jeblairfungi: i'm going to check in my current atc code to infra/config/tools/atc16:22
*** markmcclain has joined #openstack-infra16:23
mordredjeblair: ++16:23
fungijeblair: okay, reed you're ready for us to generate the list immediately right?16:23
*** dcramer_ has quit IRC16:23
jd__mordred: don't want to kill your enthusiasm, but it's still 2013 last time I checked16:23
*** ryanpetrello has joined #openstack-infra16:23
fungimordred has circled the globe so many times he's actually communicating with us from 201416:24
mordredjd__: time has no meanning for me16:24
fungitime is an illusion16:25
fungilunch time doubly so16:25
fungispeaking of which16:25
* fungi is missing, presumed fed16:26
*** therve has quit IRC16:26
reedfungi, yes16:29
openstackgerritFlavio Percoco proposed a change to openstack-infra/config: Make PY3K a voting for python-marconiclient  https://review.openstack.org/4808916:30
*** therve has joined #openstack-infra16:30
*** morganfainberg has quit IRC16:31
jeblairclarkb, mordred: is there a heat plan i should review before the meeting?  (I don't see an email about it)16:32
jeblairclarkb: good morning!16:32
openstackgerritA change was merged to openstack/requirements: change pyparsing to >= 2.0.1  https://review.openstack.org/4801416:32
*** dolphm has joined #openstack-infra16:33
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Initial checkin of ATC scripts  https://review.openstack.org/4809116:33
jeblairfungi: ^ it may also need the project list updated16:34
mordredjeblair: I think clarkb was going to work on writing up notes - the biggest todo-list task is handling caching of things needed by dib in the d-g caching process16:35
mordredjeblair: which may involve some d-g scripting, and also some additional features to dib16:35
*** svarnau has joined #openstack-infra16:35
*** dcramer_ has joined #openstack-infra16:36
*** morganfainberg has joined #openstack-infra16:37
*** senk has quit IRC16:37
fungijeblair: okay, having a look16:38
fungireed: thanks16:38
clarkbjeblair: yes I started an etherpad https://etherpad.openstack.org/testing-heat-trove-with-dib16:38
clarkbmordred: ^16:39
*** hashar has joined #openstack-infra16:40
*** hashar has quit IRC16:41
*** svarnau has joined #openstack-infra16:42
*** zehicle_at_dell has quit IRC16:44
*** dcramer_ has quit IRC16:46
*** boris-42 has quit IRC16:46
*** reed has quit IRC16:47
*** svarnau_ has joined #openstack-infra16:47
*** marun has joined #openstack-infra16:47
*** thomasbiege has joined #openstack-infra16:48
*** ndipanov is now known as ndipanov_gone16:49
*** svarnau has quit IRC16:50
clarkbchmouel: I don't think your question was answered. That number is the first 7 characters of the hex representation of a 128bit random uuid16:53
*** EmilienM has quit IRC16:53
chmouelclarkb: ah thanks16:53
clarkbchmouel: each job is assigned a random uuid by zuul/jenkins and we use that instead of job numbers since we have multiple jenkins masters now16:53
*** matty_dubs|lunch is now known as matty_dubs16:54
clarkb(so job numbers can overlap now)16:54
chmouelclarkb: is there any way to get that info somewhere i.e: api16:54
*** SergeyLukjanov has quit IRC16:54
*** EmilienM has joined #openstack-infra16:54
clarkbchmouel: you might be able to get it out of jenkins if you query specific job info16:55
*** tkammer has quit IRC16:59
chmouelclarkb: i was doing that ssh -x -p 29418 review.openstack.org 'gerrit query --format=JSON --current-patch-set change:$CHANGE_ID'16:59
chmouelis there some other arguments to add16:59
clarkbchmouel: no, you are talking to gerrit there and it has no knowledge of the zuul job uuids17:00
chmouelclarkb: oh yeah, so does jenkins has an api to query those things?17:01
clarkbchmouel: jenkins has an API, it isn't well documented (or at least I have never been able to find good docs) so I am not sure what exactly it is able to do17:01
*** dcramer_ has joined #openstack-infra17:01
clarkbchmouel: is there something specific you are trying to accomplisH/17:01
clarkbmight be easier to think in terms of goals rather than capability17:02
chmoueljust getting a link for the latest job to console.html.gz17:02
chmouelsure i can screenscrap17:02
clarkbdidn't someone write that tool and post it to -dev?17:02
chmouelAFAICS it  has different purpose17:03
clarkbmaybe? that sounds familiar17:03
chmouelit goes to things like that http://logs.openstack.org/5a/5a5023bea0fad8e0c1461fe5763d3befc5bde514/17:03
chmouelbut i want to go there http://logs.openstack.org/47/45447/9/check/gate-tempest-devstack-vm-full/1f9814e/17:03
clarkbya so it is looking at the post merge jobs not premerge17:04
*** dolphm has quit IRC17:04
jeblairchmouel: you could go to http://logs.openstack.org/47/45447/9/check/gate-tempest-devstack-vm-full/ and use the index page there to decide to fetch 1f9814e/17:04
clarkbchmouel: or get the comments from jenkins for the most recent patchset and do some minimal scraping there17:04
jeblair_slightly_ better than scraping gerrit17:04
*** dkliban has quit IRC17:04
jeblairmaybe not better.  just different.  :)17:05
chmoueljeblair: yeah i was kind of avoiding that17:05
chmouelsince I would want to get the latest17:05
chmouelbut i guess this is what i would need to do17:05
chmoueland do date comparaison etc..17:05
jeblairchmouel: it was easy when we had a stateful system with a single point of failure; it gets harder with stateless distributed systems.  uuids everywhere.  :/17:06
*** tvb has joined #openstack-infra17:06
*** tvb has joined #openstack-infra17:06
jeblairchmouel: i think jog0 recently wrote something where he took clarkb's approach and scraped the jenkins comment17:06
chmoueljeblair: make sense17:06
fungifor that matter, logs.o.o doesn't really even know which one ran most recently, just which one got uploaded most recently (though that's probably close enough)17:07
*** reed has joined #openstack-infra17:07
jeblairfungi: true.  that probably makes the gerrit/jenkins comment scraping slightly more correct.  also, the synchronization is better (no race condition for partial uploads)17:07
* fungi nods17:09
chmouelfwiw this is getting me the latest curl -s 'http://logs.openstack.org/47/45447/9/check/gate-tempest-devstack-vm-full/?C=M;O=D' | sed -n '/folder.gif/ { s,.*a href="\([^/]*\)/.*,\1,;p; q;}'17:09
fungihowever, if we end up moving logs into object storage and writing an index front-end, don't count on it being 100% backwards compatible to apache mod_autoindex17:10
pleia2clarkb: when you have a moment, looking into replying to bug 1226855 and trying to figure out a sane way to do this strace if you have any thoughts17:10
uvirtbotLaunchpad bug 1226855 in lxc "Cannot use open-iscsi inside LXC container" [Undecided,Incomplete] https://launchpad.net/bugs/122685517:10
*** dizquierdo has quit IRC17:10
*** thomasbiege has quit IRC17:10
*** zehicle has joined #openstack-infra17:11
clarkbpleia2: I think to do that you will want to start iscsid, connect strace to it, then run the iscsiadm command which should cause it to fail17:12
pleia2clarkb: right, so I think the trouble with running that the command alone is that the baremetal target doens't exist so it may fail in a different way17:13
*** ruhe has quit IRC17:13
clarkbpleia2: I think the target is failing to exist as a result of this bug17:14
clarkbpleia2: so we should be able to capture it with strace (I could be missing something important though, far too much going on)17:14
chmouelfungi: yeah it's screenscraping :) screenscaping always need to be updated17:15
portantesdague are we good to go now rechecking gate jobs?17:15
pleia2clarkb: ok, I'll give it a go17:15
portanteclarkb, mordred, is it okay for swift to start rechecking gate jobs?17:16
jeblairportante: yes; see https://wiki.openstack.org/wiki/Infrastructure_Status for the bug #17:17
chmouelthe script https://gist.github.com/chmouel/668808817:18
chmouelwould show the console.html for the current review17:18
chmouel(first arg is the job name)17:18
chmouelwithout arg list the jobs17:18
jeblairchmouel: ah cool.  well since you're doing a gerrit query anyway, i would recommend changing it to to scrape the jenkins comment (you can add --coments to the query command i think)17:20
sdagueportante: yes, should be good17:20
jeblairchmouel: then that will save a network request17:20
jeblairyeah '--comments'17:21
sdaguejeblair: so gerrit hasn't been reporting on nova changes recently, like this one - https://bugs.launchpad.net/nova/+bug/1224226 which I merged the code yesterday17:21
uvirtbotLaunchpad bug 1224226 in nova "Modules with wrong file mode bits" [Medium,Fix committed]17:21
sdagueanyone have the ability to take a look17:21
fungisdague: looking into it17:22
jeblairfungi, sdague: https://review.openstack.org/#/c/41660/ my first thought is whether that bug syntax matches17:22
jeblairfungi, sdague: it certainly is not the documented syntax17:22
fungiit definitely does not17:23
*** zehicle_at_dell has joined #openstack-infra17:23
pleia2clarkb: ok, so I'm pretty sure how this testing works is that the tripleo thing looks at the pool of mac addresses and starts booting a VM, then we use iscsi to do stuff, so we do indeed get a different error if the part where "start booting VM" isn't done and we just run the iscsiadm command http://paste.openstack.org/show/47444/17:23
clarkbI did merge ttx's change as well, though he reported it looked good last night17:23
fungiand also, if all of them are marked partially-fixes then no one of them will cause it to be marked fix-committed either17:23
pleia2(and actually #tripleo may be better for this chat, so I'll wander over there)17:23
sdaguehmmm... but it matches the gerrit linkage. I thought it was the same regex for those?17:24
clarkbpleia2: cool, let me know if I need to do anyhting on my test vm?17:24
fungisdague: no, linking to a bug and telling the commit hook which of several behaviors to follow based on properly-formed commit message headers is fairly divergent now17:24
sdaguehmmmm... that's a bummer, I doubt folks realize that17:25
fungisdague: we've sent several messages to the ml, but yeah for the most part people don't read17:25
sdaguethere is a lot of traffic on the ML :)17:25
fungiit tends to get lost in the other discussions going on, yes17:25
sdagueso is there a reason we diverged them?17:26
sdaguecan we undiverge them?17:26
fungisdague: well, based on a summit session it was decided that we needed different behavior for commits which completely fix bugs, commits which are partial fixes for bugs and commits which are merely related to a bug17:26
fungisdague: however, yes, we could consider updating the gerrit link pattern to now only work if people use the preferred header syntax and ignore the limited backward-compat patterns the hook knows about17:27
*** julim has joined #openstack-infra17:27
sdagueok, that's fine, I'm just surprised that no reporting came in.17:27
fungisdague: right, the patterns the hook recognizes for its various behaviors are maintained at http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/update_bug.py#n34617:29
*** DennyZha` has quit IRC17:30
*** svarnau has joined #openstack-infra17:30
jeblairfungi: ++ (better visual feedback there)17:30
*** svarnau_ has quit IRC17:30
fungii suspect doing that would add some incentive to people to start using the new patterns17:31
*** ruhe has joined #openstack-infra17:34
*** SergeyLukjanov has joined #openstack-infra17:36
*** kgriffs is now known as kgriffs_afk17:37
*** thomasbiege has joined #openstack-infra17:38
*** zehicle has quit IRC17:38
*** dkliban has joined #openstack-infra17:40
openstackgerritClark Boylan proposed a change to openstack-infra/config: Deploy elastic-recheck on logstash.openstack.org.  https://review.openstack.org/4749717:40
clarkbjog0: ^ if the infra cores are happy with that I will create a gerrit system user called elasticrecheck, update hiera with the key for that account, then we should be able to deploy it17:41
openstackgerritA change was merged to openstack-infra/config: Put build UUID, log_url and project in log events.  https://review.openstack.org/4659017:43
*** pblaho has joined #openstack-infra17:46
fungisdague: jeblair: i drafted up the link change while i was thinking about it, but it just dawned on me that the commentlink feature in gerrit is global to commit messages, inline code comments and review comments17:48
svarnauI am setting up an instance of openstack-infra for a different project, and have question about wiki.pp17:48
fungiwe can't fine-tune it differently for commit messages than the others17:48
svarnauAre the SSL certs set-up to link with launchpad IDs, or is that a separate issue?17:49
fungisvarnau: it uses launchpad as an openid provider, but not in any way related to ssl certs17:49
svarnauokay, thanks. So the SSL cert config can just be the server certs?17:50
fungisvarnau: yes, the ssl cert config in wiki.pp is whatever you're using to serve https for that server17:50
svarnaufungi: thanks!17:51
fungisdague: jeblair: comparing the docs for gerrit 2.6 it looks like that isn't changing in new versions either, though maybe it's a patch upstream might accept for 2.7 (differentiating scope for commentlink options)17:53
fungisdague: i think the best we can hope for at the moment is to just continue to point people at https://wiki.openstack.org/wiki/GitCommitMessages#Including_external_references (and maybe clarify it further)17:54
jeblairfungi: hrm.  well, i'd be willing to lose the non-commit-message syntax (for comments, etc), but i don't feel strongly about that.17:55
fungijeblair: mmm, well i'll go ahead and send it up for review and get opinions then17:55
openstackgerritSergey Lukjanov proposed a change to openstack-infra/config: Setup gerritbot for new Savanna repos  https://review.openstack.org/4810517:57
*** KennethWilke has joined #openstack-infra17:58
*** markmcclain has quit IRC18:00
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Limit bug links in Gerrit to recommended headers  https://review.openstack.org/4810818:03
hub_caphey where is the recheck bug XXX page that shows the rechecks and their assoc. bugs18:04
*** sarob has joined #openstack-infra18:05
fungihub_cap: http://status.openstack.org/rechecks18:05
fungihub_cap: one of the many useful tabs linked along the top of the status.o.o webpage18:05
*** acabrera has joined #openstack-infra18:06
hub_capholy moly18:06
hub_capwhy didnt i know of teh base status page18:06
*** sarob has quit IRC18:06
hub_capits the base of all the links lol18:07
fungicarefully guarded secret?18:07
hub_capthat i already use18:07
hub_capor stupid me18:07
hub_capill go w stupid me18:07
*** sarob has joined #openstack-infra18:07
*** cppcabrera has quit IRC18:07
*** jerryz has joined #openstack-infra18:10
*** jerryz has quit IRC18:11
*** sarob has quit IRC18:11
*** jerryz has joined #openstack-infra18:11
*** kgriffs_afk is now known as kgriffs18:14
anteayaUtahDave is going to be able to join us for the -infra meeting18:15
anteayaI look forward to hearing about the current Salt status18:15
fungianteaya: yay! i have some more fun for him18:15
fungiUtahDave: oh! you're here18:15
*** kgriffs is now known as kgriffs_afk18:16
fungiUtahDave: had a look last night after letting the master/minions mellow for a couple weeks... the centos minions are still working great but now the master has lost contact with all the precise minions... haven't started digging into logs yet to see if they say why or when it started18:16
UtahDaveyeah, that will be interesting to see the logs.18:18
UtahDavefungi: does restarting the salt-master help?18:18
fungiUtahDave: haven't tested that yet since i don't want to disturb its state until i collect more details18:19
*** markmcclain has joined #openstack-infra18:19
*** jhesketh_ has quit IRC18:20
UtahDavefungi: ok, ping me when you have some time to troubleshoot it.18:21
fungiUtahDave: well, i'm digging into it now, but will ping you when i have some data18:22
*** mriedem has quit IRC18:27
*** melwitt has joined #openstack-infra18:28
fungiUtahDave: so here's what i know... salt master and minion logs are empty since rotation on the 14th/15th. salt-minion processes were last restarted around the same timeframe, looks like. restarting the service on a nonresponsive minion causes it to be reachable by the master again18:29
fungiUtahDave: also, i have two ubuntu precise minions which did not go dark, even though they have similar start times in the process list18:30
fungiunlike previous issues, no unaccepted/pending minion keys, no excessive resource consumption, no mutiplicity of concurrent minion processes18:31
fungijust not responding to a salt ping and not running any commands requested by the master18:32
*** jhesketh_ has joined #openstack-infra18:32
*** mriedem has joined #openstack-infra18:33
UtahDavefungi: OK, let me go find out if we've documented this issue anywhere18:45
mordredsdague, jeblair: this morning's requirements issue ...18:46
mordredsdague, jeblair: I think we should perhaps revisit how we transactionally do some things in the mirror building18:47
mordredbecause if I'm grokking it all right- cliff released new version which had a transitive depend, and we got the new version of cliff but not the new version of pyparsing18:47
mordredwhich means there was a failure which left us half-synced18:47
mordredso while we want to have things break when upstream breaks us - that's not actually what happened here18:48
UtahDavefungi: can you pastebin the output of  salt 'minion-name' test.versions_report       from one of the previously unresponsive precise minions?18:48
mordredwe created a state in the mirror which did not exist in the world at large because of a half-applied mirroring operation18:49
mordredI'm not 100% sure I've got a good idea of what the best next algorithm should be - but some combiation of "apply all of a given dep chain or none of it" and "be able to retry individual things that didn't work the first time"18:50
*** hashar has joined #openstack-infra18:51
UtahDavefungi: also do you know if Puppet is ever trying to reload the salt-minion?  Due to some idiosyncrasies of how zeromq works Salt should always be restarted, not reloaded18:51
clarkbmordred: this has happened in the past and my understanding (after being very confused) was that this is the behavior we want18:51
clarkbmordred: because it very clearly shows us when the world has broken us18:51
mordredbut the world did not break us18:51
clarkbmordred: it did cliff required >X we required <X18:52
mordredcliff's requirement of >X should have caused >X to make it in to the mirror, no?18:52
clarkbmordred: not necessarily as pip is poor at dependency resolution, <X may have been evaluated first18:53
mordredthere _was_ a break that was valid18:53
mordredbut there was also a break in what elements of the world were present in our mirror18:53
mordredI believe that we actually  moved to agreeing with you tat the mirror should not attempt to impose additional enforcement of the reuqirment set18:53
mordrednot that we have openstack/requirements in place and gating18:53
jeblairmordred: yeah, i think the actual issue is that the mirror script was changed to support partial success (if package a breaks, don't let that stop us from uploading package b)18:55
jeblairmordred: i believe this is an unintended consequence of that18:55
jeblairmordred: and perhaps the mirror script should go back to being all or none18:56
lifelessjeblair: thanks for the reply; on capacity, we may need to only check a subset of proposals, but gate all, if we don't have enough capacity at the start. Is that feasible?18:56
mordredjeblair: hrm.18:56
mordredjeblair: I thnk I'm beginning to think that the mirror system just needs to be a mix of the two18:56
mordredjeblair: and smarter18:56
lifelessjeblair: (naturally we wouldn't *start* as gate, I mean once reliability etc have been demonstrated)18:56
mordredjeblair: so that we can get the correct all or none and appropriate re-tries to catch transient crud18:57
*** kgriffs_afk is now known as kgriffs18:57
mordredjeblair: let me noodle on it more and come back with a real suggestion rather than brainstorming directly here18:57
jeblairmordred: ok18:57
fungiUtahDave: this is one which was unresponsive until i restarted the minion service a few minutes ago... http://paste.openstack.org/show/47451/18:58
jeblairmeeting time18:59
UtahDavefungi: wow, somehow zmq 2.1.11 is being installed.  That's the problem.18:59
UtahDavefungi: I'll catch you after the meeting19:00
fungiUtahDave: awesome. and i'll find where it's coming in from19:00
UtahDavethe saltstack ppa should provide the latest zmq19:00
*** kgriffs is now known as kgriffs_afk19:02
fungiit looks like i still have some puppetry issues around getting unattended upgrades on ubuntu to not ignore third-party repositories19:04
fungione of the two which did not go on silent running was running a newer zeromq (ZMQ: 3.2.2, PyZMQ: 13.0.0)19:05
*** kgriffs_afk is now known as kgriffs19:05
fungiso this definitely supports that assertion19:05
openstackgerritClark Boylan proposed a change to openstack-infra/config: Deploy elastic-recheck on logstash.openstack.org.  https://review.openstack.org/4749719:06
*** dizquierdo has joined #openstack-infra19:09
*** pycabrera has joined #openstack-infra19:12
*** pycabrera is now known as cppcabrera19:13
*** cppcabrera has quit IRC19:14
thomasmAre the gate-tempest-devstack-* tests the same for all OS projects? Are they using the same hosts and orchestration? An update to global requirements isn't being applied completely for my grizzly patch: https://review.openstack.org/#/c/48039/19:14
clarkbthomasm: we are in our weekly meeting currently, but afterwards I am sure one of us will take a look19:15
thomasmclarkb, Oh, okay. Thanks. =]19:15
*** acabrera has quit IRC19:15
*** ruhe has quit IRC19:18
*** Ryan_Lane has quit IRC19:29
fungithomasm: bug 1229797 is still not addressed in stable/grizzly i think19:30
uvirtbotLaunchpad bug 1229797 in openstack-ci "cliff 1.4.5 & pyparsing 2.0.1 cause sadness in the gate" [Undecided,New] https://launchpad.net/bugs/122979719:30
*** ruhe has joined #openstack-infra19:31
*** mrmartin has joined #openstack-infra19:32
*** pblaho has quit IRC19:33
thomasmfungi, Ahh, right. The upstream comment. Thanks, fungi! =]19:34
fungithomasm: you're welcome19:35
*** sandywalsh has quit IRC19:35
hub_capfwiw jeblair mordred the weekly meeting is on my calendar now so ill be paying bettter attn, also in another rackspace meeting at this time lol19:37
jeblairhub_cap: from what i know of you, you should be able to handle 4 meetings simultaneously, easy.19:37
hub_capheh its 1 cup of coffee / meeting jeblair19:40
hub_capand ive only had 3 so far.... so really, the calendar just defines the # cups of coffee19:40
*** vipul is now known as vipul-away19:41
*** yolanda has quit IRC19:42
*** dcramer_ has quit IRC19:44
*** markmc has joined #openstack-infra19:46
*** boris-42 has joined #openstack-infra19:46
*** sandywalsh has joined #openstack-infra19:47
openstackgerritSean Dague proposed a change to openstack-infra/config: add first pass recheck script  https://review.openstack.org/4808719:47
openstackgerritA change was merged to openstack/requirements: Use oslo.config 1.2.0  https://review.openstack.org/4802219:59
*** sarob has joined #openstack-infra19:59
mtreinishjeblair: I didn't even realize that was on the agenda :)20:01
*** cppcabrera has joined #openstack-infra20:02
jeblairmtreinish: me neither, though it was a good idea because there's a bit of a design discussion going on in https://review.openstack.org/#/c/47491/20:03
*** gyee has quit IRC20:03
*** cppcabrera has left #openstack-infra20:03
*** tvb has quit IRC20:03
mtreinishjeblair: I was reading your comment earlier. So you're proposing we tag bugs (or leave comments with the queries) and have the bot pull that data from launchpad?20:04
*** dprince has quit IRC20:04
*** rockyg has quit IRC20:06
*** nicedice has joined #openstack-infra20:08
jeblairmtreinish: yes, that was the original idea that we brainstormed at the summit20:08
fungihave the bot retrieve comments from the bug, parse them to find the most recent containing an elastic recheck query... but yeah where do you tell elastic recheck which bugs to look at? integrate with initial gerrit recheck comments to build up a list?20:09
jeblairfungi: tag bugs with something and e-r searches for the tag20:09
openstackgerritAndreas Jaeger proposed a change to openstack-infra/config: Create new gates for openstack-manuals  https://review.openstack.org/4812820:09
*** nicedice has quit IRC20:10
fungiahh, yep. search for all bugs with the tag. got it20:10
jeblair'logstash-query' or something20:10
*** armax has joined #openstack-infra20:10
anteayayay UtahDave will be in Hong Kong20:10
*** nicedice has joined #openstack-infra20:10
fungianteaya: UtahDave: great! much collaboration will be had by all20:11
clarkbjeblair: fungi: I like the idea, I think we should start with the existing simple config file and add that feature later (I don't want to hold up deploying this thing)20:11
armaxdoes anyone know how to deal with a gerrit 'Application Error: The page you requested was not found, or you do not have permission to view this page.', after having deleted a draft change?20:11
armaxon review.openstack.org?20:11
fungii didn't know you could delete a draft20:11
UtahDavefungi: Yep!  I'm really looking forward to it.20:11
jeblairclarkb: yes, as i'm not writing the code, i'm willing to go with that since it does seem like a viable implementation.  i tried to make it clear that was a 'helpful contribution' as opposed to a blocking core review.  :)20:11
fungiarmax: i really, really don't ever use drafts. my only exposure to them is helping others troubleshoot where they go wrong20:12
*** vipul-away is now known as vipul20:12
armaxfungi: ok20:13
fungiarmax: but not familiar with what happens when you delete them, no20:13
mtreinishjeblair, fungi: that sounds like it would definitely improve the workflow and I kind of like it. But, the one key advantage I see with the file now though is the verification step and review. With bug comments theres nothing stopping people from putting bad queries that catch everything20:13
mtreinishbut I agree with clarkb we should start with what we've got now and work on that as a future item20:13
armaxusually I publish them after they are ready for prime time20:13
*** kokoisatard has joined #openstack-infra20:13
armaxthis time I hitched for the wrong button, argh20:13
*** kokoisatard has quit IRC20:13
clarkbyou can delete drafts?20:14
mtreinish(although now that I think about it we don't have it configured it to gate on running test_queries.py)20:14
jeblairmtreinish: agreed.  though i feel that the negative impact of that would be small and outweigh the benefits (intentional vandalism is very unlikely, and accidental mistakes can be easily corrected by anyone).20:14
fungimtreinish: though it's easy enough to untag or update a bug with a better query and assume people won't engage in unnecessary bugwars i think20:14
armaxclarkb: apparently yes there are two buttons20:14
jeblairmtreinish: /outweigh/be outweighed by/20:14
armaxone says delete draft change20:14
armaxthe other delete draft patch set20:14
armaxor something like that20:15
fungiclarkb: to my surprise as well20:15
*** beagles has quit IRC20:15
clarkbarmax: the simplest thing may be to repush to gerrit20:16
mtreinishjeblair, fungi: I agree, that's why I think working on using launchpad for the queries is a good future step. (I was just pointing out an advantage of the json file)20:16
armaxclarkb: I'll give it a whirl, thanks20:16
jeblairmtreinish: anyway, i'm all in favor of getting it going (after the acl typo is fixed)20:16
mtreinishjeblair: ok, let me take care of that right now20:17
*** zehicle has joined #openstack-infra20:18
*** beagles has joined #openstack-infra20:18
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Add elastic-recheck to openstack-infra  https://review.openstack.org/4749120:19
*** dcramer_ has joined #openstack-infra20:21
*** rnirmal_ has joined #openstack-infra20:22
LinuxJediwoot!  according to Stackanalysis me and half my team don't work for HP :)20:22
fungiLinuxJedi: your wish came true? ;)20:22
LinuxJedifungi: only if someone else can pay me the same ;)20:23
*** rnirmal has quit IRC20:23
*** rnirmal_ is now known as rnirmal20:23
*** bnemec has quit IRC20:23
fungiLinuxJedi: sorry, i can't spare my loose change--need to be able to do laundry20:24
*** thomasbiege has quit IRC20:25
*** bnemec has joined #openstack-infra20:26
* anteaya heads out for a walk20:28
*** Ryan_Lane has joined #openstack-infra20:29
openstackgerritSean Dague proposed a change to openstack-infra/config: add min-sev parameter  https://review.openstack.org/4813120:30
sdagueclarkb: give that a whirl & a look when you get a chance20:30
*** krtaylor has quit IRC20:31
*** tvb has joined #openstack-infra20:33
*** rockyg has joined #openstack-infra20:34
clarkbsdague: will do20:35
sdagueit's lightly tested with a cinder log I have from a while ago for both html and/or text20:36
clarkbjeblair: fyi the zuul results queue is really long, watching it now to see if it falls at all20:36
sdaguebasically add min-sev=INFO and you only get INFO and up20:36
clarkbsdague: perfect20:36
sdagueand it seems to go pretty fast20:36
jeblairclarkb: yeah, i think it may have been processing a reset20:38
*** mikal is now known as annegentle_proxy20:38
*** annegentle_proxy is now known as mikal20:38
* fungi has been talked into popping out for some evening exercise while the weather's nice... back for more hacking in a while20:41
*** tvb has quit IRC20:42
*** rnirmal has quit IRC20:43
*** atiwari has quit IRC20:43
*** julim has quit IRC20:44
*** rcleere has quit IRC20:45
*** rcleere has joined #openstack-infra20:45
lifelessis there a way to close all fix-committed bugs when a release is cut from a project ?20:47
lifelessor is it manual?20:47
*** morganfainberg has quit IRC20:47
jeblairttx: see question from lifeless ^20:48
ttxlifeless: it's a LP script20:48
SpamapSttx: but does it run with the git tag magic that makes tarballs and fresh cookies?20:49
lifelessor can it ?20:51
*** ArxCruz has quit IRC20:54
*** rwsu has joined #openstack-infra20:54
ttxlifeless: that would be manual at this point (two separate steps in the release process)20:55
ttxhttps://wiki.openstack.org/wiki/Release_Team/How_To_Release might be your friend (or not)20:55
*** HenryG has joined #openstack-infra20:56
lifelessSpamapS: ^20:56
SpamapSok seems like the git magic releaser could call the script.20:57
SpamapSlifeless: ty20:57
jeblairSpamapS: it could be made a jenkins job20:57
*** thomasbiege has joined #openstack-infra20:57
mikalDear smart people. I need adult supervision.20:58
mikalI'm interested in either a) forming a boy band with you, or b) working out the right wat to fix https://review.openstack.org/#/c/47970/220:59
lifelessmikal: why isn't that a test?21:00
mikalAssuming we go with option number B, specifically I'm trying to decide the best way to tweak the module path to import the right version of nova.exceptions in the test script21:00
mikallifeless: a unit test which walks all the exceptions?21:00
mikallifeless: cause I didn't think of that?21:00
lifelessmikal: :)21:01
jeblairlifeless's suggestion++21:01
mikalIts not a terrible idea though21:01
*** rfolco has quit IRC21:01
lifelessmikal: I have written this exact test back in bzr :)21:01
* mikal will give it a go once he's finished watching these boy band videos21:01
lifelessmikal: it works well21:01
mikalPriorities don't you know21:01
jeblairmikal: you better delegate in that case21:01
* mordred wants to see the lifeless mikal boy band21:02
lifelessmordred: you're going to keep on wanting for a loooong time21:02
mikalmordred: it would have to be down under themed.21:03
adam_gdoes the jenkins-job-builder currently support defining variables that are available globally to templates?21:08
*** kgriffs is now known as kgriffs_afk21:08
*** hashar has quit IRC21:08
*** ericlin has joined #openstack-infra21:11
*** kgriffs_afk is now known as kgriffs21:12
*** krtaylor has joined #openstack-infra21:13
*** ericlin has quit IRC21:13
*** mrmartin has quit IRC21:14
*** gyee has joined #openstack-infra21:15
*** dkranz has quit IRC21:20
*** jcoufal has quit IRC21:21
*** thomasm has quit IRC21:22
*** morganfainberg has joined #openstack-infra21:23
clarkbjeblair: I am updating the overview zuul simulation to show nnfi21:24
clarkbjeblair: I assume your javascript hasn't gotten that far yet21:24
*** Ryan_Lane has quit IRC21:28
*** markmc has quit IRC21:28
*** thomasbiege has quit IRC21:32
*** dizquierdo has left #openstack-infra21:32
*** ryanpetrello has quit IRC21:34
*** tvb has joined #openstack-infra21:39
*** tvb has joined #openstack-infra21:39
*** zehicle_at_dell has quit IRC21:40
*** zehicle_at_dell has joined #openstack-infra21:41
*** tvb has quit IRC21:43
*** dkliban has quit IRC21:44
*** weshay has quit IRC21:46
*** wchrisj has quit IRC21:51
*** ruhe has quit IRC21:52
*** julim has joined #openstack-infra21:55
*** vipul is now known as vipul-away21:57
mordredclarkb: I believe he's almost done22:02
*** matty_dubs is now known as matty_dubs|gone22:03
*** amotoki has joined #openstack-infra22:03
clarkbmordred: but he is doing js for status and not the talk simulation?22:04
*** datsun180b has quit IRC22:06
openstackgerritClark Boylan proposed a change to openstack-infra/publications: Update talk with new information and images.  https://review.openstack.org/4815122:09
openstackgerritClark Boylan proposed a change to openstack-infra/publications: Add logstash slide.  https://review.openstack.org/4815222:09
openstackgerritClark Boylan proposed a change to openstack-infra/publications: Make zuul simulation reflect NNFI.  https://review.openstack.org/4815322:09
clarkbmordred: fungi jeblair ^22:09
*** tstevenson has quit IRC22:09
clarkbmordred: http://docs.openstack.org/infra/publications/overview/#%2817%29 do you know what we are trying to express there?22:10
*** jhesketh_ has quit IRC22:13
*** jhesketh has quit IRC22:13
*** vipul-away is now known as vipul22:13
mordredclarkb: yes. we're showing the overview that you can get22:14
mordredwith the various statuses things can be in and your view of the whole22:14
*** SergeyLukjanov has quit IRC22:17
*** mriedem has quit IRC22:19
*** markmcclain has quit IRC22:19
*** zehicle_at_dell has quit IRC22:25
*** jhesketh has joined #openstack-infra22:25
*** jhesketh_ has joined #openstack-infra22:26
clarkbanyone else want to review https://review.openstack.org/#/c/47491/ (adds elastic-recheck to gerrit) before I approve?22:29
clarkbjog0: mtreinish is the source repo ready for the upstream to change?22:30
*** changbl has quit IRC22:31
*** boris-42 has quit IRC22:32
*** Ryan_Lane has joined #openstack-infra22:34
mordredclarkb: nope. go for it - if jog0 and mtreinish are ready22:34
*** prad has quit IRC22:34
*** HenryG has quit IRC22:35
jog0clarkb mordred: Its ready enough go for it22:38
*** rcleere has quit IRC22:40
openstackgerritA change was merged to openstack-infra/config: Add elastic-recheck to openstack-infra  https://review.openstack.org/4749122:42
*** armax has left #openstack-infra22:46
*** KennethWilke has quit IRC22:46
openstackgerritClark Boylan proposed a change to openstack-infra/config: Deploy elastic-recheck on logstash.openstack.org.  https://review.openstack.org/4749722:47
clarkband ^ should be ready for review22:47
*** openstackgerrit has quit IRC22:52
*** openstackgerrit has joined #openstack-infra22:52
*** rwsu has quit IRC22:53
*** dims has quit IRC22:54
*** rwsu has joined #openstack-infra22:56
clarkbok I am going to AFK to prepare for the game tonight. Need to get dinner so that I don't have to eat safeco food22:58
*** julim has quit IRC23:06
fungiclarkb: awesome updates. another good todo on the overview preso is to refactor some of the devstack-gate slides so that we plug nodepool23:06
*** dims has joined #openstack-infra23:07
clarkbgood idea23:07
*** krtaylor has quit IRC23:09
*** tvb has joined #openstack-infra23:10
*** tvb has joined #openstack-infra23:10
*** thomasm has joined #openstack-infra23:12
*** tvb has quit IRC23:14
*** amotoki has quit IRC23:18
*** flaper87 is now known as flaper87|afk23:26
*** amotoki has joined #openstack-infra23:28
reedguys, I have corrected the blog post http://www.openstack.org/blog/2013/09/openstack-governance-electing-technical-leaders-for-next-development-cycle/ but I don't think the copy on planet will update itself23:29
reedcan you confirm or manually rebuild the planet?23:29
*** Ryan_Lane has quit IRC23:29
*** Ryan_Lane has joined #openstack-infra23:34
*** vipul is now known as vipul-away23:39
*** tvb has joined #openstack-infra23:41
*** amotoki has quit IRC23:41
mordredreed: you are righ t- it will not - we need a todo list item to make that work better23:43
mordredlemme go poke it with a big stick23:43
reedthanks mordred23:43
*** UtahDave has quit IRC23:44
*** pabelanger has quit IRC23:45
*** pabelanger has joined #openstack-infra23:45
*** tvb has quit IRC23:45
mordredreed: I have removed that entry from the cache23:45
mordredreed: so next time it runs, it should re-download that entry23:45
*** vipul-away is now known as vipul23:46
*** notmyname has quit IRC23:47
*** uvirtbot has quit IRC23:47
*** crank has quit IRC23:47
*** hemnafk has quit IRC23:47
*** danger_fo_away has quit IRC23:47
*** Mithrandir has quit IRC23:47
*** Vivek has quit IRC23:47
*** mikal has quit IRC23:47
*** rahmu has quit IRC23:47
*** westmaas has quit IRC23:47
*** antonym has quit IRC23:47
*** belliott has quit IRC23:47
*** dansmith has quit IRC23:47
*** akscram has quit IRC23:47
*** Ryan_Lane has quit IRC23:47
*** marun has quit IRC23:47
*** cody-somerville has quit IRC23:47
*** ndipanov_gone has quit IRC23:47
*** jesusaurus has quit IRC23:47
*** schwicht has quit IRC23:47
*** salv-orlando has quit IRC23:47
*** mestery has quit IRC23:47
*** ianw has quit IRC23:47
*** Daviey has quit IRC23:47
*** dragondm has quit IRC23:47
*** SpamapS has quit IRC23:47
*** melwitt has quit IRC23:47
*** paul-- has quit IRC23:47
*** anteaya has quit IRC23:47
*** zul has quit IRC23:47
*** enikanorov-w has quit IRC23:47
*** lifeless has quit IRC23:47
*** bodepd has quit IRC23:47
*** Shrews has quit IRC23:47
*** MoXxXoM has quit IRC23:47
*** rainya has quit IRC23:47
*** EntropyWorks has quit IRC23:47
*** pabelanger has quit IRC23:47
*** gyee has quit IRC23:47
*** rockyg has quit IRC23:47
*** yongli has quit IRC23:47
*** dstufft has quit IRC23:47
*** klrmn has quit IRC23:47
*** kiall has quit IRC23:47
*** mtreinish has quit IRC23:47
*** soren has quit IRC23:47
*** LinuxJedi has quit IRC23:47
*** koolhead17 has quit IRC23:47
*** pleia2 has quit IRC23:47
*** openstackgerrit has quit IRC23:47
*** zehicle has quit IRC23:47
*** mrodden has quit IRC23:47
*** vipul has quit IRC23:47
*** mordred has quit IRC23:47
*** CaptTofu has quit IRC23:47
*** moted has quit IRC23:47
*** briancline has quit IRC23:47
*** cyeoh has quit IRC23:47
*** sdague has quit IRC23:47
*** jog0 has quit IRC23:47
*** fungi has quit IRC23:47
*** saschpe_ has quit IRC23:47
*** Mithrandir has joined #openstack-infra23:47
*** westmaas has joined #openstack-infra23:47
*** mikal has joined #openstack-infra23:47
*** dansmith has joined #openstack-infra23:47
*** danger_fo_away has joined #openstack-infra23:47
*** crank has joined #openstack-infra23:47
*** rahmu has joined #openstack-infra23:47
*** belliott has joined #openstack-infra23:47
*** Vivek has joined #openstack-infra23:47
*** antonym has joined #openstack-infra23:47
*** sarob has quit IRC23:48
*** akscram has joined #openstack-infra23:48
*** Vivek is now known as Guest9862223:48
*** saschpe_ has joined #openstack-infra23:54
*** fungi has joined #openstack-infra23:54
*** jog0 has joined #openstack-infra23:54
*** sdague has joined #openstack-infra23:54
*** cyeoh has joined #openstack-infra23:54
*** briancline has joined #openstack-infra23:54
*** moted has joined #openstack-infra23:54
*** CaptTofu has joined #openstack-infra23:54
*** mordred has joined #openstack-infra23:54
*** vipul has joined #openstack-infra23:54
*** mrodden has joined #openstack-infra23:54
*** zehicle has joined #openstack-infra23:54
*** openstackgerrit has joined #openstack-infra23:54
*** notmyname has joined #openstack-infra23:54
*** Ryan_Lane has joined #openstack-infra23:54
*** Ryan_Lane1 has joined #openstack-infra23:54
*** hemnafk has joined #openstack-infra23:54
*** pabelanger has joined #openstack-infra23:54
*** gyee has joined #openstack-infra23:54
*** rockyg has joined #openstack-infra23:54
*** melwitt has joined #openstack-infra23:54
*** marun has joined #openstack-infra23:54
*** cody-somerville has joined #openstack-infra23:54
*** anteaya has joined #openstack-infra23:54
*** MoXxXoM has joined #openstack-infra23:54
*** salv-orlando has joined #openstack-infra23:54
*** zul has joined #openstack-infra23:54
*** ndipanov_gone has joined #openstack-infra23:54
*** jesusaurus has joined #openstack-infra23:54
*** schwicht has joined #openstack-infra23:54
*** enikanorov-w has joined #openstack-infra23:54
*** lifeless has joined #openstack-infra23:54
*** mestery has joined #openstack-infra23:54
*** ianw has joined #openstack-infra23:54
*** Daviey has joined #openstack-infra23:54
*** yongli has joined #openstack-infra23:54
*** bodepd has joined #openstack-infra23:54
*** dstufft has joined #openstack-infra23:54
*** Shrews has joined #openstack-infra23:54
*** klrmn has joined #openstack-infra23:54
*** dragondm has joined #openstack-infra23:54
*** rainya has joined #openstack-infra23:54
*** SpamapS has joined #openstack-infra23:54
*** paul-- has joined #openstack-infra23:54
*** kiall has joined #openstack-infra23:54
*** mtreinish has joined #openstack-infra23:54
*** EntropyWorks has joined #openstack-infra23:54
*** pleia2 has joined #openstack-infra23:54
*** koolhead17 has joined #openstack-infra23:54
*** LinuxJedi has joined #openstack-infra23:54
*** soren has joined #openstack-infra23:54
*** Ryan_Lane1 has quit IRC23:54
*** Ryan_Lane has quit IRC23:54
*** Ryan_Lane has joined #openstack-infra23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!