Wednesday, 2014-03-26

*** saschpe has quit IRC00:00
*** dcramer_ has joined #openstack-infra00:00
*** mgagne has quit IRC00:01
*** Trozz has joined #openstack-infra00:03
*** nkinder has quit IRC00:03
fungii'm continuing to look at it crossways and hope it'll shape up, but if it goes on much longer i'll push up a revert00:04
boris-42mordred hi could you help Rally team to add first functional testing gate?)
mordredboris-42: not right now - but I'll look at it a little later00:05
boris-42mordred thanks=)00:05
*** nkinder has joined #openstack-infra00:06
*** saschpe has joined #openstack-infra00:07
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Add a very basic functional test
*** atiwari has quit IRC00:09
jeblairi say very basic, but it tests quite a lot actually, mostly because it's hard to test a little at this point00:13
*** krtaylor has joined #openstack-infra00:20
*** dcramer_ has quit IRC00:22
*** sabari has quit IRC00:23
*** nkinder has quit IRC00:23
*** mestery has quit IRC00:23
*** matsuhashi has joined #openstack-infra00:27
*** thuc has joined #openstack-infra00:29
*** dcramer_ has joined #openstack-infra00:29
openstackgerritAnita Kuno proposed a change to openstack-infra/meetbot: Prevents meetbot from starting meetings in non-meeting channels
clarkbis ^ a problem?00:32
clarkbalso did we revert the switchover to region b?00:32
anteayaclarkb: are you asking me about 82898?00:33
anteayain -neutron this morning00:33
anteayasomeone started a meeting00:33
*** colinmcnamara has quit IRC00:33
*** morganfainberg is now known as morganfainberg_Z00:33
anteayaso markmcclain asked me if there was a way to prevent it00:33
*** thuc has quit IRC00:33
anteayaso I went to #meetbot on oftc and this was suggested to me00:34
*** yamahata has quit IRC00:34
*** dcramer_ has quit IRC00:34
pleia2so I thought about this some today, I wonder if there are teams who have meetings in their channels and want to retain the feature00:35
anteayaI'm open to changing the conditional00:35
anteayaI can do `if "neutron" in channel: return`00:35
* pleia2 nods00:36
fungithere are definitely some who do... one stackforge project (i forget which) mentioned using it in their channel for daily scrum meetings00:36
anteayaif we want to only target neutron00:36
* anteaya changes the conditional00:36
jeblairpleia2: well, we created the meetings channels to force people to use them, and schedule meetings centrally00:36
*** SumitNaiksatam has quit IRC00:36
pleia2jeblair: yeah, but I think some do little less formal meetings and things00:36
jeblairwe could revisit that of course, but that's the current intent...00:36
clarkbanteaya: roger I missed that00:36
jeblairpleia2: if there are openstack projects that do that, then we should have a discussion as a project about how to handle that..00:37
*** mriedem has joined #openstack-infra00:37
jeblairpleia2: if there are stackforge projects that do that, i'm not particularly concerned00:37
* pleia2 nods00:37
fungiclarkb: we haven't reverted yet, but i'm starting to think it might be warranted (though not sure how we work out the cause of so many nova boot requests failing to complete)00:37
fungiother than asking hpcloud to look at their nova error logs, i guess00:38
openstackgerritAnita Kuno proposed a change to openstack-infra/meetbot: Prevents meetbot from starting meetings in openstack-neutron
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Revert "Stop using hpcloud az1-az3 in favor of region-b"
anteayaso there is an option that just prevents meetings in neutron00:39
anteayavote for your favourite00:39
jeblairanteaya: that was the same change00:39
anteayaI changed the conditional:  if "neutron" in channel:00:40
jeblairanteaya: yes, but you updated the same change in gerrit so it is not possible to "vote for your favorite" unless your favorite is the most recent patchset.00:40
anteayaoh I didnt' know that00:41
anteayafungi had previously merged a change from a stack of changes of mine00:41
anteayaso I erronously thought you could vote for any one of the three patchsets00:41
pabelangerfungi, I am noticing you are also getting the same network slowness from hpcloud-region-b:
anteayaI didn't realize you were limited to just the latest one00:42
pabelangerI can't seem to get more then 150 kB/s for apt updates00:42
*** gokrokve has joined #openstack-infra00:42
fungipabelanger: glad to know it's not just us... well, "glad" isn't quite right, but you know what i mean00:42
pabelangerfungi, Ya, trying to figure out if I was doing something wrong.  Trying to see if there is some internal mirror someplace in region-b00:43
anteayaI'll let that steep until tomorrow, if there is consenseus with jeblair view, I can offer patchset 2 back up00:43
anteayajeblair's view00:43
*** rediskin has left #openstack-infra00:43
fungipabelanger: we're currently getting a lot of instance boot requests failing to complete00:43
fungipabelanger: in addition to that less-problematic issue (from our standpoint)00:44
pabelangerfungi, just on region-b?00:45
fungipabelanger: primarily anyway, yes00:45
*** UtahDave has quit IRC00:45
fungipabelanger: we just merged a change a few hours ago to switch from the old hpcloud availability zones to region b, and have had lots of nodes there timing out after the boot call00:46
*** dcramer_ has joined #openstack-infra00:46
fungifar more than we anticipated anyway00:46
pabelangerfungi, Ya, see that today00:46
openstackgerritMatt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for heat bug 1297560
uvirtbotLaunchpad bug 1297560 in heat "*tempest-dsvm-neutron-heat-slow fails with WaitConditionTimeout" [Undecided,New]
openstackgerritJames E. Blair proposed a change to openstack-infra/nodepool: Include provider names in timeout messages
jeblairfungi: ^ (and it's not on the crazy long multinode branch)00:55
funginow you're just giving me excuses to not get around to the rest of the patches in your crazy long branch yet00:58
funginot that it's that crazy long, i've just been slack on review volume recently00:59
openstackgerritJoshua Harlow proposed a change to openstack-dev/hacking: Add a tool that can fix the import order
*** imcsk8_lap has quit IRC01:06
*** reed has quit IRC01:09
openstackgerritA change was merged to openstack-infra/config: Revert "Stop using hpcloud az1-az3 in favor of region-b"
*** changbl has joined #openstack-infra01:12
*** matsuhashi has quit IRC01:12
openstackgerritJoshua Hesketh proposed a change to openstack-infra/zuul: WIP, getting train tracks, DO NOT MERGE
*** matsuhashi has joined #openstack-infra01:14
*** sdake_ has quit IRC01:15
*** sdake has quit IRC01:16
*** krotscheck has quit IRC01:21
*** krotscheck has joined #openstack-infra01:21
*** rfolco has joined #openstack-infra01:23
*** weshay has quit IRC01:25
*** nosnos has joined #openstack-infra01:27
*** khyati has quit IRC01:29
*** krtaylor has quit IRC01:34
*** sdake has joined #openstack-infra01:35
*** sdake_ has joined #openstack-infra01:35
*** imcsk8_lap has joined #openstack-infra01:35
*** nkinder has joined #openstack-infra01:43
*** alexpilotti has quit IRC01:47
*** rfolco has quit IRC01:48
*** wenlock has joined #openstack-infra01:49
*** wenlock has quit IRC01:53
*** yjiang has joined #openstack-infra01:54
*** yamahata has joined #openstack-infra01:58
*** VijayTripathi has quit IRC01:59
*** matsuhashi has quit IRC02:00
*** wenlock has joined #openstack-infra02:03
*** mgagne has joined #openstack-infra02:03
*** matsuhashi has joined #openstack-infra02:08
*** thuc has joined #openstack-infra02:11
*** saschpe has quit IRC02:24
*** VijayTripathi has joined #openstack-infra02:25
*** harlowja is now known as harlowja_away02:27
*** saschpe has joined #openstack-infra02:27
*** thuc has quit IRC02:27
clarkbkrotscheck: jeblair: is there an existing plan for doing search in storyboard?02:28
*** thuc has joined #openstack-infra02:28
clarkbcurious to know if you plan on doing something like sphinx or elasticsearch or $otherthing02:28
pabelangerfungi, here is my issue I think:
*** vkozhukalov has joined #openstack-infra02:31
pabelangertesting the logic now02:31
*** Alexandra is now known as alex-lunch02:31
*** jeckersb is now known as jeckersb_gone02:32
*** thuc has quit IRC02:32
*** zhiyan_ is now known as zhiyan02:32
*** dcramer_ has quit IRC02:36
clarkbpabelanger: the syncs were causing problems?02:36
fungipabelanger: interesting. you think the sync calls at the end of one or more of those scripts is causing your problem, or did we miss adding one somewhere?02:36
pabelangerclarkb, fungi no, I have not been calling sync.02:37
pabelangerI am just adding it now, and testing02:37
pabelangerjust found that commit02:37
clarkbmy eyes are mostly back to normal now02:37
pabelangerafter looking at source02:37
fungiwe did indeed add it for... reasons02:37
pabelangerthat's my guess, snapshot is happening too fast, corrupting image02:38
*** khyati has joined #openstack-infra02:38
*** krtaylor has joined #openstack-infra02:38
*** BadCub01 has quit IRC02:39
*** nati_ueno has quit IRC02:42
*** thomasem has joined #openstack-infra02:53
*** imcsk8_lap has quit IRC02:55
*** gyee has quit IRC02:56
mordredclarkb: I was thinking sphinxsearch02:58
mordredclarkb: I should ping some of the sphinxsearch guys and see if they want to do that for us...02:58
pabelangerfungi, clarkb: Yup, that was it.  I needed to call sync03:01
*** todd_dsm has joined #openstack-infra03:03
lifelessthere seem to be a lot of queued nodes atm03:04
lifelessor is it just me ?03:04
lifelessqueued jobs, I mean03:04
*** dcramer_ has joined #openstack-infra03:05
*** mriedem has quit IRC03:06
*** todd_dsm has quit IRC03:08
*** zns has quit IRC03:12
*** emagana has joined #openstack-infra03:13
*** matsuhashi has quit IRC03:15
*** thuc has joined #openstack-infra03:17
*** thuc has quit IRC03:19
*** thuc has joined #openstack-infra03:20
pabelangerlifeless, I believe there is a backup from rolling back from hpcloud-region-b03:21
mordredclarkb, pabelanger, fungi: should I or anyone else communicate anything to hp?03:22
pabelangermordred, well, I only started using hpcloud today, so my feedback is pretty limited.03:24
pabelangerbut am seeing slow transfer rates to ubuntu archive for updates03:24
mordredpabelanger: welcome to the jungle03:24
*** emagana has quit IRC03:24
pabelangerI already talked to support about some internal mirror if there was one, they are getting back to me03:24
*** thuc has quit IRC03:27
*** nosnos has quit IRC03:28
clarkblifeless: yes I think you are correct03:29
clarkbmordred: possibly, I wasn't around for much of the afternoon fun, but it sounded like node boot time was terrible03:29
*** khyati has quit IRC03:36
*** gokrokve has quit IRC03:37
*** gokrokve_ has joined #openstack-infra03:38
*** emagana has joined #openstack-infra03:39
*** dkliban has quit IRC03:39
*** thomasem has quit IRC03:39
*** jepoy_ has joined #openstack-infra03:42
*** gokrokve_ has quit IRC03:42
*** alex-lunch is now known as Alexandra03:43
*** wchrisj has quit IRC03:43
*** jepoy has quit IRC03:44
*** blamar has quit IRC03:46
*** khyati has joined #openstack-infra03:46
*** thuc has joined #openstack-infra03:48
*** mestery has joined #openstack-infra03:51
*** emagana has quit IRC03:51
*** blamar has joined #openstack-infra03:56
*** blamar has quit IRC03:57
*** khyati has quit IRC03:58
*** blamar has joined #openstack-infra03:58
*** nati_ueno has joined #openstack-infra04:02
*** thuc_ has joined #openstack-infra04:03
*** thuc has quit IRC04:06
*** gokrokve has joined #openstack-infra04:07
*** todd_dsm has joined #openstack-infra04:10
*** blamar has quit IRC04:10
*** gokrokve has quit IRC04:13
*** sabari has joined #openstack-infra04:13
*** lcheng has joined #openstack-infra04:14
*** david-lyle has joined #openstack-infra04:16
*** SumitNaiksatam has joined #openstack-infra04:18
*** khyati has joined #openstack-infra04:18
*** mihgen has joined #openstack-infra04:22
*** dolphm has quit IRC04:25
*** matsuhashi has joined #openstack-infra04:25
*** SumitNaiksatam has quit IRC04:25
*** mestery has quit IRC04:25
*** dolphm has joined #openstack-infra04:27
*** vkozhukalov has quit IRC04:28
*** pcrews has quit IRC04:28
*** mrodden has quit IRC04:30
*** nosnos has joined #openstack-infra04:36
*** todd_dsm has quit IRC04:39
*** blamar has joined #openstack-infra04:46
*** basha has joined #openstack-infra04:48
*** todd_dsm has joined #openstack-infra04:49
openstackgerritMORITA Kazutaka proposed a change to openstack-infra/config: Add Jenkins jobs for swift3
*** imcsk8_lap has joined #openstack-infra04:56
*** imcsk8_lap has quit IRC04:57
*** _nadya_ has joined #openstack-infra05:02
*** _nadya_ has quit IRC05:03
*** amcrn has quit IRC05:08
*** harlowja_at_home has joined #openstack-infra05:14
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: Clean up a few errant tabs in setup.cfg
*** skraynev_afk is now known as skraynev05:22
*** thuc_ has quit IRC05:26
*** thuc has joined #openstack-infra05:27
*** harlowja_at_home has quit IRC05:27
*** _nadya_ has joined #openstack-infra05:30
*** thuc has quit IRC05:31
*** talluri has joined #openstack-infra05:39
*** emagana has joined #openstack-infra05:40
*** mgagne1 has joined #openstack-infra05:45
*** mgagne has quit IRC05:46
*** nicedice has quit IRC05:51
*** ildikov_ has quit IRC05:52
*** e0ne has joined #openstack-infra05:55
*** nati_ueno has quit IRC05:57
*** _nadya_ has quit IRC06:00
*** khyati has quit IRC06:04
*** Ryan_Lane has joined #openstack-infra06:05
*** gokrokve has joined #openstack-infra06:08
*** Ryan_Lane has quit IRC06:11
*** sdake has quit IRC06:12
*** sdake_ has quit IRC06:12
*** gokrokve has quit IRC06:13
*** khyati has joined #openstack-infra06:14
*** saju_m has joined #openstack-infra06:19
*** sdake has joined #openstack-infra06:24
*** sdake has quit IRC06:24
*** sdake has joined #openstack-infra06:24
*** tnurlygayanov_ has joined #openstack-infra06:27
*** sdake_ has joined #openstack-infra06:27
*** tnurlygayanov has quit IRC06:27
*** mrda is now known as mrda_away06:31
*** matsuhashi has quit IRC06:33
*** VijayTripathi has quit IRC06:34
*** matsuhashi has joined #openstack-infra06:35
*** thuc has joined #openstack-infra06:37
*** ichavero|zZz has joined #openstack-infra06:41
*** thuc has quit IRC06:42
*** ichavero|zZz is now known as imcsk8_lap06:42
imcsk8_laphello are you guys habing problems with name resolution on the jenkins jobs?? i'm having them:
*** mgagne1 has quit IRC06:44
*** afazekas has quit IRC06:46
*** e0ne has quit IRC06:47
*** e0ne has joined #openstack-infra06:48
*** jlibosva has joined #openstack-infra06:49
*** amcrn has joined #openstack-infra06:51
*** e0ne has quit IRC06:51
*** khyati has quit IRC07:00
*** vkozhukalov has joined #openstack-infra07:02
*** matsuhas_ has joined #openstack-infra07:03
*** matsuhas_ has quit IRC07:04
*** jcoufal has joined #openstack-infra07:04
*** matsuhas_ has joined #openstack-infra07:06
*** matsuhashi has quit IRC07:07
*** ildikov_ has joined #openstack-infra07:11
*** saju_m has quit IRC07:13
openstackgerritA change was merged to openstack-infra/storyboard: Added paging to list endpoints
SergeyLukjanovimcsk8_lap, hey, it's a known issue -
uvirtbotLaunchpad bug 1270382 in openstack-ci "Name resolution failure on gate job" [Medium,Confirmed]07:18
*** wenlock has quit IRC07:20
openstackgerritA change was merged to openstack-infra/storyboard: DataBase Token Storage added
*** afazekas has joined #openstack-infra07:21
*** afazekas has quit IRC07:25
*** sabari has quit IRC07:35
*** markmcclain has joined #openstack-infra07:37
*** lcheng has quit IRC07:37
*** afazekas has joined #openstack-infra07:40
*** Alexandra is now known as alex-gone07:41
*** yolanda has joined #openstack-infra07:46
*** mihgen has quit IRC07:50
*** johnthetubaguy has joined #openstack-infra07:51
*** johnthetubaguy1 has joined #openstack-infra07:54
*** johnthetubaguy has quit IRC07:54
*** medieval1 has quit IRC07:57
*** flaper87|afk is now known as flaper8707:57
*** markmcclain has quit IRC07:57
*** mihgen has joined #openstack-infra07:57
*** matsuhas_ has quit IRC07:59
*** medieval1 has joined #openstack-infra08:01
*** vkozhukalov has left #openstack-infra08:02
*** matsuhashi has joined #openstack-infra08:04
*** gokrokve has joined #openstack-infra08:08
*** gokrokve has quit IRC08:12
*** mihgen has quit IRC08:13
openstackgerritA change was merged to openstack-infra/storyboard-webclient: Added prod convenience invocation
*** pblaho has joined #openstack-infra08:18
*** hashar has joined #openstack-infra08:23
*** jgallard has joined #openstack-infra08:31
*** jgallard has quit IRC08:33
*** jgallard has joined #openstack-infra08:33
*** emagana has quit IRC08:33
*** chandankumar_ has joined #openstack-infra08:42
*** dizquierdo has joined #openstack-infra08:43
*** rlandy has joined #openstack-infra08:48
*** amotoki has quit IRC08:51
*** saschpe has quit IRC08:54
*** jcoufal has quit IRC08:59
*** jcoufal has joined #openstack-infra09:00
*** saschpe has joined #openstack-infra09:03
*** jpich has joined #openstack-infra09:03
*** mihgen has joined #openstack-infra09:06
*** yassine has joined #openstack-infra09:08
*** yassine has quit IRC09:08
*** yassine has joined #openstack-infra09:08
*** derekh has joined #openstack-infra09:11
*** AJaeger has quit IRC09:12
*** gokrokve has joined #openstack-infra09:13
*** gokrokve has quit IRC09:13
*** gokrokve has joined #openstack-infra09:14
*** gokrokve has quit IRC09:16
*** gokrokve has joined #openstack-infra09:17
*** AJaeger has joined #openstack-infra09:17
*** gokrokve has quit IRC09:21
*** andreaf has joined #openstack-infra09:22
*** johnthetubaguy1 has quit IRC09:23
*** ominakov has joined #openstack-infra09:24
*** dizquierdo has quit IRC09:27
*** hashar has quit IRC09:28
*** Trozz has quit IRC09:28
*** Trozz has joined #openstack-infra09:29
*** Trozz_ has joined #openstack-infra09:30
*** Trozz_ has quit IRC09:30
*** Trozz has quit IRC09:30
*** Trozz has joined #openstack-infra09:31
*** Trozz_ has joined #openstack-infra09:31
*** mich5225 has joined #openstack-infra09:32
*** Trozz has quit IRC09:33
*** Trozz_ has quit IRC09:33
*** mich5225 has quit IRC09:33
*** Trozz has joined #openstack-infra09:33
*** hashar has joined #openstack-infra09:35
*** e0ne has joined #openstack-infra09:39
*** Trozz has quit IRC09:39
*** Trozz has joined #openstack-infra09:40
*** derekh has quit IRC09:42
*** derekh has joined #openstack-infra09:42
*** Trozz has joined #openstack-infra09:43
*** pbelanyi has joined #openstack-infra09:44
*** gokrokve has joined #openstack-infra09:44
*** Trozz has quit IRC09:45
*** bauzas has joined #openstack-infra09:51
*** yjiang|2 has joined #openstack-infra10:00
*** yjiang has quit IRC10:00
*** jp_at_hp has joined #openstack-infra10:01
*** AJaeger has quit IRC10:03
*** matsuhashi has quit IRC10:09
*** AJaeger has joined #openstack-infra10:12
*** yamahata has quit IRC10:15
*** dizquierdo has joined #openstack-infra10:17
*** chandankumar_ has quit IRC10:22
*** dkliban has joined #openstack-infra10:23
*** Trozz has joined #openstack-infra10:31
*** Trozz has quit IRC10:34
*** yjiang|2 has quit IRC10:35
*** matsuhashi has joined #openstack-infra10:39
*** Trozz has joined #openstack-infra10:41
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: DB api refactoring
*** Trozz has quit IRC10:42
*** Trozz has joined #openstack-infra10:44
*** Trozz has quit IRC10:45
*** hashar has quit IRC10:45
*** hashar has joined #openstack-infra10:48
ttxsdague: Hi! could you +2/APRV before I start -2ing all requirements changes ?10:49
*** jepoy_ has quit IRC10:53
*** jepoy has joined #openstack-infra10:53
sdaguettx: done10:54
*** che-arne has joined #openstack-infra10:54
sdague-2 away!10:54
*** hashar has quit IRC10:54
ttxI'm going through them -1ing the ones I think we could consider, -2ing the ones we probably should freeze10:54
*** hashar has joined #openstack-infra10:54
ttxsdague: do you happen to know how functional ">=1.6.1,<2.0,>=2.1" is @ ?10:55
sdaguethat I don't know10:55
sdagueI've seen !=10:55
ttxsdague: will ask mordred about it10:55
sdaguettx: works for me10:56
*** jepoy_ has joined #openstack-infra10:57
sdagueso testing it on the command line, it does what you think it might do, even if it doesn't make logical sense from a math perspective10:57
ttxsdague: maybe I should just +2 and make it pass just below the bar. Opinion ?10:58
*** jepoy has quit IRC10:58
sdagueI could go either way on that. Realistically, stuff isn't going to use it until juno10:58
ttxok, can wait a week10:58
sdagueI honestly think it's ok that certain parts of our process slow down during release. Getting people to focus on the core release is goodness, instead of running full speed ahead on stuff that's juno only.11:00
sdagueso something like this has almost no chance of negatively impacting the release technically, but it's fine to wait as well11:00
*** lcostantino has joined #openstack-infra11:01
ttxyep, -2ed11:01
*** jgallard has quit IRC11:10
ttxsdague: about -- is the gate using the lower bound or the upper bound in testing ? Your comment seems to suggest it uses the upper bound ?11:11
sdagueso... it depends11:12
sdaguebasically if the requirement is satisfied by 12.04 packages, we use that11:12
sdagueotherwise it's a pip install11:12
sdaguewhich means it's going to be the newest version11:12
ttxsdague: I guess &2.04 has a psutil1 in that case11:13
sdaguewe can see in a log11:13
ttxsdague: ok thx11:13
sdagueor not... maybe neutron needs it?11:14
sdagueno, it's removed there as well11:15
sdagueso what concerns me about this is it was being brought in because of some rather complicated incubator code11:16
*** jepoy_ has quit IRC11:16
sdaguewhich is why it had a version lock11:16
sdaguebut we're not good about ever getting past version locks11:16
sdagueso I really want to avoid them11:16
*** jepoy has joined #openstack-infra11:16
sdagueand instead have the code written to support multiple versions11:16
ttxLooks like glance wants psutil11:17
ttxand neutron11:17
sdagueI guess they take from the distro11:17
ttxgot pulled from neutron, so maybe it's only in glance11:19
*** saju_m has joined #openstack-infra11:20
*** jepoy has quit IRC11:21
*** e0ne has quit IRC11:23
*** Trozz has joined #openstack-infra11:23
*** e0ne has joined #openstack-infra11:23
*** dkliban_ has joined #openstack-infra11:27
*** dkliban has quit IRC11:27
*** basha_ has joined #openstack-infra11:27
Mithrandirhmm, so, for j-j-b, would people be ok with extending it so it can manage plugins and such too?11:28
*** ociuhandu has quit IRC11:29
*** e0ne has quit IRC11:29
*** mwagner_ is now known as mwagner_notHere11:29
*** Trozz has joined #openstack-infra11:29
*** basha has quit IRC11:29
*** basha_ is now known as basha11:29
*** ominakov has quit IRC11:30
*** dkliban_ is now known as dkliban11:31
*** Trozz has quit IRC11:36
ihrachysjust got dns failure at when resolving Any ideas?11:36
sdagueBug 1270382 - Name resolution failure on gate job - it's being worked on by jeblair11:39
uvirtbotLaunchpad bug 1270382 in openstack-ci "Name resolution failure on gate job" [Medium,Confirmed]
ihrachysok, tnx11:39
sdaguerax has intermittent dns issues11:39
sdagueroot cause unknown11:39
*** yamahata has joined #openstack-infra11:40
ihrachyssdague: safe to recheck, right?11:40
sdagueihrachys: yes11:41
imcsk8_lapihrachys: hello, one quick question. how do i trigger the recheck?11:42
sdagueimcsk8_lap: leave a comment that is11:43
sdaguerecheck bug ######11:43
sdaguewhere ##### is the bug number11:43
*** hashar has quit IRC11:43
ihrachysimcsk8_lap: the link is posted by jenkins in reviews when they fail in gate11:43
imcsk8_lapsaju_m, ihrachys thanks!11:43
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Fix 009 migration
*** gokrokve has quit IRC11:45
*** gokrokve has joined #openstack-infra11:46
*** gokrokve has quit IRC11:50
*** jcoufal has quit IRC11:52
*** ociuhandu has joined #openstack-infra11:53
*** wenlock has joined #openstack-infra11:55
*** jcoufal has joined #openstack-infra11:55
*** rfolco has joined #openstack-infra11:56
*** gokrokve has joined #openstack-infra11:57
*** bogdando has quit IRC11:58
*** wenlock has quit IRC11:58
*** e0ne has joined #openstack-infra12:00
*** mwagner_lap has quit IRC12:00
*** Trozz has joined #openstack-infra12:02
*** Trozz has quit IRC12:03
*** wchrisj has joined #openstack-infra12:04
*** e0ne has quit IRC12:04
*** ArxCruz has joined #openstack-infra12:04
*** e0ne has joined #openstack-infra12:05
*** yamahata has quit IRC12:06
openstackgerritAnita Kuno proposed a change to openstack-infra/meetbot: Prevents meetbot from starting meetings in non-meeting channels
*** basha has quit IRC12:06
anteayaSergeyLukjanov: care to share your thoughts with me?12:09
SergeyLukjanovanteaya, hey12:09
anteayacan you expand on your concerns so that I can understand them?12:09
SergeyLukjanovanteaya, okay, I'll try to explain:)12:10
*** e0ne has quit IRC12:10
SergeyLukjanovanteaya, #1 is that it's quite useful IMO to run meetings in project channels12:10
*** matsuhashi has quit IRC12:10
SergeyLukjanovanteaya, #2 is that blacklisting channels will not help in general12:10
*** bogdando has joined #openstack-infra12:10
SergeyLukjanovsomeone could start meeting in #.*meeting.*12:11
anteayaI'm not sure I agree with your use of the term blacklisting12:11
*** matsuhashi has joined #openstack-infra12:11
anteayaso regarding running meetings in project channels12:11
SergeyLukjanovanteaya, do you like whitelisting? :)12:11
anteayamy understanding of jim's point last night is that he feels that running meetings in project channels is a bad idea12:11
anteayasince that is why we have the meeting channels12:12
SergeyLukjanovshame on me, I've skipped reading scrollback today's morning12:12
anteayamind you I am paraphrasing, but that is the take away I had regarding jim's points from last nigh12:12
sdagueSergeyLukjanov: I'm with jeblair on that12:12
sdaguethe reason for the meeting channels is so there is a "public square" where meetings happen12:12
*** pdmars has joined #openstack-infra12:13
anteayathere was some discussion to not worry about what the stackforge projects do12:13
*** matsuhas_ has joined #openstack-infra12:13
*** matsuhashi has quit IRC12:13
SergeyLukjanovanteaya, agreed12:13
anteayain that case, I am open to changing the conditional to also checking if "stackforge" is in the channel topic12:13
SergeyLukjanovsdague, ++ for public square12:14
*** e0ne has joined #openstack-infra12:14
* persia really appreciates being able to review multiple project meeting backscroll as a way to see what is happening in projects for which the channels are too high volume to follow directly, which only works with *-meeting-* channels.12:14
SergeyLukjanovmy main concern is this it doesn't solve root issue - anyone could start meeting12:14
SergeyLukjanovanteaya, and I agree with you that we don't like to store lists of chairs in infra12:15
SergeyLukjanovso, I think I'm going to +2 your change12:15
SergeyLukjanovbecause of lack of solutions12:15
anteayaanyone can start a meeting in a meeting channel12:15
anteayawhich is the design, yes12:16
anteayasince chairs for meetings are to be decided by the folks holding the meeting12:16
anteayaupon checking the list I have for channels meetbot logs to eavesdrop:
anteayaI don't see any stackforge channels there12:16
sdagueanteaya: murano ?12:17
*** dims_ has quit IRC12:18
SergeyLukjanovanteaya, murano, climate, dns, sdks(?)12:18
*** dcramer_ has quit IRC12:19
anteayaah so they don't put "stackforge" in the channel topic12:20
sdagueanteaya: when did neutron run a meeting in their channel?12:20
sdaguehas this been a real issue?12:20
fungiright, there's currently no "easy" way to discern stackforge from non-stackforge channels since lots of them have names which start with #openstack-12:20
anteayasomeone showed up and started a meeting12:21
anteayamark asked me is there a way to prevent it in future12:21
*** dims_ has joined #openstack-infra12:21
anteayaI didn't think it would become what it is12:21
anteayanow they were probably not paying attention to what channel they were in12:21
anteayaand ended the meeting when it was brought to their attention12:22
sdaguefungi: something is really weird with nodepool btw12:22
anteayabut I was asked could something be done, so I tried12:22
*** e0ne_ has joined #openstack-infra12:23
openstackgerritArx Cruz proposed a change to openstack-infra/devstack-gate: Correcting the right path to mysql log in Fedora 19
*** e0ne has quit IRC12:23
fungisdague: agreed, looks like maybe there aren't as many operable nodes as it thinks. i wonder if we confused it by shuffling quota between old and new hpcloud regions and back again last night12:24
fungichecking for stale nodes now12:24
*** e0ne has joined #openstack-infra12:25
*** aysyd has joined #openstack-infra12:26
*** david-lyle has quit IRC12:27
fungisomehow we ended up with a ton of py3k-precise nodes in a ready state in hpcloud for many hours now12:28
*** e0ne_ has quit IRC12:28
sdaguethis would be one of those places where some monitoring would be nice for sanity, so it doesn't require a person to alert folks that everything is wonky12:28
fungiout of curiosity, what would your wonkameter for this particular issue look like?12:29
*** jcoufal has quit IRC12:30
fungithe py3k-precise nodes do actually appear to be getting used, i think nodepool has merely misestimated the ratio of them it needs to keep in hpcloud12:30
sdagueso the thing that sticks out is the number of ready nodes vs. queued12:30
*** jcoufal has joined #openstack-infra12:30
openstackgerritDerek Higgins proposed a change to openstack-infra/config: Add the second tripleo provider
fungii'm checking the logs now to see if it still thinks it's supposed to be adding a bunch of them12:31
fungimost of those ready nodes are in that state for about 10.75 hours, so i think nodepool spun up the majority when the quota for the old hpcloud azs was reintroduced last night12:34
fungithat roughly coincides with the overall rise in building/ready nodes and workers lines on the graphs12:36
fungithen the comments left by the auto-abandon cron job triggering check pipeline runs plus the periodic jobs firing at the same time around 6.5 hours ago accounts for the spike in waiting jobs which never subsided because we're working through them too slowly12:37
openstackgerritNikita Konovalov proposed a change to openstack-infra/storyboard: Moving to soft delete
fungipicking a ready py3k-precise node at random from that time period, it's up, reachable via ssh and in jenkins waiting to grab a job, so i think we're really just looking at an early miscalculation on node types needed by nodepool, probably due to there having been so many jobs waiting on py3k-precise nodes prior to that (because of the situation with all the build timeouts in region b)12:39
fungianyway, i can speed things up by just killing off a bunch of the py3k-precise nodes we don't currently need so we can use that quota for more necessary purposes12:40
*** dprince has joined #openstack-infra12:41
sdaguejust seems we could come up with what normal operating would be for nodepool, and see if we are staying inside the lines with normal over time12:42
*** jepoy has joined #openstack-infra12:42
fungii agree, it does seem that way... curious whether it actually turns out to be the case12:43
*** dkranz has joined #openstack-infra12:44
fungianyway, i'm deleting any nodes which have been waiting in a ready state for >3 hours... that gets us back about 300 nodes of available capacity for other purposes12:44
fungipresumably nodepool will recalculate what it actually needs based on current demand now12:44
sdaguefungi: sure, but we'll never know unless we try :)12:45
fungii believe, looking at the logs, that it was already doing so... it just doesn't have a provision yet for "i have too many of *this* type, i should delete some)12:45
fungiand we were reclaiming available slots in the quota too slowly because we don't normally use very many py3k-precise nodes compared to other types12:46
*** jeckersb_gone is now known as jeckersb12:47
openstackgerritMORITA Kazutaka proposed a change to openstack-infra/config: Add Jenkins jobs for swift3
*** hashar has joined #openstack-infra12:49
*** jepoy has quit IRC12:50
*** jepoy has joined #openstack-infra12:51
*** jaypipes has quit IRC12:51
*** thomasem has joined #openstack-infra12:52
*** gokrokve has quit IRC12:53
*** gokrokve has joined #openstack-infra12:53
*** mrodden has joined #openstack-infra12:53
*** lcheng has joined #openstack-infra12:55
*** jaypipes has joined #openstack-infra12:56
fungiwe've also got a little over 70 nodes which have been in a delete state for more than 3 hours, so i'm trying to more forcibly deleting them now12:57
*** thuc has joined #openstack-infra12:57
*** gokrokve has quit IRC12:57
*** jhesketh__ has quit IRC12:58
*** thuc_ has joined #openstack-infra12:58
*** thuc has quit IRC13:02
*** bknudson has joined #openstack-infra13:03
*** wenlock has joined #openstack-infra13:03
*** unicell has quit IRC13:04
*** mfer has joined #openstack-infra13:04
*** weshay has joined #openstack-infra13:05
*** e0ne_ has joined #openstack-infra13:07
*** gokrokve has joined #openstack-infra13:07
*** matsuhas_ has quit IRC13:07
*** jgallard has joined #openstack-infra13:07
*** CaptTofu has quit IRC13:07
*** CaptTofu has joined #openstack-infra13:08
*** johnthetubaguy has joined #openstack-infra13:09
*** CaptTofu has quit IRC13:09
*** CaptTofu has joined #openstack-infra13:09
*** e0ne has quit IRC13:10
*** johnthetubaguy1 has joined #openstack-infra13:12
*** mwagner_lap has joined #openstack-infra13:13
*** johnthetubaguy has quit IRC13:13
*** CaptTofu has quit IRC13:14
*** CaptTofu_ has joined #openstack-infra13:14
*** julim has joined #openstack-infra13:16
*** adalbas has joined #openstack-infra13:16
*** basha has joined #openstack-infra13:17
*** mihgen has quit IRC13:17
*** yamahata has joined #openstack-infra13:18
*** zhiyan is now known as zhiyan_13:19
*** thomasem has quit IRC13:19
*** zhiyan_ is now known as zhiyan13:19
openstackgerritThierry Carrez proposed a change to openstack-infra/release-tools: Fix wait_for_tarball 'None' error
*** lcheng has quit IRC13:20
*** bknudson has quit IRC13:23
*** unicell has joined #openstack-infra13:25
*** signed8bit has joined #openstack-infra13:31
*** yassine_ has joined #openstack-infra13:32
*** mrodden has quit IRC13:35
*** nosnos has quit IRC13:35
*** afazekas_ has joined #openstack-infra13:36
*** afazekas has quit IRC13:39
*** jeckersb is now known as jeckersb_gone13:39
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: be extra explicit for people reading logs
sdaguettx: ^^^13:47
*** jeckersb_gone is now known as jeckersb13:47
*** nkinder has quit IRC13:49
sdaguettx: so the issue is, that failure specifies the devstack log only if devstack is the load13:50
sdaguebut d-g is used for things besides devstack13:50
sdaguegate_hook is a variable13:50
sdagueand is replaceable13:51
*** sahumada_ has joined #openstack-infra13:53
*** pcrews has joined #openstack-infra13:55
*** skraynev is now known as skraynev_going_h13:59
*** skraynev_going_h is now known as skraynev_afk13:59
*** thuc_ has quit IRC14:00
ttxsdague: ok +1ed -- "exit code" should also hint at "look just above"14:00
*** thuc has joined #openstack-infra14:01
sdagueso, honestly if "please see relevant logs" and 3 lines up is "logs will be in..." aren't enough14:03
sdaguethen I'm not sure what will be14:03
*** eharney has joined #openstack-infra14:04
*** prad_ has joined #openstack-infra14:04
*** thuc has quit IRC14:05
openstackgerritSergey Lukjanov proposed a change to openstack-infra/devstack-gate: Prepare for Juno dev
openstackgerritSergey Lukjanov proposed a change to openstack-infra/devstack-gate: Enable sahara as default in gate
*** thuc has joined #openstack-infra14:05
*** thuc has quit IRC14:06
SergeyLukjanovttx, sdague, could you please take a look at
*** imcsk8_lap has quit IRC14:06
*** thuc has joined #openstack-infra14:07
*** imcsk8_lap has joined #openstack-infra14:07
sdaguesure, but putting the ? in breaks the url :)14:07
ttxSergeyLukjanov: still cleaning up I see14:07
SergeyLukjanovsdague, hmm, it works ok in my irc client, will know14:08
SergeyLukjanovttx, yeah, I have several CRs for final cleanup14:08
openstackgerritA change was merged to openstack-infra/config: Remove extra g722 sounds
SergeyLukjanovand I've opened CRs for enabling someday in J sahara in gate14:09
SergeyLukjanovI think it's the last non-approved cleanup
*** jgrimm has joined #openstack-infra14:11
sdaguexchat puts the ? in the url14:12
*** sandywalsh has joined #openstack-infra14:12
sdaguebecause it's a valid character14:13
*** talluri has quit IRC14:13
SergeyLukjanovsdague, I'm using textual14:13
SergeyLukjanovit's parsed as url14:13
*** basha has quit IRC14:14
SergeyLukjanovheh, it just always ignore ? in the end of url :)14:14
sdagueSergeyLukjanov: in xchat I can click them14:14
*** CaptTofu_ has quit IRC14:15
SergeyLukjanovsdague, mee too, but my client doesn't include trailing ? to the link, nice14:15
sdaguewell, it's valid14:15
*** CaptTofu has joined #openstack-infra14:16
*** dcramer_ has joined #openstack-infra14:16
*** amotoki has joined #openstack-infra14:18
openstackgerritA change was merged to openstack-infra/config: Remove savanna from logstash/jenkins-log-client.yaml
*** CaptTofu has quit IRC14:20
SergeyLukjanovfungi, thank you sir!14:20
*** thuc has quit IRC14:23
*** thuc has joined #openstack-infra14:23
*** thuc has quit IRC14:28
*** rcleere has joined #openstack-infra14:30
*** e0ne has joined #openstack-infra14:36
*** alexpilotti has joined #openstack-infra14:39
*** e0ne_ has quit IRC14:39
*** branen_ has quit IRC14:40
*** chandan_kumar has quit IRC14:42
*** nkinder has joined #openstack-infra14:42
*** sahumada_ has quit IRC14:43
*** skraynev_afk is now known as skraynev14:44
*** chandankumar_ has joined #openstack-infra14:44
jeblairsdague: can we get an idea of the iops needed for devstack runs from the dstat logs we have?14:45
jeblairsdague: also, good morning :)14:45
sdaguejeblair: good morning14:46
*** chandankumar_ has quit IRC14:46
sdaguelets see what's in a log...14:46
*** mestery has joined #openstack-infra14:46
jeblairso looks like we can get disk throughput at least14:47
*** chandan_kumar has joined #openstack-infra14:47
sdaguewe can probably enable other options14:47
sdaguewhat are you looking for ?14:47
jeblaircool, i'll try to take a median of that...14:48
*** chandan_kumar has quit IRC14:48
jeblairsdague: i'm talking with someone who might have some cloud to give us14:48
*** chandan_kumar has joined #openstack-infra14:48
sdague       -r, --io14:48
sdague              enable I/O request stats (read, write requests)14:48
sdaguewe could turn that on as well14:48
sdagueif it was useful14:48
*** thedodd has joined #openstack-infra14:48
*** reed has joined #openstack-infra14:48
jeblaircool, i'll throw that in a d-g change, and at least we'll have that run (or, well, 16 runs or whatever :)14:48
*** ominakov has joined #openstack-infra14:48
sdagueI think we are running the dstat defaults.... just 'cuz no one really had other bits they wanted14:48
sdagueyeh, actually in devstack change14:49
sdagueif there is anything else in dstat you think would be useful for either diagnose or scaling, please just propose it14:49
sdaguedstat seems pretty light impact on the system, so it's fine to collect that14:50
jeblairsdague: ack, thx14:50
*** CaptTofu has joined #openstack-infra14:50
jeblairno such file or directory14:52
*** hashar has quit IRC14:52
*** david-lyle has joined #openstack-infra14:53
*** emagana has joined #openstack-infra14:55
*** chandan_kumar has quit IRC14:55
*** chandan_kumar has joined #openstack-infra14:56
sdagueprobably worth adding +214:58
*** sweston has quit IRC14:58
*** emagana has quit IRC15:00
*** julim_ has joined #openstack-infra15:00
*** julim has quit IRC15:01
openstackgerritNathan Kinder proposed a change to openstack-infra/config: Add the openstack-security-notes project
*** kmartin has quit IRC15:02
Mithrandirhmm, is dbailey on irc?  I'd really like something like to move forward.15:02
*** chandan_kumar has quit IRC15:03
*** yamahata has quit IRC15:03
*** VijayTripathi has joined #openstack-infra15:03
jeblairMithrandir: i don't think he is normally  :(15:03
*** atiwari has joined #openstack-infra15:04
*** thuc has joined #openstack-infra15:04
Mithrandirhmm, is Khai Do on irc, then?  I think dbailey might be spurred into action if he gets a reply to the question there15:05
Mithrandiror somebody else could bring it forward.15:05
*** thuc_ has joined #openstack-infra15:05
*** mestery has quit IRC15:05
jeblairMithrandir: zaro is khai do15:05
*** weshay has quit IRC15:05
Mithrandirjeblair: great, thanks.15:05
*** yamahata has joined #openstack-infra15:06
Mithrandirzaro: any chance you could respond to dbailey's question at the end of ?15:06
Mithrandirzaro: I'd really like to see something like that feature go in, so.. :-)15:06
anteayaMithrandir: he might not be around for an hour or so15:06
anteayahe is on PST15:06
*** hashar has joined #openstack-infra15:06
fungiMithrandir: darraghb seems to be /join'd on weekdays 09:00-17:00 utc (ireland i think)15:06
Mithrandiranteaya: right, hopefully he reads backlog.15:07
Mithrandirfungi: thanks. :-)15:07
*** mihgen has joined #openstack-infra15:07
fungioh, though last time i saw him in #openstack-dev was a year ago, nevermind15:07
Mithrandiroh, ok15:07
Mithrandirthat's a while ago indeed.15:07
anteayaMithrandir: he usually does I think15:07
*** vhoward has joined #openstack-infra15:07
fungiunless he changed his nick, 2013-04-26 17:46:21 was the last time he /part'd15:07
jeblairsdague: ha, our median disk read throughput is 0.  :)15:08
sdagueyeh, not surprising15:08
*** thuc has quit IRC15:09
jeblairwe read nothing 2/3 of the time15:09
*** yamahata has quit IRC15:09
*** chandan_kumar has joined #openstack-infra15:09
sdagueonce we get the cirros image into page cache that takes away a lot of our reads15:09
*** yamahata has joined #openstack-infra15:10
jeblairour median write throughput is 16M/s15:10
*** thuc_ has quit IRC15:12
*** thuc has joined #openstack-infra15:13
*** UtahDave has joined #openstack-infra15:13
*** yolanda has quit IRC15:13
*** mflobo has joined #openstack-infra15:15
*** sweston has joined #openstack-infra15:16
*** wchrisj has quit IRC15:18
*** lcostantino has quit IRC15:18
jeblairsdague: also to add memory stats15:19
*** markmc has joined #openstack-infra15:19
*** UtahDave has left #openstack-infra15:20
*** weshay has joined #openstack-infra15:21
sdaguejeblair: right, volumes :)15:23
jeblairmean and median cpu usage values are about 35%, max 96%15:26
jeblairmaybe when i'm done with this script, i could tack it onto devstack-gate to output a summary for each job15:26
sdagueI think that would be handy15:27
*** bknudson has joined #openstack-infra15:30
mordredsdague: reading scrollback...15:31
mordred11:12:43          sdague | basically if the requirement is satisfied by 12.04 packages, we use that15:31
mordred11:12:52          sdague | otherwise it's a pip install15:31
mordredjust to bring it up again - I'd REALLY like that ^^ to stop being the answer at some point15:31
sdaguemordred: sure15:31
sdagueit's just lower priority right now15:32
openstackgerritgordon chung proposed a change to openstack-infra/devstack-gate: fix trove merge
sdaguehonestly, that may be part of the big drag down knock out test matrix conversation at summit15:33
*** johnthetubaguy1 is now known as johnthetubaguy15:33
sdaguebecause with multinode, mongo jobs, max and min versions15:33
sdaguewe're talking about a lot of new combos15:33
sdaguethat people would like to see15:33
mordredbtw - I don't care about min versions, for the record, and I don't think we should either15:34
sdaguemordred: why?15:34
mordredbecause I don't think it provides value to15:34
fungiit's also not feasible to implement15:35
zaroMithrandir: lookin at it now15:35
mordredthat too15:35
sdagueit doesn't provide value to know that we work with sqla 0.7?15:35
mordrednot at all15:35
*** mihgen has quit IRC15:36
jeblairdoes in provide value to know that we don't work with it?15:36
fungiat least as long as we're using pip and pip continues to lack any real dependency solver, we can't just say "install the lowest versions of stuff" unless we manually spider the entire dependency tree first (and need to figure out what we do about things which specify no min version too, including on transitive deps)15:36
clarkbmordred jeblair fungi sdague et al holger is looking for feedback on tox bug 150 so that it can possibly be fixed in 1.7.1. I am indifferent to the suggested fix as it requires updating all tox.ini files and we can use 1.7.0 today of we do that15:36
uvirtbotLaunchpad bug 150 in baz "change arch_escape_classes" [Medium,New]
sdagueok, I'm trying to reconcile that with putting min versions global requirements15:36
jeblairbecause i think it would be great of the graphite devs knew that graphite doesn't work with the min version of django they specify15:36
sdaguefungi: I'm not saying it's trivial15:37
clarkbany chance you guys can catch up on the thread and provide feedback if you feel more strongly?15:37
*** lcheng has joined #openstack-infra15:37
mordredjeblair: ok. good point - let me rephrase the thing I'm trying to say15:37
*** jepoy_ has joined #openstack-infra15:37
Mithrandirzaro: great, thanks.15:38
*** emagana has joined #openstack-infra15:38
mordredwhich is that I _do_ think there is value in asserting that we don't work with version X - I think that given the way pip's dep solver works (per fungi) the work to try to infer which old versions we might not work with anymore is going to outweigh that value from an automated system15:39
sdagueif we actually don't care at all about anything less than latest, we can just fix that with adding -U to the pip install in devstack right?15:39
mordredso while I'm fine with someone filing a bug saying "we dont' work with sqla 0.7 anymore, we should bump the lower bound" - I think trying to discover that will be almost imposibble15:40
sdaguemordred: I don't think we'd discover a perfect answer15:40
mordredsdague: well, -U has some interesting semantics ... but yeah15:40
sdaguebut I do think we'd discover a rough answer that was useful15:40
clarkb is the link to the tox bug15:40
mordredsdague: k. I'm not opposed to rough answer15:41
clarkbI intend on commenting as soon as I am actually awake15:41
*** jepoy has quit IRC15:41
clarkbbut more feedback is better :)15:41
mordredI just don't want to try to gate on somethign that looks like the collected set of mins15:41
fungikeep in mind the corollary to my assertion is that we *could* potentially test this by either not using pip to discover and install the deps we need, or by implementing a good dep solver in pip15:41
*** mflobo has quit IRC15:41
sdagueit would be interesting how many shadow deps are still in there15:41
fungithe latter of which, i think, would be of great benefit to a lot of people, not just openstack15:42
sdaguewe've gotten more and more explicit over time15:42
sdaguethere is also the issue around python with native bindings15:42
sdaguethe pip -U is going to add a ton of time to gate jobs15:42
sdague3 minutes to build numpy, for instance15:42
mordredwe have wheels15:42
clarkbsdague we are logging "shadow" deps now15:42
sdaguefor everything?15:43
mordredsdague: yes15:43
clarkbwe know what isnt listed15:43
mordredthe mirror has wheels for everything15:43
fungisdague: we do a pip freeze15:43
mordredand we build them for the platforms we devstack on15:43
fungithough mordred... with modern pip should we be doing pip list instead of pip freeze?15:43
mordredfungi: probably15:43
*** saju_m has quit IRC15:43
fungiit seems to pick up things freeze doesn't, sometimes, from what i've seen15:43
dstufftthey are for different purposes15:44
mordredsdague: anyway - my main point in opening this can of worms, is that I think some-things-from-apt-some-things-from-pip is a thing we should get away from15:44
sdaguewell, now you make me curious15:44
fungidstufft: is there a good writeup somewhere for what their respective purposes are?15:44
dstufftpip freeze is used to create a requirements file to "freeze" an environment15:44
sdagueSean Dague proposed a change to openstack-dev/devstack: attempt pip install -U
mordredespeically since dstufft tells us it's a use case completely unsuppoed by him15:44
dstufftpip list is more of a UX thing15:44
dstuffter UI15:44
mordredsdague: so - the biggest problem with -U15:44
anteayaclarkb: after tox files are updated, is 1.7.1 better for us in any way, compared to 1.7.0?15:45
dstufftwe only exclude a few things from pip freeze15:45
mordredsdague: is that it changes how transitive ordering is going to work :)15:45
dstufftmostly becuase they are things you probably don't want frozen15:45
fungidstufft: ahh, setuptools being one example15:45
dstufftlike pip itself, setuptools15:45
dstufftI think are the main 315:45
sdaguemordred: sure, well lets see what happens15:45
zaroMithrandir: i've commented, but the change is in abandoned state.15:45
sdagueit will at least be data in the explosion15:45
fungimakes sense. so from a logging perspective, i think we *do* want to switch to pip list15:45
mordredsdague: since later declared deps will -U their transitive deps, whereas without -U, the transitive deps of later deps that were installed transitively by earlier declared deps will not re-install15:45
mordredsdague: ++15:45
clarkbanteaya: don't know yet, they are looking for feedback on the proposed fix. IMO helps preserve the old behavior with new config so it is a reasonable translation compared to the alternative of embedding a shell script15:46
dstufftI think we also ignore some stdlib stuff that used to be third party modules and still have packaging metadata in the stdlib for them15:46
dstufftlike argparse in 2.7 or something15:46
Mithrandirzaro: I'm happy to revive it if it doesn't get picked up.15:46
anteayaclarkb: k15:46
dstufftfungi: yea if you want something for diagnostics pip log is reasonable15:46
mordredsdague: oh - you _will_ see a slower build there - because I dont' believe we're configuring the mirror to pick up the wheel dirs15:46
MithrandirI don't know what the usual procedure for that is, I'd like to move it forward, but not step on toes.15:46
dstufftalso yes, mve away from half apt to half pip15:46
mordredbut the data on what would get installed and if it woudl work will still be good15:46
* clarkb goes ahead and tries to write up a comment now15:46
clarkbbefore hpk afks15:47
anteayaclarkb: I am logged into lp and don't have permission to view:
uvirtbotLaunchpad bug 150 in baz "change arch_escape_classes" [Medium,New]15:47
clarkbanteaya: that is the wrong bug15:48
dstufftmordred: do you use the same wheel house between consecutive runs?15:48
clarkbtox hosts bugs on bitbucket, I dropped a proper link not too long ago in sb15:48
anteayaah urvibot served that up15:48
mordreddstufft: I don't follow the question?15:48
dstufftmordred: I assume you're building wheels on the mirror with `pip wheel` ?15:48
mordreddstufft: yes!15:48
anteayayes, I got the bitbucket one, just wanted to ensure I wasn't missing something15:48
dstufftmordred: that has a destination directory (the wheel house)15:49
anteayauvirbot is being too helpful15:49
dstufftdo you use the same one between runs15:49
mordredI do not believe we're using the same wheelhouse15:49
mordredalthough I don't think we're opposed to15:49
dstufftI think if you set --find-links to that wheel house, iyou won't rebuild wheels you already have built15:49
jeblairthat is one really mixed metaphor, btw.15:50
mordredyes - I know we do that during the run - so taht we don't double-build wheels for repeated deps15:50
mordredjeblair: right/15:50
dstufftmordred: well just saying if you do that between consecutive runs it shouldn't inflate your mirror run time too much15:51
dstufftsince most/all of your wheels should already be built then15:52
*** wchrisj has joined #openstack-infra15:52
mordreddstufft: indeed15:52
*** mgagne has joined #openstack-infra15:58
*** mestery has joined #openstack-infra16:01
openstackgerritKhai Do proposed a change to openstack-infra/jenkins-job-builder: Added config options to not overwrite jobs desc
clarkbI responded to tox bug 150, please do chime in if you want16:01
uvirtbotLaunchpad bug 150 in baz "change arch_escape_classes" [Medium,New]
zaroMithrandir: i think you'll need to get the owner to un-abandone16:01
clarkbMithrandir: zaro: a gerrit admin can restore changes, what change needs restoring?16:02
Mithrandirand thanks.16:03
zaroclarkb: Mithrandir: review in question is
* Mithrandir goes to the shop to find some grub to eat16:03
*** pblaho has quit IRC16:04
*** wchrisj has quit IRC16:05
anteayafor some reason git clone git:// hangs for me16:06
anteayagit farm is quiet:
*** wchrisj has joined #openstack-infra16:07
anteayahttps:// works though16:07
fungianteaya: are you able to clone other repositories over git:// protocol? might be something blocking that port outbound16:09
*** Sukhdev has joined #openstack-infra16:10
*** _nadya_ has joined #openstack-infra16:10
*** sahumada has joined #openstack-infra16:11
*** wchrisj has quit IRC16:12
*** mgagne1 has joined #openstack-infra16:12
*** mgagne has quit IRC16:12
*** skraynev is now known as skraynev_afk16:14
*** weshay has quit IRC16:14
*** weshay has joined #openstack-infra16:15
openstackgerritAnita Kuno proposed a change to openstack-infra/gerritlib: Add empty-commit flag when creating a project with no commits
anteayafungi: good question16:15
anteayayes just cloned with: git clone git://
dhellmanncan I get another pbr-core member to take a look at please? I would like to cut a release before our RC period starts.16:16
jeblairsdague: dstat is python!16:16
jeblairsdague: so, "16E", as in, 16 exabytes, comes up as a value in network throughput far too often for me to believe.16:18
*** Ryan_Lane has joined #openstack-infra16:18
clarkbjeblair: we should run elasticsearch on those VMs :)16:19
sdaguejeblair: yes it is16:19
*** maxbit has joined #openstack-infra16:19
sdagueit also has a well defined plugin api16:19
fungianteaya: interesting... i just successfully cloned gerritlib via git protocol directly from each of the 5 git farm backends16:19
sdaguejeblair: presumably there is an overrun somewhere16:19
anteayafungi: let's assume derp until we here otherwise16:20
jeblairsdague: yeah, i'm looking to see if it's using the 16 or 32 bit counters16:20
jeblairi mean 32 or 6416:20
anteayaand thanks for checking the git farm16:20
*** e0ne_ has joined #openstack-infra16:21
clarkbanteaya: anyone can reach the backends as well. Just git clone git:// for example16:21
clarkbI really should set up a plan to switch the backend ports to the non offset port numbers16:21
anteayaclarkb: ah thanks16:22
*** khyati has joined #openstack-infra16:23
openstackgerritA change was merged to openstack/requirements: Remove obsolete python-savannaclient
*** e0ne has quit IRC16:24
*** boris-42 has quit IRC16:24
*** SumitNaiksatam has joined #openstack-infra16:24
* SergeyLukjanov reading scrollback16:27
*** hogepodge has quit IRC16:27
*** rfolco has quit IRC16:29
*** rfolco has joined #openstack-infra16:29
*** bauzas has quit IRC16:29
*** hogepodge has joined #openstack-infra16:29
ttxsdague: another cleanup in requireemnts:
*** jepoy_ has quit IRC16:30
sdaguettx: sure16:30
*** jepoy has joined #openstack-infra16:30
openstackgerritA change was merged to openstack-infra/release-tools: Fix wait_for_tarball 'None' error
clarkbsdague: re any desire to implement jeblair's suggestion before I approve?16:30
*** afazekas_ has quit IRC16:30
zaroclarkb: if you are in reviewing mode, could you please take a look at ?16:31
pabelangerjeblair, I found a side affect of setting min-ready = 0, once everything is setup and processing. If a new job is submitted into zuul, zuul will error out with lost jobs:16:31
pabelanger2014-03-26 16:24:58,241 ERROR zuul.Gearman: Job <gear.Job 0x7fdce013c650 handle: None name: build:gate-kickstandproject-ci-puppet-puppet-syntax unique: 44980fb4bb104d81bb9ce37a0d45dddf> is not registered with Gearman16:31
pabelangersorry, should have been pastebin URL16:31
sdagueclarkb: sure, I can fix that16:31
clarkbzaro: I may have intentionally avoided that one. I feel like if I review it I will -2 it16:32
clarkbzaro: but there is clearly interest in it so a -2 is probably not fair16:32
openstackgerritSean Dague proposed a change to openstack-infra/devstack-gate: be extra explicit for people reading logs
clarkbzaro: it appears to exploit a possible bug in jenkins to do a thing that JJB arguably shouldn't do16:33
*** jasondotstar has joined #openstack-infra16:33
jeblairpabelanger: yeah, the job isn't listed in gearman until a worker shows up and registers it.  once it is, it stays there, regardless of whether any workers are registered16:34
*** luisg has quit IRC16:34
jeblairpabelanger: zuul has that check to make sure that if you typo a job name, you don't end up with jobs stuck waiting around forever16:34
pabelangerjeblair, ya, so the chicken / egg issue I guess16:34
*** jepoy has quit IRC16:35
*** freyes has joined #openstack-infra16:35
jeblairpabelanger: we were discussing that it would be nice to have a timeout in zuul, so if zuul enqueues a job and nothing runs it after, say, 8 hours, zuul considers it failed16:35
jeblairpabelanger: if that existed, then i think it would be safer for zuul to skip the 'does job exist' check...16:35
jeblairpabelanger: or we could make that check optional in a config file16:35
*** rlandy has quit IRC16:35
*** boris-42 has joined #openstack-infra16:36
*** wchrisj has joined #openstack-infra16:36
*** Ryan_Lane has quit IRC16:36
sdaguemordred: pip install -U fails in new and unexpected ways -
sdaguebecause of an inability to find olso.config  after doing that16:37
zaroclarkb: it's been around for a while so we should accept or reject, don't want to ignore.16:37
*** kmartin has joined #openstack-infra16:38
* anteaya goes for a walk16:38
zaroinfra is opposed to new gerrit change screen, correct?16:39
pleia2good morning16:39
openstackgerritThierry Carrez proposed a change to openstack-infra/storyboard: Mention load_projects in dev install
zarofrom WIP plugin doc: Old change screen doesn't support JS API. So that the popup dialog is not shown if "Work In Progress" and "Ready In Review" buttons are used and no comments can be provided. New change screen should be used for best experience with WIP Workflow plugin16:39
zarojeblair, fungi , clarkb ^16:40
*** melwitt has joined #openstack-infra16:41
boris-42fungi hi16:41
boris-42fungi could you pls help with merging this patch =)16:42
jeblairzaro: well, that's not going to work16:42
boris-42fungi we would like to add our first functional test=)16:42
Mithrandirclarkb: re the "don't overwrite job descs", I'd rather do that by having the token be configurable and have token = None be "don't add anything and ignore description"16:45
*** bhuvan has joined #openstack-infra16:45
derekhI've been trying to get a handle on job failures and run times for tripleo jobs
derekhthen jogo pointed me at graphite16:46
*** atiwari has quit IRC16:46
Mithrandirclarkb: though, as I note in the change, I think the real fix is to have a "notes" plugin where you can add random notes.16:46
Mithrandirand not abuse the description.16:46
jeblairderekh: i may be able to help there16:46
derekhthe plot I've been trying doesn't give the times I expect,%20%2712hours%27)16:46
derekhjeblair: cool, also I wondering if  there is any precedence for ci jobs sending their own statsd stats?16:47
*** pbelanyi has quit IRC16:47
jeblairderekh: i think times are in milliseconds, so you need to scale by .00001666666666666666 to get minutes16:47
derekhjeblair: I havn't used graphite (or statsd) befor, but am willing to read up if pointed in the right direction16:48
*** hashar has quit IRC16:48
*** signed8bit has quit IRC16:48
derekhjeblair: ahh ok, so maybe the times are correct16:48
jeblairderekh: sorry 0.00001666666666666666 (graphite really wants that leading 0)16:48
jeblairderekh: so that looks something like that ^16:48
openstackgerritA change was merged to openstack-infra/config: Add check-rally-install-* template and jobs
*** mestery has quit IRC16:49
jeblairderekh: so changes mostly stay in that pipeline for around 1 hour16:49
Mithrandirit feels a lot like
derekhjeblair: ahh that looks better, so that hour is included in the total time16:49
jeblairderekh: (resident time is time change spent in queue, which should be ~= max(runtimes of all jobs) if the system is not busy, or much higher if it is busy and waiting on nodes)16:50
*** jasondotstar has quit IRC16:50
*** vkozhukalov has joined #openstack-infra16:51
derekhjeblair: so is there any timer available that doesn't include the time spent in the queue waiting?16:51
*** luisg has joined #openstack-infra16:51
jeblairderekh: yes16:52
*** luisg has quit IRC16:52
jeblairderekh: that's run time of successful check-tripleo-undercloud-precise jobs in the check-tripleo pipeline16:52
mordredsdague: wow. that is both new and unexpected16:53
jeblairderekh: you can put '*' in place of success to get success and failure on two lines, and you can use the 'summarize' them to include both values on one line16:54
*** vkozhukalov has left #openstack-infra16:54
sdaguemordred: given the other issues around namespace packages vs. install -e, I'm not entirely surprised16:55
derekhjeblair: great I'll dig a big more and see what I can pull together, I may be wrong but I'm thinking I'll leave the Failures out for this since they could happen at any stage and scew results16:55
*** gokrokve has quit IRC16:55
jeblairderekh: yeah, that's part of why they're broken out like that; same thing for the pipeline name -- that way we can include gate runs but exclude check runs...16:56
jeblairderekh: there's a change in review that will let us specify "the integrated gate" for these kinds of stats, which will be very helpful16:56
*** sabari2 has joined #openstack-infra16:57
*** chandan_kumar has quit IRC16:57
*** harlowja_away is now known as harlowja16:58
mordredsdague: I still don't grok the namepsace package issue and I'm sad I was away when it came up16:58
*** gokrokve_ has joined #openstack-infra16:58
sdagueI still don't grok it entirely either16:59
openstackgerritJason Kincl proposed a change to openstack-infra/jenkins-job-builder: Add support for Copy Artifact project copy permissions
*** atiwari has joined #openstack-infra17:00
derekhjeblair: ok thanks, btw the script to generate that HTML page is here if infra want it and thing it would be helpful, otherwise if I get what I'm looking for from graphite the script will just probably go away at some stage17:00
*** jlibosva has quit IRC17:00
*** nati_ueno has joined #openstack-infra17:00
jogoclarkb: nova log are now much smaller
*** sabari2 has quit IRC17:02
*** khyati has quit IRC17:02
*** sabari2 has joined #openstack-infra17:02
*** sabari2 is now known as sabari17:03
*** wchrisj has quit IRC17:05
*** krotscheck_ has joined #openstack-infra17:07
*** krotscheck has quit IRC17:08
*** krotscheck_ is now known as krotscheck17:08
openstackgerritNadya Privalova proposed a change to openstack-infra/devstack-gate: Fix ceilometer-anotification and trove start
*** mestery has joined #openstack-infra17:09
SergeyLukjanovsdague, please, take a look at ^^17:10
*** mkoderer has quit IRC17:11
SergeyLukjanovsdague, it means that trove isn't enabled in gate at all17:11
mordredreally? whatever happend to those tests ... vipul? (I don't see hub_cap around)17:12
*** lcostantino has joined #openstack-infra17:13
SergeyLukjanovsdague, correction
SergeyLukjanovmordred, lack of punctuation17:14
*** e0ne_ has quit IRC17:15
*** amcrn has quit IRC17:15
*** mkoderer has joined #openstack-infra17:15
fungiSergeyLukjanov: yeah, that's definitely unfortunate. just double-checking that havana failure on it before i approve17:16
mordredSergeyLukjanov: that's impressive17:16
*** sweston has quit IRC17:17
fungiSergeyLukjanov: is that failure because havana trove doesn't work with postgres? worried about breaking stable/havana with that17:18
mordredfungi: should we have trove enabled in havana?17:18
jeblairfungi: hrm, the service addition should only apply to master..17:18
fungiwasn't there a change up to fix the branch override issue which was causing trove to get started in havana even though it wasn't supposed to be? did that merge? if so, doesn't seem to be working17:19
SergeyLukjanovfungi, it should be only for master17:19
* fungi goes on a review hunt17:19
jeblairfungi: this one?
fungiahh, yep17:20
fungi+2'ing that again17:20
fungiokay, so nothing new there17:20
clarkbMithrandir: ++ to both ideas17:21
jeblairseparately, i wonder about: setup_localrc "new" "$GRENADE_OLD_BRANCH"17:21
clarkbjogo: woot thanks17:21
jeblairi mean, that looks really suspicious17:21
clarkbjogo: was that a collection of changes to do that?17:21
clarkbzaro: I agree JJB in general shouldn't ignore that change and hasnt which is why I have been able to17:22
vipulmordred: I believe a patch just went in a few days ago enabling trove in devstack-gate17:22
vipulit's likely we didn't actually notice it wasn't really working17:23
mordredok. cool17:23
* mordred is behind on the sitch17:23
jeblairoh, no i understand why $GRENADE_OLD_BRANCH is there, that makes sense.  we should comment that though.  :)17:24
jeblair approved17:24
ArxCruzjeblair: clarkb
*** _nadya_ has quit IRC17:26
ArxCruzon my ssystem, running Fedora, my vms have 2 cpus17:27
ArxCruznprocs shows and assign 217:27
ArxCruzbut it's entering in this if17:27
*** Ryan_Lane has joined #openstack-infra17:27
ArxCruzchanging > 3 to -gt 3 works17:27
jeblairArxCruz: bash is hard! :)17:27
ArxCruzjeblair: small patch is welcome? :)17:27
clarkbArxCruz: yup I think that is a valid bug17:28
fungithis is where we've been getting our dns advice from, right?
jeblairArxCruz: yes, but before you do that, you may want to consider whether the patch should fix that bug, or make it do what it does now and always /= 217:28
ArxCruzjeblair: my ci is failing because, although it has 2 cpus, it always dividing by 2 because it's entering there17:29
jogoclarkb: just a change to oslo-incubator and a change to nova17:29
ArxCruzjeblair: using -gt it doesn't because check 2 -gt 317:29
jeblairArxCruz: i don't think we've ever run tempest on <4 cpus; that check was in there because we wanted to avoid overloading the host17:29
jogohave patches out for cinder and ceilometer to do the same17:29
jeblairArxCruz: so i'm wondering if using both of your cpus would do that17:29
clarkbArxCruz: jeblair: right theory was with all of the services running + mysql + rabbit, there wasn't enough cpu time with tempest running on each cpu17:30
jeblairArxCruz: anyway, whatever you think best -- we don't have the experience with <= 2 vcpu machines17:30
clarkbgiving tempest half the cpus allows openstack and mysql and rabbit and so on to use the cpus too17:30
zaroclarkb: it needs your expertise.  i'm the only one that +2ed-it17:30
openstackgerritA change was merged to openstack-infra/elastic-recheck: Add query for heat bug 1297560
uvirtbotLaunchpad bug 1297560 in heat "*tempest-dsvm-neutron-heat-slow fails with WaitConditionTimeout" [Undecided,New]
ArxCruzjeblair: understand, I can test it and report for you guys my experiences17:31
ArxCruzbut the code is bugged anyway right ?17:31
jeblairclarkb: we might want to -=2 cpus rather than /=217:31
zaroOk.  i'm gonna drop my investigation to WIP plugin.17:31
jeblairArxCruz: definitely;  should be fixed one way or the other.  :)17:31
ArxCruzokay, cool :)17:31
*** nicedice has joined #openstack-infra17:31
jeblaircpu median 39 mean 40 max 9917:32
jeblairpercentages according to dstat ^17:32
clarkbjeblair: max(1, numcpus-2) or similar17:32
clarkbsdague: mtreinish ^ how does that sound to you guys?17:33
mordredwhen did I become the only person who thinks that the presentation laptops at linux events should actually run linux/17:34
mordredI mean, especially when said events are run by the linux foundation17:34
clarkbmordred: I am with you on that17:34
clarkblca was actually reasonably good about it17:34
mordredlca are good humans17:35
pleia2mordred: sigh17:35
zarocan you dual boot macbook into linux?17:36
zarohmm, worth investigating i guess.17:37
pleia2I love my non-mac laptop that doesn't need a silly dongle to connect to projectors :)17:37
zaroi'm with you on that :)17:38
mordredpleia2: ++17:38
jeblairpleia2: i never got that -- surely someone at apple would have said "hey guys, people try to give presentations with our computers all the time and no one ever has the right dongle so they all start half an hour late with apoligies for how their computer doesn't work with the projector"17:39
jeblair"that looks bad for us"17:39
jeblairbut i guess that's why i don't work at apple17:39
mtreinishclarkb: sounds good to me17:40
clarkbmordred: is there a current plan for ? I don't want to rereview it until we are committed to getting it through, otherwise it is just chrun17:40
*** gyee has joined #openstack-infra17:40
Mithrandirjeblair: well, vga connector is large17:40
clarkbjeblair: on the flip side their machines are art17:40
mtreinishclarkb: what about max(2, numcpus-2)?17:41
*** gokrokve_ has quit IRC17:41
clarkbmtreinish: will that work on single cpu or two cpu machines?17:41
mtreinishI guess there wouldn't be a real difference there17:41
*** lcostantino has quit IRC17:41
*** sweston has joined #openstack-infra17:41
jeblairclarkb: "instead of the presentation i was going to give, let's admire my computer for 40 minutes"17:41
mordredif I tweet the following:17:41
mtreinishI guess not but do people actually run with parallel in those cases?17:41
clarkbjeblair: exactly17:41
clarkbmtreinish: they do if running devstack gate17:42
mordredAt Linux events, MAYBE someone should consider using Linux for the presentation laptops. #yolo17:42
mordreddo you think the #yolo tag is too much?17:42
mtreinishclarkb: well it depends on the env variables. There are still a few serial tempest jobs17:42
mtreinisherr or maybe only 217:42
clarkbmordred: you only live once badump tssh17:42
mordredyeah. that was how I was reading it17:43
*** prad__ has joined #openstack-infra17:44
openstackgerritArx Cruz proposed a change to openstack-infra/devstack-gate: Altering the > to -gt to check number of processors
Mithrandirzaro,clarkb : btw17:45
Mithrandirfor the "don't overwrite job descs"17:45
MithrandirI think that's a massively simpler fix17:45
*** prad_ has quit IRC17:45
Mithrandiroh, yeah, tests/yamlparser/fixtures/complete001.xml needs updating for a new default string too17:45
mtreinishclarkb: yeah, you're probably right to do max(1, numprocs-2) though because I doubt anyone would ever know to use those env variables17:47
clarkbMithrandir: ++ that is a small diff, fwiw I am fine deferring on this one17:47
mtreinishclarkb: or just manually set the concurrency to 1 in those cases17:48
Mithrandirmine's very lightly tested, but I think that's so much simpler than the other hole that's being dug17:48
clarkbfungi: by the way I think I did the initial group seed for all of the new projects we added on Friday17:49
clarkbfungi: if any other bugs pop up to add users, double check but it should be done17:49
mtreinishjeblair: did you see this?:
fungiclarkb: awesome--thanks17:50
mtreinishI checked salv-orlando was correct the duplicate entries in the layout are just making 2 links to the same job17:50
clarkbfungi: at the time there were no bugs open, but pretty clear indications of who the group members should be so I went ahead and did it17:50
*** morganfainberg_Z is now known as morganfainberg17:50
fungiclarkb: works for me17:51
*** VijayTripathi has quit IRC17:51
clarkbmtreinish: interesting17:51
*** prad__ has quit IRC17:51
SlickNikSergeyLukjanov / mordred / fungi: Thanks for the catch on
clarkbmordred: did you see my comment about ?17:54
SlickNikThe trove services were running in the gate correctly since the individual services are called out in the ENABLED_SERVICES list.17:54
*** lcostantino has joined #openstack-infra17:54
*** prad_ has joined #openstack-infra17:55
*** khyati_ has joined #openstack-infra17:55
*** _nadya_ has joined #openstack-infra17:55
SlickNikSergeyLukjanov / mordred / fungi:
*** signed8bit has joined #openstack-infra17:55
*** amcrn has joined #openstack-infra17:55
*** prad_ has quit IRC17:55
jeblairmtreinish: no, thanks17:55
SlickNikSergeyLukjanov / mordred / fungi: Looks like the incorrect concatenation of "ceilometer-anotificationtrove" affected ceilometer tests though, and so gordon found the issue.17:56
anteayazaro: is there any ability to set WIP from the CLi with gerrit 2.8?17:56
*** saju_m has joined #openstack-infra17:57
zaroanteaya: if we use a vote for WIP then for sure it's doable thru CLI.17:58
anteayaif I create an initial patchset and want to mark it WIP, where does the vote come in?17:59
zaroanteaya: you would use the ssh command to set 'workflow' label to -118:00
*** bhuvan_ has joined #openstack-infra18:00
* anteaya looks up the workflow label18:00
openstackgerritBen Nemec proposed a change to openstack-dev/pbr: Add support for nested requirements files
*** yolanda has joined #openstack-infra18:01
clarkbjeblair: fungi other irc ops, do we see it as a problem that openstackgerrit has founder in #satori or should we ignore that for nwo and let the autmation clean it up?18:02
zaroanteaya: that's a custom label that we will define for WIP, it's not in gerrit docs.18:02
*** luisg has joined #openstack-infra18:02
jeblairclarkb: the automation will clean it up18:02
SergeyLukjanovjeblair, +118:02
*** luisg has quit IRC18:02
zaroanteaya: labels are referred to as votes on gerit UI18:02
*** bhuvan has quit IRC18:03
clarkbok, I will approve that change then bump the irc automation reviews to the top of my list18:03
jeblairclarkb: cool18:03
zaroMithrandir: how about push your solution to review.o.o?18:03
jeblairclarkb: oh, it's not going to clean it up unless the channel is added to the channel list18:03
jeblairfor access bot18:03
anteayazaro: browsing so far I can't see labels/votes or workflow18:04
*** derekh has quit IRC18:04
clarkbjeblair: are we worried about channels granting too many permissions?18:04
jeblairclarkb: er, what i mean is that there is now an additional step for new  channels18:05
*** dripton is now known as dripton_afk18:05
jeblairclarkb: the accessbot has its own channel config file, so all new channels need to be added to it18:05
clarkbjeblair: oh right, you mentioned it needed to be double accounted until we had something better18:06
jeblairclarkb: that should be considered a review criteria for irc related changes right now18:06
jeblairclarkb: yeah; i kinda want to wait to see what Kiall reports back on that new bot...18:06
clarkbok, I will -1 and leave a comment that it iwll need to be double accounted18:06
clarkbthe IRC stuff did merge then? I know I reviewed some of it but not all of it18:06
*** VijayTripathi has joined #openstack-infra18:06
jeblairclarkb: because one of the solutions to that problem is "use a single bot with multiple plugins for everything" which can obviously read a single config file18:07
*** jcoufal has quit IRC18:07
jeblairclarkb: if we don't do that, then it becomes "try to make a single config file that our multiple bots can read" or something like that18:07
*** dripton_afk is now known as dripton18:07
jeblairor one file that gets parsed out into multiple ones, etc.18:07
pabelangerjeblair, fungi: should jobs in jenkins have "Check if ZMQ events should be published for this project." enabled for ZMQ events?18:08
jeblairpabelanger: we enable it globally in the main jenkins config so they don't have to18:08
openstackgerritA change was merged to openstack-infra/devstack-gate: Don't enable savanna service for sahara
pabelangerjeblair, Ah, okay. I think that is part of my issue too18:09
jeblairpabelanger: we're going to try to move that to zuul/gearman instead of zmq soon too, so that'll be a little simpler18:09
*** Sukhdev has quit IRC18:10
clarkbjeblair: is the meetbots section of that config unused?18:10
clarkbjeblair: grepping for meetbots as a key in the accessbot source doesn't return anything18:10
zaroanteaya: maybe this will also help
anteayazaro: I'm not quite envisioning why we would need to vote -1 for that, it looks like we can makeup our own labels18:11
jeblairclarkb: no, meetbots get +O18:11
anteayaso I would be inclined to do a wip label with a 0 vote18:11
Mithrandirzaro: doing so now18:11
* anteaya clicks18:11
Mithrandirzaro: after I added some more code to make it more likely to actually work. :-P18:12
pabelangerjeblair, okay, cool. Sorry for all the questions, still learning the ropes on nodepool :)18:12
jeblairpabelanger: no problem, the documentation is terrible.  :)18:12
clarkbanteaya: 0 is the default, we don't want WIP by default18:12
jeblairpabelanger: half of it is still in review.  :)18:12
openstackgerritTollef Fog Heen proposed a change to openstack-infra/jenkins-job-builder: Make it possible to override the "managed by" tag
clarkbWIP is a thing you must explicitly change your state into18:12
openstackgerritA change was merged to openstack-infra/storyboard: DB api refactoring
openstackgerritA change was merged to openstack-infra/storyboard: Fix 009 migration
Mithrandirzaro: ^18:13
anteayaclarkb: I was going with the believe that using the label would convey that, if we made up our own wip label18:13
anteayabecause apparently we can make up our won18:14
clarkbanteaya: right we can make up our own labels18:14
*** dstanek is now known as dstanek_afk18:14
clarkbanteaya: approved and verified are non standard labels18:14
clarkbbut each label comes with a default state: 018:14
pabelangerjeblair, ya, going to review it too18:14
clarkbso adding a different label doesn't solve the problem of still needing to vote to change the state18:14
anteayathe vote changes it from the default state to register the change in the label18:15
clarkbyes, voting workflow -1 says make this thing WIP18:15
clarkbyou could have a WIP label but you would still have to vote on it to make the state change18:15
anteayaah okay18:15
*** dizquierdo has quit IRC18:16
anteayathe fact that stream events is under permissions is going to be a change18:17
anteayamore 3rd party fun18:17
clarkbjeblair: I see there is a meetbots mask but not sure how that maps to the per channel config yet18:18
clarkbanteaya: we will probably just open it to registered users to preserve access to the feature18:18
zaroanteaya: yes, my change to 70818 documents that.18:18
jeblairclarkb: it's not a mask, it's an acl18:19
* zaro afk for a few hrs.18:19
jeblairclarkb: the acl lists are masks only if they are used with "mask:" in the channel config, otherwise they are absolute values18:20
jeblairclarkb: so operators get _exactly_ "+Afortv", and meetbots get exactly "+O"18:20
*** mrodden has joined #openstack-infra18:20
clarkbjeblair: roger, and I assume open_stack is the actually registered name for the openstack meetbot?18:20
* clarkb pokes nickserv to find out18:21
jeblairclarkb: but everybody else gets masked to "+AVOfortv" because that's the value of channel_op_mask which is set as the global mask18:21
jeblairclarkb: yes (open_stack)18:21
*** dripton is now known as dripton_afk18:21
openstackgerritA change was merged to openstack/requirements: Remove oslo.sphinx from global requirements
clarkbyup confirmed with nickserv thanks18:22
anteayaclarkb: cool18:23
anteayazaro: k, going to read through 7081818:23
*** thuc has quit IRC18:23
*** prad_ has joined #openstack-infra18:23
* Mithrandir sighs at python2.618:25
*** jgallard has quit IRC18:25
*** mriedem has joined #openstack-infra18:26
*** pcrews has quit IRC18:27
* clarkb completely changes the subject. Assuming we get working glance in our clouds and stop using hpcloud 1.0, we could do the crazy thing of switching to centos 7 beta for testing icehouse. But as I bring it up it seems like a silly thing to do18:28
clarkbMithrandir's comment sort of triggered the idea since we can drop python 26 when rhel7 happens18:28
clarkbbut it hasn't happened yet18:28
jeblairclarkb: we can drop python2.6 when we stop supporting the last version of openstack that was released with python2.6 support18:29
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1297992
uvirtbotLaunchpad bug 1297992 in nova "deleting an interface hits a race condition and fails" [Undecided,New]
jeblairthat's not exactly when rhel7 is released...18:29
clarkbjeblair: well, I meant for icehouse18:30
*** yolanda has quit IRC18:30
clarkbolder versions will continue to be tested against python 2.618:30
*** thuc has joined #openstack-infra18:30
jeblairclarkb: i'm pretty sure icehouse will be released with support for python 2.618:30
clarkbjeblair: me too18:30
*** e0ne has joined #openstack-infra18:30
*** pcrews has joined #openstack-infra18:30
jeblairbut hopefully we can remove it in juno18:31
clarkbsince rhel7 doesn't appear to be happening by april18:31
*** prad_ has quit IRC18:31
openstackgerritA change was merged to openstack-infra/devstack-gate: fix trove merge
openstackgerritA change was merged to openstack-infra/devstack-gate: Use OVERRIDE_ZUUL_BRANCH in
fungialso, as discussed not long ago, our python 3.3 jobs should likely get replaced by python 3.4 jobs since that'll be the default py3k implementation in ubuntu 14.04 lts18:32
fungiright now we don't have anything which is part of the integrated release supporting 3.3 afaik, so ought to be able to do that without worrying about deprecation18:33
jeblairfungi: ++18:33
clarkbfungi: the only possible gotcha is whether or not we want to support clients on 3.3, but considering ubuntu lts will be 3.4 I am not worried much about 3.318:33
*** jpich has quit IRC18:33
*** prad_ has joined #openstack-infra18:34
fungiclarkb: i also think the delta between 3.3 and 3.4 is smallish enough that we'd be unlikely to merge many changes which work with 3.4 and not 3.3 (though no doubt there may be some)18:34
clarkbfungi: agreed, I don';t think it is like the difference between 3.2 and 3.318:35
dhellmannhas anyone else reported issues with missing /opt/stack/new/devstack on the neutron-large-ops job?
jeblairmtreinish, sdague: there is indeed a problem with the 'duplicate job' feature in zuul; we _do_ run multiple jobs, but zuul is only considering one of them when it reports, or even decides whether the change has failed18:36
*** dstanek_afk has quit IRC18:36
jeblairmtreinish, sdague: fixing that is non-trivial and makes me a little concerned about the complexity needed to do it correctly...18:37
openstackgerritTollef Fog Heen proposed a change to openstack-infra/jenkins-job-builder: Make it possible to override the "managed by" tag
*** _sweston_ has joined #openstack-infra18:37
jeblairmtreinish, sdague: how bad would it be to say that in this case, we need to instantiate the job multiple times in jjb?  (so neutron-pg-1, neutron-pg-2, etc?)18:37
jeblairjogo: ^18:37
*** sweston has quit IRC18:38
*** prad_ has quit IRC18:38
jeblairthat might make analysis a little harder (like in er, etc)18:38
anteayadhellmann: I have not seen that error before, no18:38
clarkbjeblair: it may make some of the rules a little more verbose, but hopefully not terribly so. and creating those jobs should be simple in JJB18:38
dhellmannthanks, anteaya, I'm not finding it in the list of rechecks18:39
clarkbjeblair: I think it is a decent plan of attack18:39
anteayadhellmann: you are the first, congratulations18:39
fungidhellmann: the workspace setup log for that job looks heinous too18:39
* dhellmann doesn't feel lucky18:39
jeblair2014-03-26 17:02:46.573 | error: Failed to connect to 2001:4800:7813:516:3bc3:d7f6:ff04:aacb: Network is unreachable while accessing
mtreinishjeblair: I'm fine with using different jobs names18:39
mtreinishI'll write up a patch to do that now18:40
clarkbsdague: mtreinish mriedem btw merged on the 13th, we can merge a change to e-r to use all of the new goodness on Friday without data loss18:40
dhellmannjeblair: ah, I see that now18:40
fungijeblair: looking at the console log there was no ipv6 global address/default route reported either when the job started18:40
anteayacould this be more git.o.o derp like I experienced earlier?18:40
fungijeblair: which would be correct for hpcloud18:40
clarkbfungi: I think this is that supposed git bug which I haven't been able to properly track down18:41
fungiso that's got to be that git fallback behavior18:41
jogojeblair: if we have only a few copies of a job it won't be so bad18:41
fungiclarkb: yeah18:41
clarkbfungi: lifeless and spamaps see it occasionally too18:41
jogobut if we have a lot it may be confusing18:41
fungiwhich suggests that maybe it had trouble reaching the ipv4 address it looked up for git.o.o first (or failed to look one up?)18:41
clarkbsdague: mtreinish mriedem I think I will take a stab at using the short_uuid today18:43
clarkbI feel like I need to write some code and that will be a good outlet18:43
jeblairdtroyer: oh hey, are you going to propose python-openstackclient as an official program?18:43
mriedemclarkb: ok, go nuts :)18:44
mriedemi've been in a class all morning so have no idea what's going on today18:44
lifelessfungi: I like your NS issue theory18:44
dhellmannfungi: should I open a bug against openstack-ci? I need to reverify my changeset anyway, but it's not clear this is a "bug" per se18:44
fungidhellmann: sure, we can just mark it incomplete until someone is able to put together a more reproducible test case18:45
mordredjeblair: I would so much prefer if dtroyer made python-openstackclient a thing18:45
dtroyerjeblair: I thought the SDK might have been further along by now, but maybe we should start with OSC?  I think ultimately the client-side stuff should be close, if not together.18:45
mordredthere's also the python-openstacksdk thing - perhaps teh two of them together are a thing?18:45
mordreddtroyer: jinx18:46
dtroyerso is me being the ptl of DevStack an issue to doing an OSC program?18:47
mordreddtroyer: I think python-openstackclient is something the project desperately needs - and I kinda thought most of the folks in the project agreed18:47
lifelessdtroyer: do you have the bandwidth, do you mean?18:47
mordreddtroyer: I don't think so - we have no rules about single PTL-ness18:47
jeblairi don't think we have limits on the number of ptl seats one can hold18:47
dtroyerno, is there something about not having one person wear more than one ptl hat?18:47
uvirtbotLaunchpad bug 1298006 in openstack-ci "network sometimes unreachable accessing our git server" [Undecided,New]18:48
fungidhellmann: thanks18:48
jeblairclarkb, mordred: should we open a ticket with hpcloud about the large number of servers that took more than 1 hour to launch?18:50
jeblairclarkb, mordred: or should we merge the nodepool logging change, restart it, bump up our regionb usage and get more data?18:50
jeblairand then open a ticket :)18:50
mordredwe might want to do the second thing18:50
anteayadtroyer: nothing in the election rules I have seen18:51
fungii'm fine with making a second go of it18:51
dhellmannmordred: heads up, I'm going to do another pbr release when lands18:51
fungijeblair: you did see the interesting miscalculation nodepool seems to have made after we switched back (in scrollback from before you were awake)?18:51
mordreddhellmann: ok18:51
jeblairfungi: not in detail18:52
*** zhiyan is now known as zhiyan_18:52
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Change names for duplicate neutron jobs
fungijeblair: current theory is that last night when we switched to region b and ended up starved for py3k nodes because nodepool kept asking region b to make some and then they'd eventually accumulate in building state for too long, the pileup of waiting pypy/python33 jobs caused nodepool to think it needed many more of that node type to service them when we switched back18:53
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Change names for duplicate neutron jobs
mtreinishjeblair: ^^^ I think that should do it correctly18:54
clarkbmordred: jeblair ++ to the second thing18:54
fungijeblair: so it made *a lot* of precise-py3k nodes, which were then hanging around in a ready state when i woke up this morning, and weren't going through them nearly fast enough to turn them over into other node types we actually needed18:54
*** jamielennox is now known as jamielennox|away18:54
jeblairfungi: ah.18:56
fungijeblair: so working as designed, but something we should watch out for18:57
fungijeblair: and maybe a wishlist feature for nodepool is to delete ready nodes it realizes it no longer needs so many of, so it can recycle their slots for node types which are actually in demand18:57
*** johnthetubaguy has quit IRC19:00
openstackgerritA change was merged to openstack-infra/devstack-gate: be extra explicit for people reading logs
fungithough the other problem, dealing with rare node starvation due to node builds taking too long (and building state counting toward the available number for a node type), is something i don't currently have good ideas for how we'd address19:01
clarkbmtreinish: mriedem sdague are the bug-number.json files currently in use by the unittests? any idea how those are generated so that I can update them with current field data?19:02
clarkbah there they are in the tests so they are used19:03
mtreinishclarkb: one of the functional tests is supposed to read them in19:03
clarkbI know I can write a small python script that loads each one and adds the appropriate fields19:03
mtreinishbut I think it's broken19:03
*** thuc has quit IRC19:05
*** thedodd has quit IRC19:05
clarkbmtreinish: I will do my best to make sure the json is correct then try to figure out if those tests need help19:05
mordredfungi: I was thinking this morning - maybe we don't need the precise3k nodes anymore - or the system-wide python3 stuff installed19:05
mordredsince we made the tox.ini changes to run things with teh python in the virtualenv19:06
mordredwe may  have solved the underlying problem differently19:06
fungimordred: well, we still need the interpreter19:06
clarkbmtreinish: actually that json is really really old19:06
clarkbmtreinish: we should maybe redo that completely and use current data with current bugs /me ignores it for now19:06
fungimordred: i think when trusty comes around, then yes19:07
mordredfungi: yah - but I thnk ALL we need is the interpreter19:07
fungimordred: but right now the py3k-precise nodes are preinstalled with ppa-provided python 3.3 and pypy interpreter backports19:07
*** ominakov has quit IRC19:07
fungimordred: though yes, i agree, we drop py3k-precise as a node type and use bare-trusty to run those as soon as we can19:08
mordredfungi: sure - but I think there was something else we did on them because of the old tox bug that made doign them side by side with other nodes tricky19:08
*** markmcclain has joined #openstack-infra19:08
*** mgagne1 is now known as mgagne19:09
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Change names for duplicate neutron jobs
fungimordred: right, we were doing puppet gyrations to get the pip provider to install tox and virtualenv into the module search path for python3 but since newer versions are bundling and our entry point for tox should work when called under python2.7 to run py33 (or py34) and pypy tox envs, shouldn't matter any longer19:09
fungimordred: something we certainly want to test to make sure, but we'll be doing that as part of teh trusty transition anyway19:10
mordred++ just wanted to make sure I wasn't missing something19:11
fungiso makes the most sense to stop using a separate node type for those jobs at that point19:11
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Revert "Allow multiple invocations of the same job"
jeblairmtreinish, sdague, jogo: ^19:14
*** wchrisj has joined #openstack-infra19:14
openstackgerritClark Boylan proposed a change to openstack-infra/elastic-recheck: Take advantage of the new build_short_uuid field.
*** malini has joined #openstack-infra19:17
clarkbjogo: mtreinish sdague ^ I haven't actually tested that, but was hoping for feedback on the variable renames for consistency and more discussion about the test json data19:17
marun < ready for review ;)19:20
marunjob is experimental only so I can test it with19:20
*** jp_at_hp has quit IRC19:20
clarkbjeblair: fungi fyi failed pep819:21
jeblairclarkb: do you inderstand what happened there?19:22
openstackgerritAnita Kuno proposed a change to openstack-infra/meetbot: Prevents meetbot from starting meetings in non-meeting channels
clarkbjeblair: no, looking at the log file it almost apepars like we used a different pep8 or something19:22
fungiclarkb: yeah, it's complaining about style differences in files which weren't touched by that change19:23
jeblairclarkb: a new pep8 was released today, but i thought it comes from pep819:23
jeblairer flake819:23
jeblairdoes flake8 not actually pin the pep8 version?19:23
openstackgerritA change was merged to openstack-infra/zuul: Revert "Allow multiple invocations of the same job"
clarkbjeblair: it may not, I am working to reproduce locally really quickly19:24
clarkbjeblair: we pin flake8 to 2.019:24
clarkbbut if flake8 doesn't pin pep8 then that may be the issue19:24
clarkbyup reproduced with tox -repep819:25
jeblair        "pep8 >= 1.4.3",19:25
jeblairis in of flake8 2.019:25
clarkbpep8==1.5.0 was used according to pip freeze19:26
fungithat's today's release19:26
jeblairso basically, once again, all of the style checks for all of the projects just now changed all at once because pep8 did a release?19:26
*** krotscheck has quit IRC19:26
clarkbjeblair: no, all of the other projects use hacking19:27
anteayamordred: can you take a look at please?19:27
*** markwash has quit IRC19:27
clarkband hacking pins flake8 and pep819:27
*** thuc has joined #openstack-infra19:27
anteayamordred: seems gerrit is happier with an --empty-commit flag when creating empty repos19:27
jeblairclarkb: oh, it pins flake8 _and_ pep819:27
jeblairclarkb: i thought it was transitive19:27
clarkbnodepool does not use hacking and thus ran into this, I will add an explicit pep8 pin to nodepool19:27
dstufftyou mean you don't want your style to change on someone elses whim?19:27
fungilooks that way...
*** thuc_ has joined #openstack-infra19:28
jeblair(someone told me that and i believed them but i shouldn't have)19:28
fungipins pyflakes too, presumably for similar reasons19:28
clarkboh right, I should pin pyflakes in nodepool too19:28
*** pcrews has quit IRC19:29
sdagueclarkb: sorry, catching up on scrollback. The json files were just things I snagged over time. I don't know if I've got the generator for them (probably should)19:30
*** SumitNaiksatam has quit IRC19:31
*** thuc has quit IRC19:32
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Pin flake8 deps so they don't change on us.
clarkbjeblair: fungi ^19:32
fungiclarkb: any reason not to pin to the same revs used by hacking?19:33
sdaguejeblair: so the n / 2 vs. n - 2 thing, I don't know that we have any data yet on 6 way tempest vs. failure results19:34
*** andreaf has quit IRC19:34
fungiclarkb: not saying we necessarily should, just wondering if pinning to the current latest working vs what hacking already settled on makes more/less sense19:34
sdagueI actually think before we do another change like that we should try to get the "master queue" running so we can run some numbers against master regularly and see19:34
jeblairfungi, clarkb: i think pinning to hacking versions makes sense19:36
clarkbwe don't use hacking19:36
jeblairclarkb: yes, but i think it's good to be in sync with the pep8 and pyflakes versions used in openstack19:36
clarkbok, I can update the change19:36
jeblairclarkb: (that's why i'm also okay with depending on hacking but ignore H errors, but apparently that confuses people)19:37
*** bhuvan_ has quit IRC19:37
*** pcrews has joined #openstack-infra19:38
*** bhuvan has joined #openstack-infra19:39
*** bhuvan has joined #openstack-infra19:39
openstackgerritClark Boylan proposed a change to openstack-infra/nodepool: Depend on hacking for its dependencies.
clarkbfungi: jeblair ^ that includes a fix for the one line that hacking's deps failed on19:40
jeblairsdague: btw, heads up, when hacking updates to pep8 1.5.0, you and i are going to want to ignore E129 everywhere:
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Add specs-cookiecutter to openstack-dev
jeblairclarkb: what rule was that?19:41
clarkbjeblair: nodepool/ E122 continuation line missing indentation or outdented19:42
*** dripton_afk is now known as dripton19:42
fungii do agree with that additional indentation, fwiw. it's inside deeper nested parens than the preceding lines so helps readability19:42
jogojeblair: we pin flake819:42
harlowjahey guys, qq, there was talk about moving away from the blueprint model, but i forgot what the suggestions where to move to instead (correct me if i am wrong)19:42
*** SumitNaiksatam has joined #openstack-infra19:42
*** sabari has quit IRC19:43
jeblairjogo: yeah, and pep8 too. i'm caught up now19:43
dstufftanyone mind me bothering them openstack's CI system for a few? :]19:43
jeblairharlowja: not so much the model as implementation19:43
jogojeblair: yeah just saw finished reading the backlog19:43
*** yolanda has joined #openstack-infra19:43
harlowjajeblair ah, a different impl?19:43
*** freyes has quit IRC19:43
clarkbdstufft: hrm? I am happy to answer questions if that is what you mean19:43
jeblairharlowja: some projects are trying out the idea of writing blueprints in a git repo and putting them through code review in gerrit (nova-specs and qa-specs)19:44
fungiharlowja: being more explicit about where and how to do the design documentation19:44
jeblairharlowja: but they still have blueprints in launchpad for reference and tracking purposes19:44
harlowjahmmm, right, thought i remembered some talk of moving to something entirely different19:44
*** freyes has joined #openstack-infra19:44
jeblairharlowja: take a look at the README's in those repos and you'll get a better idea19:44
jeblairharlowja: oh, you may be thinking of storyboard19:44
dstufftclarkb: I'm thinking about pip/virtualenv/setuptools, and making a more expansive CI and testing infra for it (Right now it just uses travis ci) I'm trying to figure out what zuul does ontop of jenkins? :/19:45
harlowjahmmm, could be19:45
jeblairharlowja: it's a unified bug/feature tracker we're developing19:45
harlowjaah, ya, that could be it19:45
jeblairharlowja: super early days, but it's in CD at
jeblairharlowja: git repo is openstack-infra/storyboard19:46
fungidstufft: the documentation at and presentation at are great introductions19:46
jeblair(and storyboard-webclient)19:46
clarkbdstufft: two major things. The first is straightforward, zuul schedules jobs on gearman queues so you can have arbitrary job runners, including jenkins (this allows us to do multimaster jenkins)19:46
clarkbthe second is a bit more complicated and is it allows us to gate19:47
dstufftclarkb: would it be the thing that lets me run pip tests if setuptools makes a change?19:48
jeblairdstufft: also, i think may be a topic at the language summit, and i'll be around pycon for chatting, etc19:48
* dstufft won't be at PyCon19:48
sdaguejeblair: ah the E125 wars :)19:48
clarkbdstufft: yes, if you create a shared integration test in a gate pipeline for setuptools and pip you can have them test against each other when changes are made to them19:48
jeblairdstufft: that's terrible, i don't see the point of even having a pycon without you!19:49
pabelangerjeblair, how did you enable zmq events globally in jenkins? I don't see an option to do so in manage jenkins.19:49
clarkbpabelanger: its in your global jenkins config if you have the zmq event publisher plugin installed19:50
clarkbpabelanger: there is a setting for the port to publish on and a check box to do it for all jobs19:50
pabelangerclarkb, I see host, port and enable gearman. All are filled in a test connection works.  Is that all that is needed?19:51
SpamapSclarkb: FYI I have not seen ipv6 fail for git in a long time19:51
pabelangerthat is gearman plugin19:51
pabelangerI see it19:51
dstufftclarkb: more or less I want a few things 1) I want to run the test suites of dependent projects (pip depends on setuptools and distlib and wheel atm) 2) I want to do this whenever someoone makes a PR on Github and report status 3) I want to be able to test on "wierd" OSes (Windows, AIX, etc)19:51
dstufftjeblair: :]19:52
*** dstanek_afk has joined #openstack-infra19:52
clarkbdstufft: zuul can't do that today but only because it does't speak github. If it was made to udnerstand github it could do all of that19:52
jeblairi would accept a patch for that19:52
*** hashar has joined #openstack-infra19:52
*** dstanek_afk is now known as dstanek19:53
dstufftI could write a patch for that19:53
dstufftwould zuul be a better thing to look at than just using jenkins or buildbot or something?19:54
*** skraynev_afk is now known as skraynev19:54
dstufftNot going to lie, my 'setup a CI" experience ths far is getting mad at jenkins and buildbot and using travis-ci19:54
clarkbdstufft: for that I think you need to decide if you need gating or if you need a better scaling story than jenkins or travis allow19:54
clarkb(I haven't used buildbot so am not super familiar with its ability to talk to a hunddred slaves)19:55
dstufftclarkb: gating just means checking a patch prior to it landing right? or is it more involved?19:56
clarkbdstufft: that is the key bit but it is a bit more involved. zuul does all of that checking in parallel by building the desired end state and testing that for each change instead of waiting for things to go through one at a time19:56
clarkbdstufft: so say setuptools and pip have 2 PR each. and the code reviews on github give the go ahead to gate at about the same time. Zuul will construct a tree for that scenario (in the actual order it received the events) and test each PR in that scenario19:57
clarkbdstufft: for example setuptoolsA <- pipB <- pipC <- setuptoolsD. That last PR (setuptoolsD) will be tested with A, B, C as well19:58
fungimainly because one pr might contain changes which don't merge-conflict with the other but manage to break functionality of the program when combined19:58
*** bhuvan has quit IRC19:58
clarkbdstufft: and if B fails for some reason C and D will have their tests restarted to exclude B which means A, B get tested, and A, C, D get tested19:58
*** jergerber has joined #openstack-infra19:58
openstackgerritA change was merged to openstack-infra/nodepool: Depend on hacking for its dependencies.
fungidstufft: if you look at you can see it in action (currently integration-testing changes for python-keystoneclient and swift together)20:00
*** khyati_ has quit IRC20:00
*** mrodden has quit IRC20:01
*** mrda_away is now known as mrda20:01
dstufftclarkb: so if we pretend for a minute that pip/setuptools/etc don't get near as many PRs at one time as openstack does (having more than one at a time is rare tbh) probably that bit isn't super important I'd guess?20:01
mordreddstufft: if you wrote the github trigger for zuul, then we could do the "Run openstack tests on changes to setuptools"20:01
* mordred has been meaning to do that for about 6 months20:02
*** sabari has joined #openstack-infra20:02
dstufftmordred: oh that'd be cool20:03
clarkbdstufft: ya it probably isn't super important20:03
*** sabari has left #openstack-infra20:03
*** julim_ has quit IRC20:03
dstufftthat's another thing i'd like to figure out how to do TBH, is run tests against popular projects to make sure things still install (to prevent stuff like setuptools deprecating Feature to be a big surprise that lots of people ended up using it)20:04
*** bhuvan has joined #openstack-infra20:04
dstufftah, I have to run to pick up my daughter from the bus stop, be back in a bit :) Thanks y'all :)20:05
clarkbdstufft: the other thing that comes with gating is the actual tests must pass before code merges rules which you may not get out of other tools either without writing code20:06
clarkbgating would still be useful with less PR volume in that way20:06
*** e0ne has quit IRC20:07
*** ildikov_ has quit IRC20:08
*** thedodd has joined #openstack-infra20:08
openstackgerritJoe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1291677
*** bauzas has joined #openstack-infra20:10
uvirtbotLaunchpad bug 1291677 in nova "delete interface times out in check-tempest-dsvm-neutron" [Undecided,New]
mordredyup. and as I understand it, github has an api command for "merge this PR" - so you could almost certainly write drivers for most of the peces of zuul20:10
clarkbjogo: can you do a quick braindump for me on how you run e-r locally? I can do that locally then write docs for it20:11
clarkbjogo: mostly wondering if you run the bot with it joining a private channel?20:11
openstackgerritMatthew Treinish proposed a change to openstack-infra/config: Add specs-cookiecutter to openstack-dev
clarkbjogo: or is there some way to have it report to the console and not do irc?20:11
jogoclarkb: I like how build_short_uuid cauased so many changes20:11
*** saju_m has quit IRC20:11
clarkbjogo: yup we had two different names across tools...20:12
jogoclarkb: so we have a mode to run it locally without commenting anywhere20:12
clarkbjogo: unfortunately that didn't get caught in review of the lgostash/es change20:12
jogoclarkb: tox -erun20:12
clarkbjogo: does that require I copy the sample config and point it at valid gerrit keys?20:12
jogoyou just need to createa elasticRecheck.conf file20:12
clarkbroger thanks20:13
jogoclarkb: yup20:13
mtreinishmordred: I'm not sure we want to add this yet, but I figured I'd throw a review up for it anyway20:13
jogoclarkb: also control-C doesn't kill it for some reason20:13
jogoso I usually just do a 'kill'20:13
*** marktraceur is now known as ba20:13
clarkbjogo: probably a non daemon thread hanging around20:14
jeblairjogo: ctrl-\ should do it in those situations20:14
jogojeblair: oh neato didn't know that one20:14
jeblair(though making threads daemon is probably a good idea)20:14
*** ba is now known as marktraceur20:14
jogoclarkb: your testing out ?20:15
clarkbjogo: yup20:15
*** yassine_ has quit IRC20:18
*** krotscheck has joined #openstack-infra20:18
*** malini is now known as malini_afk20:20
clarkbjogo: woot its running, now I wait20:22
fungi"HEAD is now at e2fff8c Set paramiko version > 1.9.0"
*** yassine has quit IRC20:23
fungiexcept HEAD was already something else a good 15 minutes prior to that20:23
fungii wonder whether we've got replication lagging again?20:23
pabelangerclarkb, when do zmq events get published for a job?  Once it is created in jenkins?\20:24
fungithat run also doesn't seem to have used the pin change20:24
pabelangeror only when jenkins launches the job20:24
fungieven though in theory it shouls have been included in the zuul ref20:24
clarkbpabelanger: on job start, job completion, and job finalization20:24
clarkbpabelanger: the socket won't be opened until the first job runs so you may lose the onStarted event for the first job but all other events shoudl show up20:25
*** ominakov has joined #openstack-infra20:26
clarkbfungi: why would it have been included in the zuul ref?20:26
fungiclarkb: zuul should have based the ref on the state of the repo in gerrit at the time the ref was being built20:26
clarkbfungi: it looked like jeblkair reran the check job before the other change merged so the zuul ref would not have included it20:26
*** lcostantino has quit IRC20:26
clarkbI think zuul did what is expected in this case20:26
jeblairoh right, i wanted to enqueue it in gate but i can't do that anymore.  :(20:27
clarkbif not what is desired20:27
fungiclarkb: ahh, yep20:27
fungiclarkb: i misread the report time from jenkins as the start time not the end time20:27
pabelangerclarkb, thanks20:27
clarkbpabelanger: no problem. if that bothers you, we can probably update the plugin to start the pub thread when jenkins starts20:28
clarkbpabelanger: but I went the cheap route and simply check the config each time an event is ready and deal with ti then20:28
clarkbthere are only about 100k events per day across all of our masters on a busy day which isn't many20:29
*** saju_m has joined #openstack-infra20:29
*** _nadya_ has quit IRC20:30
pabelangerclarkb, Ya, trying to see if some event could be triggered when jenkins created the job, so zuul knows of it.20:30
pabelangerso, was trying to better understand what the plugin did20:30
clarkbpabelanger: you when the job is created, hrm. You could probably extend the zmq event publisher to do that20:32
clarkbpabelanger: you would have to set up a second jenkins event listener for config changes, then pass the same queue the publisher thread pops events off of to that listener20:32
clarkbbut it doesn't do that today20:32
pabelangerI'm just thinking about ways, but will wait to see how jeblair handles it20:33
*** jeckersb is now known as jeckersb_gone20:34
*** david-lyle has quit IRC20:34
*** ildikov_ has joined #openstack-infra20:36
clarkbjeblair: what was the problem with storyboard db tests using your fixture?20:36
*** saju_m has quit IRC20:37
lifelessclarkb: did you see jeblair's email about it ?20:38
*** afazekas has joined #openstack-infra20:38
clarkblifeless: I did, but yesterday he said he found the reason the fixture wasn't workign and a patch was incoming, reading his change in that email the particular bug fix doesn't pop out at me20:39
jeblairclarkb: hrm, i may have said a lot of things yesterday, some of them might even have been correct...  :)20:40
jeblairclarkb: at any rate, i believe the current state is that the patch works as intended as far as mysql isolation for parallel testr20:40
clarkbjeblair: awesome20:41
*** _nadya_ has joined #openstack-infra20:41
clarkbjeblair: re the miyiasm thing, can sqlalchemy enforce an engine for us or possibly the fixture when it creates the schema?20:41
clarkbI suppose for that I should respond to the thread20:42
*** atiwari has quit IRC20:42
*** nkinder has quit IRC20:42
jeblairclarkb: but the schema is created with myisam tables on the slaves because sqlalchemy is used and the engine is not specified; alembic specifies the engine but alembic is only used in production, not for creating the schema for the normal unit tests20:42
clarkbI see. according to mysql docs with mysql 5.5 and onward innodb is the default table type20:43
jeblairthere are two obvious solutions for that; i prefer the one that mirrors production use20:43
clarkbjeblair: I think that means we are ok on our ubuntu hosts but not the centos hosts20:43
jeblairclarkb: we set it to myisam on the slaves specifically so that we catch errors like this.  :)20:43
clarkboh right20:43
clarkbin the pupept20:43
jeblairit worked20:43
clarkbjeblair: shouldn't storyboard nodepool et al do to force innodb?20:45
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Add StackTach project
*** julim has joined #openstack-infra20:45
clarkbI don't like relying on alembic to enforce that, the orm should20:45
openstackgerritA change was merged to openstack-infra/nodepool: Include provider names in timeout messages
jeblairclarkb: yes, they should do that; definitely nodepool.20:46
openstackgerritA change was merged to openstack-infra/config: Add a neutron job for api testing
jeblairclarkb: however, i still think alembic is the more important bit in storyboard because that's what's _actually_ used to create the database in production.20:46
jeblairi think ruslan is suggesting that if you have a test that proves that the result of creating the db with alembic is the same as creating it with the model, then it's okay.  i could buy that.20:48
jeblairhowever, that does start to get into another project i'm interested in: removing foreign keys from the database20:48
jeblairone way to do that is to have the sqla model define the foreign keys, but not define them in alembic.  that makes it easy to use sqla, but you don't actually end up with the constaints in the db.20:49
*** ominakov has quit IRC20:49
jeblairit's still possible to use sqla without declaring fk's in the model (nodepool does it, actually) but it's a little more verbose.20:49
jeblairmordred wants to make a patch to sqla that lets you specify fks but then tell sqlalchemy to not actually tell the db to create them20:49
jeblairthe really short version of why that is desirable is: your app needs to do the enforcement anyway, so don't waste database cpu resources on it (which are more constrained than your app cpu resources).  and besides, they usually cause more problems than they solve.20:50
jeblairbut perhaps it's possible to write the sqla == alembic schema check to disregard foreign keys20:52
ruhejeblair: re fks in storyboard. krotscheck rised a valid question in
*** signed8bit has quit IRC20:54
jeblairruhe: thanks, hadn't seen that yet20:55
*** pdmars has quit IRC20:56
*** weshay has quit IRC20:56
*** e0ne has joined #openstack-infra20:57
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Add StackTach project
sdaguejeblair: for what's it's worth, in nova we found that if you create db in 2 different ways, there are often subtle differences20:58
sdagueso I'd say use alembic20:58
*** branen has joined #openstack-infra20:58
clarkbsdague: is that re table engine or fks?20:58
sdagueI thought there were even table engine thigns20:58
sdagueit's been a while, there were real differences though20:59
*** boris-42 has quit IRC20:59
ruhesdague: but using alembic means that every developer has to have local mysql instance. i don't mind to have mysql instance for unit-tests, but do others like this idea?20:59
sdagueI'm not sure how developing a db driven app, having local mysql is an undo burden21:00
*** signed8bit has joined #openstack-infra21:00
jeblairi'm okay with that requirement; especially since we'll run tests for people if they push patches21:01
sdaguejeblair: is there no escape valve for sqlite like we have in migrate?21:01
sdaguewhere you can specify an alternate migration for a version21:01
sdaguefor sqlite21:02
sdaguewhich is needed for all the alters that drop things21:02
sdaguebecause sqlite doesn't do that21:02
*** mfer has quit IRC21:02
jeblairsdague: i don't know; i'm not very familiar with alembic (i've almost learned how to invoke it).  but that seems like the sort of thing i'd rather not do (extra work to support a db that isn't really used)21:03
sdaguejeblair: totally fair21:03
sdagueI think the only issue is having a standard way to provide creds for tox to pick up21:03
ruhesdague: in StoryBoard we now expect openstackci_test user:password21:04
sdagueruhe: sure, then you can do that21:04
sdaguejust like from nova21:04
sdaguejeblair: presumbly you could also be nice and skip the unit tests that are db related if you detect they don't have a db21:05
sdagueit would run the rest of them21:05
ruheyes, we have the same opportunistic DB migration tests - if mysql or postgres are present and have this user/password, then run the tests, otherwise skip21:05
*** aysyd has quit IRC21:05
sdagueruhe: cool, I wasn't clear how much of that copied in21:05
sdagueI'm glad that pattern remains useful :)21:06
jeblairsdague: yeah, as long as we have the "require that the db tests be run" env flag set in the gate21:06
ruhebut, if we really decide to run ALL the unit tests on real DB, then we'll have to enforce developers to have local mysql instance21:06
jeblair(but i'm also okay being mean and saying to run the tests you need mysql; i won't argue against that though)21:07
sdaguejeblair: also, IIRC, we are explicitly setting the gate mysql instances to myisam to detect when nova fails to specify engine in table definition21:07
jeblairsdague: yes, that's something i'm clarifying in my response to SpamapS now21:07
sdaguethat was all part and parcel of the opportunitistic testing21:07
*** e0ne has quit IRC21:07
sdaguewhen I put that together near the end of folsom21:08
jeblairsdague: because that's responsible for some of the behavior i described21:08
sdaguespecifying and engine is good though, if you depend on behavior of one21:08
jeblairsdague: i view it as successful -- it helped me realize that storyboard did not set the engine in its sqla model21:08
sdagueagreed :)21:08
sdaguedouble plus good21:08
*** ameade has joined #openstack-infra21:08
*** ArxCruz has quit IRC21:09
*** wchrisj has quit IRC21:11
*** skraynev is now known as skraynev_afk21:14
*** signed8bit has quit IRC21:15
openstackgerritA change was merged to openstack-dev/pbr: Remove pip version specifier
*** signed8bit has joined #openstack-infra21:15
*** boris-42 has joined #openstack-infra21:16
openstackgerritKhai Do proposed a change to openstack-infra/config: fix installation of nodepool on jenkins-dev
*** marun has left #openstack-infra21:18
*** lifeless has quit IRC21:19
*** dprince has quit IRC21:20
zaromattoliverau: afternoon21:21
*** e0ne has joined #openstack-infra21:22
*** CaptTofu has quit IRC21:24
*** CaptTofu has joined #openstack-infra21:25
*** CaptTofu has quit IRC21:26
*** CaptTofu_ has joined #openstack-infra21:26
*** dstanek has quit IRC21:26
zaroclarkb, jeblair: your feedback to would be appreciated.21:27
*** dstanek has joined #openstack-infra21:27
clarkbjogo: sdague mtreinish has been tested locally and I am fairly ocnfident it is working21:29
*** moted has joined #openstack-infra21:29
sdagueclarkb: let's do it21:29
clarkbwoot thanks21:31
*** mrodden has joined #openstack-infra21:31
clarkbgoing to push a change to the contributing file spelling out how local fucntional testing works21:31
*** markmc has quit IRC21:32
openstackgerritA change was merged to openstack-infra/elastic-recheck: Take advantage of the new build_short_uuid field.
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Prepare for deletion of stable/grizzly branches
fungithat's ^ the first step toward grizzly eol21:33
openstackjeblair: Error: "!!!!!!!!" is not a valid command.21:33
jeblairi was so happy and now i am sad21:34
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Add StackTach project
*** whoops has joined #openstack-infra21:34
fungianyway, once it's merged, i'll tag/delete the corresponding branches on all projects and then un-wip the cleanup patches i'm preparing to push up21:35
*** jamielennox|away is now known as jamielennox21:35
*** ociuhandu has quit IRC21:35
*** markmc has joined #openstack-infra21:35
* jeblair remembers how to use bzr21:35
anteayamorning mattoliverau21:36
anteayazaro so workflow label21:36
anteayazaro: since workflow label +1 == Approved21:37
anteayazaro: does this mean the workflow label can only be used by core?21:37
clarkbjeblair: hacking on python jenkins?21:37
clarkbfungi: will review shortly21:37
jeblairclarkb: yeah, reviewing zaro's change21:38
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Add StackTach project
fungianteaya: why would it mean that? we're able to set arbitrary vote ranges in acls (remember core reviewers can review vote -2..+2 but registered users only -1..+1 for example)21:38
openstackgerritClark Boylan proposed a change to openstack-infra/elastic-recheck: Document process to run elastic-recheck locally
anteayafungi: just wondering21:39
fungianteaya: so we'd set the workflow label via acl allowing change owner to set -1..0 and core reviewers -1..+121:39
*** mrodden has quit IRC21:39
anteayaif we can set the voting range in the acl so that everyone can -1 workflow and only core can +1 workflow that answers my question21:39
anteayaokay, thanks21:39
fungithat's exactly the plan, yes21:39
jeblairnot everyone can -1, only change owner and core21:39
fungi"everyone" can -1 or 0 their own change (e.g. each change owner)21:40
*** exz has quit IRC21:41
clarkbfungi: reviewed21:41
anteayaokay thanks, getting a clearer picture now21:42
SpamapSjeblair: I so agree on ORMs being a horrible abstraction layer. :)21:42
jeblairSpamapS: yeah, they're great at it as long as you take care to specify all the things that could possibly be different.  :)21:42
*** freyes has quit IRC21:43
SpamapSjeblair: right, like a concierge that speaks two languages, neither of which are the one you speak.21:43
jeblairSpamapS: Kiel mi vin helpos?21:44
clarkbzaro: for 78930 I think you captured the issues21:45
fungiclarkb: responded21:45
clarkbzaro: we should see if we can get plugin version detection going before we break backward compat with that plugin21:45
*** hashar has quit IRC21:45
jeblairi want this counted as code review in stackalytics!21:46
jeblairthis is hard!21:46
SpamapSjeblair: ich bin ein berliner21:46
clarkbfungi: thanks, responded21:47
clarkbfungi: I guess there isn't much we can do about the grenade thing, but I still thik that change should remove the unneeded jobs at the same time21:47
clarkbjeblair: ha21:48
*** sandywalsh has quit IRC21:48
*** mriedem has quit IRC21:48
fungiclarkb: i'd rather not have to figure out which jobs are safe to remove along with that change, rather than just remove them all as soon as the last grizzly changes are abandoned21:49
jeblairzaro: why did you choose to key on longName instead of shortName?21:50
clarkbfungi: so I have the opposite idea, by removing them jobs from JJB you get tests that make sure you didn't remove too much from zuul and vice versa21:50
jgriffithfungi: jeblair anything change today in how unit tests jobs are checked in the gate for cinder?21:50
clarkbfungi: basically by removing on both sides you get gate tests that make sure you didn't imbalance them21:50
clarkbjgriffith: no21:50
jgriffithfungi: jeblair we're getting check config errors21:50
jgriffithbut I can't figure out where the diff are coming from21:50
clarkbjgriffith: can you link to a failure?21:50
fungijgriffith: it's just the same old "keystoneclient changed options again, regenerate your sample configs" issue, right?21:51
jgriffithfungi: yeah, but that trick didn't work it seems21:51
clarkbyup thats the one21:51
jgriffithfungi: OH21:51
jgriffithdid that push a new client?21:51
clarkbI wish that check would diaf21:51
jgriffithI'm considering taking it away21:51
fungijgriffith: there's a whole mailing list thread going about ripping those sample configs out of everyone's git repos. i'm personally a fan of generating them when api docs are built21:52
* jgriffith sheepishly walks away21:52
fungior at least in a similar way to how api docs are built21:52
jgriffithfungi: I would agree21:52
clarkbfungi: going back to the grizzly thing, it should be safe to remove all grizzly jobs21:52
jgriffithalright... so update the keystone client and regenerate should do the trick I'm guessing21:53
jgriffithbut now everybody's hosed behind the keystone change21:53
clarkbfungi: and that is desired to avoid the wedged state you describe as a possibility21:53
jgriffithI think I'll just disable the thing now and merge it21:53
clarkbjgriffith: well I am not sure you can straight up disable it21:53
jgriffithclarkb: sure I can :)21:53
clarkbjgriffith: because things depend on that conf file (devstack)21:53
fungiclarkb: removing all grizzly-specific jobs doesn't prevent zuul from testing grizzly changes though, right? am i just overthinking it?21:54
clarkbjgriffith: and if you disable ti you will see failures elsewhere in harder to debug places21:54
jgriffith#/bin/bash exit21:54
jgriffithnot saying remove the conf file, remove the check21:54
clarkbjgriffith: I think that is inviting more trouble than its worth21:54
clarkbjgriffith: right,21:54
clarkbif that conf file doesn't have the correct data in it it will break stuff21:54
jgriffithYou're my voice of reason for the day :)21:54
clarkbwhich is why the check exists, its just poorly implemented imo21:54
jgriffithclarkb: oh yeah I get that for sure21:54
fungiit's checking in the wrong place at the wrong time, basically21:55
clarkbfungi: I think that is overthinking it, changes to grizzly will use the non special grizzly jobs21:55
jgriffithclarkb: no +2 for patches that change config, but you're right, you're right ;)21:55
zaroanteaya: no, who can vote on that label depends on how we configure gerrit permissions21:55
*** dcramer_ has quit IRC21:55
jgriffithok... well I know how to reset at least thanks to you all21:55
jgriffithappreciate it21:55
clarkbfungi: basically changes to grizzly use gate-python27 and friends and normal devstack for the most part21:55
anteayazaro: great thank you21:55
zarojeblair: are you referring to my patch for python-jenkins?21:56
anteayafungi outlined some scenarios for me about how permissions might be set21:56
jeblairzaro: yes21:56
fungiclarkb: meh. i'm still unconvinced that changing the way grizzly changes are tested before deleting teh grizzly branches is the technically correct way to do this (thinking in terms of future eol cycles which might be more tightly timed), but am willing to do so if that will get this approved faster21:57
clarkbfungi: I think we are maybe talking about two different things21:57
clarkbfungi: you are talking about how grizzly changes are tested and I am saying remove the unneeded grizzly specific jobs21:57
clarkbgrizzly specific jobs aren't used to test grizzly changes iirc21:57
jeblairzaro: left a comment in lp; you might want to reload21:58
fungiclarkb: but i still then need a separate cleanup change to remove the grizzly branch filters from the layout.yaml anyway21:58
*** b3nt_pin has joined #openstack-infra21:58
fungidoing so before the branches are deleted will change the way they're tested21:58
clarkbfungi: hrm, good point. so you are suggesting to do both job removal and job branch filter updates in one change?21:59
fungisince we have filters saying "only run this job on grizzly changes" or others saying "never run this job on grizzly changes"21:59
clarkbfungi: I was not suggesting you updated the job branch filters fwiw21:59
*** nkinder has joined #openstack-infra22:00
fungiand there are still plenty of grizzly-specific jobs (particularly old-docs jobs) which shouldn't be deleted before the branches are deleted22:00
*** b3nt_pin is now known as beagles22:00
jgriffithfungi: so I'm curious, could we just automate the whole conf thing as part of the merge process?22:01
jgriffithfungi: ie generate the conf file on succesful verify rigth before the merge?22:01
fungijgriffith: no, because that would change the commit being merged22:01
jgriffithnever mind22:01
jgriffithgot it22:01
jgriffithback under my rock I go22:01
*** _nadya_ has quit IRC22:02
clarkbfungi: ok, I will +2 this change and we can do all of them at once22:02
fungijgriffith: ideally we do something to stop including those files in the repo at all, and have devstack generate the sample configs it needs, and have a publish job that generates and sticks a copy in a useful place for deployers/operators to consult and docs to link to22:02
clarkbfungi: its just weird to me that we would stop running tests permanently as part of a change and not delete those tests too :)22:02
zarojeblair: i see now.  will update that.  just need to figure out how to work with lp/bzr again :)22:03
*** dhellmann is now known as dhellmann_22:03
*** blamar has quit IRC22:03
*** bhuvan_ has joined #openstack-infra22:03
fungiclarkb: it's part of a series... overall plan it stop running things which will cause negative impact if the branches are deleted first, then tag/delete the branches, then remove jobs and related job filters for everything grizzly-specific in multiple repos (config, d-g, et cetera)22:04
clarkbfungi: right22:04
zaroanteaya: my chage earlier shows the permissions required for workflow.  we basically want to allow 'change owner' and 'admin' set workflow label for all projects.22:05
fungii'm stacking the config cleanup change on top of that one, so we can still see whether things are heinous22:05
anteayazaro: I must have missed that, which patch has that?22:05
fungibut will wip/-2 it until remaining changes are abandoned and branches are deleted22:06
anteayazaro: looked at a few or yours22:06
clarkbfungi: its more the lack of 1:1 removal in that change that botehrs me than the planned stepping22:06
anteayak, thanks22:06
*** bhuvan has quit IRC22:06
*** bknudson has quit IRC22:09
zaroanteaya: refere that change. i've answered your questions22:09
*** rcleere has quit IRC22:09
*** ameade has quit IRC22:09
anteayazaro I think I have it now22:11
clarkbfungi: but given your plan it wfm22:11
anteayapatch owner gets -1 .. 0 voting on workflow label, core gets -1 .. +1 on workflow label22:12
*** jooools has quit IRC22:12
zaroanteaya: exactly!22:13
jeblairfungi: are you planning on removing grizzly branch specifiers in the zuul layout in a subsequent change?22:13
jeblairfungi: i think i'll interpret your comment as yes22:13
jeblairgrizzlyectomy part1 approved22:15
fungijeblair: yep, about to push that change now22:15
anteayathis was approved 10 hours ago by fungi is not on the zuul status page and never got merged:
anteayaI don't see why so I have no bug number for reverify22:16
*** moted has quit IRC22:16
clarkbjeblair: ya that is part of what fungi and I were just discussing, basically massive change post branch deletion to remove most things22:16
jeblairanteaya: that was far more than 10 hours ago.  it was likely during the time when gerrit was stuck.22:17
*** moted has joined #openstack-infra22:17
*** eharney has quit IRC22:17
openstackgerritA change was merged to openstack-infra/config: Prepare for deletion of stable/grizzly branches
*** mwagner_lap has quit IRC22:17
fungianteaya: reapproved, as jeblair noted that was when gerrit streaming was having issues yesterday22:18
anteayait said March 25 8:07am eastern time22:18
anteayaare we not still March 25?22:18
anteayaI know I get a lot of stuff wrong22:18
funginot on this planet anyway ;)22:18
fungiit's about to turn march 27th utc in a couple hours22:19
*** dstanek has quit IRC22:19
anteayathanks for letting me know22:19
anteayaI need to do something on the 27th22:19
*** whoops has quit IRC22:20
*** signed8bit has quit IRC22:21
*** jhesketh__ has joined #openstack-infra22:21
*** signed8bit has joined #openstack-infra22:22
anteayamorning jhesketh_22:22
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Remove grizzly jobs and job filters
fungiwip for now ^ but also i think there may be more jobs i can remove which were hanging around only to support grizzly changes so i'm hunting them down and adding to that shortly22:23
jeblairjhesketh_: good morning!  have you started looking into the streaming decompress data from swift problem?22:24
jhesketh_jeblair: yeah, I made some progress on that:
*** beagles has quit IRC22:25
*** ArxCruz has joined #openstack-infra22:25
jeblairjhesketh_: oh cool, i guess i got mixed up since that discussion didn't happen on the review22:26
jeblairi will try to look at that soon;  sdague ^22:26
jhesketh_jeblair, not sure what discussion you're referring to? I may have missed it22:26
jhesketh_jeblair: The zuul swift stuff and client stuff have been rebased and should be good to remerge22:27
jhesketh_well re-review22:27
jeblairjhesketh_: the one where sdague elucidated the problem to you and i on irc; i thought it happened after the most recent patchset, but it must have been before22:27
sdaguejhesketh_: sure, though that doesn't solve it for the compressed case22:27
*** lcheng has quit IRC22:27
*** bauzas has quit IRC22:27
sdaguewhich I thought was required to go this route22:27
jhesketh_sdague: I have compressed loading gzip in that patchset22:27
jhesketh_eh, bad speaking22:28
*** jergerber has quit IRC22:28
sdagueoh, ok, I see it now22:28
*** ameade has joined #openstack-infra22:28
jhesketh_admittedly only works for gzip22:28
*** ameade has quit IRC22:29
sdaguethat should be fine22:29
*** ameade has joined #openstack-infra22:29
jeblairjhesketh_: zuul change aprvd22:29
jgriffithOk, so I have a bit of a problem with this config file stuff22:30
jgriffithI can't seem to update it localy?22:30
*** dhellmann_ is now known as dhellmann22:31
jgriffithI run check and it fails, I run generate and no change?22:31
*** lcheng has joined #openstack-infra22:31
jgriffithso somwhere/somehow check and generate got out of sync22:31
jhesketh_thanks jeblair22:32
sdaguejhesketh_: that seems like a reasonable first pass22:32
sdaguejeblair: do we have a livish env to be able to play with this to make sure it works end to end?22:33
jhesketh_jeblair, clarkb: Also if you have time today it'd be great to get the javascript changes in, including mordred's, so future changes can work off the tail of jshint22:33
jeblairsdague: not yet, but once the zuul changes go in, we can start shoving real logs into a container22:36
openstackgerritA change was merged to openstack-infra/gitdm: Add alias email for
*** mrodden has joined #openstack-infra22:38
*** jgrimm has quit IRC22:39
*** yolanda has quit IRC22:40
sdaguecouple of minor things22:41
*** ociuhandu has joined #openstack-infra22:41
*** dhellmann is now known as dhellmann_22:41
sdaguethere was a reason why I was doing the file read and failing on it to throw the 404 instead of doing a file exists check22:41
sdaguebut, honestly, I can't remember what that was now22:42
*** blamar has joined #openstack-infra22:42
sdagueso if it's still an issue we'll probably see it pretty quickly once it lands22:42
jhesketh_hmm okay22:44
*** thedodd has quit IRC22:45
openstackgerritA change was merged to openstack-infra/zuul: Send swift upload instructions to workers
*** rfolco has quit IRC22:46
*** e0ne has quit IRC22:49
*** dstanek has joined #openstack-infra22:49
*** lcheng has quit IRC22:49
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Remove grizzly jobs and job filters
*** loquacities is now known as starving22:52
*** blamar has quit IRC22:53
jhesketh_jeblair: yay, swift is merged22:54
fungiadam_g: if you're around, any objection to me (ab)using gerrit admin privs to abandon the one remaining open grizzly change?
jhesketh_jeblair: so how do we set up a swift account.. I'm guessing you might need to do something?22:54
adam_gfungi, yeah--definitely22:56
fungiabout to delete the branch out from under it, so just wanted to be sure it's dead, dead, deadski22:57
*** alex-gone is now known as Alexandra22:58
*** harlowja has quit IRC22:58
adam_gis there some way for jenkins to blacklist a given cloud provider for specific jobs?  wondering if workaround can be avoided if check-tempest-dsvm-virtual-ironic never ended up on rackspace23:00
*** mrodden has quit IRC23:02
fungiadam_g: not trivially--we basically need special node types defined which are only on some providers and not others23:03
fungiadam_g: and that makes it hard for us to run jobs if the wrong provider is down, so we'd really rather not23:03
*** harlowja has joined #openstack-infra23:03
fungialso makes it harder to balance workloads across providers23:03
adam_gah okay23:03
adam_gsomeone had pitched the idea as a better work around, didn't know if its actually doable23:04
fungiadam_g: more likely that problem could vaporize soon if newer libvirt starts getting used?23:04
fungioh, nevermind the libvirt thing--it's a host setup issue i guess23:05
*** bknudson has joined #openstack-infra23:05
fungii was looking at the comment above it23:05
adam_gfungi, well, a newer libvirt would help--theres an update in the queue but appears to be slow moving23:05
*** starving is now known as Loquacities23:05
*** julim has quit IRC23:05
sdagueadam_g: why does that get in the way?23:05
*** markmc has quit IRC23:06
adam_gsdague, libvirt bug in saucy/UCA-icehouse -
uvirtbotLaunchpad bug 1248025 in libvirt "[SRU] libvirt-bin fails to start inside Xen" [High,In progress]23:06
adam_gwonder if we can just use UCA icehouse for that instead23:06
fungioh, that craziness again23:06
sdagueadam_g: is there a reason you guys aren't running on precise?23:07
adam_gsdague, we are, the devstack setup relies on newer OVS features in newer libvirt to get the fake baremetal networking environment setup23:07
*** signed8bit has quit IRC23:08
adam_gso libvirt comes from UCA23:08
sdagueand no way to do that with old ovs?23:08
fungiadam_g: in theory this also goes away (and we presumably replace it with a whole new set of troubles) when we move to testing on trusty for juno?23:09
*** juice has quit IRC23:09
*** Trozz has joined #openstack-infra23:10
adam_gsdague, AFAIK its libvirt features specific to OVS, not OVS itself.23:11
adam_gsdague, <virtualport='openvswitch'> in domain xml definination, for example23:12
*** dims_ has quit IRC23:12
adam_gfungi, yeah, in theory. at this point if the ironic job needs to install anything newer, it should probably be what we'll be using in juno anyway23:12
*** mrodden has joined #openstack-infra23:13
*** mburned is now known as mburned_out23:14
*** bhuvan_ has quit IRC23:15
adam_gi wonder if its still okay for me to upload to UCA.23:16
fungiadam_g: there's one way to find out... if they don't revoke your account for it (or if they do for that matter!) you'll have your answer ;)23:17
*** bhuvan has joined #openstack-infra23:18
*** flaper87 is now known as flaper87|afk23:18
openstackgerritJeremy Stanley proposed a change to openstack-infra/devstack-gate: Remove stable/grizzly provisions and workarounds
*** jepoy has joined #openstack-infra23:19
*** jepoy has quit IRC23:19
*** mkoderer has quit IRC23:21
zarojeblair: just update plugin info patch for python-jenkins,  hope i've done it correctly :)23:23
*** nati_ueno has quit IRC23:23
*** nati_ueno has joined #openstack-infra23:23
zarojeblair: update will key on both shortName and longName23:23
*** mrodden has quit IRC23:24
*** mrodden has joined #openstack-infra23:25
*** juice has joined #openstack-infra23:26
clarkbfungi: can you check my comment on the config chagne to rip stuff out?23:27
*** arosen-laptop has joined #openstack-infra23:28
clarkbfungi: pretty sure its related to the grenade branch stuff of before and am looking for clarification23:28
*** alexpilotti has quit IRC23:28
jeblairzaro: cool, thx23:28
*** mburned_out is now known as mburned23:29
*** dims_ has joined #openstack-infra23:30
fungiclarkb: good point... i guess that was only needed because grenade's stable/grizzly branch ran no voting gate jobs?23:31
fungiclarkb: i guess we can just remove that?23:31
clarkbfungi: yeah I think we can just rip it out23:31
clarkbfungi: basically grenade didn't work with grizzly or something23:31
fungibut had a grizzly branch anyway?23:32
clarkbfungi: let me do some git history detectiving to track that down23:32
*** mriedem has joined #openstack-infra23:32
clarkbfungi: aha!23:33
clarkbfungi: 94b80e00e334917f2b2dd1a7223ce2fd580ec726 it can be removed23:33
jhesketh_sdague: for making swiftclient option in loganalyze, should I check if swift is set up in the config and import it at that point23:36
jhesketh_that will mean importing it at each application request23:36
*** yamahata has quit IRC23:37
*** thuc_ has quit IRC23:37
*** mwagner_lap has joined #openstack-infra23:38
*** thuc has joined #openstack-infra23:38
sdaguejhesketh_: honestly, I'd be fine with a try at the top of the imports23:39
*** Trozz has quit IRC23:40
sdaguemostly I want to be able to run locally without swiftclient when I don't need swiftclient23:40
jhesketh_works for me23:41
*** emagana has quit IRC23:41
*** thuc has quit IRC23:42
*** mrodden has quit IRC23:43
*** mrodden has joined #openstack-infra23:45
*** mrodden has quit IRC23:47
clarkbfungi: d-g change for grizzly reviewed23:58
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Fetch tag refs from zuul in gerrit-git-prep
clarkbjhesketh_: sdague: is loganalyze going to fetch from swift, do modifications, then serve to the client?23:58
fungiclarkb: is that ^ what you had in mind for your previous comment on that change?23:58
clarkbfungi: looking23:59
sdagueclarkb: yeh23:59
sdaguejhesketh_: just implemented a generator stack to do that23:59
sdagueso the top layers look exactly the same as before23:59

Generated by 2.14.0 by Marius Gedminas - find it at!