Wednesday, 2016-02-03

fungion the other hand, there are also lots more of them, and it's a stateful service not a command-line utility00:00
notmorganfungi: that metric makes me sad. "faster tha nova's unit tests" is not the metric i'd measure things by00:00
*** dizquierdo has joined #openstack-infra00:01
*** jamielennox|away is now known as jamielennox00:02
*** rbrndt has quit IRC00:02
*** camunoz has quit IRC00:02
Sam-I-Amhmm... is something wrong with r.o.o?00:03
Sam-I-Amseems to be hanging, eventually returning service unavailable00:03
Sam-I-Amoo, now a 500 error00:03
funginotmorgan: it was a... tongue-in-cheek metric00:03
fungiSam-I-Am: database backups kick off at midnight utc00:04
*** pahuang_ has quit IRC00:05
notmorganfungi: >.>00:05
Sam-I-Amfungi: ah ha00:06
*** krotscheck1 has quit IRC00:08
*** Sukhdev has quit IRC00:09
*** rere has joined #openstack-infra00:09
*** rere has left #openstack-infra00:09
openstackgerritMerged openstack-infra/nodepool: Add launch-timeout validator into config-validation.
*** Sukhdev has joined #openstack-infra00:10
*** zz_dimtruck is now known as dimtruck00:11
*** Daisy_ has joined #openstack-infra00:13
*** Guest80711 is now known as med_00:15
*** med_ has quit IRC00:15
*** med_ has joined #openstack-infra00:15
*** dizquierdo has quit IRC00:15
*** Daisy_ has quit IRC00:16
openstackgerritIan Wienand proposed openstack-infra/nodepool: Add ipv6-preferred into config-validation
*** camunoz has joined #openstack-infra00:19
prometheanfireneat, gate is stuck00:21
jeblairprometheanfire: can you elaborate?00:21
*** pahuang_ has joined #openstack-infra00:22
prometheanfirenah, just noticed the wait time is almost 12hr now00:22
jeblairprometheanfire: okay.  the words you use have a specific meaning to some people that apparently is not the meaning you intended.  :)00:22
fungithe gate is "busy"00:23
*** yamamoto_ has joined #openstack-infra00:24
ianwgreghaynes: maybe it's obvious to everyone else looking at , but i wouldn't mind some comments to convince me how you came to the conclusion this is the right thing to do...00:24
jeblairwords like 'stuck', 'wedged', 'completely broken' tend to make us drop what we're doing and fix things.  words like 'slow because tests are flakey' not so much.00:24
*** thorst has joined #openstack-infra00:24
*** abitha has joined #openstack-infra00:25
*** diana_clarke has joined #openstack-infra00:25
jeblairfungi, whereiskrotscheckanyway: exists00:25
fungii agree, and even more importantly, apache did not fall over!00:25
fungiso we're safe to proceed00:25
nibalizerjeblair: did you see my timing comparisons for infracloud this morning?00:26
jeblairi'm not going to claim the lock on building servers right now if someone else wants to; but will do so later (possibly tomorrow) if no one else has00:26
jeblairnibalizer: no00:26
nibalizer 1883.0000 - 1144 = 73900:27
jeblairoh yeah i did see that00:27
jeblairi recognize those numbers :)00:27
*** sdake has joined #openstack-infra00:27
nibalizer739 seconds slower, so like 12 minutes00:27
jeblairnibalizer: what's 1144?00:29
jeblairnibalizer: and is this ~= gate-tempest-dsvm-full ?00:29
nibalizeryou linked a console-log00:29
jeblairnibalizer: yeah, but it was a random one just to show you where the number was00:29
nibalizerfrom a random test and that timing was 114400:29
jeblairi don't even know what job or branch it was :)00:30
*** yamamoto_ has quit IRC00:30
openstackgerritgreghaynes proposed openstack-infra/nodepool: Handle dib image deletion during periodic cleanup
*** yamamoto_ has joined #openstack-infra00:30
greghaynesianw: hah, I was currently in the process of writing a new commit msg for that :)00:31
*** annegentle has joined #openstack-infra00:31
jeblairnibalizer: i'm looking at some recent nova runs now, and i think that may be well within our current variation00:31
nibalizer looks like the right job00:31
nibalizerright job right branch00:31
*** thorst has quit IRC00:32
nibalizerjeblair: thats good news00:32
*** ybathia has joined #openstack-infra00:32
jeblairnibalizer: yeah, so we may be in the ovh/internap range but not as fast as rax00:32
nibalizercan we take the exploratory step of setting up nodepool creds for omfra then? give it a node? start up a mirror?00:32
jeblairnibalizer: yeah, i think so.00:33
*** jed56 has quit IRC00:33
jeblairnibalizer: here's a much slower internap node:
fungianother infra-cloud00:33
jeblairnibalizer: and ovh looks closer to omfra at 1900
openstackgerritJoshua Harlow proposed openstack/requirements: Bump up eventlet to at least 0.18.1
jeblairi think 'close to ovh performance' is good :)00:34
*** yamamot__ has joined #openstack-infra00:34
*** yamamoto_ has quit IRC00:34
*** annegentle has quit IRC00:36
* nibalizer will make with the patches00:37
nibalizercrinkle: any objection to me putting into dns?00:38
fungii thought i already had?00:38
*** jamielennox is now known as jamielennox|away00:38
crinklefungi: controller00 is in dns00:38
crinklenibalizer: there isn't anything on controller0100:38
fungiah! right00:38
fungifinally someone who, like me, enjoys starting host numbers at 0 and not 1 ;)00:39
clarkbI do think we should try to get as that seems to get stuff working on bluebox00:39
*** jamielennox|away is now known as jamielennox00:39
clarkbit is the switch to vxlan from gre00:39
crinklefungi: i think that started with SpamapS :P00:39
* fungi is not surprised in the least00:39
nibalizerah yes I am using controller00 in all my openstack config files00:40
clarkbbluebox floating ip listings look good still00:40
crinklethese machines are almost definitely going to be moved and probably renumbered so i wouldn't spend a lot of effort dnsing them yet00:40
fungiclarkb: seems a bit unfortunate to have to do that, but i guess it's an okay workaround while we try to get someone to figure out if bbox is just missing a conntrack plugin or something00:40
clarkbI do not think it was a persistent issue whatever was causing hte leak00:40
*** thorst has joined #openstack-infra00:40
SpamapScrinkle: aye, 'twas the only way I knew. :)00:41
clarkbfungi: ya I don't want to have to wait on that if we can make it work with an acceptable alternative00:41
fungiSpamapS: numbers start at 000:41
fungiof course00:41
clarkbfungi: I just keep looking at the health dashboard and bleubox fail rate is really high compared to everything else00:42
*** thorst has quit IRC00:42
clarkba large chunk of that appears to be multinode00:42
openstackgerritJoshua Harlow proposed openstack/requirements: Bump up eventlet to at least 0.18.1
*** lucasagomes has quit IRC00:42
SpamapSdo the multiple nodes communicate via the floating ips?00:43
fungiclarkb: it seems like gate-tempest-dsvm-multinode-full consistently failed that change. what's up with that, any idea?00:43
clarkbSpamapS: yes, tempest sshes to the floating IPs and may have to do so to a VM on the other hypervisor00:43
clarkbfungi: I think we are masking other issues with the consistent fail to ssh00:43
clarkbfungi: it does pass on other regions00:43
SpamapSThat shouldn't be super terrible, but it is definitely susceptible to slowdowns on the l3 agent.00:43
clarkbfungi: is running on ovh now and I expect that to pass00:44
clarkbSpamapS: it all works great when the floating IP is local :)00:44
SpamapSmeaning a not-floating IP?00:44
clarkbno a floating IP NAT'd on the local host00:45
clarkbbecause then you just hit networking in the kernel and never jump machines and it just works (tm)00:45
*** alivigni has quit IRC00:45
clarkbbut if that floating IP is NAT'd on the other hypervisor (as with nova net multihost or neutron DVR) you need a working overlay network00:45
SpamapSoh, so some rule that DNAT's to
*** dchen has joined #openstack-infra00:46
clarkbSpamapS: floating IPs NAT (floating IPs) to the libvirt 10.whatever addrsses00:46
clarkbif iptables is doing that on the same machine as tempest it works00:46
*** yamamot__ has quit IRC00:46
SpamapSOh I see, it's the floating ip.. of the VM00:46
SpamapSof the vm inside the vm00:46
SpamapS^^ exactly how I feel righ tnow00:47
*** lucasagomes has joined #openstack-infra00:49
*** bpokorny_ has quit IRC00:50
clarkbthere are many turtles involved00:51
clarkbbut tl;dr is outside of neutron we set up an overlay to route the floating IPs independent of the external world's networking00:51
*** bpokorny has joined #openstack-infra00:51
clarkbwe have been using GRE but bleubox seems to filter those packets00:51
clarkbvxlan on the other hand works except that tests still fail but they don't fail due to ssh00:51
SpamapSfilter, or MTU fail?00:51
clarkbpretty sure it isn't an mtu fail because they are tiny pings00:52
*** ashleighfarnham has quit IRC00:53
clarkband we have set mtus to 145000:53
clarkbwhich should be plenty small for both gre and vxlan00:53
*** jpr has quit IRC00:54
*** bhunter71 has quit IRC00:55
greghaynesSounds like they have an explicit list of l4 protos allowed00:56
*** pvaneck has quit IRC00:56
SpamapS greghaynes the question is.. why?00:57
clarkbI also confirmed that it wasn't just arp that was broken by manually populating arp tables00:57
*** mriedem_machell is now known as mriedem_afk00:58
*** eil397 has quit IRC00:58
*** ybathia has quit IRC01:01
*** SumitNaiksatam has quit IRC01:02
fungigreghaynes: mgagne mentioned that if you're neutron-based then iptables explicitly wants to state-track all communication through the interfaces, but if you don't explicitly load the conntrack plugin for gre then it can't set up state and doesn't allow the responses back in01:02
openstackgerritTony Breeds proposed openstack-infra/yaml2ical: Add functionality to batch the meetings list
greghaynesfungi: hrm, we should see the first outbound packet get there then01:03
clarkbgreghaynes: not on our vm though01:04
clarkband I can't tcpdump the hypervisors01:04
nibalizerinfra-root, so it looks like we have 2 tenants per cloud? one that has 'jenkins' in the name and is the nodepool, and one that does not and we use to hold mirrors etc?01:04
fungiwell, we might since there's no session establishment for gre?01:04
fungior... who knows01:04
funginibalizer: correcto01:04
clarkbnibalizer: correct01:04
*** rajinir has quit IRC01:05
funginibalizer: that way we don't have to entrust nodepoold with the keys to the tenant which houses non-nodepool-managed hosts01:05
jeblairnibalizer: given a choice, we try to call them openstackjenkins and openstackci01:05
*** kzaitsev_mb has quit IRC01:05
jeblairand yeah, that's probably going to live long after we stop using jenkins :)01:05
fungiat least it's not openstackhudson?01:06
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: WIP: Try a boostrap grid to list all the meetings
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Exploit the new batch_meetings function
openstackgerritYushiro FURUKAWA proposed openstack-infra/project-config: Add python3-job and check-requirements for networking-fujitsu
jeblairit barely missed being called that :)01:06
clarkbfungi: we had one fail with volume backed live migration01:06
clarkbdoesn't appear to be network related?01:06
clarkbI want to say that this is something that nova semi expects01:07
jeblairfungi: did we decide on a name for the wheel mirror slave?01:07
jeblairmaybe we should call it wheel-mirror-trustyx64.slave.o.o to use the same shortened form as the afs volume01:08
fungijeblair: i thought krotscheck had theoretically come up with a scheme... he was asking earlier today about underscores (and i referred him to the appropriate rfcs on allowed characters in hostnames as a result)01:11
fungibut i don't know what he arrived at, if anything01:12
*** Swami_ has quit IRC01:12
fungijeblair: in the interest of maintaining my personal no-bikeshed zone, i'm inclined to just agree to your first suggestion so we can move along01:15
jeblairok.  i'm still not starting the launch (and am in fact going afk), so it's still up for grabs :)01:17
jeblairprobably looking at tomorrow if it's me01:17
funginoted. i have beer here so likely not going to get to it until tomorrow either01:18
*** sputnik13 has quit IRC01:19
* nibalizer prepping omfracloud patches01:21
*** gongysh has joined #openstack-infra01:22
*** pradk has quit IRC01:22
*** sridhar_ram has quit IRC01:26
*** abitha has quit IRC01:30
*** jamielennox is now known as jamielennox|away01:30
*** shashank_hegde has quit IRC01:32
*** achanda has joined #openstack-infra01:32
greghaynesfungi: cheers01:35
*** thorst has joined #openstack-infra01:36
*** Daisy has joined #openstack-infra01:38
*** Sukhdev has quit IRC01:39
*** flwang has quit IRC01:39
*** jsavak has joined #openstack-infra01:40
*** weshay has quit IRC01:40
*** dimtruck is now known as zz_dimtruck01:40
*** baoli has joined #openstack-infra01:42
*** baoli has quit IRC01:43
*** SumitNaiksatam has joined #openstack-infra01:44
*** thorst has quit IRC01:44
*** thorst has joined #openstack-infra01:45
*** mtanino has quit IRC01:45
*** jsavak has quit IRC01:45
*** jsavak has joined #openstack-infra01:45
*** zz_dimtruck is now known as dimtruck01:47
*** krotscheck1 has joined #openstack-infra01:50
*** jpr has joined #openstack-infra01:52
*** flwang has joined #openstack-infra01:53
*** thorst has quit IRC01:53
openstackgerritMerged openstack-infra/nodepool: Add ipv6-preferred into config-validation
openstackgerritPaul Belanger proposed openstack-infra/project-config: Always clone openstack/windmill for jobs
*** ok_delta has joined #openstack-infra01:59
*** ok___delta has joined #openstack-infra01:59
*** ybathia has joined #openstack-infra02:00
*** ybathia has quit IRC02:01
*** rossella_s has quit IRC02:02
*** rossella_s has joined #openstack-infra02:03
*** sdake has quit IRC02:05
*** ok___delta has quit IRC02:07
*** ok_delta has quit IRC02:07
*** camunoz has quit IRC02:08
*** salv-orl_ has joined #openstack-infra02:09
*** pahuang_ has quit IRC02:10
*** bpokorny has quit IRC02:11
*** jpr has quit IRC02:11
openstackgerritMerged openstack-infra/irc-meetings: Remove glance-drivers meeting
*** salv-orlando has quit IRC02:12
* nibalizer adds omfracloud credentials to hiera02:14
*** EricGonczer_ has joined #openstack-infra02:14
nibalizer(in other words, I am taking the hiera lock)02:14
openstackgerritMerged openstack-infra/irc-meetings: Move Glance artifacts meeting
nibalizerand done02:17
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Change chair for Glance's meeting
*** tiswanso has joined #openstack-infra02:19
*** tiswanso has quit IRC02:19
*** tiswanso has joined #openstack-infra02:20
*** camunoz has joined #openstack-infra02:20
*** zhurong has joined #openstack-infra02:21
*** Jeffrey4l has joined #openstack-infra02:22
openstackgerritPaul Belanger proposed openstack-infra/project-config: Always clone openstack/windmill for jobs
*** pahuang_ has joined #openstack-infra02:23
openstackgerritMerged openstack-infra/irc-meetings: Change chair for Glance's meeting
*** jsavak has quit IRC02:24
*** amotoki has joined #openstack-infra02:24
*** jsavak has joined #openstack-infra02:24
angdraugAJaeger: the promised plugins post:
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add OmfraCloud to puppetmaster_clouds
*** jamielennox|away is now known as jamielennox02:31
fungiclarkb: oh hey! see that update on the osic environment being ready by friday?02:31
clarkbfungi I hadnt bjt will check email02:32
*** kzaitsev_mb has joined #openstack-infra02:32
funginot urgent. nothing new needed from us02:32
*** angdraug has quit IRC02:32
*** woodster_ has joined #openstack-infra02:37
*** tphummel has quit IRC02:37
*** Somay has joined #openstack-infra02:37
*** EricGonczer_ has quit IRC02:37
*** EricGonczer_ has joined #openstack-infra02:37
*** jsavak has quit IRC02:38
*** Thelo has quit IRC02:39
*** dims_ has quit IRC02:40
*** Apoorva has quit IRC02:41
*** dimtruck is now known as zz_dimtruck02:41
*** Jeffrey4l has quit IRC02:42
*** zz_dimtruck is now known as dimtruck02:42
*** FallenPegasus has joined #openstack-infra02:43
*** Somay has quit IRC02:43
*** EricGonczer_ has quit IRC02:44
*** yamamoto has joined #openstack-infra02:46
*** yamamoto has quit IRC02:49
*** thorst has joined #openstack-infra02:52
*** bhunter71 has joined #openstack-infra02:54
*** sam_wan has joined #openstack-infra02:54
*** thorst has quit IRC02:58
sam_wanhello, someone please tell me the email alias of openstack-infra team02:58
sam_wanI need to enable my CI account02:59
*** flwang1 has left #openstack-infra03:01
*** sdake has joined #openstack-infra03:06
*** links has joined #openstack-infra03:06
nibalizersam_wan: what ci system do you run03:08
sam_wanEMC isilon03:09
tonybDid jenkins get confused earlier today?  All of the reviews against nova kilo went into merge conflict at the same time.03:10
tonybIt's not impossible but it is improbable, and the 2 I've checked manually just fast-forward03:11
*** kzaitsev_mb has quit IRC03:11
*** Daisy has quit IRC03:11
*** aeng has joined #openstack-infra03:12
sam_wannibalizer: do you know who i should contact?03:12
nibalizersam_wan: I can do it hold on03:14
tonybsam_wan: email with all the details03:14
nibalizerwhy was your system shutdown?03:14
sam_wannibalizer: thanks03:14
nibalizersam_wan: i'm not finding the email we sent out when disabling your account03:15
nibalizerdo you hvae it handy?03:15
sam_wanit's disabled by Jim because we checked devstack-gate project03:15
sam_wanI've altered my zuul configuration so we don't check devstack-gate again03:16
nibalizertis review03:16
sam_wanI don't mean a specific patch, i enabled check for devstack-gate project before03:18
sam_wanbecause I once uploaded a patch for devstack-gate and I thought maybe I need one of our CI's to check it03:19
*** yuanying_ has quit IRC03:20
sam_wanI've got an email from 'James E. Blair <>'  on 1/2903:20
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Fix memory leak reloading triggers
jheskethjeblair, fungi: ^ that should fix our memory leaks03:22
nibalizersam_wan: ok03:22
nibalizerI will re-enable your account03:22
jheskethsorry it took so long to track down!03:22
nibalizerwhat is the gerrit name?03:23
nibalizerhi jhesketh03:23
jheskethHey nibalizer03:23
sam_wanthanks nibalizer03:23
nibalizerhow are you? are you at lca?03:24
jheskethnibalizer: I'm good thanks. Indeed I am, currently sitting in a talk no less ;-)03:24
bkeronibalizer: we should go next year. It's in Tazmania!03:24
*** yuanying has joined #openstack-infra03:24
mtreinishjhesketh: learning about power?03:25
jheskethmtreinish: trying ;-)03:25
jheskethbkero: that's my home town :-)03:25
bkeroYou mean Hobart?03:25
openstackgerritMerged openstack/requirements: Exclude xvfbwrapper 0.2.8 from global requirements
fungijhesketh: convenient ;)03:26
jheskethbkero: correct03:26
nibalizertaking the hiera lock again03:26
fungialso, nice find on tracking down the leak!03:26
jheskethit's a trivial fix, but was less trivial to find03:26
fungithose sorts of bugs often are03:27
jheskethalthough we should probalby see how it goes before saying it's fixed03:27
bkerojhesketh: Nice. I've been to a few LCAs, but skipped this year and hoped to double down for LCA Hobart next year. :)03:27
nibalizerlock released03:27
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add omfracloud to nodepool
jheskethbkero: cool, hope to see you there then!03:28
*** aeng has quit IRC03:28
*** gyee has quit IRC03:28
*** camunoz has quit IRC03:29
bkerojhesketh: Did you take the ferry over from Devonport?03:30
jheskethbkero: nah I flew over.. it's much quicker and in some ways cheaper03:31
jheskethparticularly since I don't need a car here03:31
bkeroYeah, I guess the only reason you'd do it is car.03:31
bkeroBut having a car in Geelong is probably a good thing. I remember I was thankful to have a car in Ballarat.03:31
jheskethbkero: the campus in the city so it's not quite the same as ballarat03:32
jheskeththe student accom is a while away and a car there would probably be handy, but there are buses03:32
jheskethand I'm staying in the city anyway03:32
bkerojhesketh: My partner is there, and I hear the accom is like a 2 hour walk away.03:32
bkeroI told them to stay in the student housing, and the first two nights they didn't have hot water. I'm kind in hot water over that. -_-03:33
bkeroI thought Geelong was supposed to be hosted at some kind of Casino, not a Uni like usua03:33
jheskethyep, it's not perfect but there are buses on loops03:33
*** pahuang_ has quit IRC03:33
jheskethoh I didn't hear about no hot water03:33
jheskeththat sucks :-(03:33
bkeroSuper grumpy, especially since one has to be up long beforehand to take the bus to the venue03:34
jheskethbkero: Hobart will be at a convention centre at a hotel which happens to have a casino03:34
jheskethie next year03:34
bkeroOooh, that's the casino thing. Thanks for setting me straight.03:34
*** jamespd_ is now known as jamespd03:34
*** doug-fish has quit IRC03:34
*** yamamoto has joined #openstack-infra03:34
*** doug-fish has joined #openstack-infra03:35
jlvillaltonyb, I changed title on:  So it no longer says WIP :)03:35
*** sam_wan has quit IRC03:35
*** doug-fish has quit IRC03:36
tonybjlvillal: okay.03:36
tonybjlvillal: I was hoping that things would "just happen" but ....03:37
tonybjlvillal: anyway things are somewhat moving now03:37
*** sam_wan has joined #openstack-infra03:37
jlvillaltonyb, I think it looks better and other people seem to agree.03:37
jlvillalMaybe someone can make it look even better after this :)03:37
tonybjlvillal: Maybe, TBH I find the whole page too busy so rather than look for what I need I just do in page search03:38
tonybjlvillal: did you see my alternate sort version?03:38
jlvillaltonyb, I should probably do that03:38
*** gyee has joined #openstack-infra03:38
jlvillalI have not03:38
*** pavel_bondar has quit IRC03:39
nibalizersam_wan: i think i have reactivated your account03:39
tonybjlvillal: it's basically your work I just altered the way the lists are generated03:39
sam_wanhi nibalizer, yes, thanks a lot03:39
jlvillaltonyb, Looks good to me.  Oh great. I like going down the columns!03:40
jlvillaltonyb, I didn't know how to do that. And didn't spend time trying to figure it out.03:40
jlvillalI like that much better. Maybe only downside is maybe on phones. But not sure.03:41
tonybjlvillal: that's how you do it :)03:41
tonybjlvillal: If you're using that page on a phone you've already lost ;P03:41
tonybjlvillal: I *wanted* to do it all in jinja2 but that'll do03:42
*** shashank_hegde has joined #openstack-infra03:42
jlvillaltonyb, I like it. Thanks03:42
tonybjlvillal: at least it isn't tightly coupled03:42
*** camunoz has joined #openstack-infra03:42
tonybjlvillal: we'll see if ttx agrees.03:42
tonybjlvillal: If he does there will probably be some rebasing etc but I'll do that for you.03:43
jlvillalThank you.03:43
*** sam_wan has quit IRC03:43
*** maishsk has quit IRC03:43
*** sam_wan has joined #openstack-infra03:44
*** unicell1 has quit IRC03:46
*** sam_wan has quit IRC03:46
openstackgerritSpencer Krum proposed openstack-infra/project-config: Use omfracloud in nodepool
*** esp_ has joined #openstack-infra03:47
*** deva_ has joined #openstack-infra03:47
*** hichihara has joined #openstack-infra03:47
*** tphummel has joined #openstack-infra03:48
*** NobodyCa1 has joined #openstack-infra03:48
*** pahuang_ has joined #openstack-infra03:49
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add omfracloud to nodepool
*** esp_ has quit IRC03:51
*** pavel_bondar has joined #openstack-infra03:51
*** deva_ has quit IRC03:53
*** hdd has quit IRC03:53
*** NobodyCa1 has quit IRC03:53
*** mrmartin has joined #openstack-infra03:53
*** thorst has joined #openstack-infra03:56
openstackgerritJamie Lennox proposed openstack-infra/project-config: Run the identity v3 only jobs as part of integrated gate
*** gongysh has quit IRC04:00
*** FallenPegasus has quit IRC04:00
*** thorst has quit IRC04:03
*** yuanying has quit IRC04:06
*** gyee has quit IRC04:07
*** yuanying has joined #openstack-infra04:07
*** kzaitsev_mb has joined #openstack-infra04:08
*** jamielennox is now known as jamielennox|away04:09
*** sdake has quit IRC04:10
*** deepakcs has joined #openstack-infra04:21
*** mriedem_afk has quit IRC04:22
deepakcssdague: Hi, reg:
deepakcssdague: is it possible that the reason for the higher fail rate is bcos the job runs against the plugin project as well, and might fail for patches against them as well04:23
cody-somervilletonyb: You could use some CSS so that the long strings are truncated (nicely) and don't cause that ugly irregular whitespace.04:24
cody-somervillealso, alternating row colouring along with a little padding might also make it easier to read, allowing for one to easily keep track as they eye vertically along the dataset.04:26
*** achanda has quit IRC04:28
*** dimtruck is now known as zz_dimtruck04:29
*** Daisy has joined #openstack-infra04:38
*** FallenPegasus has joined #openstack-infra04:41
*** achanda has joined #openstack-infra04:43
*** Daisy has quit IRC04:43
*** amotoki has quit IRC04:57
*** kzaitsev_mb has quit IRC04:58
*** maishsk has joined #openstack-infra04:59
*** tphummel has quit IRC05:00
*** camunoz has quit IRC05:01
*** thorst has joined #openstack-infra05:01
*** dchen has quit IRC05:01
*** esker has joined #openstack-infra05:06
*** thorst has quit IRC05:08
*** esker has quit IRC05:11
*** camunoz has joined #openstack-infra05:13
*** dchen has joined #openstack-infra05:15
*** abitha has joined #openstack-infra05:16
*** amotoki has joined #openstack-infra05:18
*** FallenPegasus has quit IRC05:18
*** Sukhdev has joined #openstack-infra05:23
*** dchen has quit IRC05:24
*** rguillebert has quit IRC05:27
*** amotoki_ has joined #openstack-infra05:30
*** amotoki has quit IRC05:33
*** vgridnev has joined #openstack-infra05:44
*** maishsk has quit IRC05:46
*** abitha has quit IRC05:47
openstackgerritMerged openstack/requirements: bump python-designateclient to 2.0.0
*** jtomasek__ has joined #openstack-infra05:51
*** amotoki_ has quit IRC05:52
*** kzaitsev_mb has joined #openstack-infra05:55
*** SumitNaiksatam has quit IRC05:57
*** SumitNaiksatam has joined #openstack-infra05:57
*** Jeffrey4l has joined #openstack-infra06:02
*** rossella_s has quit IRC06:02
*** rossella_s has joined #openstack-infra06:03
*** amotoki has joined #openstack-infra06:05
*** thorst has joined #openstack-infra06:07
*** mrmartin has quit IRC06:09
*** mrmartin has joined #openstack-infra06:09
*** armax_ has joined #openstack-infra06:11
*** unicell has joined #openstack-infra06:11
*** armax has quit IRC06:12
*** armax_ is now known as armax06:12
*** jtomasek__ has quit IRC06:13
*** thorst has quit IRC06:13
*** mrmartin has quit IRC06:15
*** mrmartin has joined #openstack-infra06:16
*** abregman has joined #openstack-infra06:16
*** woodster_ has quit IRC06:16
*** mrmartin has quit IRC06:16
*** FallenPegasus has joined #openstack-infra06:17
*** amotoki has quit IRC06:18
*** FallenPegasus has quit IRC06:22
*** HeOS has joined #openstack-infra06:24
*** SumitNaiksatam has quit IRC06:24
*** abregman has quit IRC06:25
*** abregman has joined #openstack-infra06:25
*** amotoki has joined #openstack-infra06:26
*** jaypipes has quit IRC06:26
*** SumitNaiksatam has joined #openstack-infra06:28
openstackgerritMasahito Muroi proposed openstack-infra/project-config: Adds a devstack test job using Congress new architecture
openstackgerritMasahito Muroi proposed openstack-infra/project-config: Adds a tox test job for Congress new architecture
*** eliqiao has quit IRC06:31
*** eliqiao has joined #openstack-infra06:32
*** zul has joined #openstack-infra06:33
*** sflanigan has joined #openstack-infra06:34
*** sflanigan has joined #openstack-infra06:34
*** dkehn has quit IRC06:43
*** amotoki has quit IRC06:46
*** oomichi has joined #openstack-infra06:47
*** dkehn has joined #openstack-infra06:49
*** zul has quit IRC06:51
*** armax has quit IRC06:52
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Cleanup translation scripts
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Re-enable repos for translation
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Re-Enable django_openstack_auth/designate-dashboard translations
*** YorikSar_ is now known as YorikSar07:06
*** amotoki has joined #openstack-infra07:06
*** gildub has quit IRC07:08
*** kzaitsev_mb has quit IRC07:08
openstackgerritMark McLoughlin proposed openstack-infra/zuul: Fix tiny typo in scheduler comment
*** esker has joined #openstack-infra07:11
*** thorst has joined #openstack-infra07:13
AJaegeramotoki: could you look at , please?07:14
*** infra-red has joined #openstack-infra07:14
openstackgerritIan Wienand proposed openstack-infra/puppet-openstackci: nodepool : add flag to install diskimage-builder from git
*** esker has quit IRC07:16
*** lbragstad has quit IRC07:16
AJaegerjeblair: we're soon publishing *all* documents everytime now - the building of only a few documents was an optimization we did for DocBook building (and glossary is last XML in openstack-doc, we can build it every time if that helps). RST building is faster, so we build all... So, option 1 is the one to go...07:16
openstackgerritvenkatamahesh proposed openstack-infra/project-config: Fix some word spellings
*** fungi has quit IRC07:17
*** thorst has quit IRC07:18
AJaegeramotoki: same problem: - is setup.cfg broken?07:19
*** yamahata has quit IRC07:19
*** diana_clarke has quit IRC07:19
amotokiAJaeger: looking07:20
AJaegerthanks, amotoki07:20
openstackgerritIan Wienand proposed openstack-infra/system-config: Use diskimage-builder from git for nodepool
*** lbragstad has joined #openstack-infra07:21
ianwpabelanger: ^ yeah, prior change was wrong ... i think i untangled it!07:22
AJaegerianw, yolanda: Could you put the stack starting at on your review list for today/tomorrow, please?07:22
*** pahuang_ has quit IRC07:23
AJaegerianw: that's a simple but important change - and more comments than code ;)07:23
*** kushal has joined #openstack-infra07:24
amotokiAJaeger: glance setup.cfg does not contain [files] packages entry07:24
AJaegeramotoki: so, make the script more robust *and* fix glance?07:25
yolandahi Jaeger will take a look today07:25
AJaegerthanks, yolanda07:25
amotokiAJaeger: I think so too.07:25
amotokiAJaeger: i will send infra patch soon.07:25
AJaegersame problem for glance_store I guess. Then let me tackle those two setup.cfgs...07:26
AJaegeramotoki: ironic-ui is treated as python project: - could you check that one as well, please?07:26
*** _nadya_ has joined #openstack-infra07:27
AJaegeramotoki: ah, only glance...
AJaegerflaper87: could you put 2755339 on a speed path, please? ^07:29
amotokiAJaeger: thanks. will double check07:29
*** ianw has quit IRC07:29
*** ianw has joined #openstack-infra07:30
AJaegeramotoki: quite a few repos with broken setup.cfg - but most are not translated...07:31
*** infra-red has quit IRC07:32
*** fungi has joined #openstack-infra07:32
*** infra-red has joined #openstack-infra07:32
*** esker has joined #openstack-infra07:32
AJaegeramotoki: ignore ironic-ui, that's an old log07:33
AJaegeramotoki: so, only glance was broken07:34
amotokiAJaeger: in my local test, ironic-ui works well.07:34
AJaegeramotoki: log file was from 18th January ;( My error07:34
*** pahuang_ has joined #openstack-infra07:35
*** rlandy has quit IRC07:35
*** esker has quit IRC07:37
amotokiAJaeger: no problem :)07:37
amotokiAJaeger: are you checking failure jobs in jenkins.o.o?07:38
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual: WIP: Add information about enabling translations
AJaegeramotoki: yes, I do - - and then check the first list for failures07:39
AJaegerand also for success whether those really worked as they should...07:39
AJaegeramotoki: I updated the infra manual to mention the packages entry in 27375907:40
amotokiAJaeger: nice!07:40
*** coreycb` is now known as coreycb07:40
*** rkukura_ has joined #openstack-infra07:41
*** rkukura has quit IRC07:41
*** rkukura_ is now known as rkukura07:41
tonybcody-somerville: Once I get the base infrastructure in place I'll poke you about doign that.07:42
*** abregman has quit IRC07:47
openstackgerritAkihiro Motoki proposed openstack-infra/project-config: get-modulename: Handle no packages entry in setup.cfg
amotokiAJaeger: ^^07:49
*** esker has joined #openstack-infra07:53 proposed openstack-infra/jenkins-job-builder: Add HockeyApp Plugin support.
*** Sukhdev has quit IRC07:57
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
*** esker has quit IRC07:58
*** SumitNaiksatam has quit IRC08:00 proposed openstack-infra/jenkins-job-builder: Add HockeyApp Plugin support.
AJaegeramotoki: thanks. Looks fine and handles glance. I'm just wondering whether we need a warning message or not.08:04
*** kzaitsev_mb has joined #openstack-infra08:05
yolandaAJaeger, about, you say "re-enable django translations.." but all i see in that patch is removal of liberty and translation functions. Can you clarify that to me?08:05
*** jcoufal has joined #openstack-infra08:06
AJaegeryolando, remove thes08:07
AJaeger django_openstack_auth|designate-dashboard08:07
amotokiAJaeger: good idea even though it is not easy to find it :-(08:07
AJaegerand the upstream_translation file has the same08:07
ttxtonyb: looks good.08:07
AJaegeramotoki: yeah, I know.08:07
yolandaah i see, it removes the project disabled case08:08
ttxtonyb: we should probably refactor the yaml2ical-core list to include you, but only infra-core can08:08
amotokiAJaeger: i noticed the stdout is used in the translation shell script. stderr needs to be used. is it okay?08:08
AJaegeramotoki: yes, seems the only option..08:09
AJaegeryolanda: thanks for review08:09
*** salv-orlando has joined #openstack-infra08:10
ttxtonyb: because currently we are a bit stuck on second approval in yaml2ical08:10
openstackgerritEvgeny Antyshev proposed openstack-infra/ciwatch: Ability to select specific CI results
*** salv-orl_ has quit IRC08:12
*** ifarkas has joined #openstack-infra08:13
*** jistr has joined #openstack-infra08:14
openstackgerritAkihiro Motoki proposed openstack-infra/project-config: get-modulename: Handle no packages entry in setup.cfg
*** salv-orlando has quit IRC08:14
*** esker has joined #openstack-infra08:14
*** salv-orlando has joined #openstack-infra08:14
*** ihrachys has joined #openstack-infra08:15
*** thorst has joined #openstack-infra08:16
AJaegeramotoki: thanks. yolanda, could you review the change above as well, please? ^ That one is a bug fix for glance08:16
*** jcoufal_ has joined #openstack-infra08:16
*** rubasov has quit IRC08:16
*** esikachev has joined #openstack-infra08:16
*** boris-42 has joined #openstack-infra08:18
*** jcoufal has quit IRC08:19
*** esker has quit IRC08:19
*** amaretskiy has joined #openstack-infra08:20
*** markus_z has joined #openstack-infra08:21
*** thorst has quit IRC08:22
*** jistr is now known as jistr|mtg08:22
*** Hal has joined #openstack-infra08:23
*** Hal is now known as Guest8947208:23
*** mikelk has joined #openstack-infra08:25
*** shashank_hegde has quit IRC08:25
*** amotoki has quit IRC08:25
*** nikhil has joined #openstack-infra08:25
AJaegerthanks a lot!08:27
*** nmagnezi has joined #openstack-infra08:28
*** nikhil_k has quit IRC08:28
*** mrmartin has joined #openstack-infra08:29
*** chaitu has quit IRC08:29
*** ihrachys has quit IRC08:30
*** rockyg has joined #openstack-infra08:30
yolandano problem08:31
*** vgridnev has quit IRC08:32
*** rubasov has joined #openstack-infra08:32
*** Jeffrey4l has quit IRC08:33
nibalizerhiya yolanda AJaeger08:33
nibalizeri put up some patches on the infra-cloud topic to start using it in nodepool08:33
nibalizerid appreciate reviews08:33
yolandaah cool08:33
yolandais that fix for ansible-puppet on place?08:33
yolandanibalizer, also infra-cloud is still being refactored, for example in terms of dhcp, glean, the bridge... how are we going to live with that?08:35
esikachevttx: flaper87 hi! this is patch ready for merge
*** esker has joined #openstack-infra08:35
*** jtomasek__ has joined #openstack-infra08:35
*** fhubik has joined #openstack-infra08:36
*** ihrachys has joined #openstack-infra08:36
*** hichihara has quit IRC08:37
nibalizeryolanda: the infea cloud region i have access to works great08:37
nibalizerhas for a long time08:37
nibalizerabd with hpcloud gone we are seeing long queues08:37
yolandanibalizer yes, but for example the dhcp needs to be refactored, we are using elements there, and not glean to configure the network for example08:37
yolandawe can test that changes in east anyway08:38
*** ihrachys has quit IRC08:38
yolandaalso as i saw from Colleen yesterday, we are going to change rabbitmq to work with https08:38
*** oomichi is now known as oomichi_away08:38
AJaegernibalizer: any special topic to review?08:39
yolandaso there could be some changes, i guess some downtimes won't hurt us so much...08:39
*** bnemec has joined #openstack-infra08:39
openstackgerritMerged openstack-infra/project-config: get-modulename: Handle no packages entry in setup.cfg
nibalizerAJaeger: infra-cloud is the topic i think08:39
*** jaosorior has joined #openstack-infra08:39
nibalizeryolanda: yes if we have to take a downtime to change it we can08:39
*** yaume has joined #openstack-infra08:40
*** zul has joined #openstack-infra08:40
nibalizerbut thats no reason not to add it to puppetmaster and nodepool08:40
AJaegeresikachev: AFAIK new repos for governance need PTL+1, so ask the sahara PTL to +1 it - but your change depends-on another change that is not merged, so your change cannot go in.08:40
*** esker has quit IRC08:41
nibalizerset up a mirror, etc08:41
*** abregman has joined #openstack-infra08:41
AJaegernibalizer: what is with 268363 and 268366? Both merge conflict and same subject?08:41
yolandanibalizer ok. Main blocker i had on the automation is the missing environments, that was fixed on your ansible-puppet patch. Apart from that , all patches worked for me in east as well08:41
*** dingyichen has quit IRC08:42
*** chmouel has left #openstack-infra08:42
nibalizerAJaeger: oops. ill fix that once i get a real computer08:42
nibalizeron phone irc right bow08:42
*** matrohon has joined #openstack-infra08:43
AJaegeryolanda: there's also that has a merge conflict and you as co-author. Do you want to rebase?08:43
*** jlanoux has joined #openstack-infra08:43
*** ihrachys has joined #openstack-infra08:44
yolandanibalizer, also you need to consider that we are in process of moving these blades08:45
yolandathey are scheduling the move right now08:45
yolandaso there will be a downtime again for it, also network settings will change08:45
AJaegeryolanda: compare 246739 with - why are those two so different?08:45
ttxesikachev: yes I will process it this morning. Wanted to give a last chance to TC members to chime in after the last mod08:45
*** ihrachys has quit IRC08:46
yolandaAJaeger, hpuswest has been managed mostly by crinkle, greghaynes... they started working on it, and at that time, glean and bifrost were failing, so the setup had to be done different. When we joined to work on east, we did some fixes on glean and bifrost, so the config will be different now08:46
yolandacrinkle is working as well on some refactor of the west zone08:46
openstackgerritMasahito Muroi proposed openstack-infra/project-config: Adds a devstack test job using Congress new architecture
openstackgerritMasahito Muroi proposed openstack-infra/project-config: Adds a tox test job for Congress new architecture
AJaegeryolanda: ah, ok...08:47
yolandaeast still not ready, i need to work on some patch to create bridges08:48
*** ihrachys has joined #openstack-infra08:49
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add baremetal node definition for hpuseast
*** salv-orlando has quit IRC08:52
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add baremetal node definition for hpuseast
*** esikachev has quit IRC08:54
*** esikachev has joined #openstack-infra08:55
*** zeih has joined #openstack-infra08:55
*** ihrachys has quit IRC08:56
*** zeih has quit IRC08:56
*** zeih has joined #openstack-infra08:56
*** esker has joined #openstack-infra08:56
*** esikachev has quit IRC08:59
*** esker has quit IRC09:01
*** zhurong has quit IRC09:01
*** henrynash has joined #openstack-infra09:09
*** ihrachys has joined #openstack-infra09:10
*** ihrachys has quit IRC09:10
*** MCoLo has joined #openstack-infra09:11
*** sshnaidm has quit IRC09:12
*** achanda has quit IRC09:14
*** sdake has joined #openstack-infra09:15
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add baremetal node definition for hpuseast
*** esker has joined #openstack-infra09:18
*** achanda has joined #openstack-infra09:18
*** chlong has quit IRC09:20
*** thorst has joined #openstack-infra09:21
*** markus_z has quit IRC09:22
*** esker has quit IRC09:23
*** jordanP has joined #openstack-infra09:24
*** achanda has quit IRC09:24
*** dguitarbite has quit IRC09:25
*** dizquierdo has joined #openstack-infra09:25
*** dguitarbite has joined #openstack-infra09:27
*** thorst has quit IRC09:28
*** keedya has quit IRC09:29
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add baremetal node definition for hpuseast
*** esp_ has joined #openstack-infra09:32
*** tcammann_ has joined #openstack-infra09:33
*** NobodyCa1 has joined #openstack-infra09:34
*** dtantsur|afk is now known as dtantsur09:37
*** dguitarbite has quit IRC09:37
*** tcammann_ has quit IRC09:38
*** esp_ has quit IRC09:39
*** NobodyCa1 has quit IRC09:40 proposed openstack-infra/jenkins-job-builder: Add HockeyApp Plugin support.
*** sshnaidm has joined #openstack-infra09:44
*** e0ne has joined #openstack-infra09:45
*** tzn has joined #openstack-infra09:46
*** dguitarbite has joined #openstack-infra09:47
yolandanibalizer, i reviewed your patches09:48
*** kzaitsev_mb has quit IRC09:49
*** AJaeger has quit IRC09:49
yolandaone thing, it will be nice if we can use the same cacert for east and west ... but i need the private key to sign the certs for east using tha CA. Can you or crinkle send me?09:49
*** vgridnev has joined #openstack-infra09:51
*** zeih has quit IRC09:53
*** jaosorior has quit IRC09:53
*** jaosorior has joined #openstack-infra09:54
*** dizquierdo has quit IRC09:57
*** gildub has joined #openstack-infra09:58
*** esker has joined #openstack-infra09:59
*** vivekd has joined #openstack-infra10:01
*** jordanP has quit IRC10:01
*** jordanP has joined #openstack-infra10:01
*** rossella_s has quit IRC10:02
*** rossella_s has joined #openstack-infra10:03
*** zul has quit IRC10:03
*** esker has quit IRC10:04
*** nijaba has quit IRC10:05
*** e0ne has quit IRC10:05
*** zeih has joined #openstack-infra10:06
*** zul has joined #openstack-infra10:06
openstackgerritMerged openstack-infra/project-config: Fix some word spellings
*** jaosorior has quit IRC10:09
*** zeih_ has joined #openstack-infra10:10
*** rguillebert has joined #openstack-infra10:10
*** AJaeger has joined #openstack-infra10:11
*** zeih has quit IRC10:11
*** nijaba has joined #openstack-infra10:12
*** vgridnev_ has joined #openstack-infra10:14
*** vgridnev has quit IRC10:14
*** rhallisey has joined #openstack-infra10:14
*** jaosorior has joined #openstack-infra10:16
*** salv-orlando has joined #openstack-infra10:16
openstackgerritMerged openstack-infra/project-config: Adding gerritbot and accessbot support for openstack-product
openstackgerritMerged openstack-infra/project-config: Switch ansible-role-diskimage-builder jobs voting
*** ociuhandu has joined #openstack-infra10:18
nibalizeryolanda: you are a rooter and can log into :)10:19
yolandanibalizer, i can get the data from hiera, but i need to create my own certs for hpuseast nodes, using that cacert10:20
yolandaand i guess it's protected with key to sign? or not?10:20
yolandathe password i mean10:20
*** salv-orlando has quit IRC10:21
nibalizeroh i don't know10:24
nibalizercrinkle: is probably the only one who knows10:24
yolandaok i'll ping her, i'd prefer if we use same cacert in both regions for simplicity10:25
yolandaalso, in terms of dns, how are you going to manage? add the 100 entries to our openstack dns ? in east we have 100 nodes10:25
*** thorst has joined #openstack-infra10:26
nibalizeri think we added the controller to main dns and the controller runs a dnsmasq that knows the other 100 ips10:27
nibalizerthat maybe isn't ideal but its a thing10:27
*** ildikov has quit IRC10:27
*** ldnunes has joined #openstack-infra10:27
*** yamamoto has quit IRC10:27
*** hashar has joined #openstack-infra10:28
yolandanibalizer, that's confusing me. Because nameservers in controller or compute are not pointing to controller dnsmasq, but to google10:28
*** fhubik is now known as fhubik_brb10:29
*** fhubik_brb is now known as fhubik10:29
*** thorst has quit IRC10:33
*** mrmartin has quit IRC10:33
yolandanibalizer also... we need to run puppet apply on the compute nodes so puppetmaster needs to know how to resolve these compute servers addresses right?10:37
*** khappone has quit IRC10:38
*** khappone has joined #openstack-infra10:38
*** rfolco has joined #openstack-infra10:40
*** zeih_ has quit IRC10:40
*** aysyd has joined #openstack-infra10:40
*** esker has joined #openstack-infra10:41
*** Daisy has joined #openstack-infra10:42
nibalizerso it will probably all end up in global dns10:42
*** d0ugal has quit IRC10:42
*** d0ugal has joined #openstack-infra10:42
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add support for --overcloud-update and use it for HA job
*** esker has quit IRC10:45
*** fhubik is now known as fhubik_brb10:46
*** fhubik_brb is now known as fhubik10:47
*** fhubik is now known as fhubik_brb10:47
*** aysyd has quit IRC10:47
*** aysyd has joined #openstack-infra10:48
*** abregman is now known as abregman|lunch10:48
*** zeih has joined #openstack-infra10:49
*** fhubik has joined #openstack-infra10:50
yolandanibalizer so i think it will be better if we complete that requiremetns in term of dns, controlling everything with puppet, etc... then enable the cloud on nodepool10:50
*** fhubik_brb has quit IRC10:50
*** FallenPegasus has joined #openstack-infra10:51
*** e0ne has joined #openstack-infra10:51
*** vgridnev_ has quit IRC10:52
*** vgridnev_ has joined #openstack-infra10:54
*** jaosorior has quit IRC10:58
*** electrofelix has joined #openstack-infra10:58
AJaegerwe seem to have some problems with an apt-mirror, see
*** jaosorior has joined #openstack-infra10:59
*** deepakcs has quit IRC10:59
*** zul has quit IRC11:00
AJaegerThis seems to happen with many jobs - didn't have time to look further into it.11:00
AJaegerFails on rax-iad, works fine on ovh-bhs111:00
*** zul has joined #openstack-infra11:01
AJaegerworks on internap11:01
*** tiswanso has quit IRC11:01
*** esker has joined #openstack-infra11:02
AJaegerbut that's a small sample only.11:02
*** zeih has quit IRC11:03
yolandaerrors are not showing the real failure of apt_update, looks as a timeout11:03
sdaguehmm, so all mirrors are broken now?11:03
AJaegersdague: rax-iad only11:04
AJaegerBut we need more data ;)11:04
AJaegerJust looked at 5 more fails - all rax-iad11:04
AJaegerSorry, can't look into this further now11:05
*** esker has quit IRC11:07
*** salv-orlando has joined #openstack-infra11:07
*** annegentle has joined #openstack-infra11:08
*** infra-re_ has joined #openstack-infra11:09
AJaegersdague: could you review , please?11:10
*** dizquierdo has joined #openstack-infra11:10
yolandaseems we are having network problems11:11
yolanda2016-02-03 10:32:18.496 |   Resolving (   2600:3c03::f03c:91ff:fedb:6b1d,
yolanda2016-02-03 10:34:25.759 |   Connecting to (|2600:3c03::f03c:91ff:fedb:6b1d|:80...   failed: Connection timed out.11:11
yolandathat also on rax-iad11:12
*** annegentle has quit IRC11:12
sdagueah, their ipv6 path is bonkers?11:13
sdagueis there a rax status page up?11:13
*** infra-red has quit IRC11:13
*** deepakcs has joined #openstack-infra11:13
yolandai'm not aware of problems but i'm not getting any emails from providers, is something that i'll need to work on to get subscribed to11:13
sdaguejohnthetubaguy: any idea if rax-iad is basically toast ^^^ ?11:13
*** sfinucan has joined #openstack-infra11:13
johnthetubaguysdague: taking a peak now11:14
*** arxcruz has joined #openstack-infra11:14
openstackgerritMerged openstack-infra/project-config: Remove redundant tempest job for heatclient
*** gnuoy_ has joined #openstack-infra11:20
johnthetubaguysdague: so I am just seeing roses for the infra tenant right now, but there could be a little time delay11:21
*** jaosorior has quit IRC11:21
johnthetubaguysdague: oh, so you are seeing networking related issues?11:21
*** infra-red has joined #openstack-infra11:22
*** esker has joined #openstack-infra11:23
sdaguewe can't hit any of the apt mirrors11:23
*** infra-re_ has quit IRC11:24
openstackgerritIgor Belikov proposed openstack-infra/puppet-jenkins: Use 'ruby' instead of 'ruby1.9.1' for Debian
*** jistr|mtg has quit IRC11:27
*** esker has quit IRC11:28
*** gnuoy_ has quit IRC11:29
*** infra-re_ has joined #openstack-infra11:30
johnthetubaguysdague: I guess its just a subset of the hosts in iad? I am just trying to find a uuid or something in the logs11:30
*** abregman|lunch is now known as abregman11:30
sdagueyolanda: can you get johnthetubaguy more info?11:30
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Add a non-voting inspector job for ironic-python-agent
yolandalet me check logs11:31
johnthetubaguythere was talk of one of the iad cells having networking issues, but just hoping more info will help them pin things down11:31
*** thorst has joined #openstack-infra11:32
*** yamamoto has joined #openstack-infra11:32
yolandai have a pair of errors with timeouts but cannot get the server ids from there, these are temporary nodepool servers11:33
*** infra-red has quit IRC11:33
*** yamamoto has quit IRC11:33
sdagueyolanda: is node uuid from nodepool or openstack?11:35
sdagueNode UUID: 368d44c3-3dc4-4801-8c80-33f0beb28ef111:35
sdaguein the console logs11:35
yolandaah that's better11:36
yolandajohnthetubaguy, that helps more?11:37
johnthetubaguyah, console logs, perfect, let me look that one up11:38
*** thorst has quit IRC11:38
sdaguewell, jenkins console log11:38
openstackgerritIgor Belikov proposed openstack-infra/system-config: Get debian kernel headers based on architecture
*** deepakcs has quit IRC11:39
openstackgerritIgor Belikov proposed openstack-infra/puppet-jenkins: Use 'ruby' instead of 'ruby1.9.1' for Debian
AJaegerjohnthetubaguy: do you need more console logs? Thanks for looking into this!11:42
openstackgerritIgor Belikov proposed openstack-infra/project-config: Add debian-jessie image to nodepool
*** boris-42 has quit IRC11:43
*** vivekd_ has joined #openstack-infra11:44
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Add a non-voting inspector job for ironic-python-agent
*** mrmartin has joined #openstack-infra11:45
*** vivekd has quit IRC11:45
*** vivekd_ is now known as vivekd11:45
johnthetubaguyAJaeger: yolanda: sdague: unsure whats needed, asking the support folks, its not in the cell they were wondering about, so I think they are digging11:46
*** kzaitsev_mb has joined #openstack-infra11:46
*** _amrith_ is now known as amrith11:53
johnthetubaguyyolanda: so I am out of date here, do we reconfigure the VM not to use the rackspace mirror, so I guess its hitting the default ubuntu address?11:54
yolandajohnthetubaguy i need to double check but yes, i think it uses default ubuntu address11:54
*** zeih has joined #openstack-infra11:54
yolandalet me hold a node to confirm11:54
*** deepakcs has joined #openstack-infra11:55
*** deepakcs has quit IRC11:55
*** pkoniszewski has quit IRC11:56
*** pkoniszewski has joined #openstack-infra11:56
yolandayep, i see there, on apt sources11:56
AJaegerI think we do - see nodepool/scripts/ in project-config11:57
yolandayes, it connects to us.archive, and actually my holded node is timing out11:57
yolandajohnthetubaguy, i have a holded vm with that problem, so if you need to debug something it can be useful11:58
*** jcoufal_ has quit IRC11:58
johnthetubaguyyolanda: that could be useful, but I just updated the sources.list and I think I see errors talking to now11:59
sdagueipv6 only?11:59
yolandai can ping succsesfully with ipv4, but not with ipv612:00
yolandaping us.archive.ubuntu.com12:00
yolandaPING ( 56(84) bytes of data.12:00
yolanda64 bytes from ( icmp_seq=1 ttl=51 time=14.3 ms12:00
*** vivekd_ has joined #openstack-infra12:00
yolandaping6 us.archive.ubuntu.com12:00
yolandaPING 56 data bytes12:00
sdagueright, so it's an ipv6 fail12:00
yolandalooks like12:00
sdagueI think that's the important bit12:00
sdagueto help rax narrow it12:00
yolandaapt-get -o Acquire::ForceIPv4=true update works12:02
yolandaso yes, definitely ipv6 problems12:02
AJaegerIs there an easy way for us to stop using rax-iad or should we wait?12:03
*** julim_ has quit IRC12:03
yolandai can set the max-servers to zero12:03
yolandaso we stop using that12:03
sdagueyolanda: yeh, it's probably worth doing that12:03
yolandaok going to send  a change12:03
sdaguebecause we're just auto failing all changes at this point12:03
AJaegeryes, better than all the fails and we can easily revert...12:04
*** vivekd has quit IRC12:04
*** vivekd_ is now known as vivekd12:04
*** esker has joined #openstack-infra12:04
openstackgerrityolanda.robla proposed openstack-infra/project-config: Set max-servers to 0 to rax-iad
yolandalast time i looked, puppet on nodepool was disabled, going to double check if that's still the case12:05 proposed openstack-infra/jenkins-job-builder: Add HockeyApp Plugin support.
AJaegeryolanda: thanks. Once that's in we can prepare a revert and mark it WIP...12:06
*** ildikov has joined #openstack-infra12:06
sdaguethough, that does drop our works by 1/3 which sucks12:07
sdaguebut sucks less than everything dying12:07
*** zeih_ has joined #openstack-infra12:07
AJaegersdague: exactly12:07
yolandachange has lots of chances of not passing, so going to apply on nodepool directly12:08
johnthetubaguyyeah, failed to get a decent answer, so I have raised a ticket12:09
*** jcoufal has joined #openstack-infra12:09
*** esker has quit IRC12:09
sdaguewe could also do this to all our nodes - to prefer v4 over v612:10
johnthetubaguyI basically confirmed only in IAD, you can't do the IPv6 stuff to access doing apt-get update, so its nice a reproducible, it works just fine in ORD12:10
AJaegersdague, yolanda : On the other hand I see failures only for devstack-trusty12:10
*** ihrachys has joined #openstack-infra12:10
AJaegerWe could also keep bare-trusty on iad and only disable devstack-trusty.12:10
sdagueAJaeger: we could, I don't know all the details in what that fallout does12:10
yolandaAJaeger, i bet i saw one failing on bare12:10
yolandalet me double check12:11
*** zeih has quit IRC12:11
sdagueAJaeger: do you have a logstash signature for this?12:11
AJaegersdague: nope ;(12:11
*** julim has joined #openstack-infra12:11
AJaegersdague: do you want to create one from
*** links has quit IRC12:11
yolandamm, it was devstack as well12:11
johnthetubaguyso leaving the rackspace mirror does seem to work, but that opens up a whole other can of worms12:12
sdaguemessage:"Failed to update apt repos, we're dead now"12:13
yolandaupdating the mirrors to use rax is not a simple change for us12:13
*** ihrachys has quit IRC12:13
yolandaas we'd need to do depending on providers12:13
yolandait may be easier to send a change to enforce ipv4 in apt12:14
sdagueyolanda: if ipv6 is down apt is just one place it will fail12:15
yolandayes, we may see another issues12:15
*** zz_dimtruck has quit IRC12:15
johnthetubaguyso I am seeing wget fail, and curl work12:16
johnthetubaguyI guess that could be the ip v4 vs v6 thing?12:16
*** fhubik is now known as fhubik_brb12:17
yolandayes, curl -6 fails as well12:17
AJaegersdague: so, only devstack-trusty - but there might be different errors for other types12:17
yolandafailures shoud be happening independently of the label12:17
yolandaAJaeger, it may be due to the jobs executed per label12:18
johnthetubaguyso contacting google via ipv6 works just fine12:18
yolandado you see apt updates on some of the bare ones?12:18
sdaguethe signature on bare nodes will be different, this is a devstack function that builds a good summary12:18
AJaegersdague: It's just that my jobs that failed only failed in the dsvm tests but not in others. But that's a small sample...12:19
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: failure for apt mirror
yolandai set max-servers to 0 manually at the meantime12:20
AJaegerthanks, yolanda12:20
sdagueright, that will at least keep a few things running12:20
*** weshay_xchat has joined #openstack-infra12:21
yolandareal change passed check now12:21
*** zul has quit IRC12:24
openstackgerritMerged openstack-infra/project-config: Set max-servers to 0 to rax-iad
*** zeih has joined #openstack-infra12:25
*** gildub has quit IRC12:25
AJaegerlet me do a revert as reminder...12:25
*** lucasagomes is now known as lucas-hungry12:25
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Revert "Set max-servers to 0 to rax-iad"
yolandagoing to reenable puppet on nodepool as the real change landed12:27
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Source undercloud environment variable from a file
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Split the deploy script into its own file
yolandajohnthetubaguy, i also can ping with v6 to different hosts, but not to canonical one12:27
*** zeih_ has quit IRC12:28
johnthetubaguyyolanda: yeah, seeing the same thing12:28
johnthetubaguyI added notes in the support ticket, I hope they don't ignore mine because its an internal account12:28
* johnthetubaguy runs away for some lunch12:29
*** trown|outttypeww is now known as trown12:29
sdagueyolanda: how long until that kicks in?12:30
*** ociuhandu_ has joined #openstack-infra12:30
AJaegersdague: it's active already, see our graphfs12:30
yolandayes, that happened12:30
*** rhallisey has quit IRC12:31
*** dims has joined #openstack-infra12:32
AJaegerdtantsur: is still relevant? In that case let's ask sdague or yolando for a +2...12:33
*** mrmartin has quit IRC12:34
*** ociuhandu has quit IRC12:34
*** ociuhandu_ is now known as ociuhandu12:34
dtantsurAJaeger, yep, still relevant, I've forgot about it. thanks for reminder12:34
yolandawill approve12:34
*** jpr has joined #openstack-infra12:34
AJaegerdtantsur: could you review for me, please? It's ironic related change12:35
*** rhallisey has joined #openstack-infra12:35
yolandaAJaeger, sdague, are you normally sending status alerts, when it's not a failure, but something affecting to capacity? such as "jobs will run slower"12:35
*** rhallisey has quit IRC12:35
*** zeih has quit IRC12:36
*** rhallisey has joined #openstack-infra12:36
*** zeih has joined #openstack-infra12:36
* AJaeger does not know12:36
* dtantsur looks12:37
AJaegeryolanda: you have the power now, make use of it to double check it.12:37
*** amrith is now known as _amrith_12:37
AJaegerand a #status notice would be fine in this situation...12:37
*** jpr has quit IRC12:38
yolanda#status notice Infra running with lower capacity now, due to a temporary problem affecting one of our nodepool providers. Please expect some delays in your jobs. Apologies for any inconvenience caused.12:39
openstackstatusyolanda: sending notice12:39
*** mase_x200 has joined #openstack-infra12:39
openstackgerritMerged openstack-infra/elastic-recheck: failure for apt mirror
-openstackstatus- NOTICE: Infra running with lower capacity now, due to a temporary problem affecting one of our nodepool providers. Please expect some delays in your jobs. Apologies for any inconvenience caused.12:40
yolandaAJaeger, sdague, i'm going to have lunch now, but will be back in a while12:40
*** baoli has joined #openstack-infra12:40
*** zeih_ has joined #openstack-infra12:40
AJaegeryolanda: get yourself something yummy ;) Thanks!12:40
openstackstatusyolanda: finished sending notice12:41
openstackgerritMerged openstack-infra/project-config: Switch ironic-lib and python-ironicclient gates to IPA
*** zeih has quit IRC12:42
*** baoli_ has joined #openstack-infra12:43
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Reduce timeout of ironic jobs
AJaegerdtantsur: thanks for review, updated12:44
*** baoli has quit IRC12:45
*** esker has joined #openstack-infra12:46
*** thorst_ has joined #openstack-infra12:46
*** henrynash has quit IRC12:50
*** esker has quit IRC12:51
*** zeih_ has quit IRC12:52
*** daemontool has joined #openstack-infra12:52
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add support for --overcloud-update and use it for HA job
*** zeih has joined #openstack-infra12:53
*** julim has quit IRC12:54
*** Jeffrey4l has joined #openstack-infra12:55
*** zul has joined #openstack-infra12:55
*** julim has joined #openstack-infra12:57
*** jaosorior has joined #openstack-infra12:57
*** dguitarbite has quit IRC12:57
*** jpr has joined #openstack-infra12:58
*** rlandy has joined #openstack-infra13:00
openstackgerritValeriy Ponomaryov proposed openstack-infra/project-config: Add new experimental job for manila-image-elements project
*** openstackgerrit has quit IRC13:02
*** annegentle has joined #openstack-infra13:02
*** openstackgerrit has joined #openstack-infra13:03
openstackgerritValeriy Ponomaryov proposed openstack-infra/project-config: Add new experimental job for manila-image-elements project
*** fhubik_brb is now known as fhubik13:05
*** gildub has joined #openstack-infra13:06
*** annegentle has quit IRC13:07
*** ihrachys has joined #openstack-infra13:08
openstackgerritIgor Belikov proposed openstack-infra/puppet-jenkins: Use 'ruby' instead of 'ruby1.9.1' for Debian
nibalizerso our rax-iad lost ipv6?13:09
smcginnisFYI, the nice warning that things are lower capacity didn't get sent the cinder channel. Not sure why, but just thought I should point that out.13:11
nibalizersmcginnis: thanks13:12
nibalizerthats #openstack-cinder ya?13:12
smcginnisnibalizer: Yep13:12
smcginnisScanning through the channels I'm in, some got it and some didn't.13:12
*** jistr|mtg has joined #openstack-infra13:12
nibalizeryolanda: i have ssh'd into your help rax-iad node fyi13:13
*** claudiub has joined #openstack-infra13:13
*** moravec has quit IRC13:14
nibalizerI can't see the ipv6 fails?13:15
*** moravec has joined #openstack-infra13:16
AJaegernibalizer: ping6 is the fail, not a general fail13:20
AJaegersmcginnis: let me check something for you...13:20
krotscheck1AJaeger: Thanks for all those +2's yesterday :)13:20
pabelangerAJaeger: left a comment on the review, let me know if it is easier to chat here.13:21
*** tiswanso has joined #openstack-infra13:21
*** samuelBartel has joined #openstack-infra13:21
AJaegersmcginnis: #openstack-cinder looks setup correctly, hope somebody else can debug the issue why statusbot does not show up at your channel13:22
AJaegerkrotscheck1: you're welcome13:22
AJaegerpabelanger: let me check your latest comment13:22
*** xyang1 has joined #openstack-infra13:23
AJaegerpabelanger: yes, this should work. Gave a +2. Thanks for explaining13:23
AJaegersmcginnis: check - there'S teh status message13:23
pabelangerAJaeger: great! Thanks for the follow up13:24
nibalizerAJaeger: nibz@bare-trusty-rax-iad-7675327:~$ ping6 us.archive.ubuntu.com13:25
nibalizerPING 56 data bytes13:25
nibalizerseems working?13:26
nibalizer64 bytes from icmp_seq=1 ttl=50 time=13.3 ms13:26
nibalizer64 bytes from icmp_seq=2 ttl=50 time=13.2 ms13:26
sdaguenibalizer: maybe it got resolved?13:26
sdagueit was definitely failing for folks earlier13:26
nibalizer doesn't show any isuse13:26
*** jsavak has joined #openstack-infra13:26
AJaegerlet's wait for yolanda to return from lunch and test - and then enable iad again...13:27
nibalizerwe could also bring it up to 10 nodes13:27
*** zeih has quit IRC13:27
nibalizeror something like that13:27
sdaguenibalizer: we know, see the conversation with johnthetubaguy earlier13:27
johnthetubaguysdague: it does seem to be fixed now13:27
sdaguejohnthetubaguy: ok13:28
pabelangerAJaeger: odyssey4me: So, easier to ask here. Is tox -elinters going to be a default now?13:28
sdaguejohnthetubaguy: any idea why rax didn't notice it themselves?:13:28
pabelangerAJaeger: odyssey4me: for all projects?13:28
zigopabelanger: Hi there! Could you review this one again?
bauzasjohnthetubaguy: sdague: cool, do that mean we can recheck ?13:28
johnthetubaguysdague: it seems to have been isolated to ubuntu, AFAIK, and all our images use our mirror13:28
AJaegerpabelanger: this would need some more discussion - we have enabled it as option and let teams choose...13:28
odyssey4mepabelanger it's an option available, but won't be forced - many projects are consolidating linting under the pep8 tox target, but 'linters' seems more appropriate for our needs13:28
johnthetubaguysdague: I don't think we have done any changes on our side13:28
johnthetubaguysdague: but honestly, not 100% sure13:29
*** gordc has joined #openstack-infra13:29
johnthetubaguysdague: the reply to my ticket was "hmm, this sounds familiar, let me go check with someone"13:29
*** lucas-hungry is now known as lucasagomes13:29
*** ihrachys has quit IRC13:30
AJaegersdague, nibalizer, yolanda : If you want to +2A the revert:
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Reduce timeout of ironic jobs
johnthetubaguyyeah, the VM I reproduced this one seems to have fully recovered, sorry about that, not sure what way, I have asked support to let me know what changed13:31
pabelangerzigo: 1 question, left in comments13:32
pabelangerzigo: otherwise, code looks good. Just reviewing build log13:32
*** zeih has joined #openstack-infra13:32
AJaegersdague: could you review 272481 and as well, please? - two small cleanups...13:32
*** ihrachys has joined #openstack-infra13:32
sdagueI don't understand why the timeout drops13:33
*** zeih has quit IRC13:34
*** zeih has joined #openstack-infra13:34
AJaegersdague: It was just a copy from earlier jobs in that file - and those use 120...13:34
zigopabelanger: Cheers !13:34
nibalizerinfra-root taking the hiera lock13:35
* AJaeger wants to stop the cargocult of copying always 2hours there...13:35
*** mrmartin has joined #openstack-infra13:35
sdagueAJaeger: sure, but plenty of headroom is fine13:35
sdaguewhen people over optimize this stuff, they then add tests, and start random failing over the timeout13:35
sdagueand keep rechecking because they don't realize what's going on13:36
pabelangerzigo: no issues uploading the images to rax or ovh?13:36
sdaguekeeping a big buffer in there is fine13:36
pabelangerzigo: eg: networking works13:36
*** zeih_ has joined #openstack-infra13:36
sdaguethe big buffer only hurts if tests are deadlocking regularly, and we need to whack things because of it13:36
nibalizerinfra-root removing lock13:36
zigopabelanger: redirects to the closest mirror, why is it then a problem to not use It will normally select best location ...13:36
AJaegersdague: I thought our default is 60 mins - and those jobs never run longer than 60 mins (rarely 50), so I put 70 in...13:37
sdaguehonestly I'd almost refactor the per job timeout into a global thing for 3 hours, and be fine with it.13:37
zigopabelanger: If the selected mirror is wrong, we can always ask Raphael Guesler (the maintainer of the service) to fix it.13:37
sdagueright, but the only reason you care about job timeout is deadlock13:37
sdaguethe job is wedge so that it will never end13:37
sdaguethat is *such* a rare occurance, you just want a backstop so it can't happen13:37
pabelangerzigo: mostly because it is random. There have been issues with mirrors in the past, pinning it to a specific mirror just helps troubleshooting. Once we have AFS mirrors, it won't be an issue.13:37
sdaguebecause it would get stuck forever13:38
AJaegersdague: Indeed - but isn't it also a question of not having too long running jobs?13:38
zigopabelanger: Though this way, it's going to select the US mirrors even in France with OVH, that's bad, no?13:38
*** edmondsw has joined #openstack-infra13:38
zigopabelanger: Also, "" is a moving target, it's not a single repo, as much as I know.13:38
sdagueAJaeger: what is going to happen if the job starts tripping a timeout?13:38
zigopabelanger: Therefore, which mirror do you suggest?13:39
*** daemontool has quit IRC13:39
sdaguesomeone is going to push a timeout bump, and we're going to accept it13:39
sdagueso reducing timeouts is all just random busywork13:39
zigopabelanger: is quite fast, and hosted by a serious company (the 2nd largest ADSL provider in France), and it uses a cluster of servers in the backend.13:39
pabelangerzigo: In theory, either way just a comment, not a -1.  You can default to httpredir for now, but keep in mind of the pinning.  I'll let other reviewers comment too13:39
*** zeih has quit IRC13:40
zigopabelanger: I trust you, and I prefer to change the patch to do what you suggest. I'm just trying to find the best way here.13:40
zigopabelanger: So, would work?13:40
AJaegersdague: Ok, let me double check with others later. I'll mark the change as WIP for now - and might change my review pattern...13:40
AJaegerthanks, sdague13:40
zigo(it's very close for OVH...)13:40
pabelangerzigo: That's the thing, I don't know which mirror is the best.  I'm just saying we might want to pick one for all clouds. Last I checked, we pinned everything to a US mirror.  But, I might be wrong.13:41
*** mase_x200 has quit IRC13:42
pabelangerzigo: Either way, I don't see an issue trying out httpredir to start with13:42
zigopabelanger: As much as I know, for Ubuntu, it's easier to choose, as Canonical sponsors them and make sure they are in good shape. For Debian, it's all sponsored ...13:42
zigopabelanger: Ok. then +2 the patch? :)13:42
pabelangerzigo: can't, not core13:43
zigooh ...13:43
pabelangerjust providing feedback13:43
zigopabelanger: Thanks a lot.13:44
zigopabelanger: I was wondering, could you point me to a patch which adds a check job, so that I can write my build script env script + build packages?13:45
zigoOr to the relevant doc if any...13:46
*** annegentle has joined #openstack-infra13:46
*** tiswanso has quit IRC13:46
yolandahi, so rax working again?13:46
*** vivekd has quit IRC13:46
pabelangerzigo: look into jenkins/jobs/infra.yaml, they are pretty decent. gate-openstackci-beaker-{ostype}-dsvm provides a good mix of JJB syntax13:48
AJaegeryolanda: we hope so;)13:48
zigopabelanger: Cheers.13:48
*** Jeffrey4l has quit IRC13:48
yolandawe didn't take any action from infra side to solve?13:48
smcginnisAJaeger: Thanks. That's really odd. It doesn't show up in my client, but there it is in the eavesdrop. Huh.13:49
pabelangerzigo: I did have some code up a while ago for puppet: but abandoned it.13:49
*** Jeffrey4l has joined #openstack-infra13:49
pabelangerzigo: all depends where you want things to live I guess. Adding into JJB is a good start13:49
nibalizeryolanda: i didn't do anything13:50
nibalizeri logged into your held node and ipv6 seems to work13:50
nibalizerand johnthetubaguy says rax says that its fine13:50
yolandawell, that's nice13:50
*** trown is now known as trown|afk13:50
pabelangernibalizer: the best kind of fixing things!13:50
*** fhubik is now known as fhubik_brb13:51
*** infra-red has joined #openstack-infra13:52
*** amitgandhinz has joined #openstack-infra13:53
*** infra-re_ has quit IRC13:54
*** claudiub has quit IRC13:55
*** fhubik_brb is now known as fhubik13:56
*** amotoki has joined #openstack-infra13:57
AJaegeryolanda: 275666 was approved and is in the queue - but not getting a node right now ;/13:57
*** erlon has joined #openstack-infra13:57
*** isaacb has joined #openstack-infra13:57
*** ihrachys has quit IRC14:01
sdagueAJaeger: yep, the joys of not enough nodes14:02
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add omfracloud to nodepool
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add OmfraCloud to puppetmaster_clouds
*** rossella_s has quit IRC14:02
*** Kevin_Zheng has quit IRC14:02
*** rossella_s has joined #openstack-infra14:03
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add omfracloud to nodepool
openstackgerritSpencer Krum proposed openstack-infra/project-config: Use omfracloud in nodepool
*** julim has quit IRC14:05
*** annegentle has quit IRC14:06
*** henrynash has joined #openstack-infra14:06
*** alivigni has joined #openstack-infra14:06
yolandanibalizer, i'm curious? why "omfracloud" ? :)14:06
pabelangeryolanda: do you mind helping review: 27547114:07
yolandapabelanger sure14:08
nibalizeryolanda: it was a typo from jeblair I think, and it sortof stuck14:08
pavel_bondarsdague: hi14:08
*** zeih_ has quit IRC14:08
nibalizerthat patch is an appropriate place to say that we chould call int infracloud14:08
*** skraynev has quit IRC14:08
*** isaacb has quit IRC14:09
*** zeih has joined #openstack-infra14:09
*** _amrith_ is now known as amrith14:09
yolandaah, much better with regions now14:09
yolandaonly 4 max-servers?14:09
openstackgerritMerged openstack-infra/project-config: Revert "Set max-servers to 0 to rax-iad"
yolandawe have over 40 nodes in west?14:10
openstackgerritMarton Kiss proposed openstack-infra/groups: Fix broken map on frontpage
nibalizeryolanda: just a number to get us started14:10
nibalizerwe probably don't want to go hog wild at the start14:10
*** daemontool has joined #openstack-infra14:11
*** thiagop has joined #openstack-infra14:11
*** yamamoto has joined #openstack-infra14:11
openstackgerritMarton Kiss proposed openstack-infra/groups: Fix broken map on frontpage
*** kgiusti has joined #openstack-infra14:14
pabelangergood news everybody! python-grafyaml has been uploaded to fedora rawhide. That is all14:14
*** trown|afk is now known as trown14:16
*** amotoki has quit IRC14:17
*** dkranz has joined #openstack-infra14:18
nibalizerpabelanger: neat14:19
*** dprince has joined #openstack-infra14:20
*** jistr|mtg is now known as jistr14:20
*** amotoki has joined #openstack-infra14:21
openstackgerritMerged openstack-infra/groups: Fix broken map on frontpage
*** tiswanso has joined #openstack-infra14:24
*** peristeri has joined #openstack-infra14:25
*** vgridnev_ has quit IRC14:27
*** daemontool has quit IRC14:27
*** vgridnev_ has joined #openstack-infra14:27
*** sabeen3 has joined #openstack-infra14:28
pabelangeryolanda: fungi: Odd, another review that has +1 workflow and seems to be stuck: 27547114:29
openstackgerritMerged openstack-infra/project-config: Remove jobs from gantt and python-ganttclient
fungipabelanger: why do you say that one's "stuck"?14:31
fungilooks like it entered the gate 20 minutes ago when yolanda approved it14:31
pabelangerfungi: wait, so why didn't zuul leave a comment?14:31
*** ihrachys has joined #openstack-infra14:31
fungiit did--that's how i know14:32
*** tiswanso has quit IRC14:32
pabelanger /faceplam14:32
pabelangertoggle CI14:32
*** tiswanso has joined #openstack-infra14:32
pabelangerfungi: sorry for the noise14:32
*** burgerk has quit IRC14:33
*** skraynev has joined #openstack-infra14:35
anteayaclarkb: ah floating ip exhaustion, thank you14:36
*** Swami has joined #openstack-infra14:36
anteayazaro: yes figuring out what is going on with gerrit is useful, thanks for the explanation14:36
anteayaarmax: I'm here not but you are probably not, have to run errands in about an hour that will likely take me most of my morning, I will look for you upon my return14:37
*** ihrachys has quit IRC14:37
*** annegentle has joined #openstack-infra14:38
*** annegentle has quit IRC14:38
*** annegentle has joined #openstack-infra14:39
*** regXboi has joined #openstack-infra14:41
*** fhubik is now known as fhubik_brb14:43
*** fhubik_brb is now known as fhubik14:43
*** yamamoto has quit IRC14:43
*** yaume_ has joined #openstack-infra14:44
openstackgerritMerged openstack/requirements: Change OSprofiler constraints to >=1.0.0
openstackgerritMerged openstack-infra/project-config: Always clone openstack/windmill for jobs
openstackgerritMikhail S Medvedev proposed openstack-infra/ciwatch: Ability to select specific CI results
*** yamamoto has joined #openstack-infra14:47
*** tongli has joined #openstack-infra14:47
*** yaume has quit IRC14:47
*** julim has joined #openstack-infra14:48
*** fhubik is now known as fhubik_brb14:49
*** fhubik_brb is now known as fhubik14:49
*** anteaya has quit IRC14:49
*** zeih has quit IRC14:51
*** yamamoto_ has joined #openstack-infra14:52
*** yamamoto has quit IRC14:52
*** anteaya has joined #openstack-infra14:52
*** annegentle has quit IRC14:53
*** Daisy has quit IRC14:53
*** Daisy has joined #openstack-infra14:54
*** mriedem has joined #openstack-infra14:54
*** jsavak has quit IRC14:55
anteayawould it make sense to have the mirrors, such as this one:, in cacti?14:55
*** Daisy has quit IRC14:55
*** jsavak has joined #openstack-infra14:56
fungii thought they were14:56
* anteaya looks again14:56
fungilooks like their predecessors are present14:57
*** fhubik is now known as fhubik_brb14:57
anteayaso they are14:57
fungii guess the migration involved building replacements14:57
*** yamamoto_ has quit IRC14:57
fungiso anyway, yeah if there's not already a change in flight to take those out of cacti and add the new ones, then we'll get one in shortly14:59
krotscheck1Does that particular instance have a cinder volume attached? It seems like disk usage has been creeping up for the last few days, and I don't know of anything that would do that other than the mirror cache.14:59
*** pradk has joined #openstack-infra14:59
krotscheck1I _think_ it does.14:59
krotscheck1(because it has the /cache mount.14:59
krotscheck1But that one' snot moving, while the on-disk is.15:00
anteayaoh, I guess I don't know what the new ones are15:00
fungioh, wait the new ones are in the list15:00
fungialong with the old ones15:00
krotscheck1mirror.* are the new ones. pypi.* are the old ones.15:00
fungikrotscheck1 just linked to one15:00
* fungi needs to get his eyes checked15:00
krotscheck1We're keeping the old ones around just in case the world explodes.15:00
anteayakrotscheck1: ah15:00
anteayafungi: glad I'm not the only one with morning eyes15:00
krotscheck1Because if the world explodes, we want to have a backup mirror.15:00
krotscheck1For... urm... space mirrors?15:01
anteayais that what we want?15:01
*** yamamoto has joined #openstack-infra15:01
krotscheck1mirrors in space?15:01
anteayathey can reflect light15:01
anteayaso that's a plus15:01
* krotscheck1 feels like that needs a soundtrack.15:01
krotscheck1Mirrors. IiIIIIN SPAAAAACE!15:01
anteayaoh so many muppet refereneces15:02
*** fhubik_brb is now known as fhubik15:02
fungikrotscheck1: yeah, the afs cache is mounted at /var/cache/openafs on a logical volume from a cinder-supplied block device15:03
krotscheck1fungi: Gotcha. So how come the / disk is slowly increasing?15:03
fungigood question. /var/log?15:03
krotscheck1fungi: Is it just misreporting the /afs content?15:03
fungimy guess would be apache access logs, since every single access to the mirror is logged15:04
fungi8.2gb of apache logs so far15:04
krotscheck1fungi: Do I need to add logrotate to that or will apache handle it itself?15:04
krotscheck1That would be it.15:04
anteayakrotscheck1: apache's default is 52 weeks of logs15:05
anteayaI'm in favour of logrotate15:05
fungilooks like it's rotating weekly15:05
anteayaoh, awesome15:05
*** Daisy has joined #openstack-infra15:05
fungiprevious log rotated on february 115:05
*** dprince has quit IRC15:05
fungicurrent log size is 6gb for the past couple days worth of requests15:05
* krotscheck1 wonders how many copies of war & peace that is.15:06
*** zeih has joined #openstack-infra15:06
fungias for why the afs cache is not being shown in cacti, it's possible snmpd was not restarted after that filesystem was mounted15:06
fungii'll restart snmpd on that host now and see if it pops up in the graphs soon15:06
krotscheck1fungi: I see it.15:07
krotscheck1fungi: It's only about 1GB though15:07
krotscheck1Which I find AMAZING15:07
krotscheck1Of all the terabytes of things that we mirror. Actual thigns we use? A fraction.15:07
fungioh, indeed, it was already there15:07
fungiback to needing to get my eyes checked...15:07
*** fhubik is now known as fhubik_brb15:07
anteayaoff running errands back in a few hours15:08
*** henrynash has quit IRC15:08
*** amotoki has quit IRC15:08
*** infra-red has quit IRC15:08
AJaegeryolanda, fungi: The nodepool puppet run to re-enable iad has not happened yet, do you want to kick it somehow or wait?15:09
yolandaAJaeger, it merged some time ago right?15:10
*** Daisy has quit IRC15:10
*** eharney has joined #openstack-infra15:10
fungiwhat happened in iad?15:10
AJaegeryolanda: yes15:10
yolandadevstack nodes were failing due to ipv6 problems15:10
AJaegerfungi: all dsvm jobs failed15:11
yolandaAJaeger, i'll update manually until puppet lands15:11
fungioh, rackspace announced they were having a network issue in iad?15:11
AJaegerfungi: no, they just broke it - and suddenly it worked again...15:12
yolandawe holded a node and detected that ipv6 errors, and then after a while it started to work again15:12
yolandawe were in contacst with johnthetubaguy for that, he opened a ticket15:12
fungiyeah, doesn't indicate anything other than some issues with monitoring15:12
*** annegentle has joined #openstack-infra15:13
*** jed56 has joined #openstack-infra15:13
johnthetubaguyfungi: it didn't seem like support changed anything for me, but it certainly appeared to be IAD specific somehow, but also seemed a little ubuntu specific somehow15:16
*** zeih has quit IRC15:16
*** boris-42 has joined #openstack-infra15:16
johnthetubaguyit seemed like I had access to most ipv6 things, just not ubuntu + ipv615:16
*** kurtmartin has joined #openstack-infra15:17
fungiare the symptoms mentioned in scrollback here? i'll read through in a bit15:17
yolandafungi, jeblair, it's strange that i haven't seen puppet run on nodepool in more than one hour15:17
yolandaand it's not disabled15:17
fungiyolanda: is it running elsewhere? did you look at the run_all log on the puppetmaster?15:18
*** dtantsur is now known as dtantsur|brb15:18
* fungi needs to step away from the computer for a few minutes, brb15:19
*** mtanino has joined #openstack-infra15:19
sigmavirus24_awadims: you're welcome :)15:19
sdaguefungi: is the bug that catches most of it15:22
sdaguebasically apt-get update timed out15:22
*** jsavak has quit IRC15:22
sdaguetrying to get openvswitch installed in d-g15:22
yolandafungi, i see run_all running regularly on puppetmaster15:22
*** jsavak has joined #openstack-infra15:23
yolandagoing to check run_all , i see some entries for nodepool but need to check the time15:23
*** ihrachys has joined #openstack-infra15:23
yolandaoh, it's difficult to grok last time puppet ran on nodepool with that log15:24
*** zeih has joined #openstack-infra15:24
*** sigmavirus24_awa is now known as sigmavirus2415:24
*** annegentle has quit IRC15:24
yolandalast entry in reports for nodepool is 2 hours ago15:25
*** fhubik_brb is now known as fhubik15:26
jeblaircaught up with scrollback15:29
jeblairsyslog on nodepool looks really weird with dib running15:30
*** rbrndt_ has joined #openstack-infra15:31
jeblairFeb  3 15:16:16 nodepool corvus: os-prober: debug: running /usr/libexec/os-probes/mounted/90solaris on mounted /dev/mapper/main-opt15:31
jeblairi definitely did not run that15:31
AJaegerjeblair: did you install new kernel?15:31
AJaegerOr grub?15:31
jeblairAJaeger: that's from dib15:31
openstackgerritDaniel Wallace proposed openstack-infra/shade: granting and revoking privs to users and groups
openstackgerritDoug Hellmann proposed openstack-infra/project-config: add a job to send automated announcements of releases
*** Swami_ has joined #openstack-infra15:32
*** esker has joined #openstack-infra15:32
jeblairyolanda: it looks like ansible is running puppet on
jeblairyolanda: that's based on running ps15:33
yolandaand is stuck there?15:34
jeblairyeah, from 13:50 utc15:34
openstackgerritValeriy Ponomaryov proposed openstack-infra/project-config: Add new experimental job for manila-image-elements project
yolandawhy does it take so long?15:34
jeblairyolanda: i think es07 rebooted15:34
*** zeih has quit IRC15:34
jeblairi don't see any indication as to why15:36
yolandai applied the max-servers change manually to nodepool until we wait15:36
*** Swami has quit IRC15:36
*** zeih has joined #openstack-infra15:36
jeblairpossibly something wrong with the nova host it was on, though we usually get emails about that15:37
fungisounds similar to the 2+ hour timeout we had on ansible trying to reach the hpcloud pypi mirror when it finally went offline15:39
*** doug-fish has joined #openstack-infra15:39
fungiit's possible we may want to figure out how to shorten that timeout15:40
yolandajeblair, wanted to ask you, how can i receive that alerts in email? i'm not receiving15:40
*** mrmartin has quit IRC15:40
fungiyolanda: rackspace only e-mails it to the account "owner" (which for one of our accounts is mordred i think, pvo is the owner of one of our accounts as well, not sure about the third--would have to check)15:41
jeblairit's possible i'm not an owner of this account, but maybe one of our owners did get an email15:41
*** Sukhdev has joined #openstack-infra15:41
jeblairyolanda: i'm going to try killing pid 20357 on puppetmaster15:41
fungii've been curious whether we can switch those to a role address somewhere, but i haven't looked into it15:41
yolandajeblair ok15:42
jeblairansible is moving again15:42
yolandafungi looks as a good idea, having some generic adress we can all receive15:42
*** dimtruck has joined #openstack-infra15:42
*** annegentle has joined #openstack-infra15:43
*** zul has quit IRC15:46
*** EricGonczer_ has joined #openstack-infra15:51
*** salv-orl_ has joined #openstack-infra15:54
*** clif_h has joined #openstack-infra15:55
clif_his it still possible to view our submitted talks to the austin summit?15:55
openstackgerritIgor Belikov proposed openstack-infra/project-config: Add 'create reference' to fuel-plugin-detach-* ACL
*** salv-orlando has quit IRC15:57
*** burgerk has joined #openstack-infra15:59
*** annegentle has quit IRC15:59
*** annegentle has joined #openstack-infra16:00
*** zeih has quit IRC16:00
*** jgriffith_away is now known as jgriffith16:00
*** jcoufal has quit IRC16:00
prometheanfire doesn't seem to be going through workflow even with rechecks16:01
*** gildub has quit IRC16:02
*** vgridnev_ has quit IRC16:02
*** peristeri has quit IRC16:03
*** lennyb has quit IRC16:03
*** dtantsur|brb is now known as dtantsur16:04
*** zeih has joined #openstack-infra16:04
*** esker has quit IRC16:05
*** woodster_ has joined #openstack-infra16:05
*** sshnaidm has quit IRC16:05
*** esker has joined #openstack-infra16:05
*** hdd has joined #openstack-infra16:06
openstackgerritIllia Khudoshyn proposed openstack-infra/project-config: Add gate job for Rally against Keystone v2
jeblairprometheanfire: it needs to be reapproved because it depends-on a patch that it doesn't share a gate queue with16:08
*** zeih has quit IRC16:09
prometheanfiredon't think it depends on a patch anymore16:10
prometheanfireit used to, but now doesn't with other changes having already gone in16:10
*** infra-red has joined #openstack-infra16:11
jeblairnibalizer: i think i'd prefer infracloud in puppet/nodepool16:12
jeblairrather than omfracloud16:12
*** rajinir has joined #openstack-infra16:12
crinkleyolanda: nibalizer i have been using very small unencrypted keys to sign these certs, i really want a rooter to regenerate them officially, so i don't want to use my uswest root ca to sign certs for east16:13
clarkbjeblair: is the fix for the zuul leak16:13
jeblairclarkb: yeah, i read and responded to the email, thanks16:13
clarkbshould I go ahead and approve the change or do you want to review it?16:14
jeblairclarkb: i was planning on reviewing it16:14
zigoclarkb: jeblair: Hi there! Would you mind reviewing ?16:15
*** lennyb has joined #openstack-infra16:16
fungicrinkle: i suspect we'll be using unencrypted keys regardless, but they'll be larger and presumably we'll be managing them through hiera?16:17
jeblairkrotscheck1, fungi: what's the latest on wheel building slaves?16:17
fungijeblair: i have not started building the builders16:17
yolandafungi jeblair, how can we proceed with the certs for infra-cloud? if we want to start using them in nodepool, i think we first need a trusted ca, so we can just use the same cert for both regions16:19
cody-somervillejeblair: clarkb: Would it make sense to add a test along with that fix?16:20
*** exploreshaifali has joined #openstack-infra16:20 proposed openstack-infra/jenkins-job-builder: Add HockeyApp Plugin support.
*** ihrachys has quit IRC16:20
clarkbzigo: ya I can try building a debian image locally with that16:21
clarkbzigo: the curl thing is curious because it should use the build system's curl not the chroots iirc16:21
fungiyolanda: when we discussed continuing to use self-signed/snakeoil certs for the api endpoint, i don't recall whether anyone identified how we were going to tackle that on the nodepool end. add the cert to the local trust on the server in /etc/ssl? will that work for nodepool's use case?16:21
yolandafungi, if we want to use it on nodepool, we need to add the ca cert to cloud configuration16:22
zigoclarkb: It's still nice to have either curl or wget available, IMO.16:22
fungiyolanda: as in add the path to that cert in our clouds.yaml?16:22
clarkbzigo: sure I just don't think it is necessary there16:22
yolandafungi yes16:23
clarkbzigo: also will break ubuntu I think16:23
fungiyolanda: okay, that's pretty straightforward then16:23
yolandacrinkle , why do you prefer a official rooter?16:23
*** shashank_hegde has joined #openstack-infra16:23
clarkbthere is no linux-headers-amd64 on ubuntu16:23
crinkleyolanda: i want to make sure it's done by someone who knows what they're doing and that it's managed the same way as every other https service16:24
*** matrohon has quit IRC16:24
fungiyolanda: we're going to need to put it in hiera and she doesn't have access to the hiera files, so someone in infra-root may as well regenerate it while adding16:24
yolandafungi i think spencer has been doing that already16:25
fungisounds good16:25
clarkbstill trying to track down how that all plays out, the branching there isn't well documented16:25
yolandabut i wanted to use same ca for east/west, so we don't have two files. Also concerns that crinkle says sound legit16:25
fungiyolanda: he's an infra-root, so if needed he can generate a fresh and stronger keypair16:25
*** dtantsur is now known as dtantsur|afk16:25
jeblaircody-somerville: do you have a suggestion on how to structure it?16:25
zigoclarkb: In Ubuntu, linux-headers-amd64 is a virtual package (ie: only Provides: by some other packages)16:26
clarkbzigo: I may be mistaken that ubuntu takes that branch at all too. Checking with factor16:26
*** sc68cal has joined #openstack-infra16:26
*** hashar has quit IRC16:26
zigoclarkb: It's just in a "if ($::operatingsystem == 'Debian') {" thing, so it should be fine.16:26
clarkbzigo: yup I am mistaken, there are two levels of debianness there, one that applies to ubuntu and one that doesn't16:27
zigoclarkb: "operatingsystem" is really the name of the os, not the familly of os.16:27
jeblairkrotscheck1: do you have a hostname preference for the wheel building slave?  if not, how about wheel-mirror-trustyx64.slave.o.o  ?16:27
yolandaif we go with self-signed, i'd prefer to have a single CA for both regions, and regenerate certs based on that16:27
jeblairyolanda: ++16:27
fungiyolanda: easy enough to do in hiera too16:27
crinklefungi: yolanda i didn't realize we were going to continue to use self-signed certs, that will mean we need another configuration step in puppet to add the ca chain to the trusted certs on the host, because puppet needs to access these endpoints16:27
fungicrinkle: according to yolanda we can just puppet it somewhere on the machine and then add the path to it in our clouds.yaml16:28
jeblaircrinkle: i don't understand "puppet needs to access these endpoints" ?16:28
crinklefungi: jeblair puppet itself makes API calls and it doesn't use --insecure or --cacert16:28
jeblaircrinkle: got it16:28
yolandafungi, that's for nodepool side16:29
fungisorry, i was thinking you meant ansible needed it for inventorying before calling puppet16:29
jeblairlike nibalizer's change to do that for nodepool16:29
crinkleso i had been building this with the assumption that the CA would be signed16:29
*** dtardivel has joined #openstack-infra16:29
yolandabut yes, crinkle is right, that we need to add the config in puppet for infracloud as well16:29
fungiansible is tackled easily by the clouds.yaml addition, but puppet itself less so16:29
*** dizquierdo has quit IRC16:30
*** ifarkas has quit IRC16:30
jeblairwell, it's not hard to drop it into the host's trusted certs, right?16:30
fungi /etc/ssl/certs/*splat*16:30
cody-somervillejeblair: Will need to look a bit more to see if we have the right data types used, but if # of cache objects is deterministic I was thinking of using gc module to to count how many cache objects there are before and after reload or something like that.16:30
yolandait will need some changes in puppet for infra cloud, to place the cacert in trusted hosts16:30
yolandai wasn't successful with my tries in east, but crinkle got it working on west.. so it will be a matter of puppetizing that properly16:31
fungiyolanda: presumably just adding a file resource with content supplied from hiera?16:31
fungior does puppet not actually trust what's in /etc/ssl/certs?16:31
jeblairfungi, yolanda, crinkle: like that patch and its parent ^16:31
yolandafungi, when i tested, i had complains about openstack client, not trusting it, but i bet i did something wrong when generating the cacert16:32
jeblairkrotscheck1: also, did you add logrotate to the mirrors?16:32
yolandacrinkle, you just added the ca to /etc/ssl/certs and worked for you?16:32
*** bpokorny has joined #openstack-infra16:32
crinklei was never really successful with just dropping it in /etc/ssl/certs, it seemed like an update-ca-certificates needs to be run as well16:32
clarkbfungi: yolanda the openstack clients use python requests which ignores the system certs list by default. This means you either have to set env vars to make it trust a specific path or pass certs directly iirc16:33
clarkbcrinkle: even that isn't sufficient with python-requests now that they vendor their own certs16:33
yolandaclarkb, looks as i was hitting that16:33
jeblairthis is the worst thing i've heard16:33
jeblairand i've heard a lot of bad things in this channel16:33
yolandaclarkb, but i didn't find any way to pass the path with puppet manifests. crinkle, do you know about it?16:34
clarkbjeblair: they have chosen to do this to reduce the number of bugs around differing system cert lists or something16:34
fungii'm sure i can come up with something worse, but lack time to devote to such a task ;)16:34
cody-somervillejeblair: though just checking that the old stuff is deleted might be better incase there are other "leaks" of the cache object in the reload codepath.16:34
yolandai ended tweaking the files manually to make it work, but i didn't spend so much time on that16:34
yolandaalso, seems that puppet-openstack modules doesn't allow to pass a --insecure flag easily to just ignore the problems16:34
crinkleclarkb: that doesn't sound right, how does anything auth against https without passing a cert path?16:34
krotscheck1jeblair: In a meeting, sec.16:34
*** ihrachys has joined #openstack-infra16:34
clarkbcrinkle: because most things work with their vendored list16:35
krotscheck1jeblair: I think no on the logrotate16:35
*** diana_clarke has joined #openstack-infra16:35
*** bpokorny has quit IRC16:35
yolandacrinkle but you have your controller working with https and self-signed cert , how did you fix that auth problem?16:35
fungiso it's sounding like it might make more sense to reverse direction on the original snakeoil decision if we need to hack around multiple disparate systems each with their own idea of how to extend trust16:35
crinkleclarkb: aha, well this is another reason to get a real cert then16:35
*** bpokorny has joined #openstack-infra16:36
fungihowever i still think if it's not too hard to solve, it may be of benefit to the wider community who may want to try to stand up a similar test environment without bothering with an external ca16:36
crinkleyolanda: i have no idea16:36
jeblairlet's see how hard it is to tell requests to just use system certs16:37
crinkleyolanda: the uswest cloud seems to be working but recently i've been having a ton of trouble in my dev env with it16:37
clarkbcrinkle: yolanda well te other side of this is that python on trusty doesn't use ssl properly :)16:37
*** kzaitsev_mb has quit IRC16:37
yolandamy experience with self-signed is that some calls worked, but the ones based on openstack client failed16:37
clarkbwhich is where the whole insecure platform warning thing comes from16:37
crinkleyolanda: neutron client behaves the same i think16:37
openstackgerritDevananda van der Veen proposed openstack-infra/project-config: Allow ironic jobs to run with tempest plugin
yolandayep, failures in keystone and neutron16:38
SwansonTrying to get a nodepool node to actually run cinder tests.  I'm failing due to network issues.  Getting a error running the nova bridge stuff.  Address already in use.  This seen?  Easy fix?16:38
jeblairah here we go:
cody-somervilleThis might be a stupid question but would it be possible to use for this?16:38
*** diana_clarke has quit IRC16:39
*** yumapath has joined #openstack-infra16:39
yolandaah, also rapidssl was offering some free certs?16:39
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API
jeblairREQUESTS_CA_BUNDLE is the var we set16:39
jeblaircody-somerville: that will likely be much more work16:39
clarkbSwanson: thats a known issue if oyu reuse devstack gate machines. There was a fix pushed whicc hI though I approved a while back16:40
*** mriedem is now known as mriedem_afk16:40
crinkleyesterday my puppet kept complaining about ssl, today it magically works /tableflip16:40
*** ashleighfarnham has joined #openstack-infra16:41
Clintgroundhog day error16:41
jeblaircrinkle: do you think you can try setting REQUESTS_CA_BUNDLE when running puppet and see if it can make the api calls it needed16:41
jeblaircrinkle: oh16:41
jeblairthat does make experimentation harder16:41
crinklebut maybe yolanda can try16:41
Swansonclarkb, I see.  Well, I'll fire up a new one.16:42
yolandacrinkle, i don't have my nodes on a healthy state now, because i was playing with bridge, but i can try to fix and setup tomorrow morning16:42
* jeblair picks up ssl certs from the floor16:42
krotscheck1jeblair: Yes, I think there was a specific name required for the wheel. sec if I can find it16:42
clarkbSwanson: yes should be corrected with
*** julim has quit IRC16:42
Swansonclarkb, Thanks!16:43
*** Swami__ has joined #openstack-infra16:43
*** julim has joined #openstack-infra16:44
*** tonyb has quit IRC16:45
fungicody-somerville: yolanda: the goal of using snakeoil certs in these cases is to eliminate unnecessary dependence on an external party. if the goal were merely to "save a few bucks" then we wouldn't be having this discussion16:45
*** armax has joined #openstack-infra16:46
*** tonyb has joined #openstack-infra16:46
clarkbfungi: responded to comment on with example runs that fail in the same way on ovh today using gre16:46
*** kevinbenton has quit IRC16:46
*** mahatic_ has quit IRC16:46
*** mugsie has quit IRC16:46
*** mugsie has joined #openstack-infra16:46
fungiclarkb: thanks, i was going to look for some but hadn't found the time yet16:47
*** Swami_ has quit IRC16:47
*** mahatic has joined #openstack-infra16:47
clarkbfungi: I am fairly confident that that is a fail in openstack and not with the overlay network16:47
yolandai can give a try tomorrow morning16:47
*** NikitaKonovalov has quit IRC16:47
crinkleyolanda: actually now that i think about it i haven't had problems with keystone, because keystone's ssl is being managed by apache16:47
fungiclarkb: i suspected as much, which is why i had +2'd it anyway16:48
yolandacrinkle i've had with keystone auth16:48
crinkleyolanda: i wonder what's different :/16:48
*** aarefiev has quit IRC16:48
yolandabasically openstack calls were failing with non-trusted ca16:48
yolandai generated the ca using the same links you sent me16:48
*** aarefiev has joined #openstack-infra16:49
*** diana_clarke has joined #openstack-infra16:49
*** annegentle has quit IRC16:49
*** NikitaKonovalov has joined #openstack-infra16:49
*** kevinbenton has joined #openstack-infra16:51
*** jordanP has quit IRC16:52
*** annegentle has joined #openstack-infra16:52
*** annegentle has quit IRC16:52
*** annegentle has joined #openstack-infra16:53
*** sridhar_ram has joined #openstack-infra16:53
*** jordanP has joined #openstack-infra16:53
*** infra-re_ has joined #openstack-infra16:53
*** kzaitsev_mb has joined #openstack-infra16:54
openstackgerritValeriy Ponomaryov proposed openstack-infra/project-config: Add LXD experimental Tempest job for Manila
*** hashar has joined #openstack-infra16:56
*** fitoduarte has joined #openstack-infra16:56
*** infra-red has quit IRC16:56
*** henrynash has joined #openstack-infra16:57
*** sshnaidm has joined #openstack-infra16:57
*** jistr has quit IRC16:57
krotscheck1jeblair: OK! Done with meeting.16:58
krotscheck1jeblair: Checking for server name16:58
*** e0ne has quit IRC16:59
cznewthello everyone, who can please help with adding a few people to gerrit group? For the
yolandafungi, jeblair, also for preparing for infra cloud. We need to add dns entries for that. In case of east, it's 100 servers. Is that ok to add to our dns servers?17:00
krotscheck1jeblair: wheel-mirror-ubuntu-trusty-amd6417:00
krotscheck1jeblair: It needs to be "wheel-mirror-" + the suffix from
*** Apoorva has joined #openstack-infra17:01
*** gyee has joined #openstack-infra17:01
*** sbalukoff has quit IRC17:02
*** fhubik is now known as fhubik_brb17:02
*** yamahata has joined #openstack-infra17:02
jeblairyolanda: yeah, we may want to use the rackdns client for that;17:03
*** infra-red has joined #openstack-infra17:03
crinkleyolanda: these machines are going to move and most likely be renumbered soon, so probably don't want to do all that work yet?17:03
jeblairkrotscheck1: we can do wheel-mirror-ubuntu-trusty-amd64 if you want, but why does it need to be that?17:04
*** dims has quit IRC17:04
yolandacrinkle , raising that because of the changes for nodepool preparation, that nibalizer raised this morning17:04
yolandai know they are going to be moved soon, but cannot tell about the schedule17:04
krotscheck1jeblair: Line 3517:05
krotscheck1jeblair: The job defenition is here
jeblairkrotscheck1: got it17:05
*** dims has joined #openstack-infra17:06
jeblairkrotscheck1: i will create wheel-mirror-ubuntu-trusty-amd64.slave.openstack.org17:06
jeblairkrotscheck1: oh17:06
krotscheck1Too long?17:06
jeblairkrotscheck1: actually that's a node label which is not necessarily its hostname17:06
*** infra-re_ has quit IRC17:06
jeblairbut i think it's our usual pattern to make the hostname the label17:07
jeblairmight even be a jenkins default17:07
yolandacrinkle if we want to add to nodepool now, we could go with something simple, such as bifrost, a controller and a pair of computes17:07
jeblairit's been so long since i made a slave by hand :)17:07
krotscheck1Well, the node label is /.*wheel-mirror-.*\.openstack\.org/17:07
jeblairkrotscheck1: that's puppet17:07
jeblairkrotscheck1: so wheel-mirror-<anything> will run the puppet code17:07
fungijeblair: i think by default there is no node label, but you can interchangeably refer to nodes by name or by node label in the same places in configuration17:08
jeblairkrotscheck1: in the link you sent, that's jjb telling jenkins to run this job on this node17:08
krotscheck1jeblair: Yes. The job label is different from the puppet label?17:08
prometheanfirecan someone reworkflow this?
*** jlanoux has quit IRC17:08
jeblairkrotscheck1: right, so based on what fungi says, if we spin up the node, attach it to jenkins, give it no node label, the hostname will be used as a node label, so the job will run on that slave if we use the name you suggest17:09
krotscheck1Sounds good to me.17:09
*** fhubik_brb is now known as fhubik17:09
jeblairkrotscheck1, fungi: i'm creating the slave now17:09
jeblairkrotscheck1: do you know how big a single-arch wheel mirror will be after it's built?17:09
fungiwell, we're normally naming slaves for their fqdn but we've been trimming that for node labels to the hostname portion of the fqdn. anyway, it's trivially solvable17:10
krotscheck1jeblair: maybe, let me check.17:10
* krotscheck1 might still have that vm around17:10
*** ihrachys has quit IRC17:10
fungiand now that i say that, i'm less confident in my memory so double-checking17:10
*** nmagnezi has quit IRC17:10
jeblairkrotscheck1: also, answers like "20G should be (more than | not) enough" are useful :)17:10
fungiyeah, the slave named connects to host but has a node label manually added of "release"17:11
jeblairok i'll manually add the hostname then17:11
krotscheck1jeblair: I don' thave that image anymore. I do believe that 20G should be plenty.17:11
*** fhubik has quit IRC17:11
krotscheck1jeblair: Note, though, that the job right now does _not_ clean up after itself.17:11
* krotscheck1 can add that17:12
*** dims has quit IRC17:12
jeblairkrotscheck1: do we want it to?17:12
*** samuelBartel has quit IRC17:13
krotscheck1jeblair: RIght now wheels are built into /tmp/wheelhouse17:13
krotscheck1And then moved into AFS folder structure from there.17:13
krotscheck1Since /tmp is usually only cleaned up on reboot, and this is going ot be a long-lived slave....17:13
fungiwhat sort of mess do we anticipate that job accumulating? also you don't want to rely on the job "cleaning up after itself" since that may not happen for various reasons. if you need to clean it up, do so at the start of the next run17:13
*** Swami__ has quit IRC17:13
jeblaircan it re-use previous work if we don't clean it up?17:14
krotscheck1jeblair: I believe it does.17:14
mwhahahahey openstack-infra folks, how can you determine capacity for a specific job associated with a project? I ask because we've got some changes stuck for fuel-library because gate-fuel-library-puppet-lint is QUEUED for all of the jobs. Are there no nodes currently running that can do that job?17:14
jeblairif so, that seems like a reason to count on having it not clean up17:14
krotscheck1jeblair: The downside is that that directory will grow over time.17:14
fungialso, is it safe to re-use previous work (for example, if library versions change behind the scenes due to distro package updates)?17:14
jeblairfungi: right, which would be a good argument for cleaning up :)17:15
*** [HeOS] has joined #openstack-infra17:15
*** annegentle has quit IRC17:15
*** kzaitsev_mb has quit IRC17:15
fungimwhahaha: it depends on what node type is needed for that particular job17:15
jeblairkrotscheck1: how much ram/cpu should the slave have?17:16
fungii wonder if anyone's working on node counts by type as an addition to
mgagnefungi can this info be graphed? is ready node per type available in graphite?17:16
mgagnefungi hehe17:17
jeblairfungi: i don't know.  mgagne: yes, it's in graphite.17:17
*** nt has quit IRC17:17
*** yamamoto has quit IRC17:17
krotscheck1jeblair: I was running it reasonably well on a 2VCPU virtualbox guest17:17
fungiyeah, shouldn't be hard to add to grafana since we already track it in graphite17:17
*** HeOS has quit IRC17:17
mgagneI like graphs =)17:17
krotscheck1jeblair: I _did_ run into a memory constraint though17:17
clarkbkrotscheck1: jeblair our default slave size should be plenty as they all make wheels today17:17
krotscheck1jeblair: Lemme see what I set it to.17:18
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Source undercloud environment variable from a file
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Split the deploy script into its own file
*** tphummel has joined #openstack-infra17:18
*** dims has joined #openstack-infra17:18
krotscheck1jeblair: 512MB is not enough. 2G was fine.17:18
jeblairclarkb: you're suggesting 8G?17:18
clarkbjeblair: as a conservative upper bound17:18
fungikrotscheck1: clarkb: jeblair: in particular, the constraints updates some other requirements jobs do it for the entire global-requirements.txt set, so holding one of those and looking after completion should give us some idea of what size a base wheel cache for the set would be17:19
*** nt has joined #openstack-infra17:19
mwhahahai guess the next question would be, how does one determine node type :D in the jobs.yaml node is just set to '{node}'17:19
jeblairi'm not too worried about disk space17:19
mwhahahait appears this is also impacting the  puppet-* repos as well17:19
jeblairshould i be? :)17:19
mgagnemwhahaha check projects.yaml the variable is in there17:20
fungiahh, you were asking how much ram. well, i agree with clarkb in that we do it for requirements changes now17:20
jeblairkrotscheck1: can you move the wheel building from /tmp to /opt ?17:20
krotscheck1jeblair: Yep17:21
krotscheck1jeblair: Does that require a puppet update to make sure the directory exists?17:21
jeblairhow do we feel about 4G/4cpu/40GB /opt ?17:21
*** kzaitsev_mb has joined #openstack-infra17:21
mwhahahadoes it inherit that information from the project itself? if so i guess it would be bare-trusty17:21
krotscheck1Seems good17:21
*** harlowja_at_home has joined #openstack-infra17:22
jeblairkrotscheck1: either puppet or in the job via sudo; /opt will exist, but you'll need to make /opt/wheelhouse (or whatever) and chown it to jenkins17:22
krotscheck1jeblair: Willdo17:23
jeblairkrotscheck1: if you decide to build from scratch each time, then the answer is certainly 'do it in the job instead of puppet'17:23
fungimwhahaha: also you can confirm from a recent run, for example
mgagnemwhahaha it does17:23
jeblairkrotscheck1: so maybe that's the way we should go regardless of whether we decide to clean up or not, so that we have more flexibility17:23
mgagnemwhahaha variables are inherited from projects17:23
mwhahahaso next questions, how can i check bare-trusty capacity :)17:24
mgagnemwhahaha otherwise node wouldn't exist and JJB would crash17:24
fungimwhahaha: capacity is likely 0 right now. it's a fairly popular node type, and changes in the gate pipeline get priority over changes in the check pipeline17:24
fungia better question might be how many are in use right now17:25
krotscheck1jeblair: Cando.17:25
*** jgriffith is now known as jgriffith_away17:25
mgagnewhen you know what metrics to look for, it looks so easy :O17:26
*** kzaitsev_mb has quit IRC17:26
mwhahahaok, i guess we'll just continue to wait. and if it hits 4+ hours i'll come back, so far the oldest appears ~2.5 hours old17:27
jeblairwe document the stats for zuul, but i don't think we do for nodepool17:27
krotscheck1jeblair: I'd have to remove revoke-sudo from the template17:27
*** achanda has joined #openstack-infra17:27
jeblairkrotscheck1: hrm, we probably don't actually want to use sudo for anything do we...17:28
mwhahahais it stuck because of the number of nodes pending deletion?17:28
krotscheck1jeblair: I should be able to just empty the directory, even though it's a bit more annoying.17:28
jeblairkrotscheck1: so maybe we need to make /opt/wheelthingy in puppet and chown it to jenkins, and then build in /opt/wheelthingy/actual_build_dir17:28
krotscheck1That works17:28
fungimwhahaha: nah, nodes in delete state don't satisfy nodepool's demand calculations, they just count against total quota in our providers across all node types17:29
fungimwhahaha: the bigger issue is the overall number of bare-trusty nodes in use is small because most of the quota is occupied by devstack-trusty nodes which have a much higher demand across all projects and we're running at full quota capacity in all our providers under the current workload17:30
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add wheel working directory to wheel slave
*** hashar is now known as hasharDinnerTime17:31
fungimwhahaha: right now there are almost 6x as many devstack-trusty nodes running jobs as there are bare-trusty nodes17:31
mwhahahayea i figured it was a quota issue17:31
*** mrmartin has joined #openstack-infra17:31
mwhahahawhat's annoying is that the job it's waiting on probably takes a fraction of the time to allocate the bare-trusty nodes themselves17:31
*** alivigni has quit IRC17:32
mwhahahaer sentence fail, the job takes a fraction of the time when compared to the time it takes to allocate a node17:33
fungiof the 462 nodes running jobs right now, 293 (63%) are devstack-trusty while 58 (13%) are bare-trusty17:33
*** julim has quit IRC17:33
jeblairmwhahaha: well, yes, we're running over capacity so it's going to take a long time to get a node.17:34
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add wheel mirror to
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added wheel-release job
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror
mgagne for grafana json17:34
fungiand yes, the turnover rate of jobs which run on bare-trusty is higher, which means the build and burn overhead for those node types is also higher, which eates into the overall availability of them proportionally as well17:34
krotscheck1Seem good?17:35
jeblairfungi: not a huge amount though; our current providers build nodes *fast*17:35
jeblairmgagne: you want to translate that to yaml and put it in project-config/grafana?17:35
fungijeblair: true, i only see a few bare-trusty nodes building for 30 minutes. most are a handful of minutes17:35
mgagnedoing it atm17:35
*** sbalukoff has joined #openstack-infra17:35
jeblairmgagne: woot!17:35
jeblairfungi: and likely spending a lot of time waiting on nodepool actually17:36
jeblairkrotscheck1: lgtm17:36
zaroclarkb: hey, i was not able to come up with an optimum heap number from reading gerrit info.17:36
*** unicell has quit IRC17:36
zaroclarkb: was the intention to change VM flavors with more RAM+CPU?17:37
jeblairzaro: yes, we're considering that17:37
zaroclarkb: if so i think it would help to increase VCPU as well as bump up jvm heap.17:37
fungijeblair: are we mounting the ephemeral disk at /opt or is this just on the rootfs?17:38
fungii guess we can worry about that after watching filesystem graphs17:38
zaroclarkb: since the 503 indicates a degration in ability to repsond to requests.17:38
jeblairzaro: we're not actually cpu bound except when we're dealing with GC17:38
*** jgriffith_away is now known as jgriffith17:39
jeblairzaro: examine the graphs on
*** Swami has joined #openstack-infra17:39
*** ashleighfarnham has quit IRC17:40
jeblairfungi: i believe launch-node mounts ephemeral on opt17:41
jasondotstarhi all17:41
jasondotstar has been merged17:41
openstackgerritMathieu Gagné proposed openstack-infra/project-config: grafana: Add Nodepool nodes capacity per type
jasondotstarwe're looking to get cznewt added at PTL so that he can add the other core team members17:41
*** sfinucan has quit IRC17:41
*** achanda has quit IRC17:42
cznewthi, we need to add ppl to the openstack-salt-core team, pls anyone can help?17:42
jasondotstarjeblair, dhellmann ^^17:42
*** julim has joined #openstack-infra17:42
jeblairmgagne: gertty bug!17:42
zarojeblair, clarkb : yeah, i see that.  but i was thinking that if we were to change flavors anyways might as well bump up VCPU.  there's only 8 on review.o.o correct?17:42
*** harlowja_at_home has quit IRC17:43
mgagnejeblair there is?17:43
jeblairzaro: we probably don't have a choice, more ram usually means more cpu17:43
mgagnejeblair trying a new repo setup on my side17:43
dhellmannjasondotstar : you need someone on the infra team with gerrit admin rights. I know that includes fungi, clarkb, and jeblair but there may be others.17:43
jeblairmgagne: when i diff that i get UnicodeEncodeError: 'ascii' codec can't encode character u'\u0301' in position 92: ordinal not in range(128)17:43
zarojeblair, clarkb: that makes sense and sounds good.17:43
*** fedexo has joined #openstack-infra17:44
jeblairzaro: yeah, i'm just making the point that more cpu will be nice, it will help us handle more requests in parallel and so will support a higher load, but it will not actually help with the 503 errors, as those are driven by the garbage collector, and more cpus won't help there17:44
jasondotstarfungi, clarkb, jeblair - can either of you guys assist us?17:45
tonybttx: That'd be helpful (adding me to yaml2ical-core) Do you want to request that or should I and use this log as backup?17:45
jeblaircznewt: what's your email address in gerrit?17:45
zarojeblair, clarkb : so unless we can do testing against the review.o.o to verify optimum heap then maybe we can just keep bumping up heap and continue to monitor?  maybe a good start is 20G?17:46
*** jaosorior has quit IRC17:46
jeblairtonyb, ttx: see fungi about that17:46
cznewtjeblair: ales.komarek@tcpcloud.eu17:46
zarojeblair, clarkb : switch to a flavor that provides adequate RAM for cushion?17:46
jeblaircznewt, jasondotstar: done17:46
mgagnejeblair could it be caused by my name having an accent?17:46
*** yamahata has quit IRC17:46
fungitonyb: ttx: i'm actually looking now to confirm i'm okay with the request. just a moment17:46
cznewtjeblair: thanks a lot17:47
tonybfungi: thanks.17:47
mgagnethat unicode character is  ́17:47
*** esikachev has joined #openstack-infra17:47
jeblairzaro: we can increase heap on current review.o.o if needed, but it will eat into disk cache.  however, we are spending almost no time in iowait, so we can probably safely do that.  we would need to keep an eye on iowait to make sure it doesn't significantly increase.  cc: clarkb17:48
*** abregman has quit IRC17:48
jeblairmgagne: not sure yet, doing 4 things at once :(17:48
clarkbjeblair: the other memory consumer is the git mirror17:48
clarkbjeblair: its possible we can just send that off to git.o.o via a redirect?17:49
jeblairclarkb: i do not immediately know the implications of that.17:49
jeblairclarkb: i'd probably start by checking to see if we're still using it in CI17:50
*** shashank_hegde has quit IRC17:50
zaroclarkb: do you mean the local replication?17:51
clarkbzaro: yes, we serve that out via apache and it uses a chunk of the memroy on that host17:51
fungittx: looking at changes to that repo over the past year (both open and merged) i agree reviews look consistent and thorough. tonyb: welcome to yaml2ical-core, don't let the power go to your head ;)17:51
fungion a related note, there are 9 core reviewers for that repo, almost none of whom are reviewing from what i can tell. what were the criteria for populating that?17:52
tonybfungi: Thanks.17:52
tonybfungi: I wasn't around then17:53
tonybfungi: but I think it was a intern project or similar which was done $somewhere_else17:53
*** esikachev has quit IRC17:54
tonybfungi: then when it was brought into our infrastructure all the current authors got core17:54
*** jordanP has quit IRC17:54
tonybfungi: but I don't *know* that17:54
tonybfungi: I think it shoudl be reduced to ttx and I17:55
zaroclarkb: is there a way to tell how many request review.o.o gets for that?17:55
jeblairmgagne: it's the unicode combining acute accent in your name in the commit message... is it possible that your commit message is not utf8?17:55
mgagnejeblair maybe it's repo that badly configured git17:55
clarkbzaro: we can check the apache logs, is there something specific you are trying to determine?17:56
mgagnewill retry from my old setup17:56
*** Daisy has joined #openstack-infra17:56
tonybfungi: I'd be happy to mail them all and say "Umm yeah you're not really doing anything with yaml2ical, we we're going to remove you core status" If you like.17:56
fungitonyb: that sounds likely. i guess we can revisit when i get around to running actual blanket stats across all our repos. there are probably more in similar situations17:56
tonybfungi: okay17:57
fungiit's on my neverending to-do list17:57
jeblairmgagne: how new is it?  i see the problem on some of your older changes...17:57
mgagnejeblair new from today17:57
zaroclarkb: wehter you can tell where it's coming from and if requests are low do we need to worry about it?17:57
*** sputnik13 has joined #openstack-infra17:57
mgagnewhich changes?17:58
*** lucasagomes has quit IRC17:58
openstackgerritMathieu Gagné proposed openstack-infra/project-config: grafana: Add Nodepool test nodes capacity per type
mgagnejeblair this is with old setup17:58
jeblairmgagne: i'll try that17:58
mgagnejeblair this one is from old setup17:58
mgagnemeeting now, bbl17:59
jeblairmgagne: same problem; seems unlikely to be your side, i'll keep digging17:59
*** jtomasek__ has quit IRC17:59
mgagnecould be git-review I updated last week17:59
*** Sukhdev has quit IRC17:59
jeblairmgagne: i checked a change from december and still see it18:00
*** Daisy has quit IRC18:00
*** mikelk has quit IRC18:00
tonybfungi: So speaking of power I'm going to +W my open patches to yaml2ical18:01
jprasselin_: I'm getting an error about a null user id when jenkins tries to test the ssh connection with a slave on a stock openstackci config18:01
tonybfungi: Also would running reviewstats over openstack-infra/* be a helpful start?18:02
*** arxcruz has quit IRC18:02
*** trown is now known as trown|lunch18:02
jeblairmgagne: it's something related to the recent py3 changes18:02
*** rossella_s has quit IRC18:02
jeblairthe string we're manipulating is already a unicode string18:03
*** rossella_s has joined #openstack-infra18:03
jprasselin_: the jenkins username is configured on the nodepool side and the creds in hiera work against the jenkins account on the slave, but i dont' see anything related to these credentials defined in the jenkins manage credentials ui.  having difficulty finding where to look.18:03
fungitonyb: if they're languishing then go ahead. sounds like ttx already +2'd them18:03
tonybfungi: Yeah they all have his +218:04
fungitonyb: as for reviewstats, it's less useful for infra because of our council arrangement. the granularity assumptions are vastly different18:04
*** lucasagomes has joined #openstack-infra18:04
fungii've been wanting to look into ways to extend it for our use case, but more likely this time around i'll be playing with lots of my own stats generation first just to figure out what insights different aggregations of review data are providing at all18:05
*** bnemec has quit IRC18:05
tonybfungi: Ahh ok, I'll leave it to you then I just thought it might be a helpful first step that I could do for you.18:06
fungii do look at reviewstats, it's just very muddy with our hundred or so repos in widely varying areas of expertise18:07
fungii mostly want to see which non-cores seem to have significant interest in individual repos, especially ones with lagging/under-served core review activity18:07
*** rhallisey has quit IRC18:07
fungiand also see if i can spot trends of reviewing across multiple repos and multiple disciplines18:08
*** mrmartin has quit IRC18:08
fungiall those provide possible starting points for figuring out whose reviews i should be looking through for what subgroups18:08
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter
openstackgerritVolodymyr Samotiy proposed openstack-infra/project-config: Adding broadview-lib project and jobs.
tonybfungi: Yeah, I wish I had something more helpful to say, but that sounds great and hard.  I think the interested non-cores aspect it probably doable with either reviewstats as a base.18:11
tonybfungi: but as you clearly know it'll be non-trivial.18:11
*** _nadya_ has quit IRC18:12
fungiright. i want to add support for it in an existing tool if i can, but step 1 is figuring out what sorts of collections/aggregations of these stats are actually useful18:13
*** shashank_hegde has joined #openstack-infra18:14
fungitooling to collect the data is not the hard part, deciding what to analyze is18:14
*** ybathia has joined #openstack-infra18:18
tonybfungi: yeah.  I guess an additional challenge is quality but I s'pose that can be investigated once you've found someeone to investigate.18:18
tonybfungi: I'm trying hard not to say "I'll help you" as I know I don't really have enough time to do it right18:18
*** Apoorva is now known as apoorvad18:18
*** ybathia has left #openstack-infra18:19
*** jsavak has quit IRC18:20
* anteaya prevents tonyb from jumping under another bus18:20
tonyb*sigh* it's true18:20
clarkbany other takers for ? it should address a major source of failures on bluebox18:21
jeblairfungi: wow, the py3k changes from jaypipes obliterated the unicode safety we had :(18:22
*** e0ne has joined #openstack-infra18:24
*** _nadya_ has joined #openstack-infra18:25
*** Jeffrey4l has quit IRC18:25
AJaegerfungi, you should be easily able to create some groups for reviewstats, for example a puppet group, a project-config one, and a system-config one.18:26
*** jsavak has joined #openstack-infra18:26
*** pvaneck has joined #openstack-infra18:27
AJaegerfungi, if you want to play with that, I can create a review request so that you have a basis to play with...18:27
*** sbalukoff has quit IRC18:27
*** sabeen3 has quit IRC18:29
*** achanda has joined #openstack-infra18:29
openstackgerritVolodymyr Samotiy proposed openstack-infra/project-config: Adding broadview-lib project and jobs.
*** vgridnev has joined #openstack-infra18:31
fungiAJaeger: yeah, again, that's an easy but small problem. what i want to do is data mining to determine whether there are emergent patterns and groups rather than starting the analysis from an existing rigid group structure18:32
AJaegerfungi, infra.json in reviewstats is not usable, has the wrong core teams...18:33
*** electrofelix has quit IRC18:33
AJaegerfungi: seems you need to do it from scratch on your own ;)18:33
*** unicell has joined #openstack-infra18:33
fungiindeed, a conclusion at which i had already arrived18:34
fungijeblair: which py3k changes? nodepool?18:34
jeblairfungi: gertty18:34
AJaegerfungi, one idea: Go to and dig into individuals18:34
jeblairfungi: i think i've found the changes that prompted those, and i think i've just about corrected it18:35
fungii think i probably hadn't looked at his python 3 support patches18:35
jeblairfungi: i have been reading eavesdrop from 1.5 years ago18:35
clarkbzaro: I am tailing the access logs for apache on /p/18:35
fungithat's some serious detective work18:35
fungiclarkb: does it seem like we have any of our own processes or jobs hitting that?18:35
fungior is it all from other stuff not under our control?18:36
clarkbfungi: ya zuul is occasionally hitting it18:36
jeblairclarkb: the mergers?18:36
*** unicell1 has joined #openstack-infra18:36
clarkbjeblair: no zuul.o.o according to dns18:37
*** fitoduarte has quit IRC18:37
clarkb2001:4800:7815:101:3bc3:d7f6:ff04:e07f - - [03/Feb/2016:18:35:52 +0000] "GET /p/openstack/tempest/info/refs?service=git-upload-pack HTTP/1.1" 200 2049431 "-" "Python-urllib/2.7"18:37
fungipolling to check merged state of a change?18:37
AJaegerjasondotstar: update for all repos .gitreview like done with
jeblairclarkb: just hitting info/refs?18:37
jeblairfungi: yeah, i think so if that ^ is true18:37
*** unicell1 has quit IRC18:37
clarkbjeblair: so far yes18:37
*** unicell has quit IRC18:37
*** unicell1 has joined #openstack-infra18:37
jeblairfungi, clarkb: which is an oops, since that should actually hit real gerrit18:37
fungiright, there was a race...18:37
jeblairi think i remember18:38
fungiwhere sometimes gerrit's api would report merged state before it actually merged the change on the backend18:38
jeblairi think we actually did want to test the mirror so that we would know that not only it had merged, but propogated18:38
jeblairthat thinking still predates the mirror farm though.18:38
fungii wonder if we should be checking the mirror farm, yeah18:38
*** unicell1 has quit IRC18:39
*** unicell has joined #openstack-infra18:39
*** eil397 has joined #openstack-infra18:39
fungiit's possible some of that race window was gerrit not getting around to queuing mirroring the update18:39
*** exploreshaifali has quit IRC18:39
fungior the queue being backed up18:39
jeblairi kind of think we should change it to point to real gerrit18:40
openstackgerritMerged openstack-infra/yaml2ical: Add support for skipping/excluding meetings
*** jsavak has quit IRC18:40
jeblairthe mergers pull from real gerrit now, and the critical parts of the changes under test pull from the mergers18:40
openstackgerritMerged openstack-infra/yaml2ical: Add one off events for skipped meetings
openstackgerritMerged openstack-infra/yaml2ical: Remove argparse from requirements
jeblairso we're not really subject to mirror races like we were18:40
*** jsavak has joined #openstack-infra18:40
krotscheck1~400 zuul queue items is sadpanda18:43
*** yamahata has joined #openstack-infra18:43
*** fedexo has quit IRC18:44
dmsimardkrotscheck1: last I know jenkins was happy :P18:44
fungikrotscheck1: rackspace iad was offline for a while earlier, which has made the backlog worse than it would have been18:44
openstackgerritJames E. Blair proposed openstack/gertty: Fix unicode regressions
openstackgerritJames E. Blair proposed openstack/gertty: Fix typo in expire-age setting
AJaegerkrotscheck1: yeah ;( rax-iad made all devstack-trusty fail earlier, then we took it down, now it's up and everybody rechecked I guess...18:45
*** dizquierdo has joined #openstack-infra18:45
*** slogan621 has joined #openstack-infra18:45
*** doug-fish has quit IRC18:45
dmsimardkrotscheck1: yeah, I know :/18:45
krotscheck1Fun times, fun times.18:45
*** jsavak has quit IRC18:46
krotscheck1Do we have historical data for that queue btw?18:46
* krotscheck1 wants to get a screenshot of "here's what happened when HPCloud went away"18:46
fungikrotscheck1: as in queue size over time? sure18:46
*** jsavak has joined #openstack-infra18:46
clarkbkrotscheck1: it is in graphite18:46
clarkbthat is what renders the sparklines18:46
fungithose sparklines at the headers... yeah18:46
fungiwe have history into the wayback18:47
anteayaclarkb: the code looks fine to my eyes on 274821 and will have to take your word that this may improve things for bluebox18:47
anteayadougwig: care to join me in a review of 274821?18:47
cody-somervilleI wonder what's the carbon footprint of OpenStack CI.18:47
jeblairfungi, mgagne:
* anteaya offers to share her cold remedies with dougwig18:47
fungicody-somerville: massive, i'm sure18:47
*** sdake has quit IRC18:47
dougwiganteaya: looking18:47
AJaegerjeblair, fungi, clarkb, anteaya : sdague and myself had earlier today a discussion about default timeouts - do you have time now for some discussion and to educate me, please?18:47
anteayadougwig: thanks18:47
anteayaAJaeger: I can listen18:48
anteayaAJaeger: I'm not sure how much I can offer by way of compelling argument18:48
jeblaircody-somerville: or did you mean carbon *dioxide* ?18:48
*** ashleighfarnham has joined #openstack-infra18:48
cody-somervillejeblair: the latter <grins>18:48
AJaegeranteaya: asking good questions is also helpfull ;)18:49
fungiAJaeger: yeah, saw the discussion. summary was "some timeout is needed, but cropping timeouts close means updating more often/failing jobs periodically as their scopes or runtimes creep"18:49
jeblaircody-somerville: large but smaller than the alternative i'd wager18:49
anteayaAJaeger: I will endeavour to do so18:49
AJaegerjeblair, fungi, clarkb, anteaya, sdague: This was triggered by .18:49
AJaegerWhat should I review and accept as default timeout for jobs? I remember that default should be 1 hour.18:49
AJaegerAnd I see some cargo cult in devstack-gate where everybody uses 120 mins as timeout.18:49
clarkbAJaeger: unfortunately I think we have regressed well past an hour at this point :/18:50
clarkbbut an hour has bene our goal18:50
fungiAJaeger: the counterargument of course is that projects ignore glaring ineffciiencies in their jobs until they start hitting a timeout, so many jobs will rapidly erode toward whatever timeout we set18:51
AJaegerclarkb: so, should we accept any timeout? Any length for running?18:51
anteayaAJaeger: yeah I think if an hour is our goal having the default timeout that folks can cargo cult be 70 minutes make sense to me18:51
*** deva_ has joined #openstack-infra18:51
notmorganclarkb, sdague: would you rather i roll up a change for devstack to make milliseconds happen for keystone logs or is second resolution good enough? I am dropping microseconds in either case.18:51
*** tcammann has joined #openstack-infra18:51
anteayawell having no limits on things doesn't tend to work well for us18:51
*** esp_ has joined #openstack-infra18:51
jeblairthere's significant variation in run times, especially with more clouds, so the timeout should accomodate that.18:52
anteayaagreeing to a limit and then educating folks about the purpose of it is useful I think18:52
*** thiagop has quit IRC18:52
AJaegersdague mentioned earlier that we should use a timeout of 3 hours - to protect against endless loops and that if people hit the timeout, they will just recheck which is bad18:52
*** NobodyCa1 has joined #openstack-infra18:52
jeblair3 hours seems very long18:52
clarkbnotmorgan: seconds is good enough for me if you think you can debug the interactions between keystone and other services using only second resolution18:52
fungiand lower timeouts provide more obvious pushback on projects whose jobs are becoming less efficient, while also penalizing projects whose jobs are simply getting more robust and increasing coverage18:52
anteayajeblair: that is a fair point, do our run times have any suggestions about what a good time out might be?18:52
fungihard to strike a balance18:52
*** trown|lunch is now known as trown18:52
clarkbnotmorgan: mostly just wanted to reduce the number of unique keys in elasticsearch18:52
clarkbnotmorgan: as we were hitting trouble with the total volume there18:52
notmorganclarkb: absolutely, hence the drop of microseconds in either case18:53
anteayafungi: yes18:53
AJaegerfor ironic I checked for some samples and found majority of runtime betwene 30 and 50 mins, dtantsur mentioned not seeing more than 60 mins - so I took 70 there...18:53
*** jgriffith is now known as jgriffith_away18:53
notmorganclarkb: i don't have a strong feeling that we can/can't debug. i'm content to stick with oslo_log in devstack if it'll help others in your experience18:53
notmorganclarkb: it's a... 2 line change instead of a 1 line change to get milliseconds iirc18:54
cody-somerville90 minutes is probably fair default. (1 hour "target" for all jobs with 30 minutes as grace).18:54
anteayajeblair: :)18:54
anteayacody-somerville: that isn't what happens though18:54
cody-somervilleIf we have jobs that intentionally require longer than that we should probably want to have a conversation about how to best support that and ensure that use of valuable resources is returning appropriate value.18:55
*** infra-re_ has joined #openstack-infra18:55
* AJaeger has started asking during reviews that use the cargo cult of 120mins for runtime of similar jobs and reduce if possible.18:55
*** kushal has quit IRC18:55
*** thiagop has joined #openstack-infra18:55
jeblairkrotscheck1, fungi: exists18:55
AJaegerjeblair: great!18:56
jeblair/dev/xvde2 on /opt type ext4 (rw,errors=remount-ro,barrier=0)18:56
krotscheck1jeblair: WOOHOO18:56
jeblairAFS on /afs type afs (rw)18:56
fungiinteresting but probably insane thought. i wonder if we could implement "soft" timeouts where a job fails if it runs past a certain threshold but continues running up to the hard timeout where we then wrap up the job and collect artifacts18:56
*** yumapath has quit IRC18:56
cody-somervilleIt would almost be nice if we could have additional grace time when running in the gate to avoid the nasty cost of a gate reset.18:56
anteayaAJaeger: seems we agree that having a default is valuable, as well as agreeing that deciding what the default should be is hard18:56
jeblairfungi: we do that already18:56
AJaegerjeblair: how?18:56
anteayaAJaeger: seems we also agree that having a default of 3 hours is excessive18:56
fungijeblair: sort of. we still terminate the job workload at the current "soft" timeout18:56
fungii'm talking about an even softer timeout18:57
*** mrmartin has joined #openstack-infra18:57
krotscheck1jeblair: Now all we ahve to do is wait for to merge.18:57
fungiwell, we terminate the inner workload at the soft timeout presently18:57
jeblairfungi: oh.  for what purpose?18:57
*** jgriffith_away is now known as jgriffith18:57
AJaegerjeblair: idea would be not to terminate but mark still as failure for the "fungi-softtimeout"18:57
*** infra-red has quit IRC18:57
*** tcammann has quit IRC18:57
fungimaking it easier to analyze what parts of the job are slow by allowing the tests to hopefully run to completion even though the job will fail for running longer than we want18:58
jeblairfungi: has anyone expressed an interest in doing that?18:58
*** esp_ has quit IRC18:58
cody-somervilleI think that is already supported by just looking at the data for outliers or upper quarter of runtimes.18:58
fungino, just brainstorming how else to provide pushback on long-running jobs without a hard-stop to what's being tested18:58
jeblairfungi: also, we do copy as much as has run already to the timeout, are we certain that we need more data than that to determine why it's slow?18:59
cody-somervilleStackViz is all about being able to visualize the runtime of dsvm jobs.18:59
fungiyep, was a crazy idea as i predicted18:59
*** NobodyCa1 has quit IRC18:59
cody-somervilleI believe they're still working to implement that so it runs for all jobs?18:59
anteayatonyb: thank you for 23231218:59
*** deva_ has quit IRC19:00
tonybanteaya: yw19:00
cody-somervilleala this would be the output:
cody-somervilleanyhow, that's a tangent.19:00
fungibasically at a loss for how to balance job performance regression against job coverage increases19:00
tonybanteaya: I'll rebase my other patches and then work with ttx, dhellman to get a release of yaml2ical out the door19:01
*** notnownikki has quit IRC19:01
tonybanteaya: once that's done you'll be able to use the skip feature19:01
fungiso that it's clear to projects when they should be looking at increasing their timeouts or splitting their job scopes, vs when they should be taking a hard look at the performance of their tests19:01
tonybanteaya: so it should be ready for Austin :)19:01
anteayatonyb: wooot, and I've agreed to review more patches especially ones that use the skip feature19:01
anteayatonyb: yay!!19:01
tonybanteaya: I19:01
anteayathank you thank you thank you19:01
tonybll poke you if needed ;P19:02
anteayado that please19:02
cody-somervillefungi: jeblair: Would you say that the system works best with super-consolidated long running jobs vs. more granular but still lengthy jobs? (assuming we already know that many small jobs is not efficient due to the overhead of vm creation/deletion)19:02
tonybanteaya: I don't really know how well used it'll be.  I think it's most usful for 'edge consumers' people that arrn't on IRC / don't read os-dev19:03
tonybanteaya: but time will tell.19:03
clarkbzaro: jeblair fungi I am resonably confident that the majority of the things hitting the /p/ url are instances of zuul. Our own and for third parties19:03
clarkbI don't see much other traffic against that /p/ prefix19:03
jeblaircody-somerville: i don't think it's worth worrying about too much19:03
fungithe common story is projects coming to us wanting a "temporary" timeout bump so that they can work out the performance issues they've grown (which tey weren't concerned with until it started regularly reaching the timeout), then coming to us again not long after asking for another "temporary" increase because their performance had regressed further and they hadn't worked on solving it19:03
anteayatonyb: well I'll use it for the third-party meetings, as I do feel that group fits your description19:03
tonybanteaya: I have additional features that I'd very much like to add19:03
anteayatonyb: I look forward to patches19:04
*** infra-red has joined #openstack-infra19:04
anteayaor etherpads19:04
tonybanteaya: totally :) that's why we wrote the code :)19:04
*** julim has quit IRC19:04
anteayafungi: yes19:04
clarkbzaro: jeblair fungi with that in mind a better alternative to redirecting to git.o.o maybe to simply have gerrit serve those refs directly and stop using a local replica?19:04
clarkbthen we can allocate more memory to gerrit on the whole19:05
AJaegeropen review for timeout increase:
fungicody-somerville: technically yes, by a small margin (a reduction in the amount of quota occupied by building and deleting nodes)19:05
fungicody-somerville: but really a premature optimization19:05
fungiclarkb: yes, i think so, and also adjust zuul to go to the right url rather than hitting the transparent rewrite19:06
jeblairi much prefer the approach of people just building the jobs they need and not worrying about that for anything less than, say, a 5 minute job.19:06
*** infra-re_ has quit IRC19:06
cody-somervillejeblair: fungi: thinking more from a queue theory perspective vs. optimizing away that particular overhead.19:06
jeblaircody-somerville: ah, well in theory there's no difference :)19:06
*** julim has joined #openstack-infra19:06
*** yaume_ has quit IRC19:07
clarkbI can whip up that change and we can argue about it in gerrit I guess :)19:07
fungiclarkb: with the goal of eventually dropping the rewrite when we see its use trail off19:07
*** bhunter71 has quit IRC19:08
jeblaircody-somerville: in practice, the answer is very difficult (we've discussed the merits of greater nodepool efficiency with long jobs, but neglected the greater zuul efficiency (early aborting of changes) with short jobs)19:08
*** mrmartin has quit IRC19:08
clarkbfungi: yup19:08
*** infra-re_ has joined #openstack-infra19:08
jeblaircody-somerville: in fact, we were so concerned with the latter at some point that we wanted to parse the subunit stream to abort changes *even earlier*19:08
dougwigfungi: can't folks debug those kinds of jobs by commenting out some tests and making the job smaller, until its sorted? i'm not sure that's an infra problem.19:09
jeblairi still think that would be neat, but it doesn't seem pressing19:09
clarkbI just have to find where we set that :)19:09
anteayaAJaeger: thank you for bringing up this topic for discussion19:10
anteayaAJaeger: we don't seem to have a clear happy outcome yet but at least we are discussing it19:10
AJaegeranteaya: yeah...19:11
fungidougwig: yep, and also there's the "we shouldn't have to police this" aspect. let projects be accountable for making their tests efficient, and assume good faith19:11
*** dizquierdo has quit IRC19:11
*** infra-red has quit IRC19:12
jeblairheh, i forgot about tmpreaper; we actually do clean up slave tmp dirs frequently :)19:12
fungiooh! the zuul job backlog _seems_ to be steadily dropping over the past hour19:12
cody-somervillefungi: providing a "budget" isn't necessarily a bad idea though, no?19:12
fungicody-somerville: right, though also figuring out who gets what budget becomes a political quagmire19:13
*** sbalukoff has joined #openstack-infra19:13
fungiin some ways it's easier to let the community collectively provide pushback against the rare selfish actors rather than setting rigid guidelines which make things inconvenient for everyone (including on those who have to enforce them)19:14
jeblairkrotscheck1, fungi: is online19:14
*** Sukhdev has joined #openstack-infra19:14
anteayafungi: except when the community doesn't provide pushback19:14
AJaegerand that's also a question, fungi: Should we enforce or ask them to explain their timeout value?19:15
fungianteaya: also correct19:15
jeblairkrotscheck1: /etc/wheel.keytab is root-owned; you'll need to make it owned by jenkins i think19:15
fungiAJaeger: i think expecting an explanation is entirely warranted19:15
krotscheck1jeblair: Hrm, good point.19:15
jeblairkrotscheck1, fungi: unless we want to make a sudo rule for that?19:15
cody-somervilleluckily the gate is relatively stable. long actual runtimes + flaky jobs in the integrated gate is killer for throughput.19:15
anteayaif it is other than the default19:15
fungiAJaeger: for any proposed change, not just timeout increases19:15
jeblaircody-somerville: i don't think that's luck :)19:16
AJaegerSo, default value that we review for is 60 mins (the wrapper is then 65 mins)?19:16
anteayaAJaeger: I commented on
fungijeblair: krotscheck1: i prefer relying on the filesystem permissions if we can19:16
anteayaAJaeger: I can support that19:16
jeblairfungi: owned by jenkins?19:16
openstackgerritClark Boylan proposed openstack-infra/zuul: Don't use /p/ path for info/refs
*** deva_ has joined #openstack-infra19:16
clarkbsomething like that19:16
cody-somervillejeblair: Good point. Lots of hard work has been put into that. :)19:16
*** annegentle has joined #openstack-infra19:16
krotscheck1on it19:16
AJaegerand if there is a larger timeout, we ask for an explanation - and trust a good explanation19:16
jeblairfungi: i was mostly exploring the idea that maybe we wanted to have a sudo rule so that the key isn't actually available to jenkins19:17
*** tcammann_ has joined #openstack-infra19:17
jeblairfungi: but i think actually closing that gap would be very difficult19:17
*** achanda has quit IRC19:17
*** esp_ has joined #openstack-infra19:17
*** annegentle has quit IRC19:17
dougwigAJaeger: don't most dsvm jobs use 125/120 ?  Or did that change?19:17
AJaegerSo, what should be done with the ironic change? ? Do you want +2  it besides sdague -119:17
fungijeblair: i suppose it's a question of relying on there not being a bug in sudo or the tools being allowed by sudo which could leak the file, vs just accepting the user has access to that file and planning accordingly19:17
AJaegerdougwig: many do - and new jobs just copy that value without thinking....19:18
*** annegentle has joined #openstack-infra19:18
*** jgriffith is now known as jgriffith_away19:18
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Changed wonership of wheel.keytab to jenkins
jeblairfungi: yeah, i lean toward the latter for now (we already decided to do that, i just hadn't really thought of sudo before now)19:18
anteayaAJaeger: well I'm really not a fan of +2'ing something someone else has -1'd19:18
AJaegerdougwig: the usual cargocult - use what's available without checking what is sensible...19:18
*** NobodyCa1 has joined #openstack-infra19:18
dougwigAJaeger: that's what I do, didn't even realize it was incorrect. though i've never seen a >60 job that didn't need to be shot in the head, come to think of it.19:18
jeblairanteaya: disagreement is permitted :)19:18
krotscheck1jeblair, fungi: ^^ something something 400 things in zuul something19:19
* dougwig adjusts his internal constraints.19:19
anteayajeblair: true19:19
AJaegeranteaya: we don't need to +A that one...19:19
krotscheck1Doh type19:19
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Changed ownership of wheel.keytab to jenkins
jeblairkrotscheck1: even your typos have typos19:19
*** sabeen1 has joined #openstack-infra19:19
AJaegerAnd I guess we have to continue this discussion another time based on the reviews coming in.19:19
anteayaAJaeger jeblair how would we feel if we give that change 24 hours to give sdague a chance to read backscroll if he would like to19:19
krotscheck1jeblair: Yo dawg, I heard you like typo's....19:19
AJaegeranteaya: perfectly fine with me - I can keep my WIP for a day or two...19:20
anteayaAJaeger: thanks for understanding, it at least gives him an opportunity to read the discussioni19:20
AJaegeranteaya: sure. I'd loved to have him here as well...19:21
openstackgerritDaniel Wallace proposed openstack-infra/shade: granting and revoking privs to users and groups
anteayaAJaeger: understood, me too, async working times and all19:21
*** sabeen2 has joined #openstack-infra19:21
fungiAJaeger: comes down to reviewer judgement and how thorough of an explanation you're looking for. and if there are other core reviewers who are willing to disregard your request and +2/+3 over top of you, then those are expectation discussions to be reraised as a group19:21
*** _nadya_ has quit IRC19:21
*** esp_ has quit IRC19:21
AJaegerfungi: so far, I just asked questions and didn't vote on these...19:22
*** hasharDinnerTime is now known as hashar19:22
AJaegerOnce I was happy with answer - or the change - I gave my +2 ;)19:22
*** tcammann_ has quit IRC19:22
dougwigAJaeger: gotta juice that disagreement stat!19:22
anteayaha ha ha19:22
anteayaI'm sure taht is at the top of AJaeger's mind when he reviews19:23
*** deva_ has quit IRC19:23
fungias for picking a sane "default" timeout for a given template, it's a tuning question (how often do we want to be reviewing requests to increase timeouts on specific jobs vs drawbacks of just increasing the starting/default timeout expectation)19:23
*** NobodyCa1 has quit IRC19:23
*** sabeen1 has quit IRC19:24
AJaegerwhat I've learned is that a full neutron setup needs more than 1 hour... But if neutron is not in the equation or only a small subset is run - like in the ironic case, we should be fine with an hour...19:24
jeblairkrotscheck1: i also aprvd the wheel build job since the slave exists; i'm waiting on the release job until after we manually vos release a few times19:24
krotscheck1jeblair: ...ooookaaaaaaay.19:25
AJaegerfungi, I'm not looking for a thorough explanation, I'm looking that they thought about the value and have some comparasions done - instead of blindly copying a value...19:25
anteayaAJaeger: +119:25
krotscheck1jeblair: That's going to fail a few times if wheel.keytab's not set up yet.19:25
fungii assume all the bare uuid hosts now appearing on are infra-cloud servers?19:25
slogan621I assume only people on infra team can review this: ? That's ok if so, just wondering.19:26
slogan621BTW, hi :-)19:26
AJaegerslogan621: everybody can review19:26
jeblairkrotscheck1: it's been approved too, and i'm hoping it lands before jjb gets around to updating, but yeah, they are racing.19:26
krotscheck1go patch go!19:26
crinklefungi: no i don't think so19:26
jeblairelephant race19:26
fungii'll see if i can track some down then19:26
dougwigAJaeger: oh indeed, some neutron are cresting an hour. that needs some investigation, i think.19:27
AJaegerthanks all for the comments on the job run times, I'll feel better continuing to review...19:27
slogan621I'd suppose I'd prefer to let the experts on infra do the review, though I certainly want to support it, or vote if that matters.19:27
zaroclarkb: comment on that zuul change.19:27
*** bpokorny_ has joined #openstack-infra19:29
*** jgriffith_away is now known as jgriffith19:29
fungialso we seem to have a lot (~50) of nodepool nodes stuck in delete in rackspace dfw and ord for between 1 and 6 hours now19:29
anteayaAJaeger: thanks for bringing it up for discussion :)19:30
*** bpokorny_ has quit IRC19:30
*** bpokorny_ has joined #openstack-infra19:30
sdagueback from food19:30
anteayaslogan621: you might find this blog post of mine enjoyable:
anteayasdague: welcome back19:31
sdaguemy point is the 3 hour timeout is the backstop for runaways19:31
sdagueno one wants jobs that run that long19:31
sdagueevery time anything crests an hour regularly it, rightly, causes angst19:31
jeblair1.5 or 2x should be good for backstop, though, yeah?19:32
*** bpokorny has quit IRC19:32
sdagueand the gate fails that happen when a 60 minute job becomes 61 minutes, and times out uploading files is just terrible experience19:32
fungiwow, here's a nova list result for an example of one of the "stuck" nodes... weird: | aa512172-d34f-4114-af1b-2300f47b1e0a | bare-trusty-rax-ord-7677301                 | BUILD  | deleting   | NOSTATE     | public=2001:4801:7829:101:be76:4eff:fe11:cd99,; private= |19:32
*** bpokorny_ has quit IRC19:32
sdaguejeblair: sure, if you are going to keep updating as things shift around19:33
jeblairsdague: we shouldn't timeout uploading files; if we are, we can adjust the cushion in devstack-gate for that19:33
sdaguemy point was mostly that a large common timeout that ensures things return, eventually, is fine. No one runs up to that load intentionally.19:34
sdagueit just seems like a lot of extra work to judge every job19:34
jeblairsdague: yeah, i agree with you on that, i just lean toward 2h instead of 3h for that purpose.  :)19:34
*** bpokorny has joined #openstack-infra19:34
AJaegerwe have two timeouts for the devstack jobs, the wrapper one and the devstack variable - the wrapper is an extra 6 mins. We can easily enlarge that wrapper one.19:34
jeblairAJaeger: oh, that's in p-c, right, sorry i misplaced it :)19:35
*** e0ne has quit IRC19:35
*** achanda has joined #openstack-infra19:35
jeblairwe should definitely do that if we are having upload timeouts; sdague do you know if that's a current problem?19:35
sdaguesure, 2h is probably fine. There are some 3h ones in there for some jobs that end up on slower nodes and already start at > 1 hour19:35
sdaguejeblair: I don't know if we are right now, I've seen it in the past19:35
openstackgerritMerged openstack-infra/zuul: Fix memory leak reloading triggers
sdaguethe dual timeout management bit is just a little goofy19:35
jeblairsdague: ok; well if you do see it, it's easily fixable19:36
sdaguedidn't zaro build a jenkins module to make us not have to do that19:36
AJaegerjeblair: p-c?19:36
slogan621anteaya: thanks19:36
jeblairAJaeger: project-config19:36
AJaegerjeblair: ah - yeah, jenkins/jobs/devstack-gate.yaml...19:36
* AJaeger needs to step out for 10 mins19:36
jeblairi'm going to go work on goofy things19:36
*** ihrachys has joined #openstack-infra19:37
zarosdague: i remember fixing stuff in timeout plugin but don't remember what it was atm19:38
zaroclarkb: have we ever seen this issue?
*** daemontool has joined #openstack-infra19:38
clarkbsdague: zaro the plugin work happened but no one updated the jobs19:38
clarkbzaro: that may be related to apache 500ing19:39
zaroclarkb: i remember updating the macro upon the fix to the plugin19:39
ikhudoshynHello everybody! Could you please take a look at
ikhudoshynclarkb: jeblair AJaeger sdague ^^19:40
zaroclarkb: the disccusion [1] has a specific stack trace.19:40
*** annegentle has quit IRC19:40
ikhudoshynwe're about to move Rally testing to keystone api v3 and to add a dedicated job for v219:40
anteayaslogan621: welcome, do join us and look at a few patches19:41
anteayaslogan621: starting out by saying "I looked at this." is fine19:41
krotscheck1jeblair: Daycare mode. Back when possible.19:42
*** krotscheck1 is now known as krotscheck_dcm19:42
clarkbzaro: I can check for that in a bit19:42
*** achanda has quit IRC19:43
*** annegentle has joined #openstack-infra19:43
*** vgridnev has quit IRC19:43
*** boris-42 has quit IRC19:43
*** _nadya_ has joined #openstack-infra19:44
*** Sukhdev has quit IRC19:44
zarosdague, clarkb : RE: jenkins timeout macro,
*** sdake has joined #openstack-infra19:47
*** ociuhandu has quit IRC19:47
clarkbzaro: ah ok so the missing piece is in devstack-gate to do the soft timeout19:48
clarkbthenremove all the vars for setting in the jobs19:48
*** infra-red has joined #openstack-infra19:49
slogan621andreas: would you suggest setting up pypi now is important, or should we just remove references to pypi and revisit at a later time? I don't believe pyi is a requirement of all projects, correct?19:50
anteayaslogan621: pypi is not a requirement19:51
AJaegerslogan621: you can revisit at a later time.19:51
anteayaslogan621: if you are uncertain remove the job for now19:51
slogan621okay, we'll do that, and thanks for the catch19:51
*** daemontool has quit IRC19:51
AJaegerpersonally, i would do it now so that nobody else takes the name ;) but that's your call.19:51
anteayaand as AJaeger says, add it when you want to support it19:51
anteayaslogan621: you can register the name on pypi but still hold off on the pypi job if you like19:52
anteayaand as AJaeger getting the name is useful19:52
*** infra-re_ has quit IRC19:52
AJaegerzaro, we do use build-timeout ...19:52
jeblairzaro, clarkb: i will propose a d-g patch19:52
*** daemontool has joined #openstack-infra19:53
*** rajinir has quit IRC19:55
AJaegerjeblair: so, missing piece is to take in d-g BUILD_TIMEOUT variable? And then change project-config to not have two timeouts?19:55
clarkbzaro: [2016-02-02 23:13:16,765] WARN : Could not process key for channel java.nio.channels.SocketChannel shows up pretty frequently in yesterday's error log19:55
clarkbzaro: does not show up in todays but we restarted gerrit which may explain it19:55
*** diana_clarke has quit IRC19:57
*** jsavak has quit IRC19:58
*** jsavak has joined #openstack-infra19:58
*** achanda has joined #openstack-infra19:59
*** baoli_ has quit IRC19:59
zaroclarkb: wouldn't that issue cause incoming connections to bloat? did we see that?20:00
*** sdake_ has joined #openstack-infra20:00
*** annegentle has quit IRC20:01
AJaegerikhudoshyn: what identity version do you currently test against (without the change)?20:01
clarkbno I think this is a symptom of incoming connections bloating20:01
*** slogan621 has quit IRC20:02
*** annegentle has joined #openstack-infra20:02
*** vgridnev has joined #openstack-infra20:02
ikhudoshynAJaeger: once sdague reverted that patch to Devstack, we currently test against v220:02
*** diana_clarke has joined #openstack-infra20:02
AJaegerikhudoshyn: so, your change does two things: 1) Switch default for all jobs to v3; 2) Add new v2 job. Correct?20:03
ikhudoshynAJaeger: with my pathc we're about to run everything about v3 explicitly20:03
ikhudoshynand have one job that runs against v2, explicitly as well20:03
*** annegentle has quit IRC20:03
*** sdake has quit IRC20:03
*** annegentle has joined #openstack-infra20:04
zaroclarkb: alright, want to try to increase maxQueued to 200 before increasing memory?20:04
fungias for the bare uuid hosts showing up in puppetdb, it looks like we've somehow started showing no hostnames for unreported hosts20:04
zaroi can create the changes for that20:04
AJaegerikhudoshyn: I'll comment on the patch - and will also request a better commit message.20:04
clarkbzaro: we can, though I think it is all tied together, bigger queues require more memory20:04
clarkbzaro: but its worth a shot20:04
*** annegentle has quit IRC20:05
zaroclarkb: alright gotta afk for a little bit but changes should be up before end of day.20:05
fungidid we merge something in the past day or so which could alter how puppetdb tracks reporters?20:05
*** annegentle has joined #openstack-infra20:05
ikhudoshynAJaeger: yep, thanks for pointing that. I'll update the message right now20:06
AJaegerikhudoshyn: wait for my comments ;)20:06
AJaegerikhudoshyn: added them...20:06
clarkbzaro: it is also entirely possible that queues are small cost and queuing up avoids 500s that make apache unhappy20:07
clarkbzaro: so ya I would go ahead and give it a shot20:07
openstackgerritCostin Galan proposed openstack/requirements: Remove "args[0]" before trying to rename the file
*** annegentle has quit IRC20:08
jeblairzaro, clarkb: i don't see evidence that BUILD_TIMEOUT is set on this page:
*** annegentle has joined #openstack-infra20:08
clarkbjeblair: I agree, I see it in the config but not the injected vars list20:09
*** ihrachys has quit IRC20:10
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate: Use BUILD_TIMEOUT
jeblairclarkb, zaro: ^ i included a debugging echo, so we can check that it's actually there20:11
jeblairsdague: ^20:11
*** ybathia has joined #openstack-infra20:12
fungiso... it appears we have duplicate keys in the puppetdb postgres backend... this is starting to trigger a bit of déjà vu for me20:12
jeblairfungi: you and the dbms apparently20:13
fungistarted showing up in the log at 2016-02-03 18:41:5920:13
fungijeblair: ha! yes20:13
*** imcsk8 has joined #openstack-infra20:14
sdaguejeblair: cool20:14
*** nmagnezi has joined #openstack-infra20:14
sdagueseems like we might want to bump up the default to 120 in d-g at the same time, as 60 is not a great default for it if not provided20:15
sdaguethat that will be super helpful to not have 2 values in the template20:15
AJaeger"BUILD_TIMEOUT is 7500000"20:16
*** ldnunes has quit IRC20:16
jeblairsdague: yeah probably so; was just trying to keep change minimal20:16
jeblairAJaeger: HA!20:16
*** jsavak has quit IRC20:16
fungibeginning at 18:40:47 jenkins02 reports (for example) are now indicated in the logs reporting as 44dbb9fb-dce7-4694-9365-91e25f97eb1f instead20:16
fungiand also as jenkins0220:16
clarkbit is over 9000020:16
*** annegentle has quit IRC20:16
sdagueAJaeger: nice20:16
AJaegerjeblair: but the job has timeout: 12520:17
jeblairAJaeger: assuming that's milliseconds (java standard), that converts to 125 minutes20:17
jeblairso that seems to match up20:17
imcsk8hello, where can i download a centos image like the one is used on the gate jobs?20:17
sdagueah, yeh, that makes a ton of sense20:17
openstackgerritIllia Khudoshyn proposed openstack-infra/project-config: Run all Rally jobs against Keystone API v3. Add job for v2.
jeblairso we just need to divide by 60k20:17
fungiso somehow we've got hosts now reporting into puppetdb as both their hostname and their nova uuid, and i think that's the source of the duplicate keys it's complaining about20:17
*** jsavak has joined #openstack-infra20:17
AJaegerjeblair: yes, should be milliseconds - your math is right. It works!20:18
jeblairfungi: are they doing it during the same runs?20:18
anteayadougwig: you might want to take a peek at
AJaegerjeblair: output the value in minutes, please ;)20:18
fungineed to see what changes we've made today that would have kicked in an hour or so ago20:18
jeblairfungi: hrm; i might expect some hosts to switch between name and uuid as we spun up a replacement but then deleted an old one; but both at once sounds buggy.20:19
fungialso the example i'm looking at is jenkins02. it's not been replaced in ages20:20
fungioh, actually its in the log with a "replace facts" under the uuid but "store report" under the hostname20:21
fungiso maybe this is not a duplicate report20:21
fungithough the duplicate key tracebacks look, from the index column name, like they're probably report unique identifiers20:22
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate: Use BUILD_TIMEOUT
*** fitoduarte has joined #openstack-infra20:23
AJaegeranteaya: could you review after a rebase again, please? The 3 stacked changes are fine to merge now20:23
jeblairAJaeger, clarkb, zaro, sdague: this looks right, so i'm going to drop the debug lines now:
AJaegerjeblair: really two hours now as default?20:25
AJaegerjeblair: yes, it's fine.20:25
jeblairAJaeger: i think that makes sense for devstack-gate itself, yeah.  this won't affect us running in jenkins so we don't need to bikeshed too much over it :)20:26
*** Sukhdev has joined #openstack-infra20:28
AJaegerjeblair: so, what would we set in jenkins? I agree, jenkins value is important...20:28
imcsk8pabelanger: hello, do you have the centos gate image for download? or a way for me to generate it?20:28
jeblairAJaeger: well, that's the earlier conversation :)20:28
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate: Use BUILD_TIMEOUT
jeblairAJaeger, clarkb, zaro, sdague: ^ should be ready20:30
*** sabeen2 has quit IRC20:31
*** sabeen1 has joined #openstack-infra20:31
sdaguejeblair: +220:31
clarkbjeblair: trade you ? which is another d-g reliability improvement20:31
clarkbsdague: ^ you may be intrested in that one as well20:31
*** jsavak has quit IRC20:31
jeblairclarkb: do we want to follow up with bluebox?20:32
funginibalizer: do you have much experience delving into puppetdb database issues?20:32
clarkbjeblair: I think we do, but at the same time we don't want to wait on that20:32
jeblairclarkb: are we ready to say the letters j l k? :)20:32
jeblairclarkb: i wasn't clear on whether we still thought the conntrack module could still be an issue20:32
sdagueclarkb: oh, is that why internap is an issue?20:33
clarkbsdague: and bluebox20:33
jeblairsdague: internap should be fixed, right clarkb ?20:33
clarkbjeblair: I think it may still be the issue20:33
clarkbjeblair: maybe, the health dashboard is buggy so I wasn't able to say definitively20:33
fungii'm not finding the credentials to log into the puppetdb postgres instance to perform queries interactively20:33
clarkbjeblair: it looked like we had a 1:1 pass fail ratio but some passes were actually fails20:33
*** jsavak has joined #openstack-infra20:33
jeblairclarkb: logstash?20:34
clarkbjeblair: I hvaen't asekd it yet but can do so really quick20:34
sdagueclarkb: do you have multinode jobs passing on one of them with this?20:34
sdaguethe full job was ovh in that run20:34
clarkbsdague: the grenade one passed on bluebox on one of the rechecks, the non  grenade one is independently broken20:35
clarkbvolume based live migration is failing frquently regardless of gre/vxlan20:35
openstackgerritMerged openstack-infra/project-config: Re-Enable django_openstack_auth/designate-dashboard translations
sdagueclarkb: sure, just that one test was failing though20:35
clarkbright, we no longer fail on ssh issues20:36
clarkbwe fail on other things that were masked by ssh issues20:36
jeblairclarkb: so i should see a gate-grenade-dsvm-multinode on bb?20:36
jeblair(i haven't found it yet)20:36
clarkbjeblair: it may have been internap let me look at the logs20:37
tristanCgreeting folks, is it normal if recheck comment doesn't retrigger checks ?20:37
anteayatristanC: what patch?20:37
jeblairclarkb: i'm just seeing rax and ovh for that job on that change20:38
fungitristanC: the usual reason it wouldn't is if it's already running and hasn't reported yet20:38
clarkbhrm I thought I got it to run on either internap or bb for each of the three multinode jobs20:38
*** ybathia has quit IRC20:39
clarkbya I don't see it either so I must've misremembered20:39
clarkbit definitely ran the nova net job on bluebox and failed20:39
tristanCanteaya: it's , but fungi nailed it and it's still queued20:39
clarkbbut that seemed "normal"20:39
jeblairclarkb: oh there's a neutron:
jeblairone each success/failure20:40
anteayatristanC: ah ha20:40
fungitristanC: if it's a security fix and needs to get merged urgently, we can bypass the check pipeline and enqueue it directly into the gate since there's a pretty lengthy backlog in check20:41
openstackgerritIllia Khudoshyn proposed openstack-infra/project-config: Run all Rally jobs against Keystone API v3. Add job for v2.
jeblairsdague: why the recheck?20:41
clarkblogstash says internap multinde grenade is not working20:41
jeblairroaet-: on
tristanCfungi: then yes please, I'd like to recheck the master fix since the failure seems unrelated20:41
jeblairsdague: on
jeblairsdague: just want to know what you're looking for20:42
sdaguejeblair: to try to get multinode to hit different nodes20:42
*** tiswanso has quit IRC20:42
jeblairsdague: do you want to see at least one internap result?20:42
jeblairok, so i think we have ovh, rax, and bb covered; just waiting on internap then20:42
tristanCfungi: not master, I meant stable/liberty which is change 275736 << this one could use a recheck probably because of bug 150061520:43
openstackbug 1500615 in OpenStack Compute (nova) "Large Ops scenario is taking too long" [High,Confirmed]
openstackgerritMerged openstack-infra/shade: Fix normalize_role_assignments() return value
clarkbsdague: jeblair will show us20:44
fungitristanC: yeah, i did 27573620:44
sdagueclarkb: great20:44
fungitristanC: it's in the gate pipeline now and should get workers allocated as soon as the window progresses there by a couple changes20:44
jeblairclarkb, sdague: i +2d but did not +A and left a comment about current status20:44
* jeblair lunches20:44
AJaegeranteaya: thanks for the review! 275140 and 275225 are the next on the stack if you have further review time, please20:44
anteayaAJaeger: okay20:44
clarkbthe big differene between vxlan and gre is vxlan uses udp so if udp is allowed vxlan should work20:45
clarkbgre on the othre hand is its own ip protocol and subject to specialness20:45
tristanCfungi: thank!  I'm checkin the other patch now, they'll probably need another patchset20:45
*** gordc has quit IRC20:46
nibalizerfungi: not.much20:46
nibalizerand i am way past eod, but whats up?20:46
funginibalizer: oh, right, you're in eu20:47
*** sridhar_ram has quit IRC20:47
Shrewsjeblair: i have a shade question for you about something i *think* you added (inner exceptions) when you get time20:47
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: Run packstack jobs inside a root shell
*** dtardivel has quit IRC20:48
funginibalizer: puppetdb started complaining a couple hours ago about duplicate report keys in pgsql, and now a lot of our hosts are showing up in puppetboard as unreported and with their nova uuids instead of their hostnames20:48
*** annegentle has joined #openstack-infra20:48
funginibalizer: anyway, don't sweat it. enjoy your evening!20:49
* AJaeger thanks anteaya20:49
anteayaAJaeger: welcome, thanks for your patience with me20:49
anteayaAJaeger: you are doing the lion's share of the work on project-config and I'm sorry I don't have the energy to help you out more20:49
anteayaAJaeger: thanks for understanding20:49
AJaegerchallenge is if I have patches on my own - thanks for reviewing those, anteaya !20:51
AJaegerpabelanger: could you review a grafana change, please?
openstackgerritDaniel Wallace proposed openstack-infra/shade: granting and revoking privs to users and groups
nibalizerfungi: ya i think.we should.patch the upload_to_puppetdb ansible-puppet to take a dubug flag that will cause it to keep the json blob aroubd20:52
anteayaAJaeger: :) welcome20:52
nibalizerthat way we can inspect those for correctness20:53
*** apoorvad has quit IRC20:53
AJaegerfungi,  clarkb, jeblair: Does every job need a timeout? Do we have an implicit timeout? I see jobs like gate-manila-buildimage that have no timeout and wondered whether that's fine...20:54
openstackgerritMerged openstack-infra/project-config: Re-enable repos for translation
funginibalizer: i'm also noticing in our puppet_run_all.log that the hostname for, e.g., stopped being mentioned around the timeframe this started and its nova uuid began showing up instead20:54
funginibalizer: mostly thinking we've made a change of some sort somewhere that should be fixed/reverted, but so far not found any good potential candidates20:55
*** infra-re_ has joined #openstack-infra20:55
*** dims has quit IRC20:56
nibalizerdoes 02 show up ansible all --list-hosts ?20:56
clarkbAJaeger: there is a default timeout in defaults.yaml20:56
openstackgerritMerged openstack-infra/project-config: Cleanup translation scripts
funginibalizer: nope, bit its uuid does :/20:56
nibalizerin the last we saw ansible fall back to uids when we had hostname duplicates20:57
fungithe pgsql duplicate key constraint exceptions may just be a symptom of this20:57
nibalizerid agree with that20:57
fungii checked that first actually, there is one and only one in nova20:58
AJaegerclarkb: thanks, found it - 30 mins20:58
AJaegerthis is fine20:58
nibalizera conundrum20:58
*** infra-red has quit IRC20:58
nibalizermordred: Shrews any thoughts?20:58
funginibalizer: at least in rax-dfw... i don't know how to easily scan across all providers/regions20:59
fungibut it seems unlikely that we would have stood up duplicates of _most_ of our infrastructure in one of our other accounts known to ansible20:59
Shrewsnibalizer: i have no idea what crazy things ansible does outside the OS modules  :)20:59
AJaegerdhellmann: could you review again the solum reno change, please?
dhellmannAJaeger : sure, after this meeting21:01
AJaegerdhellmann: no hurry - thanks21:01
funginibalizer: Shrews: i wonder what would happen if i moved the current inventory cache out of the way and allowed it to get regenerated21:01
AJaegerfungi: is the rename of .log to .log.txt now fine to merge?
openstackgerritsebastian marcet proposed openstack-infra/openstackid: Fix on username generation
funginibalizer: Shrews: looks like it was last modified shortly before these symptoms began21:03
fungiAJaeger: i couldn't really tell if lifeless's comments were weak objections or merely observations. at any rate i don't have a strong feeling on it either way, it was proposed as one potential simple way out of the current challenge of viewing tox logs without having to download them to your local filesystem first21:05
*** dims has joined #openstack-infra21:05
*** kurtmartin has quit IRC21:06
*** amitgandhinz has quit IRC21:06
nibalizerfungi: shade itself has an inventory command you coukd run21:06
fungiwe could instead go with a more focused solution which just affected the tox logs and not every file with a .log extension, but i don't have time right now to write that if so21:06
fungiAJaeger: ^21:06
prometheanfirecan someone reworkflow this?
funginibalizer: what's shade's cli entrypoint?21:07
clarkbfungi: >>> import shade21:07
*** _nadya_ has quit IRC21:07
fungioh! there's a "shade-inventory" entrypoint21:07
fungitab completion ftw21:07
*** baoli has joined #openstack-infra21:07
*** EricGonc_ has joined #openstack-infra21:08
fungiit's no wonder we cache this. not exactly quick21:08
*** miqui has joined #openstack-infra21:08
AJaegerfungi, so problem is ? That works just fine in firefox and chromium, both open in the browser21:09
fungialso very verbose. i should have dumped it to a file21:09
clarkbsdague: jeblair is running on internap too21:09
*** amrith is now known as _amrith_21:09
*** EricGonczer_ has quit IRC21:10
AJaegerfungi, the change only takes care of tox files, nothing else. It's only in script21:10
fungiAJaeger: odd, for me in firefox it says i've chosen to open a file with type x-application/log and wants to know if i wish to download it or which application should handle it (and thinks libreoffice writer is an excellent default)21:10
fungiAJaeger: lifeless was pointing out that when you run testr-based tests under tox you will also end up with subunit streams with .log extensions21:11
fungiand that you may wish for a browser default of downloading those rather than trying to display them raw21:11
clarkbthey are binary so can't be treated as text21:12
fungialso the tox logs sort of are as well since they contain ansi escapes21:12
*** EricGonczer_ has joined #openstack-infra21:12
fungior potentially can at least21:12
anteayajeblair: 275483 needs a zuul restart to be in production, yes?21:12
*** infra-re_ has quit IRC21:12
*** apoorvad has joined #openstack-infra21:13
*** clayton has quit IRC21:14
*** jesusaurus has joined #openstack-infra21:14
*** boris-42 has joined #openstack-infra21:14
AJaegerfungi: I'll update to only get .tox/ log files - and then let other judge whether t otake it21:15
fungiAJaeger: awesome, thanks21:15
fungii'm cool with that as a slightly more focused solution21:15
*** tcammann_ has joined #openstack-infra21:16
*** EricGonc_ has quit IRC21:16
*** kgiusti has left #openstack-infra21:16
*** esp_ has joined #openstack-infra21:16
funginibalizer: Shrews: ansible-inventory on the puppetmaster also only reports one jenkins02, but it's possible something went weird when it last updated the cache i guess21:16
*** Swami has quit IRC21:17
AJaegerfungi, done.21:17
AJaegerAnd time to leave!21:17
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rename .log to .log.txt in run-tox
Sam-I-AmAJaeger: waiiiit! :)21:17
* AJaeger waves good night21:17
AJaegerSam-I-Am: leave it as comment, I'll answer tomorrow21:17
anteayaAJaeger: good night21:18
Sam-I-Amok, its in -doc21:18
Sam-I-Amtox problem i'm trying to solve21:18
*** auggy has left #openstack-infra21:18
anteayaSam-I-Am: ah a -doc thing21:18
Sam-I-Amalthough maybe someone else here can help21:18
Sam-I-Amtrying to figure out the problem here -
anteayaSam-I-Am: ask away21:18
Sam-I-Amtox -egenconfig works fine when i run it locally21:18
Sam-I-Amyet its breaking in the gate21:18
funginibalizer: Shrews: so anyway, i have a feeling if i let it regenerate the cache it'll go back to normal. but wondering how to track down what might have caused this21:18
*** vgridnev has quit IRC21:18
anteayaSam-I-Am: what is the link to the tox file used for that test?21:19
fungianyway, i'll save the old cache and see what happens on the next pass21:19
*** gordc has joined #openstack-infra21:20
*** tcammann_ has quit IRC21:20
*** esp_ has quit IRC21:21
Shrewsfungi: i'm fairly certain shade inventory code doesn't much around with server names. did this happen with a single provider? or all?21:21
fungiShrews: hard to tell, but looks like it may have only happened for hosts in rax-dfw. unfortunately that's where most of our hosts are so it's hard to rule out it being random21:22
*** jesusaurus has quit IRC21:22
*** amitgandhinz has joined #openstack-infra21:22
*** auggy has joined #openstack-infra21:22
*** Swami has joined #openstack-infra21:23
fungiShrews: oh, actually it did happen for a couple in other clouds too21:23
Shrewsfungi: copypasta me a few examples?21:24
*** daemontool_ has joined #openstack-infra21:25
gtmanfredis there a git review command to go get a clone a pr down to look at it?21:25
fungialso after moving the ansible-inventory.cache out of the way, the new one has the correct hostnames again21:25
fungiShrews: putting that together now21:25
Shrewsgtmanfred: git review -d <review#>21:25
anteayaSam-I-Am: the only other project I see that uses oslo-config-generator --namespace is watcher21:25
fungigtmanfred: git review -d or -x depending on whether you want to fetch or cherry-pick21:25
gtmanfredcool, thanks21:25
anteayaSam-I-Am: your tox.ini doesn't look that different to my eyes:
Sam-I-Amanteaya: yeah... this is strange.21:26
fungialso git review --help if the manpage is installed in your manpath21:26
fungior git-review --help if it's not21:27
sdaguejust noticed this is +A -
sdaguethat's going to make me do a happy dance once the whole stack is in21:27
*** julim has quit IRC21:27
gtmanfredjeblair: cool, that is exactly what I was looking for, but couldn't find it :P21:27
*** daemontool has quit IRC21:27
jeblairsdague: yeah, i want to vos release a couple of times manually to check out the timing on that, and also probably some manual tests, before landing the next 221:28
*** amitgandhinz has quit IRC21:28
Sam-I-Amanteaya: know any other tox experts?21:28
Sam-I-AmAJaeger left for the night :/21:28
anteayaSam-I-Am: networking-ovn has its own script: while watcher doesn't21:28
*** zul has joined #openstack-infra21:29
anteayaSam-I-Am: mordred and lifeless, but mordred is traveling somewhere (no idea) and lifeless is at lca21:29
Sam-I-Amanteaya: hummmm21:29
dhellmannanteaya, Sam-I-Am : most projects use a separate configuration file with the namespace options in it, but it should work the same way. I see an error coming from in that log, and the line "export ZUUL_BRANCH=${ZUUL_BRANCH-$BRANCH}" looks wrong. Probably should be ":-" not just "-".21:29
Sam-I-Amdhellmann: oddly no one has touched this file in 2 months21:30
anteayaah thanks dhellmann21:30
dhellmannanteaya, Sam-I-Am : although the error is being reported on line 2121:30
anteayaSam-I-Am: did it work before?21:30
anteayaa lot of the neutron drivers appear to have their own tox install file:
fungiShrews: so the bad ansible inventory cache is and then the good one it just generated after i moved that out of the way is
Sam-I-Amdhellmann: line 21 is just a cwd21:31
fungiShrews: the differences are pretty striking21:31
dhellmannSam-I-Am : yeah. it's interesting that there's no output from that script.21:32
fungiShrews: i'm leaning toward "rackspace's nova api returned crazy" as an answer, but don't know where else this might go wrong21:32
*** kzaitsev_mb has joined #openstack-infra21:32
*** jsavak has quit IRC21:32
Sam-I-Amdhellmann: odd that locally running tox -egenconfig works, yet running it within devstack appears to break21:33
Sam-I-Amdhellmann: even running devstack locally21:33
*** abitha has joined #openstack-infra21:33
fungiShrews: and looking at that, i do think it was just rax-dfw after all21:33
anteayaSam-I-Am: and this script uses zuul-cloner which we have discovered only works in the check and gate pipelines currently, though that may be tangental21:33
anteayaSam-I-Am: what pipeline did this run in?21:33
*** jsavak has joined #openstack-infra21:33
fungiShrews: i think the others i thought were affected were really just missing from the puppetdb21:33
Sam-I-Amanteaya: i dont know21:33
dhellmannSam-I-Am : what version of tox do you have?21:33
Sam-I-Amdhellmann: locally 2.3.121:34
*** sabeen1 has quit IRC21:34
fungiShrews: and the ones in dfw i thought weren't affected were likely reporting from a previous cache version21:34
*** sabeen1 has joined #openstack-infra21:34
clarkbdhellmann: Sam-I-Am could be missing header packages21:35
dhellmannclarkb : the error is "Bad substitution"21:35
Shrewsfungi: looking21:35
dhellmannSam-I-Am : try changing the shebang line in your script to use bash instead of sh21:35
Sam-I-Amdhellmann: last time i saw an obscure error like that, it was a tox version problem21:35
*** achanda has quit IRC21:35
*** ybathia has joined #openstack-infra21:36
Sam-I-Amdhellmann: in the tox_install script?21:36
fungiShrews: my guess is that the dfw endpoint returned empty hostnames for everything when nova list was called21:36
dhellmannSam-I-Am : yes21:36
dhellmannSam-I-Am : also move the set -ex above all of the other statements, so we can see if any of the first few is being run21:36
Sam-I-Amdhellmann: done21:37
Sam-I-Amdhellmann: submitting a patch now21:37
Sam-I-Amunless theres something else to fix21:37
dhellmannthose should help debug21:38
*** nmagnezi has quit IRC21:38
Shrewsfungi: is it possible there could have been a change in shade versions between runs?21:39
*** aysyd has quit IRC21:39
*** sabeen3 has joined #openstack-infra21:39
*** daemontool_ has quit IRC21:39
clarkbsdague passed on internap if you eant to approve21:40
*** daemontool_ has joined #openstack-infra21:40
*** weshay_xchat has quit IRC21:40
*** zul has quit IRC21:40
sdagueclarkb: done21:40
Shrewsfungi: there is a 'use_hostnames' option that controls that behavior. config changes?21:40
openstackgerritMerged openstack-infra/system-config: Add wheel working directory to wheel slave
*** thingee has quit IRC21:41
fungiShrews: shade update seems unlikely. we're using "shade==1.4.0  # git sha 457ea84" from 4 weeks ago21:41
Sam-I-Amdhellmann: patch is gating soon, but not sure a simple change to tox.ini will trigger the dsvm jobs21:41
Sam-I-Amdhellmann: so i might need to touch a random file somewhere21:41
*** sabeen1 has quit IRC21:41
dhellmannSam-I-Am : oh, I thought you might insert the patch under
Shrewsfungi: k. config files?21:42
*** clayton has joined #openstack-infra21:43
Sam-I-Amdhellmann: good point. so... just to make sure i'm thinking straight... just checkout that patch on top of the tox patch?21:43
Shrewsfungi: notably the clouds.yaml21:43
*** achanda has joined #openstack-infra21:43
fungiShrews: only place we seem to set it is here (digging in git blame now)21:43
fungiShrews: been there since merged a month ago21:44
dhellmannSam-I-Am : do "git review -d 275392" then cherry pick your patch onto the branch created then rebase -i and swap the order then git review to resubmit21:44
*** dkranz has quit IRC21:45
dhellmannSam-I-Am : of course if that first patch is yours, you probably have a copy locally and don't need to do the git review -d part21:46
Sam-I-Amdhellmann: thats what i was wondering, since it is mine21:46
dhellmannSam-I-Am : ah, ok, then just get your tox changes into the same branch and rebase to swap the order21:46
*** jistr has joined #openstack-infra21:46
Shrewsfungi: hrm, dunno then. unless somebody ran inventory manually with a different clouds.yaml and modified the cache accidentally. or rax just went cuckoo for a bit21:47
*** jistr has quit IRC21:47
fungiShrews: that last one seems the most likely21:47
fungii'll try to be on the lookout for future occurrences and maybe a pattern will emerge21:48
*** thingee has joined #openstack-infra21:48
Shrewsyeah. i'm not even certain that first scenario is even possible21:48
*** dkranz has joined #openstack-infra21:49
*** FallenPegasus has quit IRC21:51
*** FallenPegasus has joined #openstack-infra21:52
Sam-I-Amdhellmann: i'm having trouble with this21:53
anteayaSam-I-Am: what is happening now?21:53
openstackgerritMerged openstack-infra/project-config: Create jobs for a wheel mirror
Sam-I-Amanteaya: git and i dont get along21:53
Sam-I-Amespecially for things i dont use very often21:53
anteayajust now or most days21:53
dhellmannSam-I-Am : submit your two patches and I'll get them into the right order for you21:53
jeblairdhellmann, Sam-I-Am: have you seen 'git restack' ?21:53
dhellmannthat is, submit the new one separately21:53
*** salv-orlando has joined #openstack-infra21:54
Sam-I-Amdhellmann: the new one is 27593921:54
dhellmannjeblair : no, have a reference?21:54
jeblairwe don't need to further derail this now, but it may make this sort of situation easier21:54
fungisomeone mentioned it on a popular mailing list recently21:54
jeblairdhellmann: lemme git one21:54
dhellmannjeblair : I see what you did there21:54
*** FallenPegasus has quit IRC21:54
funginice typo (or was it?!?)21:55
*** jsavak has quit IRC21:55
jeblairi said 'large' but it also works with 2 patches.21:55
dhellmannSam-I-Am : resubmitted21:55
lifelessSam-I-Am: whats up ?21:56
fungiheck, it even works with one patch, though it's pretty uninteresting21:56
jeblairanteaya: you may remember that from december ^; finally released it as a tool21:56
Sam-I-Amdhellmann: thanks a bunch21:56
dhellmannjeblair : that looks very nice, I'll put that on my list to set up21:56
*** salv-orl_ has quit IRC21:56
Sam-I-Amlifeless: trying to resolve some funky tox problem in the gate... and only when -egenconfig is run within devstack apparently21:56
*** dkranz has quit IRC21:57
jeblairdhellmann: thanks; it's one of those "you totally don't need this because it's so simple to just type a bunch of long git commands" things that once you use is just so much nicer :)21:57
*** dkranz has joined #openstack-infra21:58
dhellmannjeblair : yes, I see myself using this a lot21:59
anteayajeblair: yay21:59
anteayajeblair: thank you21:59
*** dkranz has quit IRC22:00
ikhudoshynAJaeger: I've missed your +2 and updated the patch so that ALL devstack jobs ec22:00
*** dkranz has joined #openstack-infra22:00
ikhudoshyn...except dedicated one run against keystone api v322:01
ikhudoshynAJaeger: could you pls re-approve
*** dkranz has quit IRC22:02
*** rossella_s has quit IRC22:02
*** rossella_s has joined #openstack-infra22:03
*** dkranz has joined #openstack-infra22:04
anteayaikhudoshyn: he is off for the night22:05
*** thorst_ has quit IRC22:05
ikhudoshynanteaya: thanks22:05
*** PsionTheory has joined #openstack-infra22:05
*** thorst_ has joined #openstack-infra22:06
*** sdake_ has quit IRC22:06
ikhudoshyncould you pls advise me somebody who I could address22:06
anteayastart with patience22:06
anteayaif someone has the time they will look22:06
krotscheck_dcmjeblair: Looks like the job merged before the system-config changes did.22:07
ikhudoshynanteaya: fair enough, thanks22:07
openstackgerritDoug Hellmann proposed openstack-infra/project-config: add a job to send automated announcements of releases
anteayathank you22:07
*** thorst__ has joined #openstack-infra22:08
dhellmannfungi, jeblair: rebasing this patch ^^ is starting to turn into a bit of a mess. Can you move it up your review queue for me, please?22:08
*** trown is now known as trown|outttypeww22:08
dhellmannI had to make a lot of changes across one of the most actively changed files, so I'm hitting lots of merge conflicts. :-/22:08
krotscheck_dcmfun times!22:09
*** thorst_ has quit IRC22:09
openstackgerritRichard Jones proposed openstack/requirements: Revert "Exclude xvfbwrapper 0.2.8 from global requirements"
*** lane_kong has joined #openstack-infra22:11
*** lane_kong has left #openstack-infra22:11
*** _amrith_ is now known as amrith22:12
*** HeOS_ has joined #openstack-infra22:12
*** [HeOS] is now known as HeOS22:12
tristanCinteresting, 275736 is in both gate and check pipeline, it's a schrödinger change :)22:12
*** thorst__ has quit IRC22:12
anteayadon't look in the box22:14
fungitristanC: yeah, the one in check will report into gerrit but will not impact gating22:14
*** dkranz is now known as dkranz-brb22:14
anteayastill working on getting the better of this cold, calling it a night, thanks everyone, see you tomorrow22:15
pigmejtristanC: lol22:15
jeblairi just spent 10 minutes trying to figure out why my workstation stopped responding22:15
*** dkranz-brb is now known as dkranz22:15
jeblairit's because tox doesn't pass through variables anymore22:15
*** henrynash has quit IRC22:16
jeblairso my instruction to run zuul unit tests on a tmpfs was not honored22:16
*** gordc has quit IRC22:16
openstackgerritIan Wienand proposed openstack-infra/puppet-openstackci: nodepool : add flag to install diskimage-builder from git
*** achanda has quit IRC22:16
fungidhellmann: how was it determined which entries in the projects.yaml should get that job-group? just any using publish-to-pypi or openstack-server-release-jobs in the layout?22:16
*** krotscheck_dcm has quit IRC22:17
dhellmannfungi : yes22:17
*** krotscheck1 has joined #openstack-infra22:17
fungii take it there wasn't another job-group which it could be added to for similar effect?22:17
*** dims_ has joined #openstack-infra22:17
fungiseems like python-jobs would have probably worked since that's the one which also instantiates the tarball job template22:18
jeblairdhellmann: yeah, you're going to hate me for this but i was just coming to the same conclusion as fungi22:19
fungiit's a bit of a balancing act between instantiating the template for some projects which won't be running the job, vs adding another job-group which lots of projects will need to remember to add22:19
*** kzaitsev_mb has quit IRC22:19
dhellmannthe announcement doesn't only apply to python things22:19
dhellmannxstatic javascript packages don't have python-jobs, right?22:19
openstackgerritMerged openstack-infra/nodepool: Remove the unused 'reset' setting from the doc
*** dims has quit IRC22:21
jeblairdhellmann: i think they do (they are python packages)22:21
fungii would be surprised if the projects using publish-to-pypi and openstack-server-release-jobs didn't add python-jobs22:21
dhellmannok, well, my point was that I thought we would want this on additional things22:21
dhellmannwe can deal with that when it comes up, I'll redo the patch22:21
fungiyou don't currently, at least, seem to be running it on non-python things22:21
jeblairdhellmann: i think if we do, we could add it to the 'javascript-jobs' job group or something22:22
dhellmannyeah, I was trying to plan ahead by not baking python assumptions in22:22
dhellmannfungi : where do we stand on the releases.o.o work? I gather there were some fires with other things today?22:22
fungimostly concerned people are cargo-culting old project addition changes (which is bad but we don't seem to be able to break them of), and they're going to miss adding this and we're going to somewhat often end up troubleshooting why their announcements aren't happening22:22
fungidhellmann: yes, sorry, i've not really started in on building that worker yet22:23
dhellmannfungi : ack, I'll redo the announcement patch22:23
jeblairdhellmann: good news!  the next revision will be shorter!22:24
*** kzaitsev_mb has joined #openstack-infra22:24
fungiand no longer prone to merge conflicts!22:24
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Expose br_name in controller and compute classes
*** dkranz has quit IRC22:25
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node
pabelangerimcsk8: you can use tools/ in project-config to build the image22:26
*** jsavak has joined #openstack-infra22:26
openstackgerritMerged openstack-infra/project-config: Add centos7 and fedora-23 to bindep-fallback
*** dims_ has quit IRC22:27
imcsk8pabelanger: thanks i was already on it :)22:27
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add Infra Cloud compute node definition
*** gildub has joined #openstack-infra22:28
pabelangerAJaeger: ianw left comments, using templates in grafyaml might be better22:29
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Remove ML reporting for periodic-qa jobs
*** tzn has quit IRC22:30
ianwpabelanger: seems like we're about 75% of the way there with the existing log config scripts22:30
fungipabelanger: thanks for doing that centos and fedora addition for the bindep-fallback jobs22:30
fungithat'll get us a good picture of how it's faring across additional node types before we start deciding to make at least the ubuntu one run in check/gate22:31
*** thorst has joined #openstack-infra22:31
*** jsavak has quit IRC22:31
*** jsavak has joined #openstack-infra22:31
pabelangerfungi: sure, now I need to test them and see if they work22:32
pabelangerianw: indeed22:32
*** sdake has joined #openstack-infra22:32
fungipabelanger: well, the ubuntu version didn't when i wrote that job, because someone had added a centos-only package to the list without marking it as only being available there22:33
fungiso i wouldn't be surprised to turn up similar discrepancies22:33
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Fix neutron ssl relationship
pabelangerfungi: sure, I'll have some spare time tomorrow to dig into it and see the current state22:34
pabelangerfungi: then review trusty and move it (non)voting22:34
fungipabelanger: good news is the job makes those easy to spot22:34
*** EricGonczer_ has quit IRC22:35
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Add pypi publish job to tempest
pabelangerfungi: Indeed22:35
*** EricGonczer_ has joined #openstack-infra22:35
*** thorst has quit IRC22:35
openstackgerritDoug Hellmann proposed openstack-infra/project-config: add a job to send automated announcements of releases
dhellmannfungi , jeblair : reworked ^^22:37
ianwclarkb: doh, where did we get to with the nodepool logging output22:39
jeblairdhellmann: are we going to release release-tools some day?  i'm wondiring about changing the zuul-cloner to a pip install at some point...22:39
clarkbianw: havent had a chance to debug, now that multinode shpuld work again I can dig in22:40
ianwclarkb: i'll take alook, i just got distracted yesterday and forgot :(22:40
mgagnewhat was it again the limitations of masterless projects in gerrit?22:40
clarkbmgagne: HEAD is not replicated is the first one22:41
ianwclarkb: it might be worth a cron job to rm -rf the log dir just before the next build, there's a lot of cruft in there now22:41
clarkbso requires manual intervention to upfate22:41
ianwactually, should probably puppet something that clears out files not touched in certain date22:41
clarkbianw: or delete old logs with find22:41
mgagneclarkb tried to clone a masterless project imported by jeepyb. it didn't go well22:41
ianwclarkb: yeah, ok, leave that with me too :)22:41
*** thorst has joined #openstack-infra22:42
dhellmannjeblair : maybe, but I have no immediate plans to release them22:43
fungimgagne: yeah, we were supposed to not be approving project imports if they lacked a master branch22:43
fungithough i don't think we have a job checking for that specifically22:44
*** sdake has quit IRC22:44
mgagnefungi you have no authority on my private gerrit install :D22:44
*** ok_delta has joined #openstack-infra22:44
jeblairfungi is lord of all gerrits22:44
*** ok___delta has joined #openstack-infra22:44
mgagne looks to be 2.12 only22:44
fungimgagne: oh! not _our_ gerrit22:45
mgagnefungi well, I learn a lot from your (bad) experience guys =)22:45
jeblairfungi, dhellmann: that will add 1120 new jobs22:45
jeblairwe may need to babysit that one22:46
fungiyeah, we should probably not approve until we're ready to handhold jjb updates22:46
dhellmannjeblair : wow22:46
fungiinteresting that it's adding more jobs than we have projects22:46
jeblairindeed, we have 93722:47
jeblairi'm looking into the discrepancy22:48
*** krotscheck1 has quit IRC22:49
jeblairthe job gear-announce-release shows up in the diff 3 times, as an example22:50
jeblairjjb claims to have genarted 420 more jobs22:53
openstackgerritDaniel Wallace proposed openstack-infra/shade: granting and revoking privs to users and groups
clarkbmgagne: my suggestion wpuld be to have a master branch that isnt used22:54
clarkbits cheap with git and makes tooling happy22:55
mgagnewe try to avoid so people don't think they can use it22:55
mgagneit's not about cheap =)22:55
openstackgerritDaniel Wallace proposed openstack-infra/shade: granting and revoking privs to users and groups
*** jsavak has quit IRC22:55
crinklejeblair: yolanda turns out the reason why the puppet in my dev env "worked" was that it was using production credentials and I'd forgotten to set /etc/hosts to point to localhost >.<22:56
mgagne"git symbolic-ref HEAD refs/heads/foobar" works just fine. just have to make sure it's updated to our replication22:56
crinklejeblair: yolanda and the difference between the cloud in uswest and the cloud on my laptop appears to be kilo versus liberty22:56
*** annegentle has quit IRC22:56
jeblairfungi: ok i don't know what about diff is making it show some files 3 times.  but i need to pop this off my stack.  i assume 420 is the correct number.  think that needs babysitting?22:57
fungimgagne: alternative is a master branch with only a readme saying "no development happens on this branch"22:57
fungijeblair: that seems marginal. i'm happy to do it and keep an eye on it22:58
fungijust to be on the safe side22:58
*** sabeen3 has quit IRC22:58
fungifeel free to approve and i'll babysit22:58
openstackgerritMerged openstack-infra/system-config: Changed ownership of wheel.keytab to jenkins
jeblairfungi: done22:58
openstackgerritIan Wienand proposed openstack-infra/puppet-nodepool: Add periodic cleanup of log files
fungithanks. i'll get screen sessions going with updates once it merges22:59
*** burgerk has quit IRC22:59
*** jpr has quit IRC22:59
*** ok_delta has quit IRC22:59
*** ok___delta has quit IRC22:59
*** ccrouch has joined #openstack-infra23:00
dhellmannfungi , jeblair: thanks!23:00
*** piet has quit IRC23:01
openstackgerritDaniel Wallace proposed openstack-infra/shade: granting and revoking privs to users and groups
*** regXboi has quit IRC23:03
openstackgerritIan Wienand proposed openstack-infra/puppet-openstackci: nodepool: Enable periodic cleanup of dib logs
ccrouchlifeless: you mind elaborating on this a little bit:
ccrouchis that work underway? how can people help out?23:06
*** erlon has quit IRC23:06
*** thorst has quit IRC23:07
*** thorst has joined #openstack-infra23:07
*** xyang1 has quit IRC23:08
fungiccrouch: there are some jobs using it already. neutron is using it for most tox-based jobs that run in check/gate/experimental pipelines23:09
*** achanda has joined #openstack-infra23:09
*** eharney has quit IRC23:09
*** Sukhdev has quit IRC23:10
*** dingyichen has joined #openstack-infra23:11
*** thorst has quit IRC23:12
*** achanda has quit IRC23:12
openstackgerritJames E. Blair proposed openstack-infra/zuul: Pass ZUUL_TEST_ROOT through tox
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add job mutex support
openstackgerritJames E. Blair proposed openstack-infra/project-config: Mutex the trusty wheel mirror/release jobs
jeblairokay, i broke my zuulv2 fast for this ^.  i don't think it's critical, but it does make the afs stuff a little more atomic and predictable.23:13
*** EricGonczer_ has quit IRC23:14
pabelangerjeblair: interesting23:18
*** Sukhdev has joined #openstack-infra23:18
ccrouchooh fungi thanks for that23:18
ccrouchso you not aware of any big blockers?23:19
openstackgerritJames E. Blair proposed openstack-infra/zuul: Pass ZUUL_TEST_ROOT through tox
fungiccrouch: the implementation depends on cloning an additional repository for all jobs so that the correct (current, proposed, et cetera) version of the constraints file can be available for the corresponding branch, and we have one good tool for this right now (zuul-cloner) but it doesn't yet support all the pipelines in which we use tox-based jobs23:22
*** rfolco has quit IRC23:23
*** e0ne has joined #openstack-infra23:24
fungiccrouch: oh, also devstack is currently using constraints for everything it pip installs for devstack-based jobs23:26
*** sabeen has joined #openstack-infra23:27
clarkbexcept for setuptools and pip :)23:27
fungibut that was somewhat easier since it was already checking out the appropriate state of the corresponding repository containing the constraints file23:27
*** sridhar_ram has joined #openstack-infra23:30
*** sigmavirus24 is now known as sigmavirus24_awa23:30
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add job mutex support
jeblairthere's the zuulv3 version :)23:32
*** baoli has quit IRC23:33
*** e0ne has quit IRC23:33
openstackgerritTony Breeds proposed openstack-infra/yaml2ical: Clarify KeyError() in Schedule.__init__()
openstackgerritTony Breeds proposed openstack-infra/yaml2ical: Clarify ValueError() in Schedule.__init__()
jeblairtonyb: are you familiar with voluptuous?23:35
tonybjeblair: I an not23:36
*** baoli has joined #openstack-infra23:36
tonybjeblair: googling23:36
fungiwe use it for validating zuul's yaml configuration, for example23:36
jeblairtonyb: it's a framework for data validation we use in many of our projects to validate yaml23:36
jeblairtonyb: it also supports transformations (grafyaml uses it as well for this purpose heavily)23:37
*** hashar has quit IRC23:38
jeblairtonyb: so not only can you use it to validate frequency names, or dates, you can also use it to normalize capitalization, etc.23:38
*** angdraug has joined #openstack-infra23:40
tonybjeblair: Interesting.23:40
tonybjeblair, fungi: I'll have a play and see if it is a win.23:40
*** baoli has quit IRC23:41
*** dimtruck is now known as zz_dimtruck23:42
clarkbjeblair: greghaynes fwiw the gearman client disconnects seem consistent23:44
clarkbwe may also want to look into why that is happening beyond hndling it in the client23:44
jlvillalsdague: I almost think you are having too much fun with PS4 :)23:44
jeblairclarkb: absolutely -- this is not going to work well if it disconnects every day.  we'll never get images built.23:45
jeblairclarkb: what's consistent about them?23:45
clarkbjeblair: it happens every day23:45
greghaynesclarkb: I suspect it happens when we spam the upload and build jobs23:45
*** FallenPegasus has joined #openstack-infra23:45
clarkbgreghaynes: it seems to happen later23:45
jeblairgreghaynes: why do you suspect it happens then?23:45
greghaynesclarkb: which is when we would fill TCP buffers23:45
clarkb2016-02-03 15:22:29,511 ERROR gear.Server: Exception in poll loop: is about an hour after the spamming23:46
greghaynesjeblair: the time seemed to line up23:46
clarkbgreghaynes: spamming happenst at 141423:46
clarkbgreghaynes: it actually seems to happen once we start uploading an image23:46
greghaynesThe time is consistentish, right?23:46
clarkbno yesterday it happened several times through the day23:47
*** jamielennox|away is now known as jamielennox23:47
clarkbaround 1056-1058 and 232723:47
greghaynesSo, maybe we should tcpdump to a pcap file and filter on gearman port23:47
greghaynesThe post mortem look at the pcap23:47
fungiif it's unpredictable, i have a feeling that would be a huge capture23:48
clarkbwe can try that, it will be a fairly large capture with all the job listings23:48
clarkbfungi: yes23:48
fungias in large enough you might not have available space to store it (no idea really)23:49
jeblairclarkb: what's the log signature?23:49
fungioh, though we can rotate pcap files23:49
*** edmondsw has quit IRC23:49
greghaynesYes, it supports rotating23:49
fungitcpdump supports truncating and moving captures23:49
fungiright that23:49
jeblairthe function listing is about 1M23:49
fungiand has a retention setting i believe23:49
clarkb is what gearman server says23:49
greghaynesWe can also set snaplen to something small23:49
fungiso as long as we catch the error before we rotate out the capture which covered it, we might have something23:50
greghaynesSince really we only care about headers for the most part23:50
jeblairclarkb: that's not in the time frame you mentioned23:50
fungigreghaynes: sure, though default snaplen is already pretty small23:50
greghaynesAh, right23:50
openstackgerritIan Wienand proposed openstack-infra/system-config: Cleanup of nodepool builder logging
clarkbjeblair: yes it is23:50
fungiso maybe i'm only thinking of if we set a large -s23:50
jeblair23:47 < clarkb> around 1056-1058 and 232723:51
clarkbjeblair: yesterday23:51
*** pradk has quit IRC23:51
clarkbtoday is what I pasted (and had earlier pasted a timestamp for)23:51
jeblairclarkb: okay, i don't understand what you're trying to say about time, but i'll move on.  how do you know that's nodepool?23:51
clarkbjeblair: I don't yet, but when I looked at this a week and a half ago I was able to track that error to corresponding nodepool logs complaining about being disconected from gearman23:52
clarkbI am still working back on the nodepool side23:53
jeblair(i mean, it probably is, because it's doing a status request, and if it were zuul, we would have noticed)23:53
jeblair(though it could be a human)23:53
fungii think the gist of the time point was that he's seen it happening at various times with no particular pattern, and so "consistently" actually meant "frequently, and at least daily"23:54
*** bmjen has quit IRC23:54
*** yuanying has quit IRC23:54
clarkb is what the nodepool side says which may have silently reconnected without us noticing there?23:54
clarkbwe definitely only noticed it the last time in nodepool because submitting a job and trying to get the listings of queues were failing due to being disconnected23:55
greghaynesOr maybe the builder doesn't disconnect at all?23:55
*** Hunner has quit IRC23:55
*** crinkle has quit IRC23:55
*** bmjen has joined #openstack-infra23:56
jeblairclarkb: the status request will come from nodepoold23:56
*** yuanying has joined #openstack-infra23:56
jeblairso maybe check the other log file?23:56
jeblairgreghaynes: ^ right23:56
clarkboh right23:56
* clarkb checks other log23:56
*** Hunner has joined #openstack-infra23:56
*** Hunner has quit IRC23:56
*** Hunner has joined #openstack-infra23:56
*** crinkle has joined #openstack-infra23:56
* greghaynes is in Dr office so will randomly afk23:57
prometheanfiregreghaynes: mind +w this to get it to gate again?
greghaynesprometheanfire: heh, can't right now23:57
prometheanfiregate broken?23:57
Sam-I-Amgreghaynes: go doctor harder :)23:58
prometheanfireSam-I-Am: hi23:58
fungiprometheanfire: see just before your question where he says he's irc'ing from a doctor's office23:58
Sam-I-Amprometheanfire: moo.23:58
Sam-I-Amfungi: we all need to learn how to disconnect more :/23:58
openstackgerritJohn L. Villalovos proposed openstack-infra/glean: Stop confusing vim's tiny brain
Sam-I-Ammaybe he's seeking treatment for irc addiction23:59
greghaynesSam-I-Am: haha trying to Dr as hard as I can23:59
openstackgerritMerged openstack-infra/openstackid: Fix on username generation
prometheanfirefungi: was away for a while23:59
fungiprometheanfire: it was literally the line before your question ;)23:59

Generated by 2.14.0 by Marius Gedminas - find it at!