Thursday, 2018-02-22

*** olaph1 has quit IRC00:00
*** yamamoto has joined #openstack-infra00:08
andreafpabelanger : thanks - trying that00:10
ianwhmm, subscribing to the vcs repo for nodepool/zuul install isn't quite right https://git.openstack.org/cgit/openstack-infra/puppet-nodepool/tree/manifests/init.pp#n119 ... if install fails we don't detect that.  we really shoud use the pip provider i guess00:11
clarkbianw: you mean if the git update fails we don't detect that?00:12
ianwclarkb: no, if that intsall fails, when we re-run we don't try to reinstall because the git tree hasn't changed00:13
mnaseranyone against adding the link to infrastructure status in the topic?00:14
mnaseri feel like its a nice place for it there00:14
*** yamamoto has quit IRC00:14
clarkbianw: gotcha00:15
fungiianw: problem with the pip package provider is that it lacks support for setting a non-default upgrade strategy, so will unconditionally fight with distro-installed packages of any of your dep tree00:15
ianwfungi: yeah, i think it's sufficiently corner case to not worry ... but how do i reset vcsrepo's idea of what it's done?00:16
fungiwe really need it to have some means of specifying --upgrade-strategy=only-if-needed00:16
fungiianw: easiest brute-force method is to git reset --hard HEAD^1 and then let puppet do its thing all over again00:17
ianwlarge hammer, i like it :)00:18
fungiianw: good news is https://github.com/pypa/pip/pull/4500 seems to be targeted for pip 1000:18
*** lbragstad has quit IRC00:18
mnaserinfra-root: does anyone have a status update on the mirror rebuilding indices?  just to know if i can recheck or not yet :)00:19
pabelangerstill working on it00:19
mnaserpabelanger: not to rush things but any approximate ETA to know how long so i dont bug anyone :p00:19
pabelangermnaser: unknown, likely some time tomorrow00:20
pabelangermnaser: I'll ping once it is fixed00:20
fungipabelanger: is it still trying to resync the afsdb?00:20
mnaserpabelanger: ok cool, thank you so much and good luck :)00:20
pabelangerfungi: no, I've stopped it because there was an issue with AFS00:21
pabelangerabout to start it up again00:21
pabelangeralso00:21
pabelanger| opensuse-tumbleweed-0000000001 | opensuse-tumbleweed | nb02    | qcow2,vhd | ready | 00:00:05:27 |00:21
*** tosky has quit IRC00:21
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Reduce duplicate data in nodepool configuration  https://review.openstack.org/54681900:24
mnaseri always thought tumbleweed was a super cool name for a rolling release00:25
mnaserlol00:25
ianwok, nb03 has puppetted itself, nodepool is there, config looks "right" (is probably wrong, but is deployed correctly)00:26
ianwnodepool-builder doens't start, debugging now00:26
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Bring online opensuse-tumbleweed images  https://review.openstack.org/54682100:30
pabelangerianw: clarkb: mnaser: ^next patches to boot opensuse-tumbleweed image.00:31
clarkbpabelanger: https://review.openstack.org/#/c/546821/1/nodepool/nl02.openstack.org.yaml didn't we remove the images from citycloud?00:33
clarkbI would not add new images there at least00:33
*** dmellado has joined #openstack-infra00:33
pabelangerclarkb: from nodepool.yaml file, that stops them from getting uploaded00:33
pabelangerwhich is builders00:33
clarkbI see so we only did it to the builders, I guess that makes sense00:34
pabelangerclarkb: can you also get the parent too00:35
pabelangera clean up of data00:35
*** HeOS has quit IRC00:35
*** liusheng has quit IRC00:36
*** liusheng has joined #openstack-infra00:37
*** andreww has quit IRC00:41
*** xarses has joined #openstack-infra00:41
*** Miouge has quit IRC00:43
clarkbfungi pabelanger ianw we are about ~15 minutes away from what has become a daily gerrit slowdown. I've got a call I've got to be on but if you can maybe keep an eye out for unusual gerrit behavior and what might be causing this it would be helpful00:44
fungiclarkb: yup, not sure what to look for but i'll see if we can catch it00:45
*** d0ugal has quit IRC00:47
ianwrun an iostat maybe?00:47
pabelangerfungi: up for 2 reviews? https://review.openstack.org/546819/ and https://review.openstack.org/546821/ nodepool changes00:47
clarkbfungi: ianw pabelanger my hunch is its something like stackalytics running a bunch of expensive queries00:49
clarkbthe behavior I have been able to isolate using melody is that we haev a large spike in threads but not a spike in http requests00:49
fungiianw: good call. i have `iostat -t 10` going in a root screen session on review.o.o now00:50
fungiclarkb: i've got cacti and javamelody up too, though i assume those were of no help in previous instances00:51
clarkbnot really. The only thing melody helped with was showing a large spike in threads and that it didn't appear to be garbage collection related00:51
pabelangerI've restarted mirror-update.o.o to see if that help with AFS kernel modules00:54
pabelangerwe seem to be losing network access a fair bit00:54
*** cuongnv has joined #openstack-infra00:54
clarkbpabelanger: do you see similar packet loss outside of afs?00:54
clarkbmtr is a nice tool to monitor that00:54
fungi83 tasks according to show-queue but a lot of those look related to replication for the ancient git tags which got pushed by release automation a little while ago00:54
fungif88e687b              22:23:28.880      [18949c8c] push cgit@git02.openstack.org00:55
fungi:/var/lib/git/openstack/nova.git00:55
clarkbcompletely unrelated but I think that grenade may be broken00:56
clarkbhttp://logs.openstack.org/00/546200/2/gate/neutron-grenade/dbcbf60/logs/grenade.sh.txt.gz#_2018-02-21_23_43_04_98800:56
fungiso yeah, i think we have a replication backlog for nova for a while00:56
clarkbmy guess is beacuse we use stestr now00:56
clarkbianw: ^00:56
pabelangerclarkb: I'll check00:56
mnaserclarkb: https://review.openstack.org/#/q/Ic1fa3a98b6bcd151c489b078028687892655a19b00:56
mnaserpabelanger: ^00:56
clarkbmnaser: thanks00:56
mnaseri backported the master fix00:57
clarkbI don't appear to have approval rights on that maybe tonyb can help us00:57
*** d0ugal has joined #openstack-infra00:57
*** daidv has joined #openstack-infra00:58
ianwme either .. #openstack-qa?01:00
ianwoh, the call is out01:00
pabelangerokay, reboot seems to help, reprepro is now reporting db corruption. I'm going to copy good database from RO volume, into RW, then run reprepro again01:01
tonybque?01:01
pabelangershould then add missing debs into the database01:01
clarkbtonyb: https://review.openstack.org/#/q/Ic1fa3a98b6bcd151c489b078028687892655a19b stable reviews for that01:01
clarkbtonyb: to make grenade happy with stsetr01:01
clarkb(I can't spell)01:01
fungiwho needs spilling?01:02
tonybGah no dice sorry01:02
*** stevebaker has joined #openstack-infra01:03
fungigerrit queue has doubled to 14101:03
mnaseroh gerrit is fully dead01:04
mnaser"Reason: Error reading from remote server"01:04
tonybLooks like grenade is limited https://review.openstack.org/#/admin/groups/188,members + https://review.openstack.org/#/admin/groups/425,members01:04
tonybsorry01:04
ianwpabelanger: ok, i hope it's not necessary but https://docs.openstack.org/infra/system-config/reprepro.html#advanced-recovery-techniques we expanded after the last issues01:04
*** gyee has quit IRC01:05
ianwthat's worryingly small for a project that runs so much01:06
pabelangerianw: yah, I'll fallback to that01:06
openstackgerritIan Wienand proposed openstack-infra/system-config master: Add linaro cloud to nodepool clouds.yaml  https://review.openstack.org/54683401:09
*** yamamoto has joined #openstack-infra01:10
funginot seeing any significant spikes in cacti graphs for review.o.o01:13
clarkbheisenbug01:14
funginot heise-enough because mnaser reported it going out to lunch at 01:0401:14
*** claudiub has quit IRC01:15
*** olaph1 has joined #openstack-infra01:15
*** olaph has quit IRC01:16
*** yamamoto has quit IRC01:16
fungiso cacti does show a relatively anomalous write spike to xvdb atarting up right at 01:0001:18
fungisame for 24 hours ago01:18
clarkbis it bup maybe?01:19
fungixvdb is the pv for the main vg01:19
clarkbwhich hosts the git repos iirc01:19
clarkbsomeone doing clones maybe?01:19
fungiso something writing to /home/gerrit201:19
fungiit's write activity, not read01:20
fungiaccording to cacti01:20
clarkbhrm01:20
clarkbpushes then? or logging?01:20
fungiinstead of ianw's iostat suggestion (which just told us what block devices were seeing activity) i should have had iotop running instead01:25
fungiwhich i do now, but it's probably too late to be useful01:26
fungibut perhaps tomorrow01:26
fungicurrently the bulk of the write activity is the gerrit jvm, it looks like01:27
*** xarses has quit IRC01:27
clarkboh!01:27
clarkbcould it be the garbage collecting and packing?01:28
fungigerrit does that now, right? we don't cron it any longer01:28
clarkbya its the jvm doing it now iirc01:28
fungithough there is also this:01:28
fungino, wait, i was reading cron fields backwards01:29
fungiso yeah, very well may be what you're saying01:29
fungiwe don't seem to have any cronjobs around that timeframe at any rate01:29
pabelanger generating universe/Contents-amd64...01:30
pabelangerbetter01:30
*** kiennt26 has joined #openstack-infra01:31
pabelangerhopefully not much longer to finish01:31
pabelangerthen I'll manually vos release01:31
fungiclarkb: threadcount didn't really spike my more than maybe 20% at 01:0001:33
fungi(if even that much)01:33
*** dhill__ has quit IRC01:34
*** slaweq has quit IRC01:34
clarkbfungi: ok it was more dramatic in the past01:34
clarkbseveral hundred percent according to melody01:34
*** slaweq has joined #openstack-infra01:35
fungii must be looking at a different graph01:35
*** esberglu has quit IRC01:35
fungithe "threads count" graph says it jumped from around 425 threads to 500 at 01:0001:35
clarkbfungi: https://review.openstack.org/monitoring?part=graph&graph=activeThreads01:36
clarkbsorry active threads01:36
fungioh, active threads01:36
clarkband it spiked here too01:36
fungiyes, though the bigger spike there was apparently closer to 01:2001:36
fungispiking up to around 55-ish01:37
*** slaweq has quit IRC01:39
*** Kevin_Zheng has joined #openstack-infra01:40
*** salv-orlando has joined #openstack-infra01:41
openstackgerritIan Wienand proposed openstack-infra/system-config master: Add linaro cloud to nodepool clouds.yaml  https://review.openstack.org/54683401:42
fungiio ops to /home/gerrit2 are still pretty heavy and iotop says it's mostly the jvm01:45
fungiso daily git repacking is seeming a likely explanation01:45
*** salv-orlando has quit IRC01:46
*** jlabarre has quit IRC01:47
*** namnh has joined #openstack-infra01:50
*** ykarel|afk has joined #openstack-infra01:52
*** lbragstad has joined #openstack-infra01:56
*** owalsh has quit IRC01:56
*** owalsh has joined #openstack-infra01:56
ianw| ubuntu-xenial-arm64-0000000003 | ubuntu-xenial-arm64 | nb03    | qcow2     | building | 00:00:04:22 |01:58
*** kien-ha has joined #openstack-infra01:58
persiaThat looks promising :)01:58
pabelangerneat02:01
*** owalsh has quit IRC02:01
ianwit's a start.  it's going to take a while to cache everything in02:02
*** annp has joined #openstack-infra02:02
*** eharney has quit IRC02:05
*** owalsh has joined #openstack-infra02:06
*** olaph1 is now known as olaph02:12
*** yamamoto has joined #openstack-infra02:12
*** hongbin has joined #openstack-infra02:13
pabelangerokay, DB recreated, just checking pools now02:15
pabelanger(reprepro)02:16
*** dhill__ has joined #openstack-infra02:16
*** dklyle has joined #openstack-infra02:17
*** yamamoto has quit IRC02:18
openstackgerritMerged openstack-infra/project-config master: Reduce duplicate data in nodepool configuration  https://review.openstack.org/54681902:19
*** david-lyle has quit IRC02:21
*** inc0 has quit IRC02:21
*** inc0 has joined #openstack-infra02:21
pabelangerreprepro successful02:22
pabelangerI'm going to vos release now02:22
fungiwoo! thanks for sticking with it02:22
pabelangerthe reboot did help, no AFS issues that time02:22
openstackgerritMerged openstack-infra/project-config master: Bring online opensuse-tumbleweed images  https://review.openstack.org/54682102:23
*** olaph1 has joined #openstack-infra02:24
*** olaph has quit IRC02:25
*** zhongjun has joined #openstack-infra02:27
*** dave-mccowan has joined #openstack-infra02:28
pabelangerReleased volume mirror.ubuntu successfully02:29
pabelangeryay02:29
pabelanger#status log mirror.ubuntu reprepro has been repaired and back online02:30
openstackstatuspabelanger: finished logging02:30
pabelangermnaser: ^02:30
*** ykarel|afk has quit IRC02:31
*** mriedem has quit IRC02:32
*** zhenguo has joined #openstack-infra02:32
*** yamahata has quit IRC02:34
*** kien-ha has quit IRC02:34
melwittdoes anyone know if reviewday is still supported at http://status.openstack.org/reviews ? I notice it says "Page refreshed at 2018-02-02"02:37
*** dhill__ has quit IRC02:40
clarkbmelwitt: it should be, its likely just broken for one reason or another02:41
melwittcool, thanks02:42
pabelanger| 0002664319 | inap-mtl01 | opensuse-tumbleweed | d7390c45-35c9-4c30-bb1e-4dc634f2a3d0 | 198.72.124.114  |                                        | ready    | 00:00:00:11 | unlocked |02:47
pabelangerclarkb: dirk: ^02:47
mnaserpabelanger: awesome thank you02:51
mnaser#thanks pabelanger and infra for getting ubuntu mirrors repaired and backup quickly!02:52
openstackstatusmnaser: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks)02:52
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add opensuse-tumbleweed testing to ozj  https://review.openstack.org/54684402:55
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add opensuse-tumbleweed testing to ozj  https://review.openstack.org/54684402:57
*** dave-mccowan has quit IRC02:58
*** wolverineav has quit IRC03:00
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: DNM  https://review.openstack.org/54684603:00
*** andymccr has quit IRC03:00
*** andymccr has joined #openstack-infra03:03
*** rlandy|rover|bbl is now known as rlandy|rover03:03
*** lbragstad has quit IRC03:04
*** andreas_s has joined #openstack-infra03:06
*** andreas_s has quit IRC03:10
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add fedora-27 testing to ozj  https://review.openstack.org/54684703:12
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: DNM  https://review.openstack.org/54684803:13
*** yamamoto has joined #openstack-infra03:14
openstackgerritMatt Riedemann proposed openstack-infra/openstack-zuul-jobs master: Don't run neutron-grenade job on stable/ocata changes  https://review.openstack.org/54685003:18
*** ramishra has joined #openstack-infra03:19
*** yamamoto has quit IRC03:20
*** mriedem has joined #openstack-infra03:23
mriedemzuulv3 question: the neutron-grenade-multinode job is defined in neutron, but not in the stable/ocata branch for neutron. nova has the neutron-grenade-multinode job in it's check queue via project-config,03:25
mriedemwe need to not run the grenade job on ocata changes since newton is eol,03:25
mriedembut i'm not sure if i need to add a branch restriction to project-config, or neutron's job def (on master?), or other03:25
mriedemneutron-grenade is defined in openstack-zuul-jobs and is part of the integrated-gate template, and is defined to not run on stable/newton (and i have a patch up to make it stop running it against ocata changes too)03:26
mriedemso a bit confused about where the branch restriction should live03:26
mriedemactually i guess it's also defined in neutron stable/ocata https://github.com/openstack/neutron/blob/stable/ocata/.zuul.yaml#L137 so that probably needs to be deleted,03:28
mriedemand nova's re-def of it in project-config also needs to go03:28
mriedemtonyb: ^ you might care since ocata is blocked atm03:28
tonybmriedem: Thanks.   I'd like to know how to avoid making this same mistake (if?)when ocata goes EOL03:30
mriedemwe've always had to just update branch restrictions for running grenade after a branch goes eol,03:31
tonybmriedem: but I have no idea.  I kinda thought it should be modified in openstack-zuul-jobs on the octata branch (or as you say that shoudl be deleted) but I'm just confused03:31
mriedembut now it's spread out more since the jobs can be defined in one place, and then re-defined (sort of) still in project-config per project03:31
mriedemi'm gonna remove the job def from neutron in ocata03:31
tonybmriedem: Yeah, this time I'd like to do it right after the branches get deleted but it a matter of knowing what "it" is03:32
mriedemit is it03:32
mriedemwhat is it03:32
tonybback and forth I sway with the wind ?03:33
*** wolverineav has joined #openstack-infra03:33
tonybGah wrong song :(03:33
* tonyb queues up album as a corrective measure03:34
mriedemhttps://review.openstack.org/54685503:34
mriedemah crap03:34
mriedemthat's master03:34
*** yamamoto has joined #openstack-infra03:37
*** wolverineav has quit IRC03:37
mriedemhttps://review.openstack.org/#/c/546857/03:37
clarkbthe best place to put the exclusion is on thr main job def if it will be gloal03:37
clarkband I think these grenade exclusions are effectively global right?03:38
*** dave-mccowan has joined #openstack-infra03:38
clarkbthat said I think the best thin here is to evwntually get to having the job list on each branch in each projecf maybe?03:38
clarkbthen you just update the jobs a project wants to run and thats an easy direct exvlusion03:39
mriedembut then you have to remove this job from every project that defines it once a branch goes eol, which sucks03:39
*** wolverineav has joined #openstack-infra03:40
mriedemi'm not really sure what happens if we remove the job def from neutron's stable/ocata branch, will nova still attempt to run it against ocata changes and if so, which job def would it be using?03:40
mriedemhttps://github.com/openstack-infra/project-config/blob/master/zuul.d/projects.yaml#L1033903:41
mriedemthat's where nova says it wants to run that job right now03:41
clarkbthats a good question I think it would be an error without removing the job from other branches that use it. Grenade may be a special case  because we drop it a cycle early03:41
*** ramishra has quit IRC03:41
clarkbso adding exclusion to main job def is probably best for grenade03:42
mriedemdoes that go in the main job def in the neutron stable/ocata branch though?03:42
*** xarses has joined #openstack-infra03:43
mriedemb/c that seems bonkers - the job def in stable/ocata would have to say, don't run this in stable/ocata03:43
*** salv-orlando has joined #openstack-infra03:43
*** ramishra has joined #openstack-infra03:43
clarkbya thats a good point this may be a corner case we have to figure out03:43
*** andreww has joined #openstack-infra03:43
mriedemok so for now, i'm going to push a project-config patch to say don't run that job on stable/ocata chnages03:43
*** andreww has quit IRC03:44
*** wolverineav has quit IRC03:44
*** andreww has joined #openstack-infra03:44
mriedemthere is still a newton branch for devstack, not sure why03:45
mriedemor if that works03:45
mriedemtonyb: ?03:45
*** xarses has quit IRC03:47
*** cuongnv has quit IRC03:47
*** salv-orlando has quit IRC03:49
*** rlandy|rover has quit IRC03:50
openstackgerritMatt Riedemann proposed openstack-infra/project-config master: Don't run neutron-grenade-multinode on newton or ocata changes  https://review.openstack.org/54685803:51
mriedemok ^ says don't run that job on projects that have stable branches (excluded tempest since it's branchless)03:51
*** wolverineav has joined #openstack-infra03:51
tonybmriedem: because $some projects are still using it.03:54
tonybmriedem: I'll look into which ones but I suspect we can drop it and clean up the mess later03:55
mriedemand ocata is marked for eol next week :)03:56
mriedemyeehaw!03:56
mriedemalright i'll check things out in the morning, gotta drop03:56
*** gema has quit IRC03:57
*** mriedem has quit IRC03:57
tonybmriedem: Yeah but we have a session at the PTG to work out if that's the reality03:57
*** wolverineav has quit IRC03:57
tonybI suspect it'll get another 3-6 months but that depends who shows up03:58
*** rlandy has joined #openstack-infra03:59
*** rlandy is now known as rlandy|rover03:59
*** slaweq has joined #openstack-infra03:59
*** d0ugal has quit IRC04:00
*** rlandy|rover has quit IRC04:01
*** slaweq has quit IRC04:04
*** udesale has joined #openstack-infra04:04
*** ykarel|afk has joined #openstack-infra04:04
*** udesale_ has joined #openstack-infra04:07
*** dave-mccowan has quit IRC04:09
*** olaph has joined #openstack-infra04:10
*** olaph1 has quit IRC04:13
*** wolverineav has joined #openstack-infra04:17
*** wolverineav has quit IRC04:22
*** sree_ has joined #openstack-infra04:25
*** sree_ is now known as Guest9694704:26
*** VW has joined #openstack-infra04:26
*** d0ugal has joined #openstack-infra04:27
*** links has joined #openstack-infra04:30
*** udesale_ has quit IRC04:31
*** udesale_ has joined #openstack-infra04:31
*** udesale has quit IRC04:32
*** olaph1 has joined #openstack-infra04:33
*** psachin has joined #openstack-infra04:34
*** olaph has quit IRC04:34
*** olaph has joined #openstack-infra04:37
*** olaph1 has quit IRC04:38
*** olaph1 has joined #openstack-infra04:41
*** zhenguo has quit IRC04:42
*** olaph has quit IRC04:42
*** salv-orlando has joined #openstack-infra04:45
*** salv-orlando has quit IRC04:49
*** pgadiya has joined #openstack-infra04:53
*** ykarel|afk is now known as ykarel04:54
*** VW has quit IRC04:55
*** VW has joined #openstack-infra04:55
*** dhajare_ has joined #openstack-infra04:57
*** VW has quit IRC05:00
*** hongbin has quit IRC05:02
*** rosmaita has quit IRC05:06
*** andreww has quit IRC05:18
*** wolverineav has joined #openstack-infra05:21
*** andreww has joined #openstack-infra05:22
*** xarses has joined #openstack-infra05:24
*** slaweq has joined #openstack-infra05:25
*** ihrachys_ has joined #openstack-infra05:25
*** ihrachys has quit IRC05:26
ianwpabelanger: 485748 ... i just ... dunno.  which is why i guess this sat in review, because it's not *obviously* correct and so nobody wants to break odd things ...  hopefully based on that there's a convincing argument for why this and not before=basic.target?05:26
*** andreww has quit IRC05:27
*** wolverineav has quit IRC05:27
*** janki has joined #openstack-infra05:27
*** slaweq has quit IRC05:30
*** oidgar has joined #openstack-infra05:32
*** eernst has quit IRC05:32
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Add /label-list to the webapp  https://review.openstack.org/53556305:35
openstackgerritJens Harbott (frickler) proposed openstack-infra/devstack-gate master: Add neutron-tempest-plugin-api job  https://review.openstack.org/52900005:37
*** bhujay has joined #openstack-infra05:45
*** salv-orlando has joined #openstack-infra05:45
*** claudiub has joined #openstack-infra05:47
*** aeng has quit IRC05:49
*** salv-orlando has quit IRC05:50
*** oidgar has quit IRC05:51
*** olaph has joined #openstack-infra05:51
*** olaph1 has quit IRC05:53
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: bindep: use shell instead of command with executable  https://review.openstack.org/54686905:56
*** bhujay has quit IRC06:00
*** jogo has quit IRC06:03
*** bhujay has joined #openstack-infra06:04
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/54687006:05
*** cuongnv has joined #openstack-infra06:10
*** pfalleno1 has quit IRC06:10
*** tdasilva has quit IRC06:10
*** kjackal has joined #openstack-infra06:12
*** lpetrut has joined #openstack-infra06:18
*** salv-orlando has joined #openstack-infra06:19
*** tdasilva has joined #openstack-infra06:25
*** slaweq has joined #openstack-infra06:27
*** gema has joined #openstack-infra06:28
*** gema has joined #openstack-infra06:28
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: bindep: use shell instead of command with executable  https://review.openstack.org/54686906:30
*** dbecker has quit IRC06:31
dirkpabelanger: nice! I'm creating some test jobs today06:31
*** slaweq has quit IRC06:31
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: bindep: use shell instead of command with executable  https://review.openstack.org/54686906:41
*** alexchadin has joined #openstack-infra06:41
*** dbecker has joined #openstack-infra06:44
*** salv-orlando has quit IRC06:51
*** salv-orlando has joined #openstack-infra06:51
*** jtomasek has joined #openstack-infra06:56
*** salv-orlando has quit IRC06:56
openstackgerritTristan Cacqueray proposed openstack-infra/zuul-jobs master: ara: check for return code instead of succeed  https://review.openstack.org/54689206:57
*** bhujay has quit IRC06:57
*** jogo has joined #openstack-infra07:00
*** ianychoi has quit IRC07:01
*** ianychoi has joined #openstack-infra07:01
*** bhujay has joined #openstack-infra07:07
*** agopi has quit IRC07:11
*** lpetrut has quit IRC07:11
*** bhujay has quit IRC07:13
*** Swami has quit IRC07:13
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: Add abstract attribute to base jobs  https://review.openstack.org/54560307:16
*** hashar has joined #openstack-infra07:16
AJaegerdirk: see also   https://review.openstack.org/546846 , both tumbleweed tests fail07:17
*** rcernin has quit IRC07:20
openstackgerritMerged openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/54687007:24
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: Replace sphinx_check_warning_is_error.py with ini lookup  https://review.openstack.org/52881207:25
*** snapiri1 has joined #openstack-infra07:28
*** dhajare_ has quit IRC07:28
openstackgerritMerged openstack-infra/storyboard master: Create a StoryBoard gui manual  https://review.openstack.org/32547407:32
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: Replace sphinx_check_warning_is_error.py with ini lookup  https://review.openstack.org/52881207:33
*** andreas_s has joined #openstack-infra07:33
ameedaMorning :) , should I reverse os-xenapi version from 0.3.1 to 0.1.0 as this bug mentioned " https://bugs.launchpad.net/nova/+bug/1718606 "07:34
openstackLaunchpad bug 1718606 in OpenStack Compute (nova) "doc: Outdated XenServer document" [Low,Confirmed]07:34
*** ykarel is now known as ykarel|lunch07:34
*** snapiri1 is now known as snapiri-07:35
*** pcaruana has joined #openstack-infra07:37
*** lpetrut has joined #openstack-infra07:37
*** HeOS has joined #openstack-infra07:38
*** slaweq has joined #openstack-infra07:45
*** slaweq_ has joined #openstack-infra07:47
AJaegermordred: I fixed your change https://review.openstack.org/528812 - could you double check it, please?07:48
*** armaan has quit IRC07:48
*** slaweq has quit IRC07:48
*** armaan has joined #openstack-infra07:48
AJaegerameeda: there's an #openstack-nova channel for nova discussion and questions.07:49
*** slaweq has joined #openstack-infra07:49
ameedaAJaeger: Thanks for reply :)07:50
*** salv-orlando has joined #openstack-infra07:52
*** slaweq_ has quit IRC07:52
*** Miouge has joined #openstack-infra07:54
*** kjackal has quit IRC07:56
*** salv-orlando has quit IRC07:56
*** slaweq_ has joined #openstack-infra08:00
*** slaweq_ has quit IRC08:04
*** lpetrut has quit IRC08:09
*** shu-mutou-AWAY is now known as shu-mutou08:12
*** threestrands_ has joined #openstack-infra08:15
*** liusheng has quit IRC08:17
*** liusheng has joined #openstack-infra08:18
*** tesseract has joined #openstack-infra08:18
*** threestrands has quit IRC08:18
*** dhajare_ has joined #openstack-infra08:27
*** ralonsoh has joined #openstack-infra08:27
*** ykarel|lunch is now known as ykarel08:28
*** slaweq_ has joined #openstack-infra08:28
*** rossella_s has joined #openstack-infra08:31
*** gfidente has joined #openstack-infra08:36
*** kjackal has joined #openstack-infra08:38
andreaftobiash corvus mordred ianw AJaeger some more improvements on stage-output https://review.openstack.org/#/c/544606/08:38
*** electrofelix has joined #openstack-infra08:40
*** florianf has joined #openstack-infra08:41
*** pblaho has joined #openstack-infra08:44
*** liusheng has quit IRC08:44
*** liusheng has joined #openstack-infra08:44
*** slaweq_ has quit IRC08:45
*** jpena|off is now known as jpena08:50
*** kjackal has quit IRC08:51
*** kjackal has joined #openstack-infra08:52
*** xarses has quit IRC08:52
*** salv-orlando has joined #openstack-infra08:52
*** salv-orlando has quit IRC08:57
*** salv-orlando has joined #openstack-infra08:57
*** vivsoni has quit IRC08:59
*** vivsoni has joined #openstack-infra08:59
*** jpich has joined #openstack-infra09:02
*** alexchadin has quit IRC09:03
*** oidgar has joined #openstack-infra09:04
*** hjensas has joined #openstack-infra09:04
slaweqhi all09:12
slaweqon zuul.openstack.org I see error "status.json: Not Found" and no jobs are displayed09:13
slaweqis it some already known issue or should I search on my side?09:13
*** amoralej|off is now known as amoralej09:13
*** annp has quit IRC09:14
*** daidv has quit IRC09:14
*** cuongnv has quit IRC09:14
AJaegerslaweq: we updated javascript, do a hard reload to invalidate the cached javascript09:14
*** daidv has joined #openstack-infra09:14
*** cuongnv has joined #openstack-infra09:14
*** annp has joined #openstack-infra09:14
slaweqAJaeger: thx, that helped :)09:15
*** dtantsur|afk is now known as dtantsur09:17
*** hamzy_ has quit IRC09:20
*** hamzy_ has joined #openstack-infra09:20
*** pfallenop has joined #openstack-infra09:23
*** mgoddard_ has joined #openstack-infra09:31
*** kjackal has quit IRC09:32
*** kjackal has joined #openstack-infra09:32
*** kiennt26 has quit IRC09:36
*** threestrands_ has quit IRC09:37
*** e0ne has joined #openstack-infra09:38
*** jaosorior has joined #openstack-infra09:39
*** pfallenop has quit IRC09:39
*** pfallenop has joined #openstack-infra09:44
*** shu-mutou is now known as shu-mutou-AWAY09:54
*** danpawlik has joined #openstack-infra09:55
*** jpich_ has joined #openstack-infra10:03
openstackgerritHoang Trung Hieu proposed openstack-infra/system-config master: Fix dead link  https://review.openstack.org/54693010:04
openstackgerritJean-Philippe Evrard proposed openstack-infra/project-config master: Add OSA os_panko repo base jobs  https://review.openstack.org/54657310:04
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Added new endpoint to seed default event types per summit  https://review.openstack.org/54693110:04
openstackgerritJean-Philippe Evrard proposed openstack-infra/project-config master: Add OSA nspawn host/container repo base jobs  https://review.openstack.org/54629410:04
AJaegerevrardjp: note we still have not recovered from your broken imports, hope corvus can help later today. We might need a Zuul restart.10:05
evrardjpResubmitting -- I guess I could :)10:05
evrardjpOh darn I thought it was done during the night, silently!10:05
openstackgerritMerged openstack-infra/openstackid-resources master: Added new endpoint to seed default event types per summit  https://review.openstack.org/54693110:05
evrardjpduring my night at least :)10:05
evrardjpI am sorry for that then10:06
AJaegerevrardjp: I rechecked one chagne this morning and it was still failing.10:06
evrardjpoh I haven't seen the rechecks, that's why I rebased.10:06
*** yamamoto has quit IRC10:06
evrardjpProbably need moar coffee then.10:06
AJaegerand I didn't see any mention in IRC that it's fixed either10:06
AJaegerevrardjp: yeah, drink some coffee - and ask the US based team later today again, please.10:07
*** jpich has quit IRC10:07
evrardjpyup will do.10:07
*** cuongnv has quit IRC10:08
evrardjpsorry, it wasn't my intent to offend .10:08
*** links has quit IRC10:09
*** links has joined #openstack-infra10:10
AJaegerevrardjp: no offense taken or understood - I just wanted to explain since it looked like you weren't away.10:10
evrardjpthanks10:11
AJaegers/away/aware/10:11
* AJaeger needs better typing skills ;(10:11
*** jpich_ is now known as jpich10:15
*** olaph1 has joined #openstack-infra10:16
*** lpetrut has joined #openstack-infra10:17
*** olaph has quit IRC10:17
*** rossella_s has quit IRC10:20
*** tosky has joined #openstack-infra10:22
*** annp has quit IRC10:23
*** rossella_s has joined #openstack-infra10:24
*** kjackal has quit IRC10:26
*** kjackal has joined #openstack-infra10:27
*** vipul has quit IRC10:28
*** spiffxp has quit IRC10:29
*** pcrews has quit IRC10:30
danpawlikcoreycb, zigo, beisner: Hi, I have small question to you. Is possible to create Octavia packages for Ubuntu? I create a storyboard for it https://storyboard.openstack.org/#!/story/200156610:30
zigodanpawlik: Sure it's possible, do you have the needed skills?10:31
zigodanpawlik: I can sponsor the upload in Debian if you want, then it will migrate to Ubuntu.10:31
danpawlikI guess I don't have such skills :(10:32
*** yamahata has joined #openstack-infra10:32
*** kjackal has quit IRC10:33
*** zoli is now known as zoli|lunch10:33
*** zoli|lunch is now known as zoli10:33
*** alexchadin has joined #openstack-infra10:34
danpawlikzigo: maybe I will try do it tomorrow10:34
zigodanpawlik: My advice would be to just look how other packages are made.10:35
zigodanpawlik: For Queens, everything is uploaded in Experimental until the final releases, so just use Sid + Experimental.10:35
zigodanpawlik: FYI, at the moment, I'm almost finished with Queens, and I'm uploading services in RC1 versions.10:36
*** pcrews has joined #openstack-infra10:36
zigo(packages are already prepared, doing the final uploads ...)10:36
*** spiffxp has joined #openstack-infra10:36
danpawlikzigo: nice :)10:37
danpawlikzigo: so I try to do it tomorrow base on some other queens package10:37
danpawlikzigo: could you tell me what is the storyboard for cloud archive?10:37
*** vipul has joined #openstack-infra10:38
zigodanpawlik: I don't do Ubuntu, I do Debian.10:39
zigodanpawlik: So I use the Debian BTS, and salsa.debian.org as a Git repository.10:39
danpawlikah, ok10:39
zigodanpawlik: I can add you in the team if you create an account there.10:40
zigodanpawlik: https://signup.salsa.debian.org/10:40
zigoGive me your account, and I'll add you there.10:40
zigoWe're using a Git tag packaging workflow, I'm not sure if you know how this work ...10:40
zigodanpawlik: How many source package will you need?10:41
danpawlikzigo: user @daniel.pawlik-guest10:42
danpawlikzigo: I know the workflow on RDO10:42
danpawlikbut here not10:43
danpawlikzigo: what do you mean "How many source package will you need?"10:43
danpawlikzigo: I try to create debianize for Octavia10:43
danpawlikOpenstack Octavia service10:43
zigodanpawlik: So, just one?10:43
*** panda|off is now known as panda|sick10:44
*** kjackal has joined #openstack-infra10:44
zigoThere's already pytohn-octaviaclient that I did.10:44
danpawlikzigo: but its just a client :P10:44
danpawlikjust one10:44
zigoRight.10:44
zigoI'm creating the repo for you.10:44
danpawlik:D10:45
danpawlikzigo: thanks10:45
*** dhajare_ is now known as dhajare10:46
zigodanpawlik: You can also file the ITP bug. Do you know how to do that?10:48
danpawlikzigo: maybe come on priv10:48
*** yee37935 has joined #openstack-infra10:48
*** yee379 has quit IRC10:49
*** panda has joined #openstack-infra10:57
*** purp_too has joined #openstack-infra10:57
*** scarpino has joined #openstack-infra10:58
*** ericyoung has quit IRC10:59
*** witek has quit IRC10:59
*** panda|sick has quit IRC10:59
*** Zara has quit IRC10:59
*** ilpianista has quit IRC10:59
*** EmilienM has quit IRC10:59
*** purp has quit IRC10:59
*** links has quit IRC10:59
*** mattoliverau has quit IRC10:59
*** Zara has joined #openstack-infra10:59
*** scarpino is now known as ilpianista10:59
*** olaph has joined #openstack-infra10:59
*** witek has joined #openstack-infra10:59
*** ericyoung has joined #openstack-infra10:59
*** olaph1 has quit IRC11:00
*** mattoliverau has joined #openstack-infra11:00
*** links has joined #openstack-infra11:00
*** EmilienM has joined #openstack-infra11:00
*** salv-orl_ has joined #openstack-infra11:00
*** yamahata has quit IRC11:00
*** Qiming has quit IRC11:01
*** kambiz has quit IRC11:01
*** purp_too has quit IRC11:01
*** salv-orlando has quit IRC11:03
*** namnh has quit IRC11:03
*** kambiz has joined #openstack-infra11:04
*** purp has joined #openstack-infra11:04
*** Qiming has joined #openstack-infra11:05
*** yamamoto has joined #openstack-infra11:06
*** pcaruana has quit IRC11:07
*** yamamoto has quit IRC11:13
openstackgerritSimon Westphahl proposed openstack-infra/zuul master: Allow using remote refs to find commits for change  https://review.openstack.org/54496411:16
*** panda is now known as panda|sick11:17
openstackgerritSimon Westphahl proposed openstack-infra/zuul master: Allow using remote refs to find commits for change  https://review.openstack.org/54496411:18
*** tbarron_ has joined #openstack-infra11:23
*** tbarron has quit IRC11:24
*** Zara_ has joined #openstack-infra11:25
*** kaisers1 has joined #openstack-infra11:25
*** olaph1 has joined #openstack-infra11:26
*** Qiming_ has joined #openstack-infra11:26
*** mattoliverau_ has joined #openstack-infra11:27
odyssey4meMorning all - apologies for the drama yesterday with the various new repo imports. I've learned some things which will hopefully make it a smoother process next time.11:31
odyssey4meAJaeger evrardjp I see that we're still not quite done as the base jobs aren't yet importing. I guess we're waiting for the US crew to wake to determine next steps?11:32
*** efried is now known as efried_rollin11:32
*** Faster-Fanboi_ has joined #openstack-infra11:32
*** olaph has quit IRC11:34
*** Zara has quit IRC11:34
*** kaisers has quit IRC11:34
*** mattoliverau has quit IRC11:34
*** rbergeron has quit IRC11:34
*** Qiming has quit IRC11:34
*** bradm has quit IRC11:34
*** Faster-Fanboi has quit IRC11:34
*** rwsu has quit IRC11:34
*** ldnunes has joined #openstack-infra11:34
*** rwsu has joined #openstack-infra11:34
*** rbergeron has joined #openstack-infra11:35
*** luzC has quit IRC11:36
*** aviau has quit IRC11:39
*** aviau has joined #openstack-infra11:39
*** luzC has joined #openstack-infra11:39
*** armaan has quit IRC11:39
*** armaan has joined #openstack-infra11:40
*** rbergero1 has joined #openstack-infra11:41
*** pbourke has quit IRC11:42
*** shardy has quit IRC11:42
*** Zara_ is now known as Zara11:44
*** bradm has joined #openstack-infra11:44
*** jaosorior has quit IRC11:45
*** jaosorior has joined #openstack-infra11:45
*** danpawlik has quit IRC11:45
*** rbergeron has quit IRC11:46
*** armaan has quit IRC11:46
*** pbourke has joined #openstack-infra11:47
*** snapiri- has quit IRC11:47
*** snapiri1 has joined #openstack-infra11:48
*** danpawlik has joined #openstack-infra11:49
*** udesale_ has quit IRC11:50
*** pbourke_ has joined #openstack-infra11:52
*** armaan has joined #openstack-infra11:57
*** jpena is now known as jpena|lunch11:57
*** pbourke has quit IRC11:58
*** snapiri1 has quit IRC11:58
*** danpawlik has quit IRC12:00
*** thorre_se has joined #openstack-infra12:03
*** lihi has joined #openstack-infra12:04
*** danpawlik has joined #openstack-infra12:05
*** thorre has quit IRC12:06
*** thorre_se is now known as thorre12:06
*** yamamoto has joined #openstack-infra12:09
*** pcaruana has joined #openstack-infra12:09
*** thorre_se has joined #openstack-infra12:09
*** thorre has quit IRC12:13
*** thorre_se is now known as thorre12:13
*** yamamoto has quit IRC12:14
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Added get tracks by summit endpoints  https://review.openstack.org/54696212:15
openstackgerritMerged openstack-infra/openstackid-resources master: Added get tracks by summit endpoints  https://review.openstack.org/54696212:16
openstackgerritHoang Trung Hieu proposed openstack-infra/zuul-jobs master: Update and replace http with https for doc links  https://review.openstack.org/54696512:26
*** dave-mccowan has joined #openstack-infra12:27
*** janki has quit IRC12:27
*** yamamoto has joined #openstack-infra12:27
*** links has quit IRC12:29
evrardjpodyssey4me: yes12:30
*** pcichy has quit IRC12:32
openstackgerritHoang Trung Hieu proposed openstack-infra/zuul master: Update and replace http with https for doc links  https://review.openstack.org/54697012:37
*** links has joined #openstack-infra12:41
*** pcichy has joined #openstack-infra12:48
*** zul has quit IRC12:51
*** olaph1 is now known as olaph12:51
*** tbarron_ is now known as tbarron12:52
*** wxy has quit IRC12:52
*** rosmaita has joined #openstack-infra12:52
*** wxy has joined #openstack-infra12:52
*** hrybacki has quit IRC12:53
*** hrybacki has joined #openstack-infra12:54
*** zul has joined #openstack-infra12:55
*** jlabarre has joined #openstack-infra13:01
*** CrayZee has joined #openstack-infra13:03
*** salv-orl_ has quit IRC13:10
*** salv-orlando has joined #openstack-infra13:11
*** jcoufal has joined #openstack-infra13:12
*** amoralej is now known as amoralej|lunch13:13
*** electrofelix has quit IRC13:14
*** betherly has quit IRC13:15
*** salv-orlando has quit IRC13:15
*** salv-orlando has joined #openstack-infra13:15
*** betherly has joined #openstack-infra13:15
*** ying_zuo has quit IRC13:16
*** dprince has joined #openstack-infra13:16
coreycbdanpawlik: happy to help you to get octavia into ubuntu as well. it would need to get into ubuntu bionic by march 1st for queens.13:16
*** hamzy_ is now known as hamzy13:16
*** ying_zuo has joined #openstack-infra13:17
*** olaph1 has joined #openstack-infra13:17
*** berendt has quit IRC13:18
*** olaph has quit IRC13:19
*** berendt has joined #openstack-infra13:21
openstackgerritOmer Anson proposed openstack-infra/project-config master: Dragonflow: Add requirement for neutron-dynamic-routing in tests  https://review.openstack.org/54697813:21
openstackgerritSimon Westphahl proposed openstack-infra/zuul master: Allow using remote refs to find commits for change  https://review.openstack.org/54496413:22
*** jpena|lunch is now known as jpena13:27
*** zhipeng has joined #openstack-infra13:28
*** vivsoni has quit IRC13:28
*** vivsoni has joined #openstack-infra13:28
*** ralonsoh_ has joined #openstack-infra13:32
*** udesale has joined #openstack-infra13:32
*** yamamoto has quit IRC13:34
*** ykarel is now known as ykarel|away13:35
*** efried_rollin is now known as efried13:35
*** ralonsoh has quit IRC13:35
*** rlandy has joined #openstack-infra13:36
*** rlandy is now known as rlandy|ruck13:36
*** sshnaidm|afk is now known as sshnaidm13:37
fungievrardjp: odyssey4me: it's possible approving another new project will get things back on track, or a manual zuul reconfig13:38
fungii'll see if i can figure something out in a sec13:38
*** CrayZee has quit IRC13:41
AJaegerfungi: in that case: Oldest not approved new project change is: https://review.openstack.org/#/c/537653/13:42
AJaegergood morning, fungi!13:42
*** mriedem has joined #openstack-infra13:42
openstackgerritSimon Westphahl proposed openstack-infra/zuul master: Allow using remote refs to find commits for change  https://review.openstack.org/54496413:42
openstackgerritSimon Westphahl proposed openstack-infra/zuul master: Allow using remote refs to find commits for change  https://review.openstack.org/54496413:43
fungiAJaeger: thanks! that saved me looking one up13:44
evrardjpgood mornign fungi :)13:44
evrardjpmorning*13:44
*** yamamoto has joined #openstack-infra13:44
*** yamamoto has quit IRC13:44
AJaegerfungi: if you do approve it, please review it's sibling https://review.openstack.org/537802 as well ;)13:44
openstackgerritSimon Westphahl proposed openstack-infra/zuul master: Allow using remote refs to find commits for change  https://review.openstack.org/54496413:44
fungiAJaeger: will do13:46
fungiAJaeger: mmm... do you think this is safe to import? https://github.com/pinodeca/tatu-dashboard/blob/master/.zuul.yaml13:46
fungii'm looking now for where that template is defined13:47
AJaegerfungi: let's recheck - it should fail ;)13:48
*** shardy has joined #openstack-infra13:48
fungiyeah, i believe that is yet another broken zuul config13:48
* fungi looks for a new project not importing prewritten zuul configuration13:49
*** pgadiya has quit IRC13:49
*** kgiusti has joined #openstack-infra13:50
*** annp has joined #openstack-infra13:50
AJaegerhttps://review.openstack.org/546260 ?13:50
AJaegerhas no imports ;)13:50
*** VW has joined #openstack-infra13:51
*** briancurtin has quit IRC13:52
AJaegerfungi, 537802 has no zuul config13:52
*** briancurtin has joined #openstack-infra13:52
AJaegermy test did not work on tatu-dashboard - why?13:53
*** arxcruz|ruck is now known as arxcruz|rover13:53
*** mwhahaha has quit IRC13:54
*** eernst has joined #openstack-infra13:55
odyssey4memorning fungi :) apologies for all that drama yesterday... I guess this is how we learn, and how zuul development gets its feedback :)13:55
fungiAJaeger: 537802 has https://github.com/pinodeca/python-tatuclient/blob/master/.zuul.yaml13:55
fungisimilarly broken too i think13:55
fungiodyssey4me: yep!13:55
*** ralonsoh_ is now known as ralonsoh13:55
*** oidgar has quit IRC13:55
*** mwhahaha has joined #openstack-infra13:56
*** zxiiro has quit IRC13:56
AJaegerfungi: yeah ;( Sorry, looked wrong place13:56
evrardjpodyssey4me: wait, it's not over yet :D13:56
*** GregHouse has quit IRC13:56
fungiAJaeger: since 546260 doesn't import a repo, i'm reviewing that on13:56
AJaegerfungi: my test is broken, fix coming...13:56
fungie13:56
*** imacdonn has quit IRC13:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Fix typo in tools/check_valid_gerrit_projects.py  https://review.openstack.org/54698813:57
AJaegerfungi: care to +2A simple typo, please?13:57
AJaegerthat should block the two reviews you mentioned now13:57
*** robcresswell has quit IRC13:57
*** zxiiro has joined #openstack-infra13:58
*** amoralej|lunch is now known as amoralej13:58
*** GregHouse has joined #openstack-infra13:58
*** Kevin_Zheng has quit IRC13:58
*** robcresswell has joined #openstack-infra14:00
*** zhongjun has quit IRC14:01
*** zhongjun has joined #openstack-infra14:01
*** bstinson has quit IRC14:01
*** Kevin_Zheng has joined #openstack-infra14:02
*** clayg has quit IRC14:02
*** bstinson has joined #openstack-infra14:03
*** clayg has joined #openstack-infra14:05
*** ihrachys_ has quit IRC14:05
*** ihrachys has joined #openstack-infra14:06
openstackgerritMerged openstack-infra/project-config master: Add new project 'osel'  https://review.openstack.org/54626014:06
*** Goneri has joined #openstack-infra14:06
openstackgerritMerged openstack-infra/project-config master: Fix typo in tools/check_valid_gerrit_projects.py  https://review.openstack.org/54698814:06
fungionce puppet updates zuul01.o.o for the osel addition we should hopefully be back on track14:06
*** imacdonn has joined #openstack-infra14:06
* fungi needs to step away and do morning things for a few minutes14:07
*** alexchadin has quit IRC14:08
*** lbragstad has joined #openstack-infra14:10
*** oidgar has joined #openstack-infra14:11
*** zhipeng has quit IRC14:12
*** lathiat has quit IRC14:13
*** zhipeng has joined #openstack-infra14:14
*** derekh has joined #openstack-infra14:15
pabelangerspacex launch now, FYI14:19
pabelangerAJaeger: dirk: we'll also need mirrors for opensuse-tumbleweed14:20
pabelangerthat's likely why job it failing14:20
dirkpabelanger: ehm, sorry, which job is failing?14:20
dirkgood morning btw14:20
pabelangerdirk: see https://review.openstack.org/546846/14:21
pabelangerzypper looks to be configured wrong14:22
*** myoung|afk is now known as myoung14:23
*** vabada has quit IRC14:27
*** agopi has joined #openstack-infra14:27
*** baoli has joined #openstack-infra14:27
*** vabada has joined #openstack-infra14:27
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool master: Add additional builder debug logging  https://review.openstack.org/54630314:28
*** baoli has quit IRC14:28
*** lathiat has joined #openstack-infra14:28
*** baoli has joined #openstack-infra14:29
*** dhill__ has joined #openstack-infra14:29
pabelangermnaser: fungi: ubuntu mirror looks good this morning too, we've released with no issues over night. And AFS errors (lost connection to cell) are all but gone today.  We haven't lost connection so far14:30
*** pblaho has quit IRC14:30
openstackgerritMerged openstack-infra/nodepool master: Add /label-list to the webapp  https://review.openstack.org/53556314:30
mnaserpabelanger: sweet. I think my jobs went throug without a hitch too14:31
*** esberglu has joined #openstack-infra14:31
pabelangermnaser: great14:32
*** ykarel|away has quit IRC14:35
*** olaph has joined #openstack-infra14:35
*** alexchadin has joined #openstack-infra14:35
*** olaph1 has quit IRC14:35
*** Guest96947 has quit IRC14:36
*** sree_ has joined #openstack-infra14:36
*** sree_ is now known as Guest6481014:37
*** dtantsur is now known as dtantsur|brb14:38
openstackgerritHoang Trung Hieu proposed openstack-infra/zuul master: WIP: Update and replace http with https for doc links  https://review.openstack.org/54697014:38
*** dklyle has quit IRC14:39
*** caphrim007_ has quit IRC14:39
*** caphrim007 has joined #openstack-infra14:40
*** Guest64810 has quit IRC14:41
*** pcichy has quit IRC14:42
*** caphrim007 has quit IRC14:44
*** yamamoto has joined #openstack-infra14:45
*** eernst has quit IRC14:45
*** eharney has joined #openstack-infra14:46
*** r-daneel has joined #openstack-infra14:46
*** psachin has quit IRC14:49
*** ameeda has quit IRC14:51
*** yamamoto has quit IRC14:51
*** auristor has quit IRC14:52
*** auristor has joined #openstack-infra14:54
*** eernst has joined #openstack-infra14:54
thingeehi infra, just one more +2 on adding a core group for the project sphinx-feature-classification. already has the ptl +1 too14:54
thingeehttps://review.openstack.org/#/c/545861/14:55
*** e0ne has quit IRC14:57
*** cmurphy has quit IRC14:57
*** olaph has quit IRC14:58
*** olaph has joined #openstack-infra14:58
mnaserthingee: voila14:59
*** e0ne has joined #openstack-infra14:59
*** ykarel|away has joined #openstack-infra14:59
*** ldnunes has quit IRC15:00
*** wolverineav has joined #openstack-infra15:01
odyssey4mepabelanger fungi yeah, thanks for sorting out the ubuntu mirrors - any idea why the update took so long?15:02
fungiodyssey4me: there had apparently been some disconnect with the afs backend db, and the resulting recovery was causing writes to go slowly. couple that with a largeish set of security updates for ubuntu and a mirror sync ran longer than the allotted time for its kerberos ticket so got aborted leaving a lockfile behind15:04
odyssey4meAJaeger are we good to go to recheck https://review.openstack.org/546573 & https://review.openstack.org/546294 now?15:04
*** annp has quit IRC15:05
pabelangerYup, when the database becomes corrupt in reprepro, we need to reindex everything again from AFS. And if info isn't in cache, we depend on network bandwidth to fetch new file info15:05
thingeemnaser: thank you15:05
fungithen cleaning up the lockfile and rerunning manually we noticed the slowness, realized the situation with the cache repair wasn't progressing as expected, rebooted the server which seemed to get the cache sync back underway...15:05
odyssey4melooks like the previously mentioned tests added are now working to validate a repository contents before allowing a merge15:05
*** baoli has quit IRC15:05
*** wolverineav has quit IRC15:05
odyssey4mefungi ah, makes sense - thanks all for following that through15:06
*** baoli has joined #openstack-infra15:06
fungiit was mostly pabelanger15:06
fungiodyssey4me: AJaeger: yeah, job working as designed! http://logs.openstack.org/53/537653/7/check/project-config-gerrit/82bf64d/job-output.txt.gz#_2018-02-22_14_44_16_81756315:07
fungithanks for adding that test, AJaeger!15:07
openstackgerritMerged openstack-infra/project-config master: Set sphinx-feature-classification own core group  https://review.openstack.org/54586115:07
pabelangeras for 546573 and 546294, I think we need a full reload zuul configuration. I was holding off from doing that until corvus was online, incase he wanted to debug why it still fails15:07
fungipabelanger: i approved another project addition a little while ago15:08
fungipabelanger: hoping to see if that triggers a reconfig when puppet kicks it again15:08
pabelangerkk15:08
openstackgerritGraham Hayes proposed openstack-infra/project-config master: Add certbot-dns-openstack repo  https://review.openstack.org/54702215:08
fungipabelanger: for the record, that was https://review.openstack.org/54626015:08
AJaegerfungi: great!15:09
*** tpsilva has joined #openstack-infra15:09
odyssey4meok, good - looks like the plan right now is to simply block *any* pre-implemented zuul config from being imported - which works... that's certainly easier than any other options I can think of15:10
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Add zuul config for new project 'osel'  https://review.openstack.org/54642115:10
openstackgerritGraham Hayes proposed openstack-infra/project-config master: Add zuul entry for certbot-dns-openstack  https://review.openstack.org/54702315:11
pabelangerodyssey4me: there also been some discussion in #zuul about maybe no loading a project in zuul, if the configuration is bad too. Will be a topic at PTG I believe.15:11
AJaegerodyssey4me: yes, big axe - let's import these files via a normal review step.15:11
odyssey4meI support this course of action.15:16
fungiaha, i think the big gcc update we saw in ubuntu was the retpoline enablement backport15:16
odyssey4meOf course ideally zuul could protect itself without requiring an external test, but I can understand that's hard to do and certainly the axe works as a stop gap to buy more time.15:17
*** udesale has quit IRC15:19
*** mriedem has left #openstack-infra15:20
*** mriedem has joined #openstack-infra15:20
*** links has quit IRC15:20
pabelangerdmsimard: are you aware of an issue with fedora-26 jobs in ozj? http://logs.openstack.org/48/546848/1/check/openstack-infra-multinode-integration-fedora-26/9c7aa43/job-output.txt.gz15:22
pabelangerTimeout (32s) waiting for privilege escalation prompt:15:22
pabelangerwhen reloading iptables15:22
dmsimardpabelanger: yes, I have held nodes to troubleshoot it15:22
dmsimardpabelanger: Haven't gotten to the bottom of it yet15:22
AJaegerfungi, we're green again with the new repos!15:22
dmsimardpabelanger: If it's blocking something, we can put it non-voting15:23
AJaegerdmsimard, fungi , pabelanger, could you review https://review.openstack.org/545234 to refactor static publish jobs as well as https://review.openstack.org/545442 and https://review.openstack.org/545603 to mark base jobs as abstract, please?15:23
pabelangerdmsimard: okay, I don't fully understand what the jobs is doing. Not blocking, wanted to add fedora-27 nodes, because we want to delete fedora-2615:24
dmsimardpabelanger: that particular test is for iptables rules persistence integration testing -- we run the multinode roles from zuul-jobs, flush iptables and then restart it to see if the rules are back15:25
dmsimardpabelanger: I don't have any objection to move forward with f2715:25
*** dtantsur|brb is now known as dtantsur15:26
*** alexchadin has quit IRC15:26
fungiodyssey4me: yep, i believe the plan is that zuul will skip loading repos with invalid configuration rather than reverting to an old layout15:27
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Set openstack-infra-multinode-integration-fedora-26 non-voting  https://review.openstack.org/54702615:27
pabelangerdmsimard: ^15:27
*** alexchadin has joined #openstack-infra15:27
AJaegerfungi: and that's what happened in this case. The repo was completely ignored - but that also meant, we could not run any tests to merge anything using the gerrit workflow.15:29
*** slaweq has quit IRC15:30
*** slaweq has joined #openstack-infra15:30
*** alexchadin has quit IRC15:31
*** e0ne has quit IRC15:34
*** slaweq has quit IRC15:35
*** e0ne has joined #openstack-infra15:35
fungiyeah, i guess the solution is more nuanced than that, but for discussing in #zuul15:36
*** alexchadin has joined #openstack-infra15:37
*** krtaylor has quit IRC15:38
*** yamahata has joined #openstack-infra15:40
*** alexchadin has quit IRC15:42
*** zhipeng has quit IRC15:42
openstackgerritPaul Belanger proposed openstack-infra/glean master: Testing  https://review.openstack.org/54703415:42
openstackgerritGraham Hayes proposed openstack-infra/project-config master: Add certbot-dns-openstack repo  https://review.openstack.org/54702215:42
openstackgerritGraham Hayes proposed openstack-infra/project-config master: Add zuul entry for certbot-dns-openstack  https://review.openstack.org/54702315:42
corvuspabelanger: debug why what still fails?15:45
openstackgerritAndrea Frittoli proposed openstack-infra/devstack-gate master: Add stable/pike Tempest bitrot job  https://review.openstack.org/54703515:46
*** hongbin has joined #openstack-infra15:46
pabelangercorvus: ya, we force merged changes to remove the bad zuul.d folders, however zuul still complained about project conflict. It looks like we got a full reconfiguration @ Last reconfigured: Thu Feb 22 2018 09:46:54 GMT-0500 (EST). So, I guess we recheck and see if it works15:47
*** yamamoto has joined #openstack-infra15:47
pabelangerwasn't sure if that was the required workflow or some other issue15:47
pabelangerokay, https://review.openstack.org/546573/ is good15:48
pabelangerso, guess we needed a reload then15:48
openstackgerritMerged openstack-infra/project-config master: Add zuul config for new project 'osel'  https://review.openstack.org/54642115:48
corvuspabelanger: yes, zuul only reloads its tenant config file on a full reconfiguration.15:49
pabelangercorvus: ack, thanks15:49
*** david-lyle has joined #openstack-infra15:50
openstackgerritAndrea Frittoli proposed openstack-infra/project-config master: Remove legacy tempest bitrot jobs for pike  https://review.openstack.org/54703715:51
*** rossella_s has quit IRC15:51
*** yamamoto has quit IRC15:51
openstackgerritSean McGinnis proposed openstack-infra/project-config master: Limt jobs on release-tools presentations  https://review.openstack.org/54704115:55
openstackgerritAndrea Frittoli proposed openstack-infra/openstack-zuul-jobs master: Cleanup tempest pike legacy jobs.  https://review.openstack.org/54704215:56
*** VW has quit IRC15:56
*** VW has joined #openstack-infra15:57
*** eventingmonkey has quit IRC15:58
pabelangerdynamic reloads looked to have climbed up a little: http://paste.openstack.org/show/682153/15:58
pabelanger20 seconds now15:58
fungipabelanger: yeah, memory utilization is climbing in the past hour too16:00
fungihttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all16:00
pabelangerso it is16:01
zxiiroWhen OpenStack was still using Jenkins. Did jobs run using ansible playbooks? if so how was that done, did they use the ansible jenkins plugin?16:01
andreafAJaeger fungi corvus pabelanger tosky tonyb three patches to replace legacy tempest jobs with zuulv3 jobs for pike (on stable and bitrot jobs) now that devstack patches have been backported to pike https://review.openstack.org/#/q/status:open+branch:master+topic:stable_pike16:02
pabelangerzxiiro: we wrote zuul-launcher which loaded JJB files at run time, and converted them into ansible-playbooks. Then the launchers run ansible-playbook locally16:02
fungizxiiro: no, we switched from jenkins ssh plugin running jobs to zuul v2's experimental ansible launcher which basically replaced our jenkins masters16:03
fungior what pabelanger also said16:03
zxiiroah ok. so even before that they were basically regular freestyle jobs with shell16:04
fungithe ansible solution in v2 was our replacement/alternative for jenkins, basically16:04
pabelangerzxiiro: yah, you can look at 2.6.0 release of zuul to see how it worked.16:04
pabelanger(not that I recommend running it in production)16:04
pabelangerbut like fungi said, jenkins was gone at that point16:05
openstackgerritSean McGinnis proposed openstack-infra/project-config master: Switch notifications for stable branches to match all  https://review.openstack.org/54669016:05
fungiright, we limited our jenkins plugin utilization as much as possible, anticipating a future where we switched to something else and wouldn't want to have to reimplement various jenkins plugins in some other framework16:05
*** rossella_s has joined #openstack-infra16:05
zxiirogood to know. thanks. mostly just curious. I'm exploring ways to make Jenkins launch Ansible. The Ansible Jenkins plugin seems to be the current frontrunner.16:06
openstackgerritMerged openstack-infra/release-tools master: stub in presentation for rocky ptg  https://review.openstack.org/54506616:06
openstackgerritMerged openstack-infra/release-tools master: draft of presentation for rocky ptg  https://review.openstack.org/54510516:06
fungiwhich made translating our mostly-shell job definitions from jjb to ansible playbooks tractable16:06
VWhey folks - our AUP department is getting a takedown request for this http://paste.openstack.org/raw/665906/16:06
*** caphrim007 has joined #openstack-infra16:06
VWsupposedly the decompiler offered there is a no no16:07
hughsaundersHey, is there a timescale for a nodepool release post zuulv3 branch becoming master?16:07
pabelangerzxiiro: you might also be interested in https://etherpad.openstack.org/p/zuulv3-jenkins-integration which is some ideas on how zuulv3 might integrate with jenkins16:07
pabelangerhughsaunders: already done! we merged feature/zuulv3 into master last month16:07
fungiVW: i can likely delete it from the database, but we also tell search engines not to index that service so i'm curious how it was even found16:07
pabelangerhughsaunders: but, v3.0 release should be coming in the next few weeks16:07
VWyeah - that I don't know, fungi16:08
pabelangerhughsaunders: https://storyboard.openstack.org/#!/board/53 is last items to finish off16:08
hughsaunderspabelanger: ahh, ok I was going to say 0.5.0 was 9th Jan and merge was after (18th?)16:08
VWI just got hit up because I'm listed as the primary on the account you all use16:08
hughsaundersThanks for storyboard link16:08
fungiVW: oh! our robots.txt isn't being served now for some reason. that might explain it16:08
VWwell, there you go16:08
VW:)16:08
*** eventingmonkey has joined #openstack-infra16:09
pabelangerhughsaunders: yes, that sounds right. Let me see if I can find the review16:09
fungiVW: thanks! cleaning that up now and also figuring out why robots.txt is missing16:09
VWcool - thanks fungi16:09
VWI'll inform the Brute Squad16:09
pabelangerhughsaunders: https://review.openstack.org/535435/16:10
hughsaundersyeah thats the one16:10
AJaegerfungi, the two tatu repos should be fixed now - want to remvoe your -1 from https://review.openstack.org/#/c/537653/ ?16:10
hughsaundersI just noticed theres quite a large delta between last release and master (due to that merge). Will use a SHA for now while waiting for the v3 tag.16:11
*** ramishra has quit IRC16:11
andreafpabelanger not using delegate_to and doing push/pull was the working solution https://review.openstack.org/#/c/528074/19/roles/sync-devstack-data/tasks/main.yaml16:12
*** sshnaidm is now known as sshnaidm|off16:12
pabelangerhughsaunders: yah, there is some breaking changes to have pre feature/zuulv3 nodepool works too. The configuration format will be a breaking change.16:12
hughsaundersnot to mention the disappearance of zmq...16:13
pabelangerandreaf: good to know16:14
*** olaph has quit IRC16:14
*** olaph has joined #openstack-infra16:14
pabelangerhughsaunders: yah, 0.5.0 does support zookeeper for builders, IIRC. But yah, zmq dropped too16:14
*** shoogz has joined #openstack-infra16:15
*** sshnaidm|off has quit IRC16:17
*** rossella_s has quit IRC16:20
smcginnisLooks like zuul events are getting backed up. Anything to be concerned about?16:21
*** rossella_s has joined #openstack-infra16:23
pabelangerfor some reason, reloads are taking a little longer now. Which means we are not starting builds as fast: http://grafana.openstack.org/dashboard/db/zuul-status16:25
openstackgerritJavier Peña proposed openstack-infra/system-config master: Move AFS mirror code to puppet-openstackci  https://review.openstack.org/52903216:26
smcginnisGerrit UI seems slow too. Network issues?16:26
*** agopi has quit IRC16:29
fungi#status log deleted http://paste.openstack.org/raw/665906/ from lodgeit openstack.pastes table (paste_id=665906) due to provider aup violation/takedown notice16:30
openstackstatusfungi: finished logging16:30
fungiVW: ^16:30
*** yolanda has quit IRC16:30
openstackgerritJeremy Stanley proposed openstack-infra/puppet-httpd master: Allow access to docroot for proxy exclusions  https://review.openstack.org/54706216:30
*** yolanda has joined #openstack-infra16:30
fungiinfra-puppet-core: ^ should make our robots.txt work on the paste.o.o site16:31
*** tosky has quit IRC16:31
VWthanks fungi!16:32
fungiVW: thanks for bringing it to our attention! we had intentionally disallowed crawlers on that server for precisely that reason16:32
VWmy pleasure.  I'll let you know if I hear anything else16:32
fungiplease do16:33
*** armaan has quit IRC16:33
*** armaan has joined #openstack-infra16:33
*** oidgar has quit IRC16:35
*** d0ugal has quit IRC16:35
*** andreas_s has quit IRC16:38
*** slaweq has joined #openstack-infra16:40
*** d0ugal has joined #openstack-infra16:40
*** david-lyle is now known as dklyle16:42
*** andreas_s_ has joined #openstack-infra16:43
*** slaweq has quit IRC16:45
openstackgerritFabien Boucher proposed openstack-infra/zuul master: Tenant config can be read from an external script  https://review.openstack.org/53587816:46
*** e0ne has quit IRC16:46
*** andreas_s has joined #openstack-infra16:47
*** andreas_s_ has quit IRC16:47
openstackgerritMerged openstack-infra/project-config master: Add new project for Tatu (SSH as a Service) Horizon Plugin.  https://review.openstack.org/53765316:47
*** yamamoto has joined #openstack-infra16:48
*** cmurphy has joined #openstack-infra16:48
openstackgerritFabien Boucher proposed openstack-infra/zuul master: Tenant config can be read from an external script  https://review.openstack.org/53587816:48
*** eernst has quit IRC16:49
*** oidgar has joined #openstack-infra16:50
*** pblaho has joined #openstack-infra16:51
*** andreas_s has quit IRC16:51
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: Refactor status functions, add web endpoints, allow params  https://review.openstack.org/53630116:52
*** yamamoto has quit IRC16:52
openstackgerritFabien Boucher proposed openstack-infra/zuul master: Tenant config can be read from an external script  https://review.openstack.org/53587816:53
*** eernst has joined #openstack-infra16:54
*** wolverineav has joined #openstack-infra16:55
*** wolverineav has quit IRC16:55
*** wolverineav has joined #openstack-infra16:55
*** ihrachys has quit IRC16:56
dirkpabelanger: is there a way for me to download the qcow2 file (opensuse-tumbleweed) somehow?16:56
*** ihrachys has joined #openstack-infra16:56
dirkpabelanger: I am interested what is in there16:56
*** dklyle has quit IRC16:56
*** rossella_s has quit IRC16:57
pabelangerdirk: nothing public by default, I can expose it manually. Something your DIB dosn't have?16:57
*** rossella_s has joined #openstack-infra16:59
*** VW_ has joined #openstack-infra17:00
*** VW has quit IRC17:00
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: Refactor status functions, add web endpoints, allow params  https://review.openstack.org/53630117:00
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands  https://review.openstack.org/53630317:00
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: webapp: add optional admin endpoint  https://review.openstack.org/53631917:00
*** david-lyle has joined #openstack-infra17:01
Zarahm, not sure where to ask- I'm getting my ptg ticket now; I have a tsp code but the cost is still displayed as $400 after applying it. is that normal? worried I'll  accidentally charge myself or am otherwise confused. :)17:02
*** jpich has quit IRC17:04
*** links has joined #openstack-infra17:04
fungiZara: good question, i'll find out who you should contact17:05
Zarathanks :)17:07
clarkbgood morning17:07
pabelangero/17:08
openstackgerritGraham Hayes proposed openstack-infra/project-config master: Add certbot-dns-openstack repo  https://review.openstack.org/54702217:09
openstackgerritGraham Hayes proposed openstack-infra/project-config master: Add zuul entry for certbot-dns-openstack  https://review.openstack.org/54702317:09
*** zoli is now known as zoli|gone17:11
*** zoli|gone is now known as zoli17:11
fungiZara: kendall waters said it shouldn't charge you, but suggested you e-mail ptg@openstack.org and she can help you out17:11
*** armaan has quit IRC17:12
*** armaan has joined #openstack-infra17:13
*** agopi has joined #openstack-infra17:16
*** agopi_ has joined #openstack-infra17:18
*** agopi has quit IRC17:20
*** agopi_ has quit IRC17:22
*** baoli has quit IRC17:24
*** baoli has joined #openstack-infra17:24
*** gyee has joined #openstack-infra17:25
Zarafungi: great, thanks, will do :)17:27
*** dprince has quit IRC17:28
*** slaweq has joined #openstack-infra17:29
jlvillalHey clarkb. I did the puppet-gerritbot change to install via git. It is passing Zuul. But I'm mostly cargo-culting my changes. So a review from someone who knows puppet is appreciated :) https://review.openstack.org/#/c/546700/17:32
clarkbjlvillal: ok17:32
jlvillalThanks.17:33
openstackgerritMerged openstack-infra/nodepool master: Refactor playbooks/nodepool-zuul-functional/pre.yaml  https://review.openstack.org/54627217:34
mnaserclarkb: jlvillal ok if i review from a puppet pov? :)17:34
jlvillalmnaser, Yes please :)17:35
clarkbmnaser: yes!17:35
fungimnaser: speaking of puppet module changes, https://review.openstack.org/547062 is semi-urgent17:36
mnaserfungi: will look after this17:36
clarkbjlvillal: left some thoughts but you amy want to wait for mnaser to comment too17:36
jlvillalclarkb, thanks17:37
*** olaph1 has joined #openstack-infra17:38
*** olaph has quit IRC17:39
mnaserjlvillal: left a few comments but it's mostly ok! :)17:40
AJaegerconfig-core, could you review a refactor of our static publish jobs, please? https://review.openstack.org/54523417:40
jlvillalmnaser, Thanks!17:40
AJaegermnaser: could you review these initial jobs for repos: https://review.openstack.org/#/c/546573/ https://review.openstack.org/538307 https://review.openstack.org/54629417:43
AJaeger, please?17:43
mnaserAJaeger: sure, ill have a look and fungi it looks good to me but cant +A :)17:44
AJaegerthanks!17:44
fungimnaser: no problem, just appreciate the look!17:44
*** agopi has joined #openstack-infra17:44
*** olaph1 is now known as olaph17:45
mnaserAJaeger: all done17:45
AJaegergreat17:45
*** lpetrut has quit IRC17:46
mnaserhttps://review.openstack.org/#/c/546690 is an easy one for any config-core :)17:46
AJaegermnaser: time for a job removal as well, please? https://review.openstack.org/#/c/546699/17:47
AJaegermnaser: will check...17:47
fungiinfra-puppet-core reviewers: i'm looking for a second +2/approval on 547062 to fix serving robots.txt on paste.o.o so spam/scams/illegal content indexed there stop getting picked up by search engines17:47
*** slaweq has quit IRC17:48
*** yamamoto has joined #openstack-infra17:48
clarkbfungi: oh interesting this is if it uses the built in template rather than providing one of its own (whcih we typically do instead)17:50
clarkbfungi: I've approved it17:50
fungithanks clarkb17:51
*** armaan has quit IRC17:51
fungiand yeah, the built-in one was basically broken (at least for apache 2.4 on xenial, may have been working-ish on 2.2 without explicit access allowances)17:51
*** jpena is now known as jpena|off17:52
*** armaan has joined #openstack-infra17:52
openstackgerritJohn L. Villalovos proposed openstack-infra/puppet-gerritbot master: Change gerritbot to install from git  https://review.openstack.org/54670017:52
fungimy guess is it worked on an implicit lack of acl until we upgraded paste.o.o to trusty17:52
fungier, to xenial17:52
fungiat which point we stopped serving robots.txt and it reverted to being a cesspool of cut-n-paste links to illicit sites, eventually drawing complaints to our provider17:54
*** yamamoto has quit IRC17:54
*** dprince has joined #openstack-infra17:55
*** mgoddard_ has quit IRC17:55
*** efoley has joined #openstack-infra17:55
fungihuh... kde started using phabricator for hosting/reviewing source code?17:56
*** derekh has quit IRC18:00
mnaserjlvillal: any reason behind the decision to install deps then gerritbot after?18:02
jlvillalmnaser, cargo-cult from puppet-zuul18:02
openstackgerritMerged openstack-infra/project-config master: Add OSA os_panko repo base jobs  https://review.openstack.org/54657318:02
fungimnaser: we want to upgrade gerritbot but not unconditionally upgrade its deps if what is already installed is satisfactory18:03
jlvillalmnaser, https://github.com/openstack-infra/puppet-zuul/blob/master/manifests/init.pp#L19318:03
mnaserjlvillal: i hate thinking of a suggestion after reviewing but i think this is really useful for you - https://github.com/openstack-infra/puppet-statusbot/blob/master/manifests/init.pp18:03
mnaserstatusbot deploys from master afaik18:03
fungithis will get simpler with pip 10 since --upgrade-strategy=only-if-needed becomes the new default18:04
jlvillalmnaser, looking...18:04
AJaegerany config-core around to review https://review.openstack.org/545234 to simplify our static publish jobs , please?18:04
jlvillalmnaser, What am I looking for?18:04
mnaserjlvillal: drawing general inspiration for future maybe :P but it's useful code to look at in comparision, pretty much same idea18:05
mnaserthe change looks fine to me if fungi mentioned the whole pip install thing is ok18:05
jlvillalmnaser, Okay. Thanks!18:05
fungiwell, mainly just explaining why it's done in such a roundabout way in some of our modules right now18:05
fungi"okay" is not the term i'd use, but it's a necessary evil in at least some situations18:06
mnaser:D18:06
fungiparticularly if you want to mix distro-packaged python modules and pypi18:06
clarkbAJaeger: part of me wonders if we want to do it as it may become easier to sneak a change that dumps the secret through since its not immediately obvious that the secret is being used in the child jobs18:07
clarkbbut that could be me being overly paranoid of potential situations18:07
* jlvillal remembers only the paranoid survive :)18:07
fungias pip historically (and still for a little while longer) will download newer versions of your deps from pypi when you specify --upgrade/-U even if you have locally preinstalled sufficient versions18:07
*** SumitNaiksatam has joined #openstack-infra18:07
*** pcichy has joined #openstack-infra18:08
clarkbspeaking of pip 10, are they still planning to break the world by not uninstalling things globally?18:08
clarkbthats going to be a fun weekend for us :/18:08
funginot sure18:08
clarkbI should resurrect my change to test it18:08
fungibut yeah, the problem there crops up when, say, you have something like numpy in your dep tree. it has no manylinux1 wheel on pypi so has to be built from sdist, requiring a number of additional c headers preinstalled and potentially taking far longer than our puppet exec timeout to complete18:09
AJaegerclarkb: according to the docs, the child has no access to it ;)18:09
AJaegerclarkb: https://docs.openstack.org/infra/zuul/user/config.html#secret has18:10
fungithe obvious workaround is to apt install python-numpy or whatever, but then pip will try to upgrade to a newer version, drag down the sdist, and break or timeout rebuilding leaving you with a nonfunctional server18:10
AJaeger"Additional pre or post playbooks which appear in child jobs will not have access to the secrets"18:10
AJaegerclarkb: ^18:10
clarkbfungi: https://review.openstack.org/513825 new patchset that undoes some of my earlier workarounds so that we can test more vanilla pip 10 behavior18:11
fungiclarkb: thanks!18:11
clarkbAJaeger: aha! that answers that then18:11
*** ralonsoh has quit IRC18:12
*** pcichy has quit IRC18:12
clarkbfungi: in our case pyyaml and python-psutil were breaking in devstack18:12
*** pcichy has joined #openstack-infra18:12
clarkbfungi: because we install pyyaml in devstack-gate for features.yaml support from the distro and not pypi and python-psutil is part of devstack's debs and rpms lists but then python packages we pip install dep on it too and try to upgrade it18:13
clarkbfungi: we should know soon enough how broken it is based on 513825's results18:13
openstackgerritMerged openstack-infra/project-config master: Zuul templates for new project tatu-dashboard.  https://review.openstack.org/53830718:14
openstackgerritMerged openstack-infra/project-config master: Switch notifications for stable branches to match all  https://review.openstack.org/54669018:14
*** oidgar has quit IRC18:14
clarkbanyone else want to review 545234 for AJaeger before I approve it?18:14
fungiclarkb: given that the default upgrade strategy is now only-if-needed in 10, that may no longer happen18:15
clarkbfungi: ya though it was broken on pip 10 when I first tested it. Do you know if that is a recent pip 10 change?18:15
clarkboh actually I bet its constraints burning us there18:15
AJaegeryeah, we can get rid of jenkins/data now - infra-root, one bindep change: https://review.openstack.org/#/c/543142/ and one for config-core on openstack-zuul-jobs: https://review.openstack.org/#/c/543141/ - please review18:15
clarkbupper constraints is almost always going to be an upgrade required compared to the distro packages18:15
*** slaweq has joined #openstack-infra18:15
*** gfidente has quit IRC18:16
AJaegeronce those two are in, we can merge https://review.openstack.org/#/c/543140/18:16
*** ykarel|away has quit IRC18:16
fungiclarkb: oh, yes, so the upgrade strategy default changed in decemberish i think, but also constraints is likely an issue if we specify a different version than the distro is packaging18:16
fungiclarkb: i stand corrected. looks like it may have gone in as early as may of last year https://github.com/pypa/pip/pull/450018:17
clarkbassuming pip doesn't change this behavior they are effectively saying you must use virtualenvs18:17
fungiso yeah, probably constraints is the issue18:17
mordredclarkb, fungi: maybe we should stop installing python things from distro packages in devstack or devstack-gate18:17
clarkbmordred: that was my workaround18:17
mordredclarkb: awesome. I think it's not a workaround- I think it's the correctthing18:17
clarkbmordred: probably is you can't completely avoid it in all cases18:18
clarkbbeacuse cloud init for example18:18
clarkbwe don't have it but other people running devstack will18:18
clarkband I disagree I think pip's behavior is completely the wrong thing18:18
clarkbyou may as well just error if not installing to a virtualenv now18:18
*** dklyle has joined #openstack-infra18:18
mordredoh - well, no arguments from me there18:18
fungii'm still not convinced that `sudo pip install ...` is a sane strategy anyway. using virtualenvs and avoiding mixing package managers makes a lot more sense to me but that train has sailed i guess18:18
clarkbor give us a flag to say we really know what we are doing and leave us a lone18:19
pabelangerfungi: yah, agree18:19
clarkbfungi: ya the problem is it has been a supported configuration for as long as pip has existed18:19
mordredI just meant in our repos we should basicaly never install python from distro packages because we install things from pip globally as well - and doing both is *always* going to break something18:19
clarkbyou can't just remove that functionality imo18:19
clarkbmaybe disable it by default18:19
*** dhajare has quit IRC18:19
pabelangerinstall everything in to virtualenv, update systemd unit files as needed, that's worked well so far with testing I've been doing18:19
fungiclarkb: a "supported" configuration where the upstream pip maintainers for the past decade have basically said "you really shouldn't do this but..."18:19
clarkbfungi: then they shouldn't have added the feature :P18:20
clarkbfungi: the problem is the ship sailed once it was allowed and people used it18:20
mordred++18:20
fungii agree on that point18:20
fungii don't think they realized what a problem it would grow into18:20
clarkbpabelanger: there are weird corner cases where it doesn't work like what we had with dib18:20
clarkbpabelanger: these are often fixable its just not something you can rely on when pulling python packages18:21
fungiand once they did and better solutions were devised, they couldn't easily take it back18:21
clarkbpabelanger: also does that mean you are volunteering to fix devstack? :P18:21
*** david-lyle has quit IRC18:22
*** yamahata has quit IRC18:22
clarkbfwiw on debuntu it really isn't an issue18:22
mordredclarkb: you know - I wonder if devstack would work if we just added --user to the pip install commands18:22
clarkbmordred: you'd have to update all the unit file paths too for example but ya its doable18:23
clarkband then fix any dib sort corner cases if you run into them18:23
mordredugh. unit files18:23
clarkb(the joy of systemd requiring fully rooted paths)18:23
mordred\o/18:23
clarkbalso this will break how we install zuul and nodepool etc etc etc18:24
clarkbits going to be a fairly painful transition18:24
openstackgerritMerged openstack-infra/puppet-httpd master: Allow access to docroot for proxy exclusions  https://review.openstack.org/54706218:25
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: remove legacy-rally-dsvm-fakevirt-heat  https://review.openstack.org/54669918:25
*** slaweq has quit IRC18:25
mordredclarkb: I'm still fully catching up - what does it break about our zuul/nodepool installs?18:25
mordredclarkb: or, rather, what's the behavior change?18:26
clarkbmordred: the breakage is if you pip install nodepool and it needs to upgrade some package that is system installed pip says no I won't do it and your install fails18:26
openstackgerritFabien Boucher proposed openstack-infra/zuul master: Tenant config can be read from an external script  https://review.openstack.org/53587818:26
clarkbsince we install zuul and nodepool globally and can't compeltely control the fact that there are common python system libs we lose18:26
clarkbso basically anywhere we overlap in dependencies between system and $application we are likely to eventually break due to pip 1018:27
mordredI thought we'd already stopped installing python libs from distro packages for nodepool/zuul18:27
*** lpetrut has joined #openstack-infra18:27
clarkbfor our things yes18:27
clarkbbut nothing stops $package from pulling things in18:27
mordrednod18:27
clarkbI expect that this will be particularly painful on centos18:27
clarkbbecause yum18:27
clarkbfedora's switch to dnf in theory makes this better?18:27
*** tesseract has quit IRC18:28
fungidoes dnf use different installation paths?18:28
clarkbfungi: no it just doesn't use python18:28
openstackgerritJohn L. Villalovos proposed openstack-infra/puppet-gerritbot master: Change gerritbot to install from git  https://review.openstack.org/54670018:28
fungiaha, got it18:28
clarkbon ubuntu our biggest headache has been cloud init which we personally do not use but others do18:28
pabelangerclarkb: ha, I mean adding it to devstack would be a good adventure18:28
fungiso saying yum itself depends on a bunch of rpms of python libs18:28
clarkbfungi: ya18:28
*** gfidente has joined #openstack-infra18:29
fungithis is one of the reasons red hat has persisted in stating that you shouldn't run any non-rh-packaged python applications using the distro-installed interpreter18:29
pabelangerclarkb: but, I'd be willing to try using virtualenv directly in devstack.  Maybe find some time at PTG to test it out18:30
fungiand instead have your own python build with your own lib search path and install your python apps and deps in there18:30
*** lamt has quit IRC18:30
clarkbpabelanger: ya in theory devstack supports it but I have no idea if it works or if it was updated to use unit files18:30
fungithe party line, at least for years, seems to have been that the packaged python interpreter and libraries are only there to support packaged python tools and applications shipped as a part of the distro18:31
fungiwhich i think is also how they justified not having python 3 available even as recently as rhel 718:32
clarkbpersonally I think the best approach would be to disable the uninstall behavior by default with a warning that pip 11 will completely remove the functionality18:32
clarkbthen give people ability to fix things on pip 10 by enabling the feature18:32
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Set openstack-infra-multinode-integration-fedora-26 non-voting  https://review.openstack.org/54702618:33
*** openstackgerrit has quit IRC18:33
AJaegerclarkb: shall I +A 545234  or anybody else reviewing the static publish job changes?18:38
clarkbAJaeger: I was hoping others would give it a look over since it changes a security related item but I'm fairly comfortable with ti based on the docs at least18:38
AJaegermordred, clarkb : we have updated images, could you review https://review.openstack.org/#/c/543141 and https://review.openstack.org/#/c/543142/  , please?18:39
*** slaweq has joined #openstack-infra18:39
AJaegerclarkb: I can wait until tomorrow morning my time and then approve if nobody reviews - ok?18:39
*** pcichy has quit IRC18:39
clarkbAJaeger: wfm18:39
AJaegerack18:40
*** openstackgerrit has joined #openstack-infra18:40
openstackgerritMerged openstack-infra/project-config master: Migrate to Storyboard for Qinling related projects  https://review.openstack.org/54068118:40
openstackgerritMerged openstack-infra/project-config master: Add individual core group for openstack-ansible-os_neutron  https://review.openstack.org/54657718:40
*** claudiub has quit IRC18:41
*** slaweq has quit IRC18:43
clarkbmordred: http://logs.openstack.org/25/513825/6/check/tempest-full/ec9c70b/job-output.txt.gz#_2018-02-22_18_34_18_732384 that shows you how pip 10 will break18:47
clarkband ^ confirms the behavior has not changed so we should be prepared for much of our world to be broken once pip 10 happens18:47
fungiugh18:47
openstackgerritMerged openstack-infra/project-config master: Add OSA nspawn host/container repo base jobs  https://review.openstack.org/54629418:47
mnasergerrit just 502'd18:48
mnaserfor what its worth18:48
clarkblooks like could be GC related looking at melody18:49
clarkbif it persists we'll probably have to restart the server18:49
mnaserit doesn't bother me to f5 but i just want to give a heads up in case you're tracking anything18:49
fungiclarkb: what's the temporary workaround flag for the refusal to uninstall distutils-based packages?18:49
clarkbfungi: I don't think there is one18:49
clarkbfungi: thats my grump18:50
mpetersonhey, I have a set of jobs that are not generating the ARA html, I think it might be a bug of the role definition but I don't have enough visibility in the logs to be able to see what ansible is doing.. can someone check? https://review.openstack.org/#/c/517359/ any of the functional or fullstack jobs18:50
clarkbmpeterson: we only generate them on failure18:50
fungiclarkb: oh, thought you said there was a temporary one which would be removed in pip 1118:50
clarkbfungi: no I'm saying thats what pip should do but as far as I know is not doing that18:50
fungiahh18:50
*** yamamoto has joined #openstack-infra18:50
fungi:(18:50
clarkbmpeterson: and looks like functional and fullstack are successful job runs so won't have ara output. We did this to conserve inodes on the log filesystem :/ there is work to have ara use a database file rather than many tiny files18:51
clarkbdmsimard: ^ is that work done? maybe we can enable ara on successes now?18:51
clarkbpabelanger: we should probably pick a night for the team dinner soonish18:52
mpetersonclarkb: oh, gotcha! thanks for the explanation, since they are also useful when jobs have succeeded18:52
*** dtantsur is now known as dtantsur|afk18:52
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add fedora-27 devstack / tempest jobs  https://review.openstack.org/54710718:52
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove fedora-26 devstack / tempest testing  https://review.openstack.org/54710818:52
*** pbourke_ has quit IRC18:53
pabelangerclarkb: yah, last I looked Tuesday seem to be the winner18:53
*** yamamoto has quit IRC18:56
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Switch devstack / tempest testing to fedora-27  https://review.openstack.org/54710918:56
*** dhajare has joined #openstack-infra18:58
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove fedora-26 devstack / tempest testing  https://review.openstack.org/54710818:58
*** yamamoto has joined #openstack-infra18:59
*** slaweq has joined #openstack-infra19:00
openstackgerritEric Kao proposed openstack-infra/system-config master: Start logging openstack-self-healing channel  https://review.openstack.org/54711019:03
*** yamamoto has quit IRC19:04
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Change windmill jobs from fedora-26 to fedora-27  https://review.openstack.org/54711219:04
*** slaweq has quit IRC19:05
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Switch bindep testing to fedora-27  https://review.openstack.org/54711319:06
pabelangerAJaeger: thanks for review, will fix in a moment19:07
clarkbhere is an interesting side effect to the pip 10 behavior19:08
clarkbthe python3 job appears to work because we install python2 system packages :)19:08
*** gfidente has quit IRC19:09
AJaegerpabelanger: can I trade in https://review.openstack.org/#/c/543141 and https://review.openstack.org/#/c/543142 , please? ;)19:10
*** links has quit IRC19:11
*** amoralej is now known as amoralej|off19:12
*** wolverineav has quit IRC19:12
*** wolverineav has joined #openstack-infra19:13
*** wolverineav has quit IRC19:13
*** wolverineav has joined #openstack-infra19:14
*** yamamoto has joined #openstack-infra19:14
mnaserinfra-root: how much of a hassle is it to run a manual mirror update for ubuntu?  lxc released new images (with newer packages) and mirrors aren't updated yet so installs are broken in the lxc containers for OSA19:16
*** wolverineav has quit IRC19:16
*** olaph has quit IRC19:16
*** wolverineav has joined #openstack-infra19:17
clarkbmnaser: it should happen automatically every 4 hours now that pabelanger fixed the mirrors19:17
*** olaph has joined #openstack-infra19:17
clarkbpabelanger: ^ maybe a lock was held?19:17
mnaserclarkb: oh i thought mirrors were every 24 hours for some reason, my bad19:17
AJaegerpabelanger: thanks19:19
clarkbmnaser: that said do we even mirror lxc images?19:19
clarkboh wait I get it19:19
*** yamamoto has quit IRC19:19
mnaserclarkb: :)19:19
mnasernew images with old mirrors19:19
clarkbits the new images which are not mirrored using new packages that we don't have so updates don't work19:19
mnaseri was investigating mirroring them right now19:20
mnaserand how much space that costs19:20
mnaserlooks like its a ~100M or so pull for every job19:20
*** rossella_s has quit IRC19:20
clarkbmnaser: we'd probably want to cache them rather than properly mirror them like we do with docker images19:20
clarkbmnaser: http://mirror.dfw.rax.openstack.org/ubuntu/timestamp.txt last update was about an hour ago19:21
mnaserclarkb: that would probably make things much easier then, because the urls are timestamped for the images so easy invalidation19:21
AJaegermnaser: could I trouble you for another review, please? https://review.openstack.org/#/c/543141 is ready...19:23
*** rossella_s has joined #openstack-infra19:23
mnaserAJaeger: is bindep-fallback confirmed to exist by any jobs right now19:24
AJaegermnaser: yes, see https://review.openstack.org/#/c/54314219:24
AJaegerit only works since we have rebuild images with that one in now19:25
mnaserAJaeger: ok cool makes sense!19:26
mnaser+A19:26
AJaegerthanks, mnaser19:27
*** rossella_s has quit IRC19:28
*** xarses has joined #openstack-infra19:28
AJaegerpabelanger: barbican and bifrost have jobs which use legacy-fedora-2619:29
*** yamamoto has joined #openstack-infra19:29
pabelangerclarkb: I can look19:31
clarkbpabelanger: I checked the timestamp and I think we are good19:31
pabelangerokay19:31
clarkbpabelanger: sorry didn't make that clear19:31
pabelangernp19:31
pabelangerAJaeger: kk, I'll update them shortly19:31
*** timburke_ is now known as timburke19:32
*** rossella_s has joined #openstack-infra19:32
*** yamamoto has quit IRC19:34
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Move jenkins/data/bindep-fallback.txt  https://review.openstack.org/54314119:36
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add fedora-27 devstack / tempest jobs  https://review.openstack.org/54710719:37
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove fedora-26 devstack / tempest testing  https://review.openstack.org/54710819:37
*** VW_ has quit IRC19:37
*** olaph1 has joined #openstack-infra19:38
*** VW has joined #openstack-infra19:38
corvusclarkb, fungi, mordred: if we have to install zuul/nodepool in venvs for pip10 we'll just need to have our deployment tooling symlink all their entrypoint scripts into /usr/local/bin19:38
clarkbI guess the othe roption is to have distros use setuptools and not use distutils19:39
clarkbbut thats probably not going to happen on xenial and may be too late for whatever b is19:39
*** olaph has quit IRC19:39
fungicorvus: yeah, that sounds reasonable (and is basically the approach i take when pip installing things on my personal systems)19:43
fungiinstall into a virtualenv, symlink into the venv's bindir from somewhere in the default command search path19:43
*** yamamoto has joined #openstack-infra19:44
*** agopi has quit IRC19:45
*** dhajare has quit IRC19:47
mnaserclarkb: apparently they're already mirrored here https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/templates/mirror.vhost.erb#L166-L16919:50
mnaseror cached rather19:50
AJaegerinfra-root, could I get another review on this bindep change to help get rid of jenkins/data, please? https://review.openstack.org/#/c/54314219:50
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove fedora-26 devstack / tempest testing  https://review.openstack.org/54710819:50
AJaegerpabelanger: with https://review.openstack.org/#/c/546848/ failing, are we really ready to make fedora-27 voting yet?19:51
*** yamamoto has quit IRC19:52
pabelangerAJaeger: no, that need to be rebased into non-voting19:52
pabelangerAJaeger: dmsimard is working on fixing19:52
AJaegerthanks, dmsimard19:52
AJaegerpabelanger: ok19:52
*** rossella_s has quit IRC19:52
dmsimardI'll look now, I have held nodes from the last failure -- not sure yet what the issue is19:52
*** rossella_s has joined #openstack-infra19:53
dmsimardwelp, it hasn't even been held that long: There were 48439 failed login attempts since the last successful login.19:54
pabelangermnaser: yah, that was added a while back. Is there an issue?19:56
mnaserpabelanger: im not sure i understand the failure fully, installs inside osa containers "debhelper : Depends: dh-strip-nondeterminism (>= 0.028~) but it is not going to be installed"19:57
*** mriedem has quit IRC19:57
*** mriedem has joined #openstack-infra19:57
mnaserif we are using cached images and download an older image and install packages with new mirrors, it should just update without an issue19:57
mnaserif we download a newer image and have old repos might be the cause i can imagine19:57
*** olaph1 has quit IRC19:58
fungihttp://mirror.dfw.rax.openstack.org/ubuntu/timestamp.txt suggests it updated ~1.5 hours ago19:58
*** olaph has joined #openstack-infra19:59
pabelangermnaser: have a log I can look at?20:00
fungiyeah, that's waaay newer than what packages.ubuntu.com claims is available on xenial at least (even xenial-backports)20:00
fungilooks like bionic20:01
fungihttps://packages.ubuntu.com/bionic/debhelper20:01
*** bmjen has quit IRC20:01
fungidepends dh-strip-nondeterminism (>= 0.028~)20:01
fungimnaser: are you trying to use bionic images?20:01
*** slaweq has joined #openstack-infra20:01
*** Hunner has quit IRC20:01
*** Hunner has joined #openstack-infra20:02
*** Hunner has quit IRC20:02
*** Hunner has joined #openstack-infra20:02
mnaseri don't think so, but maybe this was a recent osa bug or change20:02
fungiyeah, that looks like it's installing or upgrading to the debhelper in bionic20:02
fungibut a log would help20:02
*** yamahata has joined #openstack-infra20:03
mnaserfungi: http://logs.openstack.org/44/545844/1/gate/openstack-ansible-deploy-ceph-ubuntu-xenial/0095672/logs/ara/result/d268817e-7b12-41f9-962e-e20f5bd00ec7/20:03
*** slaweq_ has joined #openstack-infra20:03
mnaseri can point you to the direct log but i didnt see much more interesting info (and its cleaner in ara)20:03
mnaserlet me see what image was being used20:04
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Added add new track by summit endpoint  https://review.openstack.org/54712520:04
*** bmjen has joined #openstack-infra20:05
openstackgerritMerged openstack-infra/bindep master: Move jenkins/data/bindep-fallback.txt  https://review.openstack.org/54314220:05
*** slaweq has quit IRC20:06
mnaserok so job is running on xenial according to zuul-info20:06
AJaegerconfig-core, time to say good-bye to jenkins/data - who wants to do the honor? https://review.openstack.org/#/c/54314020:06
pabelangerfungi: mnaser: we do have bionic repos now on AFS20:07
mnaseri'm trying to make sense of whats going on exactly with this failure20:07
*** slaweq_ has quit IRC20:07
*** rossella_s has quit IRC20:08
*** rossella_s has joined #openstack-infra20:10
dmsimardpabelanger: I've reproduced the f26 issue, not sure what's causing it yet though20:11
pabelangerxenial|main|amd64: debhelper 9.20160115ubuntu320:13
pabelangermnaser: fungi: ^version of debhelper in reprepro20:13
pabelangerlooks to be insync with packages.ubuntu.com20:14
mnaserpabelanger: what about dh-strip-nondeterminism, the dependency its failing on?20:14
pabelangerxenial|main|amd64: dh-strip-nondeterminism 0.015-120:14
*** bmjen has quit IRC20:15
mnaserum20:15
*** bmjen has joined #openstack-infra20:15
pabelangerwhich is same as https://packages.ubuntu.com/xenial/dh-strip-nondeterminism20:15
openstackgerritMerged openstack-infra/openstackid-resources master: Added add new track by summit endpoint  https://review.openstack.org/54712520:15
mnaser"debhelper : Depends: dh-strip-nondeterminism (>= 0.028~) but it is not going to be installed"20:15
mnaseri wodner where it's getting that from20:15
pabelangermnaser: so, maybe repo files are not setup properly and using bionic now?20:15
fungimnaser: that's trying to resolve the dependencies for the version of debhelper in bionic20:16
jlvillalclarkb, Made the suggested changes and it is passing the gate now: puppet-gerritbot: https://review.openstack.org/#/c/546700/20:16
jlvillalAs an FYI20:16
mnaserfungi: ok thats interesting20:17
persiaDoes something automatically pull from backports?  Usual Ubuntu developer practice is to backport things like devscripts, debhelper, etc. into Ubuntu Backports so folk can run the current release whilst working on the next release.20:17
* persia remembers seeing backports not being commented out in some mirror scripts, but doesn't know if it ever gets used20:17
fungipersia: in this case the dep doesn't match the version in either xenial or xenial-backports, but matches the version in bionic20:18
fungii expect there's an incorrect sources.list20:18
*** slaweq has joined #openstack-infra20:22
clarkbjlvillal: one small thing20:22
jlvillalclarkb, looking20:23
*** slaweq_ has joined #openstack-infra20:24
jlvillalclarkb, I can change it: I did see this: https://github.com/openstack-infra/puppet-statusbot/blob/master/manifests/init.pp#L61-L6620:25
jlvillalclarkb, I'm no expert20:25
jlvillalclarkb, What you say makes sense to me as it is a chain of subscribes20:26
clarkbjlvillal: ah I think status bot might be wrong there too then20:26
clarkbbasically the git repo doesn't matter so much there as does the install itself so its better to explicitly call that out I think20:26
*** dprince has quit IRC20:27
jlvillalclarkb, Sounds good. Updating now.20:27
mtreinishpabelanger, clarkb, fungi: if you get a sec https://review.openstack.org/534953 is a super easy review20:28
openstackgerritJohn L. Villalovos proposed openstack-infra/puppet-gerritbot master: Change gerritbot to install from git  https://review.openstack.org/54670020:28
jlvillalclarkb, ^^ Hopefully Zuul likes it :)20:29
*** slaweq_ has quit IRC20:29
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: Refactor status functions, add web endpoints, allow params  https://review.openstack.org/53630120:31
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands  https://review.openstack.org/53630320:31
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: webapp: add optional admin endpoint  https://review.openstack.org/53631920:32
openstackgerritDavid Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Attempt to rescue lost SSH connection when restarting iptables  https://review.openstack.org/54713020:35
dmsimardpabelanger: attempt at fixing, dunno if it'll work ^20:35
dmsimardI don't understand the specificity to Fedora20:36
mnaserfungi: i'm a bit confused at how the broken sources.list can cause this20:36
*** rossella_s has quit IRC20:36
dmsimardbut even without ansible involved, if I just restart iptables over ssh, I lose connectivity20:36
mnaserif the sources.list pointed to bionic, it would pull debhelper from bionic and install that dependency.  if it pointed to xenial, well, that would never be an issue in the first place20:37
mnaserdmsimard: order of iptables rule might have the DROP rule before accept?20:37
odyssey4mepabelanger mnaser interestingly, the sources are right: http://logs.openstack.org/73/546773/1/check/openstack-ansible-functional-ubuntu-xenial/e232748/logs/etc/openstack/openstack1/apt/sources.list.txt.gz20:37
mnaserso as rules are being reinstalled, DROP rules come in and block everything, then ACCEPT a few seconds later20:37
odyssey4methe only other extra is http://logs.openstack.org/73/546773/1/check/openstack-ansible-functional-ubuntu-xenial/e232748/logs/etc/openstack/openstack1/apt/sources.list.d/uca.list.txt.gz20:37
*** SumitNaiksatam has quit IRC20:38
dmsimardmnaser: I don't know... rules are here: http://logs.openstack.org/15/536615/2/check/openstack-infra-multinode-integration-fedora-26/139d9d2/ara/result/c36376e0-25a2-467d-870d-eb09679d09d1/20:38
dmsimardmnaser: default policy is accept20:38
dmsimardmnaser: someone mentioned "kernel" here but I doubt that would be it: https://github.com/ansible/ansible/issues/2003320:38
pabelangerdmsimard: I usually use reload on ubuntu, doesn't fedora have the same?20:38
*** rossella_s has joined #openstack-infra20:39
dmsimardpabelanger: reload has the same behavior, breaks connectivity as well20:39
odyssey4mepabelanger mnaser aha, it's UCA that's broken it20:39
dmsimardbrb, picking up kid from school20:39
pabelangerdmsimard: ouch20:39
mnaserthats weird.. im not sure why it wouldn't come back, looks like it drops all connections20:39
mnaserodyssey4me: oh?20:39
odyssey4memnaser pabelanger http://paste.openstack.org/show/682458/20:40
odyssey4mesee the dep specification in line 1220:40
openstackgerritJohn L. Villalovos proposed openstack-infra/puppet-statusbot master: Have the 'statusbot' service subscribe to the pip install  https://review.openstack.org/54713220:40
jlvillalclarkb, ^^ Made suggested change to puppet-statusbot too20:40
clarkbre specificity to fedora is it using firewalld?20:40
mnaserodyssey4me: guess we can go back to #openstack-ansible20:40
clarkbbecause firewalld is weird and does break things in unexecpted ways20:40
fungimnaser: i was theorizing that an image with xenial packages but misconfigured with a sources.list for bionic would create this effect, but forgot that uca was yet another repository we hadn't considered checking yet20:41
mnaserfungi: yeah uca is the culprit here :(20:42
openstackgerritMatthieu Huin proposed openstack-infra/zuul-jobs master: role: Inject public keys in case of failure  https://review.openstack.org/53580320:42
odyssey4mefungi mnaser and the package UCA is providing for that dep is Version: 0.040-1~cloud2 - clearly a later version, so something is hinky20:42
openstackgerritMerged openstack-infra/release-tools master: Add StoryBoard filter and tag tools  https://review.openstack.org/53480820:43
dmsimardclarkb: we have iptables in the fedora DIB images, not firewalld20:43
pabelangerlet me check reprepro for UCA20:43
AJaegerteam, it's time to say good-bye to jenkins/data - could I get another review on https://review.openstack.org/#/c/543140 , please?20:43
mtreinishclarkb: thanks20:43
clarkbdmsimard: right but is firewalld applying them?20:43
fungiour uca mirror is up to date too at least, looks like: http://mirror.bhs1.ovh.openstack.org/ubuntu-cloud-archive/timestamp.txt20:43
dmsimardclarkb: don't think so, but I'll double check.20:44
pabelangerUCA mirror in reprepro is working20:44
fungiyeah, it's current as of 45 minutes ago20:44
mnaserso to sum it up20:45
mnaserdebhelper is being pulled from uca which depends on dh-strip-nondeterminism (>= 0.028~), dh-strip-nondeterminism_0.040-1~cloud2 exists in uca but apt doesn't like it (i guess)20:45
fungiyep, just confirmed looking in http://mirror.bhs1.ovh.openstack.org/ubuntu-cloud-archive/dists/xenial-updates/queens/main/binary-amd64/Packages20:46
clarkbAJaeger: where is /usr/local/bindep-fallback/bindep-fallback.txt created?20:46
pabelangeraptmethod got 'http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/d/debhelper/debhelper_11.1.4ubuntu1~cloud0_all.deb'20:46
pabelanger2018-02-22T16:00:01,559430775+000020:46
pabelangerthat is when we updated20:46
fungimnaser: so, yes, queens uca seems to be at fault i guess20:46
AJaegerclarkb: https://review.openstack.org/#/c/543139/6/nodepool/elements/nodepool-base/post-install.d/89-data-files20:46
mnaserfungi: cool, thanks.. time to figure out why20:46
mnaseror how to fix20:46
fungimnaser: looks like maybe they build queens uca against bionic package revs instead of xenial20:46
AJaegerclarkb: and https://review.openstack.org/#/c/543142/ shows that it's in our images20:47
clarkbAJaeger: ty20:47
clarkbAJaeger: approved20:47
AJaegerthanks, clarkb !20:47
*** yamamoto has joined #openstack-infra20:48
*** rossella_s has quit IRC20:48
fungimnaser: though the packages file for xenial queens uca claims to include dh-strip-nondeterminism 0.040-1~cloud220:48
fungiso interesting that it's not being selected20:49
AJaegerconfig-core, small cleanup for openstack-zuul-job for review, please - https://review.openstack.org/#/c/546307/20:49
*** aeng has joined #openstack-infra20:50
*** rossella_s has joined #openstack-infra20:51
fungijamespage: do you happen to know who would be able to tell us why uca's xenial-updates/queens is bringing in a debhelper version from bionic instead of xenial? seems like dh is something you would not want to update in uca, but that could just be my lack of imagination too20:52
dmsimardclarkb: yeah firewalld isn't installed, nothing weird in the systemd journal for iptables either20:53
clarkbhuh20:54
dmsimardonly fedora is exhibiting that behavior and as far as I know, it wasn't always this way20:55
*** yamamoto has quit IRC20:55
pabelangerTheJulia: mind adding https://review.openstack.org/547119/ to your review pipeline. It moves bifrost to fedora-27, we'd like to remove fedora-26 nodes.20:56
dmsimardin that upstream ansible issue, someone said on RAX it disconnected him and on Linode it didn't, I'm not sure what to make of that20:56
*** camunoz has joined #openstack-infra20:56
*** vivsoni_ has joined #openstack-infra20:57
*** vivsoni has quit IRC20:57
*** efoley has quit IRC20:58
openstackgerritMerged openstack-infra/system-config master: Add logstash worker to services table  https://review.openstack.org/53495320:58
AJaegerinfra-root, do you want to restart zuul some time today? It's getting slower IMHO ...20:59
pabelangermemory looks good still21:00
clarkbthe grafana status page shows it is getting bursty21:00
clarkbwhich appears to be related to the executor governors21:00
fungii need to disappear for a few minutes, but will return soonish21:00
clarkbload average and memory spikes which then result in not taking jobs21:00
pabelangerreloads are up, around 20s21:01
mnaserfungi: i will never understand distro versioning and never will try doing that anymore :D21:01
pabelangerclarkb: no, I think we are spending a lot of time reloading in scheduler, which is starving CPU and preventing new builds from starting21:02
clarkbpabelanger: ah that could be too though we are definitely having longish periods where executors stop accepting jobs21:02
pabelangerhttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=389&rra_id=all21:02
pabelangeryou can see we are using more CPU21:03
pabelangernot sure why however21:03
clarkb~5 minutes at a time21:03
AJaegerafter my last approval it took 7 minutes until the change was queued in gate21:03
dmsimardpabelanger: actually wait_for_connection is a thing now btw :p http://docs.ansible.com/ansible/latest/wait_for_connection_module.html21:03
dmsimardpabelanger: but yeah, +1, I'll use that instead21:04
pabelangerdmsimard: neat, TIL21:04
dmsimardpabelanger: yeah it's more convenient when testing for ssh connectivity21:04
clarkblooks like cpu incrase started about 1500UTC?21:05
AJaegerconfig-core, thanks for the reviews - we still have some backlog on the projects, reviews welcome!21:05
* AJaeger waves good night21:05
AJaegerclarkb: and memory as well21:05
AJaegerclarkb: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all21:05
clarkboh ya21:06
clarkbso some config change maybe?21:06
AJaegerno idea - I glanced at zuul.o.o at that time but didn't see anything obvious ;/21:06
*** r-daneel has quit IRC21:06
*** r-daneel has joined #openstack-infra21:06
*** mgoddard_ has joined #openstack-infra21:07
*** hashar has quit IRC21:07
dmsimardI think it'd be useful to install some diagnostic tools by default, namely iotop and sysstat (for iostat)21:08
AJaegerconfig-core, some change that would be nice to have in are https://review.openstack.org/546695 https://review.openstack.org/546432 https://review.openstack.org/545726 and https://review.openstack.org/53763421:08
* AJaeger really leaves for today21:08
pabelangerclarkb: AJaeger: that's around the time when fungi approved the change to clear our zuul errors for OSA project21:08
dmsimardpabelanger: wasn't that yesterday ?21:08
*** slaweq_ has joined #openstack-infra21:09
AJaegerpabelanger: the change was approving another new project creation which then triggered the reconfig21:09
pabelangerdmsimard: there was some this morning too21:09
pabelangeryah, that21:09
dmsimardhmm21:09
pabelangerso, zuul did full reconfiguration21:09
*** eharney has quit IRC21:10
dmsimardThere's been quite a few exceptions if you look at just today's logs http://paste.openstack.org/raw/682503/21:11
jamespagefungi: in order to ease the overhead of backporting >300 source packages back to xenial, we've backported a newer debhelper so that any new bits landing in bionic which require debhelper >= 10 don't require us to hold a patch to downgrade it21:11
jamespagefungi: 3-4 pkgs vs 30-40 patches if you see what I mean21:12
odyssey4mejamespage :) evening21:12
dmsimardwow that's a nasty exception21:12
*** r-daneel_ has joined #openstack-infra21:13
dmsimardexception from the poll loop error: http://paste.openstack.org/raw/682505/21:13
dmsimardThat's almost java levels of stack trace21:13
*** r-daneel has quit IRC21:13
*** r-daneel_ is now known as r-daneel21:13
*** slaweq_ has quit IRC21:14
odyssey4mewe hit some trouble with that in OSA, although we discovered that we didn't really know why debhelper was being installed in the first place so we have patches up to remove it... and we're not entirely sure why the installs didn't work... we add UCA/queens on xenial, we try to install debhelper, kerblooey21:14
openstackgerritMerged openstack-infra/project-config master: Move jenkins/data/bindep-fallback.txt (2/2)  https://review.openstack.org/54314021:14
clarkbdmsimard: the unknown job error is causing the other error (no job means no attribute)21:17
clarkbdmsimard: I think that may imply we still have configs that are not quite correct?21:17
clarkband we are trying to run a job that doesn't exist?21:17
clarkbit would be nice if unknown job error had the job name too21:18
*** dprince has joined #openstack-infra21:18
* clarkb goes to poke at gear to see if that is doable21:18
dmsimardI'm not sure, there's a lot of other errors as well21:18
dmsimardNot familiar enough to tell if any of those are "normal", recoverable or fatal21:18
jamespagehi odyssey4me21:19
dmsimardI've seen this one before and it's happening a lot: "ERROR zuul.Scheduler: Unable to process autohold for None" I formally created a story for it so we don't forget21:19
dmsimardThere's also "ERROR zuul.Scheduler: Exception reporting runtime stats", "ERROR zuul.nodepool: Error unlocking node" and "ERROR zuul.nodepool: Node <Node 0002678661 ['primary']:fedora-26> is not locked"21:20
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Remove jenkins-slave element from DIB images  https://review.openstack.org/51448521:20
pabelangerAJaeger: clarkb: ^now we can remove the jenkins-slave element from nodes21:20
*** iyamahat has joined #openstack-infra21:22
openstackgerritClark Boylan proposed openstack-infra/gear master: Include job name in UnknownJobError  https://review.openstack.org/54714321:23
clarkbsomething like ^ may help in debugging this21:23
pabelangerianw: https://review.openstack.org/547107/ brings online fedora-27 for tempest / devstack21:23
jamespageodyssey4me: hmmm did something break? we pushed the rc's from last week through to -updates PM today21:24
jamespagethat may have had a number of these toolchain type things with it21:24
odyssey4mejamespage ok, so something did change today21:24
*** rossella_s has quit IRC21:24
jamespageit was quite a large number of packages so might not have all go in during the same sync cycle, which would create some installability issues for an hour or so21:24
odyssey4meI'm tempted to fire up a test now to see if the issue happens outside of infra.21:25
clarkbreading the code more though its getting a status_res packet and then not finding the job associated with that packet so that it can update the job21:25
dmsimardclarkb: do we want "handle" or "job" ?21:25
clarkbdmsimard: job not existing is why we error, handle is the key we are looking for21:25
jamespageodyssey4me: issue is with debhelper installability right?21:25
dmsimardclarkb: makes sense21:25
clarkbthinking about why this may happen maybe the client is cancelling the jobs but that state hasn't made it all the way through21:25
clarkbI still think the name would help us understand why it happens21:25
odyssey4mejamespage mind if we switch into #openstack-ansible? I'd rather not interfer with the other ocnversation going on here.21:26
jamespageodyssey4me: ok21:26
ianwpabelanger: tbh, i'd prefer if we could get the stack @ https://review.openstack.org/#/c/540704/ in21:27
*** jtomasek has quit IRC21:27
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Finish golang job removal  https://review.openstack.org/54630721:28
*** rossella_s has joined #openstack-infra21:29
openstackgerritDavid Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Attempt to rescue lost SSH connection when restarting iptables  https://review.openstack.org/54713021:30
pabelangerianw: okay, I've given my +121:31
pabelanger:)21:31
clarkbdmsimard: thinking about restarting iptables if using iptables persistent I wonder if a stop iptables implies a drop all connections on fedora? maybwe can just start the iptables and avoid that?21:32
pabelangerianw: can you comment on the parts you have issue with in https://review.openstack.org/547107/ ? some jobs are using legacy-fedora-26, so we'll need to land legacy-fedora-27 for the moment21:32
ianwthanks, just need some eyes on the non-trivial stuff below that21:32
pabelangerit's possible that patch will just turn into adding nodesets21:32
dmsimardclarkb: we flush iptables rules on purpose to check if restarting iptables loads them back as a mean of testing that the persistence works -- I wonder why the other distros don't have that issue21:33
*** kgiusti has left #openstack-infra21:34
clarkbI would look at the unit file and any scripts it is calling21:34
clarkbmy guess is its doing something smart21:34
*** mgoddard_ has quit IRC21:35
clarkbI doubt its the kernel since they are pretty steadfast about no user noticeable breakages but I guess its possible the kernel is doing something21:35
ianwpabelanger: if f27 runs devstack with python3, the updown job doesn't serve much purpose any more.  it's really not worth running it with py2 on f27 i wouldn't think21:36
dmsimardyeah, I'll look after we figure out what's going on with Zuul :p21:36
dmsimardso I see that the full reconfigure appears to have been done at 14:39, starting with 2018-02-22 14:39:31,197 INFO zuul.TenantParser: Loading previously parsed configuration from openstack-infra/project-config21:37
dmsimardthat seems to correlate with the higher cpu usage21:37
odyssey4mefungi not sure if you're available, but it's plausible that the last UCA repo update happened in the middle of the UCA being updated... are there logs for when UCA last updated?21:37
clarkbdmsimard: and with the memory use jump21:37
*** camunoz has quit IRC21:37
odyssey4meor pabelanger ^ not sure who can help21:37
*** pcaruana has quit IRC21:37
clarkbodyssey4me: yes navigate to the mirror with your browser and there will be a timestamp file you can check21:38
pabelangerianw: k, is there somebody else that could confirm if we don't need it. If so, then lets delete21:38
pabelangerodyssey4me: I can look21:38
pabelangerbut it seems to be fine21:38
pabelanger1 sec21:38
ianwpabelanger: well i just put it in as a smoke test, not sure anyone but me ever cared :)21:38
pabelangerit should be running again in 22mins however21:38
pabelangerianw: ah, okay. then lets remove it21:39
ianwjust make sure the other job is doing USE_PYTHON321:39
pabelangerodyssey4me: have time to wait 20mins?21:40
odyssey4meclarkb pabelanger ah you mean http://mirror.bhs1.ovh.openstack.org/ubuntu/timestamp.txt21:40
clarkbodyssey4me: for uca it iwll be a different path, but yes21:40
odyssey4meta, thanks - happy to help urselves do more diggin :)21:40
clarkbhttp://mirror.bhs1.ovh.openstack.org/ubuntu-cloud-archive/timestamp.txt21:40
odyssey4metimestamps are UTC?21:40
*** iyamahat_ has joined #openstack-infra21:40
pabelangerya21:40
odyssey4mek thx21:41
pabelangerthey update every 2hours21:41
*** iyamahat has quit IRC21:41
odyssey4meis that timestamp a sfile that comes from the source, or placed there by the sync process when the sync is done?21:41
dmsimardclarkb: it looks like there's two reconfigure in a row..21:42
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Remove legacy-devstack-dsvm-py36-updown-fedora-26  https://review.openstack.org/54714721:44
dmsimardSeeing a lot of "WARNING zuul.GerritConnection: Unable to get change for [...]"21:44
*** slaweq_ has joined #openstack-infra21:46
*** olaph1 has joined #openstack-infra21:48
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add fedora-27 devstack / tempest jobs  https://review.openstack.org/54710721:48
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove fedora-26 devstack / tempest testing  https://review.openstack.org/54710821:48
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove devstack-dsvm-py36-updown-fedora-26 job  https://review.openstack.org/54715221:48
*** dave-mccowan has quit IRC21:49
*** olaph has quit IRC21:49
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Switch devstack / tempest testing to fedora-27  https://review.openstack.org/54710921:50
pabelangerianw: ^okay, removed the updown job21:50
clarkbdmsimard: whats the ... in there?21:50
*** slaweq_ has quit IRC21:50
clarkbdmsimard: looking at the code for the gerrit connection I expect its just content we are effectively ignoring because it isn't a change or a ref21:51
odyssey4mefungi clarkb pabelanger we got to the bottom of the package install issue - briefly: jamespage pushed a bunch of updates to the PPA for UCA, which then somewhere while it wasn't done got pulled into UCA, which then also got pulled in by the infra mirror process21:51
dmsimardclarkb: looks like this: http://paste.openstack.org/raw/682561/21:51
odyssey4mesince then, further syncs have been done and the issue is resolved21:52
clarkbdmsimard: ya those are I think just noise, we are effectively ignoring the ref-replicated events this way21:52
pabelangerodyssey4me: you can see how we generate the timestamp: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/reprepro/reprepro-mirror-update.sh21:52
clarkbsomething we might want to clean up in the logs so it doenst create distractions21:52
pabelangerodyssey4me: I guess we could expand it based on which phase it is on21:52
*** yamamoto has joined #openstack-infra21:53
dmsimardclarkb: there's a LOT of them, 19k in today's log21:53
*** dave-mcc_ has joined #openstack-infra21:53
dmsimardI'll create a story, I've created a few so far: https://storyboard.openstack.org/#!/project/67921:53
pabelangerodyssey4me: good to know21:53
odyssey4methanks, as always, for the help and support :)21:53
clarkbdmsimard: ya anytime a ref gets replicated by gerrit we'll get one (so x 8 for git mirror + 1 for github)21:53
clarkbdmsimard: 19000k/9 refs roughly I think21:54
clarkber no k there21:54
pabelangeryah, those warnings have been around for a while :)21:55
*** dhill__ has quit IRC21:56
*** yamamoto has quit IRC21:57
evrardjphello again21:59
pabelangerpushing 30mins since zuul has launched any new builds. Guess we have a lot of zuul.yaml files in pipelines right now21:59
evrardjpis there someone that can add the openstack-ansible-core group into the group https://review.openstack.org/#/admin/groups/1872,members (see also the review https://review.openstack.org/#/c/546577/)21:59
clarkbpabelanger: or possibly the gaerman issues that dmsimard found are causing us to not launch jobs22:00
clarkbis gearman running?22:00
dmsimardif you look at the web status22:00
dmsimardsome changes' jobs have all finished running but they're just staying there22:00
dmsimardhttps://i.imgur.com/Gl9M1JB.png22:01
clarkbdmsimard: thats "normal" if you look at the queue counts at the top of the page22:01
pabelangerdmsimard: right, zuul hasn't processed them yet22:01
clarkb624 results to process which are probably your changes22:01
pabelangerand there it goes now22:01
dmsimardah, okay yeah it just cleared it22:01
clarkblooks like ya it just went22:01
*** yamamoto has joined #openstack-infra22:01
pabelangerI think we just have a lot of zuul.yaml files right now22:01
pabelangerand will reloads taking 20seconds, taking longer22:02
dmsimardseeing fingergw error22:03
dmsimardhttp://paste.openstack.org/raw/682573/22:03
*** r-daneel_ has joined #openstack-infra22:05
* fungi is back again, catching up22:05
dmsimardI'm tailing logs and I'm not seeing anything obvious22:06
*** r-daneel has quit IRC22:06
*** r-daneel_ is now known as r-daneel22:06
clarkbI think we may just be seeing a further regression from the memory improvements?22:06
clarkbzuul is operating just more slowly22:06
pabelangerit was working great, up until this morning memory increase22:07
dmsimardexecutors are very spiky22:07
clarkbfungi: seems to be correlated to fixing the osa stuff?22:07
pabelangerdmsimard: starting builds graph is the once to watch, if that is flat, no new jobs are starting22:07
pabelangers/once/one22:08
dmsimardpabelanger: yeah I realize that but executors *are* picking up new builds22:08
TheJuliapabelanger: sure, can it wait a day or two? Looks like it didn't take to fedora27 very well. Likely a fix, but I won't have the requisite brain cells for at least another day.22:08
*** danpawlik has quit IRC22:08
pabelangerdmsimard: yah, once builds are started, executors run them without issue22:08
pabelangerTheJulia: Sure, just wanted to make sure you were aware22:09
*** jcoufal has quit IRC22:09
dmsimardnodepool is showing over 100 nodes ready, at least according to http://grafana.openstack.org/dashboard/db/nodepool22:10
dmsimardso zuul is just not picking up the nodes from nodepool to assign them to builds or something ?22:10
*** dhill_ has joined #openstack-infra22:10
TheJuliapabelanger: awesome, thanks!22:10
pabelangerdmsimard: zuul is busying doing dynamic reloads, and events are backing up in queue. see top left of zuul.o.o. When reloading, no new events a processed22:11
pabelangerso, nodepool will finish bringing nodes online, then wait to move then in-use by zuul22:12
dmsimardpabelanger: thus the bursts of executors picking up new jobs we've been seeing22:12
pabelangeryah22:12
*** armaan has quit IRC22:13
fungiclarkb: if what i did this morning (approving another new project creation change) is what triggered the current performance regression then we have some reason to be concerned, i suppose22:13
pabelangerreloads are taking 20 seconds now, up from 12 seconds yesterday. not sure why22:13
pabelangerfungi: clarkb: isn't there a way we can dump current layouts in memory and see if we've leaked something?22:16
pabelangerI'm not sure how that worked22:16
clarkbI think it required doing the repl socket22:17
clarkbnot sure if that is still enabled22:17
*** agopi has joined #openstack-infra22:18
*** slaweq_ has joined #openstack-infra22:18
dmsimardSo I think this isn't a perfect query but it shows recent changes for zuul files https://review.openstack.org/#/q/file:%22%255E(%255C.zuul.d%257Czuul.d%257C%255C.zuul.yaml%257Czuul.yaml).*%2524%2222:19
*** rlandy|ruck is now known as rlandy|biab22:21
*** slaweq has quit IRC22:22
*** slaweq has joined #openstack-infra22:22
*** slaweq_ has quit IRC22:23
dmsimardIn zuul-web, there's a lot of these: "Submitting job zuul:status_get with data {'tenant': 'openstack'}" and sometimes they don't come back immediately -- I suppose that's the /status endpoint.. is the apache caching working ?22:23
dmsimardI'm also not sure why that is but in the apache error log, there's this error but for like... every repo "Not a git repository: '/var/lib/zuul/git/openstack/neutron-vpnaas'"22:25
dmsimard(/var/lib/zuul/git is indeed empty on zuul01)22:25
dmsimardI have no idea where I'm going, just thinking out loud and hoping it will make someone knowledgeable think of something :D22:26
clarkbI think that may be old jobs looking for zuul refs?22:26
clarkbnot sure22:27
*** slaweq has quit IRC22:27
smcginnisclarkb: Do I remember right that one of the lunch presos next week will be about zuulv3?22:29
smcginnisclarkb: Or is that its own session?22:29
fungismcginnis: yes, tuesday (infra/qa joint update, much of which will be zuul related on both fronts)22:30
smcginnisfungi: Great, thanks22:30
dmsimardNever realized there was so many queries for status/status.json: http://paste.openstack.org/raw/682607/22:31
*** esberglu has quit IRC22:32
*** dprince has quit IRC22:33
*** kjackal has quit IRC22:34
*** tonyb has quit IRC22:35
*** tonyb has joined #openstack-infra22:36
dmsimardThere's an IP in particular that's hammering status.json, it's owned by RAX but doesn't seem to be one of our servers22:37
*** rcernin has joined #openstack-infra22:38
dmsimard¯\_(ツ)_/¯22:39
dmsimardI have no idea22:39
*** sshnaidm has joined #openstack-infra22:40
*** sshnaidm is now known as sshnaidm|off22:40
*** tosky has joined #openstack-infra22:41
*** shoogz has quit IRC22:44
fungiprobably someone's personal vm22:44
*** dave-mcc_ has quit IRC22:44
fungimaybe running a command-line status display22:44
mnasersneak in a quick rewrite rule for their ip only to show jobs saying 'pls-contact-infra'22:47
*** Goneri has quit IRC22:49
*** lpetrut has quit IRC22:50
dmsimardthey're getting 404's so I don't suppose it's hurting us too much22:58
openstackgerritDavid Moreau Simard proposed openstack-infra/zuul master: Add debug information when failing to autohold  https://review.openstack.org/54716023:00
clarkbis corvus around today? wondering if we shouldn't restart the server to see if reconfig time drops back down to 12s but worried that might destroy debugging ability23:01
corvusclarkb: i'm here23:01
clarkboh hi23:01
corvuswhat's up?23:01
clarkbcorvus: tl;dr is reconfigures are apparently up to 20s now and we're seeing higher cpu use on zuul.o.o since ~1500UTC which also came along with a jump in memory use23:02
clarkbthat time roughly correlates to when fungi fixed the osa repo23:02
corvusor rather, the reconfiguration after it was fixed i guess?23:02
clarkbas for user impacts zuul "feels" slow. The status graphs show it as being far more bursty than it was before23:02
clarkbcorvus: ya23:02
*** yamamoto has quit IRC23:02
corvushave we added any new projects other than osa since the last restart?23:03
* clarkb checks for github emails23:03
dmsimardcorvus: I created a few stories about the different errors/exceptions I've been seeing which might or might not be related: https://storyboard.openstack.org/#!/project/67923:03
clarkbyes tatu-dashboard and possibly osel23:03
corvusdid those have memory/cpu bumps?23:04
*** VW has quit IRC23:05
dmsimardneed to step away for now, I'll catch up later23:05
* clarkb does timestamp maths23:05
openstackgerritIan Wienand proposed openstack/diskimage-builder master: GPT partitioning support  https://review.openstack.org/53349023:05
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Fail if two elements provide the same thing  https://review.openstack.org/53940123:05
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Add block-device defaults  https://review.openstack.org/53937523:05
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Choose appropriate bootloader for block-device  https://review.openstack.org/53973123:05
openstackgerritIan Wienand proposed openstack/diskimage-builder master: arm64: use HWE kernel and fix console  https://review.openstack.org/54716123:05
clarkb9:15 was when I got the tatu-dashboard email23:05
clarkbwhich would be 1715UTC23:05
*** VW has joined #openstack-infra23:05
clarkbno memory jump for that one23:06
clarkbosel was at 1430UTC23:06
clarkbdoesn't look like a jump for that one either23:07
*** rossella_s has quit IRC23:07
corvusso it sounds like there may be something amiss with sucessful full reconfigurations after a failed full reconfiguration23:07
clarkbthe memory increase appears to start at 151023:07
*** rossella_s has joined #openstack-infra23:09
corvusi don't think there's much more we can do right now with the current running process; i think we should take that as a hypothesis, go ahead and restart, and, since we're not going to get to fixing it right away on account of travel, keep an eye out for any more data points that could confirm/disprove that for the next little bit.23:10
*** VW has quit IRC23:10
clarkbok I'm loading my ssh key now. Anyone else able to help with that?23:12
corvusclarkb: i'm deep into another problem but can switch if you need me23:13
clarkbok I'll ping if I run into trouble23:13
clarkbchecking there aren't any release jobs queued23:14
*** rlandy|biab is now known as rlandy|ruck23:14
clarkbya no release jobs so I'll go ahead and save queues and restart the scheduler and zuul-web23:14
fungiclarkb: corvus: approving the osel project creation is what i did to "fix" the osa repos (i figured approving another project addition would trigger a full reconfig, and that seems to have done the trick)23:14
clarkbah23:15
*** VW has joined #openstack-infra23:15
clarkbcorvus: ^ still want me to restart knowing that?23:15
clarkbI'm not sure it changes the hypothesis htough the timing is less tight23:16
corvusclarkb, fungi: then the 1430 approval probably tranlated into a 15.. something reload23:16
clarkbok moving ahead with restart then23:16
fungisounds likely since that part is apparently still puppet-driven23:16
corvuswfm23:16
corvus14:46 reload.  so there was a bit of a lag.23:17
clarkbI've told zuul to stop now just waiting for it to stop23:18
clarkb(its not an immediate response but the log did show it got the stop command on the socket)23:19
*** yee37935 has quit IRC23:20
corvusif it's in a slow reconfiguration queue run, it should stop when it gets to the end23:20
*** dbecker has quit IRC23:20
*** VW has quit IRC23:20
*** yee379 has joined #openstack-infra23:20
*** rossella_s has quit IRC23:21
*** rossella_s has joined #openstack-infra23:22
clarkblooks like it is currently processing queue items23:24
*** apetrich has quit IRC23:25
clarkblots of zuul.Pipeline.openstack.check: debug messages currnetly23:26
corvusthat seems strange23:26
corvusthere are no other pipelines it's processing23:28
clarkbya I've not seen any23:28
corvusclarkb: i suspect you should just kill it23:28
clarkbcorvus: sigint or sigkill?23:29
*** rossella_s has quit IRC23:29
clarkbI guess I can start with sigint23:29
fungii take it sigterm is what the initscript sent?23:29
corvusfungi: i think it uses the socket23:29
clarkbfungi: no it sent a command on the command socket23:29
fungioh23:29
fungikill uses sigterm by default, iirc23:30
clarkboh right not int23:30
clarkbI can start with term then23:30
clarkbhere goes23:30
*** rossella_s has joined #openstack-infra23:31
corvusit's stuck because it's not actually removing 547073,5 from the queue for some reason23:31
fungisigint is what happens when you ctrl-c in most foreground applications23:31
clarkbthat seems to have stopped the queue processing but not stopped the process23:31
clarkband now its gone23:31
clarkbstarting it up again now23:31
*** efried has quit IRC23:32
fungi547073,5 looks like just a bog-standard tempest patch23:33
*** slaweq has joined #openstack-infra23:33
corvusyeah, i don't know what's special there23:34
fungigit dependency on another tempest patch which is also typical-looking23:34
clarkbonce these cat jobs finish I'll reenqueue23:34
*** hongbin has quit IRC23:34
andreafclarkb corvus fungi is there any chance that jobs that were in progress will complete and still their logs uploaded?23:36
*** VW has joined #openstack-infra23:36
clarkbandreaf: if they finished before zuul released the nodesets yes otherwise its a race to how quickly nodepool can clean them up23:36
corvusthe executors will abort the jobs too once they lose the connection to the gearman server23:37
corvusit all happens very fast, so i'd say the chance is slim.23:37
andreafclarkb corvus ok np - can I recheck now?23:38
openstackgerritlifeless proposed openstack/gertty master: Don't lose sync requests that get bad responses  https://review.openstack.org/54716823:38
corvusandreaf: yep23:38
*** slaweq has quit IRC23:39
clarkbyes zuul is up and running now23:39
clarkbI am reequeuing too which may catch your thing23:39
*** tosky has quit IRC23:39
*** rosmaita has quit IRC23:40
andreafclarkb ok thanks23:40
*** VW has quit IRC23:41
lifelessfungi: ^ that might be a thing to merge and cut a release. It could explain a lot of 'wtf did my change go' moments :)23:41
lifelessfungi: there's also a bunch of other gertty stuff sitting there with multiple +1s etc :(23:42
*** VW has joined #openstack-infra23:42
corvuslifeless: thanks, i'll try to take a look soon23:42
lifelesscorvus: ta!23:42
lifelesscorvus: bug for it https://storyboard.openstack.org/#!/story/200157223:43
*** slaweq has joined #openstack-infra23:43
openstackgerritIan Wienand proposed openstack-infra/project-config master: Add arm64 nodes to launcher  https://review.openstack.org/54721623:44
fungilifeless: cool! i think i've probably hit that too but didn't manage to hunt down the cause23:45
*** abelur has quit IRC23:45
fungialso, while i am a sometimes reviewer/contributor to gertty, it's thoroughly a corvus production23:46
*** VW has quit IRC23:46
lifelessfungi: Its been a while, I just looked at the 'reviewers' list and grabbed the first name I remembered ;)23:46
fungi;)23:47
lifelessfungi: corvus: there may be other cases where syncs are lost - checkResponse for instance may want to look for 400's and 5xx's more generically. But this specific case wouldn't be fixed by that either, so..23:47
*** slaweq has quit IRC23:48
andreafcorvus how difficult it is to setup a local zuul and get it running with a nodepool running a couple of local virtualbox vms ?23:48
andreafcorvus it would be nice to be able to do some quicker local testing23:49
ianwarm64 getting soooo close.  wouldn't mind eyes on https://review.openstack.org/546834 which is a quick one to add linaro cloud to nodepool clouds conf ... one less hand-applied thing23:50
clarkbandreaf: there is a quick start guide in review somewhere23:50
ianwbuilds are working & uploading ... just a few tweaks to get console/config-drive working23:50
clarkbandreaf: but ya it shouldn;t be too hard let me try and find it23:51
*** rlandy|ruck is now known as rlandy|bbl23:51
clarkbandreaf: https://docs.openstack.org/infra/zuul/admin/quick-start.html23:53
clarkblooks like it doesn't go over nodepool but for that I'd configure nodepool to use some static nodes provided by whatever (virtualbox for example)23:53
corvusandreaf: if you do that, you can use the static driver in nodepool23:53
corvusandreaf, clarkb: you may prefer https://docs.openstack.org/infra/zuul/admin/zuul-from-scratch.html which is a more thorough step-by-step walkthrough for a quick example23:54
corvus(it does cover nodepool, but not static nodes (yet))23:54
*** caphrim007_ has joined #openstack-infra23:54
corvusandreaf: but if you just want to do quick local testing, why not just run ansible directly?23:55
*** caphrim007 has quit IRC23:57
andreafcorvus: that doesn't go e2e from the job definition - so I need to parse my job definition into an inventory file to supply ansible23:57
andreafcorvus merging the dicts and so23:57
corvusandreaf: you can grab an inventory file from a previously run job23:58
corvusmodify it to use your host, setup the git repos on your test node, then run the devstack playbook.  that should be about it.23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!