| *** rlandy has quit IRC | 00:03 | |
| *** eernst has quit IRC | 00:03 | |
| *** eernst has joined #openstack-infra | 00:03 | |
| clarkb | pabelanger: I've +2'd your fix in puppet-zuul. Not in a great spot to watch it go in though so haven't approved it | 00:06 |
|---|---|---|
| *** eernst_ has joined #openstack-infra | 00:06 | |
| *** eernst has quit IRC | 00:08 | |
| *** eernst_ has quit IRC | 00:08 | |
| *** eernst has joined #openstack-infra | 00:09 | |
| *** dingyichen has joined #openstack-infra | 00:09 | |
| *** eernst has quit IRC | 00:13 | |
| *** jamesmcarthur has quit IRC | 00:15 | |
| *** tosky has quit IRC | 00:18 | |
| *** jamesmcarthur has joined #openstack-infra | 00:18 | |
| *** jamesmcarthur has quit IRC | 00:20 | |
| *** rcernin has joined #openstack-infra | 00:21 | |
| *** jamesmcarthur has joined #openstack-infra | 00:23 | |
| mgagne | will need to stop nova-api on inap-mtl01 in hope to be able to fix the instances stuck in deleting state | 00:25 |
| *** jamesmcarthur has quit IRC | 00:26 | |
| *** jamesmcarthur has joined #openstack-infra | 00:27 | |
| *** jamesmcarthur has quit IRC | 00:27 | |
| *** olaph1 has joined #openstack-infra | 00:28 | |
| *** olaph has quit IRC | 00:28 | |
| *** odyssey4me has quit IRC | 00:34 | |
| *** odyssey4me has joined #openstack-infra | 00:35 | |
| mgagne | so instances are now slowly being deleted | 00:38 |
| *** jamesmcarthur has joined #openstack-infra | 00:39 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Update metadata for pypi https://review.openstack.org/553642 | 00:45 |
| *** jamesmcarthur has quit IRC | 00:48 | |
| *** harlowja has quit IRC | 00:49 | |
| mgagne | looks like situation is back to normal in inap-mtl01 | 00:54 |
| *** wolverineav has quit IRC | 00:54 | |
| *** salv-orlando has joined #openstack-infra | 00:55 | |
| *** hamzy__ has joined #openstack-infra | 00:57 | |
| *** salv-orlando has quit IRC | 01:00 | |
| *** david-lyle has joined #openstack-infra | 01:02 | |
| *** HeOS has quit IRC | 01:04 | |
| *** jamesmcarthur has joined #openstack-infra | 01:06 | |
| *** eernst has joined #openstack-infra | 01:14 | |
| *** dtruong_ has joined #openstack-infra | 01:14 | |
| *** eernst has quit IRC | 01:15 | |
| *** eernst has joined #openstack-infra | 01:16 | |
| *** wolverineav has joined #openstack-infra | 01:16 | |
| *** dtruong has quit IRC | 01:17 | |
| *** olaph has joined #openstack-infra | 01:21 | |
| *** wolverineav has quit IRC | 01:21 | |
| *** olaph1 has quit IRC | 01:21 | |
| *** wolverineav has joined #openstack-infra | 01:22 | |
| *** dizquierdo has quit IRC | 01:26 | |
| *** wolverineav has quit IRC | 01:26 | |
| *** jamesmcarthur has quit IRC | 01:28 | |
| *** eernst has quit IRC | 01:28 | |
| *** eernst has joined #openstack-infra | 01:30 | |
| *** eernst has quit IRC | 01:31 | |
| *** yamahata has quit IRC | 01:34 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: zk: use kazoo retry facilities https://review.openstack.org/535537 | 01:35 |
| *** iyamahat has quit IRC | 01:38 | |
| *** gongysh has joined #openstack-infra | 01:44 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for Cinder CapacityFilter scheduling failure bug 1756220 https://review.openstack.org/553648 | 01:45 |
| openstack | bug 1756220 in Cinder "CI: Large spike in scheduling failures with the CapacityFilter since March 13" [High,Confirmed] https://launchpad.net/bugs/1756220 | 01:45 |
| *** zerick_ is now known as zerick | 01:46 | |
| *** rosmaita has quit IRC | 01:46 | |
| *** Kevin_Zheng_ is now known as Kevin_Zheng | 01:50 | |
| *** salv-orlando has joined #openstack-infra | 01:56 | |
| dmsimard | OVH's BHS1 region is struggling :( http://grafana.openstack.org/dashboard/db/nodepool-ovh | 01:58 |
| *** felipemonteiro has joined #openstack-infra | 01:59 | |
| *** salv-orlando has quit IRC | 02:01 | |
| pabelanger | yah, have a request in #ovh to see what is going on, but no reply yet | 02:02 |
| pabelanger | we likely need to send a follow up email to confirm they know of the outage. Looks message queue related. | 02:02 |
| *** mikal_ has joined #openstack-infra | 02:03 | |
| *** jamesmcarthur has joined #openstack-infra | 02:03 | |
| *** mikal has quit IRC | 02:06 | |
| *** eernst has joined #openstack-infra | 02:08 | |
| *** gongysh has quit IRC | 02:11 | |
| *** kjackal has quit IRC | 02:12 | |
| *** bobh has joined #openstack-infra | 02:14 | |
| *** tellesnobrega has quit IRC | 02:14 | |
| *** s-shiono has joined #openstack-infra | 02:21 | |
| openstackgerrit | Merged openstack-infra/puppet-bandersnatch master: Catch subprocess output and print https://review.openstack.org/553626 | 02:21 |
| *** tellesnobrega has joined #openstack-infra | 02:23 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add limestone to cloud launcher https://review.openstack.org/552708 | 02:24 |
| *** mriedem has quit IRC | 02:26 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Change base-test to use "ara-report" role instead of "emit-ara-html" https://review.openstack.org/553659 | 02:33 |
| *** germs has quit IRC | 02:33 | |
| *** germs has joined #openstack-infra | 02:35 | |
| *** germs has quit IRC | 02:35 | |
| *** germs has joined #openstack-infra | 02:35 | |
| *** efoley has quit IRC | 02:41 | |
| *** psachin has joined #openstack-infra | 02:42 | |
| *** eernst has quit IRC | 02:48 | |
| *** psachin has quit IRC | 02:49 | |
| *** olaph1 has joined #openstack-infra | 02:50 | |
| *** felipemonteiro has quit IRC | 02:52 | |
| *** olaph has quit IRC | 02:52 | |
| *** eernst has joined #openstack-infra | 02:52 | |
| *** psachin has joined #openstack-infra | 02:54 | |
| openstackgerrit | Merged openstack-infra/zuul master: Generate symlinks during tests https://review.openstack.org/553316 | 02:54 |
| *** Goneri has quit IRC | 02:55 | |
| *** hongbin has joined #openstack-infra | 02:57 | |
| *** salv-orlando has joined #openstack-infra | 02:57 | |
| *** salv-orlando has quit IRC | 03:01 | |
| *** bobh has quit IRC | 03:02 | |
| *** adam_g has quit IRC | 03:11 | |
| *** eernst has quit IRC | 03:13 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config master: Change base-test to use "ara-report" role instead of "emit-ara-html" https://review.openstack.org/553659 | 03:14 |
| ianw | ok ... recovered mirror ... now knowing even more that i don't really want to know about reprepro. will add notes to our recovery docs :/ | 03:14 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/openstack-zuul-jobs master: Do not merge: Re-parent integration jobs to base-test https://review.openstack.org/553663 | 03:16 |
| *** adam_g has joined #openstack-infra | 03:17 | |
| dmsimard | infra-root, config-core: the ara sqlite middleware is deployed for logs-dev.o.o and I'd like to test it with base-test: https://review.openstack.org/#/q/topic:ara-sqlite-middleware | 03:18 |
| dmsimard | Good night o/ | 03:18 |
| dmsimard | oh, limestone has ended up merging, great, I'll continue tomorrow | 03:19 |
| *** olaph has joined #openstack-infra | 03:20 | |
| *** agopi has joined #openstack-infra | 03:20 | |
| *** olaph1 has quit IRC | 03:21 | |
| *** agopi has quit IRC | 03:30 | |
| *** agopi has joined #openstack-infra | 03:33 | |
| openstackgerrit | Hongbin Lu proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553673 | 03:34 |
| *** bobh has joined #openstack-infra | 03:36 | |
| *** r-daneel has quit IRC | 03:36 | |
| *** agopi has quit IRC | 03:38 | |
| *** nicolasbock has quit IRC | 03:40 | |
| *** iyamahat has joined #openstack-infra | 03:42 | |
| *** bobh has quit IRC | 03:45 | |
| *** r-daneel has joined #openstack-infra | 03:53 | |
| *** r-daneel_ has joined #openstack-infra | 03:55 | |
| *** hongbin has quit IRC | 03:57 | |
| *** r-daneel has quit IRC | 03:57 | |
| *** r-daneel_ is now known as r-daneel | 03:57 | |
| *** salv-orlando has joined #openstack-infra | 03:58 | |
| *** pgadiya has joined #openstack-infra | 03:59 | |
| *** udesale has joined #openstack-infra | 04:00 | |
| *** salv-orlando has quit IRC | 04:02 | |
| *** eernst has joined #openstack-infra | 04:03 | |
| *** germs has quit IRC | 04:05 | |
| *** eernst has quit IRC | 04:05 | |
| *** germs has joined #openstack-infra | 04:05 | |
| *** germs has quit IRC | 04:05 | |
| *** germs has joined #openstack-infra | 04:05 | |
| *** germs has quit IRC | 04:06 | |
| *** ykarel|away has joined #openstack-infra | 04:07 | |
| *** janki has joined #openstack-infra | 04:13 | |
| *** mikal_ has quit IRC | 04:17 | |
| *** olaph1 has joined #openstack-infra | 04:18 | |
| *** olaph has quit IRC | 04:19 | |
| *** mikal has joined #openstack-infra | 04:20 | |
| ianw | #status mirror-update.o.o upgraded to bionic AFS packages (1.8.0~pre5-1ppa2). ubuntu-ports, ubuntu & debian mirrors recovered | 04:26 |
| openstackstatus | ianw: unknown command | 04:26 |
| ianw | #status log mirror-update.o.o upgraded to bionic AFS packages (1.8.0~pre5-1ppa2). ubuntu-ports, ubuntu & debian mirrors recovered | 04:26 |
| openstackstatus | ianw: finished logging | 04:27 |
| ianw | now ... we wait | 04:27 |
| *** iyamahat has quit IRC | 04:28 | |
| dmsimard | ianw: bandersnatch is not happy | 04:28 |
| ianw | hmm, it was a few hours ago | 04:29 |
| dmsimard | flock: cannot open lock file /var/run/bandersnatch/mirror.lock: No such file or directory | 04:29 |
| dmsimard | flock: cannot open lock file /var/run/bandersnatch/mirror.lock: No such file or directory | 04:29 |
| dmsimard | Oops, sorry for double paste. | 04:29 |
| ianw | oh, i think we have an issue with puppet creating them | 04:29 |
| ianw | when the hosts reboot, we drop the lockfile dirs | 04:29 |
| ianw | so until puppet runs, things fail | 04:30 |
| dmsimard | in /var/run ? | 04:30 |
| dmsimard | maybe that should be in whatever runs in the cron/service | 04:30 |
| *** links has joined #openstack-infra | 04:32 | |
| ianw | yeah, it should be | 04:32 |
| *** harlowja has joined #openstack-infra | 04:35 | |
| *** sree_ has joined #openstack-infra | 04:39 | |
| *** ramishra has joined #openstack-infra | 04:39 | |
| *** sree_ is now known as Guest98337 | 04:39 | |
| *** andreas_s has joined #openstack-infra | 04:40 | |
| *** ykarel|away is now known as ykarel | 04:42 | |
| *** lpetrut has joined #openstack-infra | 04:42 | |
| *** Guest98337 has quit IRC | 04:43 | |
| *** sree has joined #openstack-infra | 04:43 | |
| *** andreas_s has quit IRC | 04:45 | |
| *** salv-orlando has joined #openstack-infra | 04:59 | |
| *** salv-orlando has quit IRC | 05:03 | |
| *** lpetrut has quit IRC | 05:17 | |
| *** zhurong has joined #openstack-infra | 05:18 | |
| *** lpetrut has joined #openstack-infra | 05:20 | |
| *** harlowja has quit IRC | 05:22 | |
| *** lpetrut has quit IRC | 05:34 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553623 | 05:38 |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553627 | 05:41 |
| *** jtomasek has joined #openstack-infra | 05:41 | |
| *** iyamahat has joined #openstack-infra | 05:44 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects for tox-cover https://review.openstack.org/553619 | 05:49 |
| *** aeng has quit IRC | 05:50 | |
| AJaeger | config-core, could you review these changes to add horizon, please? ^ | 05:57 |
| AJaeger | config-core: https://review.openstack.org/553619 https://review.openstack.org/553623 https://review.openstack.org/553621 https://review.openstack.org/553673 | 05:58 |
| *** salv-orlando has joined #openstack-infra | 05:59 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553627 | 06:03 |
| *** salv-orlando has quit IRC | 06:04 | |
| AJaeger | config-core, this one as well, please ^ | 06:05 |
| *** germs has joined #openstack-infra | 06:06 | |
| *** germs has quit IRC | 06:06 | |
| *** germs has joined #openstack-infra | 06:06 | |
| *** dhajare has joined #openstack-infra | 06:08 | |
| *** kiennt26 has joined #openstack-infra | 06:10 | |
| *** salv-orlando has joined #openstack-infra | 06:10 | |
| *** germs has quit IRC | 06:11 | |
| *** claudiub|2 has joined #openstack-infra | 06:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553623 | 06:17 |
| openstackgerrit | Merged openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553621 | 06:17 |
| openstackgerrit | Merged openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553673 | 06:17 |
| openstackgerrit | Merged openstack-infra/project-config master: Add horizon to required-projects for tox-cover https://review.openstack.org/553619 | 06:18 |
| *** kjackal has joined #openstack-infra | 06:18 | |
| frickler | ianw: no I didn't fix bup on zuul01, just received the next cron mail. would be nice if you could still look into it | 06:21 |
| *** pgadiya has quit IRC | 06:21 | |
| *** jamesmcarthur has quit IRC | 06:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553627 | 06:28 |
| *** alexchadin has joined #openstack-infra | 06:31 | |
| *** dbecker has quit IRC | 06:32 | |
| *** pgadiya has joined #openstack-infra | 06:38 | |
| *** Diabelko has quit IRC | 06:40 | |
| openstackgerrit | Shu Muto proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553690 | 06:41 |
| *** gongysh has joined #openstack-infra | 06:43 | |
| *** dbecker has joined #openstack-infra | 06:45 | |
| *** bhavik1 has joined #openstack-infra | 06:50 | |
| *** iyamahat has quit IRC | 06:50 | |
| *** jaosorior has joined #openstack-infra | 06:50 | |
| *** jtomasek has quit IRC | 06:52 | |
| ianw | frickler: will do | 06:56 |
| AJaeger | config-core, one more horizon addition with https://review.openstack.org/553690 - please add | 07:00 |
| *** jbadiapa has joined #openstack-infra | 07:02 | |
| *** jbadiapa has quit IRC | 07:02 | |
| *** jbadiapa has joined #openstack-infra | 07:02 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-zuul master: Don't remove libjs-jquery https://review.openstack.org/553697 | 07:07 |
| ianw | frickler: ^ there's the root cause of why bup has gone missing | 07:07 |
| ianw | odd one | 07:07 |
| *** ykarel has quit IRC | 07:10 | |
| *** ykarel has joined #openstack-infra | 07:11 | |
| *** pgadiya has quit IRC | 07:12 | |
| *** jcoufal has joined #openstack-infra | 07:12 | |
| *** yolanda_ is now known as yolanda | 07:16 | |
| *** lpetrut has joined #openstack-infra | 07:17 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Don't count non-live queue items in tenant list https://review.openstack.org/553702 | 07:17 |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553703 | 07:18 |
| *** lpetrut_ has joined #openstack-infra | 07:19 | |
| *** lpetrut has quit IRC | 07:19 | |
| *** rcernin has quit IRC | 07:20 | |
| *** pgadiya has joined #openstack-infra | 07:26 | |
| *** Diabelko has joined #openstack-infra | 07:26 | |
| *** liusheng has quit IRC | 07:33 | |
| *** liusheng has joined #openstack-infra | 07:34 | |
| *** armaan has joined #openstack-infra | 07:35 | |
| *** rodrigod` is now known as rodrigods | 07:43 | |
| *** dhajare has quit IRC | 07:44 | |
| *** dhajare has joined #openstack-infra | 07:44 | |
| *** olaph has joined #openstack-infra | 07:53 | |
| *** olaph1 has quit IRC | 07:56 | |
| *** zhurong has quit IRC | 07:56 | |
| *** caphrim007 has joined #openstack-infra | 08:04 | |
| *** germs has joined #openstack-infra | 08:08 | |
| *** germs has quit IRC | 08:08 | |
| *** germs has joined #openstack-infra | 08:08 | |
| *** jcoufal_ has joined #openstack-infra | 08:08 | |
| *** s-shiono has quit IRC | 08:10 | |
| *** jcoufal has quit IRC | 08:10 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Don't count non-live queue items in tenant list https://review.openstack.org/553702 | 08:11 |
| *** pblaho has joined #openstack-infra | 08:12 | |
| *** germs has quit IRC | 08:13 | |
| *** tojuvone has left #openstack-infra | 08:13 | |
| openstackgerrit | Merged openstack/diskimage-builder master: GPT partitioning support https://review.openstack.org/533490 | 08:14 |
| *** eernst has joined #openstack-infra | 08:18 | |
| *** electrofelix has joined #openstack-infra | 08:18 | |
| *** caphrim007 has quit IRC | 08:18 | |
| openstackgerrit | shangxdy proposed openstack-infra/project-config master: Fix ZUUL_USER_SSH_PUBLIC_KEY to support ssh key content https://review.openstack.org/467919 | 08:20 |
| *** florianf has joined #openstack-infra | 08:20 | |
| openstackgerrit | shangxdy proposed openstack-infra/project-config master: Fix ZUUL_USER_SSH_PUBLIC_KEY to support ssh key content https://review.openstack.org/467919 | 08:22 |
| *** ralonsoh has joined #openstack-infra | 08:24 | |
| *** ramishra has quit IRC | 08:25 | |
| *** ramishra has joined #openstack-infra | 08:25 | |
| *** hannv has joined #openstack-infra | 08:29 | |
| *** tesseract has joined #openstack-infra | 08:30 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fail if two elements provide the same thing https://review.openstack.org/539401 | 08:31 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add block-device defaults https://review.openstack.org/539375 | 08:31 |
| openstackgerrit | Merged openstack/diskimage-builder master: Choose appropriate bootloader for block-device https://review.openstack.org/539731 | 08:31 |
| openstackgerrit | Merged openstack/diskimage-builder master: arm64: use HWE kernel and fix console https://review.openstack.org/547161 | 08:31 |
| *** jchhatbar has joined #openstack-infra | 08:32 | |
| *** janki has quit IRC | 08:32 | |
| *** zoli is now known as zoli|mtg-afk | 08:34 | |
| *** hannv has quit IRC | 08:38 | |
| *** chandankumar is now known as chkumar|off | 08:42 | |
| *** eernst has quit IRC | 08:44 | |
| *** dingyichen has quit IRC | 08:46 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: WIP: Update nodepool runtime stats https://review.openstack.org/553718 | 08:48 |
| *** hashar has joined #openstack-infra | 08:49 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** alexchadin has quit IRC | 08:50 | |
| *** alexchadin has joined #openstack-infra | 08:50 | |
| *** alexchadin has quit IRC | 08:51 | |
| *** alexchadin has joined #openstack-infra | 08:51 | |
| *** alexchadin has quit IRC | 08:51 | |
| *** alexchadin has joined #openstack-infra | 08:52 | |
| *** alexchadin has quit IRC | 08:52 | |
| *** alexchadin has joined #openstack-infra | 08:53 | |
| *** alexchadin has quit IRC | 08:53 | |
| *** alexchadin has joined #openstack-infra | 08:54 | |
| *** alexchadin has quit IRC | 08:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553690 | 08:58 |
| *** ykarel is now known as ykarel|lunch | 08:59 | |
| *** armaan has quit IRC | 08:59 | |
| *** alexchadin has joined #openstack-infra | 08:59 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/553703 | 09:01 |
| *** jpich has joined #openstack-infra | 09:01 | |
| *** alexchadin has quit IRC | 09:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add horizon to required-projects https://review.openstack.org/550130 | 09:08 |
| *** dhajare has quit IRC | 09:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add tacker-specs webpage to specs.openstack.org https://review.openstack.org/552451 | 09:11 |
| *** hashar has quit IRC | 09:15 | |
| *** ykarel|lunch is now known as ykarel | 09:17 | |
| *** psachin has quit IRC | 09:22 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Ignore node_modules in flake8 https://review.openstack.org/553725 | 09:22 |
| *** hashar has joined #openstack-infra | 09:22 | |
| *** jistr is now known as jistr|mtg | 09:24 | |
| *** alexchadin has joined #openstack-infra | 09:25 | |
| *** arxcruz is now known as arxcruz|off | 09:27 | |
| *** alexchadin has quit IRC | 09:29 | |
| *** alexchadin has joined #openstack-infra | 09:30 | |
| *** psachin has joined #openstack-infra | 09:34 | |
| *** alexchadin has quit IRC | 09:35 | |
| *** efoley has joined #openstack-infra | 09:35 | |
| *** lucas-afk is now known as lucasagomes | 09:38 | |
| *** derekh has joined #openstack-infra | 09:40 | |
| *** dtantsur|afk is now known as dtantsur | 09:42 | |
| *** esberglu has joined #openstack-infra | 09:43 | |
| *** gfidente has joined #openstack-infra | 09:43 | |
| *** gfidente has joined #openstack-infra | 09:43 | |
| *** esberglu has quit IRC | 09:47 | |
| *** annp has joined #openstack-infra | 09:47 | |
| *** olaph1 has joined #openstack-infra | 09:51 | |
| *** olaph has quit IRC | 09:51 | |
| *** andreas_s has joined #openstack-infra | 09:54 | |
| *** auristor has quit IRC | 09:55 | |
| *** alexchadin has joined #openstack-infra | 09:56 | |
| *** niska has quit IRC | 09:57 | |
| *** jtomasek has joined #openstack-infra | 10:00 | |
| *** tosky has joined #openstack-infra | 10:03 | |
| *** niska has joined #openstack-infra | 10:03 | |
| *** auristor has joined #openstack-infra | 10:04 | |
| *** jcoufal has joined #openstack-infra | 10:06 | |
| *** germs has joined #openstack-infra | 10:09 | |
| *** jcoufal_ has quit IRC | 10:09 | |
| *** armaan has joined #openstack-infra | 10:10 | |
| *** germs has quit IRC | 10:14 | |
| *** olaph has joined #openstack-infra | 10:14 | |
| *** olaph1 has quit IRC | 10:15 | |
| *** armaan_ has joined #openstack-infra | 10:18 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: webapp: add optional admin endpoint https://review.openstack.org/536319 | 10:19 |
| *** markmcd has joined #openstack-infra | 10:20 | |
| *** sree has quit IRC | 10:21 | |
| *** e0ne has joined #openstack-infra | 10:21 | |
| *** sree has joined #openstack-infra | 10:22 | |
| *** armaan has quit IRC | 10:22 | |
| *** sree has quit IRC | 10:22 | |
| *** sree has joined #openstack-infra | 10:22 | |
| AJaeger | infra-root, we have quite interesting AFS behaviour - out publish job either ends after 20 mins or times out after 1h (we increased the timeout for post): http://zuul.openstack.org/builds.html?job_name=publish-openstack-manuals&project=openstack%2Fopenstack-manuals. Is there a generic problem - or should we increase timeout further? | 10:23 |
| *** kiennt26 has quit IRC | 10:24 | |
| *** bhavik1 has quit IRC | 10:24 | |
| *** efoley has quit IRC | 10:26 | |
| *** andymccr_ is now known as andymccr | 10:27 | |
| *** shardy has joined #openstack-infra | 10:28 | |
| ianw | AJaeger: looking at one ... http://logs.openstack.org/17/1787e34904e5d4786b231c93bd2352b3cd24f82e/post/publish-openstack-manuals/a5c385e/job-output.txt.gz#_2018-03-03_08_17_45_793405 | 10:30 |
| ianw | that ran on ze09 | 10:30 |
| ianw | [Sat Mar 3 08:19:59 2018] afs: Waiting for busy volume 536870991 (docs) in cell openstack.org | 10:30 |
| ianw | http://logs.openstack.org/77/77c23e061e10c5fce8cd0586c23bfa0729367063/post/publish-openstack-manuals/c97afd5/job-output.txt.gz#_2018-03-16_06_59_25_737591 was ze03 | 10:32 |
| *** efoley has joined #openstack-infra | 10:32 | |
| ianw | [Fri Mar 16 06:59:14 2018] afs: Waiting for busy volume 536870991 (docs) in cell openstack.org | 10:33 |
| AJaeger | ianw: interesting. What would help here? | 10:33 |
| ianw | well, yeah, i don't know :) it doesn't line up 100% but it seems closely correlated. i think that is when you try to do a vos release when another one is already happening | 10:34 |
| *** nicolasbock has joined #openstack-infra | 10:36 | |
| ianw | AJaeger: all i can think at the moment is that we could put in a loop that checks vos examine on the volume; e.g. http://paste.openstack.org/show/702973/ | 10:38 |
| *** armaan has joined #openstack-infra | 10:39 | |
| ianw | if you look at the examples of http://docs.openafs.org/Reference/1/vos_examine.html, it will say "Volume is currently LOCKED" I *think* if a release is in progress. we could probe for that ... | 10:39 |
| ianw | but i mean, vos release *should* theoretically be atomic and just work ... but then again there could be a bug. so possibly upgrading to the openafs 1.8 packages i've installed on mirror-update might help too | 10:40 |
| *** ralonsoh_ has joined #openstack-infra | 10:41 | |
| *** armaan_ has quit IRC | 10:41 | |
| AJaeger | let's ask corvus and rest of the team as well | 10:41 |
| ianw | trying to think of something constructive ... possibly as a first step, we can just dump the "vos examine" output into the logs? that would let us know if the state is LOCKED before we go into the release | 10:41 |
| AJaeger | thanks, ianw - I'll wait now | 10:41 |
| *** ralonsoh has quit IRC | 10:41 | |
| ianw | if we see it consistently is and that's when it hangs, well that's a pretty good data point | 10:42 |
| *** olaph has quit IRC | 10:44 | |
| *** olaph has joined #openstack-infra | 10:45 | |
| ianw | AJaeger: i won't get to it now, but i'll add it to my todo list if someone else doesn't get to it first :) ttyl | 10:45 |
| AJaeger | ianw: so, add a vos examine before we do the vos release? I can send a patch... | 10:46 |
| AJaeger | ianw: thanks | 10:46 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Add vos examine before release https://review.openstack.org/553749 | 10:47 |
| AJaeger | ianw, infra-root, ^ | 10:47 |
| *** udesale has quit IRC | 10:50 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update neutron-dynamic-routing grafana https://review.openstack.org/553635 | 10:54 |
| auristor | If the äfs: Waiting for busy volume¨message coming from a client cache manager? That is simply because the volume is offline for a volume transaction. | 10:54 |
| *** pgadiya has quit IRC | 10:55 | |
| *** olaph1 has joined #openstack-infra | 10:56 | |
| ianw | auristor: that message above was from the kernel logs | 10:56 |
| AJaeger | frickler, ianw, should we add another examine after release (as frickler proposes) or merge as is for now? | 10:57 |
| auristor | As for executing two ¨vos release¨ commands on the same volume at the same time, that is exceedingly dangerous. The VL entry lock is ignored by the vlserver if it was issued for a ¨release operation¨ and another request to lock for ¨release¨ is received. | 10:57 |
| auristor | ianw: kernel logs means from a cache manager. | 10:57 |
| *** olaph has quit IRC | 10:58 | |
| auristor | The OpenAFS cache manager upon receiving a VBUSY error will sleep for 15 seconds and retry the request. It will repeat for up to 100 retries before giving up. | 10:58 |
| *** eumel8 has joined #openstack-infra | 10:59 | |
| *** armaan has quit IRC | 11:01 | |
| *** olaph1 has quit IRC | 11:05 | |
| ianw | auristor: well that is interesting; what about copying to a releasing volume? | 11:05 |
| *** olaph has joined #openstack-infra | 11:05 | |
| ianw | it is actually http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/upload-afs/tasks/main.yaml that seems to hang | 11:05 |
| *** armaan has joined #openstack-infra | 11:06 | |
| ianw | which is https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/upload-afs/library/zuul_afs.py | 11:06 |
| *** jistr|mtg is now known as jistr | 11:07 | |
| ianw | which essentially rsyncs the data. but we know we've got the "busy volume" message around the same time | 11:07 |
| *** dhajare has joined #openstack-infra | 11:07 | |
| auristor | A release is an atomic operation. Depending on the volume configuration and vos options either a copy on write clone of the RW volume on the vice partition storing the RW for subsequent operations or the volume is kept offline until the release completes. | 11:07 |
| *** bhujay has joined #openstack-infra | 11:08 | |
| auristor | the fileserver returns VBUSY when the volume is in use by the volserver. Neither writes nor reads will complete when the volserver has exclusive use of the volume. | 11:09 |
| *** olaph1 has joined #openstack-infra | 11:11 | |
| AJaeger | ianw, auristor, so, should we merge https://review.openstack.org/553749 ot get some more debug output? Or not needed? | 11:11 |
| *** olaph has quit IRC | 11:12 | |
| ianw | well it seems very possible that one job is in https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/release-afs-volume/tasks/main.yaml and has run a vos release, and another job is rsyncing to the same volume | 11:12 |
| ianw | AJaeger: hmm, now i look, it does seem to be the "Synchronize files to AFS" task that hangs consistently? | 11:13 |
| AJaeger | ianw: yes | 11:15 |
| AJaeger | ianw: but yes, we do not limit how many jobs push to AFS in the post queue. openstack-manuals publishes a lot... | 11:15 |
| ianw | well from auristor's comments, i think we probably want to consider some sort of lock on calling "vos release", because that doesn't seem like it's safe to call that concurrently. but i think that's only a small possibility of hitting | 11:18 |
| eumel8 | the job has no end time in ara. so it seems it's is still running: http://logs.openstack.org/77/77c23e061e10c5fce8cd0586c23bfa0729367063/post/publish-openstack-manuals/c97afd5/ara/ | 11:18 |
| ianw | right, yeah, it's basically a wrapper around rsync | 11:19 |
| *** armaan has quit IRC | 11:19 | |
| ianw | i strongly suspect that when the job times out, the volume as been released while the rsync running. that would explain the "Waiting for busy volume" I think which we see around the time of the sync | 11:20 |
| *** pbourke_ has quit IRC | 11:20 | |
| *** pbourke_ has joined #openstack-infra | 11:21 | |
| ianw | what happens to cause a read or write there to just hang ... i dunno. if we can catch it live, we can lsof/strace it to see what it's waiting for | 11:22 |
| *** jamesmcarthur has joined #openstack-infra | 11:22 | |
| auristor | A VBUSY error is not delivered to the application. The request is retried by the cache manager. | 11:23 |
| *** yamamoto has quit IRC | 11:24 | |
| ianw | auristor: it could unintended behavior, though? as ajaeger says, we probably run enough that a small race could hit us semi-frequently | 11:26 |
| auristor | what do you mean by unintended behavior? | 11:27 |
| ianw | a bug | 11:27 |
| auristor | in what? | 11:27 |
| *** jamesmcarthur has quit IRC | 11:28 | |
| *** dhajare has quit IRC | 11:28 | |
| ianw | the openafs client. don't you think it's suspicious that rsync seems to hang right around the time those messages are in the logs? | 11:28 |
| auristor | as I said, the cache manager sleeps the operation for 15 seconds when it receives a VBUSY before it attempts to retry. It will retry up to 100 times before it fails. each time another VBUSY is received, the cache manager sleeps again | 11:29 |
| *** psachin has quit IRC | 11:33 | |
| *** katkapilatova has joined #openstack-infra | 11:36 | |
| *** dhajare has joined #openstack-infra | 11:36 | |
| *** katkapilatova has left #openstack-infra | 11:37 | |
| *** psachin has joined #openstack-infra | 11:38 | |
| ianw | well, i'm out of ideas. i can't see that the volume is busy for > 60 minutes leading to the job to time-out. i feel like it's pretty likely the rsync is stuck in a read() or a write() that never returns. hopefully someone can catch the job when that happens and confirm it | 11:39 |
| Roamer` | hi guys, I have a question about the process for a new image driver, namely https://review.openstack.org/#/c/137360/ - the StorPool Cinder driver is now in Queens, the Nova volume attachment driver is also in; does Flavio Percoco's position of "write the driver, maintain it externally for one cycle, then we'll talk" still hold? | 11:39 |
| ianw | Roamer`: this is really the wrong channel to talk about that, you'll want #openstack-glance to find the right people | 11:40 |
| Roamer` | oops...... sorry, I did indeed post this to the wrong channel, I did mean to post it to -glance :( | 11:41 |
| Roamer` | sorry! | 11:41 |
| ianw | np :) | 11:41 |
| *** dhajare has quit IRC | 11:41 | |
| *** ldnunes has joined #openstack-infra | 11:43 | |
| *** zoli|mtg-afk is now known as zoli | 11:54 | |
| *** hashar has quit IRC | 11:56 | |
| *** yamamoto has joined #openstack-infra | 11:58 | |
| *** gfidente has quit IRC | 12:00 | |
| *** gfidente has joined #openstack-infra | 12:00 | |
| *** gfidente has joined #openstack-infra | 12:00 | |
| *** hashar has joined #openstack-infra | 12:01 | |
| *** odyssey4me has quit IRC | 12:04 | |
| *** odyssey4me has joined #openstack-infra | 12:04 | |
| *** bhujay has quit IRC | 12:05 | |
| *** olaph has joined #openstack-infra | 12:06 | |
| *** olaph1 has quit IRC | 12:08 | |
| *** germs has joined #openstack-infra | 12:10 | |
| *** germs has quit IRC | 12:10 | |
| *** germs has joined #openstack-infra | 12:10 | |
| *** edmondsw has joined #openstack-infra | 12:12 | |
| *** dsariel has joined #openstack-infra | 12:12 | |
| *** sree has quit IRC | 12:13 | |
| *** sree has joined #openstack-infra | 12:13 | |
| *** germs has quit IRC | 12:14 | |
| *** sambetts|afk is now known as sambetts | 12:15 | |
| *** CrazySaneV0SIIR has joined #openstack-infra | 12:17 | |
| *** bobh has joined #openstack-infra | 12:19 | |
| *** dsariel has quit IRC | 12:20 | |
| *** gaurangt has quit IRC | 12:20 | |
| *** rosmaita has joined #openstack-infra | 12:20 | |
| *** gaurangt has joined #openstack-infra | 12:22 | |
| *** annp has quit IRC | 12:24 | |
| *** jpena is now known as jpena|lunch | 12:27 | |
| fungi | ianw: AJaeger: it's probably worth remembering that we used to call vos release in a separate job from writes, and only from one persistent node so effectively with a mutex | 12:28 |
| fungi | oh, though i suppose that still wouldn't prevent one job writing while a vos release is called | 12:29 |
| *** yamamoto has quit IRC | 12:31 | |
| *** shardy is now known as shardy_lunch | 12:32 | |
| *** rlandy has joined #openstack-infra | 12:32 | |
| *** dprince has joined #openstack-infra | 12:34 | |
| *** gfidente has quit IRC | 12:34 | |
| *** bobh has quit IRC | 12:35 | |
| *** yamamoto has joined #openstack-infra | 12:36 | |
| *** gfidente has joined #openstack-infra | 12:37 | |
| *** gfidente has joined #openstack-infra | 12:37 | |
| *** gongysh has quit IRC | 12:42 | |
| openstackgerrit | Alex Krzos proposed openstack-infra/grafyaml master: Add Custom Template Type https://review.openstack.org/544305 | 12:44 |
| *** jchhatbar has quit IRC | 12:47 | |
| *** jchhatba_ has joined #openstack-infra | 12:47 | |
| *** trown|outtypewww is now known as trown | 12:48 | |
| *** trown is now known as trown|ruck | 12:49 | |
| *** gongysh has joined #openstack-infra | 12:51 | |
| dmsimard | infra-root, config-core: the ara sqlite middleware is successfully deploy on logs-dev.o.o, I'm looking to test it with base-test and the following patches -- please review :) https://review.openstack.org/#/q/topic:ara-sqlite-middleware | 12:55 |
| *** gongysh has quit IRC | 12:57 | |
| *** jlabarre has quit IRC | 12:58 | |
| *** dave-mccowan has joined #openstack-infra | 12:59 | |
| *** jlabarre has joined #openstack-infra | 13:00 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move tmpdir into work root https://review.openstack.org/546698 | 13:00 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Update to Ansible 2.4 https://review.openstack.org/535781 | 13:00 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Disable action and lookup plugins from 2.4 https://review.openstack.org/535839 | 13:00 |
| *** salv-orlando has quit IRC | 13:01 | |
| *** shardy_lunch is now known as shardy | 13:03 | |
| *** gyankum has joined #openstack-infra | 13:03 | |
| *** dave-mcc_ has joined #openstack-infra | 13:04 | |
| *** dave-mccowan has quit IRC | 13:04 | |
| *** lucasagomes is now known as lucas-hungry | 13:06 | |
| *** mriedem has joined #openstack-infra | 13:06 | |
| *** Goneri has joined #openstack-infra | 13:07 | |
| *** efried is now known as fried_rice | 13:10 | |
| *** dingyichen has joined #openstack-infra | 13:11 | |
| *** esberglu has joined #openstack-infra | 13:17 | |
| *** salv-orlando has joined #openstack-infra | 13:17 | |
| *** myoung|afk is now known as myoung|rover | 13:22 | |
| *** kgiusti has joined #openstack-infra | 13:22 | |
| *** ihrachys has joined #openstack-infra | 13:24 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: tox-remote debug change https://review.openstack.org/553787 | 13:25 |
| *** andreas_s has quit IRC | 13:27 | |
| *** eventingmonkey has quit IRC | 13:30 | |
| *** eharney has joined #openstack-infra | 13:30 | |
| *** andreas_s_ has joined #openstack-infra | 13:31 | |
| *** udesale has joined #openstack-infra | 13:31 | |
| *** eventingmonkey has joined #openstack-infra | 13:32 | |
| *** jpena|lunch is now known as jpena | 13:33 | |
| *** r-daneel has quit IRC | 13:34 | |
| *** jchhatba_ has quit IRC | 13:35 | |
| *** andreas_s_ has quit IRC | 13:35 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Import systemd_mount and systemd_service to OSA https://review.openstack.org/552861 | 13:36 |
| *** e0ne has quit IRC | 13:40 | |
| *** e0ne has joined #openstack-infra | 13:40 | |
| *** eventingmonkey has quit IRC | 13:42 | |
| *** dhajare has joined #openstack-infra | 13:43 | |
| *** felipemonteiro has joined #openstack-infra | 13:44 | |
| *** eventingmonkey has joined #openstack-infra | 13:44 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: tox-remote debug change https://review.openstack.org/553787 | 13:45 |
| mriedem | i have no idea why this failed http://logs.openstack.org/08/553608/2/check/neutron-tempest-linuxbridge/8c45e56/logs/devstack-gate-setup-workspace-new.txt | 13:45 |
| mriedem | any hints? | 13:45 |
| *** david-lyle has quit IRC | 13:45 | |
| *** udesale has quit IRC | 13:46 | |
| *** andreas_s has joined #openstack-infra | 13:48 | |
| *** olaph1 has joined #openstack-infra | 13:49 | |
| *** olaph has quit IRC | 13:49 | |
| *** psachin has quit IRC | 13:51 | |
| *** andreas_s has quit IRC | 13:52 | |
| mriedem | nvm i see the problem | 13:53 |
| *** alexchadin has quit IRC | 13:54 | |
| *** sree_ has joined #openstack-infra | 13:56 | |
| *** sree_ is now known as Guest42798 | 13:56 | |
| *** olaph1 is now known as olaph | 13:56 | |
| *** links has quit IRC | 13:56 | |
| *** sree has quit IRC | 13:58 | |
| *** ykarel has quit IRC | 14:00 | |
| *** hongbin has joined #openstack-infra | 14:00 | |
| *** iyamahat has joined #openstack-infra | 14:01 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move tmpdir into work root https://review.openstack.org/546698 | 14:02 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Update to Ansible 2.4 https://review.openstack.org/535781 | 14:02 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Disable action and lookup plugins from 2.4 https://review.openstack.org/535839 | 14:02 |
| *** dhajare has quit IRC | 14:04 | |
| *** VW has joined #openstack-infra | 14:04 | |
| *** lucas-hungry is now known as lucasagomes | 14:06 | |
| *** chkumar|off is now known as chandankumar | 14:07 | |
| *** bobh has joined #openstack-infra | 14:07 | |
| *** gouthamr has joined #openstack-infra | 14:08 | |
| *** sree has joined #openstack-infra | 14:09 | |
| *** germs has joined #openstack-infra | 14:10 | |
| *** germs has quit IRC | 14:10 | |
| *** germs has joined #openstack-infra | 14:10 | |
| *** Guest42798 has quit IRC | 14:12 | |
| *** germs has quit IRC | 14:15 | |
| *** dhajare has joined #openstack-infra | 14:16 | |
| rosmaita | i want to define a job with parent devstack-tox-functional but that uses a stable/queens devstack (so stable/queens branches for all components). Is there an easy way to do that, or do i need to define project_BRANCH: stable/queens for every project? | 14:17 |
| *** felipemonteiro_ has joined #openstack-infra | 14:17 | |
| *** d0ugal has quit IRC | 14:19 | |
| *** hongbin has quit IRC | 14:20 | |
| *** felipemonteiro has quit IRC | 14:20 | |
| *** hongbin has joined #openstack-infra | 14:20 | |
| mtreinish | persia: ooh, looks like they're finally merging your mosquitto cve fixes: https://bugs.launchpad.net/ubuntu/+source/mosquitto/+bug/1752591 | 14:21 |
| openstack | Launchpad bug 1752591 in mosquitto (Ubuntu Bionic) "CVE-2017-7651 and CVE-2017-7652" [Undecided,Fix released] | 14:21 |
| *** felipemonteiro__ has joined #openstack-infra | 14:21 | |
| *** stephenfin is now known as finucannot | 14:22 | |
| *** dansmith is now known as superdan | 14:22 | |
| persia | mtreinish: Yep. That team was having the equivalent of a midcycle last week, or it would have been sooner (I received an apology last week for it). | 14:23 |
| *** r-daneel has joined #openstack-infra | 14:23 | |
| *** derekh has quit IRC | 14:24 | |
| *** felipemonteiro_ has quit IRC | 14:25 | |
| *** jcoufal has quit IRC | 14:25 | |
| *** lyarwood is now known as lyaaaaaaaaaaaaar | 14:27 | |
| *** olaph1 has joined #openstack-infra | 14:28 | |
| fungi | rosmaita: are you trying to make one which uses stable/queens of all components to check stable/queens changes, or which tests master branch changes for some repo against stable/queens branches of everything else? | 14:29 |
| *** gyankum has quit IRC | 14:29 | |
| *** olaph has quit IRC | 14:30 | |
| fungi | the former is just our usual testing for stable branch changes and shouldn't need anything special. the latter is more the backward-compat testing pattern some libraries/clients use | 14:30 |
| rosmaita | fungi: i think the latter ... the image v1 API is being removed from glance, but the glanceclient will support v1 for another cycle | 14:30 |
| rosmaita | i really only need stable/queens for glance, but figured if everything is stable/glance, it's less likely to break | 14:31 |
| fungi | rosmaita: got it, so you want to test master python-glanceclient changes against a stable/queens environment | 14:31 |
| rosmaita | exactly | 14:31 |
| * fungi checks to see if there's already something like that you can just use | 14:31 | |
| rosmaita | i tried this: https://review.openstack.org/#/c/553641/ | 14:31 |
| mnaser | does logstash archive stack.log too or just services? | 14:31 |
| *** andreas_s has joined #openstack-infra | 14:32 | |
| fungi | mnaser: i think you're looking for logs/devstack.txt.gz? | 14:32 |
| fungi | (something like that) | 14:32 |
| fungi | it may be deeper under a subdirectory for the inventory name of the node it ran on | 14:32 |
| mnaser | fungi: yes, i tried -- filename:"logs/devstack.txt" in search but no go | 14:32 |
| fungi | mnaser: oh, sorry, logstash, i misread you | 14:33 |
| fungi | not sure but i thought it did | 14:33 |
| mnaser | it looks like stuff under logs/.. is there | 14:33 |
| mnaser | maybe i just don't know how to query elasticsearch because it's a science on it's own | 14:33 |
| mnaser | filename:"logs/devstack.txt" returns nothing, but maybe the quotes mess it up? | 14:33 |
| Shrews | fungi: rosmaita: maybe look at shade's shade-functional-devstack-legacy job | 14:34 |
| mnaser | oh actually, time to cheat by looking at elastic recheck | 14:34 |
| mnaser | fungi: ok i can use tags:"job-output.txt" and grab it from there! | 14:35 |
| fungi | mnaser: i thought the stack log was separated from the job output log, but maybe that's only in legacy devstack jobs | 14:35 |
| *** jog0 is now known as jogo | 14:35 | |
| mnaser | you're right, it looks like it has :X | 14:36 |
| corvus | AJaeger, ianw: we don't release the volume in doc publication jobs. that role is only used by wheel builds. | 14:36 |
| rosmaita | Shrews: thanks for the pointer, but check out this comment: https://github.com/openstack-infra/shade/blob/master/.zuul.yaml#L98 | 14:36 |
| fungi | oh, right, i forgot we just published the rw volume for docs | 14:36 |
| corvus | fungi: we have a ro volume for docs, we just release it on a timer. every 5 minutes if it's been updated. | 14:37 |
| fungi | er, no, we do periodic vos release | 14:37 |
| fungi | right, that ;) | 14:37 |
| fungi | so... still possible the timed vos release happens in the middle of an rsync i guess. doesn't change the theory | 14:38 |
| openstackgerrit | Merged openstack-infra/zuul master: Update metadata for pypi https://review.openstack.org/553642 | 14:38 |
| *** derekh has joined #openstack-infra | 14:40 | |
| *** olaph1 is now known as olaph | 14:41 | |
| *** TaintedTrust has joined #openstack-infra | 14:41 | |
| *** felipemonteiro__ has quit IRC | 14:43 | |
| *** felipemonteiro__ has joined #openstack-infra | 14:44 | |
| rm_work | o/ heya | 14:44 |
| *** cgoncalves has joined #openstack-infra | 14:44 | |
| rm_work | RE: the removal of tox_install stuff | 14:44 |
| rm_work | (maybe I should tag AJaeger) | 14:44 |
| rm_work | devstack is branchless and so this breaks stable releases as well, right? | 14:44 |
| rm_work | but, we can't change requirements in stable repos | 14:45 |
| rm_work | or even if we did, global-reqs doesn't have neutron prior to queens, so it would never pass the requirements check gate | 14:45 |
| rm_work | does something like this look like an acceptable workaround? https://review.openstack.org/#/c/553414/4/tox.ini@20 | 14:45 |
| * mordred waves at all the nice people | 14:46 | |
| AJaeger | rm_work: where does this hit devstack? | 14:47 |
| rm_work | AJaeger: check gates for stable | 14:47 |
| *** ykarel has joined #openstack-infra | 14:47 | |
| AJaeger | hi mordred , adding neutron and horizon to requirements was lots of unwanted fun ;( | 14:47 |
| mordred | uhoh | 14:47 |
| AJaeger | mordred: there's a mailthread on openstack-dev | 14:47 |
| AJaeger | rm_work: example? | 14:48 |
| rm_work | let me find one | 14:48 |
| mordred | AJaeger: I shall read - and look forward to pulling it into my brain | 14:48 |
| rm_work | might need to recheck something | 14:48 |
| *** links has joined #openstack-infra | 14:48 | |
| AJaeger | thanks, mordred | 14:48 |
| rm_work | AJaeger: the install process though was changed in devstack, isn't that why tox_install started failing? | 14:49 |
| rm_work | and why we're removing it? | 14:49 |
| cgoncalves | AJaeger, rm_work: for instance https://review.openstack.org/#/c/553399/ | 14:49 |
| AJaeger | rm_work: no, problems are tox tests... | 14:49 |
| rm_work | ah you have one, cool | 14:49 |
| cgoncalves | specifically http://logs.openstack.org/99/553399/1/check/neutron-lbaasv2-dsvm-api/3b2604d/job-output.txt.gz#_2018-03-15_13_52_38_746891 | 14:50 |
| AJaeger | cgoncalves: http://logs.openstack.org/99/553399/1/check/neutron-lbaasv2-dsvm-api/3b2604d/job-output.txt.gz#_2018-03-15_13_52_38_804808 uses the *master* constraints file on a queens branch | 14:50 |
| AJaeger | that's just wrong ;8 | 14:50 |
| rm_work | errr | 14:50 |
| AJaeger | cgoncalves: where does the URL come from? Change it to use queens... | 14:50 |
| rm_work | does it? | 14:50 |
| rm_work | how did we end up doing that <_< | 14:50 |
| * AJaeger checks out neutron-lbaas | 14:51 | |
| rm_work | err | 14:51 |
| rm_work | ahhh that patch is pending | 14:51 |
| rm_work | literally it's this https://review.openstack.org/#/c/542223/ | 14:51 |
| rm_work | fixes it | 14:51 |
| AJaeger | rm_work: then let's get that one in first - and if it still fails, let's talk again ;) | 14:51 |
| rm_work | yeah k | 14:51 |
| rm_work | somehow i thought it went through to devstack but you're right it's just tox | 14:52 |
| AJaeger | cgoncalves, rm_work, you can base 553399 on top of 542223 | 14:52 |
| AJaeger | to quicker get results | 14:52 |
| rm_work | and yeah if the constraints don't block it, that should work fine | 14:52 |
| *** chandankumar is now known as chkumar|off | 14:52 | |
| AJaeger | exactly | 14:52 |
| cgoncalves | AJaeger: with depends-on set, right | 14:52 |
| AJaeger | cgoncalves: either depends-on or stack it on top | 14:53 |
| rm_work | i added the depends-on cgoncalves | 14:53 |
| cgoncalves | AJaeger, rm_work: great, thanks! | 14:53 |
| AJaeger | corvus: so, what should we do with the timeout of openstack-manuals? We increased already to one hour. | 14:53 |
| AJaeger | cgoncalves: please tell me whether it works ;) I'm curious | 14:54 |
| dmsimard | AJaeger: fyi I'll go through the backlog of reviews sometime today | 14:54 |
| cgoncalves | AJaeger: if you don't hear from me in a couple of hours, take it as a good sign ;) | 14:54 |
| *** amoralej is now known as amoralej|lunch | 14:54 | |
| dmsimard | AJaeger: let me know if there's specific things you'd like me to look at, otherwise I'll browse my dashboards | 14:54 |
| *** bnemec is now known as beekneemech | 14:55 | |
| AJaeger | dmsimard: just browse your dashboard- thanks a lot! | 14:55 |
| *** hashar has quit IRC | 14:55 | |
| AJaeger | mordred: https://review.openstack.org/553030 is one solution - would fix the problem by cgoncalves and rm_work as well. Removes neutron/horizon from constraints. But I'm not sure whether that's the right way is or temporary. kolla looks more complicated... | 14:56 |
| rm_work | :/ | 14:57 |
| AJaeger | corvus, ianw, I'll abandon the zuul-jobs vos examine - it's indeed only used by wheels, so won't help us | 14:58 |
| rm_work | wait, what is that going to do? | 14:58 |
| rm_work | blacklist will prevent those things from installing? | 14:58 |
| AJaeger | rm_work: blacklist will allow to put it in global-requirements but not in upper-constraints | 14:58 |
| rm_work | ah k | 14:59 |
| AJaeger | rm_work: so, allows to add it to your requirements.txt - and not biting you with upper-constraints | 14:59 |
| fungi | also prevents the reqs sync jobs and reqs check jobs from acting on entries from the blacklist | 14:59 |
| rm_work | is that really the best choice to do? rather than just fix things? | 14:59 |
| AJaeger | rm_work: it will give us some time - I'm happy to revert that again | 14:59 |
| fungi | blacklist entries are mainly intended for things like static analyzers which may differ in expected version from one project to the next | 15:00 |
| *** TaintedTrust has quit IRC | 15:00 | |
| fungi | e.g. one project may want flake8 1.11 and another wants flake8 2.3 | 15:00 |
| *** dprince has quit IRC | 15:00 | |
| *** dprince has joined #openstack-infra | 15:01 | |
| *** r-daneel has quit IRC | 15:01 | |
| *** r-daneel has joined #openstack-infra | 15:01 | |
| *** nicolasbock has quit IRC | 15:05 | |
| *** jamesmcarthur has joined #openstack-infra | 15:07 | |
| *** jamesmcarthur has quit IRC | 15:07 | |
| *** jamesmcarthur has joined #openstack-infra | 15:07 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config master: Pin networking-cisco build sphinx docs to stable/queens https://review.openstack.org/553816 | 15:07 |
| *** melwitt is now known as jgwentworth | 15:08 | |
| mordred | AJaeger: I think that's a good fix for now | 15:10 |
| *** jamesmca_ has joined #openstack-infra | 15:11 | |
| *** jamesmcarthur has quit IRC | 15:11 | |
| *** electrofelix has quit IRC | 15:12 | |
| *** electrofelix has joined #openstack-infra | 15:12 | |
| *** TaintedTrust has joined #openstack-infra | 15:13 | |
| *** electrofelix has quit IRC | 15:14 | |
| *** udesale has joined #openstack-infra | 15:16 | |
| *** dhajare has quit IRC | 15:16 | |
| AJaeger | mordred: I addeda few comments also to https://etherpad.openstack.org/p/zuulv3-remove-zuul-cloner | 15:17 |
| AJaeger | mordred: list of changes is https://review.openstack.org/#/q/topic:tox-siblings - I didn't expect that we needed to do this (the blacklist would have taken the pressure from us) | 15:18 |
| *** lpetrut_ has quit IRC | 15:19 | |
| *** lpetrut has joined #openstack-infra | 15:19 | |
| *** ramishra has quit IRC | 15:19 | |
| *** sree_ has joined #openstack-infra | 15:21 | |
| *** sree_ is now known as Guest71919 | 15:21 | |
| *** lpetrut has quit IRC | 15:23 | |
| *** sree has quit IRC | 15:24 | |
| *** jcoufal has joined #openstack-infra | 15:24 | |
| *** janki has joined #openstack-infra | 15:25 | |
| *** yamamoto has quit IRC | 15:26 | |
| clarkb | good morning. I'll try to get to all the requested reviews once I have some tea | 15:31 |
| AJaeger | bbl | 15:31 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add standard ca certificate paths https://review.openstack.org/553828 | 15:31 |
| *** links has quit IRC | 15:32 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add standard ca certificate paths https://review.openstack.org/553828 | 15:34 |
| *** TaintedTrust has quit IRC | 15:36 | |
| *** yamamoto has joined #openstack-infra | 15:36 | |
| *** d0ugal has joined #openstack-infra | 15:36 | |
| *** d0ugal has quit IRC | 15:36 | |
| *** d0ugal has joined #openstack-infra | 15:36 | |
| *** amoralej|lunch is now known as amoralej | 15:37 | |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: Set openstack-tox-cover as voting for monasca-ui https://review.openstack.org/553368 | 15:39 |
| *** derekh has quit IRC | 15:39 | |
| *** VW has quit IRC | 15:40 | |
| *** VW has joined #openstack-infra | 15:40 | |
| *** dave-mcc_ is now known as dave-mccowan | 15:40 | |
| *** yamamoto has quit IRC | 15:41 | |
| *** udesale has quit IRC | 15:44 | |
| *** VW has quit IRC | 15:45 | |
| fungi | clarkb: just to clarify the revised gerrit maintenance schedule for project renaming... reviewing our meeting log from tuesday the most concrete decision was that we would regroup on tuesday when we can clarify name choices with mordred, that i was going to put together the revised process draft (working on it today) and you were going to let the release team know when we decide on a new schedule for | 15:46 |
| fungi | that? | 15:46 |
| *** armaan has joined #openstack-infra | 15:47 | |
| *** VW has joined #openstack-infra | 15:47 | |
| clarkb | yup | 15:49 |
| clarkb | and somewhere in there we should probably scan the renamed projects for other projects using their jobs as we know that will break things | 15:49 |
| fungi | thanks! | 15:49 |
| fungi | just wanted to update the release team during their meeting (going now) | 15:50 |
| corvus | clarkb: if we decided to just force-merge things until loading broken config is safe (i think we did decide that), i don't think we need to worry about scanning for jobs | 15:50 |
| clarkb | corvus: ah ok | 15:50 |
| clarkb | corvus: we should still probably sort out what will break ahead of time so we are prepared with changes to force merge but that should be a quick resolution to the problem | 15:51 |
| *** yamamoto has joined #openstack-infra | 15:51 | |
| *** VW has quit IRC | 15:52 | |
| corvus | clarkb: if we just force-merge the change to project-config that renames the project the tenant config and projects.yaml, that should be it | 15:52 |
| *** ykarel is now known as ykarel|afk | 15:52 | |
| corvus | clarkb: (that's why the force-merge option is better right now -- we don't need to think about anything else) | 15:53 |
| *** r-daneel has quit IRC | 15:53 | |
| *** ianychoi_ has joined #openstack-infra | 15:53 | |
| *** VW has joined #openstack-infra | 15:53 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix builds queued forever after failure to get node request https://review.openstack.org/537335 | 15:54 |
| *** iyamahat has quit IRC | 15:54 | |
| *** ldnunes has quit IRC | 15:54 | |
| openstackgerrit | Brian Rosmaita proposed openstack-infra/irc-meetings master: Add Glance bug squad meeting https://review.openstack.org/553834 | 15:54 |
| *** ldnunes has joined #openstack-infra | 15:54 | |
| *** gmann__ has joined #openstack-infra | 15:55 | |
| *** icey_ has joined #openstack-infra | 15:55 | |
| *** simonmcc_ has joined #openstack-infra | 15:55 | |
| *** v1k0d3n_ has joined #openstack-infra | 15:55 | |
| *** yamamoto has quit IRC | 15:55 | |
| *** panda|off is now known as panda | 15:56 | |
| *** luzC_ has joined #openstack-infra | 15:57 | |
| *** gibi_ has joined #openstack-infra | 15:57 | |
| *** darvon_ has joined #openstack-infra | 15:57 | |
| *** pfalleno1 has joined #openstack-infra | 15:57 | |
| *** promethe1nfire has joined #openstack-infra | 15:58 | |
| *** jamesmca_ has quit IRC | 15:58 | |
| *** VW has quit IRC | 15:58 | |
| *** prometheanfire has quit IRC | 15:59 | |
| *** promethe1nfire is now known as prometheanfire | 15:59 | |
| *** andreykurilin_ has joined #openstack-infra | 16:01 | |
| *** jistr_ has joined #openstack-infra | 16:01 | |
| *** ruhe2 has joined #openstack-infra | 16:01 | |
| *** rodrigod` has joined #openstack-infra | 16:01 | |
| *** aluria` has joined #openstack-infra | 16:02 | |
| *** szaher has quit IRC | 16:02 | |
| *** njohnston_ has joined #openstack-infra | 16:02 | |
| *** jlk has quit IRC | 16:03 | |
| *** rodrigods has quit IRC | 16:03 | |
| *** ianychoi has quit IRC | 16:03 | |
| *** luzC has quit IRC | 16:03 | |
| *** bradm has quit IRC | 16:03 | |
| *** pfallenop has quit IRC | 16:03 | |
| *** lifeless has quit IRC | 16:03 | |
| *** otherwiseguy_ has quit IRC | 16:03 | |
| *** mattoliverau has quit IRC | 16:03 | |
| *** gibi has quit IRC | 16:03 | |
| *** njohnston has quit IRC | 16:03 | |
| *** icey has quit IRC | 16:03 | |
| *** simonmcc has quit IRC | 16:03 | |
| *** v1k0d3n has quit IRC | 16:03 | |
| *** gmann_ has quit IRC | 16:03 | |
| *** rajinir has quit IRC | 16:03 | |
| *** knikolla has quit IRC | 16:03 | |
| *** davidlenwell has quit IRC | 16:03 | |
| *** andreykurilin has quit IRC | 16:03 | |
| *** mpeterson has quit IRC | 16:03 | |
| *** darvon has quit IRC | 16:03 | |
| *** sambetts has quit IRC | 16:03 | |
| *** jistr has quit IRC | 16:03 | |
| *** timburke has quit IRC | 16:03 | |
| *** ruhe has quit IRC | 16:03 | |
| *** aluria has quit IRC | 16:03 | |
| *** otherwiseguy has joined #openstack-infra | 16:03 | |
| *** darvon_ is now known as darvon | 16:03 | |
| *** simonmcc_ is now known as simonmcc | 16:03 | |
| *** gmann__ is now known as gmann_ | 16:03 | |
| *** icey_ is now known as icey | 16:03 | |
| *** v1k0d3n_ is now known as v1k0d3n | 16:03 | |
| *** ruhe2 is now known as ruhe | 16:03 | |
| *** harlowja has joined #openstack-infra | 16:03 | |
| *** sambetts_ has joined #openstack-infra | 16:03 | |
| *** sambetts_ is now known as sambetts | 16:03 | |
| *** timburke has joined #openstack-infra | 16:04 | |
| *** dtruong_ has quit IRC | 16:04 | |
| *** jistr_ is now known as jistr | 16:04 | |
| *** mpeterson has joined #openstack-infra | 16:04 | |
| *** kjackal has quit IRC | 16:04 | |
| *** dtruong has joined #openstack-infra | 16:05 | |
| *** lifeless has joined #openstack-infra | 16:05 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Rework log streaming to use python logging https://review.openstack.org/541434 | 16:05 |
| *** Qiming has quit IRC | 16:05 | |
| *** yamamoto has joined #openstack-infra | 16:06 | |
| openstackgerrit | Merged openstack-infra/zuul master: Correctly document default git dirs https://review.openstack.org/535864 | 16:08 |
| *** Qiming has joined #openstack-infra | 16:09 | |
| *** rajinir has joined #openstack-infra | 16:10 | |
| *** knikolla has joined #openstack-infra | 16:10 | |
| *** davidlenwell has joined #openstack-infra | 16:11 | |
| *** yamamoto has quit IRC | 16:11 | |
| *** germs has joined #openstack-infra | 16:11 | |
| *** germs has quit IRC | 16:11 | |
| *** germs has joined #openstack-infra | 16:11 | |
| *** kashyap has joined #openstack-infra | 16:14 | |
| kashyap | clarkb: mnaser: About this ( | 16:14 |
| kashyap | Bump NOVA_READY_TIMEOUT to 120s | 16:14 |
| kashyap | ) | 16:14 |
| kashyap | clarkb: mnaser: Is it possible to do some profiling to see the start-up times? | 16:15 |
| kashyap | It's hard to believe that the libvirt *itself* is taking 1 _whole_ minute! | 16:15 |
| clarkb | kashyap: its not libvirt its qemu post spectre patching | 16:15 |
| *** germs has quit IRC | 16:16 | |
| openstackgerrit | Merged openstack-infra/zuul master: Ignore node_modules in flake8 https://review.openstack.org/553725 | 16:16 |
| kashyap | Hmm | 16:16 |
| kashyap | clarkb: But the review above from mnaser claims: | 16:18 |
| kashyap | [quote] | 16:18 |
| kashyap | This changes bumps NOVA_READY_TIMEOUT to 120s by default in Pike | 16:18 |
| kashyap | and older because the UCA shipped libvirt (2.5.0) is very slow | 16:18 |
| kashyap | to start up (taking almost a minute at times) | 16:18 |
| kashyap | [/quote] | 16:18 |
| *** jlk has joined #openstack-infra | 16:18 | |
| *** bradm has joined #openstack-infra | 16:18 | |
| *** mattoliverau has joined #openstack-infra | 16:18 | |
| clarkb | kashyap: I think it is actually qemu that also got updated at the same time that is being slow | 16:19 |
| kashyap | Yes, QEMU also got updated. | 16:19 |
| clarkb | kashyap: UCA updates that tooling all together and qemu got spectre mitigations added in | 16:19 |
| kashyap | clarkb: I just talked to libvirt devs. mnaser hasn't mentioned WHEN he has measured (if he has) the start-up time | 16:19 |
| kashyap | Is it the _first_ time when libvirtd starts? | 16:19 |
| kashyap | That is not valid. | 16:19 |
| kashyap | Because: first time around, libvirt caches the capabilities. | 16:20 |
| clarkb | kashyap: no its through the tempest testing. Our tempest jobs were failing because the how ever many VMs we boot during those jobs were timing out | 16:20 |
| *** e0ne_ has joined #openstack-infra | 16:20 | |
| clarkb | because 60 seconds was no longer sufficient to boot cirros under emulation | 16:20 |
| kashyap | Hmm | 16:20 |
| clarkb | kashyap: you should be able to test this locally booting cirros under emulation using the pre spectre patches and then doing the same with post spectre | 16:21 |
| kashyap | Yeah | 16:21 |
| kashyap | I'll check w/ RHT's perf team to see if they've got any numbers | 16:21 |
| kashyap | Or do my own testing | 16:21 |
| *** yamamoto has joined #openstack-infra | 16:21 | |
| *** e0ne has quit IRC | 16:21 | |
| *** bhujay has joined #openstack-infra | 16:22 | |
| *** e0ne has joined #openstack-infra | 16:22 | |
| kashyap | clarkb: Thanks for the quick response. | 16:23 |
| openstackgerrit | Merged openstack-infra/zuul master: Add standard ca certificate paths https://review.openstack.org/553828 | 16:24 |
| *** bhujay has quit IRC | 16:24 | |
| *** VW has joined #openstack-infra | 16:24 | |
| *** iyamahat has joined #openstack-infra | 16:25 | |
| *** e0ne_ has quit IRC | 16:25 | |
| *** bhujay has joined #openstack-infra | 16:25 | |
| *** yamamoto has quit IRC | 16:25 | |
| * mordred apologizes to everyone for the upcoming patch-bomb - also please feel free to ignore this stack for the time being | 16:27 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Split status and stream into typescript modules https://review.openstack.org/551989 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add typing to getSourceUrl https://review.openstack.org/551990 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break build list out into its own module https://review.openstack.org/551991 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break job out into its own module https://review.openstack.org/551993 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break job list out into its own module https://review.openstack.org/551994 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break tenant list out into its own module https://review.openstack.org/551995 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Break project detail and list out into their own module https://review.openstack.org/551996 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Move webpack html template to web/config https://review.openstack.org/551997 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Migrate webpack config to typescript https://review.openstack.org/551998 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use glyphicons for status balls https://review.openstack.org/551992 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Migrate status, stream and builds to angular https://review.openstack.org/553845 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Migrate project and projects to angular https://review.openstack.org/553846 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Migrate job and jobs to angular https://review.openstack.org/553847 | 16:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Migrate tenants list to angular https://review.openstack.org/553848 | 16:27 |
| mordred | yes. I was, in fact, on a plane | 16:27 |
| rosmaita | mordred: nice work | 16:27 |
| rosmaita | mordred: do you have a minute for a question? | 16:28 |
| mordred | rosmaita: yes! | 16:28 |
| *** kashyap has left #openstack-infra | 16:28 | |
| rosmaita | mordred: this patch https://review.openstack.org/#/c/553641/ is supposed to run some glanceclient tests against stable/queens (because the Image v1 api is being removed in rocky). Shrews pointed me to a way to get a stable/queens devstack checked out, but i'm getting this error about the tox testenv for v1 not being found: http://logs.openstack.org/41/553641/4/check/glanceclient-dsvm-functional-v1/3253af6/job-output.txt.gz#_ | 16:30 |
| rosmaita | 2018-03-16_16_07_57_264513 | 16:30 |
| rosmaita | sorry, too long a message | 16:30 |
| rosmaita | http://logs.openstack.org/41/553641/4/check/glanceclient-dsvm-functional-v1/3253af6/job-output.txt.gz#_2018-03-16_16_07_57_264513 | 16:30 |
| *** dtantsur is now known as dtantsur|afk | 16:31 | |
| rosmaita | i'm wondering whether the job is using the stable/queens glanceclient instead of the one being tested? | 16:31 |
| *** kjackal has joined #openstack-infra | 16:31 | |
| fungi | rosmaita: http://logs.openstack.org/41/553641/4/check/glanceclient-dsvm-functional-v1/3253af6/zuul-info/inventory.yaml says it checked out stable/queens of python-glanceclient | 16:34 |
| *** felipemonteiro_ has joined #openstack-infra | 16:35 | |
| *** derekh has joined #openstack-infra | 16:35 | |
| rosmaita | fungi: i am admittedly vague on how this actually works, but i was hoping that would get overwritten by the gerrit patch at some point? | 16:35 |
| *** yamamoto has joined #openstack-infra | 16:36 | |
| *** felipemonteiro__ has quit IRC | 16:38 | |
| clarkb | rosmaita: have a link to the job definition? | 16:39 |
| fungi | clarkb: it's in https://review.openstack.org/553641 | 16:39 |
| rosmaita | clarkb: what fungi said | 16:40 |
| fungi | i think the override-checkout there must be causing it to checkout stable/queens for python-glanceclient | 16:40 |
| fungi | is it typical for a job defined on a project to specify that project in required-projects? | 16:41 |
| *** yamamoto has quit IRC | 16:41 | |
| fungi | i guess that doesn't get automagically inferred? | 16:41 |
| clarkb | fungi: that was going to be my question. I wonder if it does the right thing if not in required-projects | 16:41 |
| clarkb | but having it in required-projects allows you to run it against changes to other projects | 16:41 |
| *** kashyap has joined #openstack-infra | 16:42 | |
| kashyap | clarkb: Hey, a critical question I forgot to ask: | 16:42 |
| kashyap | clarkb: On the hosts, do you have ALL the qemu-* system emulators binaries? | 16:42 |
| *** felipemonteiro_ has quit IRC | 16:42 | |
| kashyap | If so, that's the killer. | 16:42 |
| kashyap | Because: | 16:42 |
| kashyap | clarkb: CI starts fresh VMs for _each_ job, yes? | 16:43 |
| fungi | clarkb: right, i wasn't sure whether the fact that the job is defined in the glanceclient repo magically includes it in required-projects when other repos reuse it | 16:43 |
| kashyap | clarkb: In that case, the upstream CI is hitting the full performance hit of libvirt probing *all* QEMU binaries | 16:43 |
| clarkb | kashyap: I'm not sure but we can check a package listing from a job | 16:43 |
| kashyap | clarkb: The solution here is to ensure we _only_ have `qemu-system-x86_64` binary on the system. That makes all the difference between 3 seconds and 60 seconds. | 16:44 |
| kashyap | mriedem: ^ | 16:44 |
| kashyap | mriedem: (Context: Why libvirtd slowdown is observed on Gate.) | 16:44 |
| fungi | (or aarch64 if it's an arm system i suppose) | 16:44 |
| clarkb | kashyap: and that probing is significantly slower post spectre patching? | 16:45 |
| clarkb | kashyap: also should that only happen the first time a VM is booted? or each time? | 16:45 |
| *** d0ugal has quit IRC | 16:45 | |
| kashyap | clarkb: Let me confirm the probing behaviour post-Spectre | 16:45 |
| clarkb | kashyap: http://logs.openstack.org/41/553641/2/check/legacy-tempest-dsvm-neutron-src/e04690e/logs/dpkg-l.txt.gz we appear to install a bunch of them (mips, ppc, sparc, etc) | 16:45 |
| kashyap | clarkb: Yep, that's it | 16:46 |
| kashyap | Not installing them all will / should resolve it, actually. | 16:46 |
| *** e0ne has quit IRC | 16:46 | |
| kashyap | clarkb: Haven't benchmarked the probing behaviour post-Spectre, but it wouldn't be surprising if it got slowed down | 16:47 |
| *** trown|ruck is now known as trown|lunch | 16:47 | |
| kashyap | superdan: You also might find the last 15 messages or so useful (given the discussion on #openstack-nova yesterday) | 16:47 |
| kashyap | superdan: In short: | 16:48 |
| *** iyamahat has quit IRC | 16:48 | |
| *** iyamahat has joined #openstack-infra | 16:48 | |
| kashyap | superdan: The slowdown we see of libvirtd dameon starting itself is (largely) because: the host systems have a lot of QEMU binaries. And libvirt probes them all. Ensuring only the presence -x86 binary alleaviates that. | 16:49 |
| superdan | kashyap: that's a different reason than mnaser said yesterday | 16:49 |
| clarkb | kashyap: this isn't startup time of the libvirt daemon though | 16:49 |
| clarkb | its startup of the cirros VMs themselves | 16:50 |
| kashyap | superdan: Do you recall what did mnaser say yesterday? | 16:50 |
| kashyap | clarkb: Yeah, to that point — | 16:50 |
| superdan | kashyap: yeah, he said that older versions of qemu didn't report cpu features so libvirt was starting a tiny guest to probe those features, which took a long time | 16:50 |
| kashyap | clarkb: Yes, also wouldn't be surprised if plain emulation itself gotten slowed post-Spectre :-( | 16:50 |
| *** yamamoto has joined #openstack-infra | 16:51 | |
| *** fried_rice is now known as fried_rolls | 16:52 | |
| kashyap | clarkb: superdan: So it's a problem with _one_ specific hosting provider, or you see in all envs | 16:53 |
| clarkb | I believe we saw it pretty consistently once uca updated | 16:53 |
| *** armaan has quit IRC | 16:53 | |
| kashyap | clarkb: Also, the hosting providers should have the 'PCID' flag | 16:54 |
| kashyap | clarkb: https://review.openstack.org/#/c/534384/ | 16:54 |
| kashyap | (Still fixing unit tests) | 16:54 |
| clarkb | kashyap: ya that will help with meltdown patching | 16:54 |
| *** yolanda has quit IRC | 16:55 | |
| kashyap | clarkb: I'll just be working on it this / next week. Been a bit slow | 16:55 |
| *** lpetrut has joined #openstack-infra | 16:55 | |
| clarkb | rosmaita: fungi reading the docs override-checkout will override teh change. What you can do instead is provide a list of required-projects with project specific override checkouts | 16:56 |
| *** yamamoto has quit IRC | 16:56 | |
| clarkb | rosmaita: fungi so I think what you want to do in this case is drop the top level override then list all of the required projects with a stable/queens checkout except for glanceclient | 16:56 |
| fungi | clarkb: i wonder how the oslo libs do that for their backward-compat testing | 16:56 |
| clarkb | fungi: probably something similar to ^ ? | 16:57 |
| rosmaita | clarkb: thanks, i did not think to RTFM | 16:57 |
| clarkb | fungi: may also not be using the new native stuff yet in oslo land | 16:57 |
| *** iyamahat has quit IRC | 16:58 | |
| fungi | looks like shade is attempting to do the same in shade-functional-devstack-legacy but has a "# TODO(mordred): This does not seem to work" | 16:58 |
| clarkb | that iptables bug we had on fedora and tumblweed? totally just hit me on my local machine after enabling firewalld after a transition off of the old suse firewall | 16:58 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 16:58 |
| rosmaita | clarkb: can you look at the existing job at line 40 in https://review.openstack.org/#/c/553641/4/.zuul.yaml | 16:59 |
| mordred | rosmaita, clarkb, fungi: so - the thing that doesn't work in the shade repo is booting the old devstack in the first place | 16:59 |
| *** VW has quit IRC | 16:59 | |
| clarkb | rosmaita: that job isn't overriding anything so should be fine right? | 16:59 |
| mordred | rosmaita: in this case, what you want is for devstack to run with stable/queens but then for your tox tests to use master right? | 16:59 |
| rosmaita | mordred exactly | 17:00 |
| *** VW has joined #openstack-infra | 17:00 | |
| rosmaita | clarkb i was wondering about having glanceclient as a required-project | 17:00 |
| rosmaita | since its the thing we want to test | 17:00 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix plugin injection vulnerability https://review.openstack.org/553854 | 17:00 |
| clarkb | rosmaita: that is implied for changes against glanceclient. But if you run the job against say glance changes you will need the required project for glanceclient otherwise it will be installed from pypi instead | 17:01 |
| mordred | rosmaita: just take out the LIBS_FROM_GIT line | 17:01 |
| fungi | though if you're only going to run it on glanceclient changes, then you shouldn't need to have glanceclient in required-projects as zuul assumes the project for the change being tested is required | 17:01 |
| mordred | that way devstack won't try to install glanceclient from git, which you don't want anyway | 17:01 |
| mordred | then, put override-checkout: master in the required-projects list - so that zuul will check out master of glanceclient for you | 17:02 |
| fungi | mordred: though the inventory for the job run indicates zuul also overrode the glanceclient checkout to stable/queens | 17:02 |
| fungi | ahh, yes that bit | 17:02 |
| mordred | yes. that's because of the top-level override-checkout | 17:02 |
| fungi | right | 17:02 |
| fungi | so supplying an override-checkout for master to glanceclient | 17:02 |
| fungi | and then we'll end up on the local master branch where zuul prepped the change being tested | 17:03 |
| mordred | fungi, rosmaita, clarkb: liek this: paste.openstack.org/show/703053 | 17:03 |
| mordred | fungi: exactly | 17:03 |
| *** esberglu has quit IRC | 17:04 | |
| clarkb | ah thats a bit easier than needign to list everything explicitly to use stable queens | 17:04 |
| *** kashyap has left #openstack-infra | 17:04 | |
| *** VW has quit IRC | 17:04 | |
| mordred | if we DID want to install devstack with stable/queens glanceclient from git, then I think we'd just need to put in a (cd ~/src/git.openstack.org/openstack/python-glanceclient ; git checkout master) after devstack but before tox (like in a local pre-playbook) | 17:04 |
| * mnaser is coming into office in a minute | 17:04 | |
| *** harlowja has quit IRC | 17:05 | |
| mordred | since zuul *will* prepare the repo with both master and stable/queens state - so we *can* do both things in the same job ... but in this case we don't need to best I can tell | 17:05 |
| mordred | rosmaita: if you want, I can push those changes up to your patch and see if they make it work - or I can leave it for you to | 17:05 |
| *** gyee has joined #openstack-infra | 17:05 | |
| *** neiljerram has quit IRC | 17:06 | |
| rosmaita | mordred: feel free if you have time, i'm in another meeting and not following the discussion atm | 17:06 |
| *** gfidente has quit IRC | 17:06 | |
| *** stevebaker has quit IRC | 17:06 | |
| *** yamamoto has joined #openstack-infra | 17:06 | |
| *** yamamoto has quit IRC | 17:07 | |
| *** yamamoto has joined #openstack-infra | 17:07 | |
| *** yamamoto has quit IRC | 17:07 | |
| *** udesale has joined #openstack-infra | 17:07 | |
| *** Guest71919 has quit IRC | 17:07 | |
| mordred | rosmaita: cool. I'll push it up and maybe when you get back it'll just be working | 17:08 |
| *** sree has joined #openstack-infra | 17:08 | |
| rosmaita | mordred: you da man! | 17:08 |
| *** olaph1 has joined #openstack-infra | 17:08 | |
| mordred | rosmaita: well... let's wait to see if it works first :) | 17:08 |
| *** olaph has quit IRC | 17:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/550978 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/projects routes https://review.openstack.org/550979 | 17:10 |
| rosmaita | mordred: if it works, you will be promoted from "da man" to "ubermensch" | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Rename javascript package to zuul-dashboard https://review.openstack.org/551999 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: dashboard: add /{tenant}/job.html page to display job details https://review.openstack.org/535545 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: dashboard: add /{tenant}/projects.html web page https://review.openstack.org/537870 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule https://review.openstack.org/545671 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift html templates into components https://review.openstack.org/551327 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use arrow functions for http callbacks https://review.openstack.org/551399 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade to webpack 4 https://review.openstack.org/551987 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Express the bootstrap css depend in css https://review.openstack.org/551988 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5) https://review.openstack.org/551989 | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use glyphicons for status balls https://review.openstack.org/551992 | 17:10 |
| clarkb | dmsimard: comments on https://review.openstack.org/#/c/513874/4 | 17:11 |
| dmsimard | looking | 17:12 |
| *** sree has quit IRC | 17:12 | |
| clarkb | dmsimard: also you may be interested in https://github.com/openstack/ara/pull/8 which should get auto closed (assuming you haven't arleady seen the email for it) | 17:12 |
| *** stevebaker has joined #openstack-infra | 17:12 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix plugin injection vulnerability https://review.openstack.org/553854 | 17:13 |
| dmsimard | clarkb: oh, huh | 17:13 |
| *** zoli is now known as zoli|gone | 17:13 | |
| *** zoli|gone is now known as zoli | 17:13 | |
| dmsimard | clarkb: thanks for pointing that PR out, I would've probably missed that | 17:13 |
| *** diablo_rojo has quit IRC | 17:14 | |
| *** diablo_rojo has joined #openstack-infra | 17:14 | |
| *** iyamahat has joined #openstack-infra | 17:15 | |
| dmsimard | clarkb: good review, I'll send a new patchset, thanks a lot | 17:15 |
| clarkb | dmsimard: no problem | 17:15 |
| dmsimard | clarkb: heh, maybe openstack could start accepting pull requests with zuul v3 (half kidding) | 17:15 |
| * fungi is heading out to grab lunch and will be a while... going further up the beach than usual | 17:15 | |
| *** HeOS has joined #openstack-infra | 17:16 | |
| *** udesale has quit IRC | 17:16 | |
| *** pabelanger has quit IRC | 17:17 | |
| *** pabelanger has joined #openstack-infra | 17:17 | |
| * dmsimard opens a pull request to nova | 17:17 | |
| clarkb | dmsimard: there are tools to slurp PRs from github into gerrit, but I'm somewhat resistant to having multiple ways to contribute code to a project. I think it will only cause confusion and be yet another thing to debug when it goes wrong | 17:17 |
| dmsimard | ah yeah, I've seen that | 17:18 |
| dmsimard | they've called it.. gerritbot I think? :) | 17:18 |
| clarkb | I'm extra convinced of that after having all sorts of issues with this pip PR. I swear github accepted the push, ran new jobs due to event being emitted then simply lost the code | 17:18 |
| clarkb | dmsimard: gerrithub is the service, I'm not sure what the plugin is called | 17:18 |
| clarkb | of course that could've all been pebkac but it ran the jobs after I pushed, just didn't seem to update the code :/ | 17:21 |
| *** rpittau has quit IRC | 17:22 | |
| *** ykarel|afk has quit IRC | 17:24 | |
| *** ianychoi__ has joined #openstack-infra | 17:24 | |
| *** lyaaaaaaaaaaaaar is now known as lyarwood | 17:27 | |
| *** ianychoi_ has quit IRC | 17:27 | |
| *** jaosorior has quit IRC | 17:28 | |
| * clarkb grabs breakfast will be back to rereview ara things | 17:28 | |
| *** sree has joined #openstack-infra | 17:29 | |
| *** VW has joined #openstack-infra | 17:30 | |
| *** yamahata has joined #openstack-infra | 17:31 | |
| *** jpich has quit IRC | 17:32 | |
| *** camunoz has joined #openstack-infra | 17:33 | |
| *** stevebaker has quit IRC | 17:33 | |
| *** jtomasek has quit IRC | 17:33 | |
| *** sree has quit IRC | 17:33 | |
| *** myoung|rover is now known as myoung|bbl | 17:34 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Make Zuul able to start with a broken config https://review.openstack.org/535511 | 17:34 |
| *** VW has quit IRC | 17:35 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:35 | |
| *** lucasagomes is now known as lucas-pto | 17:35 | |
| *** olaph1 has quit IRC | 17:37 | |
| *** olaph has joined #openstack-infra | 17:38 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix builds queued forever after failure to get node request https://review.openstack.org/537335 | 17:40 |
| *** VW has joined #openstack-infra | 17:41 | |
| *** stevebaker has joined #openstack-infra | 17:42 | |
| *** andreas_s has quit IRC | 17:42 | |
| *** imacdonn has quit IRC | 17:44 | |
| *** sree has joined #openstack-infra | 17:44 | |
| *** imacdonn has joined #openstack-infra | 17:45 | |
| *** VW has quit IRC | 17:46 | |
| *** weshay is now known as weshay_brb | 17:47 | |
| *** sree has quit IRC | 17:49 | |
| *** pblaho has quit IRC | 17:50 | |
| clarkb | dmsimard: I also left a comment on the base-test update. But I +2'd it anyway. Will let you decide if it deserves an update | 17:50 |
| *** rodrigod` has quit IRC | 17:57 | |
| *** rodrigod` has joined #openstack-infra | 17:57 | |
| *** kjackal has quit IRC | 17:58 | |
| *** r-daneel has joined #openstack-infra | 17:59 | |
| *** sree has joined #openstack-infra | 18:00 | |
| *** jpena is now known as jpena|off | 18:01 | |
| *** harlowja has joined #openstack-infra | 18:01 | |
| *** harlowja_ has joined #openstack-infra | 18:04 | |
| *** trown|lunch is now known as trown|ruck | 18:04 | |
| *** sree has quit IRC | 18:05 | |
| *** harlowja has quit IRC | 18:06 | |
| *** amoralej is now known as amoralej|off | 18:06 | |
| *** yamamoto has joined #openstack-infra | 18:08 | |
| *** hemna_ has joined #openstack-infra | 18:08 | |
| *** kjackal has joined #openstack-infra | 18:08 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix plugin injection vulnerability https://review.openstack.org/553854 | 18:09 |
| * clarkb checks when puppet should update our executors | 18:10 | |
| clarkb | just as soon as I load my ssh key apparently | 18:10 |
| *** panda is now known as panda|off | 18:11 | |
| pabelanger | looks like another 5mins for it to run | 18:12 |
| *** germs has joined #openstack-infra | 18:12 | |
| *** germs has quit IRC | 18:12 | |
| *** germs has joined #openstack-infra | 18:12 | |
| clarkb | ya 1815 will start the run then about 10 minutes or so after that should update the executors | 18:12 |
| *** bhujay has quit IRC | 18:13 | |
| *** mib_zlovm0 has joined #openstack-infra | 18:13 | |
| *** yamamoto has quit IRC | 18:14 | |
| pabelanger | ++ | 18:14 |
| *** jamesmcarthur has joined #openstack-infra | 18:15 | |
| *** ldnunes has quit IRC | 18:15 | |
| *** germs has quit IRC | 18:17 | |
| *** VW has joined #openstack-infra | 18:17 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add zuul-web endpoint for getting configuration errors for a tenant https://review.openstack.org/553873 | 18:19 |
| *** jamesmcarthur has quit IRC | 18:20 | |
| *** felipemonteiro__ has joined #openstack-infra | 18:20 | |
| *** VW has quit IRC | 18:22 | |
| clarkb | its starting the everything else puppet runs | 18:22 |
| clarkb | should be soon now | 18:22 |
| *** lyarwood is now known as lyaaaaaaaaaaaaar | 18:22 | |
| *** lyaaaaaaaaaaaaar has quit IRC | 18:22 | |
| pabelanger | yah, tailing puppet log | 18:22 |
| *** felipemonteiro_ has quit IRC | 18:23 | |
| *** olaph1 has joined #openstack-infra | 18:26 | |
| *** olaph has quit IRC | 18:26 | |
| *** ldnunes has joined #openstack-infra | 18:28 | |
| jlvillal | gerritbot review request: https://review.openstack.org/#/c/545469/ Some cleanup/refactoring and adding unit tests. Has one +2 Thanks! | 18:30 |
| clarkb | ok I think all executors have installed the latest change | 18:30 |
| clarkb | I'm going to stop them all now, wait for them to stop, then start them with new code | 18:30 |
| clarkb | infra-root ^ fyi | 18:31 |
| pabelanger | wfm | 18:31 |
| *** ralonsoh_ has quit IRC | 18:31 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:32 | |
| *** felipemonteiro__ has quit IRC | 18:35 | |
| clarkb | pabelanger: also have you confirmed the update to write symlinks at run time has corrected your job issue copying zuul src? | 18:37 |
| pabelanger | clarkb: I did, was able to do a depends-on once tobiash had a patch up | 18:38 |
| pabelanger | and sync worked again | 18:38 |
| clarkb | cool, just wanted to make sure there wasn't also some other issue with that change | 18:39 |
| clarkb | waiting on 3 more executors to stop | 18:39 |
| clarkb | and restarted, we should be running the latest executor code now | 18:40 |
| *** eernst has joined #openstack-infra | 18:40 | |
| pabelanger | ack | 18:42 |
| clarkb | #status log Restarted zuul-executors to pick up fix in https://review.openstack.org/553854 | 18:42 |
| openstackstatus | clarkb: finished logging | 18:43 |
| *** sambetts is now known as sambetts|afk | 18:48 | |
| *** VW has joined #openstack-infra | 18:53 | |
| *** kjackal has quit IRC | 18:53 | |
| *** felipemonteiro_ has quit IRC | 18:54 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:54 | |
| *** florianf has quit IRC | 18:55 | |
| *** olaph1 is now known as olaph | 18:57 | |
| *** VW has quit IRC | 18:58 | |
| *** efoley has quit IRC | 18:59 | |
| *** pcichy has joined #openstack-infra | 18:59 | |
| *** weshay_brb is now known as weshay | 18:59 | |
| *** tesseract has quit IRC | 19:00 | |
| *** fried_rolls is now known as fried_rice | 19:02 | |
| pabelanger | clarkb: corvus: Regarding the restart above, would we in ever consider a trusted job to allow custom plugins? Specifically, OSA has written a connection / strategy plugin to leverage container things atop of the ssh connection | 19:04 |
| *** VW has joined #openstack-infra | 19:05 | |
| clarkb | pabelanger: I think it is just untrusted taht is restricted | 19:05 |
| *** janki has quit IRC | 19:05 | |
| clarkb | ya the checking is specifically under if not trusted: | 19:06 |
| pabelanger | clarkb: right, we'd have to create a job in project-config now for connection / strategy plugins. I wasn't sure if we (openstack-infra) wanted to do that or not | 19:06 |
| pabelanger | since we'd need to be okay with that code running on executors | 19:07 |
| pabelanger | in bwrap | 19:07 |
| clarkb | I think as long as it came from a known source and we reviewed it it would be fine? | 19:07 |
| pabelanger | yah, i think there might be some design issue today, as it would allow a job to arbitrary define which IP address an ansible-playbook process would SSH to | 19:09 |
| *** yamamoto has joined #openstack-infra | 19:10 | |
| clarkb | wouldn't the connection strategy apply to your inventory? | 19:10 |
| clarkb | itshouldn't be arbitrary | 19:10 |
| *** VW has quit IRC | 19:10 | |
| pabelanger | well, this plugin specifically overloads the default ssh connection to accept new variables, once set, it will use that variable, not ansible_host for SSH | 19:11 |
| *** shardy has quit IRC | 19:11 | |
| pabelanger | http://git.openstack.org/cgit/openstack/openstack-ansible-plugins/tree/connection/ssh.py | 19:11 |
| pabelanger | either way, nothing that needs to be solved today from an executor, since it jobs could just run ansible on the remove node them self and not executor | 19:12 |
| *** hemna_ has quit IRC | 19:12 | |
| *** felipemonteiro__ has joined #openstack-infra | 19:13 | |
| *** felipemonteiro__ has quit IRC | 19:16 | |
| *** felipemonteiro__ has joined #openstack-infra | 19:17 | |
| *** felipemonteiro_ has quit IRC | 19:17 | |
| clarkb | seems like a better approach would be to have your playbooks create the containers as necessary then add hosts dynamically to the inventory and conenct to them using normal ssh? | 19:17 |
| *** wolverineav has joined #openstack-infra | 19:18 | |
| clarkb | if you are using "thin" containers you shouldn't be connecting to them at all and if using full os containers that should be fine? | 19:18 |
| pabelanger | clarkb: yah, that is an option but from zuul-executor we'd block adding hosts into an inventory | 19:19 |
| pabelanger | clarkb: the connection plugin uses lxc-attach IIRC | 19:19 |
| *** yamamoto has quit IRC | 19:20 | |
| *** jamesmcarthur has joined #openstack-infra | 19:23 | |
| *** jamesmcarthur has quit IRC | 19:26 | |
| *** beagles is now known as beagles_biab | 19:26 | |
| *** VW has joined #openstack-infra | 19:27 | |
| *** jamesmcarthur has joined #openstack-infra | 19:28 | |
| *** andreas_s has joined #openstack-infra | 19:29 | |
| *** pcichy has quit IRC | 19:31 | |
| *** pcichy has joined #openstack-infra | 19:31 | |
| *** VW has quit IRC | 19:32 | |
| *** andreas_s has quit IRC | 19:33 | |
| *** pcichy has quit IRC | 19:34 | |
| *** lpetrut has quit IRC | 19:37 | |
| *** eumel8 has quit IRC | 19:41 | |
| *** jtomasek has joined #openstack-infra | 19:44 | |
| *** jtomasek has quit IRC | 19:45 | |
| *** r-daneel_ has joined #openstack-infra | 19:45 | |
| *** r-daneel has quit IRC | 19:47 | |
| *** r-daneel_ is now known as r-daneel | 19:47 | |
| mnaser | quiet day today | 19:48 |
| corvus | mnaser: "could be worse. could be raining." | 19:49 |
| *** myoung|bbl is now known as myoung|rover | 19:50 | |
| *** mib_zlovm0 has quit IRC | 19:51 | |
| fungi | clarkb: as far as pip pull requests, https://mail.python.org/pipermail/distutils-sig/2018-March/032047.html suggests there's probably not room to get much else in before they tag 10.0.0 | 19:55 |
| *** bobh has quit IRC | 19:55 | |
| clarkb | fungi: seems like they want people to test it and provide fixes at least? | 19:56 |
| fungi | and arguments in favor of even providing a workaround to distutils package "uninstall" failures haven't seemed to sway the maintainers, so i expect we're stuck doing virtualenvs sooner rather than later | 19:56 |
| clarkb | fun | 19:56 |
| *** eumel8 has joined #openstack-infra | 19:57 | |
| mnaser | corvus: :D | 19:59 |
| *** eumel8 has quit IRC | 20:02 | |
| *** sree has joined #openstack-infra | 20:07 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Offset cron jobs slightly https://review.openstack.org/553894 | 20:09 |
| *** felipemonteiro_ has joined #openstack-infra | 20:10 | |
| ianw | pabelanger: so, overnight it seems ubuntu-ports managed to get itself corrupted, BUT we at least have something different in the logs for afs | 20:10 |
| ianw | "find_preferred_connection: no connection and !create" | 20:10 |
| pabelanger | yah, haven't seen that before | 20:11 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix runtime stats reporting for noop job https://review.openstack.org/537432 | 20:11 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix self fulfilling empty node requests https://review.openstack.org/544236 | 20:11 |
| ianw | https://github.com/openafs/openafs/blob/master/src/afs/afs_conn.c#L73 ... while i'm not sure what that means exactly; it might a clue for a tunable however? will have to ask on the afs lists probably | 20:11 |
| *** sree has quit IRC | 20:11 | |
| *** germs has joined #openstack-infra | 20:13 | |
| mnaser | ouch | 20:13 |
| *** felipemonteiro__ has quit IRC | 20:14 | |
| AJaeger | config-core, if you have some spare cycles, please go over the config repos and review and approve open changes. We have again a long backlog... | 20:16 |
| * AJaeger wishes everybody a great weekend and signs off for now | 20:16 | |
| *** jamesmcarthur has quit IRC | 20:16 | |
| *** yamamoto has joined #openstack-infra | 20:17 | |
| fungi | thanks AJaeger, and have a great weekend! | 20:17 |
| *** germs has quit IRC | 20:18 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: enable systemd profile for Gentoo https://review.openstack.org/552952 | 20:18 |
| fungi | prometheanfire: ^ that looks defeatist | 20:21 |
| * fungi thought gentoo was one of the last non-systemd holdouts | 20:21 | |
| prometheanfire | fungi: we allow the user to choose | 20:22 |
| *** yamamoto has quit IRC | 20:22 | |
| fungi | fair enough | 20:22 |
| prometheanfire | default continues to be openrc (and we don't plan on changing that default) | 20:23 |
| prometheanfire | I personally use systemd on my laptop and openrc on my servers | 20:23 |
| prometheanfire | a little old, but ok | 20:23 |
| prometheanfire | https://rich0gentoo.wordpress.com/2016/02/27/gentoo-ought-to-be-about-choice/ | 20:23 |
| fungi | debian did that for a while too (ostensibly still does, but there's increasing pushback from package maintainers to accept fixes for initscripts now) | 20:25 |
| fungi | granted, debian's sysvinit wasn't as advanced as openrc, so there was some increased pressure to change the default to *something* anyway | 20:26 |
| *** rlandy is now known as rlandy|biab | 20:27 | |
| fungi | but once systemd became the default, that seems to have spelled the beginning of the end for ubiquitous sysvinit support | 20:27 |
| *** dprince has quit IRC | 20:29 | |
| *** hemna_ has joined #openstack-infra | 20:29 | |
| *** olaph1 has joined #openstack-infra | 20:33 | |
| *** jcoufal has quit IRC | 20:34 | |
| *** olaph has quit IRC | 20:34 | |
| ianw | pabelanger: oh, i might have been too hasty blaming afs ... i think the ubuntu-ports error was actually probably the remote mirror "Wrong checksum during receive of 'http://ports.ubuntu.com/ubuntu-ports/dists/bionic/main/binary-arm64/Packages.xz'" | 20:37 |
| ianw | infra-root: (hopefully) easy one i'd like sanity checked -> https://review.openstack.org/553697 . we're removing packages that end up removing bup, so backups aren't working on zuul | 20:39 |
| *** bobh has joined #openstack-infra | 20:40 | |
| *** Goneri has quit IRC | 20:41 | |
| *** hemna_ has quit IRC | 20:42 | |
| corvus | ianw: i think it's safe to remove the removals now that we're well across the line to the new js stuff. | 20:42 |
| ianw | thanks; i thought so, but there's a lot of deep javascript magic going on now :) | 20:43 |
| *** claudiub|2 has quit IRC | 20:47 | |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config master: Add pypi jobs to sphinx-feature-classification https://review.openstack.org/553897 | 20:48 |
| *** kgiusti has left #openstack-infra | 20:49 | |
| *** eernst has quit IRC | 20:49 | |
| logan- | dmsimard: re: next steps for my cloud.. is there anything I can help with? happy to hack on some patches if it would help | 20:52 |
| dmsimard | logan-: hey, the next step is likely to stand up the mirror and then we can start on the nodepool bits | 20:54 |
| dmsimard | logan-: unfortunately my day has been pretty busy and I'm on something I can't drop right now | 20:54 |
| *** olaph has joined #openstack-infra | 20:55 | |
| *** ldnunes has quit IRC | 20:55 | |
| *** ldnunes has joined #openstack-infra | 20:56 | |
| *** owalsh has quit IRC | 20:57 | |
| *** olaph1 has quit IRC | 20:57 | |
| logan- | ok np. thanks. | 20:58 |
| *** owalsh has joined #openstack-infra | 20:58 | |
| fungi | looking through http://codesearch.openstack.org/?q=collectd-ceilometer-plugin there don't seem to be in-repo zuul configs which will spontaneously break (though they do have some jobs in there they'll probably want to rename after), and one reference in devstack's plugin-registry documentation, but in theory if we plan to just force merge a project-config change then we shouldn't have any further | 20:59 |
| fungi | complication right? | 20:59 |
| fungi | and did i read correctly that we have consensus behind monolithic project-config change force merged rather than two to do the removal before scheduler restart and two more to reintroduce the new name after? | 20:59 |
| *** trown|ruck is now known as trown|outtypewww | 21:00 | |
| *** dave-mccowan has quit IRC | 21:01 | |
| *** camunoz has quit IRC | 21:01 | |
| *** owalsh_ has joined #openstack-infra | 21:01 | |
| *** edmondsw has quit IRC | 21:03 | |
| *** owalsh- has joined #openstack-infra | 21:03 | |
| *** owalsh has quit IRC | 21:03 | |
| clarkb | fungi: sounds like it just to reduce the number of moving parts | 21:05 |
| *** felipemonteiro_ has quit IRC | 21:05 | |
| *** owalsh_ has quit IRC | 21:06 | |
| *** rlandy|biab is now known as rlandy | 21:06 | |
| *** owalsh- has quit IRC | 21:10 | |
| *** owalsh has joined #openstack-infra | 21:15 | |
| *** ldnunes has quit IRC | 21:15 | |
| thingee | diablo_rojo: I'm not sure if the storyboard migration already covers this, but openstack/releases is tracking launchpad/storyboard ids in deliverable files https://review.openstack.org/#/c/553900/ | 21:15 |
| *** gouthamr has quit IRC | 21:16 | |
| fungi | starting to wonder, if we bypass code review for the project-config change is there any reason to even stop the zuul scheduler? now that we can do online reindexing i wonder if we can't just leave zuul running for the few seconds we take gerrit down running the rename_repos playbook | 21:18 |
| *** yamamoto has joined #openstack-infra | 21:18 | |
| fungi | so 1. stop puppet, 2. run ansible playbook, 3. bypass code review to merge project-config change, 4. do github renames/transfers, 5. start puppet again | 21:19 |
| fungi | infra-root: wagers on what i'm missing there that's dangerous not to stop/start zuul for? | 21:20 |
| tobiash | fungi: I think you should be able to let zuul alive if you trigger a full reload afterwards | 21:20 |
| tobiash | (sighup) | 21:20 |
| dmsimard | fungi: we're working on a story in zuul-security but we don't have anything yet -- tobiash and corvus can probably chime in | 21:20 |
| fungi | dmsimard: yeah, i've been following that one | 21:21 |
| fungi | my question is definitely non-urgent ;) | 21:21 |
| *** jamesmcarthur has joined #openstack-infra | 21:22 | |
| *** yamamoto has quit IRC | 21:23 | |
| clarkb | fungi: zuul could race gerrit for reporting results | 21:24 |
| clarkb | fungi: which could lead to losing merges | 21:24 |
| clarkb | its probably not a major ocncern if done during a quiet period though | 21:24 |
| fungi | yeah, i suppose the real concern is if there are changes being tested for the repos we're renaming at the time | 21:24 |
| *** e0ne has joined #openstack-infra | 21:25 | |
| fungi | and zuul attempts to report on or merge them between when the playbook brings gerrit back online and when we bypass code review to merge the project-config change | 21:25 |
| fungi | which we could keep really short, but i guess it depends on how paranoid we want to be about it | 21:26 |
| corvus | fungi: i *think* it'll be okay. the force-merge will trigger a tenant reconfiguration which will fail, but the subsequent full reconfiguration (which puppet will cause, or we can speed up by manually triggering) should then succeed. | 21:28 |
| corvus | (a failed tenant reconfiguration is okay in this case, zuul will just continue using the old config until the full reconfiguration) | 21:29 |
| fungi | so as tobiash indicates, as long as we follow the project-config merge with an immediate zuul reload we should be fine? | 21:29 |
| fungi | do we need to delay long enough for zuul to process any queued events/results before reloading? | 21:30 |
| corvus | fungi: (re 21:29) yep, we should be fine -- even if we don't do it right away, it just means config changes won't take effect between then and the full reconfiguration. so it's not time-critical. just "the sooner the better" | 21:31 |
| diablo_rojo | thingee, don't know that it does, but that is interesting. Thanks for the heads up | 21:31 |
| *** rkukura has quit IRC | 21:32 | |
| corvus | fungi: (re 21:30) no, we should be able to relead at any time. the only things that would be weird related to that is if the renamed projects were active. but hopefully it should just ignore any events for "unknown projects" | 21:32 |
| *** iyamahat has quit IRC | 21:34 | |
| *** iyamahat has joined #openstack-infra | 21:35 | |
| fungi | seems like something worth trying. if it works, then waaay simpler than our old process | 21:35 |
| *** eernst has joined #openstack-infra | 21:35 | |
| *** olaph has quit IRC | 21:42 | |
| *** olaph has joined #openstack-infra | 21:46 | |
| *** d0ugal has joined #openstack-infra | 21:47 | |
| *** d0ugal has quit IRC | 21:47 | |
| *** d0ugal has joined #openstack-infra | 21:47 | |
| *** jamesmcarthur has quit IRC | 21:48 | |
| *** stevebaker has quit IRC | 21:56 | |
| *** VW has joined #openstack-infra | 21:58 | |
| *** myoung|rover is now known as myoung|afk | 21:59 | |
| *** VW has quit IRC | 22:02 | |
| *** imacdonn has quit IRC | 22:07 | |
| *** HeOS has quit IRC | 22:07 | |
| *** knikolla has quit IRC | 22:07 | |
| *** lifeless has quit IRC | 22:07 | |
| *** hongbin has quit IRC | 22:07 | |
| *** gaurangt has quit IRC | 22:07 | |
| *** Diabelko has quit IRC | 22:07 | |
| *** adam_g has quit IRC | 22:07 | |
| *** slaweq has quit IRC | 22:07 | |
| *** krtaylor has quit IRC | 22:07 | |
| *** SergeyLukjanov has quit IRC | 22:07 | |
| *** AJaeger has quit IRC | 22:07 | |
| *** tpsilva has quit IRC | 22:07 | |
| *** pahuang has quit IRC | 22:07 | |
| *** larainema has quit IRC | 22:07 | |
| *** olivierbourdon38 has quit IRC | 22:07 | |
| *** dkehn has quit IRC | 22:07 | |
| *** _Cyclone_ has quit IRC | 22:07 | |
| *** spiffxp has quit IRC | 22:07 | |
| *** Krenair has quit IRC | 22:07 | |
| *** imacdonn has joined #openstack-infra | 22:07 | |
| *** HeOS has joined #openstack-infra | 22:07 | |
| *** knikolla has joined #openstack-infra | 22:07 | |
| *** lifeless has joined #openstack-infra | 22:07 | |
| *** hongbin has joined #openstack-infra | 22:07 | |
| *** gaurangt has joined #openstack-infra | 22:07 | |
| *** Diabelko has joined #openstack-infra | 22:07 | |
| *** adam_g has joined #openstack-infra | 22:07 | |
| *** dkehn has joined #openstack-infra | 22:07 | |
| *** slaweq has joined #openstack-infra | 22:07 | |
| *** krtaylor has joined #openstack-infra | 22:07 | |
| *** SergeyLukjanov has joined #openstack-infra | 22:07 | |
| *** AJaeger has joined #openstack-infra | 22:07 | |
| *** tpsilva has joined #openstack-infra | 22:07 | |
| *** pahuang has joined #openstack-infra | 22:07 | |
| *** larainema has joined #openstack-infra | 22:07 | |
| *** olivierbourdon38 has joined #openstack-infra | 22:07 | |
| *** _Cyclone_ has joined #openstack-infra | 22:07 | |
| *** spiffxp has joined #openstack-infra | 22:07 | |
| *** Krenair has joined #openstack-infra | 22:07 | |
| *** lifeless has quit IRC | 22:08 | |
| *** lifeless has joined #openstack-infra | 22:08 | |
| *** r-daneel has quit IRC | 22:08 | |
| *** bandini has quit IRC | 22:09 | |
| *** ruhe has quit IRC | 22:09 | |
| *** olaph has quit IRC | 22:09 | |
| *** d0ugal has quit IRC | 22:10 | |
| *** bobh has quit IRC | 22:11 | |
| *** bobh has joined #openstack-infra | 22:11 | |
| *** olaph has joined #openstack-infra | 22:12 | |
| *** efoley has joined #openstack-infra | 22:12 | |
| *** bandini has joined #openstack-infra | 22:12 | |
| *** ruhe has joined #openstack-infra | 22:12 | |
| *** e0ne has quit IRC | 22:13 | |
| *** germs has joined #openstack-infra | 22:14 | |
| *** germs has quit IRC | 22:14 | |
| *** germs has joined #openstack-infra | 22:14 | |
| *** e0ne has joined #openstack-infra | 22:15 | |
| *** salv-orlando has quit IRC | 22:15 | |
| *** salv-orlando has joined #openstack-infra | 22:16 | |
| *** bobh has quit IRC | 22:16 | |
| *** olaph1 has joined #openstack-infra | 22:18 | |
| *** germs has quit IRC | 22:18 | |
| *** e0ne has quit IRC | 22:18 | |
| *** e0ne has joined #openstack-infra | 22:18 | |
| *** olaph has quit IRC | 22:19 | |
| *** yamamoto has joined #openstack-infra | 22:20 | |
| *** e0ne has quit IRC | 22:20 | |
| *** yamamoto has quit IRC | 22:26 | |
| *** ihrachys_ has joined #openstack-infra | 22:28 | |
| *** ihrachys has quit IRC | 22:31 | |
| *** VW has joined #openstack-infra | 22:32 | |
| *** eernst has quit IRC | 22:33 | |
| *** rlandy_ has joined #openstack-infra | 22:35 | |
| *** VW has quit IRC | 22:37 | |
| *** rlandy has quit IRC | 22:37 | |
| *** eernst has joined #openstack-infra | 22:37 | |
| clarkb | I'm answering the AJaeger call to review config projects and https://review.openstack.org/#/c/548909/ looks like one corvus should review. I think you talked to notmyname about that plan at the PTG | 22:45 |
| corvus | lgtm thx | 22:46 |
| *** fried_rice is now known as efried | 22:48 | |
| *** hongbin has quit IRC | 22:48 | |
| *** ihrachys_ has quit IRC | 22:48 | |
| clarkb | mriedem: do your comments on the devstack experimental jobs removal indicate you'd like to see the cells job remain? | 22:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 22:52 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove nodejs-npm-run-lint pipeline stanza from storyboard-webclient https://review.openstack.org/545726 | 22:56 |
| *** ihrachys has joined #openstack-infra | 22:57 | |
| *** rhagarty has quit IRC | 22:57 | |
| *** superdan is now known as dansmith | 22:58 | |
| *** rlandy__ has joined #openstack-infra | 23:03 | |
| *** VW has joined #openstack-infra | 23:05 | |
| *** rlandy_ has quit IRC | 23:06 | |
| *** rodrigod` is now known as rodrigods | 23:09 | |
| *** VW has quit IRC | 23:10 | |
| *** olaph has joined #openstack-infra | 23:11 | |
| *** olaph1 has quit IRC | 23:12 | |
| *** iyamahat has quit IRC | 23:12 | |
| *** iyamahat_ has joined #openstack-infra | 23:12 | |
| *** rlandy__ has quit IRC | 23:17 | |
| clarkb | corvus: you might also have thoughts on https://review.openstack.org/#/c/531432/2 which sets some default variable values on our best job(s) | 23:20 |
| clarkb | s/on our best/on our base/ | 23:20 |
| clarkb | they may be our best jobs too | 23:20 |
| openstackgerrit | Merged openstack-infra/project-config master: added swiftstack/ProxyFS project to zuul config to enable testing https://review.openstack.org/548909 | 23:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Import config_template to openstack(-ansible) https://review.openstack.org/546678 | 23:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Add #openstack-self-healing to gerritbot https://review.openstack.org/550002 | 23:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Retire ironic-inspector-tempest-plugin https://review.openstack.org/550537 | 23:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Setup Queens translation sync for tripleo-ui https://review.openstack.org/550814 | 23:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Publish docs for networking-generic-switch https://review.openstack.org/552547 | 23:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Add networking-sfc to periodic-stable https://review.openstack.org/552456 | 23:22 |
| openstackgerrit | Merged openstack-infra/project-config master: gerritbot: send kolla/kolla-ansible stable notifications to #osism channel https://review.openstack.org/552189 | 23:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Retire puppet-apps_site https://review.openstack.org/552884 | 23:22 |
| *** yamamoto has joined #openstack-infra | 23:22 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add document for python-tackerclient https://review.openstack.org/552448 | 23:22 |
| corvus | clarkb: i think we may want to be careful about that one. left a -1 | 23:24 |
| *** bobh has joined #openstack-infra | 23:25 | |
| *** yamamoto has quit IRC | 23:27 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: remove portage git directory https://review.openstack.org/553914 | 23:30 |
| jlvillal | mordred, Had a question on the PTI document build. There is a patch: https://review.openstack.org/#/c/553473/ trying to use the new way. | 23:37 |
| jlvillal | mordred, But notice that the job runs it the old way. Only problem is that when run tox now, it fails. But doesn't fail in the gate since it doesn't use tox. It runs the "python setup.py build_sphinx" in the gate. | 23:38 |
| jlvillal | mordred, So seems when things switch the job will fail. | 23:39 |
| *** VW has joined #openstack-infra | 23:39 | |
| *** wolverineav has quit IRC | 23:39 | |
| clarkb | jlvillal: it should be self testing | 23:39 |
| *** wolverineav has joined #openstack-infra | 23:39 | |
| clarkb | jlvillal: what makes you think the job runs the old way? | 23:39 |
| jlvillal | clarkb, Because of: https://github.com/openstack-infra/zuul-jobs/blob/d28c326b36df44a7d84891609e88d3ccf976b4d5/roles/sphinx/tasks/main.yaml#L50-L62 | 23:40 |
| jlvillal | clarkb, From the job run: http://logs.openstack.org/73/553473/9/check/build-openstack-sphinx-docs/64f6c24/job-output.txt.gz#_2018-03-16_17_23_42_449126 | 23:40 |
| clarkb | ah I see you have the legacy autodoc | 23:41 |
| clarkb | I think you just turn that off and it will use the new thing? | 23:41 |
| jlvillal | clarkb, How do we turn it off? | 23:41 |
| clarkb | jlvillal: I think you have to remove the config for it from your setup.cfg then add it into your sphinx config | 23:41 |
| jlvillal | clarkb, Thanks I will look | 23:41 |
| clarkb | jlvillal: but also note that once the change merges it will continue to use that pbr method aiui so won't break | 23:41 |
| clarkb | its just your new tox.ini command won't line up with what is run | 23:42 |
| jlvillal | clarkb, Thanks. Will investigate it more. | 23:43 |
| *** VW has quit IRC | 23:43 | |
| *** tpsilva has quit IRC | 23:46 | |
| *** r-daneel has joined #openstack-infra | 23:47 | |
| *** openstackgerrit has quit IRC | 23:48 | |
| *** jgwentworth is now known as melwitt | 23:48 | |
| *** r-daneel has quit IRC | 23:51 | |
| *** eernst has quit IRC | 23:54 | |
| *** gyee has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!