| *** heyongli has quit IRC | 00:07 | |
| *** heyongli has joined #openstack-infra | 00:07 | |
| *** cjloader has joined #openstack-infra | 00:11 | |
| cjloader | hello all | 00:11 |
|---|---|---|
| clarkb | hello | 00:15 |
| *** dingyichen has joined #openstack-infra | 00:16 | |
| *** heyongli has quit IRC | 00:17 | |
| *** heyongli has joined #openstack-infra | 00:18 | |
| *** heyongli has quit IRC | 00:27 | |
| *** heyongli has joined #openstack-infra | 00:28 | |
| *** rpioso is now known as rpioso|afk | 00:29 | |
| prometheanfire | hello | 00:29 |
| *** bobh has joined #openstack-infra | 00:31 | |
| *** gyee has quit IRC | 00:32 | |
| *** jesslampe has joined #openstack-infra | 00:32 | |
| *** Dinesh_Bhor has joined #openstack-infra | 00:37 | |
| *** heyongli has quit IRC | 00:38 | |
| *** heyongli has joined #openstack-infra | 00:38 | |
| *** bobh has quit IRC | 00:41 | |
| openstackgerrit | Merged openstack-infra/zuul master: web: fix status page flickering https://review.openstack.org/578226 | 00:46 |
| *** rwsu has quit IRC | 00:48 | |
| *** heyongli has quit IRC | 00:48 | |
| *** heyongli has joined #openstack-infra | 00:48 | |
| *** zhangfei has joined #openstack-infra | 00:53 | |
| *** masber has joined #openstack-infra | 00:58 | |
| *** heyongli has quit IRC | 00:58 | |
| *** heyongli has joined #openstack-infra | 00:59 | |
| *** mriedem has quit IRC | 01:06 | |
| *** heyongli has quit IRC | 01:09 | |
| *** heyongli has joined #openstack-infra | 01:09 | |
| *** diablo_rojo has joined #openstack-infra | 01:12 | |
| *** shu-mutow has quit IRC | 01:14 | |
| *** mriedem has joined #openstack-infra | 01:15 | |
| *** caphrim007 has joined #openstack-infra | 01:19 | |
| *** heyongli has quit IRC | 01:19 | |
| *** heyongli has joined #openstack-infra | 01:19 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move localloop to exec_sudo https://review.openstack.org/578616 | 01:23 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Call kpartx remove in umount, not cleanup https://review.openstack.org/578657 | 01:23 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Only detach device if all partitions have been cleaned https://review.openstack.org/576876 | 01:23 |
| *** caphrim007 has quit IRC | 01:23 | |
| ianw | one mock, two mock, red mock, blue mock | 01:24 |
| *** zhangfei has quit IRC | 01:24 | |
| *** caphrim007 has joined #openstack-infra | 01:25 | |
| *** heyongli has quit IRC | 01:29 | |
| *** mriedem has quit IRC | 01:29 | |
| *** heyongli has joined #openstack-infra | 01:29 | |
| *** gagehugo has joined #openstack-infra | 01:30 | |
| *** yamahata has quit IRC | 01:34 | |
| *** lastmikoi has quit IRC | 01:36 | |
| *** zhangfei has joined #openstack-infra | 01:36 | |
| *** heyongli has quit IRC | 01:39 | |
| *** heyongli has joined #openstack-infra | 01:40 | |
| *** jesslampe has quit IRC | 01:46 | |
| *** auristor has quit IRC | 01:47 | |
| *** caphrim007_ has joined #openstack-infra | 01:48 | |
| *** lastmikoi has joined #openstack-infra | 01:49 | |
| *** auristor has joined #openstack-infra | 01:49 | |
| *** heyongli has quit IRC | 01:50 | |
| *** heyongli has joined #openstack-infra | 01:50 | |
| *** jesslampe has joined #openstack-infra | 01:50 | |
| *** jesslampe has quit IRC | 01:51 | |
| *** caphrim007 has quit IRC | 01:52 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add selenium tests for the status page https://review.openstack.org/579039 | 01:55 |
| *** tonytan4ever_brb has joined #openstack-infra | 01:58 | |
| *** salv-orlando has joined #openstack-infra | 01:58 | |
| *** heyongli has quit IRC | 02:00 | |
| *** heyongli has joined #openstack-infra | 02:00 | |
| *** tonytan4ever has quit IRC | 02:01 | |
| *** salv-orlando has quit IRC | 02:03 | |
| *** tinwood has quit IRC | 02:09 | |
| *** heyongli has quit IRC | 02:10 | |
| *** heyongli has joined #openstack-infra | 02:10 | |
| *** tinwood has joined #openstack-infra | 02:10 | |
| *** heyongli has quit IRC | 02:20 | |
| *** heyongli has joined #openstack-infra | 02:21 | |
| *** dpawlik has joined #openstack-infra | 02:29 | |
| *** heyongli has quit IRC | 02:31 | |
| *** heyongli has joined #openstack-infra | 02:31 | |
| *** dpawlik has quit IRC | 02:34 | |
| *** roman_g has quit IRC | 02:34 | |
| *** heyongli has quit IRC | 02:41 | |
| *** psachin has joined #openstack-infra | 02:41 | |
| *** heyongli has joined #openstack-infra | 02:41 | |
| *** diablo_rojo has quit IRC | 02:42 | |
| *** diablo_rojo has joined #openstack-infra | 02:48 | |
| *** heyongli has quit IRC | 02:51 | |
| *** heyongli has joined #openstack-infra | 02:51 | |
| *** jamesmcarthur has joined #openstack-infra | 02:53 | |
| *** cjloader has quit IRC | 02:55 | |
| *** jamesmcarthur has quit IRC | 02:58 | |
| *** salv-orlando has joined #openstack-infra | 02:59 | |
| *** heyongli has quit IRC | 03:01 | |
| *** heyongli has joined #openstack-infra | 03:02 | |
| *** salv-orlando has quit IRC | 03:04 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: status: add pool column to nodepool list output https://review.openstack.org/579046 | 03:06 |
| *** dhill_ has joined #openstack-infra | 03:10 | |
| *** heyongli has quit IRC | 03:12 | |
| *** heyongli has joined #openstack-infra | 03:12 | |
| *** diablo_rojo has quit IRC | 03:12 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: launcher: add pool quota debug and log information https://review.openstack.org/579048 | 03:12 |
| *** dhill_ has quit IRC | 03:16 | |
| *** zhangfei has quit IRC | 03:17 | |
| *** aeng has quit IRC | 03:18 | |
| ianw | #status log migrated /opt on nb03.o.o to a new cinder volume due to increasing space requirements from new builds | 03:20 |
| openstackstatus | ianw: finished logging | 03:20 |
| *** heyongli has quit IRC | 03:22 | |
| *** heyongli has joined #openstack-infra | 03:22 | |
| *** ramishra has joined #openstack-infra | 03:25 | |
| *** rh-jelabarre has quit IRC | 03:29 | |
| *** heyongli has quit IRC | 03:32 | |
| *** heyongli has joined #openstack-infra | 03:32 | |
| *** zhangfei has joined #openstack-infra | 03:34 | |
| *** ykarel has joined #openstack-infra | 03:35 | |
| *** psachin has quit IRC | 03:37 | |
| *** hongbin has joined #openstack-infra | 03:41 | |
| *** hongbin has quit IRC | 03:42 | |
| *** heyongli has quit IRC | 03:42 | |
| *** heyongli has joined #openstack-infra | 03:43 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add selenium tests for the status page https://review.openstack.org/579039 | 03:45 |
| *** lpetrut has joined #openstack-infra | 03:45 | |
| *** udesale has joined #openstack-infra | 03:46 | |
| *** jesslampe has joined #openstack-infra | 03:51 | |
| *** heyongli has quit IRC | 03:53 | |
| *** bobh has joined #openstack-infra | 03:53 | |
| *** heyongli has joined #openstack-infra | 03:53 | |
| *** jesslampe has quit IRC | 03:56 | |
| *** heyongli has quit IRC | 04:03 | |
| *** heyongli has joined #openstack-infra | 04:03 | |
| *** rh-jelabarre has joined #openstack-infra | 04:09 | |
| *** heyongli has quit IRC | 04:13 | |
| *** heyongli has joined #openstack-infra | 04:14 | |
| *** alex_xu has quit IRC | 04:15 | |
| *** alex_xu has joined #openstack-infra | 04:15 | |
| *** heyongli has quit IRC | 04:23 | |
| *** heyongli has joined #openstack-infra | 04:24 | |
| *** masber has quit IRC | 04:29 | |
| *** diablo_rojo has joined #openstack-infra | 04:31 | |
| *** heyongli has quit IRC | 04:34 | |
| *** heyongli has joined #openstack-infra | 04:34 | |
| *** lpetrut has quit IRC | 04:34 | |
| *** jesslampe has joined #openstack-infra | 04:35 | |
| *** ykarel has quit IRC | 04:35 | |
| *** kjackal has joined #openstack-infra | 04:40 | |
| *** lpetrut has joined #openstack-infra | 04:41 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add selenium tests for the status page https://review.openstack.org/579039 | 04:42 |
| *** heyongli has quit IRC | 04:44 | |
| *** heyongli has joined #openstack-infra | 04:44 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: DNM: verify selenium test by reproducing the pathname whitelabel bug https://review.openstack.org/579052 | 04:46 |
| *** jesslampe has quit IRC | 04:50 | |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/578574/ and https://review.openstack.org/#/c/578701 to finalize removing of system-required template from project-config, please? | 04:51 |
| *** lpetrut has quit IRC | 04:51 | |
| *** heyongli has quit IRC | 04:54 | |
| *** heyongli has joined #openstack-infra | 04:54 | |
| *** armaan has joined #openstack-infra | 04:56 | |
| *** CrayZee has joined #openstack-infra | 04:58 | |
| *** rh-jelabarre has quit IRC | 04:58 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Add note about project read-only step for unretirement https://review.openstack.org/578954 | 04:59 |
| *** ykarel has joined #openstack-infra | 05:02 | |
| *** heyongli has quit IRC | 05:04 | |
| *** heyongli has joined #openstack-infra | 05:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove direct tagging from manila repositories https://review.openstack.org/578381 | 05:12 |
| *** heyongli has quit IRC | 05:15 | |
| *** heyongli has joined #openstack-infra | 05:15 | |
| *** pcaruana has joined #openstack-infra | 05:20 | |
| *** heyongli has quit IRC | 05:25 | |
| *** heyongli has joined #openstack-infra | 05:25 | |
| *** armaan has quit IRC | 05:28 | |
| *** janki has joined #openstack-infra | 05:31 | |
| *** quiquell|off is now known as quiquell|rover | 05:31 | |
| openstackgerrit | Ehud Kaldor proposed openstack-infra/bindep master: added Apk class to handle Alpine Linux package manager parsing https://review.openstack.org/579056 | 05:32 |
| openstackgerrit | Ehud Kaldor proposed openstack-infra/bindep master: added platform parsing for Alpine Linux and assignment of Apk to self.platform https://review.openstack.org/579057 | 05:32 |
| *** armaan_ has joined #openstack-infra | 05:33 | |
| *** armaan_ has quit IRC | 05:33 | |
| *** armaan has joined #openstack-infra | 05:34 | |
| *** heyongli has quit IRC | 05:35 | |
| *** heyongli has joined #openstack-infra | 05:35 | |
| *** cshastri has joined #openstack-infra | 05:36 | |
| *** armaan has quit IRC | 05:39 | |
| *** pguimaraes has joined #openstack-infra | 05:40 | |
| *** heyongli has quit IRC | 05:45 | |
| *** heyongli has joined #openstack-infra | 05:46 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a PartitionTableNode to graph https://review.openstack.org/579059 | 05:52 |
| *** heyongli has quit IRC | 05:56 | |
| *** heyongli has joined #openstack-infra | 05:56 | |
| *** ccamacho has joined #openstack-infra | 05:59 | |
| *** armaan has joined #openstack-infra | 06:03 | |
| *** jbadiapa has joined #openstack-infra | 06:04 | |
| *** heyongli has quit IRC | 06:06 | |
| *** heyongli has joined #openstack-infra | 06:06 | |
| *** nicolasbock has joined #openstack-infra | 06:10 | |
| *** kjackal has quit IRC | 06:11 | |
| *** iranzo has joined #openstack-infra | 06:15 | |
| *** heyongli has quit IRC | 06:16 | |
| *** heyongli has joined #openstack-infra | 06:16 | |
| *** lpetrut has joined #openstack-infra | 06:19 | |
| *** ccamacho has quit IRC | 06:20 | |
| *** ccamacho has joined #openstack-infra | 06:20 | |
| AJaeger | thanks, mnaser ! | 06:22 |
| *** armaan_ has joined #openstack-infra | 06:26 | |
| *** heyongli has quit IRC | 06:26 | |
| *** armaan has quit IRC | 06:26 | |
| openstackgerrit | Ehud Kaldor proposed openstack-infra/bindep master: added line to adhere to style https://review.openstack.org/579064 | 06:27 |
| *** heyongli has joined #openstack-infra | 06:27 | |
| openstackgerrit | Ehud Kaldor proposed openstack-infra/bindep master: added alpine linux fixture https://review.openstack.org/579065 | 06:27 |
| openstackgerrit | Ehud Kaldor proposed openstack-infra/bindep master: added tests for apk (alpine linux) https://review.openstack.org/579066 | 06:27 |
| openstackgerrit | Merged openstack-infra/project-config master: Cleanup system-required stragglers https://review.openstack.org/578574 | 06:27 |
| mnaser | AJaeger: no problem | 06:28 |
| *** armaan_ has quit IRC | 06:28 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add executing job INFO log https://review.openstack.org/578704 | 06:29 |
| *** armaan has joined #openstack-infra | 06:29 | |
| *** andreas_s has joined #openstack-infra | 06:30 | |
| *** quiquell|rover is now known as quique|rover|bbl | 06:31 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: status: add pool column to nodepool list output https://review.openstack.org/579046 | 06:32 |
| *** armaan has quit IRC | 06:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Test for system-required https://review.openstack.org/578701 | 06:35 |
| *** psachin has joined #openstack-infra | 06:36 | |
| *** heyongli has quit IRC | 06:37 | |
| *** heyongli has joined #openstack-infra | 06:37 | |
| *** yolanda__ has joined #openstack-infra | 06:42 | |
| *** yolanda__ is now known as yolanda | 06:43 | |
| *** tonytan4ever_brb has quit IRC | 06:44 | |
| *** heyongli has quit IRC | 06:47 | |
| *** heyongli has joined #openstack-infra | 06:47 | |
| AJaeger | jamespage: what's going on with https://review.openstack.org/#/c/529379 ? Looks strange that this was never fixed... | 06:48 |
| *** annp has quit IRC | 06:51 | |
| *** annp has joined #openstack-infra | 06:52 | |
| *** kjackal has joined #openstack-infra | 06:52 | |
| *** Dinesh_Bhor has quit IRC | 06:54 | |
| *** heyongli has quit IRC | 06:57 | |
| *** heyongli has joined #openstack-infra | 06:57 | |
| *** annp has quit IRC | 06:57 | |
| *** quique|rover|bbl is now known as quiquell|rover | 07:02 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstackci master: Add beaker tests for nodepool https://review.openstack.org/577522 | 07:02 |
| *** alexchadin has joined #openstack-infra | 07:03 | |
| *** jamesmcarthur has joined #openstack-infra | 07:05 | |
| *** amoralej|off is now known as amoralej | 07:07 | |
| *** heyongli has quit IRC | 07:07 | |
| *** heyongli has joined #openstack-infra | 07:08 | |
| *** jamesmcarthur has quit IRC | 07:09 | |
| *** tesseract has joined #openstack-infra | 07:10 | |
| *** annp has joined #openstack-infra | 07:11 | |
| *** heyongli has quit IRC | 07:18 | |
| *** heyongli has joined #openstack-infra | 07:18 | |
| *** zhangfei has quit IRC | 07:18 | |
| *** jpich has joined #openstack-infra | 07:27 | |
| *** heyongli has quit IRC | 07:28 | |
| *** heyongli has joined #openstack-infra | 07:28 | |
| *** zhangfei has joined #openstack-infra | 07:31 | |
| *** tosky has joined #openstack-infra | 07:32 | |
| *** heyongli has quit IRC | 07:38 | |
| *** heyongli has joined #openstack-infra | 07:38 | |
| *** andreas_s has quit IRC | 07:47 | |
| *** dpawlik has joined #openstack-infra | 07:47 | |
| *** andreas_s has joined #openstack-infra | 07:47 | |
| *** heyongli has quit IRC | 07:48 | |
| *** heyongli has joined #openstack-infra | 07:49 | |
| *** lpetrut has quit IRC | 07:52 | |
| *** andreas_s has quit IRC | 07:53 | |
| *** jpena|off is now known as jpena | 07:56 | |
| *** lpetrut has joined #openstack-infra | 07:56 | |
| *** florianf has joined #openstack-infra | 07:57 | |
| *** andreas_s has joined #openstack-infra | 07:58 | |
| *** heyongli has quit IRC | 07:59 | |
| *** heyongli has joined #openstack-infra | 07:59 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: gerrit: use baseurl for change uris lookup https://review.openstack.org/579086 | 08:02 |
| *** bradm has quit IRC | 08:02 | |
| *** d0ugal has joined #openstack-infra | 08:02 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: gerrit: use baseurl for change uris lookup https://review.openstack.org/579086 | 08:02 |
| *** bradm has joined #openstack-infra | 08:02 | |
| *** heyongli has quit IRC | 08:09 | |
| *** heyongli has joined #openstack-infra | 08:09 | |
| *** cfriesen has quit IRC | 08:11 | |
| *** ykarel is now known as ykarel|lunch | 08:16 | |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/elastic-recheck master: Add query for live migration test failure bug 1779270 https://review.openstack.org/579090 | 08:17 |
| openstack | bug 1779270 in OpenStack Compute (nova) "live-migration tempest test failed because the instance didn't seem to migrate to its target host" [Undecided,New] https://launchpad.net/bugs/1779270 | 08:17 |
| *** heyongli has quit IRC | 08:19 | |
| *** heyongli has joined #openstack-infra | 08:19 | |
| *** shardy has joined #openstack-infra | 08:20 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: config: enable using environ template format for secrets https://review.openstack.org/579091 | 08:20 |
| *** Shrews has joined #openstack-infra | 08:25 | |
| *** Shrews has quit IRC | 08:25 | |
| *** Shrews has joined #openstack-infra | 08:25 | |
| *** Shrews has quit IRC | 08:25 | |
| *** alexchadin has quit IRC | 08:25 | |
| *** bobh has quit IRC | 08:27 | |
| *** heyongli has quit IRC | 08:29 | |
| *** heyongli has joined #openstack-infra | 08:30 | |
| *** alexchadin has joined #openstack-infra | 08:31 | |
| *** dhajare has joined #openstack-infra | 08:37 | |
| *** bauzas is now known as PapaOurs | 08:38 | |
| *** heyongli has quit IRC | 08:40 | |
| *** heyongli has joined #openstack-infra | 08:40 | |
| *** diablo_rojo has quit IRC | 08:44 | |
| xinliang | hey guys, one pypi mirror http://mirror.regionone.linaro.openstack.org:8080/pypi/ seems doesn't work. | 08:46 |
| xinliang | here is the log: http://logs.openstack.org/59/557659/21/experimental/kolla-build-debian-source-arm64/7ae964a/job-output.txt.gz#_2018-06-28_10_03_41_336903 | 08:46 |
| xinliang | it failed with "Could not find a version that satisfies the requirement pip (from versions: ), No matching distribution found for pip" | 08:47 |
| xinliang | when running "python get-pip.py" | 08:47 |
| xinliang | ianw: do you know what's wrong? | 08:48 |
| openstackgerrit | Xinran WANG proposed openstack-infra/project-config master: Add os-acc to openstack repo https://review.openstack.org/579101 | 08:48 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a PartitionTableNode to graph https://review.openstack.org/579059 | 08:50 |
| *** heyongli has quit IRC | 08:50 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move LVM cleanup phase into cleanup https://review.openstack.org/579102 | 08:50 |
| *** derekh has joined #openstack-infra | 08:50 | |
| *** heyongli has joined #openstack-infra | 08:50 | |
| *** stephenfin is now known as finucannot | 08:51 | |
| *** electrofelix has joined #openstack-infra | 08:53 | |
| *** panda|off is now known as panda | 08:54 | |
| *** hrw has joined #openstack-infra | 08:57 | |
| hrw | morning | 08:57 |
| *** heyongli has quit IRC | 09:00 | |
| *** heyongli has joined #openstack-infra | 09:00 | |
| diablo_rojo_phon | notmyname: did you get all your StoryBoard questions answered? Sorry I didn't answer in a timely manner (in Korea currently and timezones..) but it looks like fungi was able to help. | 09:09 |
| xinliang | ianw: do you think if the pypi issue has something to do with mtu issue which we found? | 09:09 |
| *** heyongli has quit IRC | 09:10 | |
| *** heyongli has joined #openstack-infra | 09:11 | |
| *** zhangfei has quit IRC | 09:11 | |
| *** zoli is now known as zoli|lunch | 09:13 | |
| *** spsurya has quit IRC | 09:14 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add executing job INFO log https://review.openstack.org/578704 | 09:14 |
| *** sree_ has joined #openstack-infra | 09:20 | |
| *** heyongli has quit IRC | 09:21 | |
| *** sree_ is now known as Guest80303 | 09:21 | |
| *** heyongli has joined #openstack-infra | 09:21 | |
| ianw | xinliang: i don't think so, because the mtu was ok on the mirror as it was never rebooted | 09:24 |
| *** zhangfei has joined #openstack-infra | 09:24 | |
| *** Guest80303 has quit IRC | 09:27 | |
| ianw | xinliang: http://paste.openstack.org/show/724580/ ... looks like something is getting in between the connection from mirror -> mirror.rax.dfw where it proxies to | 09:28 |
| ianw | unfortunately the mirror doesn't have ssl so we can't bust that with https | 09:30 |
| *** heyongli has quit IRC | 09:31 | |
| *** heyongli has joined #openstack-infra | 09:31 | |
| ianw | what's weird is it doesn't happen from nb03 | 09:31 |
| *** cjloader has joined #openstack-infra | 09:31 | |
| *** jesusaur has quit IRC | 09:32 | |
| ianw | i don't know, short of somehow changing the ip or mac address? :/ | 09:35 |
| *** cjloader has quit IRC | 09:36 | |
| *** heyongli has quit IRC | 09:41 | |
| *** armaan has joined #openstack-infra | 09:41 | |
| *** heyongli has joined #openstack-infra | 09:41 | |
| xinliang | ianw: you mean mirror.regionone.linaro.openstack.org is synced from mirror.dfw.rax.openstack.org? | 09:44 |
| xinliang | I seems it has problem with accessing mirror.dfw.rax.openstack.org in linaro cloud | 09:44 |
| *** alexchadin has quit IRC | 09:44 | |
| xinliang | China accessing outside world often behind a proxy/fireward :( | 09:46 |
| gema | xinliang: this has nothing to do with the china wall | 09:47 |
| gema | xinliang: we have a link through shanghai | 09:47 |
| gema | I mean, hong kong :) | 09:47 |
| gema | xinliang: we'll keep everyone posted on the ticket | 09:47 |
| *** heyongli has quit IRC | 09:51 | |
| *** heyongli has joined #openstack-infra | 09:52 | |
| *** jesusaur has joined #openstack-infra | 09:52 | |
| *** armaan has quit IRC | 09:53 | |
| *** armaan has joined #openstack-infra | 09:54 | |
| finucannot | Does anyone have a good reference on the lower-constraints system most projects have in place now? Why it was necessary, how to bump stuff, etc. | 09:57 |
| *** armaan has quit IRC | 09:58 | |
| *** armaan has joined #openstack-infra | 09:59 | |
| *** heyongli has quit IRC | 10:02 | |
| *** sshnaidm is now known as sshnaidm|off | 10:02 | |
| ianw | finucannot: i never quite finished off https://review.openstack.org/#/c/568994/ but maybe it helps | 10:02 |
| *** roman_g has joined #openstack-infra | 10:02 | |
| *** heyongli has joined #openstack-infra | 10:02 | |
| ianw | gema: it does seem there's some sort of transparent proxy, though? | 10:02 |
| gema | ianw: yeah, but it shouldn't | 10:03 |
| gema | ianw: we'll continue to look into it | 10:03 |
| gema | ianw: and it makes no sense, since both machines are on the same cloud | 10:05 |
| gema | so we are technically not going out of shanghai | 10:05 |
| *** ykarel|lunch is now known as ykarel | 10:05 | |
| gema | so the traffic should happily go about life internally | 10:06 |
| ianw | gema: in this case, it's the proxy on mirror.regionone.linaro.o.o not being able to connect to mirror.dfw.rax.openstack.org | 10:08 |
| gema | ianw: ah, ok | 10:10 |
| gema | that makes more sense | 10:10 |
| *** heyongli has quit IRC | 10:12 | |
| *** heyongli has joined #openstack-infra | 10:12 | |
| *** vivsoni has quit IRC | 10:13 | |
| *** armaan has quit IRC | 10:20 | |
| *** armaan has joined #openstack-infra | 10:20 | |
| *** heyongli has quit IRC | 10:22 | |
| *** heyongli has joined #openstack-infra | 10:22 | |
| *** sree_ has joined #openstack-infra | 10:24 | |
| *** sree_ is now known as Guest35141 | 10:24 | |
| *** derekh has quit IRC | 10:29 | |
| *** derekh has joined #openstack-infra | 10:30 | |
| *** cjloader has joined #openstack-infra | 10:32 | |
| *** heyongli has quit IRC | 10:32 | |
| *** heyongli has joined #openstack-infra | 10:33 | |
| *** boden has joined #openstack-infra | 10:35 | |
| *** stakeda has quit IRC | 10:36 | |
| *** cjloader has quit IRC | 10:36 | |
| *** niska` has quit IRC | 10:42 | |
| *** heyongli has quit IRC | 10:43 | |
| *** dtantsur|afk is now known as dtantsur | 10:43 | |
| *** heyongli has joined #openstack-infra | 10:43 | |
| *** linkmark has joined #openstack-infra | 10:50 | |
| *** heyongli has quit IRC | 10:53 | |
| *** panda is now known as panda|ko | 10:53 | |
| *** heyongli has joined #openstack-infra | 10:53 | |
| *** niska has joined #openstack-infra | 10:56 | |
| *** electrofelix has quit IRC | 11:02 | |
| *** d0ugal_ has joined #openstack-infra | 11:03 | |
| *** heyongli has quit IRC | 11:03 | |
| *** heyongli has joined #openstack-infra | 11:03 | |
| *** kjackal has quit IRC | 11:03 | |
| *** d0ugal has quit IRC | 11:05 | |
| *** jpena is now known as jpena|lunch | 11:06 | |
| *** Guest35141 has quit IRC | 11:07 | |
| *** gfidente has joined #openstack-infra | 11:09 | |
| *** sree_ has joined #openstack-infra | 11:10 | |
| *** zoli|lunch is now known as zoli | 11:10 | |
| *** sree_ is now known as Guest27265 | 11:10 | |
| *** heyongli has quit IRC | 11:13 | |
| *** heyongli has joined #openstack-infra | 11:14 | |
| *** Guest27265 has quit IRC | 11:15 | |
| *** neiloy has joined #openstack-infra | 11:17 | |
| *** heyongli has quit IRC | 11:24 | |
| *** udesale has quit IRC | 11:24 | |
| *** heyongli has joined #openstack-infra | 11:24 | |
| *** sshnaidm|off has quit IRC | 11:29 | |
| *** ldnunes has joined #openstack-infra | 11:29 | |
| *** edmondsw has joined #openstack-infra | 11:32 | |
| frickler | config-core: could I get one more eye on https://review.openstack.org/578790 please? | 11:33 |
| *** heyongli has quit IRC | 11:34 | |
| *** heyongli has joined #openstack-infra | 11:34 | |
| *** tonytan4ever has joined #openstack-infra | 11:36 | |
| *** annp has quit IRC | 11:39 | |
| *** electrofelix has joined #openstack-infra | 11:40 | |
| *** kjackal has joined #openstack-infra | 11:41 | |
| *** heyongli has quit IRC | 11:44 | |
| quiquell|rover | Hello, we are checking our tripleo jobs timeouts, and mos of them are limestone provider | 11:44 |
| *** vabada has quit IRC | 11:44 | |
| *** heyongli has joined #openstack-infra | 11:44 | |
| *** vabada has joined #openstack-infra | 11:44 | |
| quiquell|rover | Do you know if there are any issue there ? | 11:44 |
| quiquell|rover | They are containerized openstack jobs | 11:45 |
| *** trown|outtypewww is now known as trown | 11:49 | |
| *** auristor has quit IRC | 11:51 | |
| *** auristor has joined #openstack-infra | 11:51 | |
| *** heyongli has quit IRC | 11:54 | |
| *** zhangfei has quit IRC | 11:54 | |
| *** heyongli has joined #openstack-infra | 11:55 | |
| openstackgerrit | Xinran WANG proposed openstack-infra/project-config master: Add os-acc to openstack repo https://review.openstack.org/579101 | 11:59 |
| *** heyongli has quit IRC | 12:05 | |
| *** heyongli has joined #openstack-infra | 12:05 | |
| *** jpena|lunch is now known as jpena | 12:07 | |
| openstackgerrit | Merged openstack-infra/bindep master: Clearer unsupported platform error message https://review.openstack.org/579029 | 12:07 |
| openstackgerrit | Saverio Proto proposed openstack/diskimage-builder master: Add support for Fedora 28 https://review.openstack.org/579131 | 12:07 |
| *** mriedem has joined #openstack-infra | 12:08 | |
| *** heyongli has quit IRC | 12:15 | |
| *** heyongli has joined #openstack-infra | 12:15 | |
| *** d0ugal_ has quit IRC | 12:15 | |
| *** d0ugal_ has joined #openstack-infra | 12:16 | |
| *** zhangfei has joined #openstack-infra | 12:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add devstack-plugin-ceph to #openstack-qa channel https://review.openstack.org/578790 | 12:20 |
| *** neiloy has quit IRC | 12:22 | |
| *** heyongli has quit IRC | 12:25 | |
| *** heyongli has joined #openstack-infra | 12:25 | |
| *** rcernin has quit IRC | 12:29 | |
| *** dhill_ has joined #openstack-infra | 12:30 | |
| *** sree_ has joined #openstack-infra | 12:32 | |
| *** sree_ is now known as Guest77234 | 12:32 | |
| *** zhangfei has quit IRC | 12:33 | |
| *** zhangfei has joined #openstack-infra | 12:34 | |
| *** heyongli has quit IRC | 12:35 | |
| *** heyongli has joined #openstack-infra | 12:36 | |
| *** tonytan4ever has quit IRC | 12:36 | |
| lucasagomes | ianw, hi, if u have some time today mind taking a look at https://review.openstack.org/#/c/578752/ ? The networking-ovn's gate is currently broken due to it and the patch is fairly simple. Thanks! | 12:36 |
| *** zul has joined #openstack-infra | 12:38 | |
| *** ccamacho has quit IRC | 12:38 | |
| *** quiquell|rover is now known as quique|rover|lch | 12:40 | |
| *** hashar has joined #openstack-infra | 12:41 | |
| *** Guest77234 has quit IRC | 12:41 | |
| *** tpsilva has joined #openstack-infra | 12:41 | |
| *** rlandy has joined #openstack-infra | 12:42 | |
| *** lpetrut has quit IRC | 12:42 | |
| *** Shrews has joined #openstack-infra | 12:44 | |
| *** kgiusti has joined #openstack-infra | 12:44 | |
| *** heyongli has quit IRC | 12:46 | |
| *** heyongli has joined #openstack-infra | 12:46 | |
| *** peereb has joined #openstack-infra | 12:46 | |
| *** ccamacho has joined #openstack-infra | 12:46 | |
| frickler | infra-root: I find the inheritance_path for build-openstack-sphinx-docs to show information that doesn't match what I see in the repos, see e.g. http://logs.openstack.org/24/558824/5/check/build-openstack-sphinx-docs/bea85d5/zuul-info/inventory.yaml | 12:47 |
| frickler | the base job isn't defined here: '<Job base branches: None source: openstack-infra/project-config/zuul.d/zuul-jobs.yaml@master#25>' | 12:47 |
| frickler | also no sign of any docs job in this file: '<Job build-openstack-sphinx-docs branches: {MatchAny:{BranchMatcher:^(?!driverfixes/).*$}} source: openstack-infra/openstack-zuul-jobs/zuul.d/zuul-legacy-project-templates.yaml@master#260>' | 12:48 |
| *** openstackgerrit has quit IRC | 12:49 | |
| AJaeger | frickler: interesting - http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n27 should be the base job | 12:49 |
| mordred | frickler: that seems quite weird | 12:49 |
| frickler | AJaeger: the question that I was originally trying to answer: can I limit build-openstack-sphinx-docs to run on certain branches only by simply adding a branch matcher? | 12:50 |
| mordred | corvus: ^^ when you awaken, I imagine you will very much want to look at that | 12:50 |
| AJaeger | frickler: I agree with mordred that the source lines are broken ;( | 12:51 |
| *** lpetrut has joined #openstack-infra | 12:51 | |
| AJaeger | frickler: yes, you could do that if you do it in project-config | 12:52 |
| frickler | AJaeger: I found a sample for devstack, seems I need to add it to a job definition then instead of to the template, correct? I was thinking whether it would be easier to move the whole thing in-repo, not sure whether we can do this for this kind of job now? | 12:55 |
| *** heyongli has quit IRC | 12:56 | |
| *** heyongli has joined #openstack-infra | 12:56 | |
| AJaeger | frickler: you can move in-repo, if the job does not use release or tag pipeline - which is fine for this template. But then add it to each branch instead of using branches: | 12:57 |
| *** yamamoto has quit IRC | 12:57 | |
| *** roman_g has quit IRC | 12:57 | |
| *** efried is now known as fried_rice | 12:58 | |
| frickler | AJaeger: the idea was to only add it to master now and then rocky when it is branched, do I need to do anything special for that? IIUC the stable branches no longer inherit from master | 12:59 |
| AJaeger | frickler: yeah, that works - don't add branches line and then when you branch rocky, it will be fine without changes | 13:01 |
| *** sree has joined #openstack-infra | 13:01 | |
| frickler | AJaeger: great, thanks | 13:01 |
| *** heyongli has quit IRC | 13:06 | |
| *** heyongli has joined #openstack-infra | 13:06 | |
| *** sree has quit IRC | 13:12 | |
| *** openstackgerrit has joined #openstack-infra | 13:12 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Revert "Add doc build job for Chef OpenStack" https://review.openstack.org/579149 | 13:12 |
| *** d0ugal_ has quit IRC | 13:12 | |
| *** amoralej is now known as amoralej|lunch | 13:13 | |
| *** d0ugal has joined #openstack-infra | 13:13 | |
| *** d0ugal has quit IRC | 13:13 | |
| *** d0ugal has joined #openstack-infra | 13:13 | |
| *** udesale has joined #openstack-infra | 13:14 | |
| frickler | AJaeger: do I need to remove the complete project stanza? or just the "templates:" part? ^^ | 13:15 |
| *** zul_ has joined #openstack-infra | 13:15 | |
| *** heyongli has quit IRC | 13:16 | |
| *** heyongli has joined #openstack-infra | 13:17 | |
| AJaeger | frickler: complete project stanza - see https://review.openstack.org/#/c/578574/ as well | 13:17 |
| *** zul has quit IRC | 13:18 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Revert "Add doc build job for Chef OpenStack" https://review.openstack.org/579149 | 13:20 |
| *** zxiiro is now known as zxiiro-pto | 13:20 | |
| AJaeger | frickler: both changes look fine now | 13:21 |
| *** eharney has joined #openstack-infra | 13:23 | |
| frickler | AJaeger: cool, thanks for confirming | 13:23 |
| ssbarnea | i think we are doomed, I have the impression that ansible-2.6 was released. | 13:23 |
| *** zul_ has quit IRC | 13:24 | |
| *** zul has joined #openstack-infra | 13:25 | |
| *** yamamoto has joined #openstack-infra | 13:26 | |
| *** heyongli has quit IRC | 13:27 | |
| *** hashar has quit IRC | 13:27 | |
| *** heyongli has joined #openstack-infra | 13:27 | |
| *** derekh has quit IRC | 13:28 | |
| *** derekh has joined #openstack-infra | 13:29 | |
| *** zul_ has joined #openstack-infra | 13:29 | |
| *** agopi has quit IRC | 13:31 | |
| *** cjloader has joined #openstack-infra | 13:31 | |
| *** caphrim007_ has quit IRC | 13:32 | |
| mordred | ssbarnea: which of us does the release of ansible 2.6 doom? | 13:32 |
| *** snapiri- has joined #openstack-infra | 13:32 | |
| *** caphrim007 has joined #openstack-infra | 13:33 | |
| *** snapiri- has quit IRC | 13:33 | |
| ssbarnea | mordred: wrong channel ;) .. usually a X.Y.0 ansible release is likely to break 4/5 jobs :D | 13:33 |
| *** snapiri- has joined #openstack-infra | 13:33 | |
| ssbarnea | not necessarly ansible fault. | 13:34 |
| *** zul has quit IRC | 13:34 | |
| *** CrayZee has quit IRC | 13:35 | |
| *** cjloader has quit IRC | 13:36 | |
| *** heyongli has quit IRC | 13:37 | |
| *** caphrim007 has quit IRC | 13:37 | |
| *** heyongli has joined #openstack-infra | 13:37 | |
| mordred | ssbarnea: :) | 13:40 |
| *** sthussey has joined #openstack-infra | 13:44 | |
| *** bnemec has joined #openstack-infra | 13:46 | |
| *** bnemec is now known as beekneemech | 13:46 | |
| *** heyongli has quit IRC | 13:47 | |
| *** heyongli has joined #openstack-infra | 13:47 | |
| *** jistr is now known as jistr|mtg | 13:54 | |
| *** heyongli has quit IRC | 13:57 | |
| *** heyongli has joined #openstack-infra | 13:58 | |
| *** gfidente is now known as gfidente|pto | 14:00 | |
| jamespage | AJaeger: I'll check with tytus | 14:01 |
| *** agopi has joined #openstack-infra | 14:02 | |
| *** agopi_ has joined #openstack-infra | 14:03 | |
| *** agopi_ has quit IRC | 14:03 | |
| *** zhangfei has quit IRC | 14:03 | |
| *** kjackal has quit IRC | 14:05 | |
| weshay | greetings, anyone notice that the limestone provider, RegionOne has more timeouts than other providers? | 14:05 |
| weshay | wondering how I should approach that | 14:05 |
| *** agopi_ has joined #openstack-infra | 14:05 | |
| weshay | elastic-recheck-query 1686542.yaml | 14:05 |
| weshay | node_provider | 14:06 |
| weshay | 41% limestone-regionone | 14:06 |
| *** quique|rover|lch is now known as quiquell|rover | 14:06 | |
| weshay | next highest is significantly lower | 14:06 |
| weshay | node_provider | 14:06 |
| weshay | 41% limestone-regionone | 14:06 |
| weshay | 15% vexxhost-ca-ymq-1 | 14:06 |
| weshay | 12% rax-dfw | 14:06 |
| quiquell|rover | weshay: I am back | 14:06 |
| weshay | 8% rax-iad | 14:06 |
| weshay | 8% rax-ord | 14:06 |
| *** agopi has quit IRC | 14:06 | |
| weshay | quiquell|rover, hey there | 14:06 |
| quiquell|rover | bj ? | 14:06 |
| weshay | is that an issue I should document in a bug? | 14:06 |
| weshay | quiquell|rover, ya | 14:06 |
| *** kjackal has joined #openstack-infra | 14:06 | |
| *** heyongli has quit IRC | 14:08 | |
| *** heyongli has joined #openstack-infra | 14:08 | |
| *** jistr|mtg is now known as jistr | 14:09 | |
| *** ccamacho has quit IRC | 14:09 | |
| fungi | we've found that bug reports serve poorly for incident tracking. do all of your jobs which run in limestone-regionone time out, or just a higher percentage of them than elsewhere? | 14:09 |
| weshay | fungi, just a higher percentage | 14:09 |
| weshay | afaik | 14:09 |
| fungi | and of the ones which complete in limestone-regionone, do many/most of them finish just short of the job timeout or well below? | 14:10 |
| weshay | quiquell|rover, let's see if we can get some data on this question ^ | 14:11 |
| fungi | trying to ascertain whether it's an issue of relative performance or something actually anomalous happening in there | 14:11 |
| weshay | sure | 14:11 |
| weshay | fungi, working that now.. will post back when we have the job time for all the limehost jobs.. apparently we have 71 jobs run there in the last 30 days | 14:12 |
| fungi | that was going to be my next question: do you have a feel for how far back the timeout disparity there extends? | 14:12 |
| fungi | the 30 days you cited, or has it cropped up much more recently? | 14:13 |
| *** d0ugal has quit IRC | 14:15 | |
| *** zhangfei has joined #openstack-infra | 14:16 | |
| *** d0ugal has joined #openstack-infra | 14:16 | |
| *** hongbin has joined #openstack-infra | 14:17 | |
| *** heyongli has quit IRC | 14:18 | |
| *** heyongli has joined #openstack-infra | 14:18 | |
| fungi | i'm wondering whether your list of percentage breakdowns is actually a measure of relative performance of our providers | 14:18 |
| fungi | looks like we're sustaining ~30 in use nodes in limestone-regionone currently: http://grafana.openstack.org/dashboard/db/nodepool-limestone | 14:20 |
| fungi | which corresponds to our nodepool max-servers setting for that provider | 14:21 |
| *** amoralej|lunch is now known as amoralej | 14:22 | |
| fungi | i wonder if we're running into resource contention at capacity and need to tune that downward some | 14:22 |
| *** cshastri has quit IRC | 14:23 | |
| fungi | that provider accounts for only ~3% of our current aggregate quota, so for it to represent 41% of timeouts definitely seems to suggest something problematic | 14:26 |
| *** ccamacho has joined #openstack-infra | 14:27 | |
| *** heyongli has quit IRC | 14:28 | |
| pabelanger | give provider is at 100% capacity, maybe seeing noise neighbours on the compute nodes | 14:28 |
| *** heyongli has joined #openstack-infra | 14:28 | |
| *** esarault has joined #openstack-infra | 14:29 | |
| fungi | where "noisy neighbors" == us | 14:29 |
| fungi | but yeah, until we know whether the jobs which are timing out there are just running slower than elsewhere or some random selection of them are hanging indefinitely until the timeout kills them, hard to guess | 14:32 |
| *** cfriesen has joined #openstack-infra | 14:34 | |
| logan- | looking into it. the only recent change that occurred is all computes received kernel updates as we were attempting to solve the nested virt issues octavia has been seeing. | 14:36 |
| mtreinish | fungi: now that the exim/cyrus fix landed do we need to restart anything to take the config changes? | 14:36 |
| fungi | mtreinish: i'll check | 14:36 |
| fungi | thanks logan-! | 14:36 |
| mtreinish | I assume lpmqtt is still running so when new mail pops up in the inbox it'll start working again | 14:37 |
| *** heyongli has quit IRC | 14:38 | |
| fungi | mtreinish: looks like the last message to land in the old mbox was at 21:12:17z yesterday | 14:38 |
| *** heyongli has joined #openstack-infra | 14:39 | |
| fungi | and there are a few messages in the new cyrus mail tree now | 14:39 |
| fungi | s/new// | 14:39 |
| fungi | new messages in the cyrus mail tree | 14:39 |
| fungi | lpmqtt is running since the 25th | 14:39 |
| *** jamesmcarthur has joined #openstack-infra | 14:40 | |
| fungi | and it's logging finding messages it wants | 14:40 |
| mtreinish | ok cool, I'll sit on the topic and wait for something to pop up | 14:40 |
| fungi | so i guess if you sub to the correct topic you'll see events pop occasionally for those now | 14:40 |
| fungi | yeah | 14:40 |
| *** psachin has quit IRC | 14:42 | |
| *** lpetrut has quit IRC | 14:42 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-manual master: Drop references to system-required template https://review.openstack.org/579180 | 14:45 |
| fungi | AJaeger: ^ i didn't see a change for that yet | 14:45 |
| AJaeger | fungi: we had one patch merged - but might have missed a place? Thanks! | 14:46 |
| fungi | i ran across it still in the project retirement instructions, so did a git grep in there and found a few others too | 14:47 |
| *** heyongli has quit IRC | 14:49 | |
| *** cjloader has joined #openstack-infra | 14:49 | |
| *** heyongli has joined #openstack-infra | 14:49 | |
| mtreinish | fungi: looks like it works now thanks. Just got a burst of a bunch of bug events | 14:52 |
| fungi | nice. glad that was the whole solution | 14:52 |
| fungi | sorry i didn't spot the problem sooner | 14:52 |
| mtreinish | heh, I'm just glad we didn't have anyone relying on that event stream (or at least I'm assuming that, because no one complained) | 14:53 |
| fungi | they thought openstack suddenly became bug-free | 14:53 |
| *** andreas_s has quit IRC | 14:54 | |
| *** andreas_s has joined #openstack-infra | 14:54 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Make websocket streaming more event-driven https://review.openstack.org/558646 | 14:54 |
| *** rpioso|afk is now known as rpioso | 14:55 | |
| *** d0ugal has quit IRC | 14:55 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Stop running jobs for the release-tools repo https://review.openstack.org/579185 | 14:56 |
| *** andreas_s_ has joined #openstack-infra | 14:57 | |
| *** cjloader has quit IRC | 14:57 | |
| *** cjloader_ has joined #openstack-infra | 14:57 | |
| *** shardy has quit IRC | 14:58 | |
| *** beagles is now known as beagles_mtg | 14:58 | |
| *** cjloader_ has quit IRC | 14:58 | |
| *** andreas__ has joined #openstack-infra | 14:58 | |
| *** cjloader has joined #openstack-infra | 14:58 | |
| *** andreas_s has quit IRC | 14:58 | |
| *** heyongli has quit IRC | 14:59 | |
| *** cjloader has quit IRC | 14:59 | |
| *** heyongli has joined #openstack-infra | 14:59 | |
| *** cjloader has joined #openstack-infra | 14:59 | |
| johnsom | Does anyone know if I can use the Ansible "lookup" with "ini" plugin in gate job playbooks? I'm getting a really strange error. A quick codesearch doesn't show anyone using it yet, just a comment about using it in the future from mordred. | 15:00 |
| johnsom | This is the error: AnsibleError: An unhandled exception occurred while templating '{{lookup('ini', 'connection section=database file=\"/etc/octavia/octavia.conf\"')}}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while running the lookup plugin 'ini'. Error was a <class 'ansible.errors.AnsibleParserError'>, original message: Invalid filename: 'None' | 15:00 |
| *** tojuvone has quit IRC | 15:01 | |
| *** d0ugal has joined #openstack-infra | 15:01 | |
| *** andreas_s_ has quit IRC | 15:01 | |
| johnsom | This works locally for me | 15:01 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/release-tools master: This repository is now retired https://review.openstack.org/579188 | 15:02 |
| *** andreas__ has quit IRC | 15:02 | |
| *** tojuvone has joined #openstack-infra | 15:03 | |
| openstackgerrit | Merged openstack-infra/zuul master: Translate zuulStartStream into typescript https://review.openstack.org/558618 | 15:05 |
| corvus | johnsom, mordred: do lookup plugins run on the ansible controller or the remote node? | 15:06 |
| *** alex_xu has quit IRC | 15:06 | |
| johnsom | Should be local | 15:07 |
| *** andreas_s has joined #openstack-infra | 15:07 | |
| *** e0ne has joined #openstack-infra | 15:08 | |
| *** alex_xu has joined #openstack-infra | 15:09 | |
| corvus | johnsom: then /etc/octavia/octavia.conf isn't going to be accessible. it's probably running into a security protection we have to keep jobs confined to their working directory on the zuul executor. usually that produces a better error message. | 15:09 |
| *** heyongli has quit IRC | 15:09 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Remove release-tools configuration https://review.openstack.org/579194 | 15:09 |
| *** heyongli has joined #openstack-infra | 15:09 | |
| *** esarault has quit IRC | 15:12 | |
| *** andreas_s has quit IRC | 15:12 | |
| johnsom | corvus Hmmm, so any thoughts on how to extract a string from that config file? I need the DB connection string to run a LB migration tool to test our ability to migrate load balancers out of neutron-lbaas and into octavia. | 15:13 |
| johnsom | It is odd, when I run it locally with a bogus filename the error is different, but maybe I have a different version of ansible | 15:14 |
| logan- | lookups are run on the ansible executor, not the host you are connecting to in inventory | 15:14 |
| corvus | johnsom: yeah, it should work as long as the file is inside the work dir | 15:15 |
| logan- | i have an idea for you though | 15:15 |
| *** zhangfei has quit IRC | 15:15 | |
| johnsom | Yeah, I just thought our gate hosts were running it all in one, I didn't realize it was "remote". | 15:16 |
| corvus | (the work dir is a scratch space zuul gives each job on its executor; the git repos are inside that space, but there's also other space to do whatever you want with) | 15:16 |
| *** janki has quit IRC | 15:16 | |
| corvus | ah yeah, "local" is on one of the zuul executors, all the nodes we get from nodepool are remote | 15:16 |
| logan- | http://paste.openstack.org/raw/724600/ | 15:17 |
| *** tojuvone has quit IRC | 15:17 | |
| johnsom | Ok, so maybe I just do this old school and spawn an awk | 15:18 |
| *** tojuvone has joined #openstack-infra | 15:18 | |
| corvus | logan-: yeah, that should work (johnsom ^) | 15:22 |
| johnsom | logan- Thanks! I will give that a shot. | 15:22 |
| *** heyongli has quit IRC | 15:22 | |
| *** heyongli has joined #openstack-infra | 15:22 | |
| *** openstack has quit IRC | 15:22 | |
| *** openstack has joined #openstack-infra | 15:23 | |
| *** ChanServ sets mode: +o openstack | 15:23 | |
| *** tojuvone has joined #openstack-infra | 15:23 | |
| *** gyee has joined #openstack-infra | 15:24 | |
| pabelanger | I don't know if tempfile will work, I think we limit the safe path to zuul.executor.work_root? | 15:25 |
| logan- | yeah there you go, if theres a var for the scratch space defined you can fetch to there instead | 15:26 |
| pabelanger | yah, I'd try tempfile, if fails then some place in work_root | 15:27 |
| *** dpawlik has quit IRC | 15:27 | |
| corvus | pabelanger: we tell ansible that the temp dir is in the work root | 15:28 |
| *** cjloader has quit IRC | 15:28 | |
| corvus | pabelanger: 'local_tmp = %s\n' % self.jobdir.local_tmp | 15:28 |
| *** cjloader has joined #openstack-infra | 15:28 | |
| *** afazekas is now known as afazekas|pto | 15:29 | |
| *** iranzo has quit IRC | 15:29 | |
| *** felipemonteiro has joined #openstack-infra | 15:29 | |
| *** heyongli has quit IRC | 15:30 | |
| pabelanger | corvus: okay, I didn't think tempfile would use that | 15:30 |
| *** heyongli has joined #openstack-infra | 15:30 | |
| *** d0ugal__ has joined #openstack-infra | 15:31 | |
| *** d0ugal_ has quit IRC | 15:32 | |
| *** tojuvone_ has joined #openstack-infra | 15:32 | |
| corvus | pabelanger: well, if it doesn't, hopefully it'll obey the standard env variable: env_copy['TMP'] = self.jobdir.local_tmp | 15:33 |
| *** tojuvone has quit IRC | 15:33 | |
| pabelanger | corvus: ah, yes. That was the piece I was missing. Thanks | 15:34 |
| *** slaweq has quit IRC | 15:36 | |
| *** slaweq has joined #openstack-infra | 15:37 | |
| *** pblaho has quit IRC | 15:39 | |
| *** heyongli has quit IRC | 15:40 | |
| *** heyongli has joined #openstack-infra | 15:40 | |
| *** ykarel is now known as ykarel|afk | 15:40 | |
| *** tesseract has quit IRC | 15:42 | |
| *** d0ugal__ has quit IRC | 15:42 | |
| *** janki has joined #openstack-infra | 15:44 | |
| *** ccamacho has quit IRC | 15:44 | |
| *** caphrim007 has joined #openstack-infra | 15:45 | |
| *** zoli is now known as zoli|gone | 15:46 | |
| *** zoli|gone is now known as zoli | 15:46 | |
| *** caphrim007 has quit IRC | 15:47 | |
| *** caphrim007 has joined #openstack-infra | 15:47 | |
| *** caphrim007 has quit IRC | 15:47 | |
| *** lpetrut has joined #openstack-infra | 15:47 | |
| *** caphrim007 has joined #openstack-infra | 15:48 | |
| *** edmondsw has quit IRC | 15:48 | |
| *** heyongli has quit IRC | 15:50 | |
| *** heyongli has joined #openstack-infra | 15:50 | |
| *** edmondsw has joined #openstack-infra | 15:52 | |
| *** pbourke has quit IRC | 15:52 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/release-tools master: This repository is now retired https://review.openstack.org/579188 | 15:53 |
| *** pbourke has joined #openstack-infra | 15:54 | |
| *** rwsu has joined #openstack-infra | 15:55 | |
| *** sshnaidm|off has joined #openstack-infra | 15:55 | |
| *** armaan has quit IRC | 15:56 | |
| *** armaan has joined #openstack-infra | 15:56 | |
| *** hrw has quit IRC | 15:59 | |
| *** heyongli has quit IRC | 16:00 | |
| *** heyongli has joined #openstack-infra | 16:01 | |
| *** Adri2000_ has joined #openstack-infra | 16:02 | |
| *** kjackal has quit IRC | 16:04 | |
| *** kjackal has joined #openstack-infra | 16:05 | |
| *** lpetrut has quit IRC | 16:05 | |
| * clarkb is having an incedibly slow start to the day | 16:05 | |
| *** olivierbourdon38 has quit IRC | 16:05 | |
| *** tobasco has quit IRC | 16:06 | |
| *** tobasco has joined #openstack-infra | 16:08 | |
| *** beekneemech has quit IRC | 16:09 | |
| *** sthussey has quit IRC | 16:09 | |
| *** weshay has quit IRC | 16:09 | |
| *** dbecker has quit IRC | 16:09 | |
| *** honza has quit IRC | 16:09 | |
| *** Guest19794 has quit IRC | 16:09 | |
| *** evgenyl_ has quit IRC | 16:09 | |
| *** kamren has quit IRC | 16:09 | |
| *** szaher has quit IRC | 16:09 | |
| *** thedac has quit IRC | 16:09 | |
| *** zoli has quit IRC | 16:09 | |
| *** mugsie has quit IRC | 16:09 | |
| *** kambiz has quit IRC | 16:09 | |
| *** dtantsur has quit IRC | 16:09 | |
| *** lewo`` has quit IRC | 16:09 | |
| *** adriant has quit IRC | 16:09 | |
| *** zigo has quit IRC | 16:09 | |
| *** quite has quit IRC | 16:09 | |
| *** lennyb has quit IRC | 16:09 | |
| *** Adri2000 has quit IRC | 16:09 | |
| *** olivierb- has joined #openstack-infra | 16:09 | |
| *** lpetrut has joined #openstack-infra | 16:10 | |
| *** heyongli has quit IRC | 16:11 | |
| *** heyongli has joined #openstack-infra | 16:11 | |
| *** lbragstad is now known as elbragstad | 16:11 | |
| *** pguimaraes has quit IRC | 16:12 | |
| *** calbers has quit IRC | 16:12 | |
| *** dpawlik has joined #openstack-infra | 16:13 | |
| *** beekneemech has joined #openstack-infra | 16:14 | |
| *** sthussey has joined #openstack-infra | 16:14 | |
| *** weshay has joined #openstack-infra | 16:14 | |
| *** dbecker has joined #openstack-infra | 16:14 | |
| *** honza has joined #openstack-infra | 16:14 | |
| *** Guest19794 has joined #openstack-infra | 16:14 | |
| *** evgenyl_ has joined #openstack-infra | 16:14 | |
| *** kamren has joined #openstack-infra | 16:14 | |
| *** szaher has joined #openstack-infra | 16:14 | |
| *** thedac has joined #openstack-infra | 16:14 | |
| *** zoli has joined #openstack-infra | 16:14 | |
| *** mugsie has joined #openstack-infra | 16:14 | |
| *** kambiz has joined #openstack-infra | 16:14 | |
| *** lewo`` has joined #openstack-infra | 16:14 | |
| *** adriant has joined #openstack-infra | 16:14 | |
| *** zigo has joined #openstack-infra | 16:14 | |
| *** quite has joined #openstack-infra | 16:14 | |
| *** lennyb has joined #openstack-infra | 16:14 | |
| mnaser | clarkb: please let me know if you'd like us to dig into the NODE_FAILUREs | 16:14 |
| *** pguimaraes has joined #openstack-infra | 16:14 | |
| *** calbers has joined #openstack-infra | 16:14 | |
| *** kambiz has quit IRC | 16:14 | |
| *** kambiz has joined #openstack-infra | 16:15 | |
| *** udesale has quit IRC | 16:15 | |
| clarkb | mnaser: I do, can I ping you in about 10-15 minutes? I need caffeine | 16:15 |
| mnaser | no problem, | 16:15 |
| clarkb | also firefox + google account u2f apparently works now | 16:16 |
| clarkb | I need to get that set up | 16:16 |
| *** adreznec has quit IRC | 16:16 | |
| *** ihar has joined #openstack-infra | 16:17 | |
| *** dhellmann[m] has quit IRC | 16:17 | |
| *** warlord77[m] has quit IRC | 16:17 | |
| *** dpawlik has quit IRC | 16:17 | |
| *** jtomasek has quit IRC | 16:18 | |
| *** aspiers[m] has quit IRC | 16:18 | |
| *** linshuicheng[m] has quit IRC | 16:18 | |
| *** smcginnis[m] has quit IRC | 16:18 | |
| *** lbragstad[m] has quit IRC | 16:18 | |
| *** austinsun[m] has quit IRC | 16:18 | |
| *** adreznec has joined #openstack-infra | 16:19 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix for pools with different labels https://review.openstack.org/578831 | 16:21 |
| *** heyongli has quit IRC | 16:21 | |
| *** heyongli has joined #openstack-infra | 16:21 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Make websocket streaming more event-driven https://review.openstack.org/558646 | 16:22 |
| *** dtantsur has joined #openstack-infra | 16:23 | |
| *** dtantsur is now known as dtantsur|afk | 16:25 | |
| *** caphrim007 has quit IRC | 16:26 | |
| *** derekh has quit IRC | 16:27 | |
| *** boden has quit IRC | 16:28 | |
| *** electrofelix has quit IRC | 16:29 | |
| clarkb | mnaser: ok | 16:29 |
| clarkb | mnaser: yesterday when I looked at the logs there was pretty consistent node failurse and for those to happen the instance has to fail to build 3 times | 16:30 |
| *** greghaynes has quit IRC | 16:30 | |
| *** jpich has quit IRC | 16:30 | |
| *** thedac has quit IRC | 16:30 | |
| clarkb | mnaser: my reading of the shade munch data was that nova was returning error for the nova boot | 16:30 |
| *** jpich has joined #openstack-infra | 16:30 | |
| mnaser | clarkb: weird, i do only have 1 hv assigned to kata so it is possible that they were using up all the resources | 16:30 |
| *** ramishra has quit IRC | 16:30 | |
| *** jpich has quit IRC | 16:30 | |
| *** ihar has quit IRC | 16:31 | |
| *** andreww has quit IRC | 16:31 | |
| mnaser | around ~20ish vms | 16:31 |
| mnaser | is it still the case now? | 16:31 |
| *** jogo has quit IRC | 16:31 | |
| *** heyongli has quit IRC | 16:31 | |
| clarkb | it was before bed when I did my last recheck but not sure of currently | 16:31 |
| *** trown is now known as trown|lunch | 16:31 | |
| *** ykarel|afk is now known as ykarel|away | 16:31 | |
| *** heyongli has joined #openstack-infra | 16:31 | |
| clarkb | let me recheck the PR again | 16:31 |
| corvus | clarkb, mnaser: we can put all the vexxhost labels in their own pool and set max-servers | 16:33 |
| mnaser | corvus: i think that also has the positive effect of getting vms quicker.. | 16:34 |
| clarkb | corvus: ya I was considering that, though it won't help this problem since it is their jenkins eating the resources I think | 16:34 |
| clarkb | mnaser: yes | 16:34 |
| clarkb | I can start on that refactoring this morning | 16:34 |
| mnaser | i could bump up the resources assigned to that | 16:34 |
| clarkb | mnaser: what should I set max servers to on our end? | 16:34 |
| mnaser | clarkb: eh, 20? | 16:35 |
| clarkb | 4? | 16:35 |
| clarkb | oh cool | 16:35 |
| mnaser | or 4 | 16:35 |
| mnaser | :p | 16:35 |
| corvus | yeah, it won't help if the quota spans multiple pools and there's someone else in the pool. so to speak. | 16:35 |
| corvus | it would need its own account | 16:36 |
| *** ykarel|away has quit IRC | 16:37 | |
| clarkb | a separate account won't help this either as we already have separate accounts, the presumed issue is the cloud just can't provide more of those VMs to the multiple accounts using the flavor | 16:37 |
| clarkb | the quotas are oversubscribed, we could make them match available resources 1:1 though | 16:37 |
| pabelanger | can you have a quota per flavor in openstack? | 16:39 |
| *** pcaruana has quit IRC | 16:40 | |
| corvus | clarkb: we do have separate accounts? | 16:41 |
| *** heyongli has quit IRC | 16:41 | |
| clarkb | corvus: yes the account we use is different than the account kata jenkins is using | 16:41 |
| corvus | clarkb: oh no that's not what i meant | 16:41 |
| *** heyongli has joined #openstack-infra | 16:42 | |
| clarkb | corvus: that is where the conflict is coming from | 16:42 |
| clarkb | kata jenkins is booting enough VMs to fill the available slots for their flavor so when nodepool tries to boot them it cannot | 16:42 |
| corvus | clarkb: i mean that if we did use the same account as kata jenkins for our vexxhost flavors, and the quota on that account were set correctly, then there should be no conflict | 16:42 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Make vexxhost specific pool for kata flavors https://review.openstack.org/579220 | 16:42 |
| clarkb | ah | 16:42 |
| *** rlandy is now known as rlandy|brb | 16:43 | |
| clarkb | Shrews: tristanC ^ does that need the launchers to be updated with the bug fix that rdo had to write? | 16:43 |
| corvus | though, probably that would mean that we'd just patiently wait for jenkins to finish before we even tried to launch a vm. :) | 16:43 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Enable heartbeat on websocket streaming https://review.openstack.org/579222 | 16:43 |
| clarkb | if we can mostly cohabitate for now and longer term plan is to delete jenkins then we may just get away with it as is | 16:45 |
| pabelanger | clarkb: no, only issue was filled debug logs with exceptions, nodes were still working | 16:45 |
| pabelanger | IIRC | 16:45 |
| Shrews | clarkb: they don't *need* updated (things will still work, we'll just see KeyError exceptions in the logs), but it wouldn't hurt to update them either | 16:45 |
| * clarkb checks if that code is landed | 16:45 | |
| Shrews | clarkb: my change for that just landed | 16:45 |
| Shrews | clarkb: https://review.openstack.org/578831 | 16:45 |
| Shrews | not sure if it's puppeted yet | 16:46 |
| clarkb | ya will have to check that pip installed it before doing restarts | 16:46 |
| clarkb | if there are no objections I'll go ahead and start working to restart the four launchers | 16:46 |
| Shrews | go 4 it | 16:46 |
| clarkb | Shrews: nodepool==3.0.2.dev81 # git sha c08eb29 is in place on all four launchers according to pbr. That looks correct to me based on the git sha for Fix for pools with different labels | 16:49 |
| clarkb | Shrews: should be safe to restart one at a time right? | 16:50 |
| openstackgerrit | Merged openstack-infra/nodepool master: zk: retry initial zookeeper connection attempts https://review.openstack.org/576047 | 16:50 |
| clarkb | hrm should we wait and restart with ^ too? | 16:50 |
| clarkb | (I know people want a tag and we like to tag what we have run first) | 16:50 |
| clarkb | corvus: ^ you may have thoughts | 16:50 |
| Shrews | clarkb: yes, let's start with that one too if we're going to put it in the next tag | 16:51 |
| Shrews | clarkb: but yeah, safe to do 1-at-a-time | 16:51 |
| clarkb | alright I'll wait for puppet to get 576047 onto the launchers first | 16:51 |
| *** fried_rice is now known as fried_rolls | 16:51 | |
| *** heyongli has quit IRC | 16:52 | |
| *** heyongli has joined #openstack-infra | 16:52 | |
| *** e0ne has quit IRC | 16:58 | |
| *** jpena is now known as jpena|off | 17:00 | |
| *** warlord77[m] has joined #openstack-infra | 17:00 | |
| *** heyongli has quit IRC | 17:02 | |
| *** heyongli has joined #openstack-infra | 17:02 | |
| clarkb | nodepool==3.0.2.dev83 # git sha fe80343 is installed on all nodepool launchers now | 17:11 |
| *** florianf has quit IRC | 17:11 | |
| clarkb | infra-root I am going to restart nl03 first and see that it works, then will restart the other three if nl03 is happy | 17:11 |
| clarkb | Shrews: ^ | 17:11 |
| fungi | thanks for the heads up! | 17:11 |
| *** e0ne has joined #openstack-infra | 17:11 | |
| *** dpawlik has joined #openstack-infra | 17:11 | |
| corvus | clarkb: yeah, restart with the retry then tag sounds good | 17:11 |
| *** apevec has joined #openstack-infra | 17:11 | |
| *** pguimaraes has quit IRC | 17:11 | |
| *** openstack has quit IRC | 17:11 | |
| *** openstack has joined #openstack-infra | 17:13 | |
| *** ChanServ sets mode: +o openstack | 17:13 | |
| clarkb | nl03 has produced ready nodes in inap | 17:13 |
| clarkb | mnaser: vexxhost nodes appear to be slow to respond on the ssh port for key scanning | 17:13 |
| *** pguimaraes has joined #openstack-infra | 17:14 | |
| fungi | dhellmann: looks like it should be set from the "description" option under general settings | 17:15 |
| clarkb | mnaser: I can hit the nodes over ipv4 | 17:16 |
| clarkb | mnaser: but ipv6 from rax and ovh both seem unhappy | 17:16 |
| *** rlandy|brb is now known as rlandy | 17:16 | |
| mnaser | clarkb: oh i think that might be a root cause | 17:16 |
| clarkb | mnaser: ya wouldn't surprise me. I think we should still make the pool change though | 17:17 |
| openstackgerrit | Merged openstack-infra/zuul master: Shift log streaming code into StreamComponent https://review.openstack.org/558619 | 17:17 |
| clarkb | 2604:e100:1:0:f816:3eff:fe01:4fec vs 199.204.45.142 for example | 17:17 |
| dhellmann | fungi : it's not set on any of the archive messages (maybe that's normal?) but also not on the one I just received via the list from dtroyer. He replied to a message I sent where I had replied to an archive message that was downloaded, so maybe that's related? | 17:17 |
| clarkb | mnaser: 6f00235c-c10a-4f67-8816-9895db6912c2 is the uuid of that host | 17:18 |
| fungi | dhellmann: are you looking at a direct reply you received from him rather than through the listserv maybe? | 17:18 |
| dhellmann | fungi : oh, he did cc me, so maybe that's it | 17:18 |
| *** felipemonteiro has quit IRC | 17:19 | |
| clarkb | infra-root other than the vexxhost ssh problem nl03 is looking happy. I am going to restart the other launchers now | 17:20 |
| corvus | dhellmann, fungi: the mbox files on the list server have: List-Id: <starlingx-discuss.lists.starlingx.io> | 17:20 |
| fungi | i'm having trouble finding where the mm2 docs explicitly state how list-id is generated, but i did find some bespoke documentation for a site running a large installation which claimed description is what's used | 17:20 |
| fungi | though i don't think it's description | 17:20 |
| dhellmann | corvus , fungi : ok, I downloaded the text archive from http://lists.starlingx.io/pipermail/starlingx-discuss/ and those messages don't seem to, but perhaps that's something to do with how the archive is built | 17:21 |
| fungi | looks like it's probably the address of the list with s/@/./ | 17:21 |
| *** heyongli has quit IRC | 17:22 | |
| corvus | dhellmann: yeah. raw mbox downloads are disabled by default. do you need them? | 17:22 |
| *** heyongli has joined #openstack-infra | 17:23 | |
| mnaser | clarkb: do you mind getting traceroutes to me please? | 17:23 |
| mnaser | ipv6 is so annoying with routing and peering bs | 17:23 |
| dhellmann | maybe this is a false alarm. I have a filter set up for incoming messages for the list based on the list-id, so if I start seeing messages in my inbox I'll check back | 17:24 |
| clarkb | mnaser: can do | 17:24 |
| clarkb | infra-root all nodepool launchers have been restarted | 17:24 |
| dhellmann | corvus : nah, I just wanted to import the history so I could reply but I was able to do that | 17:24 |
| dhellmann | I was sure I had done that for another mailing list and received all of the relevant headers, but maybe not | 17:25 |
| clarkb | mnaser: http://paste.openstack.org/show/724609/ | 17:26 |
| *** gfidente|pto has quit IRC | 17:26 | |
| fungi | dhellmann: ftr, i've discovered you can also fine the message-id/in-reply-to by looking at the page source in pipermail, you just have to url-unescape special characters before using | 17:28 |
| *** dhajare has quit IRC | 17:28 | |
| openstackgerrit | Zane Bitter proposed openstack-infra/project-config master: Add gerritbot to #openstack-service-broker IRC channel https://review.openstack.org/579235 | 17:28 |
| mnaser | what the | 17:29 |
| mnaser | how is it ending up from gtt | 17:29 |
| mnaser | hmm | 17:29 |
| mnaser | clarkb: do you mind doing a quick reverse traceroute and see where it stops inside the nodepool vm? | 17:30 |
| clarkb | mnaser: so traceroute from nodepool VM to the nodepool launcher? | 17:30 |
| mnaser | yes if possible, just to see the reverse path | 17:30 |
| clarkb | mnaser: http://paste.openstack.org/show/724610/ | 17:32 |
| *** heyongli has quit IRC | 17:33 | |
| *** heyongli has joined #openstack-infra | 17:33 | |
| johnsom | fyi: Accessing files from outside the working dir /var/lib/zuul/builds/d144fa96f17041fcbf71db0ff87e8401/work is prohibited", "path": "/tmp/ansible.E3LzjG" so tempfile does not honor those other work directory settings. | 17:35 |
| corvus | mtreinish: brainstorming question for you: i'd like to have stestr discover all the test*.py files in http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles but that directory and even the intervening directories aren't importable. do you have any ideas how i could accomplish that? | 17:35 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Remove release-tools configuration https://review.openstack.org/579194 | 17:36 |
| *** aspiers[m] has joined #openstack-infra | 17:36 | |
| *** smcginnis[m] has joined #openstack-infra | 17:36 | |
| *** dhellmann[m] has joined #openstack-infra | 17:36 | |
| *** austinsun[m] has joined #openstack-infra | 17:36 | |
| *** linshuicheng[m] has joined #openstack-infra | 17:36 | |
| *** lbragstad[m] has joined #openstack-infra | 17:36 | |
| corvus | johnsom: that's surprising -- it uses mkstemp which honors TMP: https://docs.python.org/2/library/tempfile.html#tempfile.mkstemp | 17:37 |
| corvus | johnsom: you didn't set the 'path' option or anything, right? | 17:40 |
| *** dpawlik has joined #openstack-infra | 17:40 | |
| johnsom | corvus Nope: https://review.openstack.org/#/c/578942/4/playbooks/migration-test/run.yaml | 17:41 |
| *** trown|lunch is now known as trown | 17:41 | |
| *** heyongli has quit IRC | 17:43 | |
| *** heyongli has joined #openstack-infra | 17:43 | |
| corvus | johnsom: ah! you need to delegate "create temp file" and "get octavia db" to localhost | 17:43 |
| corvus | they'll both run on the remote node by default | 17:44 |
| *** fultonj has joined #openstack-infra | 17:45 | |
| mtreinish | corvus: hmm, well stestr is written to use test discovery more or less from stdlib unittest discovery which needs things to be importable (at least relative to the top level directory, which is configurable) | 17:45 |
| *** slaweq has quit IRC | 17:47 | |
| corvus | johnsom: sorry, the set_fact probably doesn't need to be delegated, just tempfile. | 17:47 |
| corvus | johnsom, tobiash, pabelanger, mordred: do we allow delegating tempfile to localhost? :) | 17:47 |
| mtreinish | heh, although what you're looking for is how pytest actually does test discovery (just a recursive scan for test_*py files) but that's part of what makes it not 100% unittest compatible | 17:48 |
| pabelanger | path is outside of safe: /tmp/ansible.E3LzjG | 17:48 |
| pabelanger | so that is likey why it is failing | 17:48 |
| corvus | pabelanger: yeah we got that | 17:48 |
| mtreinish | which is something I've been trying to make sure stestr was | 17:48 |
| corvus | pabelanger: my question was: do we allow tempfile to delegate to localhost? | 17:48 |
| pabelanger | yes, just looking now | 17:49 |
| clarkb | https://review.openstack.org/#/c/578556/ is an easy review to update our cert checker list | 17:49 |
| corvus | mtreinish: hrm. i'm assuming i could run the find myself and write out a config or something? | 17:50 |
| corvus | or maybe i should just use pytest for this? | 17:51 |
| mnaser | clarkb: i will dig into this shortly. looks like some weird ipv6 stuff going on | 17:51 |
| mtreinish | corvus: my first thought I have is that you could manually do the discovery and built the test suite yourself after doing your own scan and loading the module. Then just do that in a load_tests entrypoint which unittest discovery will work on | 17:52 |
| pabelanger | corvus: I am not sure if we block tempfile or not. I don't believe I have tried using it on localhost | 17:52 |
| corvus | mtreinish: that sounds promising | 17:52 |
| pabelanger | but if blocked, I agree we could whitelist it if suffix is in safepath | 17:53 |
| mtreinish | corvus: the only config options for stestr I have right now to do this kind of thing is --load-list which still uses python paths to import relative to the top level. It just bypasses discovery and loads explicitly from the list | 17:53 |
| pabelanger | or path | 17:53 |
| *** heyongli has quit IRC | 17:53 | |
| *** david-lyle has joined #openstack-infra | 17:53 | |
| *** heyongli has joined #openstack-infra | 17:53 | |
| mtreinish | corvus: the manually doing discovery via load_tests is actually similar to what we do in tempest for plugins and stuff | 17:53 |
| corvus | pabelanger: yeah, i think we'd have to check prefix and path | 17:53 |
| mtreinish | corvus: https://github.com/openstack/tempest/blob/master/tempest/test_discover/test_discover.py#L26-L54 | 17:54 |
| corvus | mtreinish: awesome, i think that's enough to get me started, thanks! | 17:55 |
| *** armaan has quit IRC | 17:55 | |
| logan- | my bad johnsom: that tempfile should have had a delegate_to: localhost | 17:55 |
| mtreinish | corvus: sure np, let me know if you need any help or other questions | 17:55 |
| logan- | oh corvus already said that :P | 17:55 |
| *** dklyle has quit IRC | 17:56 | |
| logan- | I don't think you should delegate the fetch though | 17:56 |
| corvus | logan-, pabelanger, johnsom: my guess is that we're going to block tempfile on localhost in zuul. but if so, i think that's a bug and we should allow it. johnsom it would be helpful if you can verify that; if we see the error, let's patch zuul. | 17:56 |
| *** dpawlik has quit IRC | 17:57 | |
| corvus | so just add "delegate_to: localhost" to the tempfile task and see if we get an error about it being blocked | 17:57 |
| *** dklyle has joined #openstack-infra | 17:57 | |
| johnsom | corvus You got it. | 17:58 |
| *** david-lyle has quit IRC | 17:58 | |
| pabelanger | +1 | 18:00 |
| *** graphene has joined #openstack-infra | 18:00 | |
| *** dhajare has joined #openstack-infra | 18:00 | |
| *** graphene has quit IRC | 18:01 | |
| *** dhajare has quit IRC | 18:02 | |
| *** graphene has joined #openstack-infra | 18:02 | |
| *** graphene has quit IRC | 18:02 | |
| *** caphrim007 has joined #openstack-infra | 18:02 | |
| *** xarses has joined #openstack-infra | 18:03 | |
| *** greghaynes has joined #openstack-infra | 18:03 | |
| *** dhajare has joined #openstack-infra | 18:03 | |
| *** heyongli has quit IRC | 18:03 | |
| *** heyongli has joined #openstack-infra | 18:04 | |
| *** ihar has joined #openstack-infra | 18:04 | |
| *** thedac has joined #openstack-infra | 18:04 | |
| *** graphene has joined #openstack-infra | 18:04 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: drop pypy jobs from cliff https://review.openstack.org/579239 | 18:04 |
| *** graphene has quit IRC | 18:05 | |
| *** kmalloc is now known as vmalloc | 18:06 | |
| *** vmalloc is now known as kmalloc | 18:06 | |
| *** graphene has joined #openstack-infra | 18:06 | |
| *** slaweq has joined #openstack-infra | 18:10 | |
| tobiash | corvus: I think we block it | 18:12 |
| tobiash | corvus: but if we want to open it we should probably restrict it to the safe paths as /tmp is a shared location that should stay blocked | 18:13 |
| *** heyongli has quit IRC | 18:14 | |
| *** heyongli has joined #openstack-infra | 18:14 | |
| *** boden has joined #openstack-infra | 18:15 | |
| corvus | tobiash: agreed. we set the TMP variable, so i think as long as we apply the safe patch check to path and prefix we should be fine | 18:18 |
| tobiash | Yes but the tmp variable is probably gone within the module | 18:20 |
| tobiash | But it supports setting a path to operate in | 18:21 |
| corvus | tobiash: oh, hrm. well, johnsom's test should tell us that too | 18:22 |
| *** heyongli has quit IRC | 18:24 | |
| corvus | johnsom: i'm about to AFK -- after doing the tempfile test, i think you could get this to work without any changes to zuul by fetching it to a known location instead of using tempfile. so fetch to "{{zuul.work_dir}}/octavia.conf" then use that same path in the lookup. | 18:24 |
| corvus | logan-: ^ | 18:24 |
| *** heyongli has joined #openstack-infra | 18:24 | |
| logan- | ++ yeah if zuul passes a scratch dir location no need for tempfiles :) | 18:25 |
| johnsom | hmm, ok. I'm leaning towards just doing this with awk and calling it a day. The queues for these tests are a bit long to wait | 18:25 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul master: Change "core developer" references https://review.openstack.org/579241 | 18:26 |
| corvus | mtreinish: it's working locally for me, thanks! | 18:27 |
| mtreinish | corvus: ok cool, glad that worked :) | 18:29 |
| *** bobh has joined #openstack-infra | 18:29 | |
| *** bobh has quit IRC | 18:30 | |
| openstackgerrit | Merged openstack-infra/zuul master: Enable heartbeat on websocket streaming https://review.openstack.org/579222 | 18:33 |
| *** jamesmcarthur has quit IRC | 18:34 | |
| *** heyongli has quit IRC | 18:34 | |
| *** heyongli has joined #openstack-infra | 18:34 | |
| *** e0ne has quit IRC | 18:36 | |
| *** e0ne has joined #openstack-infra | 18:37 | |
| *** rwsu has quit IRC | 18:38 | |
| *** felipemonteiro has joined #openstack-infra | 18:38 | |
| *** caphrim007 has quit IRC | 18:38 | |
| *** caphrim007 has joined #openstack-infra | 18:39 | |
| *** armaan has joined #openstack-infra | 18:40 | |
| *** rpioso is now known as rpioso|reboot | 18:40 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:40 | |
| *** beagles_mtg is now known as beagles | 18:41 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update certcheck to include new domains https://review.openstack.org/578556 | 18:42 |
| clarkb | anyone willing to review https://review.openstack.org/#/c/579220/ as a next step on the kata work? | 18:42 |
| *** ykarel|away has joined #openstack-infra | 18:42 | |
| clarkb | I'm also following up on the dynamic mtu determination as I think its close at this point. Going to update devstack-gate to use the zuul-jobs stuff | 18:42 |
| *** felipemonteiro has quit IRC | 18:44 | |
| *** heyongli has quit IRC | 18:44 | |
| *** heyongli has joined #openstack-infra | 18:45 | |
| pabelanger | +2 | 18:45 |
| pabelanger | left comment too | 18:45 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Dynamically determine overlay MTU https://review.openstack.org/578146 | 18:47 |
| clarkb | pabelanger: I'm half expecting max servers to go up based on mnasers earlier message, I just didn't want to create conflicts which may have been ipv6 issues all along | 18:48 |
| clarkb | pabelanger: lets see what mnaser thinks about it when he has a chance to look up from ipv6 problems | 18:48 |
| pabelanger | wfm | 18:48 |
| *** zaneb has quit IRC | 18:51 | |
| *** heyongli has quit IRC | 18:55 | |
| *** heyongli has joined #openstack-infra | 18:55 | |
| clarkb | dtroyer: stx-config may want to remove the none voting linters job from the gate (will speed things up for you since the job isn't working at all) | 18:57 |
| *** jamesmcarthur has joined #openstack-infra | 18:57 | |
| *** jamesmcarthur has quit IRC | 18:57 | |
| fungi | well, and also non-voting jobs in the gate pipeline are fairly useless | 18:57 |
| *** jamesmcarthur has joined #openstack-infra | 18:57 | |
| fungi | another zuul wishlist item: a pipeline option to silently skip non-voting jobs rather than run them | 18:58 |
| *** rpioso|reboot is now known as rpioso | 18:58 | |
| clarkb | I'm going to pop out for lunch now. mnaser please ping if we can help at all. | 18:58 |
| fungi | then we could set that on our gate pipeline and stop having to explain to people why they should omit non-voting jobs from the gate | 18:59 |
| dtroyer | clarkb: ok, will do. I used those since it seemed to not work with just noop | 18:59 |
| clarkb | dtroyer: noop should be sufficient for getting gating to work | 18:59 |
| dtroyer | merges were not happening IIRC | 19:00 |
| dtroyer | the gate queue processed it | 19:00 |
| *** felipemonteiro_ has quit IRC | 19:00 | |
| clarkb | dtroyer: if noop isn't enough to get changes merged that would be a bug, but I'm fairly certain other projects are using a noop setup without problems | 19:00 |
| dtroyer | we'll find out :) | 19:00 |
| *** jamesmcarthur has quit IRC | 19:02 | |
| dtroyer | clarkb: let's see if I remembered wrong: https://review.openstack.org/579252 | 19:04 |
| fungi | we have quite a lot of projects configured with just noop | 19:05 |
| *** heyongli has quit IRC | 19:05 | |
| corvus | they tend to be our most stable projects in testing | 19:05 |
| dtroyer | I hate it when noop fails | 19:05 |
| fungi | and the intent is that as soon as zuul notices you've approved a change, it gets merged | 19:05 |
| *** heyongli has joined #openstack-infra | 19:05 | |
| dtroyer | it may have been something else I was lacking and fixed at the same time | 19:05 |
| dtroyer | changing multiple things at once and all that | 19:06 |
| fungi | it's the only way to party | 19:06 |
| *** peereb has quit IRC | 19:07 | |
| dtroyer | yaml still doesn't like me... | 19:07 |
| *** heyongli has quit IRC | 19:15 | |
| *** heyongli has joined #openstack-infra | 19:15 | |
| johnsom | corvus FYI: "Executing local code is prohibited" so delegate_to: localhost is a no-go | 19:18 |
| *** d0ugal__ has joined #openstack-infra | 19:18 | |
| *** yolanda_ has joined #openstack-infra | 19:19 | |
| *** yolanda__ has joined #openstack-infra | 19:22 | |
| *** yolanda has quit IRC | 19:23 | |
| *** jamesmcarthur has joined #openstack-infra | 19:23 | |
| *** yolanda_ has quit IRC | 19:24 | |
| fungi | though it leaves me increasingly confident in our security measures for the executors | 19:25 |
| *** heyongli has quit IRC | 19:25 | |
| *** heyongli has joined #openstack-infra | 19:26 | |
| *** jamesmca_ has joined #openstack-infra | 19:27 | |
| *** jamesmcarthur has quit IRC | 19:27 | |
| dtroyer | clarkb: https://review.openstack.org/#/c/579252/2 merged, I remembered wrong… I'll do the other stx repos too | 19:28 |
| *** d0ugal__ has quit IRC | 19:28 | |
| *** lpetrut has quit IRC | 19:31 | |
| *** jamesmca_ has quit IRC | 19:31 | |
| *** rossella_s has quit IRC | 19:33 | |
| *** heyongli has quit IRC | 19:36 | |
| *** heyongli has joined #openstack-infra | 19:36 | |
| smcginnis | Thank you to whoever fixes zuul status filtering to remove the extra queues. :) | 19:42 |
| fungi | i want to say that was part of mnaser's status page improvements | 19:43 |
| smcginnis | kudos | 19:43 |
| *** jamesmcarthur has joined #openstack-infra | 19:43 | |
| dhellmann | oh, there is not enough beer in the world for mnaser then | 19:43 |
| dhellmann | that has bugged me forever but I couldn't figure out how to fix it | 19:44 |
| smcginnis | Same | 19:44 |
| fungi | mailman 3 is making me learn things | 19:44 |
| mnaser | Yay! I’m glad people see the impact. | 19:44 |
| fungi | things about uwsgi | 19:44 |
| fungi | please make the learning stop! | 19:45 |
| clarkb | mnaser: I'm back from lunch fwiw, can help debug further if you need that | 19:45 |
| smcginnis | Better than going to groups.io. ;) | 19:45 |
| mnaser | I’m going to dig more about the ipv6 soon | 19:45 |
| mnaser | I’m stuck in a family lunch 😅 | 19:45 |
| fungi | the knowledge, it burns | 19:45 |
| *** heyongli has quit IRC | 19:46 | |
| *** heyongli has joined #openstack-infra | 19:46 | |
| *** jamesmcarthur has quit IRC | 19:48 | |
| clarkb | as a friendly reminder to the channel next wednesday is a big holiday in the USA, you'll probably find that things quiet down a bit next week. I expect I'll be around for most of the week though | 19:48 |
| *** jamesmcarthur has joined #openstack-infra | 19:49 | |
| *** zaneb has joined #openstack-infra | 19:50 | |
| clarkb | corvus: dhellmann did we decide to use pbx.o.o? was there a specific room number chosen? | 19:51 |
| fungi | i'll be around, though i do have some food prep and entertainment obligations on wednesday | 19:51 |
| dhellmann | clarkb : I think yes and no, respectively | 19:51 |
| fungi | quick somebody randomly generate an integer between 6000 and 7000? | 19:52 |
| dhellmann | 5999 | 19:52 |
| dhellmann | oh, so close | 19:52 |
| fungi | fsil! | 19:52 |
| fungi | er, fail | 19:52 |
| fungi | i fail-failed | 19:52 |
| dhellmann | haha | 19:52 |
| *** jamesmcarthur has quit IRC | 19:53 | |
| fungi | i'll dial in and listen, but probably put the phone on mute most of the time as i'm on the patio doing some grill prep (while trying to make apache work with a unix socket to mailman3's postorious and hyperkitty django projects) | 19:55 |
| corvus | fungi: can the "post via web interface" thing be disabled on a per-list or per-vhost basis? | 19:56 |
| * fungi shakes fist at ubuntu/debian "mailman3-web" package for lacking actual webserver config | 19:56 | |
| fungi | corvus: that is one of the questions i'd love to be able to answer! | 19:56 |
| dhellmann | why would we want to disable it? | 19:56 |
| *** heyongli has quit IRC | 19:56 | |
| clarkb | I'm in the backyard too, but only because there are sleeping kids in the house and I don't want to wake them | 19:56 |
| *** heyongli has joined #openstack-infra | 19:56 | |
| dhellmann | so, which room are we in? 6000? | 19:57 |
| *** dpawlik has joined #openstack-infra | 19:57 | |
| clarkb | wfm | 19:57 |
| *** zul_ has quit IRC | 19:59 | |
| corvus | dhellmann: i don't think it's appropriate for all communities. maybe it's appropriate for openstack. i'm not sure the kind of participation it's designed for is what we need for zuul. | 19:59 |
| dhellmann | corvus : ok, that makes sense, I guess | 19:59 |
| *** dhajare has quit IRC | 20:00 | |
| fungi | i'll dial in momentarily. my cell service has decided to hate me | 20:01 |
| *** dpawlik has quit IRC | 20:02 | |
| *** edmondsw has quit IRC | 20:02 | |
| fungi | i'm in now | 20:02 |
| fungi | gol!!! http://lists-dev.openstack.org/mailman3/postorius/lists/ | 20:03 |
| fungi | very empty, but that's because i've created exactly nothing | 20:04 |
| *** heyongli has quit IRC | 20:06 | |
| *** heyongli has joined #openstack-infra | 20:07 | |
| *** edmondsw has joined #openstack-infra | 20:09 | |
| *** fried_rolls is now known as fried_rice | 20:09 | |
| *** edmondsw has quit IRC | 20:14 | |
| *** rwsu has joined #openstack-infra | 20:16 | |
| *** heyongli has quit IRC | 20:17 | |
| *** heyongli has joined #openstack-infra | 20:17 | |
| *** yamamoto has quit IRC | 20:17 | |
| *** ykarel|away has quit IRC | 20:19 | |
| *** SpamapS has quit IRC | 20:22 | |
| *** jamesmcarthur has joined #openstack-infra | 20:23 | |
| fungi | i'm always amused at the meat-based message collision avoidance/recovery protocols which emerge from conference calls | 20:26 |
| *** heyongli has quit IRC | 20:27 | |
| fungi | so many parallels with standard protocols | 20:27 |
| *** heyongli has joined #openstack-infra | 20:27 | |
| fungi | s/standard/silicon-based/ | 20:27 |
| *** jamesmcarthur has quit IRC | 20:27 | |
| *** edmondsw has joined #openstack-infra | 20:31 | |
| *** dhajare has joined #openstack-infra | 20:32 | |
| *** heyongli has quit IRC | 20:37 | |
| *** heyongli has joined #openstack-infra | 20:37 | |
| *** xarses has quit IRC | 20:39 | |
| *** xarses has joined #openstack-infra | 20:39 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Ensure iptables-services is installed when persisting firewall config https://review.openstack.org/579283 | 20:47 |
| *** heyongli has quit IRC | 20:47 | |
| *** heyongli has joined #openstack-infra | 20:48 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Ensure iptables-services is installed when persisting firewall config https://review.openstack.org/579283 | 20:49 |
| dhellmann | test-release-openstack-python3 | 20:50 |
| dhellmann | publish-to-pypi-python3 | 20:50 |
| *** slaweq has quit IRC | 20:52 | |
| *** slaweq has joined #openstack-infra | 20:54 | |
| *** heyongli has quit IRC | 20:58 | |
| *** heyongli has joined #openstack-infra | 20:58 | |
| *** kgiusti has left #openstack-infra | 20:59 | |
| *** slaweq has quit IRC | 21:01 | |
| *** trown is now known as trown|outtypewww | 21:05 | |
| *** kjackal has quit IRC | 21:06 | |
| *** heyongli has quit IRC | 21:08 | |
| *** heyongli has joined #openstack-infra | 21:08 | |
| *** agopi_ is now known as agopi | 21:08 | |
| *** caphrim007 has quit IRC | 21:12 | |
| *** e0ne has quit IRC | 21:13 | |
| *** e0ne has joined #openstack-infra | 21:15 | |
| openstackgerrit | François Magimel proposed openstack-infra/storyboard master: Replace broken links in the doc https://review.openstack.org/579287 | 21:16 |
| *** ldnunes has quit IRC | 21:16 | |
| *** janki has quit IRC | 21:17 | |
| *** heyongli has quit IRC | 21:18 | |
| *** yamamoto has joined #openstack-infra | 21:18 | |
| *** heyongli has joined #openstack-infra | 21:18 | |
| *** nicolasbock has quit IRC | 21:18 | |
| *** yamamoto has quit IRC | 21:22 | |
| *** heyongli has quit IRC | 21:28 | |
| *** heyongli has joined #openstack-infra | 21:29 | |
| dhellmann | corvus , fungi , mordred , clarkb : are you all still on the line talking? | 21:32 |
| clarkb | dhellmann: yes | 21:32 |
| dhellmann | ok, I'll dial back in | 21:32 |
| *** zaneb has quit IRC | 21:35 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Force nodepool to use ipv4 in vexxhost https://review.openstack.org/579290 | 21:36 |
| clarkb | infra-root ^ is an option we can use with vexxhost while ipv6 isn't working | 21:36 |
| fungi | mmm, i just lost audio on the call bridge | 21:36 |
| fungi | oh, i disconnected entirely | 21:37 |
| clarkb | fungi: we heard you drop ya | 21:37 |
| fungi | back on now | 21:37 |
| fungi | i blame deutsche telecom's questionable service network in the eastern islands of nc | 21:37 |
| *** heyongli has quit IRC | 21:39 | |
| *** heyongli has joined #openstack-infra | 21:39 | |
| *** agopi is now known as agopi|off | 21:39 | |
| *** agopi|off has quit IRC | 21:44 | |
| *** owalsh has quit IRC | 21:45 | |
| *** e0ne has quit IRC | 21:46 | |
| *** owalsh has joined #openstack-infra | 21:46 | |
| *** heyongli has quit IRC | 21:49 | |
| *** heyongli has joined #openstack-infra | 21:49 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Add py35 jobs to python-tempestconf https://review.openstack.org/579293 | 21:52 |
| *** dhajare has quit IRC | 21:57 | |
| *** heyongli has quit IRC | 21:59 | |
| *** heyongli has joined #openstack-infra | 21:59 | |
| openstackgerrit | François Magimel proposed openstack-infra/storyboard master: Replace broken links and http links in the doc https://review.openstack.org/579287 | 22:06 |
| clarkb | mordred: fungi corvus we should probably consider https://review.openstack.org/579290 before the weekend just so that we get working cloud and mnaser doesn't have to feel so rushed (though I suppose it is probably broken for other users too) | 22:06 |
| *** dtantsur|afk has quit IRC | 22:06 | |
| corvus | fungi: i'm eating chocolate mushrooms. not sure how you feel about that. http://www.meijiamerica.com/products/chocorooms.html | 22:07 |
| *** dtantsur has joined #openstack-infra | 22:07 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: Add py35 jobs to python-tempestconf https://review.openstack.org/579293 | 22:07 |
| clarkb | corvus: also do we still want to do a nodepool release today? I think the launchers have been working fine | 22:07 |
| mnaser | clarkb: yeah, sorry, i am a bit overwhelemd with stuff right now and routing changes during the day is no fun | 22:07 |
| clarkb | mnaser: ya no worries, we have a toggle to fix this on our end as long as ipv4 is working (and the nodes I checked it was) | 22:07 |
| corvus | clarkb: ipv4 change lgtm. nodepool looks good after restart? | 22:08 |
| fungi | awesome, i will check out change and chocolate shrooms once i'm done with my own repast | 22:08 |
| clarkb | corvus: yes the only nodepool issue I'm aware of is being unable to talk to vexxhost nodes over ipv6 | 22:08 |
| *** kaiokmo has quit IRC | 22:08 | |
| clarkb | corvus: otherwise I think it looks good for a release | 22:08 |
| *** heyongli has quit IRC | 22:09 | |
| *** heyongli has joined #openstack-infra | 22:10 | |
| clarkb | mnaser: if you want something a bit easier maybe you want ot weigh in on https://review.openstack.org/#/c/579220/ (that is the pool change we talked about) specifically curious if you want me to address pabelanger's comment or maybe bump up max servers? | 22:10 |
| mnaser | clarkb: i'm okay with max-servers: 1 to avoid losing the +2's lol | 22:11 |
| *** cjloader has quit IRC | 22:11 | |
| clarkb | mnaser: ok | 22:11 |
| mnaser | clarkb: +w | 22:12 |
| *** hongbin has quit IRC | 22:13 | |
| *** boden has quit IRC | 22:13 | |
| clarkb | thank you | 22:13 |
| *** agopi|off has joined #openstack-infra | 22:18 | |
| *** heyongli has quit IRC | 22:20 | |
| *** heyongli has joined #openstack-infra | 22:20 | |
| fungi | corvus: do those contain actual mushroom, or are they just really squat pocky? | 22:25 |
| fungi | nevermind; english-language descriptions i've been able to find imply no mushroom content | 22:28 |
| openstackgerrit | Merged openstack-infra/project-config master: Make vexxhost specific pool for kata flavors https://review.openstack.org/579220 | 22:30 |
| *** heyongli has quit IRC | 22:30 | |
| *** heyongli has joined #openstack-infra | 22:30 | |
| corvus | fungi: rally squat pocky. but with two! (nearly indistinguishable) kinds of chocolate on the same stick! | 22:31 |
| fungi | wow, i'll have to get myself some | 22:32 |
| corvus | i. um. bought a box at costco. | 22:32 |
| fungi | there is no shame in costco chocolate | 22:32 |
| fungi | i'll likely get them from a*zon due to lack of real retail stores in my vicinity | 22:33 |
| *** tpsilva has quit IRC | 22:34 | |
| corvus | it's a display box too, so you pull open half the box and set it on the counter like a snack shop or something. i charge $0.10 for a 0.75 oz bag. | 22:34 |
| *** beekneemech has quit IRC | 22:34 | |
| fungi | heh | 22:34 |
| *** mriedem has quit IRC | 22:35 | |
| fungi | opening a japanese 7-11 in your kitchen? | 22:35 |
| fungi | as long as you have rice balls and suntory highball in a can, i bet you can turn a profit | 22:36 |
| corvus | apparently i'd need to charge $0.42/ea to break even. i am bad at business. | 22:37 |
| *** heyongli has quit IRC | 22:40 | |
| *** heyongli has joined #openstack-infra | 22:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Publish docs on release https://review.openstack.org/579300 | 22:42 |
| clarkb | mordred: I have reviewed the config mgmt spec | 22:44 |
| openstackgerrit | Erich Cordoba proposed openstack-infra/project-config master: Add StarlingX documentation repository https://review.openstack.org/579301 | 22:44 |
| clarkb | overall I think it spells out a reasonable path forward, definitely bite sized chunks without trying to solve everything all at once \o/ did have some questions and comments though for when you have a moment (they are in the change review) | 22:45 |
| openstackgerrit | Merged openstack-infra/system-config master: Force nodepool to use ipv4 in vexxhost https://review.openstack.org/579290 | 22:46 |
| corvus | ianw: you have a pending message to zuul-announce, but it suggests deprecating a role at the beginning of this week. do you want to resend that with a date in the future? i'll be around to approve it this time. | 22:47 |
| openstackgerrit | Erich Cordoba proposed openstack-infra/project-config master: Add Zuul job for stx-docs https://review.openstack.org/579303 | 22:48 |
| *** heyongli has quit IRC | 22:50 | |
| *** heyongli has joined #openstack-infra | 22:51 | |
| corvus | clarkb: the add-host part of the 'run puppetmaster ansible jobs on system-config in zuul' from the spec means that they'll be trusted jobs; we can prevent them from being used in other projects. | 22:52 |
| clarkb | corvus: as long as we keep it out of the inventory by not using nodepool right? | 22:53 |
| corvus | clarkb: yeah. i think mordred literally meant using the add_host ansible task (like we do for logs.o.o and tarballs.o.o). | 22:53 |
| *** rwsu has quit IRC | 22:55 | |
| *** pguimaraes has quit IRC | 22:56 | |
| clarkb | it is looking like nl03 is not happy with the split pools. | 22:56 |
| clarkb | I'm wondering if I need to restart nodepool launcher for it to properly rebuild the provider handlers for that | 22:56 |
| corvus | shouldn't, but maybe worth doing as additional debugging | 22:57 |
| clarkb | http://paste.openstack.org/show/724622/ looks similar to previous issues with this that should've been fixed but I think that is a different traceback | 22:58 |
| clarkb | vexxhost-specific is the pool with that node type specified | 22:58 |
| *** heyongli has quit IRC | 23:01 | |
| *** heyongli has joined #openstack-infra | 23:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: WIP: Add a role to return file comments https://review.openstack.org/579033 | 23:02 |
| clarkb | if pool and not node.pool == pool.name is the check we have for lining up the pools | 23:03 |
| clarkb | but we only do that if pool is set and we are calling this without a pool | 23:03 |
| clarkb | so I think this is failing because we have nodes in zk without pool set | 23:04 |
| fungi | huh, yeah that tb is not encouraging | 23:04 |
| corvus | clarkb: oh, i wonder if that's because there's still a f28-vexxhost node from outside that pool | 23:05 |
| clarkb | corvus: ya that is what I am thinking | 23:05 |
| fungi | does zk have an equivalent of database migrations? | 23:05 |
| corvus | https://review.openstack.org/579046 would have been helpful | 23:05 |
| corvus | fungi: this isn't so much of a schema change as config change | 23:05 |
| corvus | fungi: (but no) | 23:05 |
| fungi | ahh | 23:05 |
| clarkb | ya so the pool for that held node will be main and then main no longer has the info for it | 23:06 |
| corvus | so the idea is that we previously created a fedora-28-vexxhost node in the "main" vexxhost pool. but now that's not permitted by our config file, so we assume it can't happen. | 23:06 |
| clarkb | and quota calculations fail | 23:06 |
| fungi | i see. and no mechanism to move nodes from one pool to another in concert with the config update | 23:07 |
| clarkb | I could probably update the record in zk manually to change the pool value | 23:07 |
| corvus | fungi: right. | 23:07 |
| clarkb | I held that node for salvandor.fuentes to debug kata docker issues on vexxhost | 23:07 |
| corvus | clarkb: assuming you don't want to delete it, that may be the best approach. | 23:07 |
| clarkb | corvus: do you think doing manual surgery like that is advisable or should I just ask him if I can delete the node and get a new one later? | 23:08 |
| clarkb | corvus: I don't want to delete it under them if they are debugging as there may be intermediate state thye still need. | 23:08 |
| corvus | clarkb: as long as it's not locked (it should not be if it's held) should be fine. | 23:08 |
| corvus | (and, really the only issue with doing it while it's locked is that someone will later come back and overwrite the change) | 23:08 |
| clarkb | it is unlocked | 23:08 |
| fungi | is there a zk client you use to alter records? | 23:08 |
| clarkb | fungi: ya we have it documented in system-config docs | 23:09 |
| clarkb | I'm going to make an attempt at this | 23:09 |
| fungi | oh, awesome | 23:09 |
| clarkb | where this == update the pool name in the node record | 23:09 |
| fungi | i hadn't seen that but reading now | 23:09 |
| corvus | i'm going to make an attempt at handling this case in the code | 23:09 |
| *** heyongli has quit IRC | 23:11 | |
| *** heyongli has joined #openstack-infra | 23:11 | |
| openstackgerrit | Erich Cordoba proposed openstack-infra/project-config master: Add StarlingX documentation repository https://review.openstack.org/579301 | 23:11 |
| clarkb | now to find the command that lets me write a znode | 23:13 |
| clarkb | (get to get the current value) | 23:13 |
| clarkb | set to set a value looks like | 23:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Handle node no longer in pool error https://review.openstack.org/579309 | 23:14 |
| openstackgerrit | Erich Cordoba proposed openstack-infra/project-config master: Add Zuul job for stx-docs https://review.openstack.org/579303 | 23:14 |
| corvus | it would have been handly if https://review.openstack.org/579046 worked | 23:15 |
| *** linkmark has quit IRC | 23:15 | |
| clarkb | ok manually edit it | 23:15 |
| clarkb | now to see if nodepool becomes happier about it | 23:15 |
| clarkb | it appears to be building a new node now | 23:16 |
| clarkb | I think that did it | 23:16 |
| clarkb | then next puppet run should set ipv4 forced to true and we'll start booting instances that nodepool cna tlk to | 23:18 |
| corvus | i read 'cna' as 'canna' | 23:19 |
| corvus | but i'm reading a book with a lot of scottish dialect right now. | 23:19 |
| clarkb | am I typing in scotch? | 23:19 |
| corvus | i think it is time to drink some scotch | 23:20 |
| bkero | corvus: ever try to read Trainspotting? | 23:20 |
| bkero | or anything else by Irvine Welsh | 23:21 |
| corvus | no, but now i kind of want to watch the film with subtitles | 23:21 |
| clarkb | the last key error was at 2018-06-29 23:15:01,273 | 23:21 |
| fungi | i didn't even know the film was based on a book. how pedestrian am i? | 23:21 |
| *** heyongli has quit IRC | 23:21 | |
| *** heyongli has joined #openstack-infra | 23:21 | |
| bkero | fungi: not just any book | 23:22 |
| clarkb | corvus: maybe the znodes should carry all of the info necessary to do a quota calculation | 23:22 |
| bkero | fungi: https://www.thestudentroom.co.uk/attachment.php?attachmentid=329485&stc=1 | 23:22 |
| clarkb | corvus: then we can update config safely and quota lookups never go through the config but just what the znode has already recorded about itself | 23:22 |
| bkero | It's a difficult read to say the least | 23:23 |
| fungi | bkero: it's certainly no finnegan's wake! | 23:23 |
| fungi | but yes, looks a wee might distractin | 23:23 |
| bkero | Thankfully, you're right | 23:23 |
| bkero | Welsh is only about 0.66 on the Joyce scale | 23:24 |
| fungi | moreso than a clockwork orange | 23:24 |
| fungi | i tried in vain to teach myself welsh over a number of years | 23:24 |
| corvus | clarkb: hrm, would that make it hard to implement max-servers, etc? | 23:25 |
| clarkb | corvus: hrm good point | 23:25 |
| clarkb | I was thinking of it from purely cloud quota perspective but we configure a quota too | 23:26 |
| corvus | clarkb: maybe we should chat with Shrews and tobiash about this next week -- find out how important it is to have the nodepool-specific estimation, versus just "cloud quota - cloud (including nodepool) used" | 23:27 |
| clarkb | sounds good | 23:28 |
| clarkb | the fix ended up being not too terrible, you'd either delete the node or update it like we did | 23:28 |
| *** rpioso is now known as rpioso|afk | 23:29 | |
| *** heyongli has quit IRC | 23:31 | |
| *** heyongli has joined #openstack-infra | 23:32 | |
| clarkb | alright ipv4 setting should be in place now | 23:33 |
| *** graphene has joined #openstack-infra | 23:35 | |
| clarkb | my one concern is that we may not reload that config when those files update (but I'm pretty sure we do, watching now to see if we boot nodes) | 23:36 |
| *** graphene has quit IRC | 23:37 | |
| *** graphene has joined #openstack-infra | 23:38 | |
| clarkb | the logs seem to imply we ar estill using the ipv6 addresses, this might need a launcher restart | 23:38 |
| *** apevec has quit IRC | 23:39 | |
| clarkb | I'm going to restart nodepool-launcher on nl03 | 23:39 |
| clarkb | specifically what I see in the logs is that the ip: field is the ipv6 addr and not the ipv4 addr | 23:40 |
| clarkb | and that is what nodepool uses | 23:40 |
| *** rlandy has quit IRC | 23:41 | |
| fungi | so needs a launcher restart? | 23:41 |
| clarkb | I think so | 23:41 |
| corvus | is that in clouds.yaml? | 23:41 |
| clarkb | because the cloud config settings may not result in a reconfiguration I guess | 23:41 |
| clarkb | corvus: yes | 23:41 |
| corvus | yeah, probably so. | 23:41 |
| clarkb | I've done that and we should know more soon | 23:41 |
| fungi | nodepool/nl03.openstack.org.yaml | 23:41 |
| *** heyongli has quit IRC | 23:42 | |
| clarkb | yup ip is now using ipv4 not ipv6 entry | 23:42 |
| *** heyongli has joined #openstack-infra | 23:42 | |
| clarkb | so we do need to restart to pick up clouds.yaml changes | 23:42 |
| fungi | oh, the v4 setting | 23:42 |
| clarkb | there are nodes in use in vexxhost again | 23:42 |
| clarkb | fungi: ya, becase ipv6 routes are unworking | 23:42 |
| fungi | right, i glanced at the wrong review | 23:43 |
| clarkb | firefox is using an entire cpu right now. No wonder my battery life is bad | 23:44 |
| pabelanger | we had the same issue with nl03 and moving an label to another pool. We had to revert then make sure we deleted all nodes online first, then delete image upload, then bring new pool online, reupload images and node booted | 23:45 |
| pabelanger | for some reason didn't catch that on the review before | 23:45 |
| clarkb | pabelanger: it would've been fine if the node wasn't being held I think | 23:45 |
| clarkb | pabelanger: because it would've been deleted then things would work again after | 23:45 |
| pabelanger | clarkb: maybe, I don't think nodepool in rdo deleted the ready node, nothing does today. YOu have to manually delete it I think | 23:46 |
| pabelanger | there was a discussion yesterday in #zuul about this | 23:46 |
| pabelanger | I plan on writing a unit test to test this next week | 23:46 |
| clarkb | pabelanger: you can do what I did too, and just change the pool value to the new pool | 23:47 |
| pabelanger | clarkb: in zk? | 23:48 |
| clarkb | pabelanger: yes | 23:48 |
| pabelanger | Yah, try to avoid touching the data manually :) But agree | 23:48 |
| fungi | the issue we ran into with the nl03 restart was when adding code to start setting the pool column | 23:49 |
| fungi | you can't have launchers from before and after that change actively sharing zk | 23:49 |
| clarkb | ya that was the big issue we had last, this was quite minor in comparison | 23:50 |
| fungi | so on upgrade there needed to be a coordinated launcher restart | 23:50 |
| *** jamesmcarthur has joined #openstack-infra | 23:50 | |
| fungi | and that's thankfully now mentioned in the nodepool release notes | 23:50 |
| pabelanger | good to know | 23:50 |
| *** heyongli has quit IRC | 23:52 | |
| *** heyongli has joined #openstack-infra | 23:52 | |
| *** SpamapS has joined #openstack-infra | 23:54 | |
| *** graphene has quit IRC | 23:56 | |
| *** graphene has joined #openstack-infra | 23:57 | |
| clarkb | hrm I think non working ipv6 may be causing pre playbooks to fail | 23:58 |
| clarkb | I guess the other places we force ipv4 we either have no ipv6 at all (ovh) or working ipv6(rax) | 23:58 |
| clarkb | but its the weekend and we'll retry those jobs so not sure if we need to worry too much | 23:58 |
| clarkb | I'm sure mnaser will get it sorted out :) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!