| tonyb | (and thanks pabelanger) | 00:00 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Add block-device defaults https://review.openstack.org/539375 | 00:01 |
| *** camunoz has quit IRC | 00:04 | |
| *** Krinkle has left #openstack-infra | 00:06 | |
| *** liujiong has joined #openstack-infra | 00:13 | |
| *** hongbin has quit IRC | 00:18 | |
| *** zhurong has joined #openstack-infra | 00:19 | |
| *** thorre has quit IRC | 00:19 | |
| *** jamesmcarthur has joined #openstack-infra | 00:21 | |
| *** salv-orlando has joined #openstack-infra | 00:22 | |
| openstackgerrit | Merged openstack-infra/system-config master: Set status_url for zuul https://review.openstack.org/539357 | 00:24 |
| *** bfournie has joined #openstack-infra | 00:24 | |
| openstackgerrit | Chris Hoge proposed openstack-infra/irc-meetings master: Updated meeting time to remove now defunct Interop Challenge https://review.openstack.org/539378 | 00:26 |
| *** salv-orlando has quit IRC | 00:27 | |
| *** kmalloc has quit IRC | 00:29 | |
| *** calbers has quit IRC | 00:31 | |
| *** ianychoi has joined #openstack-infra | 00:31 | |
| *** thorre has joined #openstack-infra | 00:32 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add mirror02.dfw.rax.o.o to cacti https://review.openstack.org/539328 | 00:34 |
| *** jklare has quit IRC | 00:36 | |
| *** zhurong has quit IRC | 00:37 | |
| *** jklare has joined #openstack-infra | 00:38 | |
| *** markvoelker has quit IRC | 00:43 | |
| *** liujiong has quit IRC | 00:44 | |
| *** markvoelker has joined #openstack-infra | 00:44 | |
| *** markvoelker has quit IRC | 00:48 | |
| *** liusheng has quit IRC | 00:49 | |
| *** zhurong has joined #openstack-infra | 00:50 | |
| mordred | corvus: next issue to deal with (tomorrow) ... | 00:51 |
| mordred | corvus: http://logs.openstack.org/74/20974/1a14ba59b18b1d48a4957e6eb099a211e6e9bb26/third-party-check/shade-ansible-devel-functional-devstack/b0f3899/ara/result/0c3dafcd-2487-4e9a-8e7e-2d466ca09e7f/ | 00:51 |
| mordred | corvus: also - the job failed in pre but reported success to the PR - and also the details link stayed with the status page - so there's definitely some work to do, but I think we're getting somewhere | 00:52 |
| mordred | corvus: oh - nevermind- it left the log link on the failure message it left which properly says "RETRY_LIMIT" - but the reported status is ... OH | 00:53 |
| mordred | corvus: its reporting success becaues the job is marked as non-voting | 00:53 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Remove legacy ec2-api jobs https://review.openstack.org/539109 | 00:53 |
| *** olaph1 has joined #openstack-infra | 01:00 | |
| *** olaph has quit IRC | 01:01 | |
| *** cuongnv has joined #openstack-infra | 01:01 | |
| *** SumitNaiksatam has quit IRC | 01:02 | |
| *** r-daneel has quit IRC | 01:04 | |
| *** owalsh has quit IRC | 01:05 | |
| *** owalsh_ has joined #openstack-infra | 01:05 | |
| *** owalsh_ is now known as owalsh | 01:06 | |
| *** puiterwijk has quit IRC | 01:06 | |
| *** rcernin has joined #openstack-infra | 01:09 | |
| *** dave-mccowan has joined #openstack-infra | 01:13 | |
| *** slaweq has joined #openstack-infra | 01:13 | |
| *** felipemonteiro__ has quit IRC | 01:14 | |
| *** jamesmcarthur has quit IRC | 01:17 | |
| *** slaweq has quit IRC | 01:18 | |
| *** gongysh has joined #openstack-infra | 01:19 | |
| *** puiterwijk has joined #openstack-infra | 01:23 | |
| *** jamesmcarthur has joined #openstack-infra | 01:26 | |
| *** HeOS has joined #openstack-infra | 01:29 | |
| *** markvoelker has joined #openstack-infra | 01:30 | |
| *** rlandy has quit IRC | 01:32 | |
| *** zhongjun has joined #openstack-infra | 01:34 | |
| *** mriedem_afk is now known as mriedem | 01:37 | |
| *** olaph1 has quit IRC | 01:40 | |
| *** olaph has joined #openstack-infra | 01:40 | |
| *** liujiong has joined #openstack-infra | 01:48 | |
| *** salv-orlando has joined #openstack-infra | 01:53 | |
| *** kiennt26 has joined #openstack-infra | 01:54 | |
| *** ramishra has joined #openstack-infra | 01:55 | |
| *** salv-orlando has quit IRC | 01:57 | |
| *** gongysh has quit IRC | 02:03 | |
| *** dougwig has quit IRC | 02:10 | |
| *** Swami has quit IRC | 02:11 | |
| *** harlowja has quit IRC | 02:15 | |
| *** mriedem has quit IRC | 02:17 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fail if two elements provide the same thing https://review.openstack.org/539401 | 02:24 |
| *** liujiong has quit IRC | 02:27 | |
| *** liujiong has joined #openstack-infra | 02:27 | |
| *** jamesmcarthur has quit IRC | 02:29 | |
| *** liusheng has joined #openstack-infra | 02:34 | |
| *** rcernin has quit IRC | 02:35 | |
| *** HeOS has quit IRC | 02:39 | |
| *** liujiong has quit IRC | 02:42 | |
| *** d0ugal has quit IRC | 02:50 | |
| *** cshastri has joined #openstack-infra | 02:51 | |
| ianw | hmmm | 02:51 |
| ianw | mirror.dfw.openstack.org | 02:51 |
| ianw | A | 02:51 |
| ianw | 23.253.237.8 | 02:51 |
| ianw | ohh, hang on, missing a .rax | 02:51 |
| ianw | that's confusing ... i'll get rid of them | 02:52 |
| ianw | #status log remove old A and AAAA records for mirror.dfw.openstack.org (note: not mirror.dfw.rax.openstack.org) | 02:53 |
| openstackstatus | ianw: finished logging | 02:53 |
| *** olaph1 has joined #openstack-infra | 02:53 | |
| *** olaph has quit IRC | 02:54 | |
| ianw | #status log mirror.dfw.rax.openstack.org updated to mirror02.dfw.rax.openstack.org (TTL turned down to 5 minutes as we test) | 02:55 |
| openstackstatus | ianw: finished logging | 02:55 |
| *** andreas_s has joined #openstack-infra | 02:59 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: GPT partitioning support https://review.openstack.org/533490 | 03:02 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fail if two elements provide the same thing https://review.openstack.org/539401 | 03:02 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Add block-device defaults https://review.openstack.org/539375 | 03:02 |
| *** andreas_s has quit IRC | 03:03 | |
| *** Pramod has quit IRC | 03:05 | |
| *** armax has joined #openstack-infra | 03:05 | |
| *** ykarel|away has joined #openstack-infra | 03:07 | |
| *** jamesmcarthur has joined #openstack-infra | 03:10 | |
| *** armax has quit IRC | 03:10 | |
| *** armax has joined #openstack-infra | 03:16 | |
| *** dhajare_ has joined #openstack-infra | 03:17 | |
| *** hongbin has joined #openstack-infra | 03:19 | |
| *** mgagne has quit IRC | 03:22 | |
| *** melwitt has quit IRC | 03:23 | |
| *** jamielennox has quit IRC | 03:23 | |
| *** kuromagi^ has joined #openstack-infra | 03:23 | |
| *** kuromagi has quit IRC | 03:23 | |
| *** ykarel|away has quit IRC | 03:23 | |
| *** mgagne has joined #openstack-infra | 03:24 | |
| *** ykarel|away has joined #openstack-infra | 03:24 | |
| *** mgagne is now known as Guest87240 | 03:24 | |
| *** melwitt has joined #openstack-infra | 03:25 | |
| vivsoni_ | Hi Team, | 03:27 |
| vivsoni_ | i wanted to check zuul status on patch.... trying to navigate link "http://zuulv3.openstack.org/" | 03:28 |
| *** jamielennox has joined #openstack-infra | 03:29 | |
| vivsoni_ | this link is not working... please let me know whether this is changed or its same ? | 03:29 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Add block-device defaults https://review.openstack.org/539375 | 03:29 |
| ianw | vivsoni_: it's just zuul.openstack.org ... the v3 was a temporary redirect for a while | 03:30 |
| vivsoni_ | ianw: Thanks !!! will this link be a permanent one ? | 03:32 |
| ianw | vivsoni_: yep, that's it for now | 03:33 |
| vivsoni_ | ianw: ok Thanks | 03:33 |
| *** ykarel|away has quit IRC | 03:34 | |
| *** felipemonteiro has joined #openstack-infra | 03:39 | |
| *** jamesmcarthur has quit IRC | 03:39 | |
| *** liusheng has quit IRC | 03:40 | |
| *** liusheng has joined #openstack-infra | 03:40 | |
| ianw | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64868&rra_id=all <- mirror02 eth0 stats ... coming online i guess | 03:41 |
| *** felipemonteiro_ has joined #openstack-infra | 03:42 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Add block-device defaults https://review.openstack.org/539375 | 03:43 |
| *** andreww has quit IRC | 03:43 | |
| *** felipemonteiro has quit IRC | 03:45 | |
| *** felipemonteiro_ has quit IRC | 03:45 | |
| *** felipemonteiro__ has joined #openstack-infra | 03:45 | |
| *** sree has joined #openstack-infra | 03:45 | |
| *** zhurong has quit IRC | 03:46 | |
| *** felipemonteiro_ has joined #openstack-infra | 03:46 | |
| *** pahuang has quit IRC | 03:48 | |
| *** felipemonteiro__ has quit IRC | 03:50 | |
| *** pahuang has joined #openstack-infra | 03:53 | |
| *** HenryG has quit IRC | 03:54 | |
| *** salv-orlando has joined #openstack-infra | 03:54 | |
| *** HenryG has joined #openstack-infra | 03:54 | |
| *** pahuang has quit IRC | 03:55 | |
| *** pahuang has joined #openstack-infra | 03:56 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a static driver for Nodepool https://review.openstack.org/535553 | 03:59 |
| *** salv-orlando has quit IRC | 04:00 | |
| *** coolsvap has joined #openstack-infra | 04:00 | |
| *** bhavik1 has joined #openstack-infra | 04:03 | |
| *** cshastri has quit IRC | 04:04 | |
| *** cshastri has joined #openstack-infra | 04:05 | |
| *** yamahata has joined #openstack-infra | 04:06 | |
| *** iyamahat has joined #openstack-infra | 04:08 | |
| ramishra | Seems jobs are queued for hours, Is everything ok..or just the load? | 04:09 |
| ramishra | ianw: hi! ^^ | 04:09 |
| *** jamesmcarthur has joined #openstack-infra | 04:10 | |
| *** felipemonteiro__ has joined #openstack-infra | 04:11 | |
| *** felipemonteiro_ has quit IRC | 04:11 | |
| ianw | ramishra: very heavy load around release time, and we've had a few issues that should be starting to resolve out now (longer timeouts to stop jobs rechecking, mirror in dfw etc) | 04:11 |
| ianw | won't be instant fix but hopefully things are trending downwards | 04:12 |
| ramishra | ianw: thanks for the info | 04:12 |
| *** psachin has joined #openstack-infra | 04:12 | |
| *** olaph has joined #openstack-infra | 04:14 | |
| *** bhavik1 has quit IRC | 04:14 | |
| *** olaph1 has quit IRC | 04:15 | |
| *** shu-mutou-AWAY has quit IRC | 04:15 | |
| *** jamesmcarthur has quit IRC | 04:16 | |
| *** SumitNaiksatam has joined #openstack-infra | 04:17 | |
| *** david-lyle has quit IRC | 04:26 | |
| *** felipemonteiro__ has quit IRC | 04:26 | |
| *** jamesmcarthur has joined #openstack-infra | 04:27 | |
| *** david-lyle has joined #openstack-infra | 04:27 | |
| *** ykarel|away has joined #openstack-infra | 04:31 | |
| *** dave-mccowan has quit IRC | 04:32 | |
| *** jamesmcarthur has quit IRC | 04:32 | |
| *** armax has quit IRC | 04:37 | |
| *** rosmaita has quit IRC | 04:39 | |
| *** jamesmcarthur has joined #openstack-infra | 04:43 | |
| *** chkumar|off is now known as chandankumar | 04:44 | |
| *** slaweq has joined #openstack-infra | 04:45 | |
| *** jamesmcarthur has quit IRC | 04:49 | |
| *** slaweq has quit IRC | 04:50 | |
| *** rcernin has joined #openstack-infra | 04:50 | |
| *** harlowja has joined #openstack-infra | 04:51 | |
| *** rcernin has quit IRC | 04:51 | |
| *** rcernin has joined #openstack-infra | 04:52 | |
| *** jamesmcarthur has joined #openstack-infra | 04:57 | |
| *** iyamahat has quit IRC | 05:00 | |
| *** SumitNaiksatam has quit IRC | 05:02 | |
| *** jamesmcarthur has quit IRC | 05:02 | |
| *** jamesmcarthur has joined #openstack-infra | 05:05 | |
| *** markvoelker has quit IRC | 05:06 | |
| *** links has joined #openstack-infra | 05:07 | |
| *** jamesmcarthur has quit IRC | 05:09 | |
| *** olaph1 has joined #openstack-infra | 05:12 | |
| *** zerick_ has quit IRC | 05:13 | |
| *** olaph has quit IRC | 05:13 | |
| *** jamesmcarthur has joined #openstack-infra | 05:14 | |
| *** zerick has joined #openstack-infra | 05:15 | |
| *** jamesmcarthur has quit IRC | 05:18 | |
| *** slaweq has joined #openstack-infra | 05:19 | |
| *** salv-orlando has joined #openstack-infra | 05:20 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fail if two elements provide the same thing https://review.openstack.org/539401 | 05:20 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Add block-device defaults https://review.openstack.org/539375 | 05:20 |
| *** slaweq has quit IRC | 05:24 | |
| *** mylu has quit IRC | 05:24 | |
| *** jamesmcarthur has joined #openstack-infra | 05:25 | |
| *** hongbin has quit IRC | 05:26 | |
| *** olaph has joined #openstack-infra | 05:27 | |
| *** olaph1 has quit IRC | 05:28 | |
| *** jamesmcarthur has quit IRC | 05:31 | |
| *** pgadiya has joined #openstack-infra | 05:37 | |
| *** jamesmcarthur has joined #openstack-infra | 05:39 | |
| *** jamesmcarthur has quit IRC | 05:44 | |
| *** zhurong has joined #openstack-infra | 05:47 | |
| *** threestrands has quit IRC | 05:51 | |
| *** armax has joined #openstack-infra | 05:52 | |
| *** xinliang has quit IRC | 05:57 | |
| *** salv-orlando has quit IRC | 05:58 | |
| *** HeOS has joined #openstack-infra | 05:58 | |
| *** jamesmcarthur has joined #openstack-infra | 05:59 | |
| *** salv-orlando has joined #openstack-infra | 05:59 | |
| *** threestrands has joined #openstack-infra | 06:01 | |
| *** jamesmcarthur has quit IRC | 06:03 | |
| *** jamesmcarthur has joined #openstack-infra | 06:05 | |
| *** daidv has quit IRC | 06:08 | |
| *** xinliang has joined #openstack-infra | 06:09 | |
| *** xinliang has quit IRC | 06:09 | |
| *** xinliang has joined #openstack-infra | 06:09 | |
| *** jamesmcarthur has quit IRC | 06:09 | |
| *** benj_ has quit IRC | 06:14 | |
| *** jamesmcarthur has joined #openstack-infra | 06:16 | |
| *** pcichy has joined #openstack-infra | 06:19 | |
| *** xinliang has quit IRC | 06:21 | |
| *** xinliang has joined #openstack-infra | 06:21 | |
| *** jamesmcarthur has quit IRC | 06:22 | |
| *** jamesmcarthur has joined #openstack-infra | 06:29 | |
| *** dbecker has quit IRC | 06:30 | |
| *** jamesmcarthur has quit IRC | 06:34 | |
| *** armax has quit IRC | 06:40 | |
| *** jamesmcarthur has joined #openstack-infra | 06:40 | |
| *** iyamahat has joined #openstack-infra | 06:43 | |
| *** sree_ has joined #openstack-infra | 06:43 | |
| *** sree_ is now known as Guest36585 | 06:44 | |
| *** jamesmcarthur has quit IRC | 06:45 | |
| *** dbecker has joined #openstack-infra | 06:45 | |
| *** sree has quit IRC | 06:45 | |
| *** jamesmcarthur has joined #openstack-infra | 06:47 | |
| *** dsariel has joined #openstack-infra | 06:47 | |
| *** SumitNaiksatam has joined #openstack-infra | 06:49 | |
| *** zhurong has quit IRC | 06:49 | |
| *** jamesmcarthur has quit IRC | 06:51 | |
| *** slaweq has joined #openstack-infra | 06:52 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add memory awareness to system load governor https://review.openstack.org/539426 | 06:54 |
| *** jamesmcarthur has joined #openstack-infra | 06:55 | |
| *** slaweq has quit IRC | 06:56 | |
| *** jamesmcarthur has quit IRC | 06:59 | |
| *** harlowja has quit IRC | 07:01 | |
| *** zhenguo has joined #openstack-infra | 07:05 | |
| *** jamesmcarthur has joined #openstack-infra | 07:06 | |
| *** rcernin has quit IRC | 07:08 | |
| *** jamesmcarthur has quit IRC | 07:11 | |
| *** markvoelker has joined #openstack-infra | 07:11 | |
| *** salv-orlando has quit IRC | 07:12 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Finalize rack requirement https://review.openstack.org/539431 | 07:17 |
| *** jamesmcarthur has joined #openstack-infra | 07:17 | |
| *** iyamahat_ has joined #openstack-infra | 07:17 | |
| *** iyamahat has quit IRC | 07:21 | |
| openstackgerrit | Masahito Muroi proposed openstack-infra/project-config master: Add required-projects to publish job of blazar-nova https://review.openstack.org/539433 | 07:21 |
| AJaeger | config-core, could you review https://review.openstack.org/538632 https://review.openstack.org/538631 and https://review.openstack.org/538306 , please? Those remove some jobs from project-config. | 07:21 |
| *** jamesmcarthur has quit IRC | 07:23 | |
| *** pcichy has quit IRC | 07:23 | |
| *** jaosorior has quit IRC | 07:26 | |
| *** gongysh has joined #openstack-infra | 07:29 | |
| *** jamesmcarthur has joined #openstack-infra | 07:29 | |
| *** andreas_s has joined #openstack-infra | 07:30 | |
| *** e0ne has joined #openstack-infra | 07:34 | |
| *** jamesmcarthur has quit IRC | 07:34 | |
| *** gongysh has quit IRC | 07:35 | |
| *** jamesmcarthur has joined #openstack-infra | 07:36 | |
| *** zhurong has joined #openstack-infra | 07:38 | |
| *** jamesmcarthur has quit IRC | 07:40 | |
| *** markvoelker has quit IRC | 07:42 | |
| *** jaosorior has joined #openstack-infra | 07:43 | |
| *** slaweq has joined #openstack-infra | 07:43 | |
| *** ykarel|away is now known as ykarel|lunch | 07:44 | |
| *** salv-orlando has joined #openstack-infra | 07:44 | |
| *** chenying_ has joined #openstack-infra | 07:45 | |
| *** daidv has joined #openstack-infra | 07:46 | |
| *** slaweq has quit IRC | 07:48 | |
| *** jamesmcarthur has joined #openstack-infra | 07:48 | |
| *** pcaruana has joined #openstack-infra | 07:51 | |
| *** jamesmcarthur has quit IRC | 07:53 | |
| *** jamesmcarthur has joined #openstack-infra | 07:57 | |
| *** salv-orl_ has joined #openstack-infra | 07:59 | |
| *** tmorin has joined #openstack-infra | 08:01 | |
| *** salv-orlando has quit IRC | 08:01 | |
| *** amoralej|off is now known as amoralej | 08:02 | |
| *** jamesmcarthur has quit IRC | 08:02 | |
| *** benj_ has joined #openstack-infra | 08:03 | |
| *** alexchadin has joined #openstack-infra | 08:05 | |
| *** jtomasek has joined #openstack-infra | 08:05 | |
| *** jamesmcarthur has joined #openstack-infra | 08:07 | |
| *** sree has joined #openstack-infra | 08:08 | |
| *** slaweq has joined #openstack-infra | 08:08 | |
| *** b_bezak has joined #openstack-infra | 08:11 | |
| *** Guest36585 has quit IRC | 08:11 | |
| *** jamesmcarthur has quit IRC | 08:11 | |
| *** slaweq has quit IRC | 08:13 | |
| *** vivsoni__ has joined #openstack-infra | 08:14 | |
| *** vivsoni_ has quit IRC | 08:14 | |
| *** rcernin has joined #openstack-infra | 08:16 | |
| *** ralonsoh has joined #openstack-infra | 08:16 | |
| openstackgerrit | Masahito Muroi proposed openstack-infra/project-config master: Add required-projects to publish job of blazar-nova https://review.openstack.org/539433 | 08:18 |
| *** slaweq has joined #openstack-infra | 08:18 | |
| *** ijw has joined #openstack-infra | 08:19 | |
| *** slaweq_ has joined #openstack-infra | 08:19 | |
| *** ijw has quit IRC | 08:20 | |
| *** tesseract has joined #openstack-infra | 08:20 | |
| *** ijw has joined #openstack-infra | 08:20 | |
| *** jamesmcarthur has joined #openstack-infra | 08:22 | |
| *** slaweq has quit IRC | 08:22 | |
| *** ralonsoh has quit IRC | 08:23 | |
| *** salv-orl_ has quit IRC | 08:24 | |
| *** e0ne has quit IRC | 08:25 | |
| *** ralonsoh has joined #openstack-infra | 08:26 | |
| *** rcernin has quit IRC | 08:26 | |
| *** ijw has quit IRC | 08:27 | |
| *** jamesmcarthur has quit IRC | 08:27 | |
| *** HeOS has quit IRC | 08:28 | |
| *** ijw has joined #openstack-infra | 08:29 | |
| *** jamesmcarthur has joined #openstack-infra | 08:30 | |
| *** e0ne has joined #openstack-infra | 08:30 | |
| *** janki has joined #openstack-infra | 08:31 | |
| *** jamesmcarthur has quit IRC | 08:35 | |
| *** dingyichen has quit IRC | 08:36 | |
| *** jamesmcarthur has joined #openstack-infra | 08:37 | |
| *** markvoelker has joined #openstack-infra | 08:39 | |
| *** armaan has joined #openstack-infra | 08:40 | |
| *** jamesmcarthur has quit IRC | 08:41 | |
| *** jpena|off is now known as jpena | 08:43 | |
| *** ykarel|lunch is now known as ykarel | 08:44 | |
| *** salv-orlando has joined #openstack-infra | 08:45 | |
| *** zhurong_ has joined #openstack-infra | 08:45 | |
| *** jamesmcarthur has joined #openstack-infra | 08:46 | |
| *** dsariel has quit IRC | 08:47 | |
| *** jamesmcarthur has quit IRC | 08:50 | |
| *** edmondsw has joined #openstack-infra | 08:50 | |
| *** olaph1 has joined #openstack-infra | 08:53 | |
| *** namnh has joined #openstack-infra | 08:54 | |
| *** olaph has quit IRC | 08:54 | |
| *** edmondsw has quit IRC | 08:54 | |
| *** janki is now known as janki|afk | 08:55 | |
| *** jamesmcarthur has joined #openstack-infra | 09:01 | |
| *** oidgar has joined #openstack-infra | 09:01 | |
| *** iyamahat_ has quit IRC | 09:02 | |
| *** d0ugal has joined #openstack-infra | 09:04 | |
| *** rcernin has joined #openstack-infra | 09:05 | |
| *** jamesmcarthur has quit IRC | 09:05 | |
| *** ijw has quit IRC | 09:06 | |
| *** yamahata has quit IRC | 09:06 | |
| *** jamesmcarthur has joined #openstack-infra | 09:10 | |
| *** markvoelker has quit IRC | 09:12 | |
| *** e0ne has quit IRC | 09:12 | |
| *** shardy has joined #openstack-infra | 09:13 | |
| *** jamesmcarthur has quit IRC | 09:14 | |
| *** e0ne has joined #openstack-infra | 09:21 | |
| *** stakeda has quit IRC | 09:21 | |
| *** janki|afk is now known as janki | 09:21 | |
| *** jamesmcarthur has joined #openstack-infra | 09:22 | |
| *** jamesmcarthur has quit IRC | 09:28 | |
| *** jamesmcarthur has joined #openstack-infra | 09:33 | |
| *** jamesmcarthur has quit IRC | 09:38 | |
| *** jamesmcarthur has joined #openstack-infra | 09:39 | |
| *** derekh has joined #openstack-infra | 09:41 | |
| *** e0ne has quit IRC | 09:42 | |
| *** jamesmcarthur has quit IRC | 09:43 | |
| *** ganso has joined #openstack-infra | 09:48 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Updated meeting time to remove now defunct Interop Challenge https://review.openstack.org/539378 | 09:50 |
| openstackgerrit | lei zhang proposed openstack/diskimage-builder master: Remove some redundant indents https://review.openstack.org/539479 | 09:51 |
| *** armaan has quit IRC | 09:53 | |
| *** zhenguo has quit IRC | 09:53 | |
| *** armaan has joined #openstack-infra | 09:54 | |
| *** jamesmcarthur has joined #openstack-infra | 09:54 | |
| *** liujiong has joined #openstack-infra | 09:55 | |
| *** jamesmcarthur has quit IRC | 09:59 | |
| *** jamesmcarthur has joined #openstack-infra | 10:01 | |
| *** jamesmcarthur has quit IRC | 10:05 | |
| *** e0ne has joined #openstack-infra | 10:05 | |
| *** ykarel is now known as ykarel|away | 10:06 | |
| *** markvoelker has joined #openstack-infra | 10:09 | |
| *** ykarel|away has quit IRC | 10:11 | |
| *** dsariel has joined #openstack-infra | 10:11 | |
| *** annp has quit IRC | 10:14 | |
| *** namnh has quit IRC | 10:15 | |
| *** jamesmcarthur has joined #openstack-infra | 10:16 | |
| *** hjensas has quit IRC | 10:17 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix statsd documentation about events https://review.openstack.org/539482 | 10:18 |
| *** priteau has joined #openstack-infra | 10:19 | |
| *** zhurong has quit IRC | 10:19 | |
| *** jamesmcarthur has quit IRC | 10:21 | |
| *** ijw has joined #openstack-infra | 10:23 | |
| *** cuongnv has quit IRC | 10:24 | |
| *** liujiong has quit IRC | 10:24 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/devstack-gate master: Update documentation for ironic-bm-logs https://review.openstack.org/411938 | 10:25 |
| *** sambetts|afk is now known as sambetts | 10:27 | |
| *** jamesmcarthur has joined #openstack-infra | 10:30 | |
| *** kjackal has quit IRC | 10:30 | |
| *** ijw has quit IRC | 10:30 | |
| *** ldnunes has joined #openstack-infra | 10:30 | |
| *** kjackal has joined #openstack-infra | 10:31 | |
| *** jamesmcarthur has quit IRC | 10:34 | |
| *** dtantsur|afk is now known as dtantsur | 10:36 | |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support https://review.openstack.org/529376 | 10:37 |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support for RHEL/CentOS https://review.openstack.org/528739 | 10:37 |
| *** jamesmcarthur has joined #openstack-infra | 10:38 | |
| *** markvoelker has quit IRC | 10:42 | |
| *** jamesmcarthur has quit IRC | 10:43 | |
| *** pbourke has quit IRC | 10:46 | |
| *** zhurong_ has quit IRC | 10:48 | |
| *** pbourke has joined #openstack-infra | 10:48 | |
| *** lucas-afk is now known as lucasagomes | 10:48 | |
| *** jamesmcarthur has joined #openstack-infra | 10:49 | |
| *** salv-orlando has quit IRC | 10:52 | |
| *** andreas_s has quit IRC | 10:52 | |
| *** salv-orlando has joined #openstack-infra | 10:52 | |
| *** jamesmcarthur has quit IRC | 10:53 | |
| *** sshnaidm|afk is now known as sshnaidm | 10:56 | |
| *** jamesmcarthur has joined #openstack-infra | 10:56 | |
| *** andreas_s has joined #openstack-infra | 10:56 | |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack-infra/irc-meetings master: Change time for the Containers team meeting https://review.openstack.org/539491 | 10:56 |
| *** bingoarun has joined #openstack-infra | 10:57 | |
| *** salv-orlando has quit IRC | 10:57 | |
| *** tosky has joined #openstack-infra | 10:59 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: [WIP] Allow operator to set held node's TTL https://review.openstack.org/539493 | 11:00 |
| *** jamesmcarthur has quit IRC | 11:01 | |
| *** ijw has joined #openstack-infra | 11:02 | |
| *** andreas_s has quit IRC | 11:06 | |
| *** calbers has joined #openstack-infra | 11:07 | |
| *** andreas_s has joined #openstack-infra | 11:07 | |
| *** alexchadin has quit IRC | 11:09 | |
| *** andreas_s has quit IRC | 11:11 | |
| *** jamesmcarthur has joined #openstack-infra | 11:12 | |
| *** jamesmcarthur has quit IRC | 11:16 | |
| *** andreas_s has joined #openstack-infra | 11:17 | |
| *** ganso has left #openstack-infra | 11:17 | |
| *** hjensas has joined #openstack-infra | 11:18 | |
| *** rfolco|off is now known as rfolco | 11:20 | |
| *** mdbooth has quit IRC | 11:21 | |
| *** ari[m] has quit IRC | 11:21 | |
| *** andreas_s has quit IRC | 11:22 | |
| *** ari[m] has joined #openstack-infra | 11:24 | |
| *** mdbooth has joined #openstack-infra | 11:24 | |
| *** jamesmcarthur has joined #openstack-infra | 11:25 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/openstack-zuul-jobs master: Replace legacy tempest-dsvm-py35 https://review.openstack.org/539497 | 11:25 |
| andreaf | AJaeger: can you look at this one once you have a moment? ^^^ | 11:26 |
| *** kopecmartin has joined #openstack-infra | 11:26 | |
| andreaf | AJaeger: btw the grenade team is fine with hosting ansible roles in the grenade repo so I will work on that (eventually :) | 11:26 |
| *** andreas_s has joined #openstack-infra | 11:27 | |
| *** ijw has quit IRC | 11:29 | |
| *** alexchadin has joined #openstack-infra | 11:29 | |
| *** jamesmcarthur has quit IRC | 11:29 | |
| *** alexchadin has quit IRC | 11:32 | |
| *** rpittau has quit IRC | 11:35 | |
| *** andreas_s has quit IRC | 11:36 | |
| *** jamesmcarthur has joined #openstack-infra | 11:36 | |
| *** vivsoni__ has quit IRC | 11:37 | |
| *** vivsoni__ has joined #openstack-infra | 11:37 | |
| *** markvoelker has joined #openstack-infra | 11:39 | |
| *** alexchadin has joined #openstack-infra | 11:39 | |
| *** tpsilva has joined #openstack-infra | 11:41 | |
| *** jamesmcarthur has quit IRC | 11:41 | |
| *** andreas_s has joined #openstack-infra | 11:42 | |
| *** alexchadin has quit IRC | 11:44 | |
| *** pcichy has joined #openstack-infra | 11:44 | |
| *** katkapilatova has joined #openstack-infra | 11:44 | |
| *** katkapilatova has left #openstack-infra | 11:44 | |
| *** jamesmcarthur has joined #openstack-infra | 11:45 | |
| *** yamamoto_ has quit IRC | 11:48 | |
| *** alexchadin has joined #openstack-infra | 11:49 | |
| *** jamesmcarthur has quit IRC | 11:49 | |
| *** alexchadin has quit IRC | 11:52 | |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support https://review.openstack.org/529376 | 11:52 |
| *** panda|off is now known as panda | 11:53 | |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support for RHEL/CentOS https://review.openstack.org/528739 | 11:53 |
| *** salv-orlando has joined #openstack-infra | 11:53 | |
| *** alexchadin has joined #openstack-infra | 11:53 | |
| *** ykarel has joined #openstack-infra | 11:55 | |
| *** jamesmcarthur has joined #openstack-infra | 11:56 | |
| *** alexchadin has quit IRC | 11:56 | |
| *** watanabe_isao has joined #openstack-infra | 11:57 | |
| *** salv-orlando has quit IRC | 11:57 | |
| *** alexchadin has joined #openstack-infra | 11:58 | |
| *** watanabe_isao has quit IRC | 11:58 | |
| *** jamesmcarthur has quit IRC | 12:01 | |
| *** owalsh is now known as owalsh_afk | 12:01 | |
| *** jamesmcarthur has joined #openstack-infra | 12:02 | |
| *** alexchadin has quit IRC | 12:05 | |
| *** alexchadin has joined #openstack-infra | 12:06 | |
| *** jamesmcarthur has quit IRC | 12:07 | |
| *** dsariel has quit IRC | 12:07 | |
| *** threestrands has quit IRC | 12:08 | |
| *** dave-mccowan has joined #openstack-infra | 12:08 | |
| *** alexchadin has quit IRC | 12:09 | |
| *** janki has quit IRC | 12:10 | |
| *** alexchadin has joined #openstack-infra | 12:10 | |
| *** hemna_ has joined #openstack-infra | 12:12 | |
| *** markvoelker has quit IRC | 12:12 | |
| hrw | what is wrong with zuul hosts and their ssh keys... | 12:13 |
| hrw | http://logs.openstack.org/46/539246/1/gate/kolla-build-centos-source/a287ab7/job-output.txt.gz#_2018-01-31_10_30_22_666891 | 12:13 |
| hrw | 2 hours of waiting for tests to finish went to trash | 12:15 |
| *** watanabe_isao has joined #openstack-infra | 12:15 | |
| * hrw off | 12:16 | |
| *** jamesmcarthur has joined #openstack-infra | 12:18 | |
| *** watanabe_isao has quit IRC | 12:18 | |
| AJaeger | andreaf: cool! | 12:18 |
| AJaeger | andreaf: This is the only user of tempest-dsvm-py35, so remove the job completely, please | 12:21 |
| *** dsariel has joined #openstack-infra | 12:21 | |
| andreaf | AJaeger: ok | 12:21 |
| *** yamamoto has joined #openstack-infra | 12:22 | |
| *** yamamoto has quit IRC | 12:22 | |
| *** yamamoto has joined #openstack-infra | 12:23 | |
| *** jamesmcarthur has quit IRC | 12:24 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/openstack-zuul-jobs master: Replace legacy tempest-dsvm-py35 https://review.openstack.org/539497 | 12:25 |
| andreaf | AJaeger: ^^^^ | 12:25 |
| *** bingoarun has quit IRC | 12:25 | |
| *** jamesmcarthur has joined #openstack-infra | 12:27 | |
| *** bfournie has quit IRC | 12:28 | |
| *** bfournie has joined #openstack-infra | 12:28 | |
| *** bfournie has quit IRC | 12:32 | |
| *** jamesmcarthur has quit IRC | 12:32 | |
| *** jamesmcarthur has joined #openstack-infra | 12:37 | |
| *** alexchadin has quit IRC | 12:41 | |
| *** alexchadin has joined #openstack-infra | 12:42 | |
| *** jamesmcarthur has quit IRC | 12:42 | |
| *** snapiri is now known as snapiri|AFK | 12:43 | |
| *** jcoufal has joined #openstack-infra | 12:44 | |
| *** olaph1 is now known as olaph | 12:47 | |
| *** jamesmcarthur has joined #openstack-infra | 12:49 | |
| *** jpena is now known as jpena|lunch | 12:49 | |
| *** rosmaita has joined #openstack-infra | 12:51 | |
| *** watanabe_isao_ has joined #openstack-infra | 12:53 | |
| *** watanabe_isao_ has quit IRC | 12:53 | |
| *** jamesmcarthur has quit IRC | 12:53 | |
| *** ijw has joined #openstack-infra | 12:54 | |
| *** jamesmcarthur has joined #openstack-infra | 12:56 | |
| *** _ari_|brno is now known as _ari_ | 12:56 | |
| AJaeger | andreaf: thanks, two comments | 12:56 |
| *** rosmaita has quit IRC | 12:57 | |
| *** rcernin has quit IRC | 12:58 | |
| andreaf | AJaeger: do I need to maintain the driverfixes negative lookahead? what's that for? | 12:59 |
| *** ijw has quit IRC | 12:59 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Change time for the Containers team meeting https://review.openstack.org/539491 | 13:00 |
| *** jamesmcarthur has quit IRC | 13:00 | |
| AJaeger | andreaf: cinder and manila have driverfixes branches. IF you make it *positive* like I propose, you can ignore driverfixes ;) | 13:01 |
| AJaeger | If you have a negative regex, you need to exclude it. | 13:01 |
| *** agopi has quit IRC | 13:01 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/openstack-zuul-jobs master: Replace legacy tempest-dsvm-py35 https://review.openstack.org/539497 | 13:03 |
| *** bfournie has joined #openstack-infra | 13:03 | |
| andreaf | AJaeger ^^^ | 13:04 |
| AJaeger | andreaf: suggestion inline - have a look whether you agree. But what you have should work | 13:04 |
| andreaf | I didn't know it could be a list | 13:05 |
| andreaf | sure that works for me | 13:05 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/openstack-zuul-jobs master: Replace legacy tempest-dsvm-py35 https://review.openstack.org/539497 | 13:07 |
| andreaf | AJaeger ^^^ | 13:07 |
| *** rosmaita has joined #openstack-infra | 13:08 | |
| AJaeger | andreaf: lists are a new Zuul v3 feature | 13:08 |
| AJaeger | andreaf: we use it already in several places and I consider them much easier for these cases | 13:08 |
| AJaeger | andreaf: +2 | 13:08 |
| AJaeger | thanks! | 13:09 |
| *** jamesmcarthur has joined #openstack-infra | 13:09 | |
| *** markvoelker has joined #openstack-infra | 13:09 | |
| *** olaph1 has joined #openstack-infra | 13:10 | |
| andreaf | yeah sure it's more readable | 13:11 |
| andreaf | thank you | 13:11 |
| *** olaph has quit IRC | 13:11 | |
| *** hemna_ has quit IRC | 13:13 | |
| *** jamesmcarthur has quit IRC | 13:13 | |
| *** jamesmcarthur has joined #openstack-infra | 13:16 | |
| Shrews | AJaeger: i'm in a position to watch nodepool now if you want to +A https://review.openstack.org/539248 | 13:19 |
| *** hemna_ has joined #openstack-infra | 13:20 | |
| Shrews | we have only a few nodes that would be affected right now, fwiw: http://paste.openstack.org/show/658142/ | 13:20 |
| *** rlandy has joined #openstack-infra | 13:20 | |
| *** trown|outtypewww is now known as trown | 13:21 | |
| *** dprince has joined #openstack-infra | 13:22 | |
| *** edmondsw has joined #openstack-infra | 13:22 | |
| *** jamesmcarthur has quit IRC | 13:26 | |
| *** olaph has joined #openstack-infra | 13:26 | |
| *** olaph1 has quit IRC | 13:27 | |
| *** janki has joined #openstack-infra | 13:28 | |
| *** coolsvap has quit IRC | 13:29 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fix on Summit Speaker Attendance Update https://review.openstack.org/539529 | 13:30 |
| *** kgiusti has joined #openstack-infra | 13:31 | |
| *** ralonsoh_ has joined #openstack-infra | 13:31 | |
| AJaeger | Shrews: self-approve, please | 13:33 |
| AJaeger | Shrews: looking at that list: Some look really odd, we should have used them already. We have constant high demand for xenial. But fedora-27 and opensuse-423 is the corner case we should monitor. | 13:34 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fix on Summit Speaker Attendance Update https://review.openstack.org/539529 | 13:34 |
| *** ralonsoh has quit IRC | 13:35 | |
| Shrews | AJaeger: those xenial nodes were only a few minutes old and would not be affected. in fact, they've already been used since i posted that :) | 13:35 |
| AJaeger | Shrews: ah! | 13:35 |
| Shrews | but there are a few that are days old | 13:36 |
| Shrews | fedora-27, debian-jessie | 13:36 |
| *** markvoelker has quit IRC | 13:37 | |
| *** alex_xu has quit IRC | 13:37 | |
| *** markvoelker has joined #openstack-infra | 13:37 | |
| *** alex_xu has joined #openstack-infra | 13:39 | |
| *** pgadiya has quit IRC | 13:39 | |
| *** tobberydberg has quit IRC | 13:41 | |
| *** alexchadin has quit IRC | 13:41 | |
| *** salv-orlando has joined #openstack-infra | 13:42 | |
| *** watanabe_isao_ has joined #openstack-infra | 13:45 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use max-ready-age on nodepool nodes https://review.openstack.org/539248 | 13:46 |
| *** watanabe_isao_ has quit IRC | 13:47 | |
| smcginnis | We got an rsync error on a publish job: http://logs.openstack.org/5f/5f00e51c87d6f8cd935858991fb3495b2651d99b/release-post/publish-static/da600c0/job-output.txt.gz#_2018-01-31_13_30_00_750630 | 13:47 |
| *** watanabe_isao has joined #openstack-infra | 13:47 | |
| *** watanabe_isao has quit IRC | 13:47 | |
| smcginnis | More issues with log dirs? | 13:47 |
| *** jlabarre has joined #openstack-infra | 13:48 | |
| *** jamesmcarthur has joined #openstack-infra | 13:49 | |
| *** gongysh has joined #openstack-infra | 13:49 | |
| *** kiennt26_ has joined #openstack-infra | 13:50 | |
| *** jpena|lunch is now known as jpena | 13:51 | |
| *** cshastri has quit IRC | 13:52 | |
| *** dhajare_ has quit IRC | 13:56 | |
| *** ralonsoh__ has joined #openstack-infra | 13:57 | |
| *** yolanda has quit IRC | 14:01 | |
| *** ralonsoh_ has quit IRC | 14:01 | |
| *** camunoz has joined #openstack-infra | 14:03 | |
| *** yolanda has joined #openstack-infra | 14:03 | |
| *** yamamoto has quit IRC | 14:04 | |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support https://review.openstack.org/529376 | 14:05 |
| openstackgerrit | Javier Peña proposed openstack-infra/puppet-openstackci master: Add AFS mirror support for RHEL/CentOS https://review.openstack.org/528739 | 14:05 |
| *** pcaruana has quit IRC | 14:05 | |
| *** yamamoto has joined #openstack-infra | 14:06 | |
| *** mriedem has joined #openstack-infra | 14:06 | |
| *** andreas_s has quit IRC | 14:07 | |
| Shrews | AJaeger: looks like we are good. all of the rarely used node labels seem to be rebuilt and available: http://paste.openstack.org/show/658150/ | 14:08 |
| Shrews | will keep watching it through the day though | 14:09 |
| *** yamamoto has quit IRC | 14:11 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Allow operator to set held node's TTL https://review.openstack.org/539493 | 14:11 |
| *** Goneri has joined #openstack-infra | 14:11 | |
| *** chenying_ has quit IRC | 14:15 | |
| *** chenying has joined #openstack-infra | 14:16 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix lazy initialization of GithubUser with apps https://review.openstack.org/539545 | 14:17 |
| *** amoralej is now known as amoralej|lunch | 14:18 | |
| *** pcaruana has joined #openstack-infra | 14:21 | |
| *** jtomasek_ has joined #openstack-infra | 14:22 | |
| *** peterlisak has quit IRC | 14:23 | |
| *** jtomasek has quit IRC | 14:24 | |
| *** peterlisak has joined #openstack-infra | 14:25 | |
| *** links has quit IRC | 14:27 | |
| *** sshnaidm is now known as sshnaidm|mtg | 14:27 | |
| *** dhill_ has joined #openstack-infra | 14:28 | |
| *** lucasagomes is now known as lucas-hungry | 14:30 | |
| *** esberglu has joined #openstack-infra | 14:30 | |
| *** andreas_s has joined #openstack-infra | 14:31 | |
| *** andreas_s has quit IRC | 14:31 | |
| *** dbecker has quit IRC | 14:31 | |
| *** dbecker has joined #openstack-infra | 14:31 | |
| *** andreas_s has joined #openstack-infra | 14:32 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Allow operator to set held node's TTL https://review.openstack.org/539493 | 14:34 |
| *** rosmaita has quit IRC | 14:34 | |
| *** rosmaita has joined #openstack-infra | 14:35 | |
| *** rosmaita has quit IRC | 14:36 | |
| *** dhajare_ has joined #openstack-infra | 14:37 | |
| AJaeger | great, Shrews ! | 14:37 |
| *** yamamoto has joined #openstack-infra | 14:38 | |
| *** wolverineav has joined #openstack-infra | 14:41 | |
| *** eharney has joined #openstack-infra | 14:42 | |
| *** jamesmcarthur has quit IRC | 14:42 | |
| *** rosmaita has joined #openstack-infra | 14:43 | |
| pabelanger | morning | 14:45 |
| pabelanger | just catching up on backscroll | 14:45 |
| *** yamahata has joined #openstack-infra | 14:45 | |
| *** andreas_s has quit IRC | 14:46 | |
| *** yamamoto has quit IRC | 14:47 | |
| *** yamamoto has joined #openstack-infra | 14:48 | |
| *** yamamoto has quit IRC | 14:48 | |
| *** pcaruana has quit IRC | 14:48 | |
| *** iyamahat has joined #openstack-infra | 14:48 | |
| *** yamamoto has joined #openstack-infra | 14:49 | |
| *** wolverineav has quit IRC | 14:52 | |
| *** wolverineav has joined #openstack-infra | 14:53 | |
| *** wolverineav has quit IRC | 14:57 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint get speaker summit assistance CSV https://review.openstack.org/539562 | 14:57 |
| *** armax has joined #openstack-infra | 14:59 | |
| pabelanger | infra-root: https://review.openstack.org/537611/ removes tripleo pipelines from zuul, all existing jobs have been removed (OVB) and now configured as 3rd party Ci in RDO. | 15:01 |
| *** amoralej|lunch is now known as amoralej | 15:01 | |
| pabelanger | stack of patches then starts removal from nodepool | 15:02 |
| *** agopi has joined #openstack-infra | 15:03 | |
| fungi | looks like mirror02.dfw.rax started out able to burst above 100mbps but i'm concerned that since that last spike at 12:00z it's no longer able to go above 100mbps again | 15:04 |
| mriedem | pabelanger: deps are all merged for this now https://review.openstack.org/#/c/533608 | 15:06 |
| Shrews | pabelanger: are we ready to approve those 1st 2 then? | 15:06 |
| pabelanger | mriedem: great, +3 | 15:06 |
| pabelanger | Shrews: yah, lets get all clear from clarkb, then we can start | 15:07 |
| Shrews | pabelanger: sure | 15:07 |
| pabelanger | Shrews: oh, 2nd are conflict, let me fix real quick | 15:07 |
| mordred | pabelanger: woot! | 15:07 |
| mordred | pabelanger, Shrews, fungi: also - https://review.openstack.org/#/c/539563/ should fix the ansible/shade tests when triggered from ansible | 15:08 |
| fungi | infra-root: fanatical support says mirror01.dfw.rax should be fixed as of 00:52... "I have been able to take a look on the host your server is on and it appears there was an issue with QoS Configuration. I have corrected the issue on the host and I see that your server is online and responding to requests." | 15:08 |
| Shrews | mordred: i just +2 | 15:08 |
| Shrews | mordred: oh, new PS | 15:08 |
| AJaeger | config-core, could you review https://review.openstack.org/538632 https://review.openstack.org/538631 and https://review.openstack.org/538306 , please? Those remove some jobs from project-config. | 15:09 |
| pabelanger | fungi: great, to we want to fail back or have the confirm mirror02 has same issue? | 15:09 |
| AJaeger | infra-root, rack and python-rack repos are retired - but there are still open reviews. Could one of you abandon them, please? OR give me permissions to do that... See also https://review.openstack.org/539431 to finish retirement. | 15:10 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Allow operator to set held node's TTL https://review.openstack.org/539493 | 15:10 |
| *** claudiub|3 has joined #openstack-infra | 15:10 | |
| *** alex_xu has quit IRC | 15:11 | |
| fungi | pabelanger: i think next is to probably approve https://review.openstack.org/539256 (i have removed my wip vote from it now) | 15:12 |
| mordred | AJaeger: I have given you permissions to - although I'll also go adandon some | 15:12 |
| *** olaph1 has joined #openstack-infra | 15:12 | |
| *** alex_xu has joined #openstack-infra | 15:13 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint get speaker summit assistance CSV https://review.openstack.org/539562 | 15:13 |
| *** olaph has quit IRC | 15:13 | |
| pabelanger | fungi: +3 | 15:13 |
| *** pcaruana has joined #openstack-infra | 15:14 | |
| mordred | AJaeger: ok. all abandoned | 15:15 |
| efried | mriedem: Remind me (again, please): What's the placement API minimum for Q? 1.14? | 15:16 |
| mordred | AJaeger: and +2 on the retirement patch | 15:16 |
| mriedem | efried: 1.17 right now | 15:16 |
| Roamer` | hm, I just noticed that Nodepool's config has a couple of variables that would only make sense as unsigned integers (not even -1 "don't care" handling), but config_validator can only list them as "int" since Python does not really have an uint type; has somebody considered doing some >= 0 validation on these? Is it worth it? | 15:16 |
| mriedem | for the GET /allocation_candidates?required=<trait> stuff | 15:17 |
| efried | mriedem: Oh, okay, thanks. | 15:17 |
| AJaeger | mordred: thanks - better remove my permissions again ... | 15:17 |
| mordred | AJaeger: done | 15:17 |
| openstackgerrit | Major Hayden proposed openstack/gertty master: Hide Zuul comments https://review.openstack.org/514662 | 15:17 |
| AJaeger | mordred: thanks | 15:18 |
| *** cshastri has joined #openstack-infra | 15:18 | |
| *** kiennt26_ has quit IRC | 15:18 | |
| AJaeger | ok, then https://review.openstack.org/539431 is ready for review - config-core, for your next go over project-config... | 15:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove tripleo pipelines from zuul https://review.openstack.org/537611 | 15:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Disable tripleo-test-cloud-rh1 for nodepool https://review.openstack.org/537991 | 15:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove tripelo-test-cloud-rh1 https://review.openstack.org/537992 | 15:19 |
| openstackgerrit | Merged openstack-infra/project-config master: Moving nova functional test def to in tree https://review.openstack.org/533608 | 15:20 |
| fungi | Roamer`: i'm not entirely opposed to stricter scoping on the validation rules, but we mostly have those to catch egregious typos. the likelihood of someone putting a negative integer in one of those and then being surprised by the resulting behavior is probably small enough not to worry me | 15:21 |
| Roamer` | fungi, my question was mainly prompted by an incoming change, https://review.openstack.org/539493 - there a time interval is introduced as a cmdline option, and people do make mistakes when entering commands | 15:22 |
| fungi | ahh, yes stray hyphens on the command line are certainly a higher probability | 15:23 |
| *** david-lyle has quit IRC | 15:24 | |
| *** dklyle has joined #openstack-infra | 15:24 | |
| Roamer` | but yes, I do agree that it would be 1. more difficult, and 2. more easily caught in a config file | 15:24 |
| *** slaweq_ has quit IRC | 15:25 | |
| fungi | though for 539493 the difference between ignoring the mistyped value and raising a validation exception are not enough for me to really care either way (other reviewers might i suppose?) | 15:25 |
| fungi | it's not like passing a negative value there will result in strange behavior, it'll just be like not passing the option at all | 15:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Temporarily reduce max-servers in rax-dfw" https://review.openstack.org/539256 | 15:26 |
| Roamer` | true, but that would certainly not be what the operator (most probably) meant | 15:26 |
| *** Guest87240 is now known as mgagne | 15:27 | |
| *** mgagne has joined #openstack-infra | 15:27 | |
| rosmaita | hello! does anyone know if i push a docs change to stable/pike, will the stuff at https://docs.openstack.org/glance/pike/ be updated automatically, or will some intervention be required? | 15:27 |
| Roamer` | but yeah, I am a bit of a stickler for validation in most of my projects, more than most people, so "just let it go already" is a valid answer :) | 15:28 |
| fungi | Roamer`: yeah, i agree it's probably marginally more correct to return an error than to silently ignore the option, but not by enough for me to -1 it | 15:28 |
| fungi | certainly viable as a future improvement | 15:28 |
| *** jamesmcarthur has joined #openstack-infra | 15:29 | |
| *** eharney has quit IRC | 15:29 | |
| fungi | rosmaita: as long as the stable/pike branch is still open (not past eol) and there are docs jobs defined for the repo then it should in theory update... however that'll run as a job in the post pipeline which gets lowest priority when we're running at capacity, so might take a while | 15:30 |
| *** lucas-hungry is now known as lucasagomes | 15:30 | |
| mordred | pabelanger: the tripleo removal patches - it looks like there is a diskimage called tripleo-centos-7 - do we need to remove it from the config too? | 15:30 |
| rosmaita | fungi ack, ty | 15:31 |
| *** oidgar has quit IRC | 15:31 | |
| *** armaan has quit IRC | 15:32 | |
| *** sshnaidm|mtg has quit IRC | 15:32 | |
| fungi | pabelanger: from scrollback in #rdo i take it they're still not able to switch off of our mirror yet due to *.openstack.org hard-coded in the allowed hosts filter for pip? | 15:33 |
| *** sshnaidm|mtg has joined #openstack-infra | 15:34 | |
| *** tobberydberg has joined #openstack-infra | 15:34 | |
| *** armaan has joined #openstack-infra | 15:34 | |
| *** armaan has quit IRC | 15:34 | |
| *** armaan has joined #openstack-infra | 15:35 | |
| Roamer` | right, now a stupid question about Stackalytics: basically how do I get myself in there? :) I guess I propose a change to its default_data.json? Also, if people thing I'm worthy and accept the change, will it pick up my merged commits, or does it only go forward? | 15:35 |
| *** links has joined #openstack-infra | 15:35 | |
| *** psachin has quit IRC | 15:35 | |
| *** esberglu_ has joined #openstack-infra | 15:36 | |
| fungi | Roamer`: great question... i think you only have to set entries in the stackalytics config if you want to override their default handling (it's misclassifying your employer affiliation, you've used multiple addresses to commit and want them treated as one profile, et cetera) | 15:36 |
| pabelanger | mordred: no, it is just diskimage: centos-7, but labels are tripleo-centos-7. Follow up patches should get them all (in stack) | 15:36 |
| *** salv-orlando has quit IRC | 15:37 | |
| mordred | pabelanger: ah -ok. my brain just sucks then | 15:37 |
| pabelanger | fungi: yah, trying to see if they deleted the DIB and revert, or are limping along with failure. But we are bottlenecked on rebuilding DIB (poor I/O) | 15:37 |
| fungi | Roamer`: more likely if you're not seeing your activity in stackalytics, it's because their data refreshing has stalled out again and one of their admins needs to give it a kick | 15:37 |
| *** yamamoto has quit IRC | 15:38 | |
| fungi | Roamer`: we neither maintain the stackalytics source code/configuration nor manage that service so not much we can help with there, i'm afraid | 15:38 |
| *** wolverineav has joined #openstack-infra | 15:38 | |
| *** esberglu has quit IRC | 15:40 | |
| *** salv-orlando has joined #openstack-infra | 15:41 | |
| Roamer` | fungi, ok, got it, thanks for the clarifications anyway | 15:42 |
| pabelanger | fungi: okay, doesn't look like they reverted the image. I can see jobs using mirror.regionone.rdo-cloud.rdoproject.org in RDO now. So _some_ jobs are off mirror.dfw.rax.o.o now | 15:44 |
| *** felipemonteiro__ has joined #openstack-infra | 15:44 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:45 | |
| mordred | pabelanger, Shrews: had to rebase can you +A me again https://review.openstack.org/539563 ? | 15:46 |
| *** coolsvap has joined #openstack-infra | 15:46 | |
| clarkb | pabelanger: Shrews if there are no tests using the old pipelines in https://review.openstack.org/#/c/537611/5 I have no issue with removing the pipelines (and last I remember zuul explicitly checks for that and its an error if there are still jobs in those pipelines) | 15:46 |
| pabelanger | clarkb: Shrews: yah, zuul did check (and reported when we still had jobs), but all clear now! | 15:47 |
| *** oidgar has joined #openstack-infra | 15:47 | |
| *** yamamoto has joined #openstack-infra | 15:48 | |
| *** esberglu_ is now known as esberglu | 15:48 | |
| *** felipemonteiro__ has quit IRC | 15:50 | |
| Shrews | mordred: done | 15:51 |
| clarkb | pabelanger: is that all you wanted an all clear on? | 15:51 |
| *** sshnaidm|mtg has quit IRC | 15:52 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Allow operator to set held node's TTL https://review.openstack.org/539493 | 15:52 |
| *** wolverineav has quit IRC | 15:53 | |
| pabelanger | clarkb: yah, just wanted to make sure you knew about it, and were okay with it | 15:53 |
| *** yamamoto has quit IRC | 15:53 | |
| *** wolverineav has joined #openstack-infra | 15:53 | |
| pabelanger | Shrews: want to +A 537611? | 15:53 |
| clarkb | ya I think its fine. Looks like tripleo has reviewed the changes too | 15:53 |
| pabelanger | 537991 too | 15:53 |
| pabelanger | ++ | 15:54 |
| clarkb | looks like mirror02.dfw.rax.openstack.org has the dfw mirror cname and cacti makes it looks reasonably happy? | 15:56 |
| clarkb | that should be it for the known fires right? | 15:56 |
| *** slaweq has joined #openstack-infra | 15:56 | |
| mgagne | pabelanger: so at this point, inap-mtl01 has qcow2 base file on compute nodes, no more raw. let me know if you see any improvement or problem. | 15:57 |
| *** wolverineav has quit IRC | 15:57 | |
| pabelanger | mgagne: sure, let me check grafana | 15:58 |
| clarkb | AJaeger: https://review.openstack.org/#/c/539431/1 is approved | 16:00 |
| *** eharney has joined #openstack-infra | 16:00 | |
| pabelanger | mgagne: last 24 hours look great, zero launch failure. We've also uploaded a few images | 16:01 |
| *** dhajare_ has quit IRC | 16:02 | |
| pabelanger | wow, we haven't build a new ubuntu-trusty image in 2 weeks | 16:03 |
| pabelanger | looking to see why | 16:03 |
| fungi | clarkb: i'm not so sure about mirror traffic on mirror02.dfw.rax... it's not gone significantly above 100mbps in the past ~4 hours | 16:03 |
| clarkb | fungi: ya, but it also doesn't seem to have flatlined there like before either? | 16:04 |
| mgagne | pabelanger: ok thanks, lets if there is any overall improvements | 16:04 |
| fungi | clarkb: right, waiting to see what happens with the max-servers bump back to 140 there | 16:04 |
| clarkb | so those peaks may be normal? I guess as we get into the middle of this work day we should learn more | 16:04 |
| pabelanger | mgagne: ++ | 16:05 |
| fungi | clarkb: if it looks stuck around 100mbps we might should consider rolling back to mirror01 now that they've found and fixed a qos issue on the host | 16:05 |
| *** bfournie has quit IRC | 16:05 | |
| *** bfournie has joined #openstack-infra | 16:05 | |
| fungi | or, worst case, start round-robin dns between the two | 16:06 |
| *** bfournie has quit IRC | 16:06 | |
| *** bfournie has joined #openstack-infra | 16:06 | |
| clarkb | ya | 16:06 |
| *** bfournie has quit IRC | 16:08 | |
| *** efried_hexchat has quit IRC | 16:08 | |
| *** bfournie has joined #openstack-infra | 16:08 | |
| *** kiennt26 has quit IRC | 16:08 | |
| *** bfournie has quit IRC | 16:08 | |
| *** sshnaidm|mtg has joined #openstack-infra | 16:08 | |
| *** bfournie has joined #openstack-infra | 16:08 | |
| *** daidv has quit IRC | 16:09 | |
| *** bfournie has quit IRC | 16:09 | |
| *** bfournie has joined #openstack-infra | 16:09 | |
| AJaeger | pabelanger: shall I +A 537611? Or are you waiting for anybody? | 16:09 |
| pabelanger | AJaeger: sure, go for it | 16:10 |
| AJaeger | clarkb: thanks | 16:10 |
| *** kiennt26 has joined #openstack-infra | 16:10 | |
| AJaeger | pabelanger: the rest of the stack as well? Or should we merge them one by one? | 16:10 |
| *** dsariel has quit IRC | 16:11 | |
| *** daidv has joined #openstack-infra | 16:11 | |
| pabelanger | AJaeger: yah, up to WIP -1 should be fine | 16:11 |
| *** agopi has quit IRC | 16:12 | |
| AJaeger | pabelanger: done | 16:13 |
| clarkb | mordred: https://review.openstack.org/#/c/538306/1 is one that AJaeger pointed out earlier that probably interests you | 16:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove initialize-urandom from ubuntu-trusty https://review.openstack.org/539585 | 16:16 |
| pabelanger | config-core: ^ fixes ubuntu-trusty DIBs, which have been broken for 2 weeks :( | 16:16 |
| clarkb | pabelanger: thanks for checking on that (reviewing now). We might also want to check if xenial still needs that too | 16:16 |
| pabelanger | clarkb: xenial looks to be correct | 16:17 |
| clarkb | I mean xenail's kernel may no longer have the bug that required us to prime it | 16:17 |
| clarkb | (its something we'll have to test) | 16:17 |
| AJaeger | pabelanger: +2A | 16:18 |
| *** mamitchl has left #openstack-infra | 16:18 | |
| pabelanger | clarkb: Oh, ya. Maybe we can drop it | 16:18 |
| pabelanger | +1 to test | 16:19 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Allow operator to set held node's TTL https://review.openstack.org/539493 | 16:21 |
| pabelanger | AJaeger: clarkb: I think we can also +3 https://review.openstack.org/539229/ | 16:21 |
| pabelanger | restart ubuntu-xenial builds, we are setup to use AFS mirrors now | 16:22 |
| pabelanger | fungi: too^ | 16:22 |
| clarkb | looks like AJaeger got it | 16:22 |
| * AJaeger is happy to shrink our open reviews ;) | 16:22 | |
| *** janki has quit IRC | 16:22 | |
| *** wolverineav has joined #openstack-infra | 16:24 | |
| *** oidgar has quit IRC | 16:24 | |
| *** b_bezak has quit IRC | 16:25 | |
| *** ykarel is now known as ykarel|away | 16:25 | |
| *** b_bezak has joined #openstack-infra | 16:25 | |
| *** dsariel has joined #openstack-infra | 16:27 | |
| fungi | looks like traffic levels on mirror02.dfw.rax are trending back above 100mbps again, so i'm increasingly comfortable we're not generating unnecessary packet loss there now | 16:27 |
| fungi | i'll feel even better when i see some sustained periods above the line | 16:28 |
| pabelanger | great | 16:28 |
| *** r-daneel has joined #openstack-infra | 16:29 | |
| *** ykarel|away has quit IRC | 16:29 | |
| *** b_bezak has quit IRC | 16:30 | |
| fungi | gonna perform some periodic ping tests to it as well, just to be sure we're not seeing any significant losses | 16:35 |
| *** gongysh has quit IRC | 16:38 | |
| *** ralonsoh has joined #openstack-infra | 16:38 | |
| *** pcaruana has quit IRC | 16:39 | |
| *** ralonsoh__ has quit IRC | 16:41 | |
| *** andreas_s has joined #openstack-infra | 16:42 | |
| *** cshastri has quit IRC | 16:43 | |
| *** pblaho has quit IRC | 16:43 | |
| cmurphy | I noticed our unit tests were timing out randomly with some really weird symptoms https://bugs.launchpad.net/keystone/+bug/1746016 and I wasn't sure if it was just our tests being weird but I'm seeing something similar with this devstack run https://zuul.openstack.org/stream.html?uuid=73fdd07ea092476fba8b3390f144efb5&logfile=console.log | 16:43 |
| openstack | Launchpad bug 1746016 in OpenStack Identity (keystone) "unit test jobs sometimes time out" [High,New] | 16:43 |
| cmurphy | seems like the node just sort of drops out of contact for a while | 16:43 |
| pabelanger | Hmm, could be an networking issue with provider | 16:47 |
| cmurphy | the provider is inconsistent | 16:48 |
| cmurphy | sometimes rax sometimes inap | 16:48 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] zuul autohold: allow operator to specify nodes TTL https://review.openstack.org/539596 | 16:48 |
| pabelanger | cmurphy: has there been rax issues today? | 16:48 |
| pabelanger | cmurphy: we did have an overload mirror recently, wonder if related to that | 16:48 |
| cmurphy | pabelanger: not that i've seen, the devstack issue i'm looking at ran on inap and the unit test timeouts were from a couple of days ago | 16:49 |
| pabelanger | let me check mirror in inap | 16:49 |
| fungi | hrw: looking at the failure you linked earlier, the ssh host key error may be a red herring, but the job began failing earlier in the run phase when it hit connection timeouts pip installing things from the mirror in that region: http://logs.openstack.org/46/539246/1/gate/kolla-build-centos-source/a287ab7/ara/result/65662999-bd7c-4d83-aca1-85aad2450f95/ | 16:50 |
| fungi | ReadTimeoutError: HTTPConnectionPool(host='mirror.mtl01.inap.openstack.org', port=80): Read timed out. | 16:50 |
| hrw | fungi: thanks. recheck in progress | 16:50 |
| fungi | hrw: seems like cmurphy and pabelanger may be looking into a similar failure in inap as well | 16:50 |
| pabelanger | yah, seems like something is going on | 16:51 |
| pabelanger | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64427&rra_id=all | 16:51 |
| fungi | so we may be seeing a more general network connectivity issue in their cloud | 16:51 |
| pabelanger | looks pretty good | 16:51 |
| pabelanger | fungi: yah, looks like it | 16:51 |
| fungi | mgagne: ^ just a heads up, we're still correlating | 16:51 |
| hrw | ;) | 16:52 |
| hjensas | Is this job hung? http://zuul.openstack.org/stream.html?uuid=dbae41c3a7234985bc5d6766b58dc405&logfile=console.log ? review: 471969 | 16:52 |
| hrw | thanks guys | 16:52 |
| *** daidv has quit IRC | 16:52 | |
| *** kiennt26 has quit IRC | 16:52 | |
| mgagne | fungi: iirc, last time it was related to puppet reloading apache or something like that. otherwise, I'm not aware of any networking issues in inap-mtl01 | 16:52 |
| pabelanger | hjensas: yah, looks like networking issue. Lets see what happens | 16:53 |
| *** kiennt26 has joined #openstack-infra | 16:53 | |
| *** daidv has joined #openstack-infra | 16:53 | |
| *** b_bezak has joined #openstack-infra | 16:54 | |
| hjensas | pabelanger: ok. I'll try some more patience. | 16:55 |
| *** slaweq has quit IRC | 16:55 | |
| mgagne | pabelanger: oh, I deleted ~4 orphan instances ~1h40 ago | 16:56 |
| *** andreas_s has quit IRC | 16:56 | |
| fungi | that could explain the earlier failure about a mismatched ssh host key too | 16:57 |
| *** tmorin has quit IRC | 16:57 | |
| fungi | in the run hrw linked | 16:57 |
| fungi | since additional virtual machines fighting over the same ip address are the usual cause for that particular behavior | 16:58 |
| pabelanger | okay, ansible wheel is reporting failure in git.o.o, looking into that now (notice mirror.mtl01.inap.o.o hasn't run puppet is a while) | 16:58 |
| * mgagne sighs | 16:58 | |
| *** felipemonteiro_ has quit IRC | 16:58 | |
| *** e0ne has quit IRC | 16:58 | |
| fungi | but could also result in random network timeouts due to constant arp overwrites | 16:58 |
| *** felipemonteiro_ has joined #openstack-infra | 16:59 | |
| openstackgerrit | Merged openstack-infra/project-config master: Finalize rack requirement https://review.openstack.org/539431 | 16:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove tripleo pipelines from zuul https://review.openstack.org/537611 | 16:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Disable tripleo-test-cloud-rh1 for nodepool https://review.openstack.org/537991 | 16:59 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Pause ubuntu-xenial image uploads" https://review.openstack.org/539229 | 17:00 |
| openstackgerrit | Merged openstack-infra/project-config master: remove service-type-authority tox job https://review.openstack.org/538306 | 17:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Fix puppet error on centos https://review.openstack.org/539598 | 17:00 |
| pabelanger | infra-root: should fix puppet on git.o.o ^ seems puppet on centos doesn't like missing quotes. | 17:00 |
| fungi | thanks! | 17:00 |
| *** sshnaidm|mtg is now known as sshnaidm | 17:01 | |
| *** efried is now known as efried_rollin | 17:02 | |
| *** harlowja has joined #openstack-infra | 17:02 | |
| *** yamamoto has joined #openstack-infra | 17:03 | |
| *** slaweq has joined #openstack-infra | 17:03 | |
| *** ramishra has quit IRC | 17:03 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Enabled ssh retries for ansible https://review.openstack.org/537953 | 17:05 |
| *** dprince has quit IRC | 17:08 | |
| *** wolverineav has quit IRC | 17:11 | |
| *** slaweq has quit IRC | 17:11 | |
| *** wolverineav has joined #openstack-infra | 17:12 | |
| *** bhavik1 has joined #openstack-infra | 17:12 | |
| *** vivsoni__ has quit IRC | 17:12 | |
| *** dsariel has quit IRC | 17:13 | |
| *** vivsoni__ has joined #openstack-infra | 17:13 | |
| cmurphy | https://review.openstack.org/538314 still hanging in the gate queue, seems to have been that way for the last two and half hours | 17:14 |
| *** wolverineav has quit IRC | 17:16 | |
| *** yamamoto has quit IRC | 17:16 | |
| *** Cibo has joined #openstack-infra | 17:19 | |
| *** iyamahat has quit IRC | 17:19 | |
| *** ralonsoh_ has joined #openstack-infra | 17:21 | |
| *** links has quit IRC | 17:21 | |
| *** sree has quit IRC | 17:22 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove initialize-urandom from ubuntu-trusty https://review.openstack.org/539585 | 17:22 |
| *** yamahata has quit IRC | 17:22 | |
| *** sree has joined #openstack-infra | 17:23 | |
| *** myoung is now known as myoung|rabbit | 17:23 | |
| *** ralonsoh has quit IRC | 17:24 | |
| *** dprince has joined #openstack-infra | 17:29 | |
| corvus | cmurphy: that's a curious place to stop: 2018-01-31 15:48:04.419821 | TASK [process-test-results : Discover stestr repo] | 17:30 |
| corvus | cmurphy: i'll see if i can log into the vm running that job | 17:30 |
| corvus | inap-mtl01 | 17:31 |
| corvus | | 0002297127 | inap-mtl01 | ubuntu-xenial | e21e4d48-79fa-47bd-956d-077abb8e8bba | 198.72.124.228 | | in-use | 00:03:57:39 | locked | | 17:31 |
| cmurphy | oooh "locked" | 17:32 |
| corvus | that's normal -- that just means zuul is still using it | 17:32 |
| cmurphy | oh | 17:32 |
| corvus | now this is interesting | 17:32 |
| *** Swami has joined #openstack-infra | 17:32 | |
| corvus | i try to ssh into it, and about 75% of the time i get "ssh_exchange_identification: Connection closed by remote host" and the remainder it performs a key exchange, but then stops at the next step | 17:33 |
| *** b_bezak has quit IRC | 17:33 | |
| *** dsariel has joined #openstack-infra | 17:34 | |
| fungi | corvus: keep in mind that mgagne said at 16:56z that he "deleted ~4 orphan instances ~1h40 ago" there so... | 17:34 |
| *** b_bezak has joined #openstack-infra | 17:34 | |
| corvus | yeah, i suspect this is somehow related | 17:34 |
| *** bhavik1 has quit IRC | 17:34 | |
| fungi | may not want to rule out possibilities which include more than one vm instance "sharing" the same ip address | 17:34 |
| *** sree has quit IRC | 17:34 | |
| corvus | i feel like that's the more likely explanation, based on the differing behavior between subsequent ssh connection attempts | 17:35 |
| corvus | mgagne: ^ did you delete that instance, or is that ip address in use by more than one? | 17:36 |
| mgagne | corvus: deleted instances which weren't active in Nova | 17:36 |
| corvus | mgagne: (note, as far as we know, this should still be a current, live instance) | 17:36 |
| mgagne | corvus: above instance is still active and running on compute node | 17:37 |
| corvus | mgagne: any chance something else is trying to use its ip? | 17:38 |
| mgagne | checking atm | 17:38 |
| mgagne | our monitoring found one new occurence | 17:38 |
| mgagne | well, false positive in fact | 17:38 |
| *** b_bezak has quit IRC | 17:38 | |
| mgagne | 198.72.124.228 is only used once and it's on the right compute node | 17:40 |
| fungi | not seeing any arp overwrites for that logged on the gateway? | 17:41 |
| AJaeger | fungi, thanks for digging deeper into that glance docs failure than I did - good idea. | 17:41 |
| fungi | unfortunately our mirror instance isn't in the same broadcast domain so i can't spot arp overwrites for 198.72.124.228 from there | 17:41 |
| pabelanger | would some sort of keepalive in ssh help in this case? If it didn't reply in x seconds, the connection would be broken? | 17:42 |
| pabelanger | in an effort to reduce waiting for job timeout to kick in | 17:42 |
| fungi | pabelanger: not necessarily... right now we get a couple different behaviors attempting to connect to ssh on that node | 17:42 |
| *** iyamahat has joined #openstack-infra | 17:43 | |
| fungi | though so far i haven't seen more than one host key for it, at least | 17:43 |
| pabelanger | would be interesting to see if ssh connection from executor is still up, I would assume so, since ansible is hung | 17:44 |
| *** vivsoni__ has quit IRC | 17:44 | |
| *** vivsoni__ has joined #openstack-infra | 17:44 | |
| *** tesseract has quit IRC | 17:45 | |
| fungi | not much of note in the nova console output for that instance | 17:47 |
| AJaeger | infra-root, I see that http://grafana.openstack.org/dashboard/db/nodepool-infra-cloud still exists - and a couple of other remoed dashboards. anybody cares to manually delete unused dashboards, please? | 17:48 |
| fungi | lots of "systemd[1]: Failed to start Journal Service." roughly 44.5 minutes into its lifetime | 17:48 |
| *** felipemonteiro_ has quit IRC | 17:48 | |
| *** jcoufal has quit IRC | 17:49 | |
| *** jcoufal_ has joined #openstack-infra | 17:49 | |
| pabelanger | AJaeger: yah, we don't have support to delete dashboards in grafyaml yet, something I've been meaning to loop back too | 17:49 |
| fungi | prior to that, numerous "task .* .* blocked for more than 120 seconds." errors starting up around 34 minutes from boot | 17:50 |
| fungi | the instance looks like it's probably very unhappy | 17:50 |
| *** panda is now known as panda|off | 17:51 | |
| pabelanger | fungi: wonder if console log shows anything, from openstack api | 17:52 |
| fungi | pabelanger: that's what i'm talking about | 17:52 |
| pabelanger | Oh | 17:52 |
| fungi | it's what i meant by "nova console output" above | 17:53 |
| pabelanger | sorry, didn't make the connection | 17:53 |
| fungi | openstack console log show | 17:53 |
| fungi | nah, i should have been more explicit | 17:53 |
| corvus | AJaeger: how does this look? https://etherpad.openstack.org/p/Xr2hteqsig | 17:54 |
| fungi | pabelanger: `OS_CLIENT_CONFIG_FILE=/etc/openstack/all-clouds.yaml openstack --os-cloud openstackjenkins-inap --os-region-name mtl01 console log show e21e4d48-79fa-47bd-956d-077abb8e8bba` if you're interested | 17:54 |
| fungi | or the ~fungi/console-log-show file on puppetmaster has the content from that | 17:54 |
| pabelanger | thanks | 17:54 |
| *** vivsoni__ has quit IRC | 17:55 | |
| *** vivsoni__ has joined #openstack-infra | 17:55 | |
| pabelanger | fungi: Hmm, jbd2/vda1-8 doesn't look happy. Guess something is wrong with HDD? | 17:55 |
| fungi | pabelanger: that would probably explain the sshd hangs after auth | 17:56 |
| fungi | if it's failing to write to wtmp | 17:56 |
| *** jamesmcarthur has quit IRC | 17:56 | |
| corvus | doesn't it usually move past that though, after maybe 30-60s? | 17:56 |
| AJaeger | corvus: LGTM, thanks | 17:56 |
| *** sree has joined #openstack-infra | 17:56 | |
| AJaeger | corvus: want to add "topic:zuul-projects" to the subject? | 17:57 |
| fungi | corvus: oh, maybe. i'm used to 30-60 seconds when it's timing out looking up reverse dns entries, but not from failure to be able to write to /var | 17:57 |
| corvus | fungi: oh hrm | 17:57 |
| corvus | AJaeger: also, before i resume, i was going to shuffle the remaining changes, so rather than uploading all the puppet changes at once, i would upload them in random order, to reduce the chances of batched approvals. | 17:57 |
| AJaeger | corvus: what about pushing *master* first - before queens is created? | 17:58 |
| corvus | AJaeger: i can do that. random master, then random other branches. | 17:59 |
| AJaeger | corvus: please abandon https://review.openstack.org/538461 and https://review.openstack.org/538718 - those are dups | 17:59 |
| mordred | mtreinish: I think our checks for stestr output in fetch-subunit-output are incorrect ... | 17:59 |
| AJaeger | corvus: yeah | 17:59 |
| fungi | pabelanger: anyway, i wouldn't read a ton into the "task jbd2/vda1-8:214 blocked" one specifically since it's just one of many random tasks including conntrackd, systemd-journal, mysqld, dstat, loop0, kworker/u16... | 17:59 |
| pabelanger | fungi: kk, wasn't sure if possible beginnings of hardware issue for storage | 18:00 |
| *** sree has quit IRC | 18:01 | |
| *** trown is now known as trown|lunch | 18:01 | |
| fungi | i wouldn't rule it out, but there's lots of possibilities since we've got so little actual diagnostic info to go on | 18:01 |
| fungi | this is starting to seem very similar to an earlier incident when we looked into one of these. lemme find the log of our troubleshooting from that one | 18:02 |
| AJaeger | corvus: once all changes are merged, I suggest to double check that no new branches got created or changes added with name in project stanza.... | 18:02 |
| *** jamesmcarthur has joined #openstack-infra | 18:03 | |
| *** yamahata has joined #openstack-infra | 18:03 | |
| *** kopecmartin has quit IRC | 18:06 | |
| corvus | AJaeger: yep | 18:08 |
| fungi | amusing, the last time i was trying to diagnose one of these hung jobs (2018-01-05 ~2100z), the meetbot was offline or split so didn't record the discussion | 18:09 |
| corvus | AJaeger, config-core, infra-root: uploads of new changes to remove project names from zuul stanza are proceeding again. i sent an email to the -dev list to give folks a heads up, and i've also randomized the uploads so changes are no longer batched by project. | 18:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Look for 0 in .stestr directory instead of failing https://review.openstack.org/539624 | 18:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Skip checksums on stat calls in fetch-subunit-output https://review.openstack.org/539625 | 18:11 |
| *** hongbin has joined #openstack-infra | 18:12 | |
| *** HeOS has joined #openstack-infra | 18:12 | |
| *** jpena is now known as jpena|away | 18:13 | |
| mordred | gah. I can't read. | 18:13 |
| pabelanger | corvus: clarkb: fungi: mordred: I didn't add to weekly infra meeting, but wanted to bring up our migration plan to zk cluster. Given our outage on the weekend, maybe something we want to start to schedule. Is that something we might want to try this week or next? Or maybe just hold of until PTG? | 18:13 |
| fungi | corvus: pabelanger: cmurphy: http://paste.openstack.org/show/658211 is a prior similar diagnostic discussion which ended up omitted from http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-01-05.log.html due to meetbot issues | 18:15 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] zuul autohold: allow operator to specify nodes TTL https://review.openstack.org/539596 | 18:15 |
| fungi | so... errily similar (dare i say identical) symptoms that time as this | 18:16 |
| corvus | fungi: well remembered | 18:16 |
| *** olaph has joined #openstack-infra | 18:16 | |
| *** SumitNaiksatam has quit IRC | 18:17 | |
| *** e0ne has joined #openstack-infra | 18:17 | |
| *** olaph1 has quit IRC | 18:17 | |
| pabelanger | Yah | 18:17 |
| pabelanger | wonder if it is the same compute host | 18:17 |
| corvus | i'm open to suggestions about how ansible/zuul could detect this case and avoid blocking for hours | 18:17 |
| corvus | (also, someone may want to commit-msg bump that change out of the queue to get things moving) | 18:18 |
| corvus | i'll do it | 18:19 |
| fungi | that last one from the paste and now are the only times we've caught one for long enough to be able to dig this deep, but there have been a lot of others which looked like the same sorts of hangs and so far all the ones i've been involved in even a cursory look into happened in the same provider | 18:19 |
| *** e0ne has quit IRC | 18:19 | |
| AJaeger | corvus: just mass -1'ed chagnes that introduce project name stanza (checked only for .zuul.yaml files) | 18:19 |
| fungi | and seems like a pathological enough failure mode i'm not sure zuul/ansible/openssh can really do much to paper over it | 18:20 |
| corvus | AJaeger: my brain isn't working well and i can't parse your last sentence to me. | 18:20 |
| AJaeger | corvus: I reviewed changes with "is:open file:.zuul.yaml" that added a new .zuul.yaml file and gave -1 to all those project name stanzas and referenced your email | 18:21 |
| *** myoung|rabbit is now known as myoung | 18:22 | |
| AJaeger | corvus: sorry, that my brain can't utter understandable English ;( | 18:22 |
| corvus | AJaeger: ah, thank you! | 18:22 |
| corvus | AJaeger: no i really am sluggish, i'm sick and got little sleep. | 18:22 |
| corvus | smcginnis: can you re-approve 538314 ? i had to kick it out of the gate because it was stuck in a very slow timeout process | 18:23 |
| smcginnis | corvus: ack | 18:23 |
| *** pcichy has quit IRC | 18:23 | |
| *** pcichy has joined #openstack-infra | 18:24 | |
| smcginnis | corvus: Ah, it's still approved, but it's gone back to the check queue. | 18:25 |
| corvus | oh neat. i think there's a subtle gerrit behavior change there :) | 18:25 |
| corvus | it should be gtg then | 18:26 |
| openstackgerrit | Merged openstack-infra/system-config master: Fix puppet error on centos https://review.openstack.org/539598 | 18:26 |
| *** sree has joined #openstack-infra | 18:28 | |
| *** tosky has quit IRC | 18:28 | |
| fungi | corvus: bizarre... so you amended the commit message with a trailing . and then removed that in a subsequent edit (two edits total) and it retained code-review and workflow votes? | 18:30 |
| *** vivsoni__ has quit IRC | 18:31 | |
| *** vivsoni has joined #openstack-infra | 18:31 | |
| clarkb | old gerrit would not have allowed the revert | 18:31 |
| clarkb | new does and must remember the old votes in that case | 18:31 |
| smcginnis | That does appear to be the case. | 18:32 |
| AJaeger | yes, seen it also earlier - new gerrit remains code-review for this. I didn't notice workflow as well. | 18:35 |
| *** kjackal has quit IRC | 18:35 | |
| fungi | i wonder if that's only the case when updating the commit message via webui/api and not for pushing up a new commit which only alters the commit message | 18:36 |
| *** felipemonteiro has joined #openstack-infra | 18:36 | |
| pabelanger | config-core: I've just approved https://review.openstack.org/537992/ after confirming no more images / nodes in nodepool for tripleo-test-cloud-rh1 | 18:36 |
| *** sree has quit IRC | 18:36 | |
| *** dtantsur is now known as dtantsur|afk | 18:37 | |
| pabelanger | clarkb: fungi: could I get a +3 on https://review.openstack.org/536179/, fixes issue with nodepool logging script we run in system-config | 18:38 |
| *** iyamahat has quit IRC | 18:38 | |
| *** iyamahat has joined #openstack-infra | 18:39 | |
| pabelanger | https://review.openstack.org/538623/ is another easy review to enable kazoo logging in nodepool | 18:39 |
| *** iyamahat has quit IRC | 18:40 | |
| *** iyamahat has joined #openstack-infra | 18:40 | |
| *** e0ne has joined #openstack-infra | 18:40 | |
| *** iyamahat has quit IRC | 18:43 | |
| *** iyamahat has joined #openstack-infra | 18:43 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:43 | |
| *** dsariel has quit IRC | 18:47 | |
| *** harlowja has quit IRC | 18:52 | |
| *** sree has joined #openstack-infra | 18:52 | |
| *** shardy has quit IRC | 18:53 | |
| *** kjackal has joined #openstack-infra | 18:54 | |
| *** tosky has joined #openstack-infra | 18:55 | |
| *** sshnaidm is now known as sshnaidm|off | 18:55 | |
| *** sree has quit IRC | 18:57 | |
| *** priteau has quit IRC | 18:57 | |
| *** david-lyle_ has joined #openstack-infra | 18:58 | |
| *** Cibo has quit IRC | 19:01 | |
| *** dklyle has quit IRC | 19:01 | |
| *** iyamahat has quit IRC | 19:02 | |
| *** iyamahat_ has joined #openstack-infra | 19:02 | |
| *** vivsoni has quit IRC | 19:03 | |
| *** vivsoni has joined #openstack-infra | 19:04 | |
| *** lucasagomes is now known as lucas-afk | 19:04 | |
| *** freerunner has quit IRC | 19:06 | |
| *** ruhe has quit IRC | 19:06 | |
| *** SergeyLukjanov has quit IRC | 19:06 | |
| *** NikitaKonovalov has quit IRC | 19:06 | |
| *** DinaBelova has quit IRC | 19:06 | |
| *** DinaBelova has joined #openstack-infra | 19:07 | |
| *** NikitaKonovalov has joined #openstack-infra | 19:07 | |
| *** ruhe has joined #openstack-infra | 19:08 | |
| *** freerunner has joined #openstack-infra | 19:08 | |
| *** david-lyle_ is now known as dklyle | 19:08 | |
| *** SergeyLukjanov has joined #openstack-infra | 19:08 | |
| *** sambetts is now known as sambetts|afk | 19:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove tripelo-test-cloud-rh1 https://review.openstack.org/537992 | 19:20 |
| *** trown|lunch is now known as trown | 19:24 | |
| *** jpena|away is now known as jpena|off | 19:24 | |
| *** amoralej is now known as amoralej|off | 19:25 | |
| *** coolsvap has quit IRC | 19:25 | |
| *** DinaBelova has quit IRC | 19:29 | |
| *** freerunner has quit IRC | 19:29 | |
| *** NikitaKonovalov has quit IRC | 19:29 | |
| *** ruhe has quit IRC | 19:29 | |
| *** SergeyLukjanov has quit IRC | 19:29 | |
| *** ralonsoh_ has quit IRC | 19:30 | |
| *** jamesmcarthur has quit IRC | 19:31 | |
| *** andreww has joined #openstack-infra | 19:32 | |
| *** xarses_ has joined #openstack-infra | 19:33 | |
| *** andreww has quit IRC | 19:37 | |
| *** harlowja has joined #openstack-infra | 19:37 | |
| *** sree has joined #openstack-infra | 19:39 | |
| *** mriedem1 has joined #openstack-infra | 19:41 | |
| *** gyee has joined #openstack-infra | 19:42 | |
| *** DinaBelova has joined #openstack-infra | 19:42 | |
| *** mriedem has quit IRC | 19:42 | |
| *** NikitaKonovalov has joined #openstack-infra | 19:43 | |
| *** sree has quit IRC | 19:43 | |
| *** ruhe has joined #openstack-infra | 19:43 | |
| *** freerunner has joined #openstack-infra | 19:43 | |
| *** SergeyLukjanov has joined #openstack-infra | 19:44 | |
| openstackgerrit | Merged openstack-infra/zuul master: Enabled ssh retries for ansible https://review.openstack.org/537953 | 19:55 |
| *** calebb has quit IRC | 19:57 | |
| *** salv-orlando has quit IRC | 20:00 | |
| *** salv-orlando has joined #openstack-infra | 20:00 | |
| *** calebb has joined #openstack-infra | 20:00 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove support for snapshot images in nodepool https://review.openstack.org/536179 | 20:00 |
| *** jamesmcarthur has joined #openstack-infra | 20:01 | |
| *** calebb is now known as squid | 20:02 | |
| *** squid is now known as calebb | 20:02 | |
| *** armaan has quit IRC | 20:02 | |
| *** armaan has joined #openstack-infra | 20:03 | |
| *** salv-orlando has quit IRC | 20:05 | |
| *** sree has joined #openstack-infra | 20:07 | |
| smcginnis | Another rsync failure: http://logs.openstack.org/ea/ea013a8151dfba0ef82d3a4e7cca84f538834aae/release-post/publish-static/8a1b6a3/job-output.txt.gz#_2018-01-31_20_00_28_845716 | 20:09 |
| *** mriedem1 is now known as mriedem | 20:09 | |
| *** jamesmcarthur has quit IRC | 20:09 | |
| fungi | oh, right, i meant to look into the first one you linked earlier today | 20:09 |
| clarkb | did we get anywhere with the inap thing from earlier? | 20:10 |
| clarkb | (maybe handing off uuid to mgagne to check the hypervisor?) | 20:10 |
| *** sree has quit IRC | 20:11 | |
| clarkb | fungi: smcginnis it looks like /srv/static/releases/ does not exist? | 20:13 |
| smcginnis | I suppose trying to rsync something that doesn't exist might be a problem. | 20:14 |
| clarkb | looking at static.o.o we have both /srv/static/release/ and /srv/static/releases maybe its a permission issue? | 20:14 |
| fungi | looks like /srv/static/releases exists and is writeable by the jenkins user | 20:15 |
| clarkb | drwxrwxr-x 28 jenkins jenkins 4096 Jan 31 20:02 releases | 20:15 |
| clarkb | ya, I'm guessing that task is not sshing to static.o.o as jenkins user? | 20:16 |
| clarkb | it doesn't have a become so its ssh'ing as the current user I think which would be zuul? | 20:16 |
| clarkb | there is no zuul user on that server though so not sure | 20:16 |
| clarkb | oh it logged it ssh'ed as jenkins@static.o.o | 20:17 |
| fungi | looks like we use the jenkins user for publishing in other sites | 20:17 |
| *** ruhe has quit IRC | 20:17 | |
| *** DinaBelova has quit IRC | 20:17 | |
| *** NikitaKonovalov has quit IRC | 20:17 | |
| *** SergeyLukjanov has quit IRC | 20:17 | |
| *** freerunner has quit IRC | 20:17 | |
| *** DinaBelova has joined #openstack-infra | 20:18 | |
| *** NikitaKonovalov has joined #openstack-infra | 20:18 | |
| clarkb | fungi: drwxr-xr-x 22 root root 4096 Jan 27 00:28 static maybe it is trying to update the releases entry under static/ and can't? | 20:18 |
| clarkb | -rw-rw-r-- 1 jenkins jenkins 230 Jan 31 20:02 .buildinfo | 20:19 |
| clarkb | I actually think this is that race that AJaeger identified now that I see ^ | 20:20 |
| clarkb | smcginnis: can you see if the content updated to the expected content? | 20:20 |
| clarkb | smcginnis: the tl;dr is if multiple things copy into a site at the same time they can fight each otehr for the copy from tmp | 20:20 |
| clarkb | the files should be copied mostly correctly | 20:20 |
| clarkb | but will cause one of them to fail | 20:20 |
| *** felipemonteiro_ has joined #openstack-infra | 20:21 | |
| *** NikitaKonovalov has quit IRC | 20:21 | |
| *** DinaBelova has quit IRC | 20:21 | |
| *** DinaBelova has joined #openstack-infra | 20:22 | |
| *** NikitaKonovalov has joined #openstack-infra | 20:22 | |
| *** ruhe2 has joined #openstack-infra | 20:23 | |
| *** freerunner has joined #openstack-infra | 20:23 | |
| *** SergeyLukjanov has joined #openstack-infra | 20:23 | |
| *** ruhe2 is now known as ruhe | 20:23 | |
| smcginnis | clarkb: OK, I'll check in a bit and get back to you. Trying to figure out some release branching issues at the moment. | 20:24 |
| *** felipemonteiro has quit IRC | 20:24 | |
| *** vivsoni_ has joined #openstack-infra | 20:26 | |
| *** vivsoni has quit IRC | 20:26 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove old clouds from nodepool-builder logging https://review.openstack.org/539670 | 20:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove tripleo-test-cloud-rh(1|2) support https://review.openstack.org/539671 | 20:29 |
| pabelanger | clarkb: fungi: last bits for tripleo-test-cloud-rh(1|2)^ | 20:29 |
| pabelanger | we then can revert https://review.openstack.org/539263/ to increase mirror cache sizes | 20:30 |
| clarkb | pabelanger: we should be able to revert the mirror cache size change whenever we stop running jobs in rh1 which I think is already done | 20:31 |
| *** sree has joined #openstack-infra | 20:31 | |
| pabelanger | clarkb: yup, no more jobs there now | 20:31 |
| fungi | that ought to be plenty safe in that case | 20:34 |
| clarkb | the only other thing to check is that we have the room for it on the other mirrors | 20:35 |
| clarkb | remembering that the cache cleaner has a hard time keeping up at times | 20:35 |
| clarkb | (but 50GB should be safe) | 20:35 |
| *** sree has quit IRC | 20:36 | |
| fungi | fair point, though that was mostly on the tripleo cloud mirror where i/o was poor and rate of churn was unsustainable | 20:36 |
| *** links has joined #openstack-infra | 20:36 | |
| ianw | pabelanger / fungi : so do we stick with mirror02.dfw or restore mirror01? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64868&rra_id=all has a peak at ~150, which is the same point as mirror01 in the historical logs | 20:37 |
| mordred | clarkb, pabelanger, fungi: wow. that's the end of an era | 20:38 |
| pabelanger | ianw: keeping with mirror02, means our cache volumes are more simpler. 1 vs 2 | 20:38 |
| pabelanger | mordred: indeed | 20:38 |
| andreaf | mordred: https://review.openstack.org/#/c/536611 - see my comment | 20:39 |
| *** DinaBelova has quit IRC | 20:39 | |
| *** SergeyLukjanov has quit IRC | 20:39 | |
| *** NikitaKonovalov has quit IRC | 20:39 | |
| *** ruhe has quit IRC | 20:39 | |
| *** freerunner has quit IRC | 20:39 | |
| *** DinaBelova has joined #openstack-infra | 20:40 | |
| *** derekh has quit IRC | 20:41 | |
| *** NikitaKonovalov has joined #openstack-infra | 20:41 | |
| mordred | andreaf: cool. updated | 20:41 |
| *** ruhe has joined #openstack-infra | 20:41 | |
| *** freerunner has joined #openstack-infra | 20:41 | |
| openstackgerrit | Merged openstack-infra/puppet-nodepool master: Add kazoo logging to nodepool https://review.openstack.org/538623 | 20:41 |
| openstackgerrit | Merged openstack-infra/storyboard master: do not set OS_TEST_TIMEOUT in tox.ini https://review.openstack.org/539273 | 20:41 |
| *** SergeyLukjanov has joined #openstack-infra | 20:42 | |
| fungi | ianw: my vote is to just delete mirror01 and move on with our lives | 20:42 |
| *** efried_rollin is now known as efried | 20:42 | |
| ianw | fungi / pabelanger : ok, i can clean that up later today | 20:42 |
| pabelanger | ++ | 20:43 |
| fungi | thanks! | 20:44 |
| *** pcichy has quit IRC | 20:49 | |
| AJaeger | config-core, could you review https://review.openstack.org/538632 and https://review.openstack.org/538631 , please? Those remove some jobs from project-config. | 20:51 |
| *** eharney has quit IRC | 20:54 | |
| *** dougwig has joined #openstack-infra | 20:56 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Change the list of extensions to a dict https://review.openstack.org/539683 | 20:57 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Change the list of extensions to a dict https://review.openstack.org/539683 | 20:58 |
| andreaf | mordred, corvus, tobiash: https://review.openstack.org/539683 switch list of extensions to a dict | 20:59 |
| *** salv-orlando has joined #openstack-infra | 21:01 | |
| *** sree has joined #openstack-infra | 21:02 | |
| AJaeger | andreaf: devstack, tempest, networking-odl are the only users - we should merge your change, update those three, and remove the list support. | 21:02 |
| andreaf | AJaeger: yeah - I'm writing the change for devstack and tempest | 21:03 |
| andreaf | which was the intention to begin with :) | 21:03 |
| AJaeger | andreaf: cool. So, write that with depends-on and then remove from zuul-jobs with depends on these, please | 21:03 |
| *** vivsoni__ has joined #openstack-infra | 21:04 | |
| *** vivsoni_ has quit IRC | 21:04 | |
| *** salv-orlando has quit IRC | 21:06 | |
| *** sree has quit IRC | 21:06 | |
| *** armaan has quit IRC | 21:07 | |
| andreaf | AJaeger: devstack part https://review.openstack.org/539686 | 21:08 |
| andreaf | AJaeger: the Tempest change later or tomorrow morning | 21:08 |
| *** e0ne has quit IRC | 21:12 | |
| *** vivsoni__ has quit IRC | 21:12 | |
| *** vivsoni__ has joined #openstack-infra | 21:12 | |
| *** olaph1 has joined #openstack-infra | 21:17 | |
| *** olaph has quit IRC | 21:17 | |
| *** priteau has joined #openstack-infra | 21:19 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Merger: retry network operations https://review.openstack.org/539356 | 21:20 |
| *** vivsoni__ has quit IRC | 21:22 | |
| fungi | #status log filed for removal of ask.openstack.org from mailspike blacklist per https://launchpad.net/bugs/1745512 | 21:22 |
| openstack | Launchpad bug 1745512 in OpenStack Core Infrastructure "openstack email server blacklisted" [High,Confirmed] - Assigned to OpenStack CI Core (openstack-ci-core) | 21:22 |
| openstackstatus | fungi: finished logging | 21:22 |
| *** vivsoni__ has joined #openstack-infra | 21:22 | |
| clarkb | fungi: so you don't think ask.o.o sending mail as foo@openstack.org is an issue? | 21:23 |
| fungi | clarkb: potentially, yes, unless we want to add it to the spf record for openstack.org | 21:23 |
| clarkb | fungi: ok, am asking because would've thought it would be better to fix that first then get delisted | 21:24 |
| fungi | i have strong doubts that's what caused mailspike to add it to their blacklist | 21:24 |
| clarkb | ah ok | 21:24 |
| fungi | i have yet to turn up a good possible reason, looking at server logs and behavior | 21:25 |
| fungi | the spf record is just a ?all | 21:25 |
| clarkb | ya the @openstack.org was the only thing I could see (though I'm far less versed in debugging email than others) | 21:26 |
| fungi | i agree asking to have it delisted when we don't know why it was listed is marginally risky, but they also don't seem to feel inclined to tell anyone why they're listed | 21:28 |
| *** Swanson has joined #openstack-infra | 21:28 | |
| *** sree has joined #openstack-infra | 21:31 | |
| *** links has quit IRC | 21:32 | |
| *** agopi has joined #openstack-infra | 21:32 | |
| *** sree has quit IRC | 21:35 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: use a semaphore to protect static-publish job output https://review.openstack.org/539693 | 21:40 |
| dhellmann | pabelanger : maybe ^^ is better? | 21:40 |
| *** mamitchl has joined #openstack-infra | 21:43 | |
| corvus | dhellmann: that won't work -- zuul itself has no templating, only ansible | 21:43 |
| pabelanger | dhellmann: I don't think that will work, zuul.yaml doesn't support jnija2 | 21:43 |
| dhellmann | bah | 21:43 |
| corvus | dhellmann: however, as soon as we knock out the release blocking issues, the new pipeline manager for post is first on my list | 21:44 |
| dhellmann | ok, so we can add a semaphore in tree for the release repo static-publish job I guess? | 21:44 |
| *** tpsilva has quit IRC | 21:45 | |
| corvus | dhellmann: should be fine | 21:45 |
| dhellmann | ok | 21:45 |
| *** Goneri has quit IRC | 21:46 | |
| *** kgiusti has left #openstack-infra | 21:47 | |
| *** salv-orlando has joined #openstack-infra | 21:47 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: remove unused variable in test-cleanup.sh https://review.openstack.org/539695 | 21:49 |
| fungi | clarkb: the spf discussion also spilled over into https://launchpad.net/bugs/1721803 | 21:53 |
| openstack | Launchpad bug 1721803 in OpenStack Community Project "Not able to log in using google/launchpad account - confirmation email is not received" [Critical,Confirmed] - Assigned to Jeremy Stanley (fungi) | 21:53 |
| fungi | clarkb: but in short, ?all should just mean that ask.o.o sending mail as communitymngr@o.o gets treated as if the o.o spf record did not exist | 21:54 |
| fungi | in practice it only means that to mta operators who correctly follow the spf spec, of course | 21:55 |
| fungi | so i would be fairly surprised if doing that would have caused it to get onto a blacklist | 21:56 |
| dhellmann | corvus : so, about the templating limitation in the job file for zuul, is that something you'd like changed? or is it not possible for some reason? | 21:56 |
| corvus | dhellmann: i don't think we want it changed. much of zuulv3's language was designed to avoid that, which can get pretty confusing. | 21:57 |
| dhellmann | ok | 21:57 |
| corvus | dhellmann: of course, if you find places where you want to use it, let me know, and i'll be happy to work out whether there's just a non-obvious way to do it, or if we need something new added to the language (or program) | 21:58 |
| *** hrubi_ has joined #openstack-infra | 21:58 | |
| clarkb | fungi: I'm remembering that gmail refuses to accept mail over ipv6 but I thought we had a global exim config to force gmail to go via ipv4 so that is likely not the issue | 21:58 |
| clarkb | fungi: but may be worth double checking? | 21:58 |
| dhellmann | corvus : sure. I mean, that semaphore name seemed like a good example of a place. :-) | 21:58 |
| *** armaan has joined #openstack-infra | 21:59 | |
| corvus | dhellmann: (like in this case, i think the new pipeline manager will solve the issue, and provide other benefits as well. it won't take long to implement. i'm just being very strict about following our roadmap. | 21:59 |
| dhellmann | sure | 21:59 |
| dhellmann | and I guess in most cases where we want to dynamically generate a name or something we're passing values in to playbooks and they can do it | 22:00 |
| corvus | (the other benefits are that it will reduce node usage, so the post pipelines should actually move faster) | 22:00 |
| dhellmann | oh, nice | 22:00 |
| fungi | clarkb: well, the gmail.com completions and rejections both have ipv4 addresses in the logs, though i _do_ see us talking ipv6 to qq.com mtas | 22:00 |
| *** trown is now known as trown|outtypewww | 22:00 | |
| *** priteau has quit IRC | 22:00 | |
| *** vivsoni__ has quit IRC | 22:01 | |
| *** hrubi has quit IRC | 22:01 | |
| *** priteau has joined #openstack-infra | 22:01 | |
| *** vivsoni__ has joined #openstack-infra | 22:01 | |
| corvus | mordred, fungi: can you look at https://review.openstack.org/538345 ? | 22:01 |
| corvus | that's needed for https://review.openstack.org/538039 | 22:02 |
| *** priteau has quit IRC | 22:03 | |
| *** priteau has joined #openstack-infra | 22:03 | |
| *** iyamahat_ has quit IRC | 22:04 | |
| *** yamahata has quit IRC | 22:04 | |
| *** olaph has joined #openstack-infra | 22:05 | |
| *** olaph1 has quit IRC | 22:06 | |
| *** myoung is now known as myoung|off | 22:09 | |
| clarkb | fungi: do you know if we got to the bottom of the inap server thing? | 22:09 |
| *** HeOS has quit IRC | 22:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Support the fragment form of Gerrit URLs https://review.openstack.org/539705 | 22:13 |
| *** SumitNaiksatam has quit IRC | 22:13 | |
| fungi | clarkb: we took it as far as we could, but i certainly don't have any more information to pass along to mgagne than the last time he looked into one of these on january 5 | 22:15 |
| *** bfournie has quit IRC | 22:15 | |
| *** sree has joined #openstack-infra | 22:15 | |
| *** HeOS has joined #openstack-infra | 22:17 | |
| fungi | the only real takeaway i have is that the persistent ssh socket model doesn't fare well in the face of a remote system which keeps the socket open but ceases responding to commands | 22:18 |
| *** eharney has joined #openstack-infra | 22:20 | |
| fungi | if we opened a new ssh socket for every task (or even for every playbook) the builds encountering this situation would fail out fairly quickly after timing out the first playbook which straddled that event | 22:20 |
| *** sree has quit IRC | 22:20 | |
| clarkb | fungi: from following along earlier the nova console seemed to indicate that maybe the hypervisor was unhappy? | 22:20 |
| fungi | but the tradeoff is probably higher resource utilization and more failures to open new ssh sockets because of intermittent network issues | 22:20 |
| clarkb | but we'd need mgagne to look into that | 22:20 |
| fungi | clarkb: yeah, it's the same information i got from the nova console last time too | 22:21 |
| fungi | and he poked around on the hypervisor host and gave it a clean bill of health | 22:21 |
| clarkb | ah ok | 22:21 |
| fungi | i don't know whether he checked again this time, but i have no good reason to believe it would necessarily be any different | 22:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add available RAM to statsd https://review.openstack.org/539707 | 22:22 |
| mgagne | fungi: I checked the mentioned instance earlier, nothing related to filesystem or hypervisor | 22:22 |
| mgagne | if you could find a pattern (maybe a similar hostId in Nova?), I can take a look. So far, our monitoring reports nothing abnormal. | 22:22 |
| fungi | thanks mgagne | 22:22 |
| fungi | it's certainly a mystery | 22:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Create a parent job to publish to the zuul website https://review.openstack.org/538345 | 22:23 |
| *** salv-orl_ has joined #openstack-infra | 22:23 | |
| fungi | we can start trying to record a detailed server show of the impacted instance when we notice these in the future and try to correlate that i suppose | 22:23 |
| andreaf | AJaeger: the change on Tempest side https://review.openstack.org/539708 | 22:24 |
| mgagne | fungi: if you have UUID of past instances with problem, I could dig in database to see if there is a pattern | 22:24 |
| *** salv-orlando has quit IRC | 22:25 | |
| mgagne | but I'm a bit tight on time budget so anything precooked when it comes to info would be appreciated | 22:26 |
| fungi | mgagne: mgagne according to http://paste.openstack.org/show/658211 it was cf197071-c825-4771-b824-0d2eedb9b9ab | 22:27 |
| fungi | back on january 5 | 22:27 |
| *** hjensas has quit IRC | 22:28 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove old clouds from nodepool-builder logging https://review.openstack.org/539670 | 22:30 |
| openstackgerrit | Merged openstack-infra/storyboard master: only try to reset the db user password if it needs to be https://review.openstack.org/539274 | 22:31 |
| mgagne | fungi: that would be hostId 32d82360d8c2620889e619bc1082dfb78c1077cd4811c64bce9bf3e5 | 22:31 |
| fungi | mgagne: same hostid as e21e4d48-79fa-47bd-956d-077abb8e8bba from today? | 22:32 |
| *** edmondsw has quit IRC | 22:32 | |
| *** edmondsw has joined #openstack-infra | 22:33 | |
| mgagne | that one was d50b9618e3e33c1d0b08ebe16ca501226e1ee405d838e1b69a79c600 | 22:33 |
| fungi | okay, so not a match. rules that out at least. thanks! | 22:34 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove tripleo-test-cloud-rh(1|2) support https://review.openstack.org/539671 | 22:34 |
| mgagne | fungi: I'm taking note of the instance UUID and hostId, lets see if I can come up with a pattern when I get more instance UUIDs | 22:35 |
| fungi | thanks again, as always! | 22:36 |
| mgagne | sorry about the inconveniences ;) /canada | 22:36 |
| clarkb | I read that in the correct accent | 22:37 |
| *** edmondsw has quit IRC | 22:37 | |
| fungi | i'm going to disappear and find food... bbiaw | 22:40 |
| *** salv-orl_ has quit IRC | 22:42 | |
| *** salv-orlando has joined #openstack-infra | 22:43 | |
| *** felipemonteiro_ has quit IRC | 22:43 | |
| *** kjackal has quit IRC | 22:43 | |
| *** felipemonteiro_ has joined #openstack-infra | 22:43 | |
| *** priteau has quit IRC | 22:45 | |
| *** sree has joined #openstack-infra | 22:46 | |
| *** salv-orlando has quit IRC | 22:47 | |
| *** sree has quit IRC | 22:50 | |
| *** dprince has quit IRC | 22:52 | |
| *** jcoufal_ has quit IRC | 22:52 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix statsd documentation about events https://review.openstack.org/539482 | 22:53 |
| *** sree has joined #openstack-infra | 22:53 | |
| efried | mriedem: It's possible for the placement service to be running a *later* version of code than the compute service, right? | 22:54 |
| *** andreas_s has joined #openstack-infra | 22:54 | |
| mriedem | fo sho | 22:55 |
| adriant | tosky: fix for your openstacksdk bug is up for review. | 22:55 |
| tosky | adriant: oh, thanks; I will recheck it tomorrow | 22:55 |
| adriant | tosky: I've added you to it to check. | 22:55 |
| tosky | I see, thanks | 22:56 |
| *** sree has quit IRC | 22:58 | |
| *** andreas_s has quit IRC | 22:58 | |
| efried | mriedem: Good, because if you had answered no, I wouldn't have been able to open https://bugs.launchpad.net/nova/+bug/1746615 | 22:59 |
| openstack | Launchpad bug 1746615 in OpenStack Compute (nova) "Report client can try to ensure a standard resource class" [Undecided,New] | 22:59 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix lazy initialization of GithubUser with apps https://review.openstack.org/539545 | 23:00 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Update Development & Migration Docs https://review.openstack.org/539723 | 23:01 |
| *** hemna_ has quit IRC | 23:02 | |
| mriedem | efried: is there a reason this is an infra issue? | 23:02 |
| mriedem | or are you lost again? | 23:02 |
| efried | ahjeez | 23:03 |
| efried | Infra people, please ignore my nova-specific dialog with mriedem over the past ten minutes. | 23:03 |
| *** jamesmcarthur has joined #openstack-infra | 23:04 | |
| *** bfournie has joined #openstack-infra | 23:04 | |
| *** slaweq has joined #openstack-infra | 23:04 | |
| *** bfournie has quit IRC | 23:04 | |
| *** rosmaita has quit IRC | 23:04 | |
| *** bfournie has joined #openstack-infra | 23:05 | |
| *** esberglu has quit IRC | 23:07 | |
| *** rosmaita has joined #openstack-infra | 23:08 | |
| *** felipemonteiro__ has joined #openstack-infra | 23:08 | |
| *** slaweq has quit IRC | 23:09 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** felipemonteiro_ has quit IRC | 23:12 | |
| *** slaweq has quit IRC | 23:15 | |
| *** hjensas has joined #openstack-infra | 23:15 | |
| *** hjensas has joined #openstack-infra | 23:15 | |
| tonyb | clarkb, ianw, fungi: Can any of you spare a few minutes to run the eol_branch script as covered in http://lists.openstack.org/pipermail/openstack-dev/2018-January/126776.html ? | 23:16 |
| tonyb | (or add me to bootstrappers so I can do it) | 23:16 |
| clarkb | I want to say we added you to the bootstrappers group last time. I'm happy to do that again if no one objects | 23:16 |
| ianw | yep, that worked well last time | 23:17 |
| ianw | clarkb / tonyb : happy to be the swizzler ... tonyb you want to do it now or later? | 23:18 |
| tonyb | As it's only 2 repos the time commitment for the group chnage or doign the EOL is about the same | 23:18 |
| tonyb | ianw, clarkb whenever works for y'all | 23:18 |
| openstackgerrit | Merged openstack-infra/storyboard master: add a script to clean up test databases https://review.openstack.org/539275 | 23:18 |
| *** sree has joined #openstack-infra | 23:19 | |
| ianw | tonyb: ok, added you | 23:20 |
| *** d0ugal has quit IRC | 23:20 | |
| tonyb | ianw: Thanks. | 23:20 |
| *** iyamahat has joined #openstack-infra | 23:20 | |
| *** sree has quit IRC | 23:23 | |
| openstackgerrit | Merged openstack-infra/storyboard master: log database management actions in tests https://review.openstack.org/539276 | 23:25 |
| *** rcernin has joined #openstack-infra | 23:28 | |
| *** ldnunes has quit IRC | 23:29 | |
| *** ldnunes has joined #openstack-infra | 23:30 | |
| openstackgerrit | James E. Blair proposed openstack-infra/irc-meetings master: Zuul: Remove project name https://review.openstack.org/539728 | 23:31 |
| tonyb | ianw, clarkb : Thanks, I've done EOL'd newton in nova and ironic and removed myself from bootstrappers | 23:33 |
| *** rlandy is now known as rlandy|bbl | 23:35 | |
| openstackgerrit | Merged openstack-infra/storyboard master: convert list of auth_codes to single value https://review.openstack.org/539331 | 23:36 |
| *** yamahata has joined #openstack-infra | 23:36 | |
| clarkb | corvus: http://paste.openstack.org/show/658253/ any idea of what happened there? | 23:37 |
| clarkb | job apparently hit retry limit but ansible looks happy there other than the timeout? | 23:37 |
| clarkb | also was in inap | 23:37 |
| clarkb | corvus: that was on ze10 | 23:39 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Choose appropriate bootloader for block-device https://review.openstack.org/539731 | 23:39 |
| clarkb | 2018-01-31 23:02:51,185 WARNING zuul.AnsibleJob: [build: 535d69aa778c42d0a269975b9851a6d0] Ansible timeout exceeded | 23:39 |
| clarkb | ok thats the actual error I think | 23:40 |
| clarkb | looks like maybe we give it a minute to gather facts? | 23:40 |
| clarkb | is that long enough? | 23:40 |
| clarkb | mordred: ^ | 23:40 |
| clarkb | reading the logs it does appear that we log the fact results relatively slowly | 23:42 |
| clarkb | couple lines a second | 23:42 |
| *** salv-orlando has joined #openstack-infra | 23:43 | |
| clarkb | mgagne: 81249782-f315-47f6-9cbc-22498c834bd3 and 6759558a-17c5-4151-b0d8-2ee7a4f8b69d were the affected VM instances if you want to add that to your data recording | 23:44 |
| *** sree has joined #openstack-infra | 23:44 | |
| mgagne | clarkb: thanks, those were on the same node: c1f4a1dcd147205818eb25e6d2a4c7c9139610e727535340201d47f4 | 23:46 |
| mgagne | which is different from the other 2 instances earlier | 23:47 |
| *** salv-orlando has quit IRC | 23:47 | |
| mgagne | clarkb: we did not update anything major in the past days on those nodes so I'm suspecting something in the VM like the kernel version | 23:48 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Add block-device defaults https://review.openstack.org/539375 | 23:48 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Choose appropriate bootloader for block-device https://review.openstack.org/539731 | 23:48 |
| *** armax has quit IRC | 23:49 | |
| *** sree has quit IRC | 23:49 | |
| *** rcernin has quit IRC | 23:50 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Add block-device defaults https://review.openstack.org/539375 | 23:50 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] Choose appropriate bootloader for block-device https://review.openstack.org/539731 | 23:50 |
| openstackgerrit | Merged openstack-dev/pbr master: future-proof invocation of apidoc https://review.openstack.org/534890 | 23:55 |
| *** jlabarre has quit IRC | 23:55 | |
| clarkb | mgagne: ya possibly a VM issue | 23:56 |
| clarkb | I guess next step may be to check if we see it happenign on more than one of our image types | 23:56 |
| *** bcafarel has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!