| openstackgerrit | Merged openstack/project-config master: Remove puppet 3 jobs from puppet-midonet https://review.opendev.org/659912 | 00:02 |
|---|---|---|
| *** yamamoto has joined #openstack-infra | 00:02 | |
| openstackgerrit | Merged opendev/puppet-askbot master: Move to apache 2.4 access always https://review.opendev.org/659436 | 00:15 |
| openstackgerrit | Merged opendev/puppet-askbot master: Install redis 2.10.3 https://review.opendev.org/659455 | 00:16 |
| *** sreejithp has joined #openstack-infra | 00:20 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove further puppet 3 jobs https://review.opendev.org/659913 | 00:28 |
| *** zhurong has quit IRC | 00:32 | |
| openstackgerrit | Merged openstack/project-config master: Remove fuel puppet 3 jobs https://review.opendev.org/659914 | 00:32 |
| *** zhurong has joined #openstack-infra | 00:44 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove fuel puppet 3 unit jobs https://review.opendev.org/659915 | 00:55 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Fix docs jobs in periodic template https://review.opendev.org/659948 | 00:58 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Revert "Revert "web: upgrade react and react-scripts to ^2.0.0"" https://review.opendev.org/659991 | 00:59 |
| openstackgerrit | Merged openstack/project-config master: Update hacking to version 1.1 https://review.opendev.org/659730 | 01:00 |
| ianw | AJaeger: thanks for that; there was a lot of interlinked jobs and i'm not surprised i missed some :) i think all your fixes merged now | 01:07 |
| *** oyrogerg has joined #openstack-infra | 01:12 | |
| *** uberjay has quit IRC | 01:13 | |
| *** uberjay has joined #openstack-infra | 01:13 | |
| *** yamamoto has quit IRC | 01:13 | |
| *** cburgess has quit IRC | 01:46 | |
| *** sreejithp has quit IRC | 01:49 | |
| *** sreejithp has joined #openstack-infra | 01:49 | |
| *** ykarel has joined #openstack-infra | 02:01 | |
| *** yamamoto has joined #openstack-infra | 02:02 | |
| *** armax has joined #openstack-infra | 02:02 | |
| *** yamamoto has quit IRC | 02:11 | |
| *** ricolin has joined #openstack-infra | 02:14 | |
| *** threestrands has joined #openstack-infra | 02:34 | |
| *** bdodd has quit IRC | 02:35 | |
| *** bdodd has joined #openstack-infra | 02:37 | |
| *** sreejithp_ has joined #openstack-infra | 02:45 | |
| *** sreejithp has quit IRC | 02:46 | |
| *** Lucas_Gray has joined #openstack-infra | 02:53 | |
| *** sreejithp_ has quit IRC | 02:55 | |
| *** zhurong has quit IRC | 03:04 | |
| *** Goneri has joined #openstack-infra | 03:08 | |
| *** whoami-rajat has joined #openstack-infra | 03:11 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: launch.py: Fix inventory list https://review.opendev.org/660008 | 03:12 |
| *** yamamoto has joined #openstack-infra | 03:22 | |
| *** yamamoto has quit IRC | 03:22 | |
| *** yamamoto has joined #openstack-infra | 03:23 | |
| *** psachin has joined #openstack-infra | 03:30 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:35 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add ask01.openstack.org to inventory https://review.opendev.org/660012 | 03:46 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: launch.py : fix typo calling legacy dns print function https://review.opendev.org/660013 | 03:50 |
| *** factor has joined #openstack-infra | 03:55 | |
| *** dave-mccowan has quit IRC | 03:58 | |
| *** udesale has joined #openstack-infra | 04:01 | |
| *** ykarel has quit IRC | 04:04 | |
| *** auristor has quit IRC | 04:10 | |
| *** ykarel has joined #openstack-infra | 04:22 | |
| openstackgerrit | Ian Wienand proposed opendev/puppet-askbot master: Move redis definition https://review.opendev.org/660016 | 04:22 |
| *** janki has joined #openstack-infra | 04:38 | |
| AJaeger | thanks, ianw. The puppet3 stuff is indeed in now. I pushed some more changes, so if you have time for reviewing this week, it would be appreciated... | 04:39 |
| ianw | AJaeger: will do | 04:42 |
| ianw | clarkb: these timeouts getting the gpg key are becoming quite the annoyance :/ | 04:42 |
| *** Lucas_Gray has quit IRC | 04:52 | |
| *** sreejithp has joined #openstack-infra | 04:52 | |
| *** sreejithp has quit IRC | 04:56 | |
| *** psachin has quit IRC | 05:16 | |
| *** auristor has joined #openstack-infra | 05:17 | |
| *** imacdonn has quit IRC | 05:26 | |
| *** e0ne has joined #openstack-infra | 05:31 | |
| *** jbadiapa has joined #openstack-infra | 05:35 | |
| *** lpetrut has joined #openstack-infra | 05:40 | |
| *** imacdonn has joined #openstack-infra | 05:40 | |
| *** udesale has quit IRC | 05:42 | |
| *** udesale has joined #openstack-infra | 05:42 | |
| openstackgerrit | Merged opendev/puppet-askbot master: Move redis definition https://review.opendev.org/660016 | 05:43 |
| *** quiquell has joined #openstack-infra | 05:43 | |
| *** gnuoy has quit IRC | 05:45 | |
| *** chkumar|pto is now known as chandankumar | 05:58 | |
| *** e0ne has quit IRC | 06:02 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Finish moving python-jenkins jobs in-tree https://review.opendev.org/660022 | 06:03 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch some repos to PTI docs jobs https://review.opendev.org/659977 | 06:09 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove broken docs jobs from monasca-agent https://review.opendev.org/659978 | 06:09 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove fixtures-git jobs https://review.opendev.org/660024 | 06:09 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove fixtures-git jobs https://review.opendev.org/660024 | 06:11 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch some repos to PTI docs jobs https://review.opendev.org/659977 | 06:13 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove broken docs jobs from monasca-agent https://review.opendev.org/659978 | 06:13 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove git-upstream jobs https://review.opendev.org/660026 | 06:15 |
| openstackgerrit | Merged zuul/zuul master: Get playbook data from extra_vars https://review.opendev.org/659802 | 06:20 |
| *** pgaxatte has joined #openstack-infra | 06:24 | |
| *** yamamoto has quit IRC | 06:29 | |
| *** Goneri has quit IRC | 06:30 | |
| *** yamamoto has joined #openstack-infra | 06:31 | |
| *** dpawlik has joined #openstack-infra | 06:38 | |
| *** slaweq has joined #openstack-infra | 06:39 | |
| *** zxiiro has quit IRC | 06:47 | |
| *** coreycb has quit IRC | 06:47 | |
| *** jbryce has quit IRC | 06:47 | |
| *** rpioso has quit IRC | 06:47 | |
| *** TheJulia has quit IRC | 06:47 | |
| *** jamespage has quit IRC | 06:47 | |
| *** vdrok has quit IRC | 06:47 | |
| *** seyeongkim has quit IRC | 06:47 | |
| *** icey has quit IRC | 06:47 | |
| *** Shrews has quit IRC | 06:47 | |
| *** icey has joined #openstack-infra | 06:47 | |
| *** TheJulia has joined #openstack-infra | 06:47 | |
| *** nickv1985 has quit IRC | 06:47 | |
| *** rpioso has joined #openstack-infra | 06:47 | |
| *** jamespage has joined #openstack-infra | 06:47 | |
| *** seyeongkim has joined #openstack-infra | 06:47 | |
| *** vdrok has joined #openstack-infra | 06:47 | |
| *** jbryce has joined #openstack-infra | 06:47 | |
| *** Shrews has joined #openstack-infra | 06:47 | |
| *** zaro has quit IRC | 06:47 | |
| *** seongsoocho has quit IRC | 06:47 | |
| *** clayg has quit IRC | 06:47 | |
| *** MarkMaglana has quit IRC | 06:47 | |
| *** dirk--old has quit IRC | 06:47 | |
| *** nickv1985 has joined #openstack-infra | 06:47 | |
| *** coreycb has joined #openstack-infra | 06:47 | |
| *** seongsoocho has joined #openstack-infra | 06:47 | |
| *** clayg has joined #openstack-infra | 06:47 | |
| *** abelur has quit IRC | 06:48 | |
| *** mwhahaha has quit IRC | 06:48 | |
| *** geekinutah has quit IRC | 06:48 | |
| *** johnsom has quit IRC | 06:48 | |
| *** guilhermesp has quit IRC | 06:48 | |
| *** zaro has joined #openstack-infra | 06:48 | |
| *** MarkMaglana has joined #openstack-infra | 06:48 | |
| *** sadin999 has quit IRC | 06:48 | |
| *** adriancz has quit IRC | 06:48 | |
| *** mnasiadka has quit IRC | 06:48 | |
| *** ajo has quit IRC | 06:48 | |
| *** adrianreza has quit IRC | 06:48 | |
| *** guilhermesp has joined #openstack-infra | 06:48 | |
| *** geekinutah has joined #openstack-infra | 06:48 | |
| *** mwhahaha has joined #openstack-infra | 06:48 | |
| *** johnsom has joined #openstack-infra | 06:48 | |
| *** sadin999 has joined #openstack-infra | 06:48 | |
| *** abelur has joined #openstack-infra | 06:48 | |
| *** mnasiadka has joined #openstack-infra | 06:48 | |
| *** zxiiro has joined #openstack-infra | 06:48 | |
| *** ajo has joined #openstack-infra | 06:48 | |
| *** adriancz has joined #openstack-infra | 06:48 | |
| *** adrianreza has joined #openstack-infra | 06:48 | |
| *** mandre has quit IRC | 06:51 | |
| *** dciabrin has joined #openstack-infra | 06:57 | |
| *** ralonsoh has joined #openstack-infra | 06:57 | |
| *** udesale has quit IRC | 07:00 | |
| *** udesale has joined #openstack-infra | 07:00 | |
| *** lpetrut has quit IRC | 07:04 | |
| *** zigo has joined #openstack-infra | 07:08 | |
| *** rcernin has quit IRC | 07:08 | |
| *** yolanda_ has quit IRC | 07:13 | |
| *** tesseract has joined #openstack-infra | 07:13 | |
| *** rpittau|afk is now known as rpittau | 07:18 | |
| ianw | #status log removed ask-staging-old and ask-staging01 servers, and all related dns entries (see https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup) | 07:20 |
| openstackstatus | ianw: finished logging | 07:20 |
| *** yolanda_ has joined #openstack-infra | 07:20 | |
| *** udesale has quit IRC | 07:20 | |
| *** udesale has joined #openstack-infra | 07:21 | |
| AJaeger | config-core, could you review https://review.opendev.org/660026 , https://review.opendev.org/660024 and https://review.opendev.org/660022 , please? | 07:21 |
| *** threestrands has quit IRC | 07:23 | |
| *** roman_g has joined #openstack-infra | 07:23 | |
| *** pcaruana has joined #openstack-infra | 07:23 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove ask-staging* from disabled list https://review.opendev.org/659421 | 07:25 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: ask.o.o : workaround old puppet-solr package https://review.opendev.org/659473 | 07:25 |
| *** udesale has quit IRC | 07:26 | |
| *** udesale has joined #openstack-infra | 07:27 | |
| *** bhavikdbavishi has quit IRC | 07:30 | |
| *** amoralej|off is now known as amoralej | 07:32 | |
| openstackgerrit | Merged opendev/system-config master: Remove unreachable hosts from inventory https://review.opendev.org/657820 | 07:34 |
| *** ykarel is now known as ykarel|lunch | 07:35 | |
| *** dtantsur|afk is now known as dtantsur | 07:35 | |
| *** tosky has joined #openstack-infra | 07:35 | |
| *** yboaron_ has quit IRC | 07:36 | |
| *** psachin has joined #openstack-infra | 07:43 | |
| *** xek_ has joined #openstack-infra | 07:44 | |
| *** iurygregory has joined #openstack-infra | 07:45 | |
| *** lucasagomes has joined #openstack-infra | 07:47 | |
| *** e0ne has joined #openstack-infra | 07:48 | |
| *** ginopc has joined #openstack-infra | 07:49 | |
| *** jpena|off is now known as jpena | 07:52 | |
| *** sreejithp has joined #openstack-infra | 07:52 | |
| *** sreejithp has quit IRC | 07:57 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add ask01.openstack.org to inventory https://review.opendev.org/660012 | 07:57 |
| *** shardy has joined #openstack-infra | 07:58 | |
| *** dciabrin has quit IRC | 08:07 | |
| *** dciabrin_ has joined #openstack-infra | 08:07 | |
| *** pkopec has joined #openstack-infra | 08:10 | |
| *** Lucas_Gray has joined #openstack-infra | 08:12 | |
| *** tkajinam has quit IRC | 08:17 | |
| *** Lucas_Gray has quit IRC | 08:17 | |
| *** shardy has quit IRC | 08:18 | |
| *** Lucas_Gray has joined #openstack-infra | 08:20 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:21 | |
| *** dciabrin__ has joined #openstack-infra | 08:23 | |
| *** dciabrin_ has quit IRC | 08:26 | |
| *** dciabrin__ has quit IRC | 08:26 | |
| *** ykarel|lunch is now known as ykarel | 08:31 | |
| *** oyrogerg has quit IRC | 08:37 | |
| *** derekh has joined #openstack-infra | 08:37 | |
| *** ramishra has joined #openstack-infra | 08:38 | |
| *** panda|rover|off is now known as panda|rover | 08:41 | |
| *** yboaron_ has joined #openstack-infra | 08:42 | |
| *** Lucas_Gray has quit IRC | 09:00 | |
| *** Lucas_Gray has joined #openstack-infra | 09:00 | |
| *** Lucas_Gray has quit IRC | 09:08 | |
| *** dciabrin has joined #openstack-infra | 09:12 | |
| *** dciabrin has quit IRC | 09:16 | |
| *** dciabrin has joined #openstack-infra | 09:16 | |
| *** yboaron_ has quit IRC | 09:17 | |
| *** ricolin has quit IRC | 09:19 | |
| *** lpetrut has joined #openstack-infra | 09:30 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A reporter for Elasticsearch with the capability to index build and buildset results in an index. https://review.opendev.org/644927 | 09:32 |
| *** ccamacho has quit IRC | 09:33 | |
| *** ccamacho has joined #openstack-infra | 09:34 | |
| corvus | this is so weird. i'm used to feeling immediately behind when i start work on monday morning... but now, i get to be the reason that clarkb feels behind when he starts! :) | 09:35 |
| corvus | mordred, fungi: 656871 is approved :) | 09:36 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A reporter for Elasticsearch https://review.opendev.org/644927 | 09:38 |
| openstackgerrit | Merged zuul/zuul-website master: Add slides for Spring 2019 Board Update https://review.opendev.org/656182 | 09:42 |
| *** gfidente has joined #openstack-infra | 09:45 | |
| ianw | mordred: fyi https://review.opendev.org/#/c/660008/ is a fix for launch.py ... the follow is also a fix but it doesn't actually stop launch | 09:45 |
| *** dciabrin_ has joined #openstack-infra | 09:49 | |
| *** yamamoto has quit IRC | 09:50 | |
| *** rossella_s has quit IRC | 09:51 | |
| *** dciabrin has quit IRC | 09:52 | |
| *** pgaxatte has quit IRC | 09:57 | |
| paladox | corvus: lol | 09:59 |
| *** dciabrin_ has quit IRC | 09:59 | |
| *** dciabrin has joined #openstack-infra | 09:59 | |
| *** rcernin has joined #openstack-infra | 10:03 | |
| openstackgerrit | Merged opendev/system-config master: Split the base playbook into services https://review.opendev.org/656871 | 10:04 |
| corvus | infra-root: ^ | 10:10 |
| *** dciabrin has quit IRC | 10:14 | |
| *** ccamacho has quit IRC | 10:22 | |
| *** ccamacho has joined #openstack-infra | 10:26 | |
| *** kjackal has joined #openstack-infra | 10:29 | |
| *** yikun has quit IRC | 10:32 | |
| *** flaper87 has quit IRC | 10:42 | |
| *** ramishra has quit IRC | 10:43 | |
| *** panda|rover is now known as panda|lunch | 10:52 | |
| *** panda|lunch is now known as panda|rover|eat | 10:53 | |
| *** rcernin has quit IRC | 10:57 | |
| *** flaper87 has joined #openstack-infra | 11:00 | |
| *** ykarel is now known as ykarel|afk | 11:00 | |
| *** yboaron_ has joined #openstack-infra | 11:02 | |
| *** yolanda_ has quit IRC | 11:04 | |
| *** yboaron_ has quit IRC | 11:08 | |
| *** yolanda_ has joined #openstack-infra | 11:08 | |
| *** yboaron_ has joined #openstack-infra | 11:08 | |
| corvus | that's running now (after a 1 cycle delay since it's an update to the outer script) | 11:10 |
| *** jamesdenton has joined #openstack-infra | 11:11 | |
| *** udesale has quit IRC | 11:16 | |
| *** jpena is now known as jpena|lunch | 11:25 | |
| *** Lucas_Gray has joined #openstack-infra | 11:32 | |
| *** kopecmartin has joined #openstack-infra | 11:42 | |
| *** kopecmartin is now known as kopecmartin|off | 11:43 | |
| *** rh-jelabarre has joined #openstack-infra | 11:48 | |
| openstackgerrit | Markus Hosch proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 11:51 |
| *** bobh has joined #openstack-infra | 11:51 | |
| *** amoralej is now known as amoralej|lunch | 11:53 | |
| *** Lucas_Gray has quit IRC | 11:55 | |
| *** Lucas_Gray has joined #openstack-infra | 11:57 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Keep task stdout/stderr separate in result object https://review.opendev.org/650276 | 11:57 |
| *** panda|rover|eat is now known as panda|rover | 11:57 | |
| *** yamamoto has joined #openstack-infra | 11:58 | |
| *** yamamoto has quit IRC | 11:58 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Keep task stdout/stderr separate in result object https://review.opendev.org/650276 | 11:59 |
| *** kjackal has quit IRC | 12:01 | |
| *** dave-mccowan has joined #openstack-infra | 12:04 | |
| *** ykarel|afk is now known as ykarel | 12:05 | |
| *** bobh has quit IRC | 12:10 | |
| *** yboaron_ has quit IRC | 12:11 | |
| *** bobh has joined #openstack-infra | 12:12 | |
| *** ykarel_ has joined #openstack-infra | 12:16 | |
| *** bobh has quit IRC | 12:16 | |
| openstackgerrit | Christian Fetzer proposed zuul/zuul master: pass-to-parent: Fix passing multiple secrets to parent https://review.opendev.org/660105 | 12:16 |
| *** ykarel has quit IRC | 12:18 | |
| *** bhavikdbavishi has quit IRC | 12:20 | |
| *** jpena|lunch is now known as jpena | 12:22 | |
| *** ab-a has joined #openstack-infra | 12:23 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:23 | |
| *** amoralej|lunch is now known as amoralej | 12:25 | |
| *** derekh has quit IRC | 12:25 | |
| *** rlandy has joined #openstack-infra | 12:27 | |
| *** ianychoi has quit IRC | 12:28 | |
| *** edsousa has joined #openstack-infra | 12:29 | |
| *** ianychoi has joined #openstack-infra | 12:29 | |
| *** janki has quit IRC | 12:29 | |
| *** rossella_s has joined #openstack-infra | 12:32 | |
| *** rosmaita has joined #openstack-infra | 12:34 | |
| *** dciabrin has joined #openstack-infra | 12:35 | |
| *** dciabrin_ has joined #openstack-infra | 12:39 | |
| *** dciabrin has quit IRC | 12:43 | |
| *** ccamacho has quit IRC | 12:45 | |
| *** dave-mccowan has quit IRC | 12:48 | |
| *** mriedem has joined #openstack-infra | 12:51 | |
| *** rfarr has joined #openstack-infra | 12:51 | |
| *** factor has quit IRC | 12:54 | |
| *** factor has joined #openstack-infra | 12:54 | |
| fungi | corvus: thanks for the heads up, i'm still only partially caffeinated but will try to keep a closer eye than usual on things | 12:55 |
| *** aaronsheffield has joined #openstack-infra | 13:01 | |
| *** udesale has joined #openstack-infra | 13:01 | |
| *** derekh has joined #openstack-infra | 13:04 | |
| *** Guest84310 has joined #openstack-infra | 13:05 | |
| *** Guest84310 is now known as redrobot | 13:06 | |
| *** janki has joined #openstack-infra | 13:08 | |
| *** bhavikdbavishi has quit IRC | 13:13 | |
| *** bhavikdbavishi has joined #openstack-infra | 13:14 | |
| *** ykarel__ has joined #openstack-infra | 13:17 | |
| *** ykarel_ has quit IRC | 13:19 | |
| *** psachin has quit IRC | 13:20 | |
| *** yboaron_ has joined #openstack-infra | 13:22 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: add-build-sshkey: remove previously authorized build-sshkey https://review.opendev.org/632620 | 13:22 |
| *** jroll has quit IRC | 13:25 | |
| *** jroll has joined #openstack-infra | 13:26 | |
| *** sreejithp has joined #openstack-infra | 13:26 | |
| *** bhavikdbavishi has quit IRC | 13:32 | |
| *** pcaruana has quit IRC | 13:33 | |
| *** ykarel__ is now known as ykarel | 13:33 | |
| openstackgerrit | Christian Fetzer proposed zuul/zuul master: pass-to-parent: Fix passing multiple secrets to parent https://review.opendev.org/660105 | 13:34 |
| openstackgerrit | Flavio Percoco proposed zuul/zuul master: Fix typo in log message https://review.opendev.org/660118 | 13:43 |
| *** lseki has joined #openstack-infra | 13:45 | |
| *** eharney has joined #openstack-infra | 13:46 | |
| *** michael-beaver has joined #openstack-infra | 13:46 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Keep task stdout/stderr separate in result object https://review.opendev.org/650276 | 13:46 |
| *** lpetrut has quit IRC | 13:53 | |
| *** kjackal has joined #openstack-infra | 14:04 | |
| *** janki has quit IRC | 14:10 | |
| openstackgerrit | Dirk Mueller proposed opendev/system-config master: Properly mirror the lxc container images as well https://review.opendev.org/655895 | 14:11 |
| openstackgerrit | Dirk Mueller proposed opendev/system-config master: Move openSUSE Tumbleweed into a caching mirror instead https://review.opendev.org/660125 | 14:11 |
| openstackgerrit | Dirk Mueller proposed opendev/system-config master: Start mirroring openSUSE Leap 15.1 https://review.opendev.org/660126 | 14:13 |
| *** lpetrut has joined #openstack-infra | 14:15 | |
| *** nicolasbock has joined #openstack-infra | 14:15 | |
| *** dpawlik has quit IRC | 14:19 | |
| *** jamesmcarthur has joined #openstack-infra | 14:22 | |
| openstackgerrit | Dirk Mueller proposed openstack/project-config master: Switch bindep fallback testing to openSUSE Leap 15.x https://review.opendev.org/660130 | 14:23 |
| openstackgerrit | Dirk Mueller proposed openstack/project-config master: Add openSUSE 15.1 to the nodepool building as opensuse-15 https://review.opendev.org/660131 | 14:23 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate node request processing with event id https://review.opendev.org/658889 | 14:25 |
| *** lpetrut has quit IRC | 14:26 | |
| *** jamesmcarthur has quit IRC | 14:29 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate node request processing with event id https://review.opendev.org/658889 | 14:32 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Update test coverage for openSUSE/-minimal to 15.1 https://review.opendev.org/660137 | 14:32 |
| openstackgerrit | Jean Paul Gatt proposed openstack/diskimage-builder master: Add version-less RHEL element for RHEL7 and RHEL8 https://review.opendev.org/643731 | 14:32 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Update test coverage for openSUSE/-minimal to 15.1 https://review.opendev.org/660137 | 14:32 |
| *** jamesmcarthur has joined #openstack-infra | 14:37 | |
| *** jamesmcarthur_ has joined #openstack-infra | 14:43 | |
| *** pkopec has quit IRC | 14:46 | |
| *** jamesmcarthur has quit IRC | 14:47 | |
| *** pkopec has joined #openstack-infra | 14:52 | |
| corvus | fungi: i looked at the run_all output as it was doing it's first run and didn't see any problems; have you checked it since then? | 14:53 |
| *** mriedem is now known as mriedem_away | 14:53 | |
| *** pcaruana has joined #openstack-infra | 14:53 | |
| fungi | tailing run_all_cron.log right now and seems to be doing the things i expect | 14:54 |
| *** gyee has joined #openstack-infra | 14:57 | |
| fungi | corvus: quick gertty question... i have sync errors related to a queued review on a closed change (either raced a sync or initially happened offline). first, is that expected behavior and second, how do i clear it? seems to want to reattempt it every time i start the application | 14:57 |
| corvus | fungi: hrm, i'll have to look into that, i don't know the answers to either off the top of my head | 14:58 |
| fungi | presumably i could edit the db, but not sure if it's supposed to clear on its own or if there's some way i should be "undoing" the review so it doesn't continue trying | 14:58 |
| fungi | not urgent at all. also this is the traceback from the log in case it helps: http://paste.openstack.org/show/751670/ | 14:59 |
| corvus | my guess is that it would be difficult to "undo" -- but you could try it -- you could try re-reviewing with an empty comment | 14:59 |
| corvus | that may or may not clear out the pending review (which may have had other info such as votes which are no longer relevant) | 14:59 |
| fungi | yeah, the pending review it's trying to apply has votes | 15:01 |
| corvus | i bet we could catch that http response and flag the change as 'held' | 15:01 |
| fungi | re-applying just a comment doesn't seem to do anything | 15:01 |
| corvus | drat | 15:01 |
| fungi | holding the review would be a very user-friendly solution | 15:02 |
| fungi | consistent with the way it uses hold for reviews on changes where conflicting reviews appeared while you were offline | 15:02 |
| *** ykarel is now known as ykarel|away | 15:05 | |
| corvus | the conference wifi is getting spotty, so i'll take that as my cue to exit | 15:08 |
| clarkb | kata is claiming zuul not responding wih sob checks quickly. We didnt restart zuul today andlose events? I wonder if the github api is slow is biting us today | 15:12 |
| mordred | clarkb: I am not aware of zuul restarting - might be github api slowness | 15:13 |
| clarkb | any idea how ansible intendson dealing with this problem? | 15:14 |
| clarkb | they arethe trigger of it on our end | 15:14 |
| *** josephrsandoval has joined #openstack-infra | 15:18 | |
| *** rascasoft has quit IRC | 15:19 | |
| *** jcoufal has joined #openstack-infra | 15:24 | |
| *** rascasoft has joined #openstack-infra | 15:25 | |
| *** pkopec has quit IRC | 15:28 | |
| *** pkopec has joined #openstack-infra | 15:28 | |
| openstackgerrit | Merged zuul/zuul master: Fix typo in log message https://review.opendev.org/660118 | 15:29 |
| *** quiquell has quit IRC | 15:31 | |
| fungi | grafana suggests we're pretty backed up on node requests at the moment | 15:32 |
| fungi | which is likely the cause? | 15:33 |
| fungi | http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1&from=1558280008232&to=1558366408234 | 15:33 |
| *** pkopec is now known as pkopec|afk | 15:34 | |
| clarkb | the sob checks should be executor only | 15:37 |
| clarkb | though if we arethrottling those too may help explain it | 15:37 |
| *** ykarel|away has quit IRC | 15:40 | |
| fungi | gonna pop out for lunch, back as soon as i can be | 15:41 |
| *** pkopec|afk has quit IRC | 15:42 | |
| *** tdasilva has joined #openstack-infra | 15:43 | |
| clarkb | reading zuul logs on github event queue lengths they haven't been too terrible for long periods of time | 15:48 |
| *** slaweq has quit IRC | 15:48 | |
| clarkb | I've asked them for specific PRs so that we can lookup specific rather than hunting in the dark | 15:51 |
| *** dpawlik has joined #openstack-infra | 15:55 | |
| *** ricolin has joined #openstack-infra | 15:55 | |
| *** josephrsandoval has quit IRC | 15:58 | |
| *** kskels has joined #openstack-infra | 16:00 | |
| *** chandankumar is now known as raukadah | 16:01 | |
| *** ykarel|away has joined #openstack-infra | 16:03 | |
| clarkb | I'm beginning to suspect something with either that change or their zuul config or both | 16:04 |
| clarkb | I did a 'zuul-recheck' comment which should've enqueued dco-license if I'm reading their configs properly but doesn't appear to have done so | 16:04 |
| *** mattw4 has joined #openstack-infra | 16:06 | |
| *** josephrsandoval has joined #openstack-infra | 16:06 | |
| *** jcoufal has quit IRC | 16:07 | |
| pabelanger | any expections in scheduler logs? | 16:08 |
| *** ginopc has quit IRC | 16:08 | |
| pabelanger | I've seen zuul miss a recheck, because github api issue | 16:08 |
| clarkb | not that I've seen (thats grepping the log for github then looking for tracebacks) | 16:08 |
| clarkb | (there is one but for a different repo) | 16:09 |
| *** rpittau is now known as rpittau|afk | 16:10 | |
| AJaeger | config-core, please review https://review.opendev.org/660024 and https://review.opendev.org/660026 and https://review.opendev.org/659978 | 16:11 |
| *** Lucas_Gray has quit IRC | 16:12 | |
| *** josephrsandoval has quit IRC | 16:19 | |
| clarkb | heh I see my error now | 16:20 |
| clarkb | its /zuul-recheck | 16:20 |
| clarkb | because of course there must be a special fancy regex as the string isn't unique enough already | 16:20 |
| *** mattw4 has quit IRC | 16:20 | |
| *** ricolin has quit IRC | 16:22 | |
| openstackgerrit | Merged openstack/project-config master: Remove fixtures-git jobs https://review.opendev.org/660024 | 16:22 |
| clarkb | job is running now | 16:24 |
| AJaeger | thanks, clarkb | 16:24 |
| *** eernst has joined #openstack-infra | 16:27 | |
| openstackgerrit | Merged openstack/project-config master: Remove git-upstream jobs https://review.opendev.org/660026 | 16:29 |
| *** dave-mccowan has joined #openstack-infra | 16:29 | |
| clarkb | ok got kata sorted. I think zuul lost the event a few days back and they just needed to recheck which ahs been done | 16:30 |
| *** lucasagomes has quit IRC | 16:32 | |
| AJaeger | config-core, some more simple cleanups for review, please https://review.opendev.org/659917 https://review.opendev.org/659963 https://review.opendev.org/659974 https://review.opendev.org/660022 | 16:32 |
| clarkb | AJaeger: do we want GBP to ack https://review.opendev.org/#/c/659917/2 prior to merging? | 16:33 |
| clarkb | AJaeger: also https://review.opendev.org/#/c/660022/ 's depends on is unhappy | 16:35 |
| AJaeger | clarkb: 917 NEVER ever worked according to ZUUl database - so, I would not wait... | 16:36 |
| clarkb | k | 16:36 |
| AJaeger | clarkb: missed the python-jenkins failure - thanks. | 16:36 |
| *** udesale has quit IRC | 16:40 | |
| openstackgerrit | Merged openstack/project-config master: Remove obsolete playbook for placement-api-ref https://review.opendev.org/659974 | 16:41 |
| clarkb | as a followup on the sjc1 dns issues, no hits on that e-r bug since we went to ipv4 | 16:41 |
| clarkb | and one hit on the upper-constraints 404 in the last few days (that could be load related maybe) | 16:42 |
| *** slaweq has joined #openstack-infra | 16:46 | |
| clarkb | I'm going to find some tea and breakfast but I think next up on my todo list is deleting the old cgit farm | 16:46 |
| clarkb | infra-root do you think we should get snapshots of those nodes? they are largely stateless other than mirroring the git repo content whcih is now elsewhere | 16:47 |
| openstackgerrit | Merged openstack/project-config master: Remove broken legacy-group-based-policy-dsvm-rally jobs https://review.opendev.org/659917 | 16:48 |
| mordred | clarkb: I do not think it's valuable to do so | 16:48 |
| mordred | I think gitea is solid enough that I find it unlikely we'd go back to cgit - and even if we did we could just rebuild them pretty simply | 16:49 |
| clarkb | mordred: ya, the only outstanding issue I know of right now id the 404 issue with upper constraints that seems to happen once or twice every few days | 16:49 |
| clarkb | mordred: http://status.openstack.org/elastic-recheck/#1829024 | 16:49 |
| clarkb | I tried reproducing and ended up doing like 100k requests for that file without a 404 | 16:49 |
| *** eernst has quit IRC | 16:50 | |
| *** eernst has joined #openstack-infra | 16:54 | |
| mordred | clarkb: yeah | 16:56 |
| *** ykarel|away has quit IRC | 16:59 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:00 | |
| *** eernst has quit IRC | 17:01 | |
| clarkb | if anyone else needs to disable hyperthreading on intel cpus but has a bios that doesn't support doing that: http://paste.openstack.org/show/751652/ should get you sorted if you run systemd | 17:04 |
| clarkb | surprisingly onle ony of my current machiens actually supports hyperthreading and of course that bios doesn't let me chagne it | 17:05 |
| *** slaweq has quit IRC | 17:07 | |
| *** e0ne has quit IRC | 17:07 | |
| *** amoralej is now known as amoralej|off | 17:07 | |
| *** jpena is now known as jpena|off | 17:07 | |
| *** dtantsur is now known as dtantsur|afk | 17:08 | |
| clarkb | and now reboot to pick up other fixes | 17:10 |
| *** derekh has quit IRC | 17:11 | |
| *** dpawlik has quit IRC | 17:11 | |
| fungi | clarkb: i see no point in snapshotting the cgti nodes... they're easy enough to rebuild from scratch | 17:11 |
| fungi | s/cgti/cgit/ | 17:11 |
| fungi | in fact, restoring one from a snapshot might be more effort than launching a new one | 17:12 |
| *** ykarel|away has joined #openstack-infra | 17:14 | |
| *** mattw4 has joined #openstack-infra | 17:14 | |
| *** efried has quit IRC | 17:19 | |
| *** eernst has joined #openstack-infra | 17:19 | |
| *** efried has joined #openstack-infra | 17:22 | |
| *** lpetrut has joined #openstack-infra | 17:23 | |
| fungi | clarkb: i've adjusted https://review.opendev.org/657174 if you're interested in acknowledging | 17:24 |
| clarkb | fungi: +1 | 17:25 |
| fungi | thanks! | 17:26 |
| *** lpetrut has quit IRC | 17:27 | |
| *** slaweq has joined #openstack-infra | 17:29 | |
| *** ralonsoh has quit IRC | 17:31 | |
| *** jamesmcarthur has joined #openstack-infra | 17:31 | |
| *** jamesmcarthur_ has quit IRC | 17:34 | |
| *** eharney has quit IRC | 17:40 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate node request processing with event id https://review.opendev.org/658889 | 17:40 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id https://review.opendev.org/658895 | 17:40 |
| *** slaweq has quit IRC | 17:43 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Install latest git-review from PyPI in quickstart https://review.opendev.org/659674 | 17:48 |
| *** Weifan has joined #openstack-infra | 17:49 | |
| *** bhavikdbavishi has joined #openstack-infra | 17:50 | |
| *** tesseract has quit IRC | 17:51 | |
| *** raissa has joined #openstack-infra | 17:55 | |
| *** markvoelker has joined #openstack-infra | 17:57 | |
| *** ykarel|away has quit IRC | 18:04 | |
| *** xek_ has quit IRC | 18:06 | |
| *** xek_ has joined #openstack-infra | 18:06 | |
| openstackgerrit | Merged zuul/zuul master: Annotate github logs with the event id https://review.opendev.org/658645 | 18:15 |
| openstackgerrit | Dirk Mueller proposed openstack/project-config master: Handle TOX_CONSTRAINTS_FILE as well as UPPER_CONSTRAINTS_FILE https://review.opendev.org/660187 | 18:17 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Update test coverage for openSUSE/-minimal to 15.1 https://review.opendev.org/660137 | 18:21 |
| *** Weifan has quit IRC | 18:21 | |
| *** xek_ has quit IRC | 18:24 | |
| *** eharney has joined #openstack-infra | 18:27 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: WIP: Convert to using py3 pathlib https://review.opendev.org/660191 | 18:31 |
| clarkb | infra-root http://paste.openstack.org/show/751676/ is the 9 cgit cluster servers on the to be deleted list. Can you take a look at that and make sure I've got the right servers up for deletion? | 18:32 |
| clarkb | also those servers are probably ancient enough (yay centos 7) that if people have things in homedirs they don't want to lose now is a good time to grab stuff (though we shouldn't assume our homedirs are a good place to store stuff anyway) | 18:34 |
| *** kskels has quit IRC | 18:36 | |
| fungi | sudo OS_CLIENT_CONFIG_FILE=/etc/openstack/all-clouds.yaml launch-env/bin/openstack --os-cloud openstackci-rax --os-region-name DFW server list | grep `wget -qO- http://paste.openstack.org/raw/751676/|cut -d' ' -f1|sed 's/^/-e /'` | 18:38 |
| fungi | that returns the servers i expect | 18:38 |
| openstackgerrit | Merged zuul/zuul master: Annotate gerrit event logs https://review.opendev.org/658646 | 18:39 |
| *** raissa has quit IRC | 18:39 | |
| *** raissa has joined #openstack-infra | 18:39 | |
| *** raissa has quit IRC | 18:40 | |
| clarkb | I'll give monty a chance to look but then its off to the deleting | 18:40 |
| *** raissa has joined #openstack-infra | 18:40 | |
| clarkb | also https://review.opendev.org/#/c/658281/ passes testing now (create mirrors with https) | 18:41 |
| clarkb | so we should all give that a review if we get a moment | 18:41 |
| *** raissa has joined #openstack-infra | 18:41 | |
| * clarkb is doing meeting agenda things while giving people time to review git server deletion list | 18:41 | |
| *** raissa has quit IRC | 18:41 | |
| *** raissa has joined #openstack-infra | 18:42 | |
| *** raissa has quit IRC | 18:42 | |
| *** raissa has joined #openstack-infra | 18:43 | |
| *** raissa has quit IRC | 18:43 | |
| *** raissa has joined #openstack-infra | 18:44 | |
| mordred | fungi, clarkb: I concur that the server list is correct | 18:44 |
| *** raissa has quit IRC | 18:44 | |
| clarkb | k I'll proceed with it once I get the infra meeting agenda out in a few | 18:44 |
| *** e0ne has joined #openstack-infra | 18:45 | |
| clarkb | last call for meeting agenda topics | 18:48 |
| fungi | all the topics i can think of right now are properly subtopics of the opendev priority effort anyway | 18:53 |
| clarkb | I've deleted git.o.o first | 18:54 |
| clarkb | as I figure that will shake out any potential problems quickest | 18:54 |
| * fungi waits for screaming | 18:54 | |
| clarkb | fungi: want to add them as explicit subtopics? right now we just have "next steps" | 18:54 |
| fungi | well, i mean, really that's what i want to talk about | 18:56 |
| clarkb | k | 18:56 |
| fungi | where do we go now | 18:56 |
| fungi | obviously there's a bunch to do, so picking priorities to focus on | 18:56 |
| fungi | whichever services those may be | 18:56 |
| *** raissa has joined #openstack-infra | 18:57 | |
| *** e0ne has quit IRC | 18:57 | |
| *** bhavikdbavishi has quit IRC | 19:00 | |
| clarkb | ok meeting agenda sent | 19:02 |
| clarkb | how long do you think we should wait for screaming before deleting git0* | 19:03 |
| clarkb | I expect if we really want to make people happy it would be a day or two so maybe we don't wait too long | 19:03 |
| *** jamesmcarthur has quit IRC | 19:05 | |
| * mordred screams just on general principal | 19:07 | |
| *** eernst has quit IRC | 19:15 | |
| *** raissa has quit IRC | 19:17 | |
| *** raissa has joined #openstack-infra | 19:18 | |
| *** jamesmcarthur has joined #openstack-infra | 19:19 | |
| fungi | i'd just, you know, go for it | 19:24 |
| fungi | no looking back | 19:24 |
| *** derekh has joined #openstack-infra | 19:26 | |
| clarkb | ok I went for it and not looking back | 19:26 |
| clarkb | #status log Deleted git.openstack.org and git01-git08.openstack.org servers. These cgit cluster hosts have been replaced with the gitea cluster. | 19:26 |
| openstackstatus | clarkb: finished logging | 19:26 |
| * fungi applauds | 19:28 | |
| clarkb | I'll clean up dns records next (but leave git.o.o alone in dns) | 19:29 |
| clarkb | the git0x.o.o records can be removed though | 19:29 |
| fungi | sure it's a cname to a redirect now anyway | 19:29 |
| clarkb | infra-root please use the `edit-secrets` command to load up the gpg encrypted file with secrets. It will probably manage the gpg agent | 19:30 |
| fungi | maybe even properly, but no guarantees | 19:32 |
| clarkb | heh typos ftl | 19:33 |
| clarkb | dns has been cleaned up. All 8 A records and all 8 AAAA records for git01-git08 have been deleted | 19:33 |
| fungi | i wonder if we don't also need to `gpg-agent --daemon ...` when viewing that file too | 19:34 |
| clarkb | alright that leaves the next items on the git server cleanup rebuilding gitea06 (and possibly all gitea servers with more disk and otherwise smaller overall) and upgrading gitea itself | 19:34 |
| * mordred does a dance and hands clarkb an ostrich of happiness | 19:35 | |
| clarkb | fungi: yes gpg-agent --daemon emacs $filename is what it runs | 19:35 |
| fungi | clarkb: i mean in addition to editing the file, maybe we also need to do the same when merely decrypting/viewing the file? | 19:36 |
| clarkb | yup, I just use the one command for all of it (since emacs will decrypt in memory only and then I can choose to edit or not) | 19:36 |
| clarkb | the problem is anytime the key is used we get a runaway agent | 19:37 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Update test coverage for openSUSE/-minimal to 15.1 https://review.opendev.org/660137 | 19:37 |
| *** markvoelker has quit IRC | 19:41 | |
| *** slaweq has joined #openstack-infra | 19:42 | |
| *** toabctl has quit IRC | 19:43 | |
| * clarkb makes a sandwich for lunch then will be back to review the https mirrors change | 19:44 | |
| *** toabctl has joined #openstack-infra | 19:46 | |
| *** dmsimard5 has joined #openstack-infra | 19:50 | |
| *** dmsimard has quit IRC | 19:51 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Use py3 pathlib in DibImageFile https://review.opendev.org/660191 | 20:07 |
| openstackgerrit | Merged zuul/zuul master: Attach event to queue item https://review.opendev.org/658647 | 20:17 |
| *** whoami-rajat has quit IRC | 20:19 | |
| *** Lucas_Gray has joined #openstack-infra | 20:28 | |
| fungi | i'll adjust the command i use, since i don't actually enter into an interactive session on the server at all to view that file | 20:28 |
| fungi | it's usually as an ssh one-liner which runs the content to stdout | 20:28 |
| openstackgerrit | Merged zuul/zuul master: Annotate some logs in the scheduler with event id https://review.opendev.org/658648 | 20:30 |
| *** yboaron_ has quit IRC | 20:46 | |
| *** kjackal has quit IRC | 20:47 | |
| openstackgerrit | Merged zuul/zuul master: Annotate logs in the zuul driver with event ids https://review.opendev.org/658649 | 20:48 |
| *** raissa has quit IRC | 20:51 | |
| *** raissa has joined #openstack-infra | 20:52 | |
| *** raissa has quit IRC | 20:52 | |
| *** raissa has joined #openstack-infra | 20:54 | |
| *** raissa has quit IRC | 20:54 | |
| *** raissa has joined #openstack-infra | 20:54 | |
| *** raissa has joined #openstack-infra | 20:56 | |
| *** raissa has quit IRC | 20:56 | |
| *** raissa has joined #openstack-infra | 20:57 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove legacy-group-based-policy-dsvm-rally jobs https://review.opendev.org/659918 | 21:00 |
| * kmalloc_away surfaces from vacation for a moment: is it most correct to ask here or on the ML for initial addition to a given -core group in gerrit? | 21:02 | |
| *** tjgresha has joined #openstack-infra | 21:03 | |
| fungi | kmalloc_away: in most cases those groups are self-managed so you should ask wherever is appropriate to reach the members of that grou | 21:04 |
| fungi | p | 21:04 |
| kmalloc_away | fungi: this is a new repo-new new core group | 21:04 |
| *** mattw4 has quit IRC | 21:04 | |
| fungi | kmalloc_away: oh, you said "initial" | 21:04 |
| kmalloc_away | :) | 21:04 |
| fungi | i should read harder | 21:04 |
| fungi | yeah, asking in here works fine | 21:04 |
| fungi | which group? also helps to indicate which change added the group | 21:05 |
| kmalloc_away | spectroscope-core, initial member, me ;) | 21:05 |
| kmalloc_away | sec. | 21:05 |
| kmalloc_away | https://review.opendev.org/#/c/657862/ | 21:05 |
| kmalloc_away | added the group. | 21:05 |
| kmalloc_away | also, i guess -release as well. | 21:05 |
| *** tjgresha has quit IRC | 21:05 | |
| fungi | if you were the author of the group and the project isn't an official part of any existing governance then we can add the change creator easily without much justification | 21:05 |
| kmalloc_away | yup, i am the author of the change that added the group and it's not anything under normal governance at this point | 21:06 |
| kmalloc_away | this is the service that will do IDP Proxy/Brokering | 21:06 |
| kmalloc_away | it might someday follow into keystone's purview but it really is not anything yet and I didn't want to burden folks for the intial code unless they were actively contributing | 21:07 |
| fungi | kmalloc_away: added | 21:07 |
| fungi | (to both groups created by that acl addition) | 21:07 |
| kmalloc_away | this is the "got too much sun and doing a day indoors so i am not burned worse by that warm bright object in the sky" | 21:07 |
| kmalloc_away | supported by brie :) and time to duck back off irc. | 21:08 |
| fungi | the daystar, it burns | 21:08 |
| kmalloc_away | yeah, it burnt my shoulders. | 21:08 |
| kmalloc_away | it's amazing how much easier it is to burn here in the middle of that large wet object rather than in seattle. | 21:08 |
| kmalloc_away | fungi: thanks for the help | 21:08 |
| openstackgerrit | Merged zuul/zuul master: Add event id to timer events https://review.opendev.org/658650 | 21:08 |
| fungi | any time! | 21:09 |
| clarkb | kmalloc_away: I almost never get burned here in the PNW | 21:10 |
| clarkb | but 10 minutes in the sun in the tropics and am toast | 21:11 |
| kmalloc_away | clarkb: yup. here on maui, even under an umbrella and with copious amounts of SPF 50, slightly burned | 21:11 |
| kmalloc_away | in PNW, never burned. | 21:11 |
| *** mattw4 has joined #openstack-infra | 21:24 | |
| openstackgerrit | Merged zuul/zuul master: Annotate pipeline processing with event id https://review.opendev.org/658651 | 21:27 |
| *** sreejithp has quit IRC | 21:34 | |
| *** sreejithp has joined #openstack-infra | 21:35 | |
| *** igordc has joined #openstack-infra | 21:37 | |
| clarkb | ianw: left some comments on the mirror playbook chnage | 21:37 |
| clarkb | ianw: overall it looks like it shoudl work so I think this is close. Mostly just need toa ccomodate the base.yaml split which merged this morning (my time) and a firewall fix | 21:38 |
| clarkb | mordred: ^ https://review.opendev.org/#/c/658281/29 has an ansible question or two if you have a moment | 21:39 |
| *** pcaruana has quit IRC | 21:40 | |
| *** igordc has quit IRC | 21:40 | |
| *** mriedem_away has quit IRC | 21:42 | |
| *** sreejithp has quit IRC | 21:43 | |
| *** igordc has joined #openstack-infra | 21:44 | |
| *** jamesmcarthur has quit IRC | 21:45 | |
| openstackgerrit | Merged zuul/zuul master: Annotate merger logs with event id https://review.opendev.org/658652 | 21:45 |
| mordred | clarkb: responded, mostly in agreement | 21:46 |
| *** jamesmcarthur has joined #openstack-infra | 21:46 | |
| ianw | clarkb: will do | 21:50 |
| ianw | clarkb / fungi : i'd like to knock ask.o.o over hopefully even today, could you take a look at https://review.opendev.org/#/c/659473/ which installs the hacky intermediate package | 21:51 |
| clarkb | ianw: I also +2'd the ask01 inventory addition but didn't approve as I'm not sure how much hand holding it needs | 21:51 |
| * clarkb looks at that one | 21:51 | |
| ianw | yesterday i brought up ask01 with it hand-preinstalled; so you can investigate it on that if you like | 21:51 |
| ianw | clarkb: yeah, thanks i'll put it in the inventory now, it should be fine to be puppeted | 21:53 |
| clarkb | fungi: ianw mordred any idea if skopeo has updated their ppa with tristanC's fix yet? | 21:54 |
| clarkb | wonder if we can merge the revert to the pin | 21:54 |
| clarkb | https://review.opendev.org/#/c/659474/ is teh change to merge if we think that has made it in | 21:54 |
| mordred | ianw: ok to merge that change? | 21:54 |
| * clarkb tries to understand ppa versioning | 21:54 | |
| ianw | mordred: which one is "that" now? :) | 21:55 |
| mordred | the one. you know - the one change :) | 21:55 |
| mordred | ianw: https://review.opendev.org/#/c/659473 - 2x+2 | 21:55 |
| clarkb | 1.37.1 is the version of skopeo on the ppa but 1.36 is the most recent git tag | 21:55 |
| clarkb | ah but it was built on the 18th but fix merged ont he 20th | 21:56 |
| ianw | mordred: ah, yep; should do nothing on the live server as the package is already installed :) | 21:56 |
| clarkb | so that settles that, not safe to revert the pin yet | 21:56 |
| *** slaweq has quit IRC | 21:57 | |
| clarkb | ianw: re https://review.opendev.org/#/c/658228/1 should we alsoremove the dns records for grafana01? Seems they are still in place but the server is gone | 21:59 |
| ianw | clarkb: yep, can do. | 22:00 |
| openstackgerrit | Merged opendev/system-config master: launch.py: Fix inventory list https://review.opendev.org/660008 | 22:00 |
| clarkb | I just approved the testinfra master use in the testinfra-devel job and with that I think I'm largely caught up on your stacks of changes | 22:01 |
| ianw | clarkb: thanks :) if no objection, i'll give all this ask stuff an hour or two to go through and check it out, if looking good will stop the old service, migrate the db and switch the cname | 22:02 |
| ianw | oh, the only thing is if we want the ssl cert to cover ask01.o.o too ... | 22:03 |
| fungi | clarkb: see my comments on the skopeo pin revert... unless it's been updated again in the past few hours no | 22:04 |
| clarkb | ianw: I don't think the ssl cert has to cover the backend server name | 22:04 |
| clarkb | ianw: so that plan sounds great to me | 22:04 |
| clarkb | fungi: there must be another revert change? https://review.opendev.org/#/c/659474/ has no comments | 22:04 |
| fungi | oh, i had no idea there was a preexisting revert, mine was https://review.opendev.org/659908 | 22:05 |
| fungi | for some reason gertty isn't showing me that conflict | 22:06 |
| clarkb | gerrit doesn't either | 22:06 |
| clarkb | how interesting | 22:06 |
| ianw | oh, i put 659474 in the evening i promised to babysit it when i realised the package had disappeared, mostly just so there was a flag up that it wasn't working as expected | 22:06 |
| fungi | anyway, the way to figure it out is to fetch the source .tar.xz from there and check the most recent entry in the debian/changelog file which indicates the skopeo repo commit from which it was built | 22:07 |
| ianw | maybe it doesn't conflict because it's exactly the same | 22:08 |
| fungi | heh | 22:08 |
| fungi | mayhaps | 22:08 |
| *** slaweq has joined #openstack-infra | 22:09 | |
| ianw | fungi: yeah, i was just clicking on "changes file" from https://launchpad.net/~projectatomic/+archive/ubuntu/ppa/+packages but same | 22:09 |
| fungi | but anyway, the current one is built from the commit immediately prior to the one we want, so pretty sure the next build will have it | 22:11 |
| rpioso | If a change's parent has been abandoned, must it be rebased before it'll merge? | 22:12 |
| *** slaweq has quit IRC | 22:13 | |
| fungi | rpioso: yes | 22:14 |
| rpioso | fungi: Thank you! | 22:14 |
| *** tosky has quit IRC | 22:15 | |
| fungi | you're welcome! | 22:17 |
| openstackgerrit | Merged zuul/zuul master: Annotate job freezing logs with event id https://review.opendev.org/658888 | 22:18 |
| *** rfarr has quit IRC | 22:21 | |
| openstackgerrit | Merged opendev/puppet-diskimage_builder master: Use full key for gpg key https://review.opendev.org/659716 | 22:23 |
| openstackgerrit | Merged opendev/puppet-diskimage_builder master: Install backported debootstrap https://review.opendev.org/659031 | 22:23 |
| openstackgerrit | Merged opendev/system-config master: Add ask01.openstack.org to inventory https://review.opendev.org/660012 | 22:23 |
| openstackgerrit | Merged opendev/system-config master: Remove grafana01.openstack.org from inventory https://review.opendev.org/658228 | 22:23 |
| clarkb | ianw ^ fyi thats a good chunk of what I reviewed of yours | 22:27 |
| *** bobh has joined #openstack-infra | 22:31 | |
| *** slaweq has joined #openstack-infra | 22:40 | |
| *** gfidente is now known as gfidente|afk | 22:43 | |
| openstackgerrit | Merged opendev/system-config master: Add testinfra master to -devel job https://review.opendev.org/640231 | 22:43 |
| *** dmsimard5 is now known as dmsimard | 22:44 | |
| *** slaweq has quit IRC | 22:44 | |
| *** bobh has quit IRC | 22:46 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/os-loganalyze master: Add support for Magnum https://review.opendev.org/616705 | 22:51 |
| *** tkajinam has joined #openstack-infra | 22:59 | |
| *** sreejithp has joined #openstack-infra | 23:07 | |
| clarkb | ianw I think the diskimage_builder puppet changes will go into effect in about 10 minutes I'll be keeping an eye on that as it also fixes an issue with puppet-4 warnings (I think) | 23:08 |
| ianw | clarkb: did the full key help? | 23:09 |
| *** sreejithp has quit IRC | 23:09 | |
| ianw | http://logs.openstack.org/16/659716/4/gate/openstackci-beaker-puppet-4/e2c06f4/job-output.txt.gz#_2019-05-20_22_08_07_613619 | 23:11 |
| ianw | not sure if this means it used the on-disk key, or just talked to the gpg server ok and didn't log anything | 23:11 |
| clarkb | ya that I don't know | 23:11 |
| clarkb | was mostly going to check the in production behavior fixing the puppet-4 warning | 23:12 |
| clarkb | oh we are on a 30 minute cycle instead of 45 minute right now (I wonder if that is due to breakup of base.yaml) | 23:12 |
| clarkb | warning is fixed | 23:12 |
| clarkb | debootstrap 1.0.78+nmu1ubuntu1.9 is the version of debootstrap it doesn't appear to have upgraded? | 23:13 |
| clarkb | 1.0.114~bpo16.04+1 is the newer version I think | 23:14 |
| clarkb | the ensure is present not latest | 23:14 |
| clarkb | I wonder if that prevented puppet from doing an upgrade | 23:14 |
| clarkb | ianw should I do an apt-get install debootstrap to see? | 23:14 |
| ianw | clarkb: yeah, that would be good | 23:15 |
| clarkb | k trying now on nb01 | 23:15 |
| clarkb | debootstrap 1.0.114~bpo16.04+1 is the reported version now so that seems to have been it | 23:16 |
| clarkb | I'll do it on nb02 as well ( do we have an arm ppa too? ) | 23:16 |
| clarkb | we do have an arm ppa too | 23:18 |
| clarkb | all three builders have up to date debootstrap | 23:18 |
| clarkb | zigo: ^ you should be able to build debian now | 23:18 |
| ianw | it should be arch independent, it's perl | 23:20 |
| *** Lucas_Gray has quit IRC | 23:20 | |
| clarkb | ah | 23:21 |
| ianw | i think there is a change out there to add buster images | 23:21 |
| ianw | maybe not; other thing might be the mirrors | 23:22 |
| *** rcernin has joined #openstack-infra | 23:22 | |
| *** efried has quit IRC | 23:22 | |
| *** efried has joined #openstack-infra | 23:23 | |
| *** rlandy has quit IRC | 23:25 | |
| clarkb | mordred: do you remember what base image was used to build the gitea servers? | 23:27 |
| mordred | uh | 23:27 |
| clarkb | (I can probably figure this out mostly from api output) | 23:27 |
| clarkb | just remembering it doesn't include less which makes me think it might be one of our minimal images? | 23:27 |
| clarkb | (starting to page in what is necessary to rebuild the gitea servers since 06 needs a rebuild and they all need more disk) | 23:28 |
| mordred | hrm. yeah - I don't 100% remember ... but let me look real quick | 23:28 |
| mordred | (I feel like we did something weird...) | 23:28 |
| clarkb | also I think we want the larger journal size for ext4 if indeed they come from dib | 23:28 |
| clarkb | ianw: logan- ^ any idea when that might be in a dib release? | 23:29 |
| logan- | i think it is in the release ianw cut a few days ago | 23:29 |
| clarkb | oh cool so should already be done | 23:29 |
| logan- | https://opendev.org/openstack/diskimage-builder/src/tag/2.23.0 that one | 23:29 |
| logan- | confirmed its in there | 23:30 |
| clarkb | ianw: can we approve https://review.opendev.org/#/c/659215/ now? | 23:30 |
| mordred | clarkb: it was either "Ubuntu 18.04 LTS (x86_64) [2018-08-23]" or one of the two ubuntu-bionic-minimal | 23:31 |
| clarkb | rebuilding the nodes I think we want to go to 80GB disks (up from 40GB where we keep running up to that limit for various reasons) then we can do less cpu and less memory too (though more less cpu than less memory) | 23:32 |
| clarkb | network bw isn't tied to cpus in vexxhost as in other clouds so that shouldn't be an issue | 23:32 |
| clarkb | mordred: ya these hosts have glean installed so I think it was our image | 23:32 |
| mordred | clarkb: these are all boot-from-volume ceph nodes - maybe we can just expand them? | 23:32 |
| ianw | we can, i was sort of meaning to look into what happens with growroot and journal sizes | 23:32 |
| mordred | clarkb: ++ | 23:32 |
| clarkb | mordred: maybe? we need to rebuild gitea06 anyway since its sad disk state and we can't easily shrink cpu/memory without a rebuild either | 23:33 |
| clarkb | (resize is only up iirc) | 23:33 |
| clarkb | mordred: but also the journal size is wrong if we increase the disk size | 23:33 |
| logan- | https://opendev.org/openstack/diskimage-builder/commit/87a18f51e38e8a43adf4d67d30229854355a181f might be useful to look at for trimming down nodepool image sizes also | 23:34 |
| mordred | clarkb: oh - good point | 23:34 |
| clarkb | logan-: ya that may help with raw/vhd images | 23:35 |
| logan- | it should be possible to save quite a bit of upload bw each day by having less than a 60% free space buffer in the images | 23:35 |
| clarkb | qcow2 should compress the empty space away | 23:35 |
| logan- | yup | 23:35 |
| mordred | clarkb: d0edcf7a-0779-476e-8285-bcab9043b616 | 23:36 |
| mordred | clarkb: was the image id | 23:36 |
| mordred | clarkb: fwiw, I got that from doing a volume show on the attached volume, and it shows: | 23:36 |
| mordred | | volume_image_metadata | {'checksum': '31f6b2b772bbd43f67908b37721f69a8', 'min_ram': '0', 'disk_format': 'raw', 'image_name': 'ubuntu-bionic-minimal', 'image_id': 'd0edcf7a-0779-476e-8285-bcab9043b616', 'container_format': 'bare', 'min_disk': '0', 'size': '2237857792'} | 23:36 |
| clarkb | so I think the next step for us there is a new minimal image with correct journalctl size, then we can build a new gitea06, then we can work to replace the others in sequence | 23:36 |
| mordred | ++ | 23:36 |
| *** aaronsheffield has quit IRC | 23:37 | |
| clarkb | mordred: what happened to the change to have nodepool build those images for us? | 23:37 |
| clarkb | it depends on https://review.opendev.org/#/c/640027/ | 23:37 |
| clarkb | mordred: does ^ need to be rebased after the base.yaml split? | 23:38 |
| mordred | clarkb: almost certainly | 23:38 |
| clarkb | gerrit does say cannot merge so that is probably why | 23:38 |
| mordred | clarkb: I wrote that change? | 23:39 |
| clarkb | yes | 23:39 |
| mordred | february was a long time ago | 23:39 |
| mordred | lemme see about rebasing it | 23:40 |
| clarkb | it is probably a good idea to get that in place before we rebuild all the gitea nodes | 23:40 |
| clarkb | (maybe not necessary before rebuilding gitea06) | 23:40 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add control plane clouds to nodepool builder clouds.yaml https://review.opendev.org/640027 | 23:43 |
| clarkb | mordred: you'll need to remove the arm64ci cloud stuff | 23:44 |
| mordred | oh right | 23:44 |
| *** jamesmcarthur has quit IRC | 23:44 | |
| clarkb | I think packethost as well at this point :/ | 23:44 |
| clarkb | and "linaro" ? | 23:44 |
| clarkb | and maybe citycloud? | 23:45 |
| clarkb | ugh its been a bad run for some of the clouds | 23:45 |
| ianw | somehow merging that opendev mirror change base.yaml into master's new base.yaml confuses diff and makes emerge fail; i've never seen that before | 23:45 |
| clarkb | mordred: you should cross check against what we currently have in place | 23:45 |
| *** gyee has quit IRC | 23:51 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: DNM: this should cause a React exception https://review.opendev.org/660231 | 23:53 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: DNM: this should cause a React exception https://review.opendev.org/660231 | 23:54 |
| *** lseki has quit IRC | 23:55 | |
| *** igordc has quit IRC | 23:55 | |
| *** michael-beaver has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!