| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: validate-zone-db : add job and make more generic https://review.opendev.org/661138 | 00:01 |
|---|---|---|
| *** Weifan has quit IRC | 00:01 | |
| *** Weifan has joined #openstack-infra | 00:01 | |
| *** smarcet has quit IRC | 00:06 | |
| *** Weifan has quit IRC | 00:06 | |
| *** aaronsheffield has quit IRC | 00:06 | |
| *** rcernin has quit IRC | 00:07 | |
| *** smarcet has joined #openstack-infra | 00:08 | |
| *** armax has quit IRC | 00:08 | |
| *** samueldmq has quit IRC | 00:12 | |
| *** eernst has joined #openstack-infra | 00:14 | |
| *** smarcet has quit IRC | 00:17 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: trigger: add job filter event https://review.opendev.org/639905 | 00:20 |
| *** smarcet has joined #openstack-infra | 00:20 | |
| *** markvoelker has quit IRC | 00:31 | |
| *** eernst has quit IRC | 00:37 | |
| *** gyee has quit IRC | 00:39 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-zuul-ci.org master: Add zone-check job https://review.opendev.org/660967 | 00:44 |
| *** smarcet has quit IRC | 00:45 | |
| ianw | corvus / fungi : thanks; https://review.opendev.org/#/q/topic:check-zone+(status:open+OR+status:merged) updated to a more generic job | 00:55 |
| *** rcernin has joined #openstack-infra | 01:05 | |
| *** ykarel has joined #openstack-infra | 01:11 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: trigger: add job filter event https://review.opendev.org/639905 | 01:11 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: webtrigger: add initial driver and event https://review.opendev.org/555153 | 01:11 |
| *** ykarel has quit IRC | 01:15 | |
| fungi | gerrit task queue is down below 6k now | 01:29 |
| paladox | got to love the gerrit task queue :P | 01:30 |
| *** sreejithp has joined #openstack-infra | 01:31 | |
| *** yolanda has quit IRC | 01:51 | |
| *** michael-beaver has quit IRC | 01:53 | |
| *** yolanda has joined #openstack-infra | 01:54 | |
| *** ykarel has joined #openstack-infra | 01:55 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: webtrigger: add web route and rpclistener https://review.opendev.org/554839 | 01:59 |
| fungi | and now below 5k | 02:10 |
| *** ddurst- is now known as ddurst | 02:12 | |
| *** altlogbot_0 has quit IRC | 02:12 | |
| *** altlogbot_1 has joined #openstack-infra | 02:13 | |
| *** sarob has joined #openstack-infra | 02:13 | |
| *** ykarel has quit IRC | 02:15 | |
| clarkb | speeding up | 02:16 |
| *** sarob has quit IRC | 02:20 | |
| fungi | now below 4k | 02:30 |
| *** markvoelker has joined #openstack-infra | 02:32 | |
| *** armax has joined #openstack-infra | 02:38 | |
| *** gmann has joined #openstack-infra | 02:41 | |
| fungi | 3k | 02:46 |
| *** sarob has joined #openstack-infra | 02:48 | |
| fungi | 2 | 02:54 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: add build button to trigger job https://review.opendev.org/635716 | 03:05 |
| *** markvoelker has quit IRC | 03:06 | |
| *** nicolasbock has quit IRC | 03:09 | |
| *** whoami-rajat has joined #openstack-infra | 03:11 | |
| fungi | 1 | 03:14 |
| *** dave-mccowan has quit IRC | 03:14 | |
| fungi | and *finally* caught up | 03:14 |
| *** sarob has quit IRC | 03:22 | |
| *** Lucas_Gray has joined #openstack-infra | 03:25 | |
| *** ykarel has joined #openstack-infra | 03:26 | |
| *** psachin has joined #openstack-infra | 03:30 | |
| *** ricolin has joined #openstack-infra | 03:31 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:46 | |
| openstackgerrit | Merged zuul/zuul master: ansible-config: pin ara to <1.0.0 https://review.opendev.org/660859 | 03:48 |
| openstackgerrit | Merged openstack/reviewstats master: Handle all exceptions loading pickled data https://review.opendev.org/653106 | 03:53 |
| openstackgerrit | Merged openstack/reviewstats master: Switch default server to opendev.org https://review.opendev.org/655504 | 03:53 |
| openstackgerrit | Merged openstack/reviewstats master: Don't fail when writing cache either https://review.opendev.org/655505 | 03:53 |
| *** jesusaur has quit IRC | 03:54 | |
| *** jesusaur has joined #openstack-infra | 03:59 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 04:00 | |
| *** bhavikdbavishi has quit IRC | 04:01 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 04:01 | |
| *** markvoelker has joined #openstack-infra | 04:02 | |
| *** Lucas_Gray has quit IRC | 04:04 | |
| *** ykarel has quit IRC | 04:08 | |
| *** Lucas_Gray has joined #openstack-infra | 04:10 | |
| *** hwoarang has quit IRC | 04:17 | |
| *** sreejithp has quit IRC | 04:18 | |
| *** hwoarang has joined #openstack-infra | 04:18 | |
| *** bhavikdbavishi has quit IRC | 04:22 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:23 | |
| *** ykarel has joined #openstack-infra | 04:24 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: amqp: add basic trigger https://review.opendev.org/637458 | 04:27 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: amqp: add message informations to the job variables https://review.opendev.org/637666 | 04:27 |
| *** bhavikdbavishi has quit IRC | 04:30 | |
| *** auristor has quit IRC | 04:31 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:31 | |
| *** bhavikdbavishi has quit IRC | 04:31 | |
| *** markvoelker has quit IRC | 04:35 | |
| *** auristor has joined #openstack-infra | 04:44 | |
| *** raukadah is now known as chandankumar | 04:59 | |
| *** pcaruana has joined #openstack-infra | 05:08 | |
| *** pcaruana has quit IRC | 05:16 | |
| *** dpawlik has joined #openstack-infra | 05:16 | |
| *** pcaruana has joined #openstack-infra | 05:17 | |
| openstackgerrit | Joshua Hesketh proposed x/gearman-plugin master: add project parameters https://review.opendev.org/660873 | 05:19 |
| openstackgerrit | Joshua Hesketh proposed opendev/system-config master: Add tab to link from repo page to gerrit changes https://review.opendev.org/656808 | 05:21 |
| openstackgerrit | Joshua Hesketh proposed opendev/system-config master: Add tab to link from repo page to gerrit changes https://review.opendev.org/656808 | 05:22 |
| *** ramishra has joined #openstack-infra | 05:32 | |
| *** markvoelker has joined #openstack-infra | 05:32 | |
| *** tkajinam has quit IRC | 05:32 | |
| *** tkajinam has joined #openstack-infra | 05:34 | |
| *** sreejithp has joined #openstack-infra | 05:38 | |
| *** sreejithp_ has joined #openstack-infra | 05:41 | |
| *** imacdonn has quit IRC | 05:42 | |
| *** imacdonn has joined #openstack-infra | 05:42 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: git: only list head references https://review.opendev.org/661185 | 05:43 |
| *** sreejithp has quit IRC | 05:44 | |
| *** lpetrut has joined #openstack-infra | 05:45 | |
| *** Lucas_Gray has quit IRC | 05:45 | |
| *** zzehring_ has quit IRC | 05:46 | |
| *** hwoarang has quit IRC | 05:46 | |
| *** lpetrut has quit IRC | 05:46 | |
| *** lpetrut has joined #openstack-infra | 05:46 | |
| *** hwoarang has joined #openstack-infra | 05:47 | |
| *** quiquell has joined #openstack-infra | 05:48 | |
| *** sreejithp_ has quit IRC | 05:51 | |
| *** markvoelker has quit IRC | 06:05 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Switch rax.dfw mirror to opendev.org https://review.opendev.org/661187 | 06:07 |
| *** kaisers has quit IRC | 06:13 | |
| *** kaisers has joined #openstack-infra | 06:16 | |
| *** stakeda has joined #openstack-infra | 06:19 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Move openSUSE Tumbleweed into a caching mirror instead https://review.opendev.org/660125 | 06:20 |
| evrardjp | ianw: thanks for these ^ | 06:20 |
| AJaeger | config-core, want to review moving deploy-guide to tox-docs as well? https://review.opendev.org/661081 and https://review.opendev.org/661078 are ready for review, please | 06:23 |
| *** kjackal has joined #openstack-infra | 06:30 | |
| openstackgerrit | dongwenjuan proposed x/gearman-plugin master: add project parameters https://review.opendev.org/660873 | 06:31 |
| *** slaweq has joined #openstack-infra | 06:35 | |
| openstackgerrit | dongwenjuan proposed x/gearman-plugin master: add project parameters https://review.opendev.org/660873 | 06:35 |
| *** ramishra_ has joined #openstack-infra | 06:37 | |
| *** ramishra has quit IRC | 06:37 | |
| openstackgerrit | dongwenjuan proposed x/gearman-plugin master: add project parameters https://review.opendev.org/660873 | 06:43 |
| *** ramishra_ has quit IRC | 06:45 | |
| *** ramishra has joined #openstack-infra | 06:47 | |
| openstackgerrit | dongwenjuan proposed x/gearman-plugin master: add project parameters https://review.opendev.org/660873 | 06:48 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: git: only list head references https://review.opendev.org/661185 | 06:48 |
| *** yolanda has quit IRC | 06:49 | |
| *** yolanda has joined #openstack-infra | 06:49 | |
| *** pgaxatte has joined #openstack-infra | 06:50 | |
| *** rkukura has quit IRC | 06:55 | |
| *** tdasilva has quit IRC | 06:55 | |
| *** rkukura has joined #openstack-infra | 06:56 | |
| *** dpawlik has quit IRC | 06:56 | |
| openstackgerrit | Merged zuul/nodepool master: Use py3 pathlib in DibImageFile https://review.opendev.org/660191 | 06:56 |
| *** kjackal has quit IRC | 07:01 | |
| *** markvoelker has joined #openstack-infra | 07:03 | |
| openstackgerrit | dongwenjuan proposed x/gearman-plugin master: add project parameters https://review.opendev.org/660873 | 07:08 |
| *** kjackal has joined #openstack-infra | 07:08 | |
| *** gfidente has joined #openstack-infra | 07:09 | |
| openstackgerrit | Merged opendev/system-config master: Cleanup bashate errors to make them easier to understand https://review.opendev.org/661117 | 07:10 |
| openstackgerrit | dongwenjuan proposed x/gearman-plugin master: add project parameters https://review.opendev.org/660873 | 07:22 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Switch rax.dfw mirror to opendev.org https://review.opendev.org/661187 | 07:22 |
| ianw | clarkb: ^ note i added the volumes on the mirror as we discussed this morning too, so i consider it fully operational | 07:23 |
| ianw | AJaeger: thanks, i just realised i had the match on the region/provider backwards (wouldn't have broken, just wouldn't have done anything :) | 07:23 |
| *** ginopc has joined #openstack-infra | 07:24 | |
| *** jtomasek has joined #openstack-infra | 07:25 | |
| *** dpawlik has joined #openstack-infra | 07:26 | |
| *** gfidente has quit IRC | 07:27 | |
| *** gmann has quit IRC | 07:28 | |
| *** dtantsur|afk is now known as dtantsur | 07:30 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: git: only list heads and tags references https://review.opendev.org/661185 | 07:31 |
| *** icey has quit IRC | 07:34 | |
| *** markvoelker has quit IRC | 07:35 | |
| *** mnaser has quit IRC | 07:35 | |
| *** msmarcal has quit IRC | 07:35 | |
| *** edsousa has quit IRC | 07:36 | |
| *** rcernin has quit IRC | 07:36 | |
| *** guilhermesp has quit IRC | 07:37 | |
| *** lsell has quit IRC | 07:37 | |
| *** beisner has quit IRC | 07:39 | |
| *** seyeongkim has quit IRC | 07:39 | |
| *** sweston has quit IRC | 07:39 | |
| *** kjackal has quit IRC | 07:39 | |
| *** tkajinam has quit IRC | 07:41 | |
| *** davidlenwell has quit IRC | 07:42 | |
| *** lsell has joined #openstack-infra | 07:43 | |
| *** sweston has joined #openstack-infra | 07:43 | |
| *** davidlenwell has joined #openstack-infra | 07:43 | |
| *** seyeongkim has joined #openstack-infra | 07:43 | |
| *** guilhermesp has joined #openstack-infra | 07:43 | |
| *** msmarcal has joined #openstack-infra | 07:43 | |
| *** beisner has joined #openstack-infra | 07:44 | |
| *** mnaser has joined #openstack-infra | 07:44 | |
| *** jpena|off is now known as jpena | 07:50 | |
| *** ralonsoh has joined #openstack-infra | 07:50 | |
| *** andreykurilin has joined #openstack-infra | 07:51 | |
| *** e0ne has joined #openstack-infra | 07:51 | |
| *** pkopec has joined #openstack-infra | 07:52 | |
| openstackgerrit | Merged opendev/system-config master: Move openSUSE Tumbleweed into a caching mirror instead https://review.opendev.org/660125 | 08:01 |
| *** clayg has quit IRC | 08:01 | |
| *** clayg has joined #openstack-infra | 08:01 | |
| *** mnaser has quit IRC | 08:02 | |
| *** jtomasek has quit IRC | 08:02 | |
| *** iurygregory has joined #openstack-infra | 08:03 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/658564 | 08:03 |
| *** tkajinam has joined #openstack-infra | 08:04 | |
| *** portdirect has quit IRC | 08:04 | |
| *** mnaser has joined #openstack-infra | 08:05 | |
| *** geekinutah has quit IRC | 08:06 | |
| *** adriancz has quit IRC | 08:08 | |
| *** rpittau|afk is now known as rpittau | 08:08 | |
| *** portdirect has joined #openstack-infra | 08:08 | |
| *** rm_work has quit IRC | 08:09 | |
| *** geekinutah has joined #openstack-infra | 08:10 | |
| *** rm_work has joined #openstack-infra | 08:10 | |
| *** tosky has joined #openstack-infra | 08:10 | |
| *** lucasagomes has joined #openstack-infra | 08:11 | |
| *** abelur has quit IRC | 08:11 | |
| *** adriancz has joined #openstack-infra | 08:11 | |
| *** apetrich has quit IRC | 08:12 | |
| *** apetrich has joined #openstack-infra | 08:13 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support squash merge in Github https://review.opendev.org/661096 | 08:13 |
| *** abelur has joined #openstack-infra | 08:14 | |
| *** icey has joined #openstack-infra | 08:18 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Use tox for publish-deploy-guide https://review.opendev.org/661081 | 08:19 |
| AJaeger | frickler: thanks for catching this ^ | 08:19 |
| *** zbr has joined #openstack-infra | 08:19 | |
| openstackgerrit | dongwenjuan proposed x/gearman-plugin master: add project parameters https://review.opendev.org/660873 | 08:21 |
| openstackgerrit | Merged openstack/pbr master: Remove neutron-lbaas https://review.opendev.org/658643 | 08:21 |
| frickler | AJaeger: ah, great, that looks more like what I'd expected, thx | 08:21 |
| *** ykarel is now known as ykarel|lunch | 08:21 | |
| *** jtomasek has joined #openstack-infra | 08:24 | |
| *** kopecmartin|afk is now known as kopecmartin | 08:29 | |
| *** markvoelker has joined #openstack-infra | 08:32 | |
| *** trident has quit IRC | 08:33 | |
| *** trident has joined #openstack-infra | 08:34 | |
| *** Lucas_Gray has joined #openstack-infra | 08:35 | |
| *** ginopc has quit IRC | 08:39 | |
| *** ginopc has joined #openstack-infra | 08:39 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Differentiate between queued and waiting jobs in zuul web UI https://review.opendev.org/660878 | 08:43 |
| openstackgerrit | dongwenjuan proposed x/gearman-plugin master: add project parameters https://review.opendev.org/660873 | 08:47 |
| *** tesseract has joined #openstack-infra | 08:48 | |
| openstackgerrit | Hervé Beraud proposed openstack/pbr master: Read description file as utf-8 https://review.opendev.org/564874 | 08:50 |
| openstackgerrit | Hervé Beraud proposed openstack/pbr master: Read description file as utf-8 https://review.opendev.org/564874 | 08:52 |
| zbr | did anyone asked if it would be possible to enable gitea issue tracker for specific projects? | 08:53 |
| *** janki has joined #openstack-infra | 08:53 | |
| *** ccamacho has joined #openstack-infra | 08:56 | |
| *** derekh has joined #openstack-infra | 08:57 | |
| openstackgerrit | Hervé Beraud proposed openstack/pbr master: Read description file as utf-8 https://review.opendev.org/564874 | 09:00 |
| *** gfidente has joined #openstack-infra | 09:01 | |
| *** markvoelker has quit IRC | 09:06 | |
| openstackgerrit | dongwenjuan proposed x/gearman-plugin master: add project parameters and bindep file https://review.opendev.org/660873 | 09:09 |
| *** ykarel|lunch is now known as ykarel | 09:11 | |
| *** tkajinam has quit IRC | 09:36 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Differentiate between queued and waiting jobs in zuul web UI https://review.opendev.org/660878 | 09:45 |
| *** lpetrut has quit IRC | 09:46 | |
| *** sreejithp has joined #openstack-infra | 09:51 | |
| *** sreejithp has quit IRC | 09:55 | |
| *** tesseract has quit IRC | 09:56 | |
| *** hjensas has quit IRC | 09:57 | |
| *** tesseract has joined #openstack-infra | 09:59 | |
| *** tesseract has quit IRC | 10:00 | |
| *** tesseract has joined #openstack-infra | 10:01 | |
| *** markvoelker has joined #openstack-infra | 10:03 | |
| *** sarob has joined #openstack-infra | 10:04 | |
| *** electrofelix has joined #openstack-infra | 10:11 | |
| *** markvoelker has quit IRC | 10:35 | |
| *** sarob has quit IRC | 10:36 | |
| *** panda has quit IRC | 10:38 | |
| *** stakeda has quit IRC | 10:40 | |
| *** hjensas has joined #openstack-infra | 10:41 | |
| *** panda has joined #openstack-infra | 10:46 | |
| *** panda is now known as panda|rover | 10:48 | |
| *** ramishra has quit IRC | 10:49 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A reporter for Elasticsearch https://review.opendev.org/644927 | 10:53 |
| *** gmann has joined #openstack-infra | 11:01 | |
| *** roman_g has quit IRC | 11:07 | |
| *** Lucas_Gray has quit IRC | 11:09 | |
| *** tesseract has quit IRC | 11:10 | |
| *** psachin has quit IRC | 11:29 | |
| *** markvoelker has joined #openstack-infra | 11:32 | |
| *** smarcet has joined #openstack-infra | 11:37 | |
| *** psachin has joined #openstack-infra | 11:38 | |
| *** jpena is now known as jpena|lunch | 11:40 | |
| *** rosmaita has quit IRC | 11:41 | |
| *** smarcet has quit IRC | 11:41 | |
| *** dave-mccowan has joined #openstack-infra | 11:45 | |
| *** smarcet has joined #openstack-infra | 11:46 | |
| *** smarcet has quit IRC | 11:47 | |
| *** adriant has quit IRC | 11:48 | |
| *** amorin has quit IRC | 11:49 | |
| *** iokiwi has quit IRC | 11:49 | |
| *** iokiwi has joined #openstack-infra | 11:49 | |
| *** adriant has joined #openstack-infra | 11:49 | |
| *** dave-mccowan has quit IRC | 11:50 | |
| *** amorin has joined #openstack-infra | 11:51 | |
| *** rfolco is now known as rfolco|brb | 11:51 | |
| *** rfarr_ has joined #openstack-infra | 11:57 | |
| *** markvoelker has quit IRC | 11:57 | |
| *** EmilienM is now known as EvilienM | 11:58 | |
| *** ramishra has joined #openstack-infra | 11:58 | |
| *** rosmaita has joined #openstack-infra | 11:58 | |
| *** rh-jelabarre has joined #openstack-infra | 11:59 | |
| *** xek has joined #openstack-infra | 12:02 | |
| *** jaosorior has joined #openstack-infra | 12:06 | |
| *** sshnaidm is now known as sshnaidm|afk | 12:06 | |
| mordred | zbr: it is not currently possible. each gitea behind the load balancer is completely independent. there is work needed upstream for us to be able to have a single clustered install - namely around indexing | 12:07 |
| *** psachin has quit IRC | 12:08 | |
| *** rh-jelabarre has quit IRC | 12:08 | |
| *** rh-jelabarre has joined #openstack-infra | 12:09 | |
| mordred | zbr: that's not saying that once that's fixed it will be decided to support such a thing - just that right now it's physically impossible | 12:13 |
| *** derekh has quit IRC | 12:15 | |
| *** Yamini has joined #openstack-infra | 12:20 | |
| Yamini | hi team, am facing an issue in the CI system while running the CI 2019-05-24 12:21:22,231 DEBUG zuul.Repo: Cloning from ssh://cisco-cinder-ci@review.opendev.org:29418/openstack/nova to /var/lib/zuul/git/review.opendev.org/openstack/nova | 12:20 |
| Yamini | did something change hitting it only from yesterday | 12:20 |
| Yamini | the above cloning fails | 12:21 |
| Yamini | repo.reset() File "/usr/local/lib/python3.6/dist-packages/zuul/merger/merger.py", line 263, in reset self.update() File "/usr/local/lib/python3.6/dist-packages/zuul/merger/merger.py", line 427, in update repo = self.createRepoObject() File "/usr/local/lib/python3.6/dist-packages/zuul/merger/merger.py", line 256, in createRepoObject self._ensure_cloned() File "/usr/local/lib/python3.6/dist-packages/zuu | 12:21 |
| *** rosmaita has left #openstack-infra | 12:21 | |
| Yamini | 2019-05-24 12:21:22,182 ERROR zuul.Merger: Unable to reset repo <zuul.merger.merger.Repo object at 0x7f3d41ba91d0> Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/zuul/merger/merger.py", line 666, in _mergeItem repo.reset() File "/usr/local/lib/python3.6/dist-packages/zuul/merger/merger.py", line 263, in reset self.update() File "/usr/local/lib/python3.6/dist-packages/zuul/merger | 12:22 |
| Yamini | has anyone seen similar error | 12:22 |
| *** rosmaita has joined #openstack-infra | 12:25 | |
| *** eharney has joined #openstack-infra | 12:30 | |
| *** dhill_ has quit IRC | 12:35 | |
| *** rlandy has joined #openstack-infra | 12:35 | |
| *** tdasilva has joined #openstack-infra | 12:36 | |
| *** jpena|lunch is now known as jpena | 12:38 | |
| corvus | i'm going to run another gitea gc pass now | 12:41 |
| *** roman_g has joined #openstack-infra | 12:44 | |
| corvus | Yamini: can you paste the entire error and stack trace to paste.openstack.org and copy the resulting url here? | 12:47 |
| *** e0ne has quit IRC | 12:52 | |
| *** jamesmcarthur has joined #openstack-infra | 12:52 | |
| *** aaronsheffield has joined #openstack-infra | 12:53 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 12:54 |
| mordred | infra-root: I thought I was going to be AFK today - but the diving has been cancelled due to bad weather in the gulf - so I'm here today | 12:56 |
| *** e0ne has joined #openstack-infra | 13:01 | |
| *** rfarr__ has joined #openstack-infra | 13:04 | |
| *** rfarr_ has quit IRC | 13:07 | |
| *** hwoarang has quit IRC | 13:07 | |
| fungi | sorry to hear about your last-minute change of plans, but glad you're here! | 13:07 |
| corvus | mordred: i'm not glad you're not diving but i am glad you're not diving in bad weather | 13:08 |
| corvus | looks like some of the gcs are wrapping up now | 13:09 |
| *** hwoarang has joined #openstack-infra | 13:09 | |
| mordred | corvus: yes - I completely agree. it seems like spending three nights on a dive boat in the gulf of mexico in high seas would be ... unpleasant | 13:10 |
| *** jistr is now known as jistr|afk | 13:11 | |
| zbr | mordred: thanks. good to know that about gitea. | 13:12 |
| *** mriedem has joined #openstack-infra | 13:12 | |
| fungi | zbr: we do have the flexibility to point different projects to different issue trackers though (and currently use it to hook the issues link up to either launchpad or storyboard) | 13:13 |
| zbr | fungi: i know but there would be a huge benefit in terms of accessibility to have issue tracker integrated, especially for small/standalone projects. | 13:15 |
| *** sreejithp has joined #openstack-infra | 13:15 | |
| *** sreejithp_ has joined #openstack-infra | 13:17 | |
| *** rfolco|brb is now known as rfolco | 13:18 | |
| *** sreejithp has quit IRC | 13:19 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/pbr master: Allow git-tags to be SemVer compliant https://review.opendev.org/618569 | 13:22 |
| *** dpawlik has quit IRC | 13:23 | |
| *** Goneri has joined #openstack-infra | 13:29 | |
| *** ykarel is now known as ykarel|meeting | 13:29 | |
| *** quiquell has quit IRC | 13:31 | |
| *** lseki has joined #openstack-infra | 13:33 | |
| *** Goneri has quit IRC | 13:36 | |
| *** zzehring has joined #openstack-infra | 13:39 | |
| corvus | gitea gc complete | 13:39 |
| corvus | mordred, fungi: iiuc, in order to pick up the new gitweb links, we need to restart gerrit, which, ironically means another full replication? | 13:40 |
| corvus | (and yes, that's alanis-mode irony) | 13:42 |
| *** Goneri has joined #openstack-infra | 13:45 | |
| fungi | right, will take effect after the next gerrit restart | 13:51 |
| fungi | restarting it later today might make sense as we traditionally see lower volumes in the latter hours utc of friday leading into the weekend | 13:52 |
| fungi | which means 1. folks are less likely to be inconvenienced/notice, and 2. it may go faster when there's less contention for ci requests | 13:53 |
| corvus | sgtm | 13:53 |
| dmsimard | So, for some reason, "git clone --mirror" commands are failing very often from CentOS but not from Fedora. It seems like this started happening after the switch to opendev and we aren't able to reproduce on github. I'm still troubleshooting but I wanted to mention that in case it rang a bell | 13:54 |
| *** betherly has joined #openstack-infra | 13:55 | |
| dmsimard | git clone --mirror commands are apparently ran when cloning puppet modules for a Puppetfile and I can't reproduce the issue without --mirror | 13:55 |
| fungi | could be related to the vintage of git client on each platform, and possible how they handle differences in protocol implementation server-side | 13:55 |
| *** jistr|afk is now known as jistr | 13:56 | |
| dmsimard | yeah, that's why I checked fedora :D | 13:56 |
| openstackgerrit | Markus Hosch proposed zuul/zuul master: WIP: Fix case sensitivity in codeowners check https://review.opendev.org/661003 | 13:57 |
| *** quiquell has joined #openstack-infra | 13:58 | |
| *** betherly has quit IRC | 14:00 | |
| *** whoami-rajat has quit IRC | 14:00 | |
| *** bnemec is now known as beekneemech | 14:14 | |
| *** stephenfin is now known as finucannot | 14:14 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Fix case sensitivity in codeowners check https://review.opendev.org/661003 | 14:14 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 14:18 |
| *** rh-jelabarre has quit IRC | 14:20 | |
| *** _erlon_ has joined #openstack-infra | 14:22 | |
| *** jaosorior has quit IRC | 14:23 | |
| *** jamesmcarthur has quit IRC | 14:24 | |
| *** quiquell has quit IRC | 14:24 | |
| *** rh-jelabarre has joined #openstack-infra | 14:26 | |
| *** quiquell has joined #openstack-infra | 14:26 | |
| *** Anticime1 is now known as Anticimex | 14:26 | |
| *** michael-beaver has joined #openstack-infra | 14:30 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: allow the use of non-bzip compressed stages for building gentoo https://review.opendev.org/661098 | 14:33 |
| *** roman_g has quit IRC | 14:33 | |
| *** smarcet has joined #openstack-infra | 14:34 | |
| *** jchhatbar has joined #openstack-infra | 14:34 | |
| *** janki has quit IRC | 14:36 | |
| *** liuyulong has joined #openstack-infra | 14:37 | |
| *** gfidente has quit IRC | 14:39 | |
| AJaeger | config-core, want to review moving deploy-guide to tox-docs as well? https://review.opendev.org/661081 and https://review.opendev.org/661078 are ready for review, please | 14:40 |
| *** jchhatbar has quit IRC | 14:44 | |
| *** ykarel|meeting is now known as ykarel | 14:48 | |
| *** chandankumar is now known as raukadah | 14:48 | |
| *** hwoarang has quit IRC | 14:50 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Disable gc in test_scheduler.TestExecutor as done in base assertFinalState https://review.opendev.org/661316 | 14:52 |
| *** quiquell has quit IRC | 14:52 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 14:53 |
| *** jamesmcarthur has joined #openstack-infra | 14:55 | |
| *** hwoarang has joined #openstack-infra | 14:56 | |
| *** jamesmcarthur has quit IRC | 14:59 | |
| openstackgerrit | Jean-Philippe Evrard proposed opendev/system-config master: Start mirroring openSUSE Leap 15.1 https://review.opendev.org/660126 | 14:59 |
| *** roman_g has joined #openstack-infra | 15:00 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 15:04 |
| *** jamesmcarthur has joined #openstack-infra | 15:05 | |
| *** piotrowskim has quit IRC | 15:06 | |
| *** kopecmartin is now known as kopecmartin|off | 15:06 | |
| *** markmcclain has joined #openstack-infra | 15:07 | |
| *** e0ne has quit IRC | 15:08 | |
| *** diablo_rojo has joined #openstack-infra | 15:08 | |
| *** e0ne has joined #openstack-infra | 15:11 | |
| *** jamesmcarthur has quit IRC | 15:11 | |
| openstackgerrit | David Moreau Simard proposed opendev/puppet-zuul master: Pin ARA to <1.0.0 https://review.opendev.org/661329 | 15:11 |
| *** Yamini has quit IRC | 15:12 | |
| *** gyee has joined #openstack-infra | 15:13 | |
| *** e0ne has quit IRC | 15:14 | |
| *** e0ne has joined #openstack-infra | 15:15 | |
| openstackgerrit | David Moreau Simard proposed zuul/zuul master: Pin ARA to <1.0.0 https://review.opendev.org/661331 | 15:15 |
| fungi | just got around to confirming, but the "openid" tab is now gone from https://opendev.org/user/login and the old https://opendev.org/user/login/openid url it used to take you to now renders a blank page | 15:17 |
| fungi | so seems that hole is effectively plugged as far as i can see | 15:17 |
| *** jamesmcarthur has joined #openstack-infra | 15:19 | |
| *** Lucas_Gray has joined #openstack-infra | 15:22 | |
| openstackgerrit | Jason E. Rist proposed openstack/project-config master: Retiring TripleO-UI https://review.opendev.org/661332 | 15:22 |
| *** pkopec has quit IRC | 15:24 | |
| *** pgaxatte has quit IRC | 15:25 | |
| *** owalsh has quit IRC | 15:25 | |
| *** kjackal has joined #openstack-infra | 15:26 | |
| mordred | fungi: \o/ | 15:29 |
| *** owalsh has joined #openstack-infra | 15:32 | |
| clarkb | catching up, disk usage looks good after this mornings gc | 15:34 |
| clarkb | I've rechecked the cron installation change as it failed on the puppet idempotency check | 15:35 |
| clarkb | maybe we should stop caring about idempotency as it races repos like project-config merges | 15:36 |
| AJaeger | infra-root, should we use ansible 2.7 for testing in jobs ? See https://review.opendev.org/659810 and https://review.opendev.org/659811 , please | 15:37 |
| corvus | AJaeger: if this is abount ansible-lint, didn't we decide that 2.8 was too agressive? should we drop it altogether? | 15:40 |
| corvus | (if we aren't going to be able to use 2.8, i don't see the point in continuing to use 2.7) | 15:40 |
| AJaeger | corvus: it's only about *ansible*, no changes for lint. | 15:40 |
| openstackgerrit | Clark Boylan proposed opendev/puppet-openstackci master: Remove idempotency spec checks https://review.opendev.org/661341 | 15:41 |
| corvus | AJaeger: what is ansible's purpose here? | 15:41 |
| clarkb | that change should remove the idempotency checks from the openstackci beaker/spec jobs | 15:41 |
| *** roman_g has quit IRC | 15:41 | |
| corvus | (maybe the comment about why we're installing ansible needs updating too) | 15:42 |
| AJaeger | maybe... | 15:42 |
| corvus | i'm looking into the cronspam errors from the new mirror server | 15:42 |
| * AJaeger can abandon as well if it's not needed... | 15:43 | |
| corvus | AJaeger: i don't know the right course of action here, i barely understand the linting jobs | 15:43 |
| AJaeger | corvus, let me try without ansible, you have a good point there... | 15:43 |
| *** roman_g has joined #openstack-infra | 15:43 | |
| corvus | AJaeger: that should at least answer the question | 15:44 |
| corvus | regarding the mirror -- there seems to be an error in the playbook | 15:44 |
| AJaeger | corvus: since zuul does not require ansible anymore, we could remove the line.. | 15:44 |
| corvus | failed: [mirror01.dfw.rax.opendev.org] (item=wheel) => {"changed": false, "gid": 1000, "group": "ubuntu", "item": "wheel", "mode": "0755", "msg": "chown failed: [Errno 30] Read-only file system: b'/afs/openstack.org/mirror/wheel'", "owner": "daemon", "path": "/afs/openstack.org/mirror/wheel", "size": 2048, "state": "directory", "uid": 1} | 15:44 |
| corvus | the issue is that the ownership of the directory in afs is daemon/wheel whereas the playbook seems to want the link target owner to be root/root | 15:45 |
| corvus | i don't know why the playbook cares about that, but set that aside for the moment | 15:45 |
| corvus | the thing i don't understand is why this did not fail in testing | 15:45 |
| *** rpittau is now known as rpittau|afk | 15:45 | |
| corvus | here you can see that it set up the link without complaint: http://logs.openstack.org/81/658281/32/check/system-config-run-mirror/fb71a5d/bridge.openstack.org/ara-report/result/9250cb87-639c-4b12-8285-9595d53b86cf/ | 15:45 |
| corvus | oh... | 15:45 |
| corvus | what if it works the first time but not later times? | 15:45 |
| mordred | oh. piddle | 15:46 |
| *** jamesmcarthur has quit IRC | 15:46 | |
| clarkb | fitting that I would propose a change to remove idempotency checks and we discover somewhere else that having that would be nice :) | 15:46 |
| mordred | corvus: do we actually have the AFS client going in those tests? | 15:47 |
| corvus | clarkb: yeah, i was wondering if anyone would notice that :) | 15:47 |
| corvus | mordred: yes | 15:47 |
| mordred | kk. so it's not that | 15:47 |
| corvus | there's even an explicit verification that afs is mounted and working | 15:47 |
| mordred | sweet! | 15:47 |
| corvus | http://logs.openstack.org/81/658281/32/check/system-config-run-mirror/fb71a5d/bridge.openstack.org/ara-report/result/a91c23dd-b52d-4eed-90cd-543fd8922ef3/ | 15:48 |
| *** kjackal has quit IRC | 15:48 | |
| corvus | hrm, i can't back up my not-idempotent theory looking at the run logs | 15:49 |
| *** zul has quit IRC | 15:49 | |
| *** ijw has joined #openstack-infra | 15:53 | |
| *** jamesmcarthur has joined #openstack-infra | 15:55 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Fix new mirror system errors https://review.opendev.org/661346 | 15:55 |
| *** kjackal has joined #openstack-infra | 15:56 | |
| corvus | that collects two minor issues i've observed so far, and attempts to debug the perm issue | 15:56 |
| *** ykarel is now known as ykarel|away | 15:56 | |
| *** ginopc has quit IRC | 16:00 | |
| *** dtantsur is now known as dtantsur|afk | 16:01 | |
| *** jamesmcarthur has quit IRC | 16:02 | |
| *** jamesmcarthur has joined #openstack-infra | 16:03 | |
| *** e0ne has quit IRC | 16:04 | |
| *** roman_g has quit IRC | 16:07 | |
| *** betherly has joined #openstack-infra | 16:07 | |
| *** ykarel|away has quit IRC | 16:07 | |
| AJaeger | removing ansible from tox.ini, ansible-lint drags in ansible 2.8 and we get test failures, so I won't add additional changes | 16:10 |
| clarkb | we are setup to do 4 replication threads per gitea target. That explains why we use ~4 cpus on the gitea side I guess | 16:12 |
| clarkb | so we could use all 8vcpus if we bumped the number of replication threads | 16:12 |
| *** ijw has quit IRC | 16:12 | |
| *** betherly has quit IRC | 16:12 | |
| *** ijw has joined #openstack-infra | 16:13 | |
| *** Wryhder has joined #openstack-infra | 16:14 | |
| *** Lucas_Gray has quit IRC | 16:15 | |
| *** Wryhder is now known as Lucas_Gray | 16:15 | |
| corvus | oh hey, looks like the test confirms the idempotency hypothesis | 16:17 |
| clarkb | ansible bug? | 16:17 |
| corvus | maybe... this should be easy to test/reproduce | 16:18 |
| *** lucasagomes has quit IRC | 16:19 | |
| corvus | yep, reproduces locally just by creating a link to a target owned by root | 16:20 |
| *** bobh has joined #openstack-infra | 16:23 | |
| corvus | i think we just want to set "follow" to true | 16:23 |
| corvus | er false | 16:23 |
| *** bobh has quit IRC | 16:23 | |
| clarkb | we want to set the perms on the link itself not the target so that makes sense | 16:23 |
| corvus | yep, that seems to work. so yeah, i think there's still a minor bug in that in order to be idempotent, it really should check the perms after creating the link if follow is set to true (which is the default post 2.5) | 16:24 |
| corvus | i'll file that and then fixup my patch | 16:25 |
| beekneemech | Having to do extensive work on GitHub recently has made me appreciate our infra even more than I already did. | 16:26 |
| beekneemech | #thanks infra team for providing such a great place to do development | 16:26 |
| openstackstatus | beekneemech: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks) | 16:27 |
| clarkb | beekneemech: awww thanks | 16:27 |
| clarkb | This reminds me | 16:28 |
| clarkb | #success Infra puppetry now running puppet-4 everywhere but arm64 hosts and disabled (for cfgmgmt) hosts | 16:29 |
| openstackstatus | clarkb: Added success to Success page (https://wiki.openstack.org/wiki/Successes) | 16:29 |
| mtreinish | beekneemech: I definitely can second that sentiment. I miss infra all the time working mostly on github projects these days | 16:29 |
| *** ccamacho has quit IRC | 16:30 | |
| *** ijw has quit IRC | 16:30 | |
| *** iurygregory has quit IRC | 16:32 | |
| *** bobh has joined #openstack-infra | 16:33 | |
| *** jamesmcarthur has quit IRC | 16:34 | |
| clarkb | infra-root I see the certcheck emails | 16:36 |
| clarkb | Usually I try to get those down when they get down to about a weeks left | 16:37 |
| *** hjensas has quit IRC | 16:37 | |
| clarkb | that way we maximize cert usage on the old cert but give us time to get things replaced before expiration | 16:37 |
| clarkb | As with survey and git.airshipit.org I expect these will all get 2 year certs | 16:37 |
| corvus | clarkb, mordred: https://github.com/ansible/ansible/issues/56928 | 16:40 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Fix new mirror system errors https://review.opendev.org/661346 | 16:42 |
| corvus | infra-root: ^ that should fix the mirror cronspam | 16:43 |
| clarkb | corvus: bug seems to capture what I expected was the issue | 16:43 |
| clarkb | +2'd the fix on our end | 16:43 |
| corvus | yeah, arguably there's another bug which is "follow should be false by default if state is link" | 16:44 |
| fungi | sorry, popped out to the pub for a nice friday lunch. lots of highlights so catching back up now | 16:44 |
| corvus | but i seem to have misplaced my can-of-worms-opener, so that one will have to wait | 16:45 |
| clarkb | corvus: I think I left mine on the boat wednesday :) | 16:45 |
| *** kjackal has quit IRC | 16:45 | |
| corvus | clarkb: you should go back for it! let me know if you need help.... :) | 16:45 |
| *** smarcet has quit IRC | 16:46 | |
| fungi | best part of scrollback was beekneemech's praise. the rest i no longer care about! ;) | 16:47 |
| fungi | clarkb: i'd be cool with redoing survey to be survey.opendev.org and not renewing the survey.openstack.org cert for it since it's still more or less in beta | 16:48 |
| *** quiquell has joined #openstack-infra | 16:48 | |
| clarkb | fungi: too late it expired right after the ptg and I took care of it | 16:48 |
| fungi | oh, yeah that's right. i've repressed all memories of that week | 16:49 |
| beekneemech | :-) | 16:50 |
| *** manjeets has quit IRC | 16:51 | |
| *** diablo_rojo has quit IRC | 16:51 | |
| tosky | clarkb: the logs-dev change seems to work; I added few comments to the review and spammed a lot of people (https://review.opendev.org/#/c/648658/) | 16:55 |
| tosky | so now it's more an architectural question: should we assume that the logs viewer shows those files and remove the custom handling, or leave it in place because "you can never know"? | 16:56 |
| clarkb | tosky: I think we can probably enable the dotfile viewing on the production vhost | 16:57 |
| tosky | can we assume the same for any other zuul instance? I mean, are those settings part of the standard zuul configuration, or are those custom settings for logs.openstack.org?? | 16:58 |
| clarkb | tosky: zuul doesn't enforce log hosting so that is per deployment | 16:58 |
| tosky | clarkb: should the code be kept (and improved) in zuul-jobs then, as zuul-jobs is for general consumption, or does it make sense to at least document this? | 16:59 |
| clarkb | tosky: it is an apache specific behavior. We could add a note somewhere that if hosting logs with apache dotfiles may need special config to be indexed | 17:00 |
| openstackgerrit | Jason E. Rist proposed openstack/project-config master: Retiring TripleO-UI https://review.opendev.org/661332 | 17:01 |
| *** electrofelix has quit IRC | 17:02 | |
| *** ricolin has quit IRC | 17:03 | |
| *** jpena is now known as jpena|off | 17:04 | |
| *** ralonsoh has quit IRC | 17:06 | |
| clarkb | the change to add the gc cron failed on the nodepool idempotency check due ot project-config updates I think | 17:07 |
| *** quiquell has quit IRC | 17:07 | |
| clarkb | if it fails again I think I'll have to dig in more | 17:07 |
| *** quiquell has joined #openstack-infra | 17:08 | |
| *** adriancz has quit IRC | 17:08 | |
| corvus | i'm +2 on https://review.opendev.org/661341 :) | 17:09 |
| *** mattw4 has joined #openstack-infra | 17:09 | |
| fungi | and i'm +3 | 17:10 |
| fungi | still catching up on high-priority changes which came in while i was at the pub | 17:10 |
| Shrews | fungi: you were in a pub... on a friday... at the beach. Why did you leave again???? | 17:11 |
| fungi | who says i did? ;) | 17:12 |
| Shrews | lol | 17:12 |
| Shrews | probably best to leave before all of the out-of-towners arrive today, anyway | 17:12 |
| *** quiquell has quit IRC | 17:12 | |
| fungi | yeah, this was more of a last hurrah before we go into hiding | 17:12 |
| fungi | i'll stick my head out of my hole after labor day and if i see my shadow then that means six more weeks of tourists | 17:13 |
| clarkb | is that this weekend? | 17:13 |
| fungi | this weekend is memorial day | 17:13 |
| fungi | (in the usa) | 17:13 |
| fungi | the height of tourism here generally runs from usa memorial day weekend through usa labor day weekend | 17:14 |
| *** ykarel|away has joined #openstack-infra | 17:14 | |
| clarkb | I was debating a trip to the coast or similar this weekend but may have to skip that and do things around the house instead | 17:15 |
| fungi | as beaches go, we at least still have somewhat of a clear off-season. i envy locales like bar harbor though | 17:15 |
| *** mattw4 has quit IRC | 17:15 | |
| openstackgerrit | Clark Boylan proposed opendev/puppet-openstackci master: Apply autoindexing of dotfiles to prod vhost https://review.opendev.org/661364 | 17:18 |
| clarkb | tosky: ^ that should get it on the prod vhost | 17:18 |
| tosky | thanks | 17:19 |
| *** Weifan has joined #openstack-infra | 17:19 | |
| *** Lucas_Gray has quit IRC | 17:24 | |
| openstackgerrit | Merged opendev/puppet-openstackci master: Remove idempotency spec checks https://review.opendev.org/661341 | 17:28 |
| *** armax has quit IRC | 17:36 | |
| *** armax has joined #openstack-infra | 17:36 | |
| *** armax has quit IRC | 17:36 | |
| *** electrofelix has joined #openstack-infra | 17:38 | |
| *** bobh has quit IRC | 17:38 | |
| clarkb | I am able to reproduce RDO' | 17:43 |
| clarkb | er | 17:43 |
| clarkb | I am able to reproduce RDO's git fetch origin --prune problem | 17:43 |
| clarkb | however, interestingly it seems to delete refs then add them back again? | 17:45 |
| clarkb | makes me wonder if a specific backend is unhappy? | 17:45 |
| clarkb | ya it totally deleted a ref then add it back when I ran the command again | 17:45 |
| * clarkb tries against specific backends | 17:45 | |
| * corvus just verified that 06 is not in the lb config | 17:46 | |
| *** Weifan has quit IRC | 17:48 | |
| *** hjensas has joined #openstack-infra | 17:49 | |
| fungi | i also replied to their e-mail requesting some more refined scoping on when they saw the problem arise | 17:49 |
| *** electrofelix has quit IRC | 17:51 | |
| clarkb | I seem to have no trouble when talking to a specific backend too. (though I didn't clone from specific backend then prune so I'm gonna do that next) | 17:53 |
| clarkb | is it possible that the load balancer is causing the problems? | 17:53 |
| *** pcaruana has quit IRC | 17:53 | |
| *** e0ne has joined #openstack-infra | 17:54 | |
| fungi | could it be that differences in packing are causing serialized requests split between different backends to return nonsensical results to the client? and newer git is either better at dealing with that or makes its requests in such a way that haproxy is sending them all to the same place? | 17:56 |
| corvus | maybe we should switch from 'leastconn' to 'source' | 17:57 |
| clarkb | corvus: ya that is what I'm suspecting. But also I notice that there are fewer objects on gitea07 | 17:58 |
| corvus | though that may end up funnelling all $bigcorp requests to one server, thanks to pnat | 17:58 |
| clarkb | gitea07 had the extended outage from the git config lock file | 17:58 |
| clarkb | possible we didn't actually replicate everything against that one? | 17:58 |
| corvus | clarkb: seems likely... | 17:58 |
| corvus | we can trigger a replication to just that host | 17:58 |
| corvus | or we could go ahead and do the gerrit restart | 17:58 |
| clarkb | my hunch is that each gitea server is internally consistent but if requeists bounce between them 07 having fewer refs than the others can lead to trouble | 17:58 |
| corvus | clarkb: we can drop 07 from the lb config temporarily to verify the hypothesis | 17:59 |
| clarkb | and yup I did mirror clones and prunes confined to individual gitea servers and everything was fine | 17:59 |
| clarkb | corvus: That is a simple thing to do. Did you want to do it or should I? | 18:00 |
| corvus | i will | 18:01 |
| clarkb | I've been using openstack/puppet-keystone fwiw | 18:01 |
| corvus | we lost our haproxy docs | 18:02 |
| corvus | i'll re-add them in a few mins | 18:02 |
| clarkb | corvus: check my history or roots history for socat commands | 18:02 |
| clarkb | I did things for 06 during ptg | 18:02 |
| fungi | lost? did we inadvertently "clean them up?" | 18:02 |
| corvus | i checked out an old commit | 18:02 |
| corvus | yeah | 18:02 |
| fungi | ahh, i guess they were co-mingled with the cgit docs | 18:02 |
| *** Weifan has joined #openstack-infra | 18:03 | |
| corvus | clarkb: done; want to try now? | 18:04 |
| clarkb | trying | 18:04 |
| clarkb | corvus: it is still trying to delete refs | 18:05 |
| clarkb | (and hanging) | 18:05 |
| *** eernst has joined #openstack-infra | 18:05 | |
| corvus | i should clarify i only removed gitea07 from the https pool | 18:05 |
| corvus | it's still in the http pool, but that shouldn't matter | 18:05 |
| *** finucannot is now known as stephenfin | 18:05 | |
| clarkb | I cloned from https so wouldn't have hit the redirect even | 18:06 |
| clarkb | maybe the next thing to try is force replication against just puppet-keystone | 18:06 |
| clarkb | so ya removing 07 doesn't seem to have fixed it | 18:06 |
| corvus | clarkb: i'll add it back in, and let you force replication if you want to try that next | 18:07 |
| clarkb | k | 18:07 |
| corvus | it's back | 18:07 |
| openstackgerrit | Merged opendev/system-config master: Fix new mirror system errors https://review.opendev.org/661346 | 18:08 |
| clarkb | replication start openstack/puppet-keystone is the ssh command to trigger against a specific project | 18:08 |
| fungi | yep | 18:10 |
| clarkb | ok that seemed to help. I got through two cycles of clone --mirror and fetch --prune but third appears to have gotten stuck | 18:11 |
| clarkb | I notice that each time I cloned there are different object counts | 18:11 |
| clarkb | (could be due to different gc'ing? | 18:11 |
| fungi | i would suggest we don't --all and instead let the gerrit restart we're wanting to do today take care of that | 18:11 |
| clarkb | we might compare the packed refs files between the gitea nodes? | 18:12 |
| clarkb | fourth attempt also hanging | 18:12 |
| clarkb | the packed-refs files do differ according to md5sum | 18:14 |
| *** eernst has quit IRC | 18:18 | |
| fungi | yeah, possible that if part of the request ends up at a different backend then the client could get confused | 18:19 |
| clarkb | ya so I've confirmed that 07 has fewer packed refs than 01, but 07 has the refs they just happen to be in refs/changes and aren't packed | 18:19 |
| clarkb | and if the client is fetching from one location then another it may get confused? | 18:20 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add gitea docs https://review.opendev.org/661370 | 18:20 |
| clarkb | we likely do need to set the balance type to source | 18:20 |
| corvus | according to the docs, we used to | 18:20 |
| clarkb | either that or someone get the cluster into sync as far as repo format goes | 18:20 |
| fungi | maybe newer git pipelines subsequent requests? | 18:20 |
| corvus | i wonder why we changed it | 18:20 |
| fungi | but yeah, sounds like we should just go back to source hash | 18:21 |
| clarkb | s/someone/somehow/ | 18:21 |
| corvus | our gc crons are always going to race new refs :/ | 18:22 |
| clarkb | I expect that none of this would be an issue if they didn't both mirror and prune | 18:24 |
| clarkb | one or the other is likely fine | 18:24 |
| corvus | https://review.opendev.org/220340 is relevant | 18:24 |
| clarkb | that said mirroring and pruning should be valid | 18:24 |
| fungi | yeah, even if we pointed all the servers at a single backend filesystem (via afs or something) the cron could still cause this by packing refs between subrequests | 18:24 |
| corvus | fungi: though that is far less likely | 18:25 |
| clarkb | corvus: fungi it is also possible that gitea runs the server side with more state between requests? | 18:26 |
| fungi | i agree, it's a less likely race, but it is still a potential race | 18:26 |
| corvus | https://review.opendev.org/219709 is the previous change | 18:26 |
| clarkb | cgit may have cared less about this | 18:26 |
| *** e0ne has quit IRC | 18:26 | |
| clarkb | or rather git web or whatever the cgi thing was called | 18:26 |
| fungi | git smart http backend | 18:27 |
| corvus | (depends on whether they had /cgit/ in their clone urls) | 18:27 |
| corvus | https://review.opendev.org/193838 is the previous to that | 18:27 |
| fungi | oh, right, cgit also had its own | 18:28 |
| clarkb | and ya actually strace shows it hangs on a read | 18:29 |
| *** armax has joined #openstack-infra | 18:29 | |
| clarkb | so it could be that gitea1 get requiests and starts handling it then gitea02 gets next requests and doesn't have the state to properly handle it? | 18:29 |
| clarkb | and so connection hangs | 18:29 |
| clarkb | though I suppose that state could be the delta in repo structure | 18:29 |
| clarkb | we do have a lot of headroom on the cpus for the gitea servers | 18:30 |
| clarkb | we can probably try source and monitor it | 18:30 |
| corvus | https://review.opendev.org/43359 is the previous change | 18:31 |
| *** e0ne has joined #openstack-infra | 18:31 | |
| corvus | what i'm getting out of this is that we have alternated between being more concerned about data integrity vs load. :) | 18:31 |
| corvus | those changes list all the potential pitfalls -- including not only uneven load because of NAT, but also due to clones of nova piling up | 18:32 |
| corvus | ^ pitfalls of source | 18:33 |
| corvus | and of course, we previously identified the pitfalls of leastcon -- deltas between hosts | 18:33 |
| fungi | our alternating between concerns of data integrity and load distribution is not a surprising observation | 18:33 |
| fungi | but it is good to weigh in the current determination | 18:33 |
| corvus | i would really love a source-leastconn | 18:33 |
| fungi | its absence suggests there's probably a complexity in actually implementing one which we're not aware of | 18:34 |
| corvus | a lot of state | 18:35 |
| corvus | and cpu | 18:35 |
| fungi | yeah | 18:35 |
| fungi | the stereotypical villians in any high-throughput proxy tale | 18:35 |
| corvus | if we terminated tls at the lb, we might be able to use gitea session cookies | 18:36 |
| * mordred is back from his sandwich and notices fun with load balancers and git | 18:36 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Store hold requests in zookeeper https://review.opendev.org/661114 | 18:36 |
| fungi | is git smart enough to actually return cookies? | 18:37 |
| clarkb | I need to pop out now for late breakfast/early lunch. Will be back in a bit. I'm happy to try source as an easy change that will give us more data | 18:37 |
| corvus | no idea! | 18:37 |
| corvus | clarkb: if we want to do that, we should really fix up the grafana dashboard -- http://grafana.openstack.org/d/MYvSHcSiz/git-load-balancer?orgId=1 | 18:38 |
| corvus | i mean, we need to do that anyway | 18:38 |
| corvus | but i'd really like to have all our metrics before we do that | 18:38 |
| corvus | the stats socket is available on the host | 18:38 |
| corvus | it should just be a matter of getting the haxproxy-stats script running and pointed at the right location | 18:38 |
| fungi | if it helps, the daily gc cron firing across all backend son a roughly synchronous schedule coupled with fairly synchronous replication to them will probably make this less painful | 18:39 |
| clarkb | fungi: does gc always pack objects? | 18:39 |
| fungi | dunno, i really am not sure | 18:40 |
| fungi | what factors could cause it to only sometimes pack objects> | 18:40 |
| fungi | ? | 18:40 |
| corvus | could be that's why we didn't notice it before -- everything was more in sync | 18:40 |
| mordred | yah. and occasional hiccups could just be explained away as "sometimes the internet messes up | 18:41 |
| clarkb | fungi: disk usage heuristics maybe? | 18:42 |
| corvus | i also need to do lunch; unless anyone is interested in jumping on it now, i'll work on getting haproxy-statsd up after lunch | 18:43 |
| corvus | https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/git | 18:44 |
| corvus | i guess that should be a container image now | 18:44 |
| *** xek has quit IRC | 18:47 | |
| *** pcaruana has joined #openstack-infra | 18:48 | |
| fungi | firefox also continues to give me corruption/protocol errors on opendev.org urls after a gitea restart, forcing me to force reload the first new tab i open. weird browser-side caching side effects i guess | 18:48 |
| *** xek has joined #openstack-infra | 18:48 | |
| *** pcaruana has quit IRC | 18:48 | |
| openstackgerrit | Merged opendev/system-config master: Add cron to gc on gitea servers https://review.opendev.org/661102 | 18:49 |
| *** xek_ has joined #openstack-infra | 18:53 | |
| *** xek has quit IRC | 18:54 | |
| *** calebb has joined #openstack-infra | 19:01 | |
| *** xek__ has joined #openstack-infra | 19:11 | |
| *** eernst has joined #openstack-infra | 19:11 | |
| fungi | "Corrupted Content Error: The site at https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/git has experienced a network protocol violation that cannot be repaired. The page you are trying to view cannot be shown because an error in the data transmission was detected. Please contact the website owners to inform them of this problem." | 19:14 |
| *** xek_ has quit IRC | 19:14 | |
| fungi | and the "try again" button it provides just repeats that error. but a force refresh in the tab brings up the expected content | 19:15 |
| *** tosky has quit IRC | 19:16 | |
| fungi | next time i see it, i'll try going to about:debugging#workers and unregistering https://opendev.org/serviceworker.js | 19:18 |
| *** whoami-rajat has joined #openstack-infra | 19:18 | |
| fungi | see if that's involved at all | 19:18 |
| fungi | though according to the warning on that panel, i may also have to disable "multiple content processes" in the browser | 19:20 |
| corvus | multiple tabs open? | 19:20 |
| fungi | yeah | 19:20 |
| *** rfarr__ has quit IRC | 19:24 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: WIP: Add haproxy-statsd to haproxy server https://review.opendev.org/661383 | 19:34 |
| corvus | clarkb, fungi, mordred: ^ | 19:34 |
| *** ykarel|away has quit IRC | 19:35 | |
| *** eernst has quit IRC | 19:35 | |
| *** eernst has joined #openstack-infra | 19:36 | |
| corvus | i believe that's ready to go -- i only set WIP because i cut the jobs list to get a quicker first test run | 19:36 |
| *** panda|rover has quit IRC | 19:36 | |
| mordred | corvus: looks reasonable to me | 19:36 |
| mordred | clarkb: am I supposed to be doing static.o.o or status.o.o ? the two names are very similar ... | 19:37 |
| *** panda has joined #openstack-infra | 19:40 | |
| fungi | mordred: status | 19:40 |
| fungi | ianw was working on static, last i recall | 19:41 |
| mordred | fungi: awesome. | 19:41 |
| mordred | there are some things we run on status that it seems like maybe we should stop running ... | 19:42 |
| mordred | for instance - looking at apache access logs, it seems like reviewday is there solely for the bots | 19:47 |
| fungi | the nova team has been discussing reviewday in their weekly meetings at least, and i unstuck it last week at their request | 19:49 |
| mordred | oh wow | 19:49 |
| mordred | ok. I guess there is someone still using it then | 19:49 |
| mordred | it has 50 non-bot hits in the last 4 days, fwiw (by my best stab at a quick filter of that) | 19:50 |
| mordred | fungi: what about bugday? http://status.openstack.org/bugday/ | 19:50 |
| corvus | does reviewday have a maintainer to opendevify it? | 19:50 |
| fungi | looks like it's basically dprince, judging from https://opendev.org/openstack/reviewday/commits/branch/master | 19:52 |
| fungi | and last commit of substance was a year ago | 19:52 |
| fungi | mordred: no clue about bugday | 19:52 |
| corvus | though, interestingly, that commit made it more opendev friendly :) | 19:52 |
| corvus | https://opendev.org/openstack/reviewday/commit/cca8e6ed63c88eb3969ae84e85ff55e4899fe202 | 19:52 |
| mordred | yeah | 19:52 |
| fungi | yup, i thought the same | 19:52 |
| fungi | mordred: i haven't heard anyone ask about bugday in a while, so it may not be in use at all | 19:53 |
| corvus | maybe reviewday is ready, or only needs minor tweaks (like stylesheets) | 19:53 |
| mordred | mostly trying to remove cruft before updating, rather than the other way around | 19:53 |
| fungi | nova folks also may only have a passing interest in reviewday, i'm not sure | 19:54 |
| fungi | the other thing it's been providing is custom review dashboards, but those could probably get relocated | 19:54 |
| fungi | s/dashboards/one dashboard/ | 19:56 |
| fungi | the neutron one | 19:56 |
| clarkb | back | 19:56 |
| openstackgerrit | Andreas Jaeger proposed openstack/os-testr master: Fix warning message with double "to" https://review.opendev.org/661384 | 19:56 |
| clarkb | I think ttx was involvedi nreviewday once upon a time | 19:56 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove bugday from status.o.o https://review.opendev.org/661385 | 19:56 |
| mordred | there ^^ sake of argument | 19:57 |
| fungi | makes for an excellent strawman at least | 19:57 |
| Weifan | Sorry to interrupt..Does anyone know if there is a way to let zuul run py35 or later for openstack-tox-pep8? | 19:58 |
| Weifan | Currently if I configure tox_envlist to py35 or py36 in .zuul.yaml to ,it would throw an InterpreterNotFound Error. | 19:58 |
| Weifan | The issue is that, by default it used python3.4 for pep8, and it would fail when installing requriements.txt and test-requirements.txt, using upper-constraints in requirements, as requirements repo no longer care about python3.4. | 19:58 |
| Weifan | Or is it suggested we use py27 for openstack-tox-pep8? | 19:58 |
| *** rfarr__ has joined #openstack-infra | 19:58 | |
| fungi | Weifan: switch to a newer node type | 19:59 |
| fungi | ubuntu-trusty came with python 3.4, ubuntu-xenial with 3.5, ubuntu-bionic has 3.6 and 3.7 available | 19:59 |
| clarkb | corvus: the haproxy statsd container change lgtm | 19:59 |
| *** rfarr_ has joined #openstack-infra | 20:00 | |
| Weifan | fungi: thanks, i'll try that | 20:00 |
| clarkb | our default nodeset node is bionic now isn't it? | 20:01 |
| fungi | yes, but many jobs set an explicit nodeset | 20:02 |
| fungi | Weifan: for context, openstack kilo through newton used ubuntu-trusty for testing, ocata through rocky used ubuntu-xenial, and stein through now use ubuntu-bionic | 20:02 |
| *** rfarr__ has quit IRC | 20:03 | |
| Weifan | that's a bit weird, as the issue is on master | 20:03 |
| fungi | Weifan: have an example change where you're seeing it? | 20:03 |
| fungi | zuul job artifacts include all the inheritance information necessary to determine where it's being set | 20:04 |
| Weifan | i did some hack to workaround it, can't really find the build report.. but heres what the .zuul.yaml looks like: | 20:07 |
| Weifan | - project: | 20:07 |
| Weifan | templates: | 20:07 |
| Weifan | - openstack-python-jobs-neutron | 20:07 |
| Weifan | - openstack-python35-jobs-neutron | 20:07 |
| Weifan | - openstack-python36-jobs-neutron | 20:07 |
| Weifan | - openstack-python37-jobs-neutron | 20:07 |
| Weifan | project is x/networking-bigswitch | 20:08 |
| Weifan | this would fail to install requirements due to python version issue | 20:09 |
| Weifan | only for pep8 | 20:09 |
| clarkb | and you don't have logs? | 20:09 |
| clarkb | (it is really hard for us to debug things without a link to examples) | 20:09 |
| fungi | either a link to a change the failure was reported on, or a link to the logs from that | 20:10 |
| Weifan | i made a different commit so couldn't find the old one | 20:10 |
| openstackgerrit | Andreas Jaeger proposed openstack/os-testr master: Fix warning message with double "to" https://review.opendev.org/661384 | 20:10 |
| Weifan | https://review.opendev.org/#/c/661074/ | 20:10 |
| Weifan | not sure if you can find build history.. | 20:10 |
| clarkb | https://opendev.org/x/networking-bigswitch/src/branch/master/.zuul.yaml#L12 | 20:10 |
| clarkb | that may explain it? | 20:10 |
| Weifan | thats the only one | 20:11 |
| Weifan | some of the commits i was trying | 20:11 |
| Weifan | had that removed | 20:11 |
| fungi | http://logs.openstack.org/74/661074/1/check/openstack-tox-pep8/d0a17ba/zuul-info/inventory.yaml | 20:11 |
| fungi | label: ubuntu-trusty | 20:11 |
| openstackgerrit | Andreas Jaeger proposed openstack/os-testr master: Fix warning message with double "to" https://review.opendev.org/661384 | 20:11 |
| fungi | the inheritance path on that is somewhat lengthy, so will take a moment to dig through | 20:12 |
| clarkb | https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/project-templates.yaml#L855-L874 it is going through that | 20:13 |
| clarkb | so my suspicion is probably correct | 20:13 |
| clarkb | https://opendev.org/x/networking-bigswitch/src/branch/master/.zuul.yaml#L12 remove that line | 20:13 |
| fungi | yeah | 20:13 |
| fungi | i agree | 20:14 |
| Weifan | so i think if i remove the openstack-python-jobs-neutron, and just add py27 and pep8 jobs, it should be fixed? | 20:14 |
| fungi | remove openstack-python-jobs-trusty | 20:15 |
| clarkb | Weifan: those jobs are already added. https://opendev.org/x/networking-bigswitch/src/branch/master/.zuul.yaml#L15-L29 | 20:15 |
| fungi | you already include openstack-tox-pep8 and openstack-tox-py27 explicitly anyway, and those will fall through to using ubuntu-bionic | 20:15 |
| Weifan | these are the failing reasons of old failures... but for the PRs it was a bit different.. i already removed it in all my PRs | 20:17 |
| Weifan | but it was still failing due to py34 | 20:17 |
| Weifan | that is why i pasted the .zuul.yaml that i remembered | 20:18 |
| fungi | yes, py34 was available on ubuntu-trusty but on ubuntu-xenial it's py35 and ubuntu-bionic is where py36 and py37 jobs will be run | 20:18 |
| Weifan | issue was that when i looked at the log, it said python is 3.4 | 20:19 |
| Weifan | i think i tried 3.5 and 3.6 | 20:19 |
| Weifan | anyway, i'll try the new nodeset | 20:19 |
| Weifan | thanks for the help | 20:19 |
| *** slaweq has quit IRC | 20:19 | |
| fungi | then that was probably an ubuntu-trusty node, if you omit the nodeset override for those job variants then 3.6 should work | 20:20 |
| mordred | infra-root: ooh - we have a too-many-open-files notice on codesearch | 20:20 |
| fungi | because it will run on an ubuntu-bionic node where python3.6 is the default python3 | 20:20 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove python-jobs-trusty https://review.opendev.org/661389 | 20:21 |
| AJaeger | Weifan: you need this as well ^ | 20:21 |
| *** Lucas_Gray has joined #openstack-infra | 20:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove in-tree job from networking-bigswitch https://review.opendev.org/661390 | 20:23 |
| AJaeger | Weifan: and this is also important ^ | 20:23 |
| clarkb | ~9 minutes to haproxy statsd container test results | 20:24 |
| corvus | the sheer amount of stuff this is testing with no additional test-specific code is amazing :) | 20:27 |
| clarkb | I'm so happy the registry stuff is stable now (or appears to be and now I am knocking on my desk) | 20:27 |
| Weifan | Ajaeger: thanks:) | 20:29 |
| mordred | corvus: ++ | 20:29 |
| clarkb | speaking of stable registry stuff, any idea if placement has looked into that more mordred ? | 20:30 |
| AJaeger | Weifan: I commented on your review, you can remove even more. First the project-config changes need to merge before your change in networking-bigswitch can pass | 20:30 |
| mordred | clarkb: nope - no idea | 20:31 |
| corvus | it's green | 20:32 |
| corvus | i guess i didn't need to drop the jobs | 20:33 |
| *** EvilienM is now known as EmilienM | 20:33 | |
| AJaeger | Weifan: also, backport https://review.opendev.org/661074 to your stable branches... | 20:33 |
| corvus | oh, there is an error though: http://logs.openstack.org/83/661383/1/check/system-config-run-gitea/46a15d9/gitea-lb01.opendev.org/docker/haproxydocker_haproxy-statsd_1.txt | 20:33 |
| corvus | (not executable) | 20:34 |
| Weifan | Ajaeger: I was thinking about making another commit to remove the pep8 thing once those changes are merged | 20:34 |
| clarkb | corvus: yay for testing | 20:34 |
| clarkb | corvus: as a heads up I pointed airship via https://review.opendev.org/#/c/653066/5 to the zuul-jobs docker stuff | 20:34 |
| corvus | i'm not sure how to verify that it started up | 20:35 |
| Weifan | Ajaeger: for stable banches, they are a bit different from master, i'll fix them with a different pr.. | 20:35 |
| clarkb | corvus: ps? | 20:35 |
| clarkb | I guess it is a one shot thing so ps won't help much | 20:35 |
| fungi | or may be able to check references on the executable | 20:35 |
| clarkb | corvus: could tcpdump for port 8125 traffic | 20:35 |
| fungi | oh, right, not persistent process | 20:35 |
| AJaeger | Weifan: same idea ;) | 20:36 |
| fungi | so same problem as ps | 20:36 |
| corvus | well, it is persistent | 20:36 |
| corvus | but right now it's in a crash loop | 20:36 |
| corvus | so a ps or docker ps might race | 20:36 |
| clarkb | oh it does run in a while loop. | 20:36 |
| fungi | also a great point | 20:36 |
| clarkb | corvus: if you fix the crash wouldn't it be ps'able? | 20:36 |
| fungi | could result in false successes | 20:36 |
| AJaeger | Weifan: your change will not pass this run since project-config is still active, so wait until that is merged and then update ;) | 20:36 |
| clarkb | fungi: oh I see | 20:37 |
| fungi | clarkb: it could be ps'able now if you get lucky | 20:37 |
| corvus | right that | 20:37 |
| clarkb | could check that docker only started one | 20:37 |
| clarkb | via docker ps -a ? | 20:37 |
| Weifan | Ajaeger: Sure :) | 20:37 |
| corvus | oh hrm. ok i'll look into that | 20:37 |
| *** aspiers has quit IRC | 20:38 | |
| fungi | is log inspection an option? | 20:39 |
| corvus | clarkb: should i add https://docs.opendev.org/opendev/base-jobs/latest/docker-image.html to your airship change too? | 20:39 |
| corvus | fungi: yes | 20:39 |
| clarkb | corvus: oh probably | 20:39 |
| fungi | or, lots of extra work but, stand up a statsd service, configure haproxy-statsd.py to send to it, and then check that stats were received and recorded? | 20:40 |
| clarkb | fungi: ya the tcpdump idea was cheap version of ^ | 20:41 |
| clarkb | I think both will work | 20:41 |
| corvus | maybe after we rework the statsd stuff to containers? | 20:41 |
| fungi | right, both that and the tcpdump option at least confirm the service has started sufficiently to send a stats datagram | 20:42 |
| fungi | anyway, i asked about logging because i see the script is already set up to use python logging | 20:42 |
| *** aspiers has joined #openstack-infra | 20:43 | |
| openstackgerrit | Merged openstack/project-config master: Remove python-jobs-trusty https://review.opendev.org/661389 | 20:44 |
| openstackgerrit | Merged openstack/project-config master: Remove in-tree job from networking-bigswitch https://review.opendev.org/661390 | 20:44 |
| *** smarcet has joined #openstack-infra | 20:46 | |
| mordred | clarkb: there is part of me that wants to do a more extreme rework of some of the things on status - and another part of me that does not want to do that and is trying to convince the first part to shut up | 21:00 |
| clarkb | mordred: fwiw the way I've approached it is trusty is rip already. Xenial gives us ~2 years to scratch those itches | 21:01 |
| clarkb | so doing the minimum to get to xenial is a win | 21:01 |
| mordred | because the rabbit hole could go deep | 21:01 |
| mordred | yeah | 21:01 |
| mordred | clarkb: what do we use the jenkins user for on status.o.o ? | 21:03 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add haproxy-statsd to haproxy server https://review.opendev.org/661383 | 21:03 |
| clarkb | mordred: uh | 21:03 |
| mordred | clarkb: I can't find anything we use it for | 21:04 |
| corvus | that's ^ updated with a chmod a+x and a test that it's running | 21:04 |
| corvus | i used docker inspect and parsed the json in testinfra | 21:04 |
| clarkb | mordred: that may be from when we had the recheck tracker? | 21:04 |
| clarkb | corvus: do you need to tell testinfra to run docker privileged? | 21:05 |
| clarkb | otherwise that looks great | 21:06 |
| fungi | corvus: i'm probably blind, where was the chmod added? | 21:06 |
| clarkb | fungi: its on the file in git | 21:06 |
| fungi | OH! | 21:06 |
| fungi | yes, blind | 21:06 |
| clarkb | gerrit web ui shows it as new mode at top of "diff" that isn't a diff | 21:06 |
| fungi | thanks | 21:06 |
| fungi | gertty doesn't actually show the mode change, but does show an empty diff of that file which should have tipped me off | 21:07 |
| *** sreejithp_ has quit IRC | 21:07 | |
| corvus | clarkb: we have a "docker exec" call in test_gitea so i think that'll work | 21:10 |
| *** slaweq has joined #openstack-infra | 21:11 | |
| *** sreejithp has joined #openstack-infra | 21:11 | |
| clarkb | I've approved it | 21:12 |
| *** mriedem has quit IRC | 21:13 | |
| fungi | ~when do we want gerrit restarted? probably soonish i guess so we're still around in case it fails to restart | 21:13 |
| fungi | or would we rather wait for git gc to fire on them all over the weekend and then restart early in the week? | 21:14 |
| fungi | i'm up for a restart any time now, but understand if there's a concern that will muddy our investigation into the suspected load balancing behaviors | 21:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove jenkins user from status.o.o https://review.opendev.org/661399 | 21:15 |
| corvus | i'm okay putting off the restart a bit in favor of lb investigations | 21:17 |
| fungi | fine by me, it was really just for the gitweb->gitea link swap, which isn't urgent | 21:18 |
| corvus | also, as we continue or work to make more system config playbooks, we're really going to need the zuul feature that runs a job when its own configuration changes, otherwise, we'll get to the point where .zuul.yaml changes use 10% of our total quota :) | 21:19 |
| corvus | i count ~43 nodes for this change | 21:22 |
| corvus | so, currently only 5% of our quota. | 21:23 |
| *** slaweq has quit IRC | 21:24 | |
| *** rossella_s has quit IRC | 21:24 | |
| mordred | corvus: ++ | 21:25 |
| clarkb | wow | 21:25 |
| *** rossella_s has joined #openstack-infra | 21:27 | |
| *** rh-jelabarre has quit IRC | 21:27 | |
| *** whoami-rajat has quit IRC | 21:28 | |
| *** mattw4 has joined #openstack-infra | 21:31 | |
| *** xek__ has quit IRC | 21:31 | |
| clarkb | fungi: https://www.nature.com/articles/d41586-019-01629-1 | 21:36 |
| *** smarcet has quit IRC | 21:36 | |
| corvus | clarkb: i finished green mars on my recent trip -- still enjoying that, and i got the first 3 books of the expanse series (they were in a discounted box set...) | 21:39 |
| *** e0ne has quit IRC | 21:40 | |
| clarkb | corvus: I just picked up red moon and am about 100 pages into it | 21:40 |
| clarkb | I expect you'd probably enjoy it but I am nowhere near the end | 21:40 |
| fungi | i'm all about setbacks, evolutionary or otherwise | 21:40 |
| *** sarob has joined #openstack-infra | 21:41 | |
| *** sarob has quit IRC | 21:41 | |
| mordred | I'm in th emiddle of season 2 of the expanse now and enjoying it | 21:43 |
| corvus | mordred: clarkb enjoyed the first few books and persia enjoyed all of them, so i figured i'd at least grab a handful and see where i ended up :) | 21:44 |
| corvus | all jobs green -- just need to do it one more time and it'll merge :) | 21:56 |
| *** tdasilva has quit IRC | 22:06 | |
| *** eernst has quit IRC | 22:07 | |
| *** hwoarang has quit IRC | 22:08 | |
| *** slaweq has joined #openstack-infra | 22:11 | |
| *** jamesmcarthur has joined #openstack-infra | 22:12 | |
| *** hwoarang has joined #openstack-infra | 22:14 | |
| *** rlandy has quit IRC | 22:16 | |
| *** jamesmcarthur has quit IRC | 22:19 | |
| clarkb | it failed the run gitea change | 22:22 |
| clarkb | s/change/job/ | 22:22 |
| clarkb | http://logs.openstack.org/83/661383/2/gate/system-config-run-gitea/ae355f2/job-output.txt.gz#_2019-05-24_22_12_40_876021 | 22:22 |
| clarkb | does that imply it was trying to pull haproxy-statsd from dockerhub? | 22:23 |
| clarkb | possibly because docker-compose pull runs as a different user than the one we set up with the buildset registry? | 22:24 |
| *** slaweq has quit IRC | 22:25 | |
| *** rfarr_ has quit IRC | 22:25 | |
| clarkb | hrm I would expect that type of failure to affect the check jobs too | 22:30 |
| clarkb | http://logs.openstack.org/83/661383/2/gate/system-config-upload-image-haproxy-statsd/8c30efb/job-output.txt.gz#_2019-05-24_22_15_14_195643 starting frmo there we pushed a tag to buildset registry, then dockerhub, then intermedaite registry | 22:32 |
| clarkb | so the image should've been findable even if it looked at another location | 22:32 |
| clarkb | is it possible that is a race between upload to dockerhub and it being pullable? | 22:35 |
| *** _erlon_ has quit IRC | 22:36 | |
| clarkb | oh :latest is not found | 22:36 |
| corvus | it shouldn't be pulling from dockerhub there, it should still be using the buildset registry | 22:36 |
| clarkb | http://logs.openstack.org/83/661383/2/gate/system-config-run-gitea/ae355f2/bridge.openstack.org/ara-report/result/220baacd-fc00-4a84-b97a-be304b864b6d/ and that is what we pull | 22:36 |
| clarkb | ah ok well at least I can reproduce the failure locally because :latest doesn't exist on dockerhub | 22:37 |
| clarkb | so maybe it isn't pulling from buildset registry for some reason? | 22:37 |
| clarkb | http://logs.openstack.org/83/661383/2/gate/system-config-upload-image-haproxy-statsd/8c30efb/job-output.txt.gz#_2019-05-24_22_15_12_614192 the build side did tag a :latest | 22:38 |
| corvus | use-buildset-registry was skipped | 22:39 |
| *** Weifan has quit IRC | 22:39 | |
| *** Weifan has joined #openstack-infra | 22:40 | |
| corvus | ooooh | 22:40 |
| corvus | i see it | 22:40 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add haproxy-statsd to haproxy server https://review.opendev.org/661383 | 22:40 |
| corvus | legit error :) | 22:40 |
| clarkb | ah the check vs gate jobs | 22:41 |
| fungi | oh, so it fired too early? | 22:41 |
| corvus | yep, and that caused the gitea job not to be a dependency of the image build job | 22:41 |
| fungi | makes sense | 22:42 |
| *** Weifan has quit IRC | 22:44 | |
| clarkb | corvus: also we don't put the image artifact data in the inventory file? Are those -e'd in? | 22:45 |
| *** Weifan has joined #openstack-infra | 22:48 | |
| *** Weifan has quit IRC | 22:49 | |
| *** Weifan has joined #openstack-infra | 22:50 | |
| *** jamesmcarthur has joined #openstack-infra | 22:50 | |
| *** Weifan has quit IRC | 22:54 | |
| *** jamesmcarthur has quit IRC | 22:57 | |
| corvus | clarkb: i think it should end up in the inventory? but not in this case since there was no dependency | 23:01 |
| corvus | rather, not in the error case | 23:01 |
| corvus | clarkb: yeah, here it is from the previous correct check run: http://logs.openstack.org/83/661383/2/check/system-config-run-gitea/ed72e69/zuul-info/inventory.yaml | 23:02 |
| clarkb | gotcha | 23:02 |
| corvus | see "artifacts:" | 23:02 |
| *** rcernin has joined #openstack-infra | 23:04 | |
| *** hwoarang has quit IRC | 23:08 | |
| *** armax has quit IRC | 23:08 | |
| *** hwoarang has joined #openstack-infra | 23:09 | |
| *** mattw4 has quit IRC | 23:09 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** rcernin has quit IRC | 23:11 | |
| *** aaronsheffield has quit IRC | 23:13 | |
| *** hwoarang has quit IRC | 23:14 | |
| *** hwoarang has joined #openstack-infra | 23:14 | |
| *** Weifan has joined #openstack-infra | 23:16 | |
| *** slaweq has quit IRC | 23:24 | |
| *** jamesmcarthur has joined #openstack-infra | 23:33 | |
| *** hwoarang has quit IRC | 23:35 | |
| *** hwoarang has joined #openstack-infra | 23:36 | |
| *** jamesmcarthur has quit IRC | 23:37 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!