| sean-k-mooney | was there a zuul restart earlier? | 00:00 |
|---|---|---|
| fungi | yes, there was | 00:00 |
| sean-k-mooney | ok that makes sense | 00:00 |
| fungi | #status log zuul scheduler was restarted at 20:41z | 00:00 |
| sean-k-mooney | i kicked off something 6 hours ago and its still hadnt started | 00:00 |
| fungi | seems we forgot to log that | 00:01 |
| fungi | openstackstatus is also ignoring me | 00:01 |
| sean-k-mooney | no worries im going to reproduce it locally and ill check the job in the morning | 00:01 |
| openstackstatus | fungi: finished logging | 00:02 |
| fungi | oh, there it goes | 00:02 |
| fungi | that was a lengthy delay | 00:02 |
| sean-k-mooney | it might need a reboot too :) | 00:02 |
| fungi | i think it's just tired | 00:02 |
| mordred | have you tried turning it off and on again? | 00:04 |
| *** gyee has quit IRC | 00:04 | |
| sean-k-mooney | mordred: while i think of it if you have an opipion on the nameing of the nova evacuate command equivalent in osc feel free to chime in https://review.opendev.org/#/c/643578/ | 00:04 |
| sean-k-mooney | ill likely send a mail to the list about it but you expressed an interst at the fourm session so just an fyi | 00:05 |
| * sean-k-mooney or rather mordred didnt run seaming out of the room when evacuate was mentions so is interested by default | 00:06 | |
| *** jamesmcarthur has joined #openstack-infra | 00:08 | |
| *** eernst has joined #openstack-infra | 00:17 | |
| *** jamesmcarthur has quit IRC | 00:18 | |
| *** jamesmcarthur has joined #openstack-infra | 00:19 | |
| *** Weifan has quit IRC | 00:21 | |
| *** aaronsheffield has quit IRC | 00:24 | |
| *** ekultails has quit IRC | 00:24 | |
| *** jamesmcarthur has quit IRC | 00:25 | |
| pabelanger | oh, ha. zuul-scheduler was only restarted today, not executors. That explains why we don't have full 2.8 support | 00:33 |
| pabelanger | for ansible | 00:34 |
| *** hwoarang has quit IRC | 00:36 | |
| *** hwoarang has joined #openstack-infra | 00:37 | |
| *** jamesmcarthur has joined #openstack-infra | 00:52 | |
| *** eernst has quit IRC | 00:54 | |
| *** eernst has joined #openstack-infra | 00:54 | |
| *** slaweq has joined #openstack-infra | 00:57 | |
| *** jamesmcarthur has quit IRC | 00:58 | |
| *** eernst has quit IRC | 00:58 | |
| *** lseki has quit IRC | 01:00 | |
| *** slaweq has quit IRC | 01:01 | |
| *** sarob has joined #openstack-infra | 01:06 | |
| fungi | yes, i'm happy to help with executor restarts tomorrow, but it's a bit late in the evening for me now | 01:09 |
| pabelanger | wfm, also will be around to assist too | 01:15 |
| openstackgerrit | Merged opendev/system-config master: Switch to upstream and upgrade gitea to current master (pre-1.9.0) https://review.opendev.org/660848 | 01:15 |
| fungi | infra-root: we should probably keep an eye on opendev.org with that ^ merging | 01:16 |
| fungi | hopefully it deploys smoothly and there are no new regressions | 01:17 |
| *** jamesmcarthur has joined #openstack-infra | 01:28 | |
| *** whoami-rajat has joined #openstack-infra | 01:30 | |
| *** jamesmcarthur has quit IRC | 01:33 | |
| *** ricolin has joined #openstack-infra | 01:36 | |
| *** gregoryo has joined #openstack-infra | 01:41 | |
| *** jamesmcarthur has joined #openstack-infra | 01:45 | |
| fungi | Gitea Version: 6eb53ac570ab9af51fc9cbd79f1db782edce57e0 | 01:46 |
| fungi | looks like it's deployed now | 01:46 |
| fungi | dtroyer: corvus: mordred: https://opendev.org/starlingx/integ/ is showing normal content for me now | 01:49 |
| fungi | poking around the site, stuff's looking normal so far | 01:49 |
| *** jamesmcarthur has quit IRC | 01:49 | |
| fungi | unless someone else notices a problem, i think we're good | 01:50 |
| *** zhurong has joined #openstack-infra | 01:50 | |
| *** apetrich has quit IRC | 01:57 | |
| *** michael-beaver has quit IRC | 02:01 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| logan- | the new gitea upgrade has blame view! awesome | 02:11 |
| fungi | yup! | 02:12 |
| *** jamesmcarthur has joined #openstack-infra | 02:12 | |
| *** slaweq has quit IRC | 02:15 | |
| *** jamesmcarthur has quit IRC | 02:17 | |
| *** ykarel|away has joined #openstack-infra | 02:20 | |
| dtroyer | fungi, corvus: yup, looks good here too, thanks! | 02:25 |
| *** irclogbot_0 has quit IRC | 02:26 | |
| *** irclogbot_3 has joined #openstack-infra | 02:29 | |
| *** jamesmcarthur has joined #openstack-infra | 02:35 | |
| *** bobh has joined #openstack-infra | 02:39 | |
| *** sreejithp has joined #openstack-infra | 02:39 | |
| *** sreejithp_ has joined #openstack-infra | 02:41 | |
| *** sreejithp has quit IRC | 02:44 | |
| openstackgerrit | Merged openstack/pbr master: Update Sphinx requirement https://review.opendev.org/659546 | 02:49 |
| *** nicolasbock has quit IRC | 02:51 | |
| *** _erlon_ has quit IRC | 02:52 | |
| *** jamesmcarthur has quit IRC | 03:05 | |
| *** bobh has quit IRC | 03:12 | |
| *** rlandy|bbl is now known as rlandy | 03:14 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:16 | |
| *** sarob has quit IRC | 03:20 | |
| *** rlandy has quit IRC | 03:20 | |
| *** jamesmcarthur has joined #openstack-infra | 03:26 | |
| *** bhavikdbavishi has quit IRC | 03:45 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:45 | |
| openstackgerrit | dongwenjuan proposed x/gearman-plugin master: add project parameters https://review.opendev.org/660873 | 03:46 |
| *** gregoryo has quit IRC | 03:47 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: Add nodesets API route https://review.opendev.org/605877 | 03:48 |
| *** jamesmcarthur has quit IRC | 03:48 | |
| *** samueldmq has quit IRC | 03:50 | |
| *** yikun_ has quit IRC | 03:50 | |
| *** sreejithp_ has quit IRC | 03:51 | |
| *** jamesmcarthur has joined #openstack-infra | 03:56 | |
| *** bhavikdbavishi has quit IRC | 04:06 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:07 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** jamesmcarthur has quit IRC | 04:13 | |
| *** slaweq has quit IRC | 04:16 | |
| *** ykarel|away has quit IRC | 04:17 | |
| *** auristor has quit IRC | 04:23 | |
| *** udesale has joined #openstack-infra | 04:28 | |
| *** ykarel|away has joined #openstack-infra | 04:32 | |
| *** auristor has joined #openstack-infra | 04:32 | |
| AJaeger | frickler: good morning! the api jobs are broken due to sphinx/python 3, I think https://review.opendev.org/659975 and https://review.opendev.org/659976 will fix it as side effect - could you review, please? | 04:37 |
| *** auristor has quit IRC | 04:39 | |
| *** jamesmcarthur has joined #openstack-infra | 04:40 | |
| *** auristor has joined #openstack-infra | 04:42 | |
| *** jamesmcarthur has quit IRC | 04:51 | |
| *** kjackal has joined #openstack-infra | 04:55 | |
| *** jamesmcarthur has joined #openstack-infra | 04:57 | |
| *** kjackal has quit IRC | 05:00 | |
| *** jamesmcarthur has quit IRC | 05:04 | |
| *** roman_g has quit IRC | 05:05 | |
| *** lpetrut has joined #openstack-infra | 05:11 | |
| *** yboaron_ has quit IRC | 05:11 | |
| *** jamesmcarthur has joined #openstack-infra | 05:12 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Differentiate between queued and waiting jobs in zuul web UI https://review.opendev.org/660878 | 05:12 |
| *** ykarel|away is now known as ykarel | 05:20 | |
| *** sarob has joined #openstack-infra | 05:23 | |
| *** raukadah is now known as chandankumar | 05:25 | |
| *** quiquell has joined #openstack-infra | 05:27 | |
| *** sarob has quit IRC | 05:27 | |
| *** ccamacho has quit IRC | 05:29 | |
| *** janki has joined #openstack-infra | 05:31 | |
| *** zbr has joined #openstack-infra | 05:34 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Differentiate between queued and waiting jobs in zuul web UI https://review.opendev.org/660878 | 05:37 |
| *** imacdonn has quit IRC | 05:41 | |
| *** imacdonn has joined #openstack-infra | 05:42 | |
| AJaeger | config-core, please review https://review.opendev.org/#/q/topic:tox_constraints+is:open to help the requirements team move forward | 05:48 |
| *** jamesmcarthur has quit IRC | 05:57 | |
| *** jamesmcarthur has joined #openstack-infra | 05:58 | |
| *** slaweq has joined #openstack-infra | 05:59 | |
| *** hwoarang has quit IRC | 06:01 | |
| *** hwoarang has joined #openstack-infra | 06:02 | |
| *** slaweq has quit IRC | 06:03 | |
| *** sarob has joined #openstack-infra | 06:04 | |
| *** kopecmartin|off is now known as kopecmartin | 06:04 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** dpawlik has joined #openstack-infra | 06:16 | |
| *** jchhatbar has joined #openstack-infra | 06:19 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: [dnm] testing https://review.opendev.org/660888 | 06:19 |
| *** janki has quit IRC | 06:20 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Differentiate between queued and waiting jobs in zuul web UI https://review.opendev.org/660878 | 06:21 |
| *** gregoryo has joined #openstack-infra | 06:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: [dnm] testing https://review.opendev.org/660888 | 06:27 |
| *** sarob has quit IRC | 06:37 | |
| *** Lucas_Gray has joined #openstack-infra | 06:39 | |
| *** tesseract has joined #openstack-infra | 06:41 | |
| *** yboaron_ has joined #openstack-infra | 06:43 | |
| *** Lucas_Gray has quit IRC | 06:46 | |
| *** Lucas_Gray has joined #openstack-infra | 06:48 | |
| *** roman_g has joined #openstack-infra | 06:48 | |
| *** yboaron_ has quit IRC | 06:48 | |
| *** yboaron_ has joined #openstack-infra | 06:49 | |
| *** jamesmcarthur has quit IRC | 06:54 | |
| *** Lucas_Gray has quit IRC | 06:55 | |
| *** pgaxatte has joined #openstack-infra | 06:58 | |
| *** adriancz has joined #openstack-infra | 06:59 | |
| *** rcernin has quit IRC | 07:00 | |
| *** markvoelker has joined #openstack-infra | 07:03 | |
| *** pcaruana has joined #openstack-infra | 07:07 | |
| *** markvoelker has quit IRC | 07:08 | |
| *** ginopc has joined #openstack-infra | 07:09 | |
| *** sarob has joined #openstack-infra | 07:11 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 07:12 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Use openstack-tox-docs for api-* jobs https://review.opendev.org/659975 | 07:13 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Switch to using TOX_CONSTRAINTS_FILE https://review.opendev.org/660823 | 07:13 |
| openstackgerrit | Merged openstack/project-config master: Handle TOX_CONSTRAINTS_FILE as well as UPPER_CONSTRAINTS_FILE https://review.opendev.org/660187 | 07:17 |
| AJaeger | config-core, let's finish up the two changes above with merging https://review.opendev.org/#/c/657886/ , please | 07:19 |
| *** apetrich has joined #openstack-infra | 07:19 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: [dnm] testing https://review.opendev.org/660888 | 07:19 |
| *** rpittau|afk is now known as rpittau | 07:20 | |
| *** rf0lc0 has joined #openstack-infra | 07:23 | |
| *** dhellmann_ has joined #openstack-infra | 07:25 | |
| *** rfolco has quit IRC | 07:26 | |
| *** dhellmann has quit IRC | 07:26 | |
| *** geguileo has quit IRC | 07:27 | |
| openstackgerrit | Merged openstack/project-config master: Switch api publish jobs to tox-docs https://review.opendev.org/659976 | 07:27 |
| *** jchhatbar has quit IRC | 07:27 | |
| *** dhellmann_ is now known as dhellmann | 07:29 | |
| *** e0ne has joined #openstack-infra | 07:31 | |
| *** tosky has joined #openstack-infra | 07:32 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update opendev website to be more present tense https://review.opendev.org/653946 | 07:33 |
| ianw | looks good | 07:33 |
| *** ccamacho has joined #openstack-infra | 07:35 | |
| AJaeger | thanks, ianw | 07:36 |
| *** ykarel is now known as ykarel|afk | 07:37 | |
| *** janki has joined #openstack-infra | 07:44 | |
| *** sarob has quit IRC | 07:44 | |
| *** jpena|off is now known as jpena | 07:45 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] zone file vaildator https://review.opendev.org/660916 | 07:45 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: [dnm] testing https://review.opendev.org/660888 | 07:45 |
| AJaeger | mriedem, http://zuul.openstack.org/builds?pipeline=post&job_name=publish-api-ref is looking green again wiht the fixes merged and https://developer.openstack.org/api-ref/compute/ is also updated. So, should be all fine again... | 07:46 |
| *** apetrich has quit IRC | 07:48 | |
| openstackgerrit | Markus Hosch proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 07:50 |
| openstackgerrit | Merged zuul/zuul-jobs master: Switch to TOX_CONSTRAINTS_FILE https://review.opendev.org/657886 | 07:51 |
| *** sreejithp has joined #openstack-infra | 07:52 | |
| *** tosky has quit IRC | 07:55 | |
| openstackgerrit | Merged opendev/system-config master: Remove unused commit-filter script https://review.opendev.org/658657 | 07:55 |
| *** tosky has joined #openstack-infra | 07:55 | |
| *** apetrich has joined #openstack-infra | 07:55 | |
| *** gregoryo has quit IRC | 07:56 | |
| *** gregoryo has joined #openstack-infra | 07:56 | |
| *** sreejithp has quit IRC | 07:56 | |
| *** gregoryo has quit IRC | 07:56 | |
| openstackgerrit | Merged opendev/system-config master: Drop tools/owners.py https://review.opendev.org/532327 | 07:57 |
| zigo | ianw: AJaeger: In puppet-openstack-integration, there's a Puppetfile, I used to replace :ref => 'master' by for example :ref => 'refs/changes/46/660846/1' to get a particular patch which is still under review, so I can continue to work on the next patch. This doesn't seem to work anymore. What's the trick? | 07:59 |
| *** apetrich has quit IRC | 07:59 | |
| *** jtomasek has joined #openstack-infra | 07:59 | |
| zigo | Should I change the Git URL somehow? | 07:59 |
| *** apetrich has joined #openstack-infra | 08:00 | |
| *** iurygregory has joined #openstack-infra | 08:01 | |
| ianw | zigo: hrm, wouldn't that be pulling from review.opendev.org (not what used to be git.openstack.org)? | 08:02 |
| ianw | if it's not, it could be i think | 08:02 |
| *** ykarel|afk is now known as ykarel | 08:03 | |
| zigo | ianw: I can try to use that URL then. | 08:03 |
| *** lucasagomes has joined #openstack-infra | 08:04 | |
| *** trident has quit IRC | 08:04 | |
| *** ralonsoh has joined #openstack-infra | 08:04 | |
| *** jtomasek has quit IRC | 08:05 | |
| *** trident has joined #openstack-infra | 08:05 | |
| *** bhavikdbavishi has quit IRC | 08:11 | |
| *** tkajinam has quit IRC | 08:14 | |
| *** lyarwood has joined #openstack-infra | 08:15 | |
| *** ykarel is now known as ykarel|lunch | 08:16 | |
| lyarwood | Is anyone aware of any host ssh key changes to review.opendev.org? I'm getting `RSA host key for review.openstack.org has changed and you have requested strict checking.` errors suddenly. | 08:17 |
| *** tobias-urdin has joined #openstack-infra | 08:19 | |
| *** pkopec has joined #openstack-infra | 08:20 | |
| *** janki has quit IRC | 08:23 | |
| *** sarob has joined #openstack-infra | 08:30 | |
| *** Lucas_Gray has joined #openstack-infra | 08:40 | |
| *** panda|rover has quit IRC | 08:40 | |
| *** panda has joined #openstack-infra | 08:42 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report job depedencies via MQTT reporter https://review.opendev.org/660940 | 08:44 |
| *** derekh has joined #openstack-infra | 08:46 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report job depedencies via MQTT reporter https://review.opendev.org/660940 | 08:47 |
| *** bhavikdbavishi has joined #openstack-infra | 08:49 | |
| openstackgerrit | dongwenjuan proposed openstack/openstack-zuul-jobs master: install maven for building gearman-plugin https://review.opendev.org/660945 | 08:49 |
| *** jamesmcarthur has joined #openstack-infra | 08:51 | |
| *** jamesmcarthur has quit IRC | 08:55 | |
| *** gfidente has joined #openstack-infra | 08:58 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 08:59 | |
| *** bhavikdbavishi has quit IRC | 08:59 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 08:59 | |
| *** sarob has quit IRC | 09:02 | |
| *** jamesmcarthur has joined #openstack-infra | 09:04 | |
| *** markvoelker has joined #openstack-infra | 09:04 | |
| AJaeger | lyarwood, we renamed review.openstack.org to review.opendev.org and it has a new host key... | 09:08 |
| *** panda is now known as panda|rover | 09:11 | |
| *** pgaxatte has quit IRC | 09:12 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report job depedencies via MQTT reporter https://review.opendev.org/660940 | 09:13 |
| openstackgerrit | Stephen Finucane proposed openstack/pbr master: Remove neutron-lbaas https://review.opendev.org/658643 | 09:15 |
| *** ricolin has quit IRC | 09:15 | |
| lyarwood | AJaeger: right but that was a while ago and the redirect has been working for my existing remotes until this morning | 09:16 |
| frickler | lyarwood: do you see this for connects to gerrit (port 29418) or to ssh (port 22) | 09:18 |
| lyarwood | AJaeger: nvm, looks like it's just one repo where things are borked so maybe this is just a local issue | 09:18 |
| lyarwood | frickler: yeah the host key was to port 22, so that's my fault | 09:20 |
| *** jtomasek has joined #openstack-infra | 09:20 | |
| lyarwood | and the actual issue is the repo has moved with the opendev rename, apologies for missing this | 09:20 |
| dirk | frickler: pabelanger: ianw: any chance you could review https://review.opendev.org/#/c/660126/ and its dependencies? things are starting to break because 15.1 was released but the ci jobs are now not having their repositories anymore | 09:21 |
| dirk | it would be good to have a mirror of 15.1 repos shortly | 09:21 |
| openstackgerrit | Lee Yarwood proposed opendev/elastic-recheck master: Add query for Cinder volume attachment bug 1820155 https://review.opendev.org/660952 | 09:24 |
| openstack | bug 1812913 in Cinder "duplicate for #1820155 VolumeAttachment is not bound to a Session; lazy load operation of attribute 'volume' cannot proceed" [High,Fix released] https://launchpad.net/bugs/1812913 - Assigned to Gorka Eguileor (gorka) | 09:24 |
| *** jtomasek has quit IRC | 09:25 | |
| *** ginopc has quit IRC | 09:33 | |
| *** Lucas_Gray has quit IRC | 09:34 | |
| *** pgaxatte has joined #openstack-infra | 09:34 | |
| *** ykarel|lunch is now known as ykarel | 09:34 | |
| *** markvoelker has quit IRC | 09:38 | |
| *** electrofelix has joined #openstack-infra | 09:38 | |
| rpittau | AJaeger: hi! I think this change https://review.opendev.org/659975 broke a couple of things, for example http://logs.openstack.org/08/660708/1/check/build-openstack-api-ref/4a33c64/job-output.txt.gz#_2019-05-23_08_03_40_952701 the doc/build dir do not exist because tox run with the api-ref env, not docs | 09:39 |
| *** jamesmcarthur has quit IRC | 09:44 | |
| *** jamesmcarthur has joined #openstack-infra | 09:45 | |
| *** derekh has quit IRC | 09:54 | |
| *** derekh has joined #openstack-infra | 09:54 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: trigger: add job filter event https://review.opendev.org/639905 | 09:54 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: webtrigger: add initial driver and event https://review.opendev.org/555153 | 09:54 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: webtrigger: add web route and rpclistener https://review.opendev.org/554839 | 09:54 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: add build button to trigger job https://review.opendev.org/635716 | 09:54 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Show retry attempts in UI https://review.opendev.org/660960 | 09:57 |
| frickler | rpittau: dtantsur: AJaeger: ack, patch upcoming, we need to keep the sphinx_build_dir variable | 09:58 |
| *** jamesmcarthur has quit IRC | 09:58 | |
| rpittau | frickler: thanks! | 09:58 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/openstack-zuul-jobs master: Fix build-api-* jobs https://review.opendev.org/660962 | 10:00 |
| *** jamesmcarthur has joined #openstack-infra | 10:10 | |
| *** electrofelix has quit IRC | 10:12 | |
| *** electrofelix has joined #openstack-infra | 10:13 | |
| openstackgerrit | Merged opendev/system-config master: Properly mirror the lxc container images as well https://review.opendev.org/655895 | 10:13 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Zone file validation role https://review.opendev.org/660916 | 10:15 |
| *** jamesmcarthur has quit IRC | 10:16 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: Add zone-check job https://review.opendev.org/660888 | 10:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: [dnm] sample of zone misconfiguration to fail testing https://review.opendev.org/660966 | 10:19 |
| *** hjensas is now known as hjesas|afk | 10:19 | |
| *** jamesmcarthur has joined #openstack-infra | 10:21 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-zuul-ci.org master: Add zone-check job https://review.opendev.org/660967 | 10:22 |
| *** jamesmcarthur has quit IRC | 10:26 | |
| dtantsur | frickler: thank you! | 10:31 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Zone file validation role https://review.opendev.org/660916 | 10:32 |
| frickler | ianw: could you take a quick look at the api job fix https://review.opendev.org/660962 ? otherwise I'd just self-approve it in order to hopefully fix the regression merged earlier | 10:34 |
| openstackgerrit | renminmin proposed openstack/diskimage-builder master: Add DIB_LOCAL_IMAGE to use ubuntu local cloud image https://review.opendev.org/643885 | 10:35 |
| ianw | fricker: thanks, lgtm | 10:35 |
| *** markvoelker has joined #openstack-infra | 10:35 | |
| *** jamesmcarthur has joined #openstack-infra | 10:39 | |
| *** paladox has quit IRC | 10:41 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/pbr master: Set subparser argument required https://review.opendev.org/647492 | 10:42 |
| openstackgerrit | Sorin Sbarnea proposed openstack/pbr master: Add openstack-tox-py37 job https://review.opendev.org/610659 | 10:42 |
| *** jamesmcarthur has quit IRC | 10:43 | |
| *** paladox has joined #openstack-infra | 10:45 | |
| ianw | fungi / clarkb / corvus : something like https://review.opendev.org/#/q/topic:zone-check+(status:open+OR+status:merged) would have prevented the cname conflicts i merged the other day. i think it's good for hygiene because debugging is a bit of a pain, and impossible for !root | 10:46 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Log github requests with annotated events https://review.opendev.org/660800 | 10:47 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs around build completion and cancellation https://review.opendev.org/660806 | 10:47 |
| *** jamesmcarthur has joined #openstack-infra | 10:48 | |
| *** jamesmcarthur has quit IRC | 10:52 | |
| *** diablo_rojo has joined #openstack-infra | 10:53 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Fix build-api-* jobs https://review.opendev.org/660962 | 10:53 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report job depedencies via MQTT reporter https://review.opendev.org/660940 | 10:57 |
| *** jpena is now known as jpena|lunch | 11:01 | |
| *** ginopc has joined #openstack-infra | 11:04 | |
| *** diablo_rojo has quit IRC | 11:07 | |
| *** markvoelker has quit IRC | 11:08 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source https://review.opendev.org/658835 | 11:10 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Upgrade formatting of the patch series. https://review.opendev.org/660683 | 11:10 |
| *** nicolasbock has joined #openstack-infra | 11:11 | |
| *** janki has joined #openstack-infra | 11:11 | |
| *** udesale has quit IRC | 11:13 | |
| *** udesale has joined #openstack-infra | 11:13 | |
| *** jamesmcarthur has joined #openstack-infra | 11:16 | |
| *** Lucas_Gray has joined #openstack-infra | 11:22 | |
| openstackgerrit | Flavio Percoco proposed zuul/zuul master: Read multiple *.conf files from directory https://review.opendev.org/660977 | 11:24 |
| frickler | rpittau: dtantsur: latest api-ref jobs seem to be passing again, please let us know if you notice further issues | 11:27 |
| dtantsur | thx, sure | 11:28 |
| *** bhavikdbavishi has quit IRC | 11:29 | |
| *** dave-mccowan has joined #openstack-infra | 11:32 | |
| *** eharney has quit IRC | 11:34 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:42 | |
| *** bobh has joined #openstack-infra | 11:47 | |
| *** sreejithp has joined #openstack-infra | 11:52 | |
| *** _erlon_ has joined #openstack-infra | 11:54 | |
| *** sreejithp has quit IRC | 11:57 | |
| *** panda|rover is now known as panda|rover|eat | 11:59 | |
| *** markvoelker has joined #openstack-infra | 11:59 | |
| *** tosky has quit IRC | 12:01 | |
| *** tosky has joined #openstack-infra | 12:03 | |
| *** hrw has joined #openstack-infra | 12:03 | |
| hrw | morning | 12:03 |
| hrw | can someone tell me who I should pester to get https://review.opendev.org/657578 merged? it is x/networking-infoblox which looks like no one cares about but nicely blocks any Py3 work | 12:04 |
| *** janki has quit IRC | 12:04 | |
| hrw | and this is Py3 one line fix | 12:04 |
| *** janki has joined #openstack-infra | 12:05 | |
| *** dpawlik has quit IRC | 12:06 | |
| hrw | for now I just cut it off from Kolla to make it unblocking and as long as no one will complain I will be happy to ignore that x/ project | 12:07 |
| *** tosky__ has joined #openstack-infra | 12:08 | |
| *** rfarr__ has joined #openstack-infra | 12:08 | |
| *** rfarr_ has joined #openstack-infra | 12:10 | |
| *** dpawlik has joined #openstack-infra | 12:10 | |
| *** tosky has quit IRC | 12:11 | |
| *** dpawlik has quit IRC | 12:12 | |
| *** rfarr__ has quit IRC | 12:13 | |
| *** dpawlik has joined #openstack-infra | 12:13 | |
| *** dhill_ has joined #openstack-infra | 12:14 | |
| *** sreejithp has joined #openstack-infra | 12:16 | |
| *** jamesmcarthur has quit IRC | 12:18 | |
| *** jamesmcarthur has joined #openstack-infra | 12:18 | |
| *** eharney has joined #openstack-infra | 12:18 | |
| *** rf0lc0 is now known as rfolco | 12:18 | |
| *** sreejithp_ has joined #openstack-infra | 12:20 | |
| *** rh-jelabarre has joined #openstack-infra | 12:20 | |
| *** sreejithp has quit IRC | 12:21 | |
| *** sreejithp_ has quit IRC | 12:22 | |
| *** sreejithp has joined #openstack-infra | 12:25 | |
| AJaeger | frickler: thanks for fixing api-ref! dtantsur, frickler, sorry for breakage! | 12:28 |
| *** rlandy has joined #openstack-infra | 12:29 | |
| *** jpena|lunch is now known as jpena | 12:30 | |
| *** jamesmcarthur has quit IRC | 12:30 | |
| *** tosky has joined #openstack-infra | 12:33 | |
| *** tosky__ has quit IRC | 12:36 | |
| *** sreejithp has quit IRC | 12:36 | |
| *** tosky__ has joined #openstack-infra | 12:38 | |
| *** tosky has quit IRC | 12:40 | |
| *** tosky__ is now known as tosky | 12:40 | |
| *** lseki has joined #openstack-infra | 12:41 | |
| *** eernst has joined #openstack-infra | 12:47 | |
| *** ykarel is now known as ykarel|afk | 12:48 | |
| *** bobh has quit IRC | 12:49 | |
| rpittau | AJaeger frickler thanks and no worries! One change that was failing is now passing, so all good :) | 12:51 |
| fungi | hrw: it looks like these are the people capable of approving your change on that project: https://review.opendev.org/#/admin/groups/networking-infoblox-core | 12:51 |
| hrw | fungi: thx | 12:51 |
| AJaeger | hrw: you can check the core group, go on the option icon, look at access etc - here's a direct link: https://review.opendev.org/#/admin/projects/x/networking-infoblox,access ., But if those do not care, ignore them... | 12:51 |
| AJaeger | rpittau, yeah, I fixed the post job and broke the check/gate ;( Glad that everything is ggreen again... | 12:52 |
| *** jamesmcarthur has joined #openstack-infra | 12:52 | |
| fungi | (for the record, the way i found the core review group for that repo was following the same steps AJaeger mentions) | 12:52 |
| AJaeger | hrw: or use fungi's link directly... | 12:52 |
| hrw | ok. added missing ones. | 12:53 |
| hrw | thx | 12:53 |
| *** e0ne has quit IRC | 12:57 | |
| *** panda|rover|eat is now known as panda|rover | 13:00 | |
| *** mriedem has joined #openstack-infra | 13:03 | |
| *** jamesmcarthur has quit IRC | 13:05 | |
| *** eernst has quit IRC | 13:06 | |
| *** Goneri has joined #openstack-infra | 13:06 | |
| *** dave-mccowan has quit IRC | 13:12 | |
| *** ykarel|afk is now known as ykarel | 13:12 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Set the DiskAccountant log when size > self.limit as warning level https://review.opendev.org/660996 | 13:12 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Remove query for bug 1640920 https://review.opendev.org/660998 | 13:14 |
| openstack | bug 1640920 in Cinder "list manageable snapshots fails with lazy load error on snapshot model attr" [Undecided,New] https://launchpad.net/bugs/1640920 - Assigned to Gorka Eguileor (gorka) | 13:14 |
| *** dave-mccowan has joined #openstack-infra | 13:14 | |
| *** smarcet has joined #openstack-infra | 13:17 | |
| *** aaronsheffield has joined #openstack-infra | 13:18 | |
| *** janki has quit IRC | 13:31 | |
| *** ginopc has quit IRC | 13:33 | |
| *** ginopc has joined #openstack-infra | 13:34 | |
| *** samueldmq has joined #openstack-infra | 13:36 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Remove query for bug 1640920 https://review.opendev.org/660998 | 13:37 |
| openstack | bug 1640920 in Cinder "list manageable snapshots fails with lazy load error on snapshot model attr" [Undecided,New] https://launchpad.net/bugs/1640920 - Assigned to Gorka Eguileor (gorka) | 13:37 |
| *** sreejithp has joined #openstack-infra | 13:40 | |
| *** sreejithp_ has joined #openstack-infra | 13:42 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Fix case sensitivity in codeowners check https://review.opendev.org/661003 | 13:43 |
| *** sreejithp has quit IRC | 13:45 | |
| *** hrw has left #openstack-infra | 13:47 | |
| *** ricolin has joined #openstack-infra | 13:49 | |
| corvus | fungi, mordred: now that gitea should be able to handle the extra refs, we need to get them into the git repos. we could just let gerrit replication take care of things, but it will still take a while to replicate nova, so i think we may want to handle it specially. what do you think about this procedure? https://etherpad.openstack.org/p/gvQQow55E1 | 13:57 |
| *** quiquell has quit IRC | 13:58 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Zone file validation role https://review.opendev.org/660916 | 13:58 |
| *** smarcet has quit IRC | 14:01 | |
| *** yolanda__ has joined #openstack-infra | 14:02 | |
| *** iurygregory has quit IRC | 14:02 | |
| *** yolanda__ is now known as yolanda | 14:03 | |
| corvus | fungi, mordred: according to http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-04-24.log.html#t2019-04-24T23:40:15 it would take 2 hours to push nova without pre-seeding it, but should only take a few seconds to do so afterwords. should we think about pre-seeding any other repos? | 14:08 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Push all refs to gitea https://review.opendev.org/661008 | 14:10 |
| *** pcaruana has quit IRC | 14:10 | |
| fungi | corvus: if gerrit reindex times are any measure, neutron and openstack-manuals | 14:10 |
| corvus | fungi: i think they are good measures | 14:11 |
| fungi | those are consistently the top three left reindexing after all other repos are done | 14:11 |
| corvus | i believe it's directly linked to the number of changes | 14:11 |
| corvus | (so, eg, deb-nova should not be a problem) | 14:11 |
| fungi | yes, which is why i thought of it | 14:11 |
| fungi | seems like a relevant comparison | 14:12 |
| fungi | corvus: +refs/notes/*:refs/notes/* too or do you want to save those for a separate pass? | 14:14 |
| *** eernst has joined #openstack-infra | 14:14 | |
| corvus | fungi: i forgot them, will add, thx | 14:14 |
| fungi | just checking | 14:15 |
| *** sreejithp has joined #openstack-infra | 14:16 | |
| *** smarcet has joined #openstack-infra | 14:16 | |
| corvus | fungi, mordred: i took the liberty of validating that this works on gitea06 (which is out of the load balancer due to a corrupt filesystem and due to be replaced, but works well enough to test these commands) | 14:17 |
| fungi | yeah, i wondered if we should try it there | 14:17 |
| fungi | good idea | 14:18 |
| fungi | granted, it may go faster there as it's not serving any requests | 14:18 |
| *** sreejithp__ has joined #openstack-infra | 14:18 | |
| fungi | so has less competition for system resources | 14:18 |
| corvus | i think it goes slower whenever it hits bad fs blocks :) | 14:18 |
| fungi | but at least provides us with a lower bound on runtime | 14:18 |
| fungi | oh, hah ;) | 14:18 |
| corvus | it took "a few minutes" | 14:19 |
| fungi | trivial then, in the grand scheme | 14:19 |
| *** sreejithp_ has quit IRC | 14:19 | |
| *** eernst has quit IRC | 14:19 | |
| fungi | not "a few hours" or "a few days" anyway | 14:19 |
| corvus | yeah. and i really do expect it to be faster, because even "git show $something" is unusually slow. | 14:19 |
| *** sreejithp has quit IRC | 14:21 | |
| *** smarcet has quit IRC | 14:24 | |
| *** michael-beaver has joined #openstack-infra | 14:24 | |
| corvus | this may work, if you reload a couple of times: https://gitea06.opendev.org:3000/openstack/nova/commit/9c2989091b11ad90a6d87d60be1b7b7a6ced0556 | 14:26 |
| corvus | that's an intermediate patchset (not the final merged version) | 14:26 |
| corvus | refs/changes/616120/16 | 14:26 |
| corvus | it worked 1/3 times for me; given the state of 06, i think that counts as success | 14:27 |
| *** dpawlik has quit IRC | 14:27 | |
| fungi | i guess taking a while to return a result is the expected misbehavior when it runs into fs trouble? | 14:28 |
| fungi | aha, it did eventually load on the first try | 14:28 |
| corvus | fungi: yeah, i believe so | 14:28 |
| fungi | yeah, server seems... "sluggish" | 14:28 |
| fungi | maybe it needs more coffee | 14:28 |
| fungi | (and a less-corrupt filesystem) | 14:29 |
| *** pcaruana has joined #openstack-infra | 14:29 | |
| * mordred waves to fungi and corvus | 14:34 | |
| mordred | corvus: ++ to the scrollback and pre-seeding refs for things | 14:35 |
| corvus | mordred, fungi: what's the down low on changing the gerrit replication config? is a full restart needed before it sees changes to the file? | 14:36 |
| fungi | modifying the file on disk will now cause gerrit to automatically reload it (unless we reverted that) | 14:37 |
| fungi | but the frogurt is cursed | 14:37 |
| fungi | queued replication tasks will be dropped when that happens | 14:37 |
| corvus | i guess that's okay since this process ends with a replicate --all | 14:37 |
| fungi | yup | 14:38 |
| *** smarcet has joined #openstack-infra | 14:38 | |
| *** auristor has quit IRC | 14:38 | |
| corvus | okay, so we should keep the change WIP until the pre-seeding is done, then merge it, then wait for it to be in place, then replicate | 14:38 |
| fungi | just somethnig to keep in mind more generally when altering that file | 14:38 |
| *** auristor has joined #openstack-infra | 14:38 | |
| fungi | sounds right to me | 14:38 |
| mordred | yes, that sounds like the right process | 14:39 |
| corvus | given that, i'll start the pre-seeding right after breakfast. i'll do one host first and we can check it out to double check the process didn't break anything (in case there's a difference between gitea06 and a real production host) | 14:39 |
| mordred | ++ | 14:39 |
| *** pgaxatte has quit IRC | 14:39 | |
| fungi | excellent | 14:40 |
| *** smarcet has quit IRC | 14:44 | |
| fungi | also, this will be awesome once it lands: https://github.com/go-gitea/gitea/pull/6984 | 14:45 |
| fungi | though it may need a followup to handle refs/notes/gerrit in addition to just refs/notes/commits | 14:46 |
| *** sarob has joined #openstack-infra | 14:46 | |
| corvus | i like how well thought out that pr is | 14:47 |
| fungi | oh! maybe not... a recent update to that pr made the notesref configurable | 14:47 |
| fungi | so actually we should be able to just set it in configuration | 14:48 |
| mordred | \o/ | 14:48 |
| mordred | yeah- the discussion on the PR is also nice | 14:48 |
| openstackgerrit | Stephen Finucane proposed openstack/pbr master: Resolve some issue with tox.ini, setup.cfg https://review.opendev.org/660297 | 14:48 |
| fungi | i tended to rely somewhat often on cgit's behavior of displaying relevant git notes, so if anyone else who was doing the same this is one more checkbox ticked for feature parity | 14:50 |
| *** dave-mccowan has quit IRC | 14:52 | |
| *** dave-mccowan has joined #openstack-infra | 14:53 | |
| *** kopecmartin is now known as kopecmartin|afk | 14:53 | |
| *** hjesas|afk is now known as hjensas | 14:55 | |
| *** auristor has quit IRC | 14:58 | |
| *** auristor has joined #openstack-infra | 14:59 | |
| smcginnis | Is there something that can be done to fix the "No account linked to committer's email address" in our gitea instance, or is it just going to be that way since we don't have fully integrated auth? | 15:03 |
| *** auristor has quit IRC | 15:03 | |
| corvus | smcginnis: can you specify where you see that? | 15:04 |
| smcginnis | corvus: I think since I sign my commits: https://opendev.org/openstack/cinder/commit/8fbfe923bd99760293117dea47040c77746bc2d5 | 15:04 |
| *** auristor has joined #openstack-infra | 15:04 | |
| corvus | smcginnis: i wasn't aware of that until now. i'm not sure if we can do anything about it, but i think it's worth looking into. | 15:06 |
| mordred | I think that'll require having user accounts in the gitea - but I agree, I also think it's worth looking in to | 15:06 |
| corvus | conveniently, the pr fungi linked earlier has a screenshot of what it looks like when it's working: https://camo.githubusercontent.com/fdaa7a8ff7e3df82f77534b1c7aa5cb3b7a2db77/68747470733a2f2f64756d702e7468656379626572736861646f772e6e65742f65346530306135376534656364626662383039613335366561343834333835342f3136253341313025334130392d75706c6f61642e706e67 | 15:07 |
| pabelanger | it does look like one works: https://opendev.org/yatinkarel | 15:07 |
| smcginnis | With GitHub, you have your account that includes somewhere to upload your keys for validation. We might not have that. | 15:07 |
| pabelanger | not sure how that happened | 15:07 |
| *** lpetrut has quit IRC | 15:08 | |
| corvus | also, frickler seems to have made an account? | 15:08 |
| mordred | they don't show up in explore users | 15:08 |
| corvus | each server is going to be different | 15:09 |
| mordred | this is ... interesting | 15:09 |
| pabelanger | Oh, yah, if I refresh https://opendev.org/explore/users I can see it bounce users | 15:10 |
| frickler | corvus: oh, I did that while exploring things a while back and saw that it had disappeared again immediately, didn't look into it any further back then. that it's per backend server does indeed explain things | 15:11 |
| *** yboaron_ has quit IRC | 15:11 | |
| corvus | frickler: any chance you remember how you did it so we can prevent it from happening? | 15:11 |
| mordred | I just tried going to https://opendev.org/user/sign_up (which is what the registration link _would_ be if we didn't have it disabled) and it's properly showing "registration is disabled" | 15:12 |
| mordred | maybe there was a moment where our config was incomplete and it wasnt' disabled or something? | 15:12 |
| corvus | or 2 moments | 15:13 |
| corvus | yatinkarel was april 21 | 15:13 |
| frickler | mordred: corvus: yeah, I think that it's disabled now. I think it should be fine to just remove it if you have a tool for that | 15:14 |
| corvus | yatinkarel is on 1,2,5,7 j-harbott on 5,8 | 15:14 |
| corvus | yatinkarel jobined on may 1 and april 21 | 15:15 |
| corvus | so at least 3 days where that was possible | 15:16 |
| mordred | *weird* | 15:16 |
| *** chandankumar is now known as raukadah | 15:17 | |
| corvus | and yatinkarel made 1 repo: https://gitea05.opendev.org:3000/yatinkarel/scripts | 15:17 |
| clarkb | corvus: re replicating extra refs we should make sure we wont fill the disks on the gitea servers | 15:17 |
| *** ian-pittwood has joined #openstack-infra | 15:17 | |
| mordred | clarkb: speaking of - when we rebuild the gitea servers - I think we want to be sure to clone one of the existing servers' mysql and filesystem rather than rebuilding an empty fs since the repl of the refs/ stuff takes a second | 15:20 |
| corvus | clarkb: good point. the size difference is currently 2.6G with 13G free, so i think we'll be okay | 15:20 |
| *** smarcet has joined #openstack-infra | 15:21 | |
| *** ian-pittwood has quit IRC | 15:21 | |
| *** jamesmcarthur has joined #openstack-infra | 15:21 | |
| *** smarcet has quit IRC | 15:22 | |
| *** ccamacho has quit IRC | 15:22 | |
| corvus | infra-root: could someone take on the task of investigating how the yatinkarel account was created? perhaps clues in the logs or the database may suggest something. | 15:22 |
| corvus | i'd like to press on with the refs | 15:23 |
| fungi | if we're going to copy mysql filesystems around, we maybe ought to sync one new server fresh and then copy that to the others? to address the lingering users and repos which got created early on | 15:23 |
| fungi | but yes, first we probably need to know how the one was created ~3 weeks ago | 15:23 |
| fungi | ykarel: do you happen to be around? | 15:23 |
| mordred | corvus: I'll go troll through the logs | 15:24 |
| corvus | fungi: i'm sure we can delete the account through the web ui, but we should do that after investigating | 15:24 |
| ykarel | fungi, yes | 15:24 |
| ykarel | anything for me? | 15:24 |
| mordred | corvus: we do not have logs that go back that far | 15:26 |
| corvus | ykarel: we noticed that you created an account (perhaps several times) on the gitea servers at opendev.org, and we thought we had disabled the ability to do that. can you tell us how you did it so we can close that loophole? | 15:26 |
| mordred | corvus: only since the latest restart ... so we should also probably add to the list to do something about how logging is logged to places | 15:26 |
| ykarel | corvus, yes there was login button, i used that, /me looks again | 15:27 |
| clarkb | mordred: I'd really prefer we continue to treat them as stateless as much as possible | 15:27 |
| corvus | mordred: i don't think we have any log rotation configured | 15:27 |
| clarkb | mordred: we can take one out of the lb, replace it, sync, then add to the lb | 15:27 |
| corvus | mordred: which logs are you looking at? | 15:27 |
| mordred | corvus: the docker logs | 15:27 |
| corvus | mordred: look in /var/gitea/logs | 15:28 |
| mordred | ah | 15:28 |
| ykarel | corvus, https://opendev.org/user/login | 15:28 |
| corvus | (at this point, the logging bug is that anything is output to docker) | 15:28 |
| ykarel | corvus, explicitly https://opendev.org/user/login/openid | 15:29 |
| corvus | ykarel: did you use openid? | 15:29 |
| corvus | aha | 15:29 |
| ykarel | corvus, yes | 15:29 |
| corvus | ykarel: thanks, i think that explains it :) | 15:29 |
| fungi | neat, neither of those look particularly disabled | 15:29 |
| corvus | fungi: logging in with a password is okay as long as you can't create an account | 15:29 |
| ykarel | corvus, okk | 15:29 |
| *** Lucas_Gray has quit IRC | 15:29 | |
| corvus | fungi: but i think logging in with an openid carries an implicit registration | 15:30 |
| fungi | aha | 15:30 |
| mordred | clarkb: what I'm saying is that syncing from gerrit is likely to be highly inefficient. so I agree that they should be treated as stateless - but was more suggesting that we perform the "sync" step via a mechanism other that "gerrit replicate" | 15:30 |
| corvus | ykarel: fyi, we don't intend to support logins on that system, so we'll delete that account (and the scripts repo) soon | 15:30 |
| corvus | clarkb: nova alone will take >2 hours | 15:30 |
| *** priteau has joined #openstack-infra | 15:31 | |
| ykarel | corvus, okk np from me, corvus actually some urls got redirect to login page, i logged in thinking those urls will work | 15:31 |
| mordred | corvus: we have: ENABLE_OPENID_SIGNIN = true | 15:31 |
| mordred | ENABLE_OPENID_SIGNUP = true | 15:31 |
| mordred | in our config - should we change those to false? | 15:31 |
| corvus | mordred: ++ | 15:31 |
| mordred | patch coming | 15:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Disable openid login and signup https://review.opendev.org/661038 | 15:32 |
| *** smarcet has joined #openstack-infra | 15:32 | |
| *** jamesmcarthur_ has joined #openstack-infra | 15:32 | |
| *** diablo_rojo has joined #openstack-infra | 15:33 | |
| frickler | corvus: mordred: now that I see it, this openid page looks slightly familiar to me, so there's a good chance that I used that one, too | 15:35 |
| corvus | good, hopefully that's the only leak then :) | 15:35 |
| *** jamesmcarthur has quit IRC | 15:36 | |
| clarkb | mordred: ya we also need the redirect state so not totally stateless | 15:37 |
| mordred | clarkb: yah. oh - yeah - in fact, that's a _really_ good point | 15:38 |
| clarkb | corvus: gitea06 has been offline since the ptg. Not sure a fewhours is a major deal? | 15:38 |
| corvus | clarkb: i don't follow | 15:38 |
| corvus | i think i understand | 15:38 |
| mordred | clarkb: if we just do an rsync of git repo data and a mysqldump/restore from a working server - that shoudl get us redirect state and whatnot | 15:39 |
| *** ykarel is now known as ykarel|away | 15:39 | |
| corvus | clarkb: yes, we could allow replication to happen normally and then add it to the load balancer after it completes (say, the next day). but your point about the db probably supercedes that. | 15:39 |
| *** hwoarang has quit IRC | 15:40 | |
| clarkb | just pointing out 2 hours to sync nova usnt that bad compared to being out of commission for a couple weeks | 15:41 |
| *** hwoarang has joined #openstack-infra | 15:41 | |
| *** hamzy has quit IRC | 15:44 | |
| corvus | i've copied the repo data to all gitea hosts and am running the nova git fetch on 01 now | 15:46 |
| *** ykarel|away has quit IRC | 15:46 | |
| clarkb | corvus: that was out of band git push of refs/changes and refs/notes ? | 15:47 |
| corvus | clarkb: fetch, but yes: https://etherpad.openstack.org/p/gvQQow55E1 | 15:47 |
| clarkb | ah yes the big three repos | 15:48 |
| *** sarob has quit IRC | 15:48 | |
| *** whoami-rajat has quit IRC | 15:50 | |
| *** rpittau is now known as rpittau|afk | 15:51 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use gitea links in gerrit https://review.opendev.org/654034 | 15:53 |
| mordred | corvus: once the refs are replicated and we're happy with it all, there's the rebased version of "use gitea not gitweb" | 15:54 |
| *** diablo_rojo has quit IRC | 15:54 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Finish removing git.openstack.org references https://review.opendev.org/654040 | 15:57 |
| corvus | oh, i think we need to pack those refs | 15:59 |
| corvus | git upload-pack and friends are taking a long time because of all the stat calls | 16:00 |
| mordred | mmm. yes. pack refs | 16:02 |
| corvus | okay, i've done that on gitea01. that makes https://gitea01.opendev.org:3000/openstack/nova take about the same time as https://gitea02.opendev.org:3000/openstack/nova | 16:03 |
| tosky | question on zuul-jobs: before I add more python code to solve the problem highlighted during the review of https://review.opendev.org/#/c/648658/, I would like to ask: why are dotfiles not shown on logs.openstack.org by default? | 16:03 |
| clarkb | tosky: mod auto directory renders those indexes for us | 16:04 |
| clarkb | er mod auto index in paache | 16:04 |
| clarkb | possible that it ignores "hidden" files? | 16:04 |
| tosky | clarkb: that seems to be the case; I don't have recent builds but I'm pretty sure that without that change I wasn't able to see them | 16:05 |
| tosky | (now, the change is not complete, but that's a different story) | 16:05 |
| mordred | https://serverfault.com/questions/245922/how-to-configure-apache-to-view-hidden-files | 16:05 |
| corvus | infra-root: does anyone else want to poke around the nova repo on gitea01, or should i proceed with the process in https://etherpad.openstack.org/p/gvQQow55E1 for all 3 repos on all servers? | 16:05 |
| corvus | (and i think we're going to want a cron to pack-refs periodically) | 16:06 |
| clarkb | corvus: reading the process it seemed sound. The only bit that I wondered about was where is the source of the nova repo? rsync from gerrit? | 16:06 |
| corvus | clarkb: scp tarball from /opt/lib on gerrit | 16:06 |
| *** smarcet has quit IRC | 16:07 | |
| *** roman_g has quit IRC | 16:07 | |
| clarkb | tosky: http://httpd.apache.org/docs/2.2/mod/mod_autoindex.html#indexignore the default indexignore value does not seem to ignore all dotfiles just the current dir single dot dotfile | 16:08 |
| clarkb | tosky: possible our distro is overriding that though | 16:08 |
| clarkb | I think one of the gotchas here is exposing .htpasswd files? | 16:08 |
| fungi | it tends to be considered a safety precaution for that reason, yes | 16:08 |
| mordred | corvus: https://gitea01.opendev.org:3000/openstack/nova/commit/8e59a7d8be23d928290b7fcfa2a1c7f37b45632f renders | 16:08 |
| *** kjackal has joined #openstack-infra | 16:08 | |
| *** ykarel|away has joined #openstack-infra | 16:08 | |
| tosky | I'm sure that there are good reasons for closing it by default, but, unless there are some super-secret files, exposing them under logs.openstack.org shouldn't be a big issue, I think | 16:09 |
| clarkb | mordred: corvus nova loads really slowly for me on gitea01 | 16:09 |
| *** pcaruana has quit IRC | 16:09 | |
| corvus | clarkb: it loads in about the same time for me on gitea02 | 16:09 |
| fungi | gitea01 lgtm | 16:09 |
| mordred | yes. I agree - it does load slow on 01 - but I also agree, it loads slow on 02 too | 16:10 |
| clarkb | ya | 16:10 |
| mordred | at some point we should maybe talk to our gitea friends about ways to make that better | 16:10 |
| mordred | we have a lovely test-case repo they can play with :) | 16:11 |
| fungi | i need to step away to grab lunch and pick up some groceries. need ingredients for stuffing bratwurst this weekend, but will be back as soon as i can | 16:11 |
| clarkb | I can't seem to find the path to render refs/changes/47/651647/1 without the commit sha | 16:11 |
| clarkb | maybe we don't need that? | 16:11 |
| clarkb | ^ that ref maps to the commit url above | 16:12 |
| mordred | yeah - I also can't find such a thing - it's possible it doesn't exist | 16:12 |
| mordred | I believe gerrit will link to changes via the gitweb interface using sha | 16:12 |
| corvus | yeah, i don't think it's needed for the gerrit links | 16:12 |
| mordred | so I don't think we need it immediately | 16:12 |
| clarkb | k | 16:12 |
| corvus | hopefully we can git fetch it? | 16:12 |
| * clarkb tries | 16:12 | |
| mordred | it would be a nice improvement to the ui for it to be able to work | 16:13 |
| *** Romik has joined #openstack-infra | 16:13 | |
| *** pkopec has quit IRC | 16:14 | |
| *** hwoarang has quit IRC | 16:14 | |
| clarkb | seems to be working though I got a warning no common commits message with my somewhat older nova clone | 16:15 |
| *** factor has quit IRC | 16:15 | |
| clarkb | I would've expected there to be common commits somwhere | 16:15 |
| corvus | i'm starting on the other servers now | 16:15 |
| *** hwoarang has joined #openstack-infra | 16:16 | |
| *** gyee has joined #openstack-infra | 16:16 | |
| clarkb | well that explains it my repo named "nova" is actually openstack/openstack for some reason | 16:16 |
| *** mattw4 has joined #openstack-infra | 16:16 | |
| clarkb | I should wipe out my local repos and start over | 16:16 |
| *** lucasagomes has quit IRC | 16:18 | |
| mordred | clarkb: yeah - I did that with the opendev shift | 16:18 |
| mordred | figured that way I could also get rid of local clones I wasn't using | 16:19 |
| clarkb | ya I should probably go ahead and do that myself | 16:21 |
| clarkb | tosky: I think the gotcha with changing that setting is if it comes from the distro apache install then it is likely apache wide and not per vhost (still working to confirm that but I need to dig out my ssh key) | 16:22 |
| clarkb | tosky: in which case while safe for logs, we do host other things on that server which we may not want to have exposed in a similar manner | 16:23 |
| corvus | nova is done on all servers | 16:23 |
| mordred | clarkb: http://httpd.apache.org/docs/2.4/mod/mod_autoindex.html#indexignorereset | 16:23 |
| clarkb | mordred: oh cool so we could reset it just for logs | 16:24 |
| mordred | yah | 16:24 |
| mordred | however ... | 16:24 |
| mordred | are there any issues serving dotfiles as swift objects? | 16:24 |
| clarkb | https://opendev.org/opendev/puppet-openstackci/src/branch/master/templates/logs-dev.vhost.erb and https://opendev.org/opendev/puppet-openstackci/src/branch/master/templates/logs.vhost.erb are the two vhosts to change for that. Often we'll update logs-dev first, check the behavior then update logs next | 16:24 |
| clarkb | mordred: I expect swift will do what tosky wants since it has fewer built in assumptions like hiding htpassword behind a . | 16:25 |
| *** dpawlik has joined #openstack-infra | 16:25 | |
| timburke | from swift's perspective, definitely -- an object starting with a dot, or containing a '/.' anywhere in its path, is just another object | 16:26 |
| corvus | neutron done on all servers | 16:26 |
| clarkb | https://opendev.org/opendev/puppet-openstackci/src/branch/master/templates/logs-dev.vhost.erb#L62-L71 would be the section to modify | 16:27 |
| timburke | though i seem to recall that things like the log hosting are actually on radosgw, in which case i have no idea about what extra limitations there may be | 16:27 |
| corvus | okay, all 3 repos complete on all 7 servers | 16:28 |
| mordred | corvus: that is exciting! | 16:28 |
| corvus | i have +3d https://review.opendev.org/661008 | 16:28 |
| mordred | timburke: woot! | 16:28 |
| openstackgerrit | Ben Nemec proposed openstack/pbr master: Read description file as utf-8 https://review.opendev.org/564874 | 16:29 |
| corvus | next step is to wait for that to go into effect, then run 'replication start -- all' | 16:29 |
| openstackgerrit | Merged opendev/system-config master: Disable openid login and signup https://review.opendev.org/661038 | 16:29 |
| corvus | i'm going to afk for a bit while that works its way through | 16:30 |
| openstackgerrit | Clark Boylan proposed opendev/puppet-openstackci master: The logs for a job my contain dotfiles, index them https://review.opendev.org/661047 | 16:31 |
| clarkb | mordred: fungi tosky ^ we can test it out on logs-dev | 16:31 |
| *** kjackal has quit IRC | 16:38 | |
| *** kskels has joined #openstack-infra | 16:39 | |
| kskels | Hi infra team! I am having issue with airship/pegleg post job showing CONFIG_ERROR in the build set.. I can't seem to find any logs, etc. | 16:40 |
| kskels | https://usercontent.irccloud-cdn.com/file/RE42Qniw/sync.png | 16:40 |
| kskels | is there any tips you could share.. | 16:40 |
| tosky | clarkb: thanks! | 16:40 |
| tosky | now I must confess than I don't know what logs-dev is used for :) | 16:41 |
| openstackgerrit | Merged openstack/pbr master: Add openstack-tox-py37 job https://review.opendev.org/610659 | 16:41 |
| *** armax has quit IRC | 16:41 | |
| clarkb | tosky: it is a vhost that allows us to test changes to the logs.openstack.org vhost. Basically take any logs.openstack.org url and use logs-dev.openstack.org instead to use it | 16:42 |
| clarkb | kskels: which post job? | 16:43 |
| kskels | I believe it doesn't really even get to the post job | 16:43 |
| kskels | https://opendev.org/airship/pegleg/src/branch/master/.zuul.yaml | 16:44 |
| kskels | here is the .zuul file so I've been adding the git mirror job additionally to exisitng post jobs | 16:44 |
| kskels | the config works well for ariship/treasuremap repo - so bit puzzled where to even look for error logs as it doesn't get to the job itself | 16:45 |
| kskels | http://zuul.openstack.org/builds?job_name=pegleg-upload-git-mirror | 16:45 |
| kskels | shows no builds | 16:45 |
| *** zbr has quit IRC | 16:45 | |
| clarkb | kskels: looks like you used the same secret in both repos, but the secret encryption key is repo specific | 16:46 |
| tosky | clarkb: oh, different frontend for the same data? Interesting | 16:47 |
| clarkb | my hunch here is that pegleg can't decrypt the secret with its key | 16:47 |
| clarkb | tosky: yes | 16:47 |
| kskels | oh.. you are right! :) | 16:47 |
| kskels | the cert encryption was indeed project specific | 16:47 |
| kskels | thank you.. | 16:47 |
| mordred | clarkb: seems like the sort of thing that should get reported somehwere - not 100% sure where though | 16:50 |
| *** whoami-rajat has joined #openstack-infra | 16:51 | |
| *** roman_g has joined #openstack-infra | 16:51 | |
| mordred | clarkb, corvus: maybe when someone submits a zuul config change with a secret in it, if the secret is undecryptable for that project, an error message about "this secret was unable to be decrypted" or something similar? | 16:52 |
| *** Romik has quit IRC | 16:52 | |
| *** dtantsur is now known as dtantsur|afk | 16:54 | |
| clarkb | mordred: ya. THe only question would be if that violates security concerns somehow (like when you login to a service and fail they don't tell you what you got wrong) | 16:55 |
| *** ricolin has quit IRC | 16:55 | |
| *** tjgresha has quit IRC | 16:55 | |
| mordred | yah | 16:55 |
| *** tjgresha has joined #openstack-infra | 16:55 | |
| openstackgerrit | Merged openstack/pbr master: Fix white space handling in file names https://review.opendev.org/629161 | 16:56 |
| *** derekh has quit IRC | 17:00 | |
| *** pcaruana has joined #openstack-infra | 17:00 | |
| *** ykarel|away has quit IRC | 17:02 | |
| openstackgerrit | Merged openstack/pbr master: Set subparser argument required https://review.opendev.org/647492 | 17:02 |
| *** udesale has quit IRC | 17:03 | |
| openstackgerrit | Merged opendev/system-config master: Push all refs to gitea https://review.opendev.org/661008 | 17:07 |
| *** ginopc has quit IRC | 17:08 | |
| *** tosky has quit IRC | 17:08 | |
| *** Weifan has joined #openstack-infra | 17:10 | |
| *** tosky has joined #openstack-infra | 17:12 | |
| corvus | well that wasn't super fast, was it? :) | 17:12 |
| mordred | corvus: I mean - it wasn't the fastest | 17:13 |
| mordred | but it also could have been worse | 17:13 |
| *** dpawlik has quit IRC | 17:17 | |
| *** jpena is now known as jpena|off | 17:19 | |
| *** tosky has quit IRC | 17:22 | |
| openstackgerrit | Dirk Mueller proposed opendev/system-config master: Move openSUSE Tumbleweed into a caching mirror instead https://review.opendev.org/660125 | 17:23 |
| openstackgerrit | Dirk Mueller proposed opendev/system-config master: Start mirroring openSUSE Leap 15.1 https://review.opendev.org/660126 | 17:23 |
| *** tosky__ has joined #openstack-infra | 17:23 | |
| *** ykarel has joined #openstack-infra | 17:24 | |
| *** smarcet has joined #openstack-infra | 17:26 | |
| *** dpawlik has joined #openstack-infra | 17:26 | |
| *** hamzy has joined #openstack-infra | 17:28 | |
| *** e0ne has joined #openstack-infra | 17:29 | |
| *** auristor has quit IRC | 17:30 | |
| *** auristor has joined #openstack-infra | 17:33 | |
| *** sarob has joined #openstack-infra | 17:33 | |
| *** hamzy has quit IRC | 17:40 | |
| *** hamzy has joined #openstack-infra | 17:40 | |
| clarkb | mordred: on https://review.opendev.org/#/c/640027/12/playbooks/templates/clouds/nodepool_builder_clouds.yaml.j2 we actually set force_ipv4 on vexxhost now too to fix the dns resolution errors that test jobs were having | 17:43 |
| clarkb | mordred: I don't know that we need to set that in that file because builders don't launch nodes and otherwise I expect it will be fine (whereas in rax there are glean considerations) | 17:43 |
| clarkb | mordred: but wanted to call that out before I revote and get some input | 17:44 |
| clarkb | infra-root can you review https://review.opendev.org/#/c/655476/ which is our fixup change to the rename playbook. Important if we want to rename repos next week. | 17:45 |
| *** hamzy_ has joined #openstack-infra | 17:46 | |
| *** hamzy has quit IRC | 17:49 | |
| corvus | clarkb: done | 17:49 |
| corvus | and the new replication config just landed on gerrit | 17:50 |
| clarkb | I'm going to pop out for a bit to dig out some food then I'll be double checking on ask.o.o before I declare puppet-4 done | 17:50 |
| *** ralonsoh has quit IRC | 17:51 | |
| clarkb | I guess I can do an ansible command to check for which puppet && test -e /opt/puppetlabs | 17:51 |
| corvus | i just watched the gerrit replication queue and saw it replicate to openstack-helm-infra, and verified that there are now refs/changes refs for that repo on the gitea servers | 17:52 |
| corvus | so i think we're ready to trigger the really big replication | 17:52 |
| evrardjp | clarkb: could you by any chance reapply a +w on https://review.opendev.org/#/c/660126/ please? | 17:53 |
| evrardjp | or anyone, sorry, it was the name that was in the middle of my screen | 17:53 |
| corvus | infra-root: i've started replication of all refs of all projects to all gitea servers; this may take a while and therefore we will likely see a replication lag until the process is complete | 17:56 |
| corvus | it started with ~18000 replication tasks | 17:57 |
| *** lpetrut has joined #openstack-infra | 18:03 | |
| corvus | 17k | 18:06 |
| *** kskels has left #openstack-infra | 18:07 | |
| corvus | etc 3.6 hours (~21:45 utc) | 18:07 |
| *** lpetrut has quit IRC | 18:07 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Allow to select the merge method in GitHub https://review.opendev.org/617949 | 18:10 |
| *** rlandy is now known as rlandy|brb | 18:16 | |
| openstackgerrit | Merged opendev/system-config master: Omnibus rename repo playbook fixes https://review.opendev.org/655476 | 18:17 |
| *** hamzy__ has joined #openstack-infra | 18:26 | |
| *** hamzy_ has quit IRC | 18:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Use openstack-tox-docs for deploy-guide job https://review.opendev.org/661078 | 18:28 |
| *** raissa has joined #openstack-infra | 18:30 | |
| *** ykarel has quit IRC | 18:32 | |
| *** hamzy__ has quit IRC | 18:34 | |
| *** smarcet has quit IRC | 18:35 | |
| *** mriedem has quit IRC | 18:37 | |
| *** smarcet has joined #openstack-infra | 18:37 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Use tox for publish-deploy-guide https://review.opendev.org/661081 | 18:38 |
| *** rlandy|brb is now known as rlandy | 18:42 | |
| *** hamzy has joined #openstack-infra | 18:43 | |
| *** mriedem has joined #openstack-infra | 18:43 | |
| fungi | looks like we're suddenly failing bashate "E042 local declaration hides errors" on https://opendev.org/opendev/system-config/src/branch/master/run_all.sh#L48-L51 | 18:44 |
| fungi | specifically i think it's "current" and "elapsed_ms" it's bothered bu | 18:44 |
| fungi | s/bu/by/ | 18:44 |
| *** priteau has quit IRC | 18:44 | |
| *** hamzy has quit IRC | 18:47 | |
| *** electrofelix has quit IRC | 18:47 | |
| fungi | trying to figure out what it doesn't like about them | 18:47 |
| *** hamzy has joined #openstack-infra | 18:47 | |
| fungi | according to the source code it means it doesn't like =$( assignment | 18:49 |
| fungi | or =` | 18:49 |
| fungi | there have been no new releases of bashate for nearly a year so no idea why this is only showing up now | 18:50 |
| fungi | ahh, it wants the local declaration to be done separately from the variable assignment | 18:52 |
| *** e0ne has quit IRC | 18:53 | |
| fungi | the lines it's complaining about in that script haven't been touched since september | 18:54 |
| *** e0ne has joined #openstack-infra | 18:54 | |
| fungi | i'm mystified | 19:00 |
| *** armstrong has joined #openstack-infra | 19:01 | |
| clarkb | fungi: have a link to a log? | 19:01 |
| clarkb | fungi: but usually what it means is you have to do local foo\nfoo=$(ls) | 19:01 |
| fungi | http://logs.openstack.org/26/660126/2/check/tox-linters/0c4b583/ | 19:01 |
| clarkb | fungi: there is a bash behvaior issue where if you do local foo=$(some exec) you don't get the local behavior you expected on failure? | 19:02 |
| clarkb | something like that | 19:02 |
| fungi | and yeah, i see how to make it not complain, more wondering what caused it to suddenly start | 19:02 |
| fungi | i don't think the version of bashate we're using has changed in almost a year, the lines it's complaining about haven't been touched in 8 months... | 19:03 |
| mtreinish | fungi: I don't think there has been a bashate update in over a year | 19:04 |
| fungi | tox.ini was last altered a couple months back | 19:04 |
| fungi | and didn't touch the bashate invocation either | 19:04 |
| clarkb | fungi: I think those are just warnings? | 19:04 |
| clarkb | http://logs.openstack.org/26/660126/2/check/tox-linters/0c4b583/job-output.txt.gz#_2019-05-23_18_05_55_200759 is the error failing the job I think | 19:04 |
| *** hamzy has quit IRC | 19:05 | |
| fungi | 84 bashate warning(s) found, 2 bashate error(s) found | 19:05 |
| *** tesseract has quit IRC | 19:05 | |
| fungi | oh! so it is | 19:06 |
| fungi | okay, so problem with that patch | 19:06 |
| fungi | evrardjp: "opensuse-mirror-update.sh:43:1: E040 Syntax error: syntax error near unexpected token `distribution/leap/15.0/jeos' | 19:06 |
| *** Adri2000 has quit IRC | 19:06 | |
| clarkb | infra-root cmurphy in reviewing if we have any remaining puppet-3 hosts nb03.openstack.org and mirror01.london.linaro-london.openstack.org and backup01.ord.rax.ci.openstack.org are still puppet 3. The first two because they are arm64 hosts and the third because it is one we don't touch much | 19:06 |
| clarkb | I think we can call puppet-4 done when we update the backup server? Are we actively puppeting the backup server? | 19:06 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Allow to select the merge method in Github https://review.opendev.org/617949 | 19:06 |
| *** Adri2000 has joined #openstack-infra | 19:08 | |
| clarkb | fungi: I don't know why those "errors" are treated as warnings. Maybe because those files aren't touched by the change or we've ignored those rules? | 19:08 |
| fungi | i can't tell how you're expected to differentiate warnings from errors in bashate output | 19:08 |
| fungi | it's all "ENNN some text" | 19:09 |
| clarkb | yup | 19:09 |
| fungi | likely i'm just reading it wrong | 19:09 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove groups configuration https://review.opendev.org/661086 | 19:13 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove the ask.openstack.org inventory entry https://review.opendev.org/661087 | 19:13 |
| clarkb | fungi: ianw ^ can you review those since you have been involved in the related changes? | 19:13 |
| clarkb | noticed those were still in the inventory when checking puppet-4 completeness | 19:13 |
| clarkb | All three hosts are unreachable | 19:14 |
| clarkb | missed some stuff will get new ps up | 19:18 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove groups configuration https://review.opendev.org/661086 | 19:20 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove the ask.openstack.org inventory entry https://review.opendev.org/661087 | 19:20 |
| clarkb | that should be more complete | 19:20 |
| clarkb | (also I couldn't find existing changes for that stuff, but point me at them if they exist and I can abandon my changes) | 19:21 |
| clarkb | corvus: we are down to 6.8GB of disk on gitea01 | 19:26 |
| *** hamzy has joined #openstack-infra | 19:26 | |
| clarkb | though 1.3GB of that is your temp dir | 19:27 |
| clarkb | so really 8.2GB free? I guess that isn't too bad | 19:27 |
| fungi | i think we can survive on that until we rebuild them | 19:27 |
| mordred | ++ | 19:27 |
| clarkb | still about 10k replication tasks to go | 19:28 |
| clarkb | but ya I expect we'll get to a point that is still sustainable | 19:28 |
| clarkb | mordred: did you see my question earlier about the nodepool builder clouds change? | 19:28 |
| clarkb | our backup server is in the disabled group in system-config/inventory/groups.yaml | 19:29 |
| clarkb | so I'm not sure what the best way forward as far as puppet 4ing it is | 19:29 |
| mordred | clarkb: oh - yeah. I agree, I don't think it actually matters. BUT - maybe we should update it anyway for completeness | 19:31 |
| mordred | I can respin if you want | 19:31 |
| clarkb | mordred: ya I'm thinking that might ease confusion in the future if we wonder why it isn't consistent. So maybe a respin is worthwhile | 19:31 |
| mordred | kk. coming right up | 19:32 |
| corvus | clarkb: i'm about to delete the tempdirs, so we'll get some back soon | 19:32 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Allow to select the merge method in Github https://review.opendev.org/617949 | 19:33 |
| clarkb | corvus: ++ | 19:33 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add control plane clouds to nodepool builder clouds.yaml https://review.opendev.org/640027 | 19:34 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Allow to select the merge method in Github https://review.opendev.org/617949 | 19:35 |
| clarkb | also I think the flavor I had in mind for the rebuilt gitea nodes should be 4 cpu instead of 2 | 19:35 |
| clarkb | these big replication events do want cpu available | 19:35 |
| corvus | clarkb: well, most of the time the load average is <1 | 19:36 |
| clarkb | ya that is why I went with the 2vcpu flavor with 8GB ram (we do seem to use about 6-7GB of ram | 19:36 |
| clarkb | we can likely get away with that it iwll just make replication in bulk even slower (which should happen infrequently) | 19:36 |
| openstackgerrit | Dirk Mueller proposed opendev/system-config master: Move openSUSE Tumbleweed into a caching mirror instead https://review.opendev.org/660125 | 19:38 |
| openstackgerrit | Dirk Mueller proposed opendev/system-config master: Start mirroring openSUSE Leap 15.1 https://review.opendev.org/660126 | 19:38 |
| corvus | clarkb: even though our LA is high (>7) we're still 50% idle cpu (and somehow, almost no iowait?) we might be able to increase the number of replication threads even with our current setup | 19:38 |
| clarkb | corvus: ya the 50% of 8 vcpus is where I got the maybe it should be 4vcpus number | 19:38 |
| corvus | clarkb: oh, you said 4 instead of 2 :) | 19:39 |
| *** e0ne has quit IRC | 19:39 | |
| clarkb | corvus: when you were out I had suggested a 2vcpu flavor | 19:40 |
| clarkb | but now thinking 4 would be better | 19:40 |
| corvus | oh. i lacked that context. | 19:40 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support squash merge in Github https://review.opendev.org/661096 | 19:40 |
| clarkb | (no new nodes have been booted yet as we are trying to spin up nodepool built minimal images for control plane) | 19:40 |
| corvus | yes, i don't think we should go down to 2. 4 might be okay. | 19:40 |
| clarkb | corvus: basically 8GB memory seems about right based on cacti showing we use a lot of cache in memory. 8vcpu seemed like a lot. And we probably want double the disk | 19:41 |
| corvus | ++ | 19:41 |
| clarkb | also I dug up info in the docs and to resize a volume you have to detach it from an instance first. So we can't resize the existing volumes I don't think | 19:43 |
| clarkb | But I think we also decided we wanted to avoid that so that we could boot on new images with appropriately sized ext4 journals anyway | 19:43 |
| clarkb | so next steps here are merging mordred's changes to have nodepool build us control plane images, updating them to have bigger journals, then replacing gitea instances one by one | 19:43 |
| corvus | clarkb: do we have inappropriately sized journals? | 19:44 |
| mordred | clarkb: actually - next step is doing the secrets update on bridge | 19:44 |
| mordred | clarkb: but then yet, merging that change | 19:44 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: allow the use of non-bzip compressed stages for building gentoo https://review.opendev.org/661098 | 19:44 |
| clarkb | corvus: logan- discovered that dib built images had/have a 64MB journal. mkfs.ext4 will use a much larger journal for the size of disks we are running. For 80GB filesystems the journal default in mkfs is 512MB | 19:44 |
| mordred | clarkb: shall I go ahead and copy the secrets? | 19:44 |
| clarkb | mordred: ya I think we should make that copy so we are ready when the changes go in | 19:45 |
| mordred | kk. copying now | 19:45 |
| clarkb | corvus: at 128GB the journal is 1GB in size and that seems to be the size for anything bigger than that | 19:45 |
| clarkb | 256MB for 32GB to 64GB fses | 19:45 |
| clarkb | and it scales down to some number like 8MB for really small fses | 19:45 |
| corvus | gotcha | 19:46 |
| clarkb | It does make me wonder if the small journal may have contributed to gitea06's fs problems | 19:46 |
| clarkb | a bigger journal wouldn't have made the problems go away but may have helped it recover? | 19:46 |
| corvus | did we use a nodepool image for that? | 19:47 |
| corvus | or rather a dib image | 19:47 |
| clarkb | corvus: aiui mordred used a locally built dib image (no nodepool but yes dib) | 19:47 |
| corvus | ah | 19:47 |
| fungi | corvus: clarkb: also, having too small a journal for the fs churn implied significant i/o resource contention, i think? | 19:48 |
| mordred | yah. I've forgotten *why* I did that - there was a reason at the time. I think it might have been missing stuff in the stock bionic image and I didn't want to deal with trying to bootstrap package installs via cloud-init | 19:48 |
| clarkb | mordred: iirc you got mad at the broken transition to the network yaml configuration stuff | 19:48 |
| *** gfidente has quit IRC | 19:48 | |
| clarkb | mordred: so you built a new image that was post transition | 19:49 |
| mordred | infra-root: cloud creds copied to group_vars/control-plane-clouds.yaml | 19:49 |
| mordred | clarkb: oh right! | 19:49 |
| fungi | so suspicion was small ext4 journal sizes chosed by dib were suspect in reduced performance we were seeing in limestone | 19:49 |
| fungi | s/chosed/chosen/ | 19:49 |
| clarkb | corvus: latest dib allows us to set the journal size when fs is created | 19:50 |
| clarkb | corvus: we now do that for our test node images and set it to 512MB | 19:50 |
| clarkb | (to match mkfs default) | 19:51 |
| logan- | fungi: not necessarily provider specific, but i'd suspect certain jobs may have been doing more IO churn than necessary due to journal contention | 19:56 |
| logan- | DIB basically just chooses an arbitrary number, 64MB, since it has no way to know what the actual filesystem size will be after the image is booted and growpart runs | 19:58 |
| fungi | yep | 19:58 |
| clarkb | I imagine other cloud images do similar | 19:59 |
| fungi | logan-: i remember you had linked to something originally which had interesting things to say about i/o performance implications of under-sized journals, bit am now not finding it from https://review.opendev.org/633368 | 19:59 |
| clarkb | if someone had time to inspect say ubuntus images might be interesting but enotime | 19:59 |
| fungi | did i dream it? | 19:59 |
| logan- | it was on the project-config patch | 19:59 |
| logan- | https://review.opendev.org/#/c/659215/ | 20:00 |
| logan- | and the link seems to have disappeared :( | 20:00 |
| logan- | the content at the link* | 20:00 |
| fungi | ahh, right | 20:00 |
| fungi | i wonder what they did with that article | 20:01 |
| fungi | it was indeed there at the time, i remember reading it | 20:01 |
| logan- | archive.org has it https://web.archive.org/web/20190226031606/http://www.linux-mag.com/id/7666/ | 20:02 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add cron to pack refs on gitea servers https://review.opendev.org/661102 | 20:02 |
| corvus | clarkb, fungi, mordred: ^ that should tie off the pack-refs work | 20:02 |
| fungi | reminds me of something | 20:03 |
| clarkb | corvus will look in a asec | 20:03 |
| *** Goneri has quit IRC | 20:08 | |
| clarkb | corvus: fungi has a good quetion on the pack refs change | 20:08 |
| *** armax has joined #openstack-infra | 20:09 | |
| fungi | yeah, i wasn't able to find any *.t directories so am thinking it may just be a typo | 20:09 |
| corvus | heh, yeah, i think that was a copy-paste error | 20:12 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add cron to pack refs on gitea servers https://review.opendev.org/661102 | 20:12 |
| corvus | probably resized the terminal and lost some chars without noticing | 20:12 |
| mordred | doh | 20:14 |
| clarkb | corvus: did you check that git and find are both installed in the container image? | 20:14 |
| clarkb | (I don't actually know myself) | 20:15 |
| corvus | clarkb: yeah, i ran that on gitea06 | 20:16 |
| *** jamesmcarthur_ has quit IRC | 20:16 | |
| clarkb | k | 20:16 |
| clarkb | change is approved | 20:16 |
| clarkb | anyone else able to review https://review.opendev.org/#/c/661086/2 and child to put that stuff behind us? | 20:17 |
| *** tosky__ is now known as tosky | 20:18 | |
| corvus | done | 20:21 |
| clarkb | tyty | 20:22 |
| *** kjackal has joined #openstack-infra | 20:22 | |
| clarkb | back down to ~6.7GB free on gitea01 | 20:29 |
| clarkb | about 9k replication events | 20:29 |
| clarkb | s/events/tasks/ | 20:29 |
| *** whoami-rajat has quit IRC | 20:31 | |
| corvus | clarkb: oh, i forgot to remove my tarballs. just did that, free on 01 is 7.8G now | 20:33 |
| corvus | the others are all >8G | 20:33 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: allow the use of non-bzip compressed stages for building gentoo https://review.opendev.org/661098 | 20:33 |
| *** dpawlik has quit IRC | 20:34 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Store hold requests in zookeeper https://review.opendev.org/661114 | 20:40 |
| *** slaweq has quit IRC | 20:41 | |
| *** pcaruana has quit IRC | 20:47 | |
| clarkb | fungi: did you want to review the security impact of https://review.opendev.org/#/c/661047/ (re tosky's dotfile logs thing) | 20:50 |
| openstackgerrit | Merged opendev/system-config master: Remove groups configuration https://review.opendev.org/661086 | 20:50 |
| openstackgerrit | Merged opendev/system-config master: Remove the ask.openstack.org inventory entry https://review.opendev.org/661087 | 20:50 |
| clarkb | fungi: also maybe we should fix all of those warnings from bashate to make the errors more obvious? I'll take a stab at that cleanup | 20:53 |
| *** nicolasbock has quit IRC | 20:58 | |
| *** nicolasbock has joined #openstack-infra | 20:58 | |
| *** kjackal has quit IRC | 21:00 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Cleanup bashate errors to make them easier to understand https://review.opendev.org/661117 | 21:01 |
| clarkb | fungi: infra-root ^ that is a reduce confusion change by cleaning up bashate output | 21:02 |
| fungi | ahh, thanks | 21:06 |
| *** jtomasek has joined #openstack-infra | 21:12 | |
| openstackgerrit | Adam Spiers proposed openstack/cookiecutter master: Add a testenv for lower-constraints https://review.opendev.org/660732 | 21:13 |
| *** raissa has quit IRC | 21:13 | |
| *** raissa has joined #openstack-infra | 21:13 | |
| *** raissa has quit IRC | 21:14 | |
| *** raissa has joined #openstack-infra | 21:14 | |
| *** raissa has quit IRC | 21:15 | |
| *** raissa has joined #openstack-infra | 21:15 | |
| *** raissa has quit IRC | 21:15 | |
| *** armstrong has quit IRC | 21:20 | |
| *** eharney has quit IRC | 21:20 | |
| dmsimard | some changes have been merged but aren't showing up on gitea ? | 21:24 |
| clarkb | dmsimard: we are doing a full sync to pick up refs/chagnes and refs/notes on gitea so there is a backlog | 21:25 |
| dmsimard | https://opendev.org/recordsansible/ara-web/commits/branch/master doesn't show several changes that merged today https://review.opendev.org/#/q/project:recordsansible/ara-web+status:merged | 21:25 |
| *** jtomasek has quit IRC | 21:25 | |
| dmsimard | clarkb: ah, okay, I thought I was going crazy there for a moment | 21:25 |
| fungi | 9072 tasks remaining | 21:27 |
| corvus | dmsimard: you may want to add infra-root as a nick highlight if you haven't | 21:27 |
| openstackgerrit | Merged openstack/project-config master: Create 'Backport-Candidate' for Octavia repos https://review.opendev.org/657657 | 21:28 |
| *** smarcet has quit IRC | 21:35 | |
| dmsimard | corvus: ++, I've been a bit disconnected | 21:37 |
| *** Goneri has joined #openstack-infra | 21:38 | |
| openstackgerrit | Merged openstack/project-config master: End Project Gating https://review.opendev.org/660808 | 21:38 |
| corvus | we should probably ask for project names in commit messages like that ^ :) | 21:38 |
| *** rh-jelabarre has quit IRC | 21:39 | |
| openstackgerrit | Merged openstack/project-config master: Retire ansible-role-k8s-* repos https://review.opendev.org/660446 | 21:39 |
| dmsimard | made me look too. | 21:40 |
| *** bobh has joined #openstack-infra | 21:42 | |
| openstackgerrit | Merged openstack/project-config master: Switch bindep fallback testing to openSUSE Leap 15.x https://review.opendev.org/660130 | 21:43 |
| clarkb | corvus: mordred want to review https://review.opendev.org/#/c/656189/1 a change from pabelanger ot trim docker image sizes a bit | 21:43 |
| openstackgerrit | Merged opendev/puppet-openstackci master: The logs for a job my contain dotfiles, index them https://review.opendev.org/661047 | 21:44 |
| corvus | i kind of want to say that maybe we talked about keeping the caches around in the builder image to make building more efficient | 21:44 |
| corvus | let's hold approval of that for mordred's review | 21:45 |
| *** bobh has quit IRC | 21:45 | |
| corvus | okay i guess mordred is okay with it :) | 21:45 |
| mordred | we need the pip cache for the packages the builder image installs | 21:45 |
| mordred | but not the pip cache or the apt-cache for bindep itself | 21:46 |
| clarkb | mordred: corvus that change removes /var/lib/apt-lists | 21:46 |
| mordred | so I think those are alright | 21:46 |
| mordred | the followup removes pip cache for bindep | 21:46 |
| clarkb | fungi: can you squash https://review.opendev.org/#/c/653881/1 into its parent then we can merge that in for historical purposes? | 21:46 |
| corvus | mordred: you may want to +W it then | 21:46 |
| mordred | incidentally, once we've got better testing of the image build jobs, there are some techniques we can start using to make this whole "don't forget to delete dirs" thing easier | 21:47 |
| corvus | yeah, i'm planning on picking up work on image testing next week | 21:47 |
| * clarkb is trying to clear review backlogs | 21:48 | |
| fungi | clarkb: what's the reason for the squash? will it not merge otherwise? happy to regardless | 21:49 |
| clarkb | fungi: the commit message says "This should be squashed into its parent" | 21:49 |
| clarkb | I don't know why corvus thought that was necessary | 21:49 |
| fungi | ahh | 21:49 |
| fungi | well, for actual running anyway | 21:50 |
| fungi | but sure, will combine them | 21:50 |
| clarkb | http://logs.openstack.org/17/661117/1/check/tox-linters/ef91f47/job-output.txt.gz#_2019-05-23_21_19_55_288685 clean bashate \o/ | 21:55 |
| fungi | does 653138 crash anyone else's gertty? | 21:58 |
| corvus | not i | 21:59 |
| corvus | what's the traceback? | 21:59 |
| *** armax has quit IRC | 22:03 | |
| clarkb | corvus: do we need to maybe consider packing refs in the foreground to free up more disk on the gitea servers? (back down to 6.2GB with about 8800 replication tasks to go) | 22:05 |
| fungi | corvus: http://paste.openstack.org/show/752011/ | 22:05 |
| clarkb | (its hard to gauge how much disk is necessary via the replication tasks count since each repo isn't the same) | 22:05 |
| corvus | clarkb: hrm, i wouldn't expect it to be a problem based on df in /opt on review01 | 22:06 |
| clarkb | corvus: I think those repos are all packed | 22:07 |
| *** armax has joined #openstack-infra | 22:07 | |
| *** Goneri has quit IRC | 22:08 | |
| openstackgerrit | Merged opendev/system-config master: Clean up apt cache https://review.opendev.org/656189 | 22:09 |
| corvus | clarkb: ah, i think you're right and they are packed periodically (probably repacked) | 22:11 |
| corvus | we may want to look into doing repacks on gitea instead of pack-refs actually | 22:11 |
| corvus | clarkb: think i should just run the cron pack-refs command? | 22:12 |
| clarkb | corvus: ya pack files are compressed so I'm thinking that should free up a bit of disk (which I think we want given its hard to tell how much headroom we have) | 22:12 |
| corvus | clarkb: no this is just pack-refs | 22:12 |
| corvus | sorry, there's 2 things under discussion and streams may be getting crossed | 22:13 |
| clarkb | oh pack-refs only sticks the ref pointers in a file and doesn't pack objects got it | 22:13 |
| *** mriedem has quit IRC | 22:13 | |
| corvus | yeah, and pack-refs is the only thing we've done on gitea servers so far | 22:13 |
| clarkb | I think we need to pack objects or we may run out of disk sometime between now and 8k replication events from now | 22:13 |
| clarkb | (it is possible that won't happen as I said very hard to tell) | 22:14 |
| corvus | any idea where the ref packing for the /opt mirrors happens? | 22:14 |
| *** rfarr_ has quit IRC | 22:14 | |
| clarkb | ya one sec | 22:15 |
| corvus | ah found it | 22:15 |
| corvus | 17 4 * * 0 find /opt/lib/git -type d -name "*.git" -print -exec git --git-dir="{}" gc \; | 22:15 |
| fungi | yeah | 22:15 |
| fungi | crontab for gerrit2 user | 22:15 |
| clarkb | hrm I can't link it because opendev saying things like secure connection failed | 22:16 |
| *** sreejithp__ has quit IRC | 22:16 | |
| fungi | i guess git gc also packs objects? | 22:16 |
| clarkb | fungi: yes | 22:16 |
| corvus | and packs refs | 22:16 |
| *** sreejithp has joined #openstack-infra | 22:16 | |
| clarkb | https://opendev.org/opendev/system-config/ broken for anyone else or just me? | 22:16 |
| fungi | taking a while to load for me | 22:17 |
| fungi | i wonder if i'm having ipv6 connectivity issues | 22:18 |
| logan- | doesn't load for me, also opendev.org took a good 30 seconds or so | 22:18 |
| logan- | but / is fast upon refresh | 22:18 |
| logan- | system-config loaded after a minute or two | 22:19 |
| corvus | we may be running into contention with the replication | 22:19 |
| fungi | eventually loaded for me | 22:19 |
| fungi | no errors | 22:20 |
| corvus | https://gitea07.opendev.org:3000/opendev/system-config is erroring | 22:20 |
| fungi | doesn't seem to be a v6 to v4 fallback delay, i can connect over v6 to it just fine | 22:20 |
| clarkb | there is/was a container restart | 22:20 |
| clarkb | on gitea01 | 22:20 |
| clarkb | so that may be spinup time? | 22:20 |
| fungi | oh, maybe for the openid disablement? | 22:21 |
| corvus | 67693ca76264 opendevorg/gitea:latest "/usr/bin/entrypoint…" 5 minutes ago Restarting (2) 53 seconds ago giteadocker_gitea-web_1 | 22:21 |
| corvus | thats 07 | 22:21 |
| corvus | so yeah, that's probably what's happening | 22:21 |
| corvus | maybe we should orchestrate restarts with haproxy | 22:21 |
| corvus | (since we can theoretically do that now with ansible) | 22:21 |
| clarkb | corvus: also something freed up some disk (did you run the pack-refs command?) | 22:22 |
| corvus | clarkb: nop | 22:22 |
| clarkb | I wonder if gitea does stuff like that on startup | 22:22 |
| fungi | which could possibly introduce startup delays | 22:22 |
| corvus | i dunno what it does on startup.. 07 looks idle | 22:23 |
| corvus | panic: Failed to execute 'git config --global core.quotepath false': error: could not lock config file /data/git/.gitconfig: File exists | 22:23 |
| corvus | possible side effect of the container being stopped in the middle of a git command running? | 22:24 |
| clarkb | or running the fetch commands earlier? | 22:24 |
| corvus | it's only happening on one host though | 22:24 |
| clarkb | gitea01's .gitconfig is super simple | 22:25 |
| clarkb | I half expect we can move that aside on gitea07 and it would be happy | 22:25 |
| corvus | it's the lock file it's complaining about | 22:25 |
| corvus | there are no long-running git processes running on gitea07 so i will just remove the .lock file | 22:26 |
| clarkb | .gitconfig.lock for anyone else wondering (in the same dir) | 22:26 |
| clarkb | corvus: oh ya on 01 that .lock file has come and gone | 22:27 |
| clarkb | so guessing if we stop the container when the .lock is present it gets unhappy on startup | 22:27 |
| corvus | 07 is starting up now | 22:27 |
| corvus | and running | 22:27 |
| clarkb | maybe our init container needs to rm that file before starting gitea? | 22:28 |
| openstackgerrit | Merged opendev/system-config master: Use --no-cache-dir with pip https://review.opendev.org/656191 | 22:29 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add cron to gc on gitea servers https://review.opendev.org/661102 | 22:31 |
| corvus | clarkb, fungi: ^ shall i run that now on all gitea hosts? | 22:31 |
| fungi | seems prudent, yes | 22:32 |
| clarkb | should we test it on one first? | 22:32 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add opendev migration repo rename scripts https://review.opendev.org/653138 | 22:32 |
| clarkb | (mostly curious if gitea will be happy with that happening under it) | 22:32 |
| corvus | sure i'll do that on 07 | 22:32 |
| clarkb | sounds good | 22:32 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add cron to gc on gitea servers https://review.opendev.org/661102 | 22:34 |
| clarkb | corvus: re ^ you might also want to drop the -t since ansible may not ssh with a tty? | 22:35 |
| corvus | clarkb: "-t" works from the command line on bridge; "-it" does not | 22:36 |
| clarkb | ah | 22:36 |
| corvus | cron could be different, but i think this might work | 22:36 |
| clarkb | 07 is slow (due to load I expect) but I can request opendev/system-config from it and it renders pages | 22:37 |
| corvus | clarkb: it actually *just* finished gcing system-config | 22:37 |
| corvus | and yeah, still seems to work for me... | 22:37 |
| clarkb | that could explain it :) but ya I was able to browse around system-config so I think it probably is happy with the GCing under it | 22:38 |
| corvus | clarkb: happily, you may have even read from the repo during the gc which is a great test | 22:38 |
| clarkb | (if a little slow when also doing the replicate world step) | 22:38 |
| clarkb | and that is freeing up disk on 07 | 22:38 |
| corvus | i will run the same on the other hosts now | 22:39 |
| clarkb | k | 22:39 |
| fungi | excellent | 22:39 |
| *** sreejithp has quit IRC | 22:39 | |
| clarkb | note that 06 may still be quite unhappy so errors there probably don't mean much | 22:40 |
| corvus | oh yeah, i won't do it on 06 | 22:40 |
| corvus | i also didn't do the fetches on 06 | 22:40 |
| corvus | replication is very fast on 06; it seems to have been failing fast | 22:41 |
| corvus | gc running everywhere now | 22:41 |
| clarkb | I expect this will make disk usage much more closely match review.o.o:/opt which should mean we'll have plenty of disk | 22:45 |
| fungi | also possible pushes are way faster when there's no ci system making constant fetch requests | 22:45 |
| clarkb | gitea01 is doing the openstack ansible repos now (for a sense of where in the alphabetical queueing it is) | 22:47 |
| tosky | clarkb: should I already see the new setting on logs-dev.openstack.org? | 22:50 |
| clarkb | tosky: I think it is delayed due to the replication backlog in gerrit | 22:51 |
| clarkb | tosky: so it might be a little while before it applies (I checked on the server and the config isn't applied yet) | 22:51 |
| clarkb | under 8k replications events finally :) | 22:52 |
| tosky | clarkb: oh, that's fine, I was worrying that it didn't work :) | 22:54 |
| tosky | I will recheck in few hours (my morning) | 22:54 |
| tosky | thanks! | 22:54 |
| *** tkajinam has joined #openstack-infra | 22:57 | |
| *** smarcet has joined #openstack-infra | 22:57 | |
| *** panda|rover has quit IRC | 22:59 | |
| *** panda has joined #openstack-infra | 22:59 | |
| *** harlowja has quit IRC | 23:01 | |
| *** michael-beaver has quit IRC | 23:03 | |
| *** tosky has quit IRC | 23:05 | |
| *** rlandy has quit IRC | 23:06 | |
| clarkb | now down to 7.7k replication tasks. Seems to be moving a bit quicker now (possibly because it got through most of the big repos) | 23:06 |
| *** sarob has quit IRC | 23:07 | |
| clarkb | 07 is done GC'ing | 23:11 |
| clarkb | ended up with 11GB free | 23:11 |
| clarkb | er no 13. 01 has 11 and is still running | 23:11 |
| clarkb | so that looks a lot more like what we expect | 23:12 |
| clarkb | we probably want to rerun the GC'ing after the replication completes (as things that haven't yet replicated won't be packed) | 23:13 |
| clarkb | depending on how much disk space we end up with in the end it may be fine to wait for the crno to run | 23:14 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: encrypt_secret: display the full_url on error https://review.opendev.org/661134 | 23:18 |
| *** rcernin has joined #openstack-infra | 23:18 | |
| *** smarcet has quit IRC | 23:19 | |
| *** Weifan has quit IRC | 23:22 | |
| *** _erlon_ has quit IRC | 23:24 | |
| clarkb | GC has ended on gitea01 now too so is probably done nearly across the board | 23:27 |
| *** smarcet has joined #openstack-infra | 23:27 | |
| *** michael-beaver has joined #openstack-infra | 23:35 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: validate-zone-db : add job and make more generic https://review.opendev.org/661138 | 23:43 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: Add zone-check job https://review.opendev.org/660888 | 23:44 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: [dnm] sample of zone misconfiguration to fail testing https://review.opendev.org/660966 | 23:44 |
| ianw | fungi: i've found that when i save a zone.db file emacs bumps the serial number automagically | 23:46 |
| clarkb | under 7k replication events now | 23:47 |
| clarkb | things are looking good now on that front. I'm about to pop out for dadops things | 23:48 |
| clarkb | ianw: so I did a sanity check that we didn't miss any puppet-3 nodes. The two arm64 instances (mirror and builder) are puppet3. Your replacement work for the mirrors will handle one of those and the last instance I found was our backup host | 23:48 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: validate-zone-db : add job and make more generic https://review.opendev.org/661138 | 23:49 |
| clarkb | ianw: the backup host is currently in the disabled group in inventory/groups.yaml implying we don't touch it often. Any idea if that should be updated to puppet-4? | 23:49 |
| clarkb | or is that a task for the next time we build a backup host? | 23:49 |
| ianw | clarkb: yeah, i think that's on purpose because we don't want any risk of it being wiped by some automagic | 23:49 |
| ianw | but ? ... it's just always been like that since i got here :) | 23:49 |
| clarkb | ok I'll have to think about that one then | 23:50 |
| clarkb | ianw: also not sure if you saw the email from linaro. TLDR is someone new will be working with us in the next few weeks (probably) and they want to spin up a new cloud hosted on packethost | 23:50 |
| *** mattw4 has quit IRC | 23:50 | |
| clarkb | oh and they have more capacity in london if we want to push on that more | 23:50 |
| ianw | we could make a new backup host ... does it do more than install bup really? seems ansible-isable | 23:51 |
| *** trident has quit IRC | 23:51 | |
| clarkb | ianw: I think it is install users and firewall and bup so really basic | 23:51 |
| clarkb | and now I must walk the kids to the playground to keep them from falling asleep early | 23:52 |
| ianw | clarkb: yeah, saw that, will take a look at where we're at. | 23:52 |
| ianw | clarkb: did you see https://mirror.dfw.rax.opendev.org ? | 23:52 |
| clarkb | ianw: I did | 23:52 |
| ianw | i'll attach some volumes to make it have space similar to the other mirrors; then i think we can probably CNAME it into rotation and see how it goes for a bit | 23:53 |
| clarkb | nice | 23:53 |
| ianw | (when we're around to monitor it :) | 23:53 |
| *** trident has joined #openstack-infra | 23:53 | |
| clarkb | ianw: then we can add vhosts for tls'd docker hub proxy? | 23:53 |
| ianw | yep, i mean could do that now and just point everything at the one mirror for testing? | 23:54 |
| clarkb | ianw: ya that might be a good idea. | 23:54 |
| clarkb | get tseting out of the way early as possible | 23:54 |
| ianw | i'll ad the extra volumes and lvm setup today and we can take it from there | 23:54 |
| *** Weifan has joined #openstack-infra | 23:57 | |
| *** lseki has quit IRC | 23:58 | |
| *** Weifan has quit IRC | 23:59 | |
| *** Weifan has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
