| ianw | haven't released it yet, need to afk for a few minutes if anyone wants to look at /afs/.openstack.org/project/tarballs.opendev.org | 00:00 |
|---|---|---|
| clarkb | I've poked around briefly I see what you mean re being owner of files and dirs but ya I think the afs acls make that a non issue | 00:03 |
| clarkb | and afsmon seems good? | 00:03 |
| *** hillpd has joined #opendev | 00:05 | |
| ianw | yeah i dunno what the deal is there | 00:13 |
| ianw | ok i've released it, so will watch for weirdness | 00:14 |
| ianw | i've approved the apache redirects and will monitor that too | 00:15 |
| *** hamalq has quit IRC | 00:16 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix certificate issue with use buildset registry https://review.opendev.org/741584 | 00:26 |
| openstackgerrit | Merged opendev/system-config master: tarballs: Add rewrite rules for tenant moves https://review.opendev.org/755181 | 00:43 |
| *** weshay|ruck has quit IRC | 00:54 | |
| *** weshay has joined #opendev | 00:55 | |
| *** weshay is now known as weshay|ruck | 00:55 | |
| *** qchris has quit IRC | 01:12 | |
| *** dulek has quit IRC | 01:20 | |
| *** dulek has joined #opendev | 01:20 | |
| *** qchris has joined #opendev | 01:24 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: tarballs.opendev.org: better redirects https://review.opendev.org/755678 | 01:49 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: tarballs.opendev.org: better redirects https://review.opendev.org/755678 | 02:22 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Revert "Disable broken fetch-sphinx-tarball test job" https://review.opendev.org/753199 | 03:16 |
| *** hillpd has quit IRC | 03:21 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Revert "Disable broken fetch-sphinx-tarball test job" https://review.opendev.org/753199 | 03:28 |
| openstackgerrit | Merged opendev/system-config master: tarballs.opendev.org: better redirects https://review.opendev.org/755678 | 03:48 |
| *** hillpd has joined #opendev | 03:49 | |
| *** Topner has joined #opendev | 03:49 | |
| *** Topner has quit IRC | 03:54 | |
| ianw | ERROR: Package 'cliff' requires a different Python: 2.7.17 not in '>=3.6' | 04:51 |
| ianw | i've seen this a couple of times | 04:51 |
| ianw | i wonder if our bad mirror pypi is back | 04:51 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Revert "Disable broken fetch-sphinx-tarball test job" https://review.opendev.org/753199 | 04:59 |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Add backup02 in vexxhost https://review.opendev.org/755690 | 05:18 |
| frickler | ianw: yes, this also looks like bad pypi, pip on py2 trying to install latest setuptools https://zuul.opendev.org/t/openstack/build/f8146de7ac674481af95762893587a6f | 05:31 |
| frickler | all instances I've seen for this were on ovh, so likely some regional thing again | 05:32 |
| ianw | frickler: sorry i don't have time to exactly dig into the proxy cache like we did last time. i'm not sure how we can prove it ... last time it worked reaching out to dstufft directly in pypa-dev mostly | 05:33 |
| frickler | ianw: yeah, left a message over there | 05:36 |
| frickler | ianw: do you have a job pointer for the cliff failure? | 05:37 |
| ianw | umm, zuul-jobs, let me see | 05:37 |
| ianw | https://zuul.opendev.org/t/zuul/build/87db92acdc874593935c7d8e6e60c559 | 05:38 |
| ianw | i wonder if i can find that old pypi server they identified as out of sync | 05:38 |
| ianw | https://mirror.dub1.pypi.io/ was the one last time | 05:41 |
| ianw | it is saying it is in sync as of 2020-10-02 05:40:01 UTC. | 05:41 |
| ianw | which is ... now? | 05:41 |
| ianw | 60c559 was vexxhost ca-ymq-1 | 05:43 |
| frickler | IIUC the issue last time was an outdated bandersnatch mirror that got used as last fallback for the CDN, so the issue might only manifest during certain times when load is high | 05:44 |
| ianw | yeah, i don't know if maybe they have other mirror hosts | 05:45 |
| *** hamalq has joined #opendev | 06:12 | |
| *** eolivare has joined #opendev | 06:29 | |
| *** roman_g has joined #opendev | 06:31 | |
| *** roman_g has quit IRC | 06:45 | |
| *** roman_g has joined #opendev | 06:48 | |
| *** roman_g has quit IRC | 06:52 | |
| *** slaweq has joined #opendev | 07:01 | |
| *** DSpider has joined #opendev | 07:05 | |
| *** hamalq has quit IRC | 07:13 | |
| *** andrewbonney has joined #opendev | 07:20 | |
| *** fressi has joined #opendev | 07:35 | |
| *** tosky has joined #opendev | 07:43 | |
| *** mrunge has quit IRC | 07:53 | |
| *** mrunge has joined #opendev | 07:54 | |
| *** ralonsoh has joined #opendev | 07:58 | |
| *** hashar has joined #opendev | 07:58 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| *** slaweq has quit IRC | 08:11 | |
| *** roman_g has joined #opendev | 08:12 | |
| *** slaweq has joined #opendev | 08:16 | |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: ensure-docker: validate network connectivity https://review.opendev.org/755505 | 08:27 |
| *** tkajinam has quit IRC | 08:29 | |
| *** tkajinam has joined #opendev | 08:29 | |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: ensure-docker: validate network connectivity https://review.opendev.org/755505 | 08:32 |
| *** tkajinam has quit IRC | 09:18 | |
| *** ysandeep|sick is now known as ysandeep|holiday | 09:55 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Add Fedora 33 support https://review.opendev.org/755750 | 10:03 |
| *** roman_g has quit IRC | 10:31 | |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: ensure-docker: validate network connectivity https://review.opendev.org/755505 | 10:35 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils https://review.opendev.org/742736 | 10:39 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Consolidate common log upload code into module_utils https://review.opendev.org/742736 | 10:40 |
| *** draliana has joined #opendev | 10:48 | |
| *** lpetrut has joined #opendev | 11:44 | |
| *** sshnaidm is now known as sshnaidm|afk | 11:46 | |
| *** iurygregory has quit IRC | 11:55 | |
| *** iurygregory has joined #opendev | 11:57 | |
| *** roman_g has joined #opendev | 12:06 | |
| *** slaweq has quit IRC | 12:09 | |
| *** slaweq has joined #opendev | 12:09 | |
| *** draliana has quit IRC | 12:12 | |
| *** hashar has quit IRC | 13:25 | |
| *** fressi has quit IRC | 13:45 | |
| *** priteau has joined #opendev | 13:52 | |
| *** hashar has joined #opendev | 13:55 | |
| *** priteau has quit IRC | 13:58 | |
| *** roman_g has quit IRC | 14:01 | |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: ensure-docker: validate network connectivity https://review.opendev.org/755505 | 14:04 |
| openstackgerrit | Javier Peña proposed openstack/project-config master: CentOS 8 Stream initial deployment https://review.opendev.org/734791 | 14:18 |
| *** priteau has joined #opendev | 14:19 | |
| fungi | ianw: frickler: manifest when load is high or the cdn has trouble reaching the server for other reasons | 14:21 |
| *** dviroel has quit IRC | 14:26 | |
| *** dviroel has joined #opendev | 14:26 | |
| *** Eighth_Doctor has quit IRC | 14:33 | |
| *** hillpd_ has joined #opendev | 14:40 | |
| *** hillpd has quit IRC | 14:40 | |
| *** hillpd_ is now known as hillpd | 14:40 | |
| clarkb | fungi: ya the reason there is no python requires metadata is that it is just a bandersnatch mirror with essentially mod auto index type indexes I think | 14:47 |
| clarkb | previously there was discussion on how they could pregenerate indexes in object storage and fallback to that maybe (and that would include the metadata) | 14:48 |
| *** Eighth_Doctor has joined #opendev | 14:48 | |
| fungi | yep, i recall. there's discussion in #pypa-dev right now about turning it back off again | 14:51 |
| clarkb | fungi: https://review.opendev.org/#/c/748294/ would be a good one to land before doing gerrit upgrade testing so that we have an image ready with that branch | 14:54 |
| fungi | thanks | 15:02 |
| clarkb | also I've begun updating https://etherpad.opendev.org/p/gerrit-2.16-upgrade with concrete commands from my notes as well as adding in edits luca suggested | 15:02 |
| clarkb | ok I think that gets us as far as 2.16 without notedb. Then we can sort out the notedb transition on review-test once we've tested ^ | 15:06 |
| fungi | awesome | 15:07 |
| clarkb | the next step on review-test is to just turn on gerrit and see if our production snapshot from yesterday is viable. If it is we'll shut it back down again and snapshot the cinder volume | 15:08 |
| clarkb | corvus: frickler ^ fyi I expect we'll be doing that sometime in the near future. fungi and I have looked over configs and turned off exim etc etc and don't expect it to touch anything production like if turned on. If you have any concerns or would like to follow along let me know | 15:09 |
| corvus | clarkb: ack, thx. i'm heading out to get a flu vaccine, should be back in a bit. | 15:28 |
| *** lpetrut has quit IRC | 15:31 | |
| openstackgerrit | Merged opendev/system-config master: Use the new stable-2.16 branch of its-storyboard https://review.opendev.org/748294 | 15:39 |
| clarkb | we've got it turned on. Have discovered we need to update iptables rules, but so far things are ok. Will put notes on the bits we had trouble with in the etherpad once sorted | 15:42 |
| *** priteau has quit IRC | 15:45 | |
| *** priteau has joined #opendev | 15:53 | |
| openstackgerrit | zbr proposed zuul/zuul-jobs master: Update ensure-docker for new releases https://review.opendev.org/752630 | 16:13 |
| clarkb | review-tests' gerrit is off now and we've detached the cinder volume and are snapshotting it | 16:23 |
| clarkb | once that is done it will get reattached and we'll be in a good spot to start testing and timing things. There is one thing I want to check on locally first since I Think it will be quicker and easier but once that is done I expect we'll start getting hard data on the upgrade timing and process | 16:24 |
| *** hashar has quit IRC | 16:28 | |
| *** slaweq has quit IRC | 16:32 | |
| *** ralonsoh has quit IRC | 16:37 | |
| *** hamalq has joined #opendev | 16:58 | |
| clarkb | the volume is reattached and things look good. I need to go find breakfast then I'll be trying to test gerrit things locally I expect | 17:05 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Increase the default TTL in the opendev.org zone to 1h https://review.opendev.org/755550 | 17:06 |
| *** andrewbonney has quit IRC | 17:08 | |
| *** eolivare has quit IRC | 17:12 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Partial address ansible-lint E208 https://review.opendev.org/748480 | 17:18 |
| *** eolivare has joined #opendev | 17:18 | |
| *** eolivare has quit IRC | 17:19 | |
| *** Tengu has quit IRC | 17:48 | |
| corvus | that was not fast | 17:52 |
| fungi | the time from approval to merge on 748480? | 18:03 |
| clarkb | or maybe getting vaccinated | 18:03 |
| fungi | oh! right, that | 18:04 |
| fungi | context, i lost it ;) | 18:04 |
| clarkb | our pediatrician had a drive through flu vaccine clinic for the whole family so we did that and it was actually pretty good | 18:04 |
| openstackgerrit | Tristan Cacqueray proposed openstack/project-config master: infra-package-needs: add support for centos-8-stream https://review.opendev.org/755860 | 18:19 |
| clarkb | infra-root if you have time for https://review.opendev.org/755352 and https://review.opendev.org/755355 those are two gitea related improvements that would be good to get in | 18:23 |
| clarkb | I can watch them today too as I think I'm going to take a break from the gerrit stuff now (things are looking great. my concern that wanted to check out before doing prod like upgrades turned out to be a non issue in local testing) | 18:23 |
| clarkb | hoping that early next week we'll be able to test production like upgrades | 18:23 |
| clarkb | the reason for not doing them now is I expect them to be long and slow and I don't want to worry about them over the weekend | 18:24 |
| *** Tengu has joined #opendev | 18:25 | |
| *** hashar has joined #opendev | 18:43 | |
| *** roman_g has joined #opendev | 18:52 | |
| openstackgerrit | Merged openstack/project-config master: infra-package-needs: add support for centos-8-stream https://review.opendev.org/755860 | 18:58 |
| *** priteau has quit IRC | 19:17 | |
| *** diablo_rojo has quit IRC | 19:56 | |
| *** tosky has quit IRC | 19:57 | |
| *** tosky has joined #opendev | 19:58 | |
| *** tosky has quit IRC | 20:19 | |
| *** roman_g has quit IRC | 20:46 | |
| *** roman_g has joined #opendev | 20:47 | |
| *** roman_g has quit IRC | 20:47 | |
| *** roman_g has joined #opendev | 20:48 | |
| *** roman_g has quit IRC | 20:48 | |
| *** roman_g has joined #opendev | 20:48 | |
| *** roman_g has quit IRC | 20:49 | |
| *** roman_g has joined #opendev | 20:49 | |
| *** roman_g has quit IRC | 20:49 | |
| *** roman_g has joined #opendev | 20:50 | |
| *** roman_g has quit IRC | 20:50 | |
| *** roman_g has joined #opendev | 20:51 | |
| *** roman_g has quit IRC | 20:51 | |
| *** roman_g has joined #opendev | 20:51 | |
| *** roman_g has quit IRC | 20:52 | |
| *** roman_g has joined #opendev | 20:52 | |
| *** roman_g has quit IRC | 20:53 | |
| *** roman_g has joined #opendev | 20:53 | |
| *** roman_g has joined #opendev | 20:54 | |
| *** roman_g has joined #opendev | 20:55 | |
| *** roman_g has quit IRC | 20:55 | |
| *** roman_g has joined #opendev | 20:55 | |
| *** hashar has quit IRC | 20:56 | |
| *** roman_g has quit IRC | 20:56 | |
| *** roman_g has joined #opendev | 20:56 | |
| *** roman_g has quit IRC | 20:57 | |
| *** roman_g has joined #opendev | 20:57 | |
| *** roman_g has quit IRC | 20:57 | |
| *** roman_g has joined #opendev | 20:58 | |
| *** roman_g has quit IRC | 20:58 | |
| openstackgerrit | Merged opendev/system-config master: Record stacktraces when logging errors in gitea https://review.opendev.org/755352 | 21:48 |
| fungi | corvus: clarkb: northrop grunman launch next door at m.a.r.s. tonight (01:16 utc) | 22:01 |
| fungi | antares | 22:02 |
| fungi | postponed from yesterday, postponed from earlier this week | 22:09 |
| fungi | weather looks better this time though | 22:09 |
| * fungi is saddened by mission reports referencing "the united states space force" | 22:11 | |
| clarkb | looks like we don't restart giteas when we update app.ini | 22:24 |
| clarkb | I'm going to restart 01 now | 22:24 |
| clarkb | then if it looks happy roll through the others | 22:24 |
| clarkb | cool we get tracebacks now (know this beacuse it failing to talk to the db on startup while it waited for the db to start too) | 22:28 |
| clarkb | it seems to be happy so I will proceed with the others now | 22:29 |
| clarkb | #status log Restarted gitea cluster in sequence to pick up new config that emits tracebacks to logs on errors | 22:35 |
| openstackstatus | clarkb: finished logging | 22:35 |
| *** qchris has quit IRC | 22:40 | |
| *** qchris has joined #opendev | 22:53 | |
| *** DSpider has quit IRC | 22:54 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!