| *** mlavalle has quit IRC | 00:02 | |
| ianw | i really don't want to roll back the sdk version on bridge, i feel like there might be other things we need that for | 00:23 |
|---|---|---|
| ianw | i'm thinking a manual run of the cloud launcher out of a venv with an old sdk might be the best way to get this done this once | 00:23 |
| fungi | that seems reasonable | 00:38 |
| *** tkajinam has quit IRC | 00:55 | |
| *** tkajinam has joined #opendev | 00:56 | |
| ianw | https://mirror.regionone.osuosl.opendev.org/ is alive, yay! | 01:05 |
| fungi | awesome! | 01:07 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Fix typo on OSU OSL password template https://review.opendev.org/c/opendev/system-config/+/786149 | 01:09 |
| *** artom has quit IRC | 01:56 | |
| openstackgerrit | Merged opendev/system-config master: Add planet.openstack.org redirect to static https://review.opendev.org/c/opendev/system-config/+/785993 | 02:01 |
| ianw | when ^ deploys i'll switch the dns | 02:08 |
| openstackgerrit | Merged opendev/system-config master: Fix typo on OSU OSL password template https://review.opendev.org/c/opendev/system-config/+/786149 | 02:11 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add OSU OSL to nodepool configuration https://review.opendev.org/c/opendev/system-config/+/786155 | 02:20 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add OSUOSL resources to nodepool https://review.opendev.org/c/openstack/project-config/+/786156 | 02:26 |
| *** hamalq has quit IRC | 02:29 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add OSU OSL to nodepool configuration https://review.opendev.org/c/opendev/system-config/+/786155 | 02:34 |
| ianw | fungi: ^ thanks; i just renamed the var to "id" match what it is | 02:34 |
| fungi | ahh, cool | 02:35 |
| fungi | also i left a comment on the other one | 02:36 |
| ianw | yeah, that seemed to come in via https://review.opendev.org/c/openstack/project-config/+/737361 but didn't update the autogen scripts | 02:38 |
| fungi | aha, makes sense | 02:44 |
| fungi | looks like it was an experiment we never cleaned up | 02:44 |
| openstackgerrit | Merged opendev/system-config master: Stop managing planet01.openstack.org https://review.opendev.org/c/opendev/system-config/+/785994 | 04:13 |
| *** ykarel has joined #opendev | 04:14 | |
| ianw | #status log planet.openstack.org redirected to opendev.org/openstack/openstack-planet via static.o.o, server removed and dns entries cleaned up | 04:32 |
| openstackstatus | ianw: finished logging | 04:32 |
| ianw | vale planet, and our RSS based future that wasn't to be | 04:34 |
| openstackgerrit | Merged opendev/system-config master: Add OSU OSL to nodepool configuration https://review.opendev.org/c/opendev/system-config/+/786155 | 04:48 |
| *** marios has joined #opendev | 04:59 | |
| *** whoami-rajat_ has joined #opendev | 05:20 | |
| *** ralonsoh has joined #opendev | 05:21 | |
| *** cenne-away is now known as cenne | 05:39 | |
| *** tinwood has quit IRC | 06:22 | |
| *** hemanth_n has joined #opendev | 06:24 | |
| *** ysandeep|holiday is now known as ysandeep | 06:24 | |
| *** tinwood has joined #opendev | 06:25 | |
| *** zbr has quit IRC | 06:30 | |
| yoctozepto | fungi: \o/ for jeepyb, thanks! | 06:31 |
| *** zbr has joined #opendev | 06:32 | |
| *** eolivare has joined #opendev | 06:32 | |
| *** knikolla has quit IRC | 06:34 | |
| *** ildikov has quit IRC | 06:35 | |
| *** sboyron has joined #opendev | 06:35 | |
| *** knikolla has joined #opendev | 06:36 | |
| *** ildikov has joined #opendev | 06:36 | |
| *** lpetrut has joined #opendev | 06:57 | |
| *** amoralej|off is now known as amoralej | 06:59 | |
| *** amoralej is now known as amoralej|off | 06:59 | |
| *** andrewbonney has joined #opendev | 07:14 | |
| *** marios has quit IRC | 07:21 | |
| *** DSpider has joined #opendev | 07:34 | |
| *** AJaeger has joined #opendev | 07:35 | |
| AJaeger | Hi, Is publishing to docs.openstack.org broken? The promote job in https://review.opendev.org/c/openstack/openstack-manuals/+/784910 run 30 mins ago and https://static.opendev.org/docs/?C=M;O=D shows that docs.openstack.org is not updated. | 07:36 |
| AJaeger | infra-root ^ | 07:38 |
| * AJaeger tries to push the Wallaby release to docs.o.o early and run into this | 07:39 | |
| *** tosky has joined #opendev | 07:50 | |
| AJaeger | All good, docs are finally pushed - didn't remember that it takes that long | 07:51 |
| *** rpittau|afk is now known as rpittau | 07:55 | |
| *** jpena|off is now known as jpena | 07:57 | |
| *** AJaeger has quit IRC | 08:07 | |
| hrw | yay! moar aarch64 nodes! | 08:12 |
| hrw | and it started with a tweet: https://twitter.com/haerwu/status/1379713900275503105 | 08:14 |
| *** slaweq_ has joined #opendev | 08:19 | |
| *** slaweq has quit IRC | 08:19 | |
| *** marios has joined #opendev | 08:21 | |
| *** slaweq_ is now known as slaweq | 08:51 | |
| *** klonn has joined #opendev | 09:16 | |
| *** amoralej|off is now known as amoralej | 09:20 | |
| *** dtantsur|afk is now known as dtantsur | 09:45 | |
| ttx | hrw: nice! | 10:04 |
| *** fresta has joined #opendev | 10:17 | |
| *** whoami-rajat_ is now known as whoami-rajat | 10:17 | |
| *** snapdeal has joined #opendev | 10:26 | |
| *** snapdeal has quit IRC | 10:26 | |
| *** klonn has quit IRC | 11:04 | |
| *** snapdeal has joined #opendev | 11:05 | |
| *** avass has quit IRC | 11:07 | |
| *** avass has joined #opendev | 11:08 | |
| *** eolivare_ has joined #opendev | 11:12 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Revert "Revert "Temporarily serve static sites from AFS R+W vols"" https://review.opendev.org/c/opendev/system-config/+/786205 | 11:13 |
| fungi | infra-root: we should maybe consider merging that ^ | 11:13 |
| *** tkajinam has quit IRC | 11:14 | |
| fungi | AJaeger's comment earlier got me looking, and indeed there was a lag in publishing static site content at the time he observed, owing to a nonzero content update in the tarballs volume | 11:14 |
| *** eolivare has quit IRC | 11:15 | |
| fungi | the openstack release activity today represents a decidedly nontrivial update to the tarballs volume by comparison | 11:15 |
| fungi | it's possible that could hold up publication of updates to all our static sites for hours | 11:15 |
| *** avass has quit IRC | 11:31 | |
| *** jpena is now known as jpena|lunch | 11:32 | |
| *** avass has joined #opendev | 11:32 | |
| *** eolivare_ has quit IRC | 11:35 | |
| *** ykarel_ has joined #opendev | 11:37 | |
| yoctozepto | hrw: well, that was quick! | 11:39 |
| yoctozepto | congrats | 11:39 |
| *** ykarel has quit IRC | 11:40 | |
| *** zoharm has joined #opendev | 11:45 | |
| *** ykarel_ is now known as ykarel | 11:47 | |
| *** hrw has quit IRC | 11:50 | |
| fungi | i'm setting 786205 to wip for now, apparently whatever caused the 45-minute vos release of the tarballs volume earlier (someone publishing a set of container images maybe?) was an anomaly. the full openstack release was barely a blip on the radar by comparison | 11:51 |
| *** eolivare_ has joined #opendev | 12:16 | |
| *** amoralej is now known as amoralej|lunch | 12:21 | |
| *** marios is now known as marios|call | 12:23 | |
| *** hemanth_n has quit IRC | 12:24 | |
| *** AJaeger has joined #opendev | 12:28 | |
| *** jpena|lunch is now known as jpena | 12:29 | |
| *** snapdeal has quit IRC | 12:30 | |
| *** snapdeal has joined #opendev | 12:39 | |
| *** klonn has joined #opendev | 12:43 | |
| *** ysandeep is now known as ysandeep|afk | 12:46 | |
| openstackgerrit | Merged openstack/project-config master: Allow delete permissions in Gerrit ACLs https://review.opendev.org/c/openstack/project-config/+/786088 | 13:02 |
| *** amoralej|lunch is now known as amoralej | 13:02 | |
| openstackgerrit | Merged openstack/project-config master: Add puppet-cinder-core and puppet-glance-core https://review.opendev.org/c/openstack/project-config/+/784888 | 13:03 |
| *** AJaeger has quit IRC | 13:04 | |
| *** snapdeal has quit IRC | 13:13 | |
| *** tkajinam has joined #opendev | 13:14 | |
| *** ysandeep|afk is now known as ysandeep | 13:25 | |
| *** artom has joined #opendev | 13:36 | |
| *** marios|call is now known as marios | 13:36 | |
| *** hemanth_n has joined #opendev | 13:47 | |
| *** hemanth_n has quit IRC | 13:52 | |
| *** arxcruz has quit IRC | 13:57 | |
| *** arxcruz has joined #opendev | 13:59 | |
| clarkb | fungi: heh reading scrollback I was all ready to merge a chnage then kep reading and was happy to see it was unnecesasry | 14:10 |
| clarkb | fungi: and ya I suspect that the actual tarballs may not be an issue, but some other artifact | 14:10 |
| clarkb | fungi: anything else to look at? based on scrollback here and in -infra no, but figured I would ask | 14:11 |
| fungi | nah, seems we're all good | 14:11 |
| fungi | the final change to mark wallaby as released will be approved in another ~15 minutes and then the press release goes out an hour after that | 14:12 |
| fungi | er, half an hour after that | 14:12 |
| fungi | release notes jobs are taking a long time because we serialize them to avoid rsync fighting over the same doctrees, but that's not a blocker | 14:13 |
| *** slaweq has quit IRC | 14:16 | |
| *** slaweq has joined #opendev | 14:20 | |
| *** ykarel_ has joined #opendev | 14:20 | |
| *** ykarel has quit IRC | 14:22 | |
| clarkb | fungi: if you have a moment can you review https://review.opendev.org/c/opendev/system-config/+/786125 as part of the firehose cleanup? | 14:26 |
| clarkb | once that lands and applies I can go and make sure the workers are restarted on new configs. | 14:26 |
| fungi | sure, i have a sec | 14:26 |
| fungi | though need to take a break for a shower soon, been going for ~4 hours but wanted to make sure release stuff completed smoothly | 14:27 |
| clarkb | thank you for being around for that | 14:27 |
| clarkb | Once I feel awake enough I'll try to do lp bug subscription investigation again | 14:28 |
| clarkb | to get the other input to firehose disabled | 14:28 |
| *** ykarel_ is now known as ykarel | 14:29 | |
| fungi | getting 782830 merged soon would also be cool so we can start experimenting with acl inheritance in openstack fairly soon, now that the release is basically done | 14:31 |
| fungi | and the open changes for topic:lp-integration before we restart gerrit at the end of the week | 14:32 |
| clarkb | ++ I'll look at it | 14:32 |
| clarkb | I was going to followup on the zuul status plugin change this morning too to see where that was at | 14:32 |
| clarkb | looks like davido is in favor and +2'd it but waiting on luca's feedback | 14:33 |
| fungi | awesome, that's great news | 14:33 |
| clarkb | if we restarted before that change lands we would probably want to land ianw's existing change on our side then manually update all-projects | 14:33 |
| fungi | there's no urgency for restarting gerrit as far as lp-integration work is concerned | 14:34 |
| fungi | it's hand-patched into the currently running container (to make sure it actually works) | 14:34 |
| clarkb | cool, and ya we don't want to restart until friday anyway so have time to sort out a solution | 14:35 |
| fungi | if we down and up the container then having those changes merged first would be good, right | 14:35 |
| clarkb | fungi: re the meta-config project the jeepyb project creation process will create that repo with a master branch and a .gitreview file. The repo will inherit from all-projects by default. I think this means it may be in a weird spot if anyone wants to merge code to master. Maybe we half manually drop a readme in there explaining the repo's purpose and force merge it and then leave it be? | 14:36 |
| clarkb | just calling that out to see if there is any concern with that extra branch laying around | 14:36 |
| clarkb | Another thing to consider is replication. The repo will be replicated but not its config branch iirc | 14:36 |
| fungi | i expected that, yeah, hopefully the project description is sufficient for people to understand there's little point in proposing changes directly for that repo | 14:37 |
| clarkb | I don't think any of these concerns are necessarily issues. Just things to be aware of as they straddle the behavior of existing config projects and existing openstack repos | 14:37 |
| clarkb | I have +2'd it and will let you +A if you decide the concerns are merely that | 14:38 |
| mtreinish | I've got a random gitea question, I was looking at one of my commits which was signed and it says my key wasn't found in the database: https://opendev.org/openstack/reno/commit/984bcba17e4e0b46763f42015d09680e5c5d5a04 | 14:38 |
| mtreinish | is there a way I can fix that? | 14:38 |
| clarkb | mtreinish: not currently as we don't expose user auth on gitea | 14:38 |
| *** zoharm has quit IRC | 14:39 | |
| clarkb | mtreinish: I believe the gitea mechanism for that is you create a user and then add your gpg pubkey | 14:39 |
| fungi | right, making that work would mean we'd need to have user accounts in gitea | 14:39 |
| mtreinish | ah, ok | 14:39 |
| mtreinish | I was just wondering if there was something I forgot to do :) | 14:39 |
| clarkb | mtreinish: gitea operates as 8 separate installations currently so doing user management is largely a non starter | 14:39 |
| clarkb | if we manage to get it into a single gitea cluster then we could start to consider the implications of adding users for things like this | 14:40 |
| openstackgerrit | Merged opendev/system-config master: Stop publishing subunit worker data to mqtt https://review.opendev.org/c/opendev/system-config/+/786125 | 14:44 |
| *** lpetrut has quit IRC | 15:00 | |
| *** mlavalle has joined #opendev | 15:17 | |
| *** d34dh0r53 has quit IRC | 15:20 | |
| *** artom has quit IRC | 15:21 | |
| *** d34dh0r53 has joined #opendev | 15:22 | |
| *** ysandeep is now known as ysandeep|away | 15:25 | |
| clarkb | the openstack release has completed. One down and one to go before we start making bigger changes | 15:33 |
| clarkb | looking at firehose I'm not sure that anything but the exim and imap services are running | 15:52 |
| clarkb | we really should be ok to clean this up in that case | 15:53 |
| clarkb | I've restarted the subunit workers for completeness though it appears that may not have been necessary | 15:55 |
| clarkb | fungi: the openstack bugs link was the ticket on the lp side too. I went to "edit bug mail" and unsubscribed from "everything openstack" | 15:59 |
| clarkb | I'm tailing the exim log now to see if there is any indication for other subscriptions | 16:01 |
| clarkb | fungi: do you think we should send annoucnement of turning off firehose? As far as I can tell the service has been effectively off already | 16:05 |
| clarkb | also slightly confused around why certcheck didn't fail against firehose | 16:11 |
| clarkb | but certcheck is accurately reporting survey right now so not too worried | 16:11 |
| *** zoharm has joined #opendev | 16:16 | |
| mtreinish | clarkb: we did announce it on the list when it was brought up: http://lists.openstack.org/pipermail/openstack-dev/2016-September/103985.html | 16:16 |
| clarkb | mtreinish: ah in that case probably worth a small note to the list now that it is going away (even if it has been away) | 16:18 |
| * clarkb starts writing one | 16:18 | |
| *** zoharm has quit IRC | 16:19 | |
| *** rpittau is now known as rpittau|afk | 16:21 | |
| clarkb | and sent | 16:26 |
| *** hamalq has joined #opendev | 16:28 | |
| clarkb | mtreinish: re haskell you should write more :) though anymore the only haskell I dabble with is my xmonad config | 16:30 |
| clarkb | I did fix a bug in it within the last year though so thats something | 16:31 |
| *** marios is now known as marios|out | 16:32 | |
| *** marios|out has quit IRC | 16:37 | |
| fungi | sorry, back now. i disappeared for a post-release shower | 16:37 |
| fungi | also minor farming drama, most of our onions have decided to bolt in their first season, probably warm wet winter to blame | 16:37 |
| fungi | but hey, fresh onions in dinner tonight! | 16:38 |
| mtreinish | clarkb: heh, I know it has a lot of fans. I might try it for a small project at some point | 16:38 |
| mtreinish | most of my new projects have been in rust, which has been a lot of fun | 16:38 |
| fungi | if my programs develop rust, i sandblast and powdercoat them | 16:40 |
| mtreinish | hah, well I've been oxidizing slow python libs. Different philosophy I guess :) | 16:42 |
| *** ralonsoh has quit IRC | 16:45 | |
| clarkb | the last email received by the host was over half an hour ago. Though it did receive an email about 2 minutes after I unsubscribed (lag in the system maybe?) | 16:48 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/786126 is probably ready for review now though as things appear stopped on the host | 16:48 |
| fungi | launchpad api not synchronous, details at 22 | 16:48 |
| *** eolivare_ has quit IRC | 16:48 | |
| fungi | s/22/11/ | 16:49 |
| *** klonn has quit IRC | 16:56 | |
| clarkb | fungi: als not sure if you want to go ahead and approve https://review.opendev.org/c/openstack/project-config/+/782830 after our earlier conversation | 16:56 |
| *** ricolin has joined #opendev | 16:57 | |
| clarkb | fungi: re onions, if you have dry weather you should be able to dry them out and keep them edible in a cool dark spot for later too | 17:01 |
| fungi | clarkb: normally yes, but if they bolt early the tops are still green and very wet and that becomes much harder | 17:01 |
| clarkb | ah | 17:01 |
| fungi | ideally you let them go brown and fall over, but they only do that in the second season | 17:02 |
| fungi | we'll try to preserve them as we can, but probably going to make a lot of onion jam, caramelized onion, chop and freeze a bunch, et cetera | 17:02 |
| *** clarkb has quit IRC | 17:08 | |
| *** clarkb has joined #opendev | 17:08 | |
| clarkb | apparently I ping timed out | 17:09 |
| clarkb | did I miss anything good? | 17:09 |
| *** jpena is now known as jpena|off | 17:10 | |
| fungi | ctcp pings | 17:10 |
| fungi | whether those are good is for you to decide though | 17:10 |
| clarkb | heh | 17:10 |
| *** avass has quit IRC | 17:16 | |
| *** amoralej is now known as amoralej|off | 17:24 | |
| *** klonn has joined #opendev | 17:30 | |
| *** artom has joined #opendev | 17:34 | |
| *** ykarel has quit IRC | 17:35 | |
| *** dtantsur is now known as dtantsur|afk | 17:44 | |
| *** andrewbonney has quit IRC | 17:59 | |
| *** zimmerry has joined #opendev | 18:00 | |
| clarkb | fungi: any chance I can get a review on https://review.opendev.org/c/opendev/system-config/+/786126 ? then maybe can land that this afternoon and delete the server | 18:10 |
| fungi | sure thing | 18:18 |
| openstackgerrit | Merged openstack/project-config master: Add an empty project for an OpenStack base ACL https://review.opendev.org/c/openstack/project-config/+/782830 | 18:21 |
| *** avass has joined #opendev | 18:26 | |
| *** whoami-rajat has quit IRC | 18:38 | |
| openstackgerrit | Merged opendev/system-config master: Remove firehose.openstack.org https://review.opendev.org/c/opendev/system-config/+/786126 | 18:50 |
| *** fressi has quit IRC | 19:00 | |
| *** sboyron has quit IRC | 19:30 | |
| *** klonn has quit IRC | 20:16 | |
| ianw | re gerrit plugin, it looks like everyone is happy with switching the default, so i think let's merge that | 20:38 |
| clarkb | ianw: did luca end up responding? /me refreshes the chagne | 20:39 |
| ianw | oh, there's a question for luca | 20:39 |
| ianw | heh, yeah, i guess let's see if any more comments | 20:39 |
| clarkb | ya I think davido wants luca to weigh in if possible | 20:40 |
| clarkb | I'm about to eat some lunch but will followup on firehose after since that change merged | 20:41 |
| clarkb | should mostly be a matter of cleaning up the server at this point | 20:41 |
| clarkb | I have also been reminded that we should all register for the ptg if we plan on attending | 20:43 |
| clarkb | I had missed doing that (I think or maybe I'm double registered now, I'm registered at least once :) ) | 20:43 |
| openstackgerrit | Merged openstack/project-config master: Add OSUOSL resources to nodepool https://review.opendev.org/c/openstack/project-config/+/786156 | 20:49 |
| *** zimmerry has quit IRC | 20:57 | |
| ianw | does anyone else have an opinion on the openEuler mirroring changes at https://review.opendev.org/c/opendev/system-config/+/784874 ? | 20:58 |
| ianw | "Currently openEuler repo site doesn't support anonymous rsync." | 20:59 |
| ianw | i mean it's not a blocker i guess, but it just feels ... odd | 20:59 |
| *** zimmerry has joined #opendev | 21:01 | |
| *** zimmerry has quit IRC | 21:02 | |
| clarkb | is that the official method for mirroring that distro? if so I guess we can live with it? but it does feel odd to do something special. In particualr I don't want to become a mirror people rely on outside of CI (whcih seems far more probably if we're mirroring it when others cant) | 21:05 |
| fungi | clarkb: also cleaning up dns records... maybe skim the list of records and remove any others we've obsoleted recently (e.g., nodepool launchers) | 21:07 |
| clarkb | 786126 hasn't gotten to the run puppet step yet, I wanted that to finish before I remove the server. But once that is done I think I should be good to delete firehose01.openstack.org | 21:07 |
| clarkb | fungi: ++ can do once the server is removed | 21:07 |
| clarkb | ddd5b4c1-37af-4973-b49c-b2023582b75f is the uuid of the firehose server if anyone else wants to double check that | 21:08 |
| ianw | fungi: i cleared out the openstack.org nlXX entries the other day after i got myself confused why the host key had changed for them :) | 21:09 |
| fungi | ianw: oh, awesome thanks! yeah i kept trying and failing to ssh into them too | 21:09 |
| fungi | but hadn't gotten around to logging in to delete them | 21:09 |
| clarkb | sorry about that (I should've done that when I replaced them) | 21:10 |
| fungi | i'm sure there's still plenty in there i've forgotten to delete too | 21:10 |
| *** zimmerry has joined #opendev | 21:11 | |
| ianw | it's easier to see things in /var/lib/rax-dns-backup than their webui | 21:12 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: rax-dns-backup : fix cron output capture https://review.opendev.org/c/opendev/system-config/+/786326 | 21:15 |
| ianw | i don't know what i was thinking there ... must have copy pasted a command line | 21:16 |
| fungi | clarkb: ddd5b4c1-37af-4973-b49c-b2023582b75f looks right to me, yep | 21:17 |
| clarkb | thanks for checking | 21:19 |
| clarkb | afs is running then puppet then I can double check empty syslogs on firehose then do cleanups | 21:25 |
| clarkb | and after these runs the runs to enable the osu arm resources should happen | 21:25 |
| clarkb | an exciting afternoon after an exciting morning | 21:25 |
| ianw | yep i'll keep an eye on those to make sure images get in place, etc. | 21:28 |
| clarkb | remote puppet else did end up failing, but there are no new syslog entries for ansible or puppet that correspond to that run. I think that means I'm clear to delete the server. Doing that now | 21:41 |
| * clarkb double checks the puppet run log first though | 21:42 | |
| ianw | yeah i think that has been mostly working but also failing for a while | 21:42 |
| clarkb | the failures were ask, logstash.o.o, and openstackid-dev | 21:43 |
| clarkb | none of that should be related to firehose so I will proceed | 21:43 |
| clarkb | #status log Deleted firehose01.openstack.org (ddd5b4c1-37af-4973-b49c-b2023582b75f) as its deployment was unmaintained and it was never used in production | 21:44 |
| openstackstatus | clarkb: finished logging | 21:44 |
| clarkb | and dns has been cleaned up too. Going to skim the list as fungi suggested and see if any other cleanups stnad out | 21:48 |
| clarkb | the only thing that jumped out is there is a backup server in the openstack domain. I think we may have moved all of those to opendev.org now? I don't want to break backups so I will leave it alone for now until we determine it is safe | 21:51 |
| *** jaicaa has quit IRC | 21:52 | |
| fungi | ianw: oh, i meant to follow up, seems like the openstack release was barely a blip for syncing the tarballs volume... i have a feeling those longer sync times we're seeing are projects uploading large container images and the like | 21:52 |
| clarkb | now to rerun the gerrit externalid conflict audit script to prepare for tomorrows cleanups | 21:52 |
| *** jaicaa has joined #opendev | 21:53 | |
| ianw | fungi: yeah, that sounds likely as you wouldn't think a few tarballs would be that big. it's a bit of a conundrum what to do ... | 21:53 |
| ianw | the best thing would be if it could replicate at more than 1.3mb/s | 21:54 |
| fungi | well, in the case of the openstack wallaby coordinated release it wasn't "a few" tarballs... more like 60-70 | 21:54 |
| fungi | plus as many wheels | 21:55 |
| fungi | plus signatures | 21:55 |
| fungi | but yeah, all of that together is probably smaller than a kolla layer | 21:55 |
| clarkb | maybe we should move sub artifacts onto a different volume? | 21:59 |
| clarkb | basically have tarballs be top layer release artifacts, then secondary things like container images go elsewhere? | 22:00 |
| clarkb | (also I'm not quite sure why we put containers on tarballs at all since docker hub and quay etc seem to work ok?) | 22:00 |
| fungi | some projects were using that as "scratch space" for emulating the pattern which eventually became our container promotion workflow | 22:05 |
| clarkb | ah | 22:06 |
| clarkb | might be a good time to push towards the intermediate registry | 22:06 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: OSU OSL : add diskimages to nb03 https://review.opendev.org/c/openstack/project-config/+/786337 | 22:08 |
| clarkb | ianw: I think that chagne is not quite right | 22:09 |
| clarkb | need to change the cloud and provider name on one of the two blocks | 22:09 |
| mordred | clarkb: I think it's definitely time to push towards the intermediate registry | 22:09 |
| ianw | clarkb: ahh yes, copy-paste issue | 22:09 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: OSU OSL : add diskimages to nb03 https://review.opendev.org/c/openstack/project-config/+/786337 | 22:10 |
| ianw | i'm not actually sure if we need those boot flags for this cloud | 22:10 |
| ianw | or indeed, at all any more; i think nova got smarter about starting arm64 nodes over time | 22:10 |
| ianw | hrw would know for sure :) | 22:10 |
| clarkb | yes I recall nova got smarter. Is the issue with sdk doing security groups that this cloud is older though? | 22:11 |
| clarkb | also I don't think it hurts to have the extra flags | 22:11 |
| ianw | yeah i didn't actually trace back what release the sdk issues would have come in | 22:13 |
| *** DSpider has quit IRC | 22:17 | |
| ianw | actually the nova changes only got added for wallaby, so "older" is maybe too strong a term :) | 22:17 |
| ianw | s/nova/neutron | 22:17 |
| clarkb | oh wow | 22:18 |
| fungi | that's been around for, like, hours man | 22:24 |
| fungi | get with the times | 22:24 |
| openstackgerrit | Merged openstack/project-config master: OSU OSL : add diskimages to nb03 https://review.opendev.org/c/openstack/project-config/+/786337 | 22:34 |
| clarkb | the audit looks good and I've got my list of accounts to cleanup external ids for tomorrow all ready to go | 22:35 |
| *** tkajinam has quit IRC | 22:43 | |
| ianw | # grep 'kazoo.client: Connection dropped: socket connection error: Connection reset by peer' nodepool-builder.log | wc -l | 22:43 |
| ianw | 117 | 22:43 |
| ianw | nb03 seems to drop out of zookeeper about every 2 minutes | 22:43 |
| clarkb | ianw: I think the zk config is ipv4 specific (we use ips not names) | 22:45 |
| corvus | where is nb03? | 22:45 |
| ianw | corvus: in linaro-us | 22:45 |
| clarkb | and we recently had ipv4 issues with the mirror there right? | 22:45 |
| clarkb | (just wondering if that could be related) | 22:45 |
| ianw | i don't think we're actively having issues, but it certainly doesn't seem to keep a reliable connection to ZK at least | 22:46 |
| corvus | potential mitigation: a dib-compatible shell script that does the work over ssh and copies results back. make nb03 a vm in rax and ssh into linaro to do the builds. | 22:48 |
| clarkb | maybe check if the ipv4 connectivity out to something else is relatively stable? | 22:48 |
| clarkb | corvus' idea is neat but would probably suffer issues too (though I guess ssh is very resilient to issues) | 22:49 |
| corvus | (i mean, obvs "make network good" is better :) | 22:49 |
| corvus | yep; put my idea near the bottom of the list | 22:49 |
| corvus | but ssh can have pretty long timeouts | 22:49 |
| clarkb | our zk timeout is 2 minutes iirc | 22:49 |
| clarkb | which isn't very short either | 22:50 |
| ianw | if we find OSU is more stable and perhaps faster, we could move the builder there | 22:50 |
| corvus | ++ | 22:50 |
| ianw | it has 3ms pings and no packet loss on ping to zk01 | 22:51 |
| clarkb | that makes me wonder if it isn't able to service the zk pings in a timely manner (its all running on one cpu and if busy may not be able to do zk and normal duties? | 22:52 |
| *** tkajinam has joined #opendev | 22:52 | |
| clarkb | (that is essentially what happens with the scheduler when it ooms and swapps | 22:53 |
| ianw | it does say "Connection reset by peer" | 22:54 |
| ianw | does that perhaps suggest ZK decided there was nothing on the other end and closed the connection? | 22:54 |
| clarkb | ianw: ya that typically indicates that zk hit its internal ping timeout to the node and disconnects it from the zk side | 22:54 |
| corvus | or could be bad firewall/nat issuing a gratuitous RST | 22:55 |
| clarkb | if you look at syslog on the zk leader (zk02 last I checked) you will see it make those decisions | 22:55 |
| ianw | openstack.exceptions.SDKException: Image creation failed: Unable to establish connection to http://arm-openstack.osuosl.org:9292/v2/images/c2e15f04-cb0a-433d-8002-2fc791f4d620/file: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer') | 22:55 |
| ianw | that is unfun | 22:55 |
| ianw | and indeed points to local issues even more | 22:56 |
| *** hamalq has quit IRC | 22:57 | |
| *** hamalq has joined #opendev | 22:57 | |
| openstackgerrit | Merged opendev/system-config master: rax-dns-backup : fix cron output capture https://review.opendev.org/c/opendev/system-config/+/786326 | 22:58 |
| ianw | they all just failed to upload again | 23:00 |
| ianw | --upload-workers 8 porbably too much for this host | 23:03 |
| ianw | probably even | 23:03 |
| clarkb | oh that could be | 23:08 |
| clarkb | maybe turn it down to 2 and see if that helps? | 23:08 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool-builder: provide for configuration of upload works, reduce nb03 https://review.opendev.org/c/opendev/system-config/+/786341 | 23:10 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool-builder: configure upload workers, reduce nb03 https://review.opendev.org/c/opendev/system-config/+/786341 | 23:11 |
| clarkb | lgtm | 23:11 |
| ianw | Image build debian-stretch-arm64-0000100981 (external_id 47944e6a-c9a3-4c58-bc74-055b90eab1ec) in osuosl is ready | 23:13 |
| ianw | some are getting through | 23:14 |
| ianw | aliasByNode(stats.timers.nodepool.task.$region.ComputePostServers.mean) ... these stats no longer seem to be there | 23:26 |
| ianw | https://opendev.org/zuul/nodepool/src/branch/master/nodepool/driver/openstack/provider.py#L92 in theory the client should be setup to send out nodepool.task results | 23:28 |
| *** gry has joined #opendev | 23:29 | |
| *** tosky has quit IRC | 23:29 | |
| ianw | 0x0010: 68ef f0a7 b535 1fbd 003e 5343 6f70 656e h....5...>SCopen | 23:38 |
| ianw | 0x0020: 7374 6163 6b2e 6170 692e 636f 6d70 7574 stack.api.comput | 23:38 |
| ianw | 0x0030: 652e 4745 542e 7365 7276 6572 732e 6465 e.GET.servers.de | 23:38 |
| ianw | 0x0040: 7461 696c 3a38 3733 2e30 3030 3030 307c tail:873.000000| | 23:38 |
| ianw | weird, it looks like they're coming out with a generic "openstack.api.compute" prefix | 23:38 |
| ianw | i think it's not impossible openstacksdk has broken setting the prefix | 23:40 |
| ianw | relocating for a bit ... bib | 23:41 |
| *** irclogbot_2 has quit IRC | 23:50 | |
| *** irclogbot_1 has joined #opendev | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!