| ianw | not related to current discussion, but do we know why infra-prod base is failing? https://zuul.opendev.org/t/openstack/builds?job_name=infra-prod-base&project=opendev/system-config | 00:05 |
|---|---|---|
| ianw | i'm looking now | 00:05 |
| ianw | hrm, mirror01.kna1.airship-citycloud.opendev.org seems involved | 00:06 |
| ianw | ECDSA host key for 188.212.109.64 has changed and you have requested strict checking. | 00:08 |
| *** hamalq has quit IRC | 00:12 | |
| *** hamalq has joined #opendev | 00:12 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Update airship mirror address https://review.opendev.org/c/opendev/system-config/+/774015 | 00:12 |
| clarkb | oh | 00:15 |
| clarkb | I think the cloud launcher is also having trouble recently. I was going to look into that as part of inmotion cloud spinup. | 00:15 |
| clarkb | I did start that this morning and ran into a few issues. Emailed them and just recently got a response back saying things should be better, so I'll see if Ic an giveit another go tomorrow | 00:16 |
| clarkb | thinking about the haproxy thing a bit more we could also try rate limiting | 00:18 |
| clarkb | haproxy does support it though you have to build it up out of some pretty basic primitives (so it doesn't look very intuitive) | 00:18 |
| clarkb | I am now learning me an haproxy rate limit setup | 00:19 |
| clarkb | most of the examples people have written up seem to be http specific, but it seems that since you build up with simple primitives a version for tcp could be devices | 00:20 |
| clarkb | *devised | 00:20 |
| *** rchurch has quit IRC | 00:23 | |
| *** mlavalle has quit IRC | 00:26 | |
| *** rchurch has joined #opendev | 00:26 | |
| ianw | is that rate-limiting connections, or throughput? | 00:26 |
| clarkb | all of the examples use connections. If there is a primitive for throughput we can probably make that work too | 00:27 |
| clarkb | I'm trying to find where haproxy documents all of these built in primitives that you use to build up the table | 00:27 |
| *** mlavalle has joined #opendev | 00:27 | |
| clarkb | http_req_rate(10s) <- is the http request rate over a 10 second period for example | 00:27 |
| clarkb | then you can compare that to some constant you define as the limit and decide to allow new requests or not. I assume there are similar things for tcp and probably throughput but haven't found the docs for that yet | 00:28 |
| clarkb | ianw: yup there is conn_cur, conn_rate, and bytes_out_rate | 00:30 |
| clarkb | fwiw I haven't decided if I think this is a good idea, but wanted to read up on it to understand what is possible | 00:30 |
| ianw | similar to the http redirect stuff, it's probably good to have tested and there behind a switch for when we really need it | 00:31 |
| ianw | at least I mean ... if the limits are reasonable and it's working, i guess we should just keep it on | 00:32 |
| clarkb | sounds like the bytes_out_rate is more typcially used for CDN billing type purposes ( you record the data in the same tables and instaed of rejecting connections based on that you bill with it? ) | 00:33 |
| clarkb | I think the risk with using something like that to rate limit is you actually want transfers to go as quickly as possible and complete | 00:33 |
| clarkb | https://www.haproxy.com/blog/introduction-to-haproxy-stick-tables/ has been the most useful doc so far fwiw | 00:34 |
| fungi | clarkb: i thought we already added logging of the forward source port in haproxy for this very reason? | 00:37 |
| clarkb | fungi: for rate limiting? | 00:38 |
| fungi | no, source port for the forwarded socket | 00:40 |
| fungi | https://review.opendev.org/738685 | 00:40 |
| fungi | it's been in place since june | 00:40 |
| fungi | sorry, catching up from having stepped away from the keyboard for a bit | 00:40 |
| fungi | oh, nevermind, you said "logging the apache source port on the connection to gitea" | 00:41 |
| clarkb | fungi: oh yup, the problem is we have a second proxy involved now | 00:41 |
| fungi | yeah i don't think we do that yet | 00:41 |
| clarkb | we are good between haproxy and apache | 00:41 |
| clarkb | not apache and gitea, though apache and gitea log roughly the same stuff so its probably not an urgent need | 00:41 |
| openstackgerrit | Merged opendev/system-config master: Update airship mirror address https://review.opendev.org/c/opendev/system-config/+/774015 | 00:46 |
| fungi | ianw: oh! thanks for spotting that, i missed it when updating dns | 00:47 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Preliminary haproxy rate limiting ideas https://review.opendev.org/c/opendev/system-config/+/774023 | 00:57 |
| clarkb | I'm going to mark that workinprogress because I'm almost certain the rules in that change are wrong | 00:58 |
| clarkb | but I think having the framework in place will help us talk about what might be right if we want to take it further | 00:58 |
| clarkb | also I think that haproxy role might be trying to be generic and I've just shoved some super site specific info in its config. I'll have to work on making it configurable | 00:59 |
| ianw | i have https://review.opendev.org/c/opendev/system-config/+/677903 which makes the haproxy stuff more generic | 01:05 |
| ianw | that was from when we were thinking of putting static behind it | 01:05 |
| openstackgerrit | Jeremy Stanley proposed opendev/bindep master: Move all jobs in-repo https://review.opendev.org/c/opendev/bindep/+/773797 | 01:43 |
| openstackgerrit | Jeremy Stanley proposed opendev/bindep master: Build docs for OpenDev https://review.opendev.org/c/opendev/bindep/+/773796 | 01:43 |
| openstackgerrit | Jeremy Stanley proposed opendev/bindep master: Remove release note about rpm path references https://review.opendev.org/c/opendev/bindep/+/774031 | 01:43 |
| openstackgerrit | Jeremy Stanley proposed opendev/bindep master: Move all jobs in-repo https://review.opendev.org/c/opendev/bindep/+/773797 | 01:52 |
| openstackgerrit | Jeremy Stanley proposed opendev/bindep master: Build docs for OpenDev https://review.opendev.org/c/opendev/bindep/+/773796 | 01:52 |
| openstackgerrit | Jeremy Stanley proposed opendev/bindep master: Remove release note about rpm path references https://review.opendev.org/c/opendev/bindep/+/774031 | 01:52 |
| *** dviroel has quit IRC | 02:13 | |
| *** DSpider has quit IRC | 02:13 | |
| *** hamalq has quit IRC | 02:45 | |
| openstackgerrit | Merged opendev/system-config master: Remove AFS puppet https://review.opendev.org/c/opendev/system-config/+/771342 | 03:52 |
| *** ysandeep|away is now known as ysandeep | 04:31 | |
| *** ysandeep is now known as ysandeep|ruck | 04:31 | |
| *** ykarel has joined #opendev | 04:49 | |
| *** auristor has quit IRC | 05:07 | |
| *** hemanth_n has joined #opendev | 05:16 | |
| *** ykarel has quit IRC | 05:17 | |
| *** ykarel has joined #opendev | 05:18 | |
| *** whoami-rajat__ has joined #opendev | 05:23 | |
| *** auristor has joined #opendev | 05:36 | |
| *** ykarel_ has joined #opendev | 05:51 | |
| *** ykarel has quit IRC | 05:53 | |
| *** ykarel_ is now known as ykarel | 06:00 | |
| *** marios has joined #opendev | 06:04 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/774040 | 06:06 |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/774040 | 06:36 |
| *** slaweq has joined #opendev | 07:15 | |
| frickler | mnaser: any update on the IPv6 issue? let me know if you need help with that. I also found https://www.space.net/~gert/RIPE/ipv6-filters.html which recommends filtering /48s, so I would strongly suggest to you to also announce and document your full /32 | 07:23 |
| *** sboyron_ has joined #opendev | 07:25 | |
| *** eolivare has joined #opendev | 07:31 | |
| *** ralonsoh has joined #opendev | 07:36 | |
| *** rpittau|afk is now known as rpittau | 07:39 | |
| *** jpena|off is now known as jpena | 07:51 | |
| *** fressi has joined #opendev | 07:59 | |
| *** jaicaa has quit IRC | 08:09 | |
| *** jaicaa has joined #opendev | 08:10 | |
| *** hashar has joined #opendev | 08:18 | |
| ykarel | is there some known issue with limestome mirrors? i see http://mirror.regionone.limestone.opendev.org/centos/7/os/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to mirror.regionone.limestone.opendev.org:80; No route to host" in jobs | 08:21 |
| *** cgoncalves has quit IRC | 08:31 | |
| *** cgoncalves has joined #opendev | 08:33 | |
| *** haleyb has quit IRC | 08:34 | |
| *** tosky has joined #opendev | 08:46 | |
| *** andrewbonney has joined #opendev | 08:50 | |
| *** fbo|off is now known as fbo | 08:55 | |
| *** valery_t has joined #opendev | 09:11 | |
| *** ykarel is now known as ykarel|lunch | 09:15 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/project-config master: Add ansible-role-pki to zuul https://review.opendev.org/c/openstack/project-config/+/773387 | 09:17 |
| *** DSpider has joined #opendev | 09:28 | |
| *** icey has quit IRC | 09:46 | |
| *** icey has joined #opendev | 09:50 | |
| *** eolivare_ has joined #opendev | 10:14 | |
| *** ykarel|lunch is now known as ykarel | 10:16 | |
| *** eolivare has quit IRC | 10:17 | |
| *** dviroel has joined #opendev | 10:47 | |
| *** hashar is now known as hasharLunch | 10:50 | |
| *** dtantsur|afk is now known as dtantsur | 11:01 | |
| ysandeep|ruck | ykarel, i also noticed issue with limestone mirrors for some jobs , I have also opened a bug for tracking https://bugs.launchpad.net/tripleo/+bug/1914585 | 11:30 |
| openstack | Launchpad bug 1914585 in tripleo "Content provider jobs failed after failing to connect to mirrors, Error: Failed to download metadata for repo 'delorean-current': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried" [High,Triaged] | 11:30 |
| ykarel | okk let's see if some one from infra can look into it | 11:31 |
| *** auristor has quit IRC | 11:33 | |
| *** valery_t has quit IRC | 11:35 | |
| *** auristor has joined #opendev | 11:35 | |
| *** arxcruz|ruck has joined #opendev | 11:43 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:53 | |
| *** hasharLunch has quit IRC | 11:58 | |
| *** auristor has quit IRC | 12:05 | |
| *** auristor has joined #opendev | 12:06 | |
| *** hemanth_n has quit IRC | 12:22 | |
| *** jpena is now known as jpena|lunch | 12:30 | |
| *** hasharLunch has joined #opendev | 12:46 | |
| *** mtreinish has joined #opendev | 13:06 | |
| *** dtantsur is now known as dtantsur|brb | 13:14 | |
| *** hasharLunch is now known as hashar | 13:19 | |
| *** jpena|lunch is now known as jpena | 13:33 | |
| *** roman_g has joined #opendev | 13:48 | |
| roman_g | Hello team. May I ask you for the OpenStack project name which is used to launch CI instances by Zuul on KNA1 (CityCloud)? There is one called 'Airship_Opendev_Mirror', but I suspect that this one is used for the package mirrors (permanent) VM instance. Thank you! | 13:51 |
| *** JohnnyRainbow has quit IRC | 13:59 | |
| *** DSpider has quit IRC | 14:05 | |
| *** JohnnyRainbow has joined #opendev | 14:05 | |
| *** DSpider has joined #opendev | 14:05 | |
| *** dtantsur|brb is now known as dtantsur | 14:16 | |
| mordred | roman_g: https://opendev.org/opendev/system-config/src/branch/master/playbooks/templates/clouds/nodepool_clouds.yaml.j2#L162 | 14:31 |
| roman_g | mordred thank you! (I've searched in a wrong place) | 14:32 |
| mordred | roman_g: there are many places one can search :) | 14:32 |
| roman_g | I've expected it to be under openstack configs, not opendev. What's the best way to search code in all opendev.org/* projects? | 14:33 |
| roman_g | grep -Rni whatever . | 14:34 |
| jrosser | https://codesearch.opendev.org/?q=kna1 | 14:34 |
| roman_g | jrosser bookmarked. Thank you! | 14:35 |
| *** fressi has quit IRC | 15:09 | |
| fungi | ykarel: ysandeep|ruck: i wonder if there are networking problems in there. i can get to that url just fine, but sounds like maybe some nodes we boot in there can't? | 15:17 |
| *** hashar has quit IRC | 15:19 | |
| fungi | roman_g: are you having trouble with the nodes in kna1 again? we had to stop using that provider over the weekend because some network incident there caused the mirror server's network to no longer have any connectivity at all. we finally got it added back on tuesday after confirming things were reachable again | 15:20 |
| roman_g | fungi , we have noticed that. Thank you for your work. | 15:20 |
| roman_g | We found issues with mirror VM, then they disappeared (thanks to you). | 15:21 |
| fungi | okay, just making sure you were aware it was off for a few days | 15:21 |
| roman_g | And then we got a number of NODE_FAILURE's | 15:21 |
| roman_g | Now I'm working with CityCloud to check on quotes and their nova failures to schedule new instances. | 15:22 |
| fungi | unfortunately our only listed contact for that environment is someone at ericsson (i guess that's who pays the bill?) but noonedeadpunk was able to get the right folks in citycloud looking into it for us | 15:22 |
| roman_g | I'm in contact with CityCloud. We are fine here. | 15:22 |
| roman_g | However I would appreciate to know financial contact name from Ericsson (because quota affects the bill). | 15:23 |
| roman_g | To private msgs, please. | 15:23 |
| fungi | roman_g: sure, just a sec and i'll look it up | 15:25 |
| *** sshnaidm is now known as sshnaidm|afk | 15:48 | |
| openstackgerrit | Merged opendev/system-config master: Remove karbor channel from bot list https://review.opendev.org/c/opendev/system-config/+/767239 | 15:53 |
| ysandeep|ruck | roman_g, fungi, thanks! is it okay to recheck the jobs now, if issue with mirror cleared? | 15:55 |
| roman_g | ysandeep|ruck sure. | 15:56 |
| ysandeep|ruck | thanks! | 15:56 |
| fungi | ysandeep|ruck: roman_g: i think you're talking about different providers | 15:56 |
| fungi | ysandeep|ruck was seeing intermittent issues with nodes in limestone trying to reach the mirror in that environment | 15:57 |
| roman_g | I'm about kna1, CityCloud. | 15:57 |
| fungi | yeah, so unrelated | 15:57 |
| roman_g | Ahha. another one. OK. | 15:57 |
| fungi | ysandeep|ruck probably we need to check logstash to see what the hit rate for those errors is and whether we can put together more detail for logan- (who takes care of that environment) | 15:58 |
| fungi | ysandeep|ruck: for example, if we can correlate failures to a specific host-id, then that might help him pinpoint where in the environment a network problem has arisen | 15:59 |
| fungi | ysandeep|ruck: also if they're particularly frequent, we can temporarily stop booting new nodes there while we investigate further | 16:00 |
| ysandeep|ruck | fungi, i am seeing green run since ~9:00 gmt , faced majority of issue around ~07:30 | 16:09 |
| ysandeep|ruck | http://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?var-influxdb_filter=cloud%7C%3D%7Climestone&var-influxdb_filter=job_name%7C%3D%7Ctripleo-ci-centos-8-content-provider&from=now-24h&to=now | 16:09 |
| *** mlavalle has quit IRC | 16:09 | |
| clarkb | as a heads up I'm going to try and continue reviews (I started trying to catch up on them yesterday before gitea had a sad), then I awnt to look at some of those gerrit accounts and see if I can run another group through the "retire these accounts without preferred email settings" step | 16:09 |
| clarkb | I think there may be 10-15 out of the remaining 28 in that situation that we can safely turn off without much fuss | 16:10 |
| ysandeep|ruck | fungi, fyi.. I have filtered on one a of job for which i noticed this error, other jobs could be impacted as well | 16:10 |
| clarkb | ysandeep|ruck: fungi: you can use logstash to look at all jobs and filter by cloud and error message | 16:11 |
| *** artom has quit IRC | 16:12 | |
| fungi | yeah, i've been in a meeting tunnel all morning, which continues for the next few hours still | 16:13 |
| fungi | so trying to poke at stuff as i can | 16:13 |
| clarkb | node to mirror traffic in limestone should happen over ipv6 (because we have a AAAA record in dns for the mirror and linux will prefer ipv6 over ipv4 if available) | 16:14 |
| clarkb | I wonder if there is some internal ipv6 routing issue? | 16:14 |
| *** ysandeep|ruck is now known as ysandeep|away | 16:15 | |
| *** mlavalle has joined #opendev | 16:15 | |
| *** ykarel has quit IRC | 16:33 | |
| *** ykarel has joined #opendev | 16:33 | |
| fungi | or if it was just a brief network incident there | 16:35 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Preliminary haproxy rate limiting ideas https://review.opendev.org/c/opendev/system-config/+/774023 | 16:38 |
| clarkb | cool my workinprogress was preserved on that | 16:39 |
| clarkb | also yay testing saying I got haproxy configs wrong :) | 16:45 |
| *** jpena is now known as jpena|off | 17:01 | |
| *** marios is now known as marios|out | 17:15 | |
| openstackgerrit | Jeremy Stanley proposed opendev/bindep master: Overhaul Python package metadata https://review.opendev.org/c/opendev/bindep/+/774106 | 17:15 |
| openstackgerrit | Jeremy Stanley proposed opendev/bindep master: Update contributor doc and readme https://review.opendev.org/c/opendev/bindep/+/774107 | 17:15 |
| *** openstack has joined #opendev | 17:23 | |
| *** ChanServ sets mode: +o openstack | 17:23 | |
| *** marios|out has quit IRC | 17:28 | |
| *** ykarel has quit IRC | 17:28 | |
| *** bodgix has quit IRC | 17:35 | |
| *** bodgix has joined #opendev | 17:36 | |
| *** artom has joined #opendev | 17:43 | |
| fungi | weird, seeing some very large outbound network traffic spikes for review.o.o and they don't correspond with our backup cronjobs either: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=45&rra_id=all | 17:50 |
| clarkb | fungi: looking at the longer historical trends that is in line with what we had seen before | 17:56 |
| clarkb | fungi: if I had to guess git protocol v2 helped a lot with bw usage | 17:56 |
| clarkb | so maybe some v1 client was doing updates for ci or something? | 17:56 |
| fungi | could be | 17:57 |
| clarkb | I assume that https://review.opendev.org/c/openstack/diskimage-builder/+/716788/ has been applied to all our debuntu hosts by now which would allow us to land https://review.opendev.org/c/openstack/project-config/+/716789 however we did switch to preferring https with our mirrors so I'll do some checking on that before approving 716789 | 17:58 |
| *** snbuback_ has joined #opendev | 17:59 | |
| clarkb | I think the thing to check there is grepping the image build logs to see that the dib side is trying to add it in and our element noops? | 18:00 |
| clarkb | https://nb02.opendev.org/ubuntu-xenial-0000188857.log that log lgtm. I can see the baseline-tools script pull it in then later when our element runs package installs it says it is laready at the latest. Now to double check bionic and focal, but I think we can approve that change | 18:03 |
| *** snbuback_ has quit IRC | 18:04 | |
| *** snbuback_ has joined #opendev | 18:05 | |
| *** snbuback_ is now known as snbubac | 18:06 | |
| clarkb | fungi: xenial, bionic, focal, buster, and stretch all look good for apt-transport-https according to their image build logs. That should be everywhere we need that package right? | 18:07 |
| *** hashar has joined #opendev | 18:07 | |
| *** snbubac is now known as snbuback | 18:08 | |
| *** sboyron_ has quit IRC | 18:08 | |
| *** snbuback is now known as Guest17897 | 18:09 | |
| *** rpittau is now known as rpittau|afk | 18:10 | |
| * clarkb thinks that is it and goes for it. | 18:10 | |
| *** Guest17897 has quit IRC | 18:11 | |
| *** snbuback has joined #opendev | 18:11 | |
| *** dtantsur is now known as dtantsur|afk | 18:12 | |
| *** hamalq has joined #opendev | 18:14 | |
| *** ralonsoh has quit IRC | 18:18 | |
| openstackgerrit | Merged openstack/project-config master: zuul-worker: remove additional install of apt-transport-https https://review.opendev.org/c/openstack/project-config/+/716789 | 18:20 |
| fungi | yeah, sorry, got sidetracked by lunch but can look in a bit | 18:20 |
| clarkb | fungi: I managed to convince myself we were good | 18:22 |
| clarkb | the change removed package listings for only debian and ubuntu which was the last thing I needed to check to confirm I had surveyed properly | 18:22 |
| *** eolivare_ has quit IRC | 18:26 | |
| clarkb | mordred: is https://review.opendev.org/c/openstack/project-config/+/711474 still a thing now with collections and stuff? I'm happy to rebase that if so, but not sure if it even makes sense anymore | 18:30 |
| *** roman_g has quit IRC | 18:52 | |
| *** sboyron has joined #opendev | 18:59 | |
| fungi | zbr: i've got a stack of small cleanup/update changes to complete the move of bindep to the opendev tenant in zuul and modernize package metadata, correct urls, et cetera in preparation for tagging 2.9... https://review.opendev.org/q/topic:bindep-2.9 | 19:00 |
| fungi | if those look good to you i'll approve them and tag the tip of master as 2.9.0 | 19:01 |
| *** sboyron has quit IRC | 19:05 | |
| *** d34dh0r53 has quit IRC | 19:09 | |
| *** d34dh0r53 has joined #opendev | 19:12 | |
| *** d34dh0r53 has quit IRC | 19:13 | |
| *** d34dh0r53 has joined #opendev | 19:14 | |
| *** andrewbonney has quit IRC | 19:20 | |
| *** artom has quit IRC | 19:30 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 19:45 |
| *** artom has joined #opendev | 19:57 | |
| *** akrpan-pure has joined #opendev | 20:00 | |
| *** artom has quit IRC | 20:02 | |
| *** stand has quit IRC | 20:12 | |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 21:02 |
| *** hashar has quit IRC | 21:07 | |
| *** whoami-rajat__ has quit IRC | 21:10 | |
| ianw | 350 home/gerrit2/review_site/git/openstack/cinder.git/objects/pack/pack-6580ae06ed541e3a180c5600d01d79f4ed40f07a.pack | 21:20 |
| ianw | 379 home/gerrit2/review_site/git/openstack/neutron.git/objects/pack/pack-b5ac46fe2863a138711c6ae695491f257d090811.pack | 21:20 |
| ianw | 701 home/gerrit2/review_site/git/openstack/nova.git/objects/pack/pack-8675125f931d721e1aff139d2134a83cb081db91.pack | 21:20 |
| ianw | in case you're wondering where the space goes in incremental backups of review ... | 21:21 |
| ianw | i guess .pack files have the same problem as us backing up the db .gzip files ... we didn't have 700mb of changes in nova but just pushing a few changes will completely update the .pack | 21:21 |
| clarkb | ya | 21:31 |
| clarkb | though I think it is our repacking which does it | 21:32 |
| clarkb | (which is necessary for other efficiencies) | 21:32 |
| ianw | yep; i guess the flip side is that these objects are all unique, so will be pruned | 21:33 |
| ianw | i assume openstackwatch is dead? it's cron job is still firing (and writing out a log file -- i noticed from going through the non-zero delta file list in backup) | 21:37 |
| clarkb | thats a name I haven't heard in a while. I wonder if anyone was using it until it stopped working (thats the thing that makes rss/atomic feeds?0 | 21:39 |
| clarkb | my strong hunch is we can turn it off | 21:39 |
| ianw | it's just in a failing loop; it won't be deployed on the new server due to it only being setup by puppet anyway | 22:00 |
| openstackgerrit | Maksim Malchuk proposed openstack/diskimage-builder master: Fix hooks order for CentOS/Fedora when mirror used https://review.opendev.org/c/openstack/diskimage-builder/+/772350 | 22:00 |
| *** cap has quit IRC | 22:00 | |
| ianw | ok, i've removed the cronjob for it | 22:08 |
| clarkb | thanks | 22:09 |
| clarkb | I've got an inmotion cloud spinning up now. Their dashboard says it should take an hour or so | 22:12 |
| fungi | ianw: yes, we ripped out openstackwatch ages ago but apparently never cleaned up the cronjob for it on the server | 22:16 |
| fungi | thanks! | 22:17 |
| *** lbragstad_ has joined #opendev | 22:22 | |
| *** lbragstad has quit IRC | 22:24 | |
| clarkb | I'm up to 10 more accounts that I think we can just retire to fix some of these consistency problems | 22:28 |
| clarkb | but trying to find as many as possible before batching them up and updating them | 22:28 |
| *** tkajinam has quit IRC | 22:59 | |
| *** tkajinam has joined #opendev | 22:59 | |
| clarkb | anyone know what gerrit's behavior is if you remove the preferred email address for an account that otherwise has external ids for openid, email, and ssh username? | 23:00 |
| clarkb | there is at least one account where they don't seem to have duplicates they just seem to have orphaned their preferred account and i'm wondering if unsetting preferred is reasonable | 23:00 |
| fungi | at a minimum, they will cease getting e-mail notifications from gerrit | 23:01 |
| fungi | but beyond that, i have no idea if newer gerrit considers that a consistency problem | 23:01 |
| clarkb | ok I may just file that one away for now. Looking at the emails invovled they seem to have a personal one and an employer one. The employer one is orphaned so I suspect they removed it at some point but it remained preferred? | 23:02 |
| clarkb | in that case not spamming the employer email may actually be a good thing | 23:02 |
| clarkb | though maybe it is already not sending email since no external id exists for it? | 23:03 |
| *** slaweq has quit IRC | 23:03 | |
| fungi | or do you mean what happens if you delete your preferred address in the preferences ui? i don't think it will allow that, requires you to pick a secondary as preferred first, and then you can delete the address when it's a secondary one | 23:03 |
| *** lbragstad_ is now known as lbragstad | 23:03 | |
| clarkb | fungi: no I mean directly in All-Users | 23:03 |
| clarkb | basiclly the preferred email has no external ids. The emails in enxternal ids match no other accounts. The preferred email matches no other accounts | 23:03 |
| fungi | yeah, fundamentally it means no e-mail notifications can be sent, but beyond that i don't know whether gerrit will care | 23:04 |
| clarkb | I'm wondering if we can just unset their preferred email and then they can fix it later if they wish | 23:04 |
| fungi | i think they also can't be selected to add to groups or add as requested reviewers when there's no e-mail address | 23:04 |
| fungi | similar to when the address is in conflict with another account | 23:04 |
| clarkb | well there aer other emails | 23:04 |
| clarkb | (that is part of what makes this confusing and also potentailly inconsistentable, preferred email and external ids are stored in different refs so you can't atomically update them) | 23:05 |
| fungi | just thinking about the impact we had when not setting a preferred e-mail for our admin accounts, you couldn't update the accounts through the cli, gerrit would error | 23:07 |
| clarkb | ya might have to set a value via git directly? | 23:10 |
| clarkb | oh right any other settings wouldn't be settable | 23:10 |
| clarkb | so if we remove ap referred email we should also deactivate the account | 23:10 |
| fungi | potentially | 23:12 |
| *** lbragstad has quit IRC | 23:21 | |
| *** lbragstad has joined #opendev | 23:23 | |
| clarkb | ok review-test:~clarkb/gerrit-consistency-notes/further-preferred-email-cleanups has more notes on things I think we can just retire. We're down to just a small number of accounts that we may actually need to fix assuming others are comfortable with my analysis and choices in that doc | 23:51 |
| clarkb | my inmotion cloud deployment succeeded. Looks like the way you bootstrap things is when you build the cloud you give them an ssh public key. Once it is built that key can ssh in and get some info which allows you to horizon (and presumably keystone directly) | 23:59 |
| clarkb | I'm going to check basic functionality of things but then likely need to pick up actual resource provisioning tomorrow (in this case users in keystone I think and then we can use cloud launcher for everything else?) | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!