| corvus | clarkb: oh, you know what, it may be okay | 00:01 |
|---|---|---|
| corvus | i was worried about leaked instances, but as long as they both have access to the same zk data, should be fine. | 00:01 |
| openstackgerrit | Merged openstack/diskimage-builder master: Don't use hardcode while override base image file https://review.opendev.org/c/openstack/diskimage-builder/+/771978 | 00:03 |
| clarkb | corvus: oh interesting | 00:05 |
| clarkb | corvus: how does that work for rejecting node requests? | 00:05 |
| clarkb | isn't that done by provider? so if wev'e got multiple providers in different launchers taht collide would we maybe break there? | 00:05 |
| clarkb | ze12 is down to a single job. I expect it will finish up first | 00:07 |
| openstackgerrit | Merged opendev/system-config master: refstack: trigger image upload https://review.opendev.org/c/opendev/system-config/+/779289 | 00:11 |
| openstackgerrit | Merged opendev/system-config master: Fix sshfp record printing https://review.opendev.org/c/opendev/system-config/+/779044 | 00:11 |
| clarkb | ze10 just restarted a few seconds ago | 00:14 |
| fungi | i guess graceful means graceful restart not graceful stop | 00:14 |
| clarkb | fungi: yes, that was a known problem ebfore when we had docker-compose config that said restart: always, but we chagned it to restart: on-failure | 00:15 |
| clarkb | however, maybe it has to be started after the restart: on-failure config change was made to take effect? | 00:15 |
| clarkb | docker ps -a only shows the current container, I think docker compose is good at cleaing up after itself | 00:15 |
| clarkb | ya I wonder if that is it, the config at startup is what matters | 00:20 |
| fungi | ahh, interesting | 00:22 |
| *** lbragstad has quit IRC | 00:23 | |
| clarkb | ze10.openstack.org has been deleted now. Still waiting on the others | 00:23 |
| *** tosky has quit IRC | 00:28 | |
| clarkb | ze11 has been deleted now (it just restarted zuul and failed to reconnect) | 00:29 |
| clarkb | just two more to go then this set of rotations is done | 00:29 |
| clarkb | 09 is gone now too. We wait on ze12 | 00:36 |
| clarkb | https://zuul.opendev.org/t/openstack/stream/fe6fd4182ff7467a8326b142f08b43be?logfile=console.log is the last job we are waiting on. I wonder if the lack of output in the last almost 2 hours indicates it is stuck | 00:42 |
| *** lbragstad has joined #opendev | 00:52 | |
| fungi | not paused? | 00:53 |
| clarkb | correct, it is executing normally if I parse the tail of the executor log properly | 00:53 |
| clarkb | with a paused job the last thing the executor log spits out is "I'm pausing this job" but that isn't present in this case. I think it is a buggy job that is on its way to timing out | 00:54 |
| clarkb | I'll wait a few more minutes but then I need to go help with dinner | 00:55 |
| clarkb | can clean it up in the morning. If it restarts it will just try to reconnect to zk and fail and doesn't cause problems | 00:55 |
| clarkb | yup its still going. Not a big deal I'll finish this up in the morning. Thank you for the help fungi | 01:00 |
| fungi | any time! | 01:03 |
| *** mlavalle has quit IRC | 01:21 | |
| *** hamalq has quit IRC | 01:24 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add aarch64 support for rhel https://review.opendev.org/c/openstack/diskimage-builder/+/770703 | 01:52 |
| openstackgerrit | Merged openstack/diskimage-builder master: Use the same bootloader pkg-map for all redhat family https://review.opendev.org/c/openstack/diskimage-builder/+/779105 | 02:00 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add efibootmgr utility for UEFI boot menu management https://review.opendev.org/c/openstack/diskimage-builder/+/779106 | 02:11 |
| openstackgerrit | Merged opendev/system-config master: install-ansible: ensure stevedore https://review.opendev.org/c/opendev/system-config/+/778354 | 02:52 |
| openstackgerrit | Steve Baker proposed openstack/diskimage-builder master: Don't run grub2-install for efi block devices https://review.opendev.org/c/openstack/diskimage-builder/+/779107 | 03:07 |
| *** lbragstad has quit IRC | 03:43 | |
| *** dpawlik6 has joined #opendev | 03:51 | |
| *** dhellmann_ has joined #opendev | 03:54 | |
| *** raukadah has joined #opendev | 03:54 | |
| *** avass_ has joined #opendev | 03:55 | |
| *** icey_ has joined #opendev | 03:56 | |
| *** owalsh_ has joined #opendev | 03:58 | |
| *** openstackgerrit has quit IRC | 03:59 | |
| *** icey has quit IRC | 03:59 | |
| *** dhellmann has quit IRC | 03:59 | |
| *** dmellado has quit IRC | 03:59 | |
| *** avass has quit IRC | 03:59 | |
| *** chandankumar has quit IRC | 03:59 | |
| *** tbarron has quit IRC | 03:59 | |
| *** mhu has quit IRC | 03:59 | |
| *** cgoncalves has quit IRC | 03:59 | |
| *** dpawlik has quit IRC | 03:59 | |
| *** owalsh has quit IRC | 03:59 | |
| *** fbo has quit IRC | 03:59 | |
| *** dhellmann_ is now known as dhellmann | 03:59 | |
| *** dpawlik6 is now known as dpawlik | 03:59 | |
| *** fbo has joined #opendev | 04:02 | |
| *** dmellado has joined #opendev | 04:02 | |
| *** ykarel has joined #opendev | 04:23 | |
| *** whoami-rajat_ has joined #opendev | 04:41 | |
| *** whoami-rajat_ is now known as whoami-rajat | 04:46 | |
| *** raukadah is now known as chandankumar | 04:51 | |
| *** ysandeep|away is now known as ysandeep | 05:16 | |
| *** iurygregory has quit IRC | 05:26 | |
| *** marios has joined #opendev | 06:07 | |
| *** slaweq has joined #opendev | 06:50 | |
| *** cgoncalves has joined #opendev | 07:09 | |
| *** hashar has joined #opendev | 07:09 | |
| dpawlik | clarkb: hehe, no problem :) | 07:12 |
| *** eolivare has joined #opendev | 07:39 | |
| *** fressi has joined #opendev | 07:47 | |
| *** DSpider has joined #opendev | 08:04 | |
| *** lpetrut has joined #opendev | 08:04 | |
| *** DSpider has quit IRC | 08:05 | |
| *** hashar has quit IRC | 08:19 | |
| *** andrewbonney has joined #opendev | 08:19 | |
| *** rpittau|afk is now known as rpittau | 08:24 | |
| *** toomer has joined #opendev | 08:45 | |
| *** jpena|off is now known as jpena | 08:55 | |
| *** tosky has joined #opendev | 09:02 | |
| *** hashar has joined #opendev | 09:25 | |
| *** klonn has joined #opendev | 09:30 | |
| *** iurygregory_ has joined #opendev | 10:45 | |
| *** iurygregory_ is now known as iurygregory | 10:46 | |
| *** marios has quit IRC | 10:56 | |
| *** icey_ is now known as icey | 11:02 | |
| *** marios has joined #opendev | 11:03 | |
| *** hashar has quit IRC | 11:08 | |
| *** ysandeep is now known as ysandeep|afk | 11:11 | |
| *** klonn has quit IRC | 11:13 | |
| *** avass_ is now known as avass | 11:17 | |
| *** dtantsur|afk is now known as dtantsur | 11:25 | |
| *** marios has quit IRC | 11:31 | |
| *** marios has joined #opendev | 11:37 | |
| *** artom has quit IRC | 11:51 | |
| *** slaweq_ has joined #opendev | 11:52 | |
| *** brinzhang has quit IRC | 11:56 | |
| *** lpetrut has quit IRC | 11:56 | |
| *** lpetrut has joined #opendev | 11:58 | |
| *** slaweq has quit IRC | 11:59 | |
| *** ysandeep|afk is now known as ysandeep | 12:01 | |
| *** ykarel_ has joined #opendev | 12:17 | |
| *** ykarel has quit IRC | 12:20 | |
| *** ykarel_ is now known as ykarel | 12:21 | |
| *** artom has joined #opendev | 12:25 | |
| *** klonn has joined #opendev | 12:29 | |
| *** jpena is now known as jpena|lunch | 12:34 | |
| *** slaweq_ is now known as slaweq | 12:38 | |
| *** hashar has joined #opendev | 13:04 | |
| *** lbragstad has joined #opendev | 13:24 | |
| *** jpena|lunch is now known as jpena | 13:36 | |
| *** tkajinam has quit IRC | 13:53 | |
| *** artom has quit IRC | 14:49 | |
| *** ysandeep is now known as ysandeep|afk | 14:50 | |
| *** klonn has quit IRC | 15:13 | |
| clarkb | #status log Replaced ze09-12.openstack.org with ze09-12.opendev.org focal servers. This concludes the rolling replacement of the zuul executors. | 15:13 |
| openstackstatus | clarkb: finished logging | 15:13 |
| *** fressi has quit IRC | 15:28 | |
| *** hashar has quit IRC | 15:28 | |
| *** lpetrut has quit IRC | 15:29 | |
| *** ysandeep|afk is now known as ysandeep|away | 15:34 | |
| *** ykarel has quit IRC | 15:43 | |
| *** stephenfin has quit IRC | 15:53 | |
| *** stephenfin has joined #opendev | 15:54 | |
| *** hashar has joined #opendev | 15:57 | |
| toomer | What is the best way to change username in Gerrit after migration to NoteDb ? | 16:10 |
| toomer | Should I even think about changing username for the account in Gerrit ? | 16:10 |
| clarkb | toomer: you cannot change gerrit usernames | 16:12 |
| clarkb | (this was true pre notedb too) | 16:12 |
| toomer | SQL update helped from what I remember | 16:12 |
| clarkb | yes you could update the db but it had side effects that caused gerrit to strongly recommend against it iirc | 16:14 |
| clarkb | we never really offered username changes to people as a result | 16:14 |
| clarkb | (its a value you set in your .gitconfig then never have to think about again) | 16:14 |
| *** klonn has joined #opendev | 16:20 | |
| clarkb | toomer: https://groups.google.com/g/repo-discuss/c/y3Kq0YiKC7Q/m/1HT6w6QA7tQJ | 16:24 |
| clarkb | thats the really old thread explaining why it is a bad idea | 16:25 |
| clarkb | (we don't take advantage of that feature, but they have still made it very difficult to change anyway) | 16:25 |
| toomer | Thanks, I will read that. I was only doing that for new account and not the one that had patches published. | 16:35 |
| *** marios is now known as marios|out | 16:56 | |
| *** rpittau is now known as rpittau|afk | 17:05 | |
| *** hashar has quit IRC | 17:07 | |
| *** lpetrut has joined #opendev | 17:18 | |
| *** eolivare has quit IRC | 17:22 | |
| *** openstackgerrit has joined #opendev | 17:39 | |
| openstackgerrit | Elod Illes proposed opendev/system-config master: Add stable branch delete rights to release managers https://review.opendev.org/c/opendev/system-config/+/779546 | 17:39 |
| *** lpetrut has quit IRC | 18:00 | |
| *** artom has joined #opendev | 18:01 | |
| *** jpena is now known as jpena|off | 18:02 | |
| *** dtantsur is now known as dtantsur|afk | 18:13 | |
| *** klonn has quit IRC | 18:18 | |
| *** marios|out has quit IRC | 18:26 | |
| *** ralonsoh has quit IRC | 18:28 | |
| *** toomer has quit IRC | 18:29 | |
| *** hamalq has joined #opendev | 18:30 | |
| *** SWAT has quit IRC | 18:45 | |
| *** SWAT has joined #opendev | 18:53 | |
| *** hashar has joined #opendev | 18:53 | |
| *** andrewbonney has quit IRC | 19:09 | |
| *** hamalq has quit IRC | 19:09 | |
| *** hamalq has joined #opendev | 19:09 | |
| *** lbragstad has quit IRC | 20:01 | |
| clarkb | infra-root I've put the proposed external id cleanups input file at review:~clarkb/gerrit_user_cleanups/external_id_cleanups.txt.20210309.proposed if you can look that over and compare it to the classifications files in that dir that would be helpful | 20:03 |
| clarkb | I can run that today if we're happy with that list or sometime tomorrow, just want to keep this moving along | 20:04 |
| clarkb | and now lunch | 20:04 |
| *** lbragstad has joined #opendev | 20:15 | |
| *** irclogbot_2 has quit IRC | 20:31 | |
| *** irclogbot_0 has joined #opendev | 20:32 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Use upstream jitsi-meet web image https://review.opendev.org/c/opendev/system-config/+/778308 | 20:36 |
| clarkb | corvus: ^ that changes the parent job of the system-config-run-meetpad job to drop the containers which seems to assume that it will hae built images in intermediate registry but since we pull upstream that isn't the case. | 20:37 |
| *** sboyron has quit IRC | 20:38 | |
| *** whoami-rajat has quit IRC | 20:42 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update nodepool launchers to focal in testing https://review.opendev.org/c/opendev/system-config/+/776981 | 20:45 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update zookeeper servers in testing to focal https://review.opendev.org/c/opendev/system-config/+/776982 | 20:45 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update zuul01 to focal in testing https://review.opendev.org/c/opendev/system-config/+/776983 | 20:45 |
| clarkb | infra-root in addition to reviewing the contents of that list of cleanups do you think you can reivew https://review.opendev.org/c/opendev/system-config/+/777846 and possibly land it? That way we get the current set of tooling in the tree which will simplify when I switch branches around for other work :) | 20:47 |
| *** klonn has joined #opendev | 20:56 | |
| *** klonn has quit IRC | 20:57 | |
| clarkb | #status log Cleaned up old DNS records for old openstack.org zuul mergers and executors | 21:00 |
| openstackstatus | clarkb: finished logging | 21:00 |
| fungi | nice to see the records in that zone shrinking as we go | 21:00 |
| clarkb | agreed | 21:01 |
| clarkb | self.launcher_id = "%s-%s-%s" % (socket.gethostname(), os.getpid(), self.name) <- I think that will make my plan for easy mode launcher replacement tricky | 21:22 |
| clarkb | maybe we can manipulate the docker setup to select a high pid? | 21:22 |
| clarkb | and then that would work I think | 21:22 |
| clarkb | could do nl05-08, or maybe update nodepool to use the full fqdn if that is a safe transition | 21:25 |
| clarkb | reading the code I think it may be ok to have two launchers with the same provider name running at the same time as long as the launcher_id above is unique. They will fight over deleting leaked instances but they should only delete nodes unknown to nodepool | 21:32 |
| clarkb | (that is nodepool as a whole as it looks up the node id in zk first) | 21:32 |
| fungi | ahh, yeah i guess os.getpid() inside containers will be consistent | 21:39 |
| fungi | though hard to confirm since our images don't include procps | 21:41 |
| fungi | oh, right, i can `ls /proc` | 21:41 |
| clarkb | you can ps within the container I think? | 21:42 |
| clarkb | but the nodepool logs seem to confirm it looking at the values. It isn't 100% overlap but its close enough to be dangerous :) | 21:42 |
| fungi | yeah, pid 7 seems like | 21:43 |
| fungi | at least on both nl02 and nl02 | 21:44 |
| fungi | er, 01 and 02 | 21:44 |
| clarkb | and 6 on 03 I think | 21:44 |
| fungi | yep, confirmed with `cat /proc/6/cmdline` there | 21:47 |
| clarkb | discussing potential changes to nodepool in #zuul. I think I've convinced myself a change would be safe, just need to figure out what the most appropriate change would be | 21:47 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 21:57 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 22:18 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI https://review.opendev.org/c/opendev/system-config/+/776292 | 22:22 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Zuul Cache role with s3 implementation. https://review.opendev.org/c/zuul/zuul-jobs/+/764808 | 22:24 |
| corvus | clarkb: there may be some grafana dashboards that need updating with hostname changes (there might be some openstack.org names in the metrics) | 22:49 |
| corvus | (i'm thinking like executor hostname rollups) | 22:50 |
| clarkb | oh ya let me look at grafyaml | 22:50 |
| clarkb | hrm the grafyaml configs don't explicitly do anything with those names | 22:51 |
| clarkb | it must just be matching a * in the graphite stats paths? | 22:51 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI https://review.opendev.org/c/opendev/system-config/+/776292 | 22:52 |
| clarkb | I don't know that we should merge those together since they are distinct, maybe carry the old stats for a while then delete them? | 22:52 |
| *** hashar has quit IRC | 22:53 | |
| clarkb | or maybe drop the grafana zuul dashboard and haev it recreate? its not entirely clear to me how much of that is remembered by grafana | 22:53 |
| clarkb | I do see where we laias the openstack names to their hostnames | 22:54 |
| corvus | yeah, grep for openstack.org (don't escape the dot) | 22:55 |
| corvus | i think that's what's causing issues on https://grafana.opendev.org/d/5Imot6EMk/zuul-status?orgId=1 right now | 22:56 |
| corvus | see "running builds" | 22:56 |
| clarkb | ya I'm working on a change | 22:56 |
| clarkb | I think I see how to make this better | 22:56 |
| corvus | cool | 22:57 |
| corvus | fwiw, i'd recommend just cutting over to the new names at once; the old guages are stuck at their old values (will need a statsd restart to kill them) | 22:57 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Update grafana for opendev zuul executors https://review.opendev.org/c/openstack/project-config/+/779626 | 23:00 |
| *** tkajinam has joined #opendev | 23:00 | |
| clarkb | corvus: ^ yup that does a cut over or attempts to anyway. I expect the glob I've done on the metric path is valid like that? | 23:00 |
| corvus | clarkb: https://graphite.opendev.org/?showTarget=stats.gauges.zuul.executor.*_opendev_org.running_builds&width=586&height=308&target=stats.gauges.zuul.executor.*_opendev_org.running_builds&hideLegend=false looks legit | 23:02 |
| clarkb | yup graphite docs seem to say my glob is valid | 23:02 |
| clarkb | https://graphite-api.readthedocs.io/en/latest/api.html#paths-and-wildcards was what I found | 23:02 |
| clarkb | they have examples with multiple * and context between | 23:03 |
| clarkb | corvus: ianw has +2'd the chagne did you want to approve it or shoudl I go ahead and do that | 23:15 |
| corvus | done | 23:17 |
| clarkb | thanks | 23:18 |
| *** mlavalle has joined #opendev | 23:22 | |
| openstackgerrit | Merged openstack/project-config master: Update grafana for opendev zuul executors https://review.opendev.org/c/openstack/project-config/+/779626 | 23:26 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] kerberos ansible https://review.opendev.org/c/opendev/system-config/+/778840 | 23:42 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI https://review.opendev.org/c/opendev/system-config/+/776292 | 23:54 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!