| ianw | kevinz: so the linaro-us mirror is shutoff again | 00:06 |
|---|---|---|
| ianw | mirror01.regionone.linaro-us.opendev.org | SHUTOFF | 00:06 |
| ianw | it's back after i started it, but i don't know why it keeps disappearing | 00:14 |
| *** ryohayakawa has joined #opendev | 00:17 | |
| ianw | https://github.com/pyca/cryptography/pull/5341/checks?check_run_id=942805616 ... checks api is reporting on pyca | 00:21 |
| fungi | nothing in syslog/wtmp to suggest when or why it went down? (we can probably approximate shutdown time to within 5 minutes based on snmp blackout time in cacti) | 00:21 |
| corvus | ianw: looks legit | 00:21 |
| ianw | fungi: last entry is "Aug 3 10:53:18 mirror01 kernel: [291253.841199] afs: volume location server 104.130.136.20 in cell openstack.org is back up (code 0)" | 00:22 |
| corvus | ianw: i wonder if we'll have a stale 'status' on that pr after the check run is done | 00:22 |
| fungi | looks like it went dark between 10:50 and 10:55 utc today | 00:23 |
| fungi | judging from cacti's graphs | 00:23 |
| ianw | we could put a netconsole on it, and see if we get a oops (i'm guessing yes) | 00:24 |
| ianw | or, update it to focal and see if it still happens too, before we spend too much time debugging an old kernel | 00:24 |
| ianw | corvus: yeah, i guess the old one will hang around, although it's a +1 from the last good run | 00:25 |
| fungi | cacti shows a fairly large outbound traffic spike shortly before it died | 00:25 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: launch-node : add sshfp records https://review.opendev.org/743461 | 01:04 |
| ianw | fungi: if you have a sec for a backup review on https://review.opendev.org/#/c/743445/1 adds zuul to the "new" backup server | 01:11 |
| fungi | lgtm, thanks for fixing that! | 01:25 |
| ianw | il | 01:27 |
| ianw | i'll start the oe mirror and make sure the launch script spits out sshfp records with it | 01:27 |
| openstackgerrit | Merged opendev/system-config master: launch-node : add sshfp records https://review.opendev.org/743461 | 01:33 |
| openstackgerrit | Merged opendev/system-config master: Backup inventory - match zuul01.openstack.org https://review.opendev.org/743445 | 01:48 |
| *** ysandeep|away is now known as ysandeep | 02:32 | |
| openstackgerrit | Adrian Turjak proposed openstack/project-config master: Return gnocchi back to openstack https://review.opendev.org/744592 | 02:51 |
| *** fressi has joined #opendev | 04:20 | |
| *** raukadah is now known as chkumar|rover | 04:23 | |
| *** ysandeep is now known as ysandeep|afk | 04:33 | |
| *** xiaolin has quit IRC | 05:10 | |
| *** marios has joined #opendev | 05:13 | |
| *** marios is now known as marios|ruck | 05:43 | |
| *** DSpider has joined #opendev | 05:54 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/744096 | 06:10 |
| *** ysandeep|afk is now known as ysandeep | 06:18 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/744096 | 06:37 |
| *** hashar has joined #opendev | 06:49 | |
| openstackgerrit | Merged openstack/project-config master: Add notifications for openstack-stable channel https://review.opendev.org/744050 | 06:51 |
| *** dtantsur|afk is now known as dtantsur | 07:16 | |
| *** tosky has joined #opendev | 07:48 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| *** AJaeger has joined #opendev | 08:08 | |
| *** fressi has quit IRC | 08:19 | |
| *** jhesketh has quit IRC | 08:24 | |
| openstackgerrit | Merged openstack/project-config master: Add Ceph iSCSI charm to OpenStack charms https://review.opendev.org/744479 | 08:32 |
| *** fressi has joined #opendev | 08:37 | |
| *** priteau has joined #opendev | 08:43 | |
| *** fressi has joined #opendev | 09:09 | |
| *** lpetrut has joined #opendev | 09:12 | |
| openstackgerrit | Merged openstack/project-config master: Revert "Remove os_congress gating" https://review.opendev.org/742532 | 09:24 |
| *** bolg has quit IRC | 09:28 | |
| *** hashar has quit IRC | 09:42 | |
| *** auristor has quit IRC | 10:01 | |
| *** tkajinam has quit IRC | 10:12 | |
| *** sshnaidm_ has joined #opendev | 10:12 | |
| *** sshnaidm has quit IRC | 10:15 | |
| *** jhesketh has joined #opendev | 10:21 | |
| *** sshnaidm_ is now known as sshnaidm | 10:26 | |
| openstackgerrit | Daniel Bengtsson proposed openstack/diskimage-builder master: Update the tox minversion parameter. https://review.opendev.org/738754 | 10:37 |
| *** tosky_ has joined #opendev | 11:20 | |
| *** tosky has quit IRC | 11:20 | |
| *** tosky_ is now known as tosky | 11:20 | |
| *** marios|ruck has quit IRC | 11:48 | |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Retire Zuul's Kata tenant https://review.opendev.org/744687 | 11:58 |
| *** xiaolin has joined #opendev | 12:05 | |
| *** tosky has quit IRC | 12:14 | |
| *** tosky_ has joined #opendev | 12:14 | |
| *** tosky_ is now known as tosky | 12:15 | |
| *** xiaolin has quit IRC | 12:15 | |
| *** hashar has joined #opendev | 12:16 | |
| *** ryo_hayakawa has joined #opendev | 12:28 | |
| *** ryohayakawa has quit IRC | 12:29 | |
| *** ryo_hayakawa has quit IRC | 13:02 | |
| *** ryo_hayakawa has joined #opendev | 13:03 | |
| *** auristor has joined #opendev | 13:05 | |
| *** ryo_hayakawa has quit IRC | 13:10 | |
| *** ysandeep is now known as ysandeep|mtg | 13:18 | |
| *** fressi has quit IRC | 13:25 | |
| *** ysandeep|mtg is now known as ysandeep | 14:00 | |
| *** mlavalle has joined #opendev | 14:00 | |
| *** fressi has joined #opendev | 14:14 | |
| *** ysandeep is now known as ysandeep|off | 14:16 | |
| *** hashar has quit IRC | 14:29 | |
| *** fressi has quit IRC | 14:34 | |
| *** chkumar|rover is now known as raukadah | 15:10 | |
| *** lpetrut has quit IRC | 15:15 | |
| *** sdmitriev has quit IRC | 15:16 | |
| *** openstackgerrit has quit IRC | 15:20 | |
| *** fressi has joined #opendev | 15:36 | |
| *** fressi has left #opendev | 15:36 | |
| *** lpetrut has joined #opendev | 15:43 | |
| *** hashar has joined #opendev | 15:45 | |
| *** lpetrut has quit IRC | 16:21 | |
| *** mlavalle has quit IRC | 16:22 | |
| *** mlavalle has joined #opendev | 16:23 | |
| *** sshnaidm is now known as sshnaidm|afk | 16:48 | |
| *** fressi has joined #opendev | 16:50 | |
| *** priteau has quit IRC | 16:51 | |
| *** priteau has joined #opendev | 16:51 | |
| *** fressi has quit IRC | 16:58 | |
| *** dtantsur is now known as dtantsur|afk | 17:00 | |
| *** hashar is now known as hashardinner | 17:16 | |
| clarkb | I've approved ^ I don't expect any issues but this is the first time we've retired a tenant | 17:17 |
| *** openstackgerrit has joined #opendev | 17:26 | |
| openstackgerrit | Merged openstack/project-config master: Retire Zuul's Kata tenant https://review.opendev.org/744687 | 17:26 |
| *** priteau has quit IRC | 17:33 | |
| *** lpetrut has joined #opendev | 17:49 | |
| *** lpetrut has quit IRC | 17:50 | |
| clarkb | https://zuul.opendev.org/tenants doesn't show the kata tenant so I expect that all went well | 18:27 |
| fungi | yeah, seems fine and all | 18:31 |
| *** auristor has quit IRC | 19:14 | |
| *** mtreinish has quit IRC | 19:27 | |
| *** auristor has joined #opendev | 19:46 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Increate nodepool builder upload workers from 4 to 8 https://review.opendev.org/744780 | 19:57 |
| clarkb | fungi: ^ thats the upload workers fix. I think we want to land that after the current round of uploads completes as they are almost done | 19:58 |
| clarkb | ianw: ^ fyi | 19:58 |
| *** sgw2 has joined #opendev | 20:02 | |
| sgw2 | Morning gang, I need to delete a tag in the stx-tools repo, it was applied slightly prematurely. I don't appear the have Force Push rights to allow me to delete the 4.0.0 tag (refs/tags/4.0.0). | 20:03 |
| clarkb | sgw2: typically we strongly recommend against deleting tags because any downstream pullers won't have their tag updated | 20:05 |
| clarkb | that means you can end up with weird repo states. Instead we recommend pushing a 4.0.1 or similar to address the problem in a roll forward fashion. that way all remote repos have a consistent state | 20:05 |
| sgw2 | I understand, this should have limited scope as the time window has only been an couple of hours. | 20:05 |
| clarkb | is a 4.0.1 tag inappropriate for some reason? | 20:06 |
| fungi | also ci systems will likely have pulled copies of that tag already | 20:06 |
| fungi | and won't know to replace them later if the tag name is reused | 20:06 |
| sgw2 | What ci system for starlingx, unless Zull deals with something | 20:07 |
| fungi | because tags aren't altered on git pull | 20:07 |
| sgw2 | Zuul sorry. | 20:07 |
| clarkb | yes zuul would be one of them | 20:08 |
| fungi | yeah, i mainly don't know if you have any ci/cd automation triggered from tags, just pointing out that any automation fetching refs from your repos will likely wind up with incorrect tags if you delete and later replace them | 20:08 |
| clarkb | (there are potentially others) | 20:08 |
| sgw2 | no automation other than our Jenkins builds which have not fired yet. | 20:08 |
| fungi | also anyone who has run something like `git remote update` will forever have the old tag locally (even if you replace it with a new tag later) unless they know to manually delete it | 20:09 |
| fungi | so for the most part we've considered tags permanent, and can't predict what the result will be if we delete one | 20:10 |
| clarkb | right its usually better to roll forward as that is predictable | 20:10 |
| clarkb | which is why I was asking what the concerns are with that | 20:10 |
| sgw2 | I understand the challenges and as I said this is a very small window. I will double check with our build/release team. | 20:11 |
| clarkb | if we do it we'll need to rm the zuul executors local repos for stx-tools | 20:15 |
| clarkb | corvus: ^ can that be done with zuul running or do we also need to stop the executors? | 20:15 |
| clarkb | (or I guess we may also be able to manually surgery the repos instead of having zuul reclone) | 20:15 |
| fungi | i can escalate my gerrit account privs to delete refs/tags/4.0.0 from the starlingx/tools repo momentarily | 20:15 |
| corvus | clarkb: i certainly wouldn't do it with an stx job running | 20:15 |
| corvus | clarkb: but i think aside from that, it should be okay | 20:15 |
| clarkb | corvus: k | 20:16 |
| clarkb | fungi: lets see what sgw2 after asking their build/release team. I can help with executor and merger repo cleanups | 20:17 |
| sgw2 | So it is more complex than just a tag deletion, that's extra info to inform our team. | 20:18 |
| fungi | it's definitely extra work. we don't even give ourselves permission to delete tags (or push --force for that matter) by default | 20:19 |
| fungi | part of why we have the second bullet in the "note" block under Request For Enhancement | 20:20 |
| fungi | er, under https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#tagging-a-release | 20:20 |
| fungi | "Tags can’t be effectively deleted once pushed, so make absolutely certain they’re correct (ideally by locally testing release artifact generation commands and inspecting the results between the tag and push steps above)." | 20:21 |
| *** hashardinner has quit IRC | 20:34 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openedge mirror: remove for replacement https://review.opendev.org/744785 | 20:37 |
| ianw | clarkb/fungi: ^ i think we should do that before i remove it from the emergency list, so we don't have non-responding hosts in inventory | 20:38 |
| clarkb | approved | 20:39 |
| clarkb | fungi: latest patchset of the identity service spec lgtm | 20:41 |
| ianw | thanks, once that merges i'll restart the oe mirror with the bigger instance, and try getting the sshfp keys from the host | 21:09 |
| donnyd | kk | 21:11 |
| openstackgerrit | Merged opendev/system-config master: openedge mirror: remove for replacement https://review.opendev.org/744785 | 21:17 |
| donnyd | ugg... I have taskers the general just came in and handed me that have to be done today.. be back in a bit | 21:34 |
| *** DSpider has quit IRC | 21:39 | |
| ianw | hrm, deployment didn't seem to go too well anyway | 21:52 |
| ianw | infra-prod-base failed | 21:52 |
| ianw | review-test has a full fs, which killed the run : /dev/xvda1 39G 39G 0 100% / | 21:54 |
| ianw | it's /usr/local/bin/track-upstream | 21:54 |
| ianw | i thought we fixed that | 21:54 |
| ianw | https://review.opendev.org/#/c/739840/ ... we did drop it from cron, in theory | 21:56 |
| ianw | ok, that has not applied because the gerrit playbook is failing on review-test | 22:04 |
| ianw | AnsibleUndefinedVariable: 'gerrit_vhost_name' is undefined | 22:04 |
| clarkb | ianw: ya I was hoping mordred would be around for an update on that host since we ran into trouble with it during project renames too | 22:07 |
| ianw | it seems /etc/ansible/hosts/host_vars/review-test.opendev.org.yaml is not in sync? | 22:08 |
| ianw | are we supposed to be reading that directly from system-config? | 22:08 |
| clarkb | ianw: I think it was intentionally different to avoid creating confusion with a truly prod server | 22:09 |
| clarkb | so it should be in sync to a degree but not completely? | 22:09 |
| clarkb | infra-root I've got a change ready to go for gerritbot on eavesdrop instead of review.o.o and running out of a container. Before I push it is there any concern using our actual channel config if I'm supplying bogus freenode nick/password and gerrit user connection details? | 22:09 |
| ianw | clarkb: so ... inventory/service/host_vars/review-test.opendev.org.yaml:gerrit_vhost_name: review-test.opendev.org isn't deployed to /etc/ansible/hosts? | 22:10 |
| clarkb | actually what I can do is comment out the inclusion of the role from the playbook then we can review it and decide if that is safe | 22:10 |
| clarkb | ianw: no we should include the contents of both /etc/ansible/hosts and system-config side by side | 22:11 |
| clarkb | rather than write system-config data into /etc/ansible/hosts on bridge iirc | 22:11 |
| clarkb | maybe that is broken? | 22:11 |
| ianw | ... i don't want to go too far down this rabbit hole right now. for now, i've just manually deleted the crontab entry | 22:11 |
| clarkb | that seems reasonable. It would be good to sync up with mordred on that as we keep running into weird behaviors related to it | 22:12 |
| ianw | ok, having another go at the oe mirror | 22:13 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add ansible role to manage gerritbot https://review.opendev.org/744795 | 22:15 |
| ianw | donnyd: does not look like it's my day. launching the server just went into error status, and now i can't delete it either | 22:15 |
| clarkb | infra-root ^ thoughts on the testing questions there would be great | 22:15 |
| clarkb | I put TODOs in the places where I had questions | 22:16 |
| clarkb | fungi: ^ fyi since you manually updated that services config | 22:17 |
| ianw | donnyd: i thought it might be the 250g instance, but similarly doesn't work for 80gb | 22:18 |
| ianw | different error though : 504: Server Error for url: https://api.us-east.open-edge.io:8774/v2.1/os-keypairs, 504 Gateway Time-out: The server didn't respond in time. | 22:18 |
| *** qchris has quit IRC | 22:22 | |
| *** qchris has joined #opendev | 22:35 | |
| *** tosky has quit IRC | 22:50 | |
| *** dpawlik2 has quit IRC | 22:55 | |
| *** guillaumec has quit IRC | 22:55 | |
| *** frickler has quit IRC | 22:55 | |
| *** tkajinam has joined #opendev | 22:55 | |
| *** dpawlik2 has joined #opendev | 22:57 | |
| *** guillaumec has joined #opendev | 22:57 | |
| *** frickler has joined #opendev | 22:57 | |
| *** mlavalle has quit IRC | 22:59 | |
| clarkb | ianw: donnyd just noticed that an ubuntu focal upload to openedge failed | 23:05 |
| clarkb | probably not super urgent but once we've got the mirror up we may want to look into upload reliability | 23:05 |
| ianw | yeah, given the launch issues i'm guessing it would all be related | 23:06 |
| fungi | clarkb: i'll try to review in the morning, but initial concern is that we may want to prevent it from trying to connect to freenode just so we're not pestering them with bogus login attempts | 23:12 |
| clarkb | fungi: ya I can make that configurable too and point it somewhere invalid | 23:12 |
| clarkb | though I think we may test our other bots against freenode | 23:13 |
| fungi | a really neat test might be to install something like inspircd from distro package and configure it to listen on the loopback | 23:13 |
| * fungi runs the debian-packaged inspircd and has for years, rather simple to set up | 23:14 | |
| fungi | also supports things like sasl auth and comes with nickserv/chanserv and the like, so we could test some fairly complex bot interactions eventually if we wanted | 23:15 |
| fungi | i could try to find time to add something like that, though it probably won't be this week | 23:17 |
| clarkb | ya maybe we do what we can for now then can add that in later | 23:18 |
| fungi | sounds reasonable | 23:19 |
| openstackgerrit | Merged opendev/system-config master: Increate nodepool builder upload workers from 4 to 8 https://review.opendev.org/744780 | 23:23 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!