| *** cloudnull7 is now known as cloudnull | 00:23 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit docs : remove old database docs/update duplicate account info https://review.opendev.org/c/opendev/system-config/+/804183 | 01:22 |
|---|---|---|
| ianw | clarkb / fungi: ^ I think that covers what I just did | 01:22 |
| *** ykarel|away is now known as ykarel | 04:46 | |
| opendevreview | Merged openstack/diskimage-builder master: Introduce openEuler distro https://review.opendev.org/c/openstack/diskimage-builder/+/784363 | 04:57 |
| *** marios is now known as marios|ruck | 05:11 | |
| *** bhagyashris_ is now known as bhagyashris | 05:39 | |
| *** rpittau|afk is now known as rpittau | 07:23 | |
| *** jpena|off is now known as jpena | 07:32 | |
| yoctozepto | morning infra; related question to one of my recent ones - could we configure gerrit to allow the (new, gerrit's own) wip flag to be lifted by project cores? | 07:47 |
| opendevreview | Merged opendev/elastic-recheck rdo: Run elastic-recheck in container https://review.opendev.org/c/opendev/elastic-recheck/+/803932 | 08:20 |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 08:39 |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 11:10 |
| opendevreview | Merged openstack/project-config master: afsdocs_secret-tox-docs-site: Zuul 4.6.0 fix https://review.opendev.org/c/openstack/project-config/+/804170 | 11:12 |
| opendevreview | Andreas Jaeger proposed openstack/project-config master: Adjust secrets for developer.o.o https://review.opendev.org/c/openstack/project-config/+/804226 | 11:14 |
| *** rpittau is now known as rpittau|afk | 11:28 | |
| *** dviroel|out is now known as dviroel | 11:32 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** jpena|lunch is now known as jpena|off | 12:28 | |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 13:21 |
| *** ykarel is now known as ykarel|away | 13:23 | |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 13:27 |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 13:54 |
| tristanC | corvus: clarkb: it seems like eavesdrop.openstack.org:9001 is blocked from the internet, can we open the port (and how?). Otherwise could you please paste the output of `curl http://eavesdrop.openstack.org:9001/metrics` | 13:56 |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 13:56 |
| fungi | yoctozepto: that would have to be configured on a per-project basis. if you can find the right access control in the gerrit docs for it, propose a change and we'll review and try it out | 14:07 |
| yoctozepto | fungi: ack, I hoped someone tried that already :-) I will try it when I need it again and have time to research gerrit :D | 14:08 |
| corvus | tristanC: looking | 14:19 |
| corvus | tristanC: was thisk working earlier? (i thought this was how you discovered the connection issues) | 14:22 |
| fungi | #status log Killed an htcacheclean process on mirror01.bhs.ovh.opendev.org which had been squatting the flock since 2021-07-21, and then cleanly restarted the apache2 service since at least one of its workers logged a segfault in dmesg at 10:18:55 UTC when its cache volume filled completely | 14:24 |
| tristanC | corvus: it was not enabled in opendev (though i used the endpoint to diagnose the issue locally) | 14:24 |
| opendevstatus | fungi: finished logging | 14:24 |
| corvus | tristanC: oh, so that graph was from a locally running copy? | 14:24 |
| fungi | given the timing, i suspect the hung htcacheclean on the bhs1 mirror was related to our afs server restarts | 14:25 |
| corvus | for the issue at hand -- it looks like the base playbook has exited with an error, however, the logs indicate that it did write the iptables config on eavesdrop01. but the file on disk does not have 9001 in it. so it's looking like somehow the change to add the port isn't working as expected. | 14:26 |
| tristanC | corvus: yes | 14:27 |
| opendevreview | James E. Blair proposed opendev/system-config master: Test port 9001 on eavesdrop https://review.opendev.org/c/opendev/system-config/+/804247 | 14:28 |
| corvus | tristanC: ^ i will be curious what the results of those tests are | 14:28 |
| corvus | tristanC: meanwhile, i'll try to make a small playbook to manually run to help debug | 14:29 |
| opendevreview | Ananya proposed opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 14:45 |
| opendevreview | James E. Blair proposed opendev/system-config master: Remove eavesdrop from webservers group https://review.opendev.org/c/opendev/system-config/+/804255 | 14:47 |
| corvus | tristanC: ^ i suspect that's the problem. if it is, then 804247 will fail tests and 804255 will pass. if that happens, we can squash them. | 14:48 |
| tristanC | corvus: nice, thank you for looking into that! | 14:48 |
| corvus | np | 14:49 |
| opendevreview | James E. Blair proposed opendev/system-config master: Remove port 22 from webservers extra ports https://review.opendev.org/c/opendev/system-config/+/804256 | 14:49 |
| corvus | and that's unrelated cleanup ^ | 14:49 |
| *** jpena|off is now known as jpena | 14:57 | |
| clarkb | yoctozepto: fungi: there is an acl for it, but unfortunately the gerrit docs don't tell you what the raw contents are just describe the high level objects since they expect people to edit them via the web ui | 15:03 |
| clarkb | yoctozepto: fungi: I think you can create a change in the web ui and view the diff of that then abandon the change in the web ui and propose it to project-config though | 15:03 |
| yoctozepto | clarkb: I don't know about the web ui :-( | 15:05 |
| clarkb | yoctozepto: if you view the acls for a repo there should be an edit button that allows you to make changes with a gui then propose them for review. We don't use that process but if you do that you get diffs you can apply to project-config acl files iirc | 15:05 |
| clarkb | corvus: do you need to squash the test and group changes together in order for the test to pass? | 15:06 |
| corvus | clarkb: the group is a followup so it should pass. we will need to squash to merge (but i want to see the test fail first) | 15:08 |
| clarkb | got it | 15:08 |
| corvus | hrm, the job failed, but it failed restarting apache? | 15:12 |
| corvus | https://8f916be78216e964d9a4-88701ef6bf8de11ee3b199ef1b631f6f.ssl.cf2.rackcdn.com/804247/1/check/system-config-run-eavesdrop/becc5c5/bridge.openstack.org/ara-report/results/363.html | 15:13 |
| corvus | i've rechecked, but i didn't expect or understand that failure | 15:14 |
| yoctozepto | clarkb: I seem unable to do anything on https://review.opendev.org/admin/repos/openstack/kolla-ansible,access it's read-only | 15:14 |
| clarkb | yoctozepto: that must be a change since we upgraded gerrit. That is unfortunate. In that case we probably have to go read some gerrit source instead | 15:15 |
| clarkb | yoctozepto: https://gerrit-review.googlesource.com/Documentation/access-control.html#category_toggle_work_in_progress_state is the documentation fwiw | 15:15 |
| yoctozepto | clarkb: yeah, but it gives no clues | 15:16 |
| yoctozepto | don't worry though, it's no priority for us at the moment ;-) | 15:16 |
| clarkb | yoctozepto: public static final String TOGGLE_WORK_IN_PROGRESS_STATE = "toggleWipState"; | 15:16 |
| clarkb | yoctozepto: I have a checkout so grepping was easy :) I think that is the string you use to indicate the permission you want | 15:17 |
| clarkb | then it is like create, push, etc | 15:17 |
| yoctozepto | clarkb: ack, many thanks; then I will propose a change later | 15:19 |
| yoctozepto | now in a meeting | 15:19 |
| *** sshnaidm is now known as sshnaidm|afk | 15:35 | |
| *** jpena is now known as jpena|off | 15:42 | |
| clarkb | anyone know why zuul didn't run jobs against https://review.opendev.org/c/opendev/infra-specs/+/804122 ? I'm going to recheck it. Its jobs are defined in openstack/project-config | 16:11 |
| clarkb | rechecking it does seem to have queued the docs job | 16:14 |
| *** dviroel is now known as dviroel|away | 16:19 | |
| fungi | could have been pushed during a zuul restart i guess? | 16:27 |
| fungi | mmm, no there was no restart in progress at that time | 16:28 |
| *** ykarel is now known as ykarel|away | 16:33 | |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Find (s)testr more reliably https://review.opendev.org/c/zuul/zuul-jobs/+/804280 | 17:05 |
| roman_g | Hello team. CityCloud reports that they are experiencing ongoing issues in KAN1 region, their Glance API is crashing and needs to be restarted multiple times during last hours. Their operations team says that this is somehow connected to the operations being done under opendevtest account. Account has created 391 images today, and that might be the reason of glance failures, which causes problems to City Networks. | 17:09 |
| roman_g | Please, check what is going on there. Thank you. | 17:09 |
| clarkb | roman_g: nodepool will restart its uploads if previous ones fail which could cause a feedback loop | 17:10 |
| clarkb | we can disable the cloud region and stop uploads but I strongly suspect this isn't the cause of the problem | 17:10 |
| clarkb | I'm not sure what there is to check on our end other than it is likely failing in a loop | 17:11 |
| roman_g | Are there any more details available to you on why and on which operations their Glance is failing? | 17:11 |
| clarkb | roman_g: the cloud should have that information availale on their end... but we can look | 17:11 |
| roman_g | Thank you. I will forward information that we are in a loop to them. | 17:12 |
| clarkb | roman_g: openstack clients typically et very terse error messages and to do any real debugging you hvae to look at hte cloud side fwiw | 17:14 |
| roman_g | OK. Requesting. | 17:14 |
| roman_g | Thank you, clarkb. | 17:14 |
| clarkb | roman_g: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://kna1.citycloud.com:9292/v2/images/d71af024-7021-458f-a268-65b255c8f011/file: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) | 17:15 |
| clarkb | "Broken Pipe" | 17:15 |
| roman_g | Interesting. Thank you. | 17:15 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Disable airship citylcoud nodepool provider https://review.opendev.org/c/openstack/project-config/+/804281 | 17:19 |
| clarkb | infra-root ^ is that the correct way to stop image uploads? | 17:19 |
| roman_g | Endpoint https://kna1.citycloud.com:9292 responds appropriately, but I don't have creds to test, so getting http 401 not authorized. | 17:20 |
| clarkb | roman_g: the way that nodepool works for image is it builds a new image for eahc image type daily then will upload it to all he clouds in a loop | 17:20 |
| clarkb | So ya us creating a bunch of images is expected if image creations fail, but I dout that we are the source ofthe problem. Possibly amplfying it or making it more visible | 17:20 |
| corvus | clarkb: lgtm | 17:21 |
| fungi | clarkb: we should be able to interactively pause? | 17:21 |
| fungi | though maybe that only pauses builds not uploads | 17:22 |
| fungi | roman_g: based on the error, it looks like we get partway through uploading the image and then the connection is terminated | 17:22 |
| clarkb | fungi: is that documented? I'm not finding how that works with grepping pause in the docs | 17:23 |
| mordred | we've seen issues like that with clouds before due to unhappy load balancers - the lb will terminate on a timeout, and then we'll happily retry the upload which will still trigger the timeout - enter an endless loop | 17:23 |
| fungi | though "partway" may be as few as 0 bytes, i can't really tell | 17:23 |
| fungi | clarkb: https://review.opendev.org/747306 doesn't seem to have included docs, just usage output for --hel | 17:25 |
| fungi | p | 17:25 |
| fungi | and i guess it's for pausing an image not just a provider | 17:26 |
| clarkb | oh ya we want to do a specific provider in this case | 17:26 |
| fungi | yep, so what you have is what we want i think | 17:27 |
| clarkb | I'll go ahead an manually modify nb01 and nb02 as well now that the change is approved | 17:28 |
| fungi | thanks | 17:29 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Allow kolla cores to toggle kolla wipstate https://review.opendev.org/c/openstack/project-config/+/804283 | 17:29 |
| clarkb | alright it has been manually applied on the two builders now | 17:32 |
| clarkb | roman_g: ^ should stop retrying for a bit though ansible may put the old config back again depending on when the change above merges and when our hourly updates happen | 17:33 |
| roman_g | clarkb thank you. I'm waiting for reply from provider. See you tomorrow. | 17:34 |
| roman_g | I will read evening logs. | 17:34 |
| yoctozepto | clarkb: thanks again and here is the patch to merge: https://review.opendev.org/c/openstack/project-config/+/804283 | 17:35 |
| yoctozepto | it seems ooo already used it | 17:35 |
| fungi | yoctozepto: oh! good, so you can ignore my comment about it probably causing a failure in the validator, we must have already solved it when tripleo decided to give it a try | 17:37 |
| yoctozepto | fungi: yeah, I replied on the review as well | 17:38 |
| yoctozepto | I'm glad it went this smooth again | 17:39 |
| opendevreview | Merged openstack/project-config master: Disable airship citylcoud nodepool provider https://review.opendev.org/c/openstack/project-config/+/804281 | 17:39 |
| yoctozepto | thanks mordred | 17:42 |
| * mordred has been useful for the week | 17:42 | |
| yoctozepto | :-) | 17:42 |
| fungi | and it's only wednesday! | 17:43 |
| jrosser | is there a fix for git review -> error: remote unpack failed: error Missing tree 3e154d5146909cb52cf17b71f8a6630448aab485 | 17:55 |
| opendevreview | Merged openstack/project-config master: Allow kolla cores to toggle kolla wipstate https://review.opendev.org/c/openstack/project-config/+/804283 | 17:56 |
| clarkb | jrosser: sort of. The issue is in jgit not git review | 17:56 |
| jrosser | i've had this a couple of times today | 17:56 |
| clarkb | jrosser: latest git review has a --no-thin flag which can be used when you have that problem to workaround it | 17:56 |
| clarkb | the flag shouldn't be used always as it is far more computationally intensive but when you hit this issue can be supplied | 17:56 |
| jrosser | once after trying to propose a revert of a patch straight after proposing it | 17:57 |
| fungi | and that needs git-review 2.1.0 just fyi | 17:57 |
| jrosser | and just now i did a trivial fix to the commit message with commit --amend | 17:57 |
| clarkb | jrosser: yes reverts are apparently involved in one of the known reproduction cases, but upstream closed the bug and told me to go away when I asked them to reopening it :/ | 17:57 |
| jrosser | doh :( | 17:57 |
| clarkb | jrosser: https://bugs.chromium.org/p/gerrit/issues/detail?id=1582 is the bug, the issue is in jgit. git review is an innocent bystander | 17:58 |
| clarkb | but ya try the --no-thin flag | 17:58 |
| jrosser | oh thats awesome, --no-thin worked straight away | 17:58 |
| fungi | which is essentially a passthrough to git push's --no-thin option | 17:58 |
| clarkb | jrosser: https://groups.google.com/g/repo-discuss/c/AtMvu8rW8gc/m/RL31a361BQAJ is a recent thread on it if you want to read up on what I said to upstream | 17:59 |
| clarkb | there is another workaround we can do but it presents a security issue so we don't | 17:59 |
| fungi | i love that they closed the bug report because it "looks pretty old" and even though the message invited people to report if they could reproduce it on recent gerrit versions they told you privately that they didn't care? | 18:01 |
| yoctozepto | duh | 18:02 |
| clarkb | fungi: no they told me in the thread Ilinked above | 18:02 |
| clarkb | basically because I don't have a set of reproduction steps the bug isn't worth keeping open | 18:03 |
| clarkb | I responded with basically "I understand that not having a simple reproduction case makes fixing this harder but the problem seems understood by the JGit maintainers nad it still happens with our 3.2 gerrit" | 18:04 |
| clarkb | basically they set the bar so high that fixing this (what I would consider) major bug isn't important enough | 18:04 |
| clarkb | I asked becaues I couldn't reopen the bug myself fwiw | 18:06 |
| fungi | clarkb: interesting, that bug's comments skip from 2017 to 2021 so i don't see any from you | 18:06 |
| clarkb | fungi: right it is on the mailing list thread | 18:06 |
| clarkb | I didn't bothcommenting on the bug bceause it is closed and I can't reopenin it | 18:06 |
| fungi | ahh | 18:06 |
| clarkb | wow typing is hard. I didn't bother commenting on the bug because it is closed and I can't reopen it | 18:06 |
| fungi | i see now, i missed your link to the ml | 18:07 |
| fungi | i guess if we could capture an exact remote and local repository state which exhibit the issue, then we might be able to reproduce it with those? | 18:08 |
| clarkb | ya or go through the steps to reproduce in the original bug and see if they reproduce on modern gerrit | 18:09 |
| fungi | which means we'd need to tell someone to not work around it, tar up their local repository, snapshot the bare repo on the gerrit side, and then load those up in a test deployment | 18:09 |
| clarkb | I personally don't feel like users should be asked to go through that effort | 18:10 |
| clarkb | if you read the thread there is clear indication that jgit understands the exact problem | 18:10 |
| clarkb | gerrit wants to pretend it doesn't happen anymore and force end users to jump over a very high bar | 18:11 |
| clarkb | they even fixed it once but reverted because it created performance regressions | 18:11 |
| *** dviroel|away is now known as dviroel | 19:09 | |
| opendevreview | Merged zuul/zuul-jobs master: Find (s)testr more reliably https://review.opendev.org/c/zuul/zuul-jobs/+/804280 | 19:21 |
| smcginnis | I'm seeing something odd with gerrit queries, wondering if someone might know why. | 19:54 |
| fungi | please elaborate! | 19:54 |
| smcginnis | If I get the changes from the API using https://review.opendev.org/changes/?q=reviewer:purestorage-thirdparty-ci the json returned only has a small amount only against opendev/ci-sandbox. | 19:55 |
| smcginnis | But if I query in the UI with https://review.opendev.org/q/commentby:purestorage-thirdparty-ci, it returns a bunch of reviews. | 19:55 |
| smcginnis | Can't see why they would be different. | 19:55 |
| fungi | those where they voted vs those where they only commented? | 19:58 |
| fungi | it probably helps to add options to expand the response so it contains the comments, i'll check | 19:58 |
| smcginnis | Ah, wasn't thinking reviewer vs commentby. | 19:59 |
| smcginnis | They do show up in the "CC" group now, not in "Reviewers" in the UI. | 20:00 |
| clarkb | reviewer is changes that have been or need to be reviewed by a user and I think gerrit does define that as voting | 20:00 |
| fungi | "REVIEWER: Users with at least one non-zero vote on the change." | 20:03 |
| fungi | https://review.opendev.org/Documentation/rest-api-changes.html#change-info | 20:03 |
| smcginnis | Weird that this lastcomment-scoreboard code works for most CI's. None of them are voting. | 20:03 |
| fungi | "CC: Users that were added to the change, but have not voted." | 20:04 |
| smcginnis | I'll see if it works with querying by commentby instead. | 20:05 |
| fungi | i agree https://review.opendev.org/Documentation/user-search.html#_search_operators does a poor job of distinguishing those | 20:06 |
| clarkb | there is also reviewedby | 20:07 |
| fungi | yeah, which is subtly different apparently | 20:07 |
| fungi | also i find the lack of alpha-ordering for entries there... maddening | 20:08 |
| smcginnis | Looks like in this case it is equivalent since I have the (theoretical) list of CI IDs, so the script just needs to get a set of recent reviews to inspect. | 20:08 |
| smcginnis | I'll run it with both and compare output to see if there's any difference. | 20:08 |
| smcginnis | No comments found. CI SYSTEM UNKNOWN - I'll do some more debugging, but doesn't look like changing that filter to either of those worked. | 20:31 |
| clarkb | smcginnis: curl -X GET https://review.opendev.org/changes/?q=reviewedby:purestorage-thirdparty-ci returns a bunch of results for me and seems to match what I get in the dashboard | 20:34 |
| clarkb | commentby seems to do similar as well | 20:34 |
| clarkb | and reviewer. So the UI and API seems to line up | 20:35 |
| *** dviroel is now known as dviroel|ruck | 20:35 | |
| smcginnis | The script has spaces in the URL. I wonder if something is causing an issue with that. Still not sure why it is only for this CI and all the other accounts come through fine. | 20:36 |
| fungi | when looking at the changes where that account is listed as a reviewer, it does seem to have left a vote at some point on at least one patchset | 20:38 |
| smcginnis | OK, debugged some more and it looks like it is actually failing somewhere in parsing the response. Which then causes a failure that emits CI SYSTEM UNKNOWN, which in this case is a little misleading. | 20:58 |
| smcginnis | Thanks for the pointers! | 20:58 |
| smcginnis | If curious, this is what actually gets called: curl -X GET "https://review.opendev.org/changes/?q=reviewedby%3A%22purestorage-thirdparty-ci%22%20AND%20project%3A%22openstack/cinder%22&o=MESSAGES&o=DETAILED_ACCOUNTS" | 20:58 |
| smcginnis | https://opendev.org/x/third-party-ci-tools/src/branch/master/monitoring/lastcomment-scoreboard/cireporter.py#L24 | 20:58 |
| fungi | but only for that account? | 21:00 |
| smcginnis | Yeah. There are a few others failing, but they could be really old deactivated accounts. Most are working fine, this is the only one failing in this way that I've identified so far. | 21:05 |
| smcginnis | Appears this does not return anything (empty iterator): https://opendev.org/x/third-party-ci-tools/src/branch/master/monitoring/lastcomment-scoreboard/cireporter.py#L34 | 21:05 |
| clarkb | I deactivated a mellanox cinder ci account recently | 21:05 |
| clarkb | I havne't heard any screaming about it since I did it and from what I could tell it hadn't been used in a long time | 21:06 |
| smcginnis | Looks like the last run must have worked for some mellanox account - http://cinderstats.ivehearditbothways.com/cireport.txt | 21:06 |
| smcginnis | The failing ones in there are mostly expected. | 21:06 |
| clarkb | ya I think they consolidated accounts | 21:06 |
| smcginnis | At least so far, other than the pure one. | 21:06 |
| clarkb | there was a cinder specific one and now I htink they use a generic ci account | 21:07 |
| corvus | tristanC, clarkb, ianw: https://review.opendev.org/804247 finally failed on the actual issue it should have failed on (attempt #3). attempts 1 and 2 failed on an apache restart issue related to the limnoria handler. i don't understand that. but that also took out the follow up change which should have working tests. | 21:17 |
| corvus | in other words, i think there's a flaw in the eavesdrop playbook which has a 75% chance of causing test failures, but i don't understand it. | 21:18 |
| corvus | see https://zuul.opendev.org/t/openstack/build/4af4ff824cea40a88890a84703e33796 for an example failure | 21:18 |
| corvus | it doesn't look like the jobs save the journal, so we can't see the error | 21:19 |
| clarkb | that is running as an ansible handler in response to writing out the apache config for the apache server | 21:20 |
| corvus | i guess the limnoria role writes out that config? | 21:21 |
| clarkb | ya | 21:21 |
| opendevreview | James E. Blair proposed opendev/system-config master: Test port 9001 on eavesdrop https://review.opendev.org/c/opendev/system-config/+/804255 | 21:21 |
| opendevreview | James E. Blair proposed opendev/system-config master: Remove port 22 from webservers extra ports https://review.opendev.org/c/opendev/system-config/+/804256 | 21:21 |
| clarkb | internet says this can happen if the ports already have things listening on them | 21:21 |
| clarkb | that seems unlikely here unless it is the old apache process and not stopping reliably in the restart | 21:22 |
| clarkb | the other possibility is that the new vhost template is invalid for some reason | 21:24 |
| clarkb | maybe the ssl cert provisioning didn't succeed earlier with the mocked out letsencrypt? | 21:24 |
| corvus | clarkb: yes: https://zuul.opendev.org/t/openstack/build/4af4ff824cea40a88890a84703e33796/log/eavesdrop01.opendev.org/acme.sh/acme.sh.log | 21:26 |
| clarkb | ya I was looking at the ara side and it almost looks like we're not calling the script with the test flag? But I'm reading what the script does now. https://6fe7a4c45f2f7bdcea66-eeccdc6968a6b16fa4ca2e3ee7c1080d.ssl.cf5.rackcdn.com/804255/1/check/system-config-run-eavesdrop/4af4ff8/bridge.openstack.org/ara-report/results/248.html | 21:27 |
| clarkb | LETSENCRYPT_STAGING env var is how we toggle the staging flag | 21:28 |
| corvus | it's hitting the staging server, but according to the curl output, there's an ssl handshake error with the acme server | 21:29 |
| clarkb | oh I see | 21:29 |
| clarkb | so we are staging but we are failing | 21:29 |
| corvus | that's my understanding; like maybe network error or LE server issues? | 21:30 |
| clarkb | ya could be | 21:30 |
| clarkb | that could also explain why it is fine now. They fixed the upstream issue | 21:30 |
| corvus | true, success was the latest run | 21:30 |
| corvus | cloud provider does not correlate | 21:31 |
| corvus | okay, i think we should assume it's LE or Internet and see what the next run comes back with | 21:32 |
| clarkb | ++ | 21:32 |
| corvus | clarkb: thanks :) | 21:32 |
| clarkb | looking through my old changes https://review.opendev.org/c/opendev/system-config/+/791832/1/launch/make_swap.sh is one that should be easy to land. We'll just want to confirm the results the next time we boot an instance | 21:42 |
| *** dviroel|ruck is now known as dviroel|ruck|out | 21:46 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force gitea failure for interaction https://review.opendev.org/c/opendev/system-config/+/800516 | 21:52 |
| clarkb | fungi: I have just discovered https://review.opendev.org/c/opendev/system-config/+/758594/1/playbooks/rename_repos.yaml is a change I pushed to upate rename repos after the server upgrade. Notice it removes the mysql steps (sorry I wish I remembered I had pushed this change) but it also does reindexes for groups and projects | 21:56 |
| clarkb | it seems this wasn't strictly necessary after the recent rename we did, but I suspect we do need the groups reindex at least if groups change names | 21:56 |
| clarkb | also the projects reindex is needed if acls change maybe? | 21:56 |
| clarkb | I'm going to rebase that so that it is mergeable and I suspect we may want to merge it | 21:57 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add additional post project rename reindexing https://review.opendev.org/c/opendev/system-config/+/758594 | 22:00 |
| clarkb | trying to do some spring cleaning at the end of summer :) | 22:00 |
| ianw | clarkb: just while it's in my mind, would be good if you could read through https://review.opendev.org/c/opendev/system-config/+/804183 | 22:24 |
| ianw | updates docs for removing user emails | 22:24 |
| clarkb | ianw: ++ I meant to do that then it slipped my mind. Will do so now to prevent that happening again | 22:24 |
| ianw | no rush; just the relational model between bits tends to fall out of my head fairly quickly :) | 22:25 |
| clarkb | ianw: left a few notes. I'm not sure any one rise to a -1 but together it is probably worht an update | 22:29 |
| ianw | so after it's fixed, you can point the externalid at their old account? | 22:33 |
| clarkb | yes | 22:35 |
| clarkb | basically we put the file back in place for the openid then chnage the accoundId in the file to be the old account instead of the new | 22:35 |
| clarkb | currently gerrit rejects that push | 22:35 |
| clarkb | but it should accept them once the conflicts are all removed | 22:35 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit docs : remove old database docs/update duplicate account info https://review.opendev.org/c/opendev/system-config/+/804183 | 22:37 |
| clarkb | fungi: ^ did you want to look that over since you did one recently too? | 22:39 |
| clarkb | if not I think we can approve that | 22:39 |
| fungi | sure, just a sec | 22:42 |
| clarkb | https://www.eclipsestatus.io/incidents/rsr9qzb1yl2h we may find that interesting, also possible we might learn something from it too | 22:49 |
| fungi | ianw: some comments, i'm not opposed, but am mostly concerned about the bits where it's authenticating with you.admin through the rest api | 22:51 |
| ianw | argh, more stable branch debian-stable references ... openstack-ansible-nspawn-container-create-debian-stable | 23:13 |
| clarkb | oh ya that reminds me there are a bunch of x/tap-as-a-service zuul config errors in the openstack tenant | 23:14 |
| clarkb | I saw them when trying to sort out why my infra-specs change didn't run jobs | 23:14 |
| clarkb | I know we've said we won't care too much, but its hard to not notice and want to go force merge a bunch of fixes :/ | 23:14 |
| fungi | supposedly at least some of those were going away with branch eols/deletions | 23:14 |
| opendevreview | Merged opendev/system-config master: gerrit docs : remove old database docs/update duplicate account info https://review.opendev.org/c/opendev/system-config/+/804183 | 23:15 |
| ianw | i think codesearch got branch overrides, so we could setup some separate instances to index stable branches | 23:24 |
| ianw | like codesearch/wallaby/... | 23:24 |
| mordred | neat | 23:25 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!