| *** rlandy is now known as rlandy|out | 01:17 | |
| opendevreview | Merged opendev/system-config master: gitea: fix loadbalancer forwarding in testing https://review.opendev.org/c/opendev/system-config/+/848793 | 01:24 |
|---|---|---|
| opendevreview | wangxiyuan proposed opendev/system-config master: Update openEuler mirror repo https://review.opendev.org/c/opendev/system-config/+/848703 | 01:30 |
| opendevreview | wangxiyuan proposed opendev/system-config master: Update openEuler mirror repo https://review.opendev.org/c/opendev/system-config/+/848703 | 01:31 |
| opendevreview | wangxiyuan proposed opendev/system-config master: Update openEuler mirror repo https://review.opendev.org/c/opendev/system-config/+/848703 | 01:40 |
| opendevreview | wangxiyuan proposed opendev/system-config master: [WIP]Drop openEuler 20.03 LTS SP2 repo mirror https://review.opendev.org/c/opendev/system-config/+/848796 | 01:40 |
| opendevreview | wangxiyuan proposed opendev/system-config master: Update openEuler mirror repo https://review.opendev.org/c/opendev/system-config/+/848703 | 01:44 |
| opendevreview | wangxiyuan proposed opendev/system-config master: [WIP]Drop openEuler 20.03 LTS SP2 repo mirror https://review.opendev.org/c/opendev/system-config/+/848796 | 01:45 |
| opendevreview | Ian Wienand proposed opendev/system-config master: letsencrypt: selfsigned testing certs - use common CA, setup SAN https://review.opendev.org/c/opendev/system-config/+/848562 | 01:58 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea-lb: test ssl connections during testing too https://review.opendev.org/c/opendev/system-config/+/848777 | 01:58 |
| opendevreview | Ian Wienand proposed opendev/system-config master: paste : move testing host to paste99, remove https hacks https://review.opendev.org/c/opendev/system-config/+/848685 | 01:58 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: test loadbalancer to gitea https://review.opendev.org/c/opendev/system-config/+/848687 | 01:58 |
| ianw | well the os-testr/pbr thing doesn't replicate in a xenial docker image, to start | 02:56 |
| ianw | sigh, nor on a xenial image i just jumped on | 02:59 |
| *** ysandeep|out is now known as ysandeep|ruck | 03:12 | |
| ianw | ahh, https://review.opendev.org/c/openstack/diskimage-builder/+/787827 is the pre-install for glean | 03:13 |
| opendevreview | Ian Wienand proposed openstack/project-config master: nodepool elements: fix pip upgrade venv https://review.opendev.org/c/openstack/project-config/+/848800 | 03:32 |
| ianw | frickler / clarkb: ^ ... rather embarrassing error on my behalf there... | 03:33 |
| ianw | given the time i'm going to self-approve and monitor that, dropped a comment with a link to local replication | 03:42 |
| opendevreview | Merged openstack/project-config master: nodepool elements: fix pip upgrade venv https://review.opendev.org/c/openstack/project-config/+/848800 | 03:53 |
| *** raukadah is now known as chandankumar | 04:22 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: test loadbalancer to gitea https://review.opendev.org/c/opendev/system-config/+/848687 | 04:29 |
| *** ysandeep|ruck is now known as ysandeep|ruck|afk | 04:32 | |
| *** ysandeep|ruck|afk is now known as ysandeep|ruck | 05:08 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: test loadbalancer to gitea https://review.opendev.org/c/opendev/system-config/+/848687 | 06:04 |
| *** ysandeep|ruck is now known as ysandeep|ruck|lunch | 08:43 | |
| ianw | looks like xenial is creating qcows now | 09:12 |
| * frickler moos approvingly, but still doesn't understand why that error only seemed to break things a year later | 09:18 | |
| *** ysandeep|ruck|lunch is now known as ysandeep|ruck | 09:53 | |
| *** rlandy|out is now known as rlandy | 10:24 | |
| *** dviroel|out is now known as dviroel | 11:25 | |
| fungi | for a moment i thought it might be more recent fallout from https://review.opendev.org/846248 but that hasn't merged yet | 11:30 |
| *** rlandy is now known as rlandy|brb | 12:23 | |
| *** rlandy|brb is now known as rlandy | 12:42 | |
| fungi | okay, one remaining bit of weirdness i've noticed... in the message headers for mailing list posts after the conversion to https urls, the List-Archive header still says http but the others (List-Unsubscribe, List-Subscribe) correctly say https | 13:07 |
| fungi | not a huge deal of course, since the archive urls will get redirected to https anyway, but there must be something still missing somewhere | 13:08 |
| fungi | ahh, looks like DEFAULT_URL_PATTERN may not be the whole story. https://wiki.list.org/DOC/4.27%20Securing%20Mailman%27s%20web%20GUI%20by%20using%20Secure%20HTTP-SSL%20%28HTTPS%29?action=show&redirect=DOC%2F4.27+Securing+Mailman%27s+web+GUI+by+using+Secure+HTTP-SSL suggests also setting PUBLIC_ARCHIVE_URL which our current configs don't do | 13:12 |
| fungi | from the existing behavior and the wording there, i'm now inferring that mailman constructs that url with an http scheme if you don't set PUBLIC_ARCHIVE_URL at all | 13:13 |
| fungi | i'll put together a change to add that, for completeness | 13:13 |
| fungi | aha, it's inherited from Defaults.py | 13:17 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Update Mailman archive URLs to use HTTPS https://review.opendev.org/c/opendev/system-config/+/848843 | 13:25 |
| *** dasm|off is now known as dasm | 13:39 | |
| opendevreview | Merged opendev/base-jobs master: Separate swift provider selection from the swift log upload task for base-test https://review.opendev.org/c/opendev/base-jobs/+/848027 | 13:44 |
| fungi | jrosser_: ^ testing that can proceed at your convenience | 13:45 |
| jrosser_ | fungi: excellent - thanks | 13:46 |
| fungi | thanks for working through it! | 13:47 |
| *** dviroel is now known as dviroel|lunch | 14:59 | |
| *** ysandeep|ruck is now known as ysandeep|dinner | 15:28 | |
| opendevreview | Jonathan Rosser proposed zuul/zuul-jobs master: [DNM] test 848027 https://review.opendev.org/c/zuul/zuul-jobs/+/848880 | 15:38 |
| opendevreview | Jonathan Rosser proposed zuul/zuul-jobs master: [DNM] test 848027 https://review.opendev.org/c/zuul/zuul-jobs/+/848880 | 15:49 |
| clarkb | jrosser_: note I think the first patch was correct | 15:50 |
| jrosser_ | oh hmm | 15:50 |
| jrosser_ | i was digging around for an example of that | 15:50 |
| clarkb | jrosser_: the second will ensure the base-jobs repo is also copied onto the test node (which isn't necessary) | 15:50 |
| jrosser_ | as it didnt appear to run | 15:50 |
| clarkb | jrosser_: I think it did looking at https://zuul.opendev.org/t/zuul/build/12cd1331897944b2a3f55560e7ff182d/ | 15:50 |
| clarkb | because the uploads upload our logs there is a bit of a chicken and egg so things might be cut off at ^ but if you grep base-test you'll see it ran that version | 15:52 |
| jrosser_ | feel like i am asking a stupid question but how did you find the results | 15:54 |
| *** marios is now known as marios|out | 15:54 | |
| clarkb | jrosser_: I found it via the zuul status page for your change | 15:55 |
| clarkb | https://zuul.opendev.org/t/zuul/status there | 15:55 |
| jrosser_ | oh right well the reason i went messing with it a second time was i thought it had not run at all, as theres nothing reported back to gerrit | 15:55 |
| clarkb | it may not have completed fully? | 15:56 |
| jrosser_ | i went there and put 848880 in the search box and there was no sign of it | 15:57 |
| clarkb | whne you push a new ps it deletes the jobs for the old one and won't report back so if that happened before it completed | 15:57 |
| clarkb | jrosser_: were you in the openstack tenant view? | 15:57 |
| jrosser_ | oh quite possibly :) | 15:57 |
| clarkb | zuul run its jobs in the zuul tenant. So you have to switch tenant contexts in the status page | 15:57 |
| jrosser_ | doh | 15:57 |
| clarkb | In the top right of the web page is a Tenant link if you clikc that it takes you to the listing and hyou can change | 15:58 |
| jrosser_ | i think browser autocomplete got me there | 15:58 |
| clarkb | anyway I've looked at the console stream, console rendered from json and the executor debug log and I think this looks good to me. Others should confirm though | 15:58 |
| fungi | thanks! i was planning to take a look as soon as i finish lunch | 15:59 |
| clarkb | https://paste.opendev.org/show/b3PlARhoBRvLyUtgd1s7/ | 16:00 |
| clarkb | One thing I'm noticing looking at that is that {{ item }} is the "loop_var" can we override that to not be {{ item }} when using with_random_choice? | 16:00 |
| clarkb | the raeson for that is that we've found that if you end up nested somehow (maybe less of a concern with this particular change) that {{ item }} can end up with weird valeus from outer scopes | 16:01 |
| clarkb | But other than that I think this is working as expected and the {{ item }} thing is probably fine in this scenario as it isn't in a role but a playbook (which means it is top level) | 16:01 |
| opendevreview | Jonathan Rosser proposed opendev/base-jobs master: Separate swift provider selection from the swift log upload task https://review.opendev.org/c/opendev/base-jobs/+/848881 | 16:01 |
| jrosser_ | clarkb: oh yes i can change that name | 16:02 |
| clarkb | jrosser_: in zuul-jobs we try to use a zj_ var prefix for loop vars. This is opendev base jobs so maybe opendev_base_ or something | 16:03 |
| *** ysandeep|dinner is now known as ysandeep | 16:03 | |
| clarkb | but its a minor concern given this is at the top level playbook | 16:03 |
| *** dviroel|lunch is now known as dviroel | 16:08 | |
| *** ysandeep is now known as ysandeep|out | 16:08 | |
| opendevreview | Jonathan Rosser proposed opendev/base-jobs master: Avoid potential loop variable collision in base-test post-logs https://review.opendev.org/c/opendev/base-jobs/+/848883 | 16:09 |
| fungi | clarkb: jrosser_: yep, debug log on ze02 doesn't appear to have emitted any swift credentials for build 12cd1331897944b2a3f55560e7ff182d, and it does seem to have included the provider name ahead of the upload task | 16:14 |
| fungi | this should do nicely | 16:15 |
| jrosser_ | the {{ foo }} in task names is really handy | 16:16 |
| fungi | once 848883 merges and we get a good recheck on the test change, we should be ready to update the base job for real | 16:17 |
| jrosser_ | i also did this https://review.opendev.org/c/zuul/zuul-jobs/+/848153 | 16:18 |
| jrosser_ | which really is at the stage of "is this even a good idea?" | 16:18 |
| clarkb | Given that the debugging in the base job works and will work even in the timeout acse I think 848153 may not be necessary? I don't think it is problematic just not useful unless you modify the base jobs too (to run the upload twice) and we've already got a base job update that works | 16:21 |
| jrosser_ | right - i wasnt sure of the value of having the url to whatever partial upload might had happened | 16:22 |
| jrosser_ | that is something else that gets lost along the way with the kind of failures we were seeing | 16:23 |
| opendevreview | Merged opendev/base-jobs master: Avoid potential loop variable collision in base-test post-logs https://review.opendev.org/c/opendev/base-jobs/+/848883 | 16:25 |
| clarkb | I think it does add a bit more value if we need to fetch the content after the fact for further debugging | 16:26 |
| clarkb | as we'll know the exact location in that case | 16:27 |
| opendevreview | Jonathan Rosser proposed zuul/zuul-jobs master: [DNM] test 848027 and 848883 https://review.opendev.org/c/zuul/zuul-jobs/+/848880 | 16:40 |
| clarkb | https://paste.opendev.org/show/bO5NBF1aBd6mvKZ1kn2o/ has a warning in it which might be related to the loop var update? I don't understand that | 17:24 |
| clarkb | going to recheck to see what shows up in the streaming log | 17:24 |
| clarkb | the paste was for build e2106ee5f1fe4583902eed0ae3699bdb on ze10 | 17:25 |
| corvus | clarkb: i think that's a zuul bug -- check for that in the debug logs of other normal jobs | 17:25 |
| clarkb | corvus: ya its the callback plugin that zuul provides that is complaining. | 17:26 |
| corvus | clarkb: oh, i see there's a specific warning about item -- you may be right. ignore what i said :) | 17:26 |
| clarkb | corvus: fwiw that does show up for other jobs | 17:26 |
| clarkb | But ya I'm wondering if this specific change of item to something else is maybe what triggers it | 17:26 |
| corvus | yeah; unclear | 17:27 |
| corvus | that method assumes 'item' in result_dict | 17:29 |
| corvus | that could be a keyerror if it isn't there | 17:30 |
| clarkb | it should probably look at "ansible_loop_var": "opendev_base_item" to find the name instead? | 17:31 |
| clarkb | but ya that seems to be the likely cause of that. I think it is ok here? or do we want to be extra careful about it? The values operated on there are not secrets though so should be fine either way I think | 17:31 |
| corvus | maybe -- assuming that is the error | 17:31 |
| corvus | (this should be able to be tested in a zuul unit test, fwiw) | 17:32 |
| clarkb | there is a console stream difference. When we used {{ item }} the random selection task printed its selection. When using {{ opendev_base_item }} the task succeeds but doesn't record that value and we record it as part of the name of the next task | 17:33 |
| clarkb | I think this is safe but with that slightly less helpful amount of debugging (but all the info we're aiming to add is there) | 17:33 |
| corvus | i'm assuming it didn't record the value because of the callback plugin failure | 17:33 |
| clarkb | ya makes sense | 17:34 |
| clarkb | given that I'm happy to go back to item them update it if/when we fix the zuul callback | 17:34 |
| clarkb | or leave it as is and wait for when we fix it | 17:34 |
| corvus | should i be able to see the change in https://zuul.opendev.org/t/zuul/build/12cd1331897944b2a3f55560e7ff182d/log/job-output.txt ? | 17:36 |
| corvus | oh found it | 17:36 |
| corvus | https://zuul.opendev.org/t/zuul/build/12cd1331897944b2a3f55560e7ff182d/log/job-output.txt#1397-1399 | 17:36 |
| corvus | so is that run from before the variable change or after? | 17:37 |
| corvus | okay, that's the before | 17:39 |
| corvus | and https://zuul.opendev.org/t/zuul/build/e2106ee5f1fe4583902eed0ae3699bdb/log/job-output.txt#1413-1415 is the after | 17:39 |
| corvus | why does that just say `ok: "ovh_bhs"` and not `Item: ovh_bhs Runtime: ...` ? where does _log_messag() go? | 17:41 |
| clarkb | ya thats what I think is eaten by hte error? But I'm not sure | 17:41 |
| corvus | well it doesn't show up in the old version either | 17:42 |
| corvus | oh, i think that's the command case | 17:43 |
| clarkb | oh you're asking about a slightly different format of output. Where would that come from? | 17:43 |
| clarkb | ah | 17:43 |
| corvus | https://opendev.org/zuul/zuul/src/branch/master/zuul/ansible/base/callback/zuul_stream.py#L513-L517 | 17:43 |
| corvus | there, i think that's the case we should hit, and that produces the output we see in the "item" case, and that's the likely KeyError in the other case | 17:44 |
| corvus | so yeah, i think i'm 99% sure i understand this now and that's what's happening :) | 17:44 |
| clarkb | makes sense | 17:46 |
| clarkb | and I think we can do result_dict[result_dict["ansible_loop_var"]] to fix it? | 17:46 |
| corvus | yeah | 17:47 |
| corvus | clarkb: er, do you have a link to that ssh thing fedora users were having to do to talk to gerrit? | 18:15 |
| corvus | oh i think i found it https://lists.opendev.org/pipermail/service-discuss/2021-October/000291.html | 18:16 |
| clarkb | corvus: ya the openssh release notes for 8.8 have details now that upstream openssh has caught up with fedora | 18:19 |
| clarkb | Gerrit 3.6 is expected to fix this | 18:20 |
| corvus | in the mean time.. it seems jammy is affected | 18:20 |
| clarkb | ya anything with openssh 8.8 or newer will be | 18:20 |
| clarkb | fedora just happened to do it on 8.7 early | 18:20 |
| clarkb | I still contend that all of those clients are buggy and should default to rsa + sha2 if they have disabled rsa + sha1 | 18:21 |
| clarkb | I looked at the openssh code for a while and got lost trying to figure out how to update the default | 18:21 |
| opendevreview | Neil Hanlon proposed openstack/diskimage-builder master: Add Rockylinux 9 build configuration and update jobs for 8 and 9 https://review.opendev.org/c/openstack/diskimage-builder/+/848901 | 19:42 |
| *** dviroel is now known as dviroel|out | 20:50 | |
| *** dasm is now known as dasm|off | 21:36 | |
| *** rlandy is now known as rlandy|biab | 21:50 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: static: remove unused letsencrypt group variable https://review.opendev.org/c/opendev/system-config/+/848906 | 22:21 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] static: move certs to group, update testing name to static99 https://review.opendev.org/c/opendev/system-config/+/848908 | 22:49 |
| ianw | btw after i stuffed up and accidentally killed it, then re-ran it (much slower, as it checked the current setting before updating), all 2,138,256 stats finally have their xfilesfactor set to 0 on graphite now | 22:58 |
| ianw | #status log all stats on graphite.opendev.org reset with an xfilesfactor of 0; c.f. https://review.opendev.org/c/opendev/system-config/+/847876 | 23:00 |
| opendevstatus | ianw: finished logging | 23:00 |
| *** rlandy|biab is now known as rlandy | 23:01 | |
| fungi | thanks for fixing that! | 23:25 |
| *** rlandy is now known as rlandy|out | 23:37 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!