| opendevreview | Jan Marchel proposed openstack/project-config master: Add NebulOuS optimiser repo https://review.opendev.org/c/openstack/project-config/+/896070 | 08:45 |
|---|---|---|
| opendevreview | Jan Marchel proposed openstack/project-config master: Add NebulOuS optimiser-utility-evaluator repo https://review.opendev.org/c/openstack/project-config/+/896070 | 08:56 |
| opendevreview | Jan Marchel proposed openstack/project-config master: Add NebulOuS optimiser-utility-evaluator repo https://review.opendev.org/c/openstack/project-config/+/896070 | 08:56 |
| yoctozepto | https://review.opendev.org/c/openstack/project-config/+/896070 <- any project creation heroes out there? :-) | 10:46 |
| opendevreview | Merged openstack/project-config master: Add NebulOuS optimiser-utility-evaluator repo https://review.opendev.org/c/openstack/project-config/+/896070 | 12:48 |
| yoctozepto | aand this time my thanks go to... fungi | 13:03 |
| opendevreview | Merged opendev/git-review master: Warn rather than fail if HEAD already exists on the remote https://review.opendev.org/c/opendev/git-review/+/895751 | 13:04 |
| fungi | any time | 13:07 |
| opendevreview | Merged opendev/git-review master: feat(cmd): add hashtag implementation https://review.opendev.org/c/opendev/git-review/+/854649 | 13:32 |
| ildikov | Hello OpenDev team! I have a question about the mm3 migration, specifically regarding the starlingx-discuss ML. | 14:08 |
| ildikov | I'm a list admin for that mailing list, and received an email that there is a new message in the moderation queue. The notification email doesn't have a link to the admin page, even though the previous version provided that. | 14:09 |
| ildikov | I already had a mm3 account, since I'm on other mailing lists that were migrated before. But, the starlingx-discuss mailing list does not show up under that account, even though all the email addresses are listed that I'm signed up with. | 14:10 |
| ildikov | Do I need to create a new account on top of the one that I already have? Or is there any other way to have starlingx-discuss to show up on my existing account? | 14:11 |
| fungi | ildikov: which lists are displayed will depend on which domain you visit. are you clicking the login link at https://lists.starlingx.io/mailman3/lists/starlingx-discuss.lists.starlingx.io/ | 14:26 |
| fungi | the account is shared across all our mailman sites, but you have to log into each one independently (when we get an sso linked to it that may get simpler) | 14:27 |
| ildikov | fungi: this is the first time I see that link, prolly missed it in other emails before. I used this one: https://lists.opendev.org/mailman3/lists/ | 14:27 |
| fungi | oh, sorry. https://lists.starlingx.io/ is the url for the starlingx mailing lists (still the same as it was before). logging in anywhere on that site should hopefully get you to what you need | 14:28 |
| fungi | https://lists.opendev.org/ is the url for the opendev mailing lists, https://lists.airshipit.org/ is for the airship mailing lists, and so on | 14:29 |
| fungi | each one is filtered by the lists which are associated with their domain | 14:29 |
| fungi | https://lists.openinfra.dev/ is another | 14:29 |
| fungi | as is https://lists.zuul-ci.org/ | 14:30 |
| fungi | and https://lists.katacontainers.io/ too | 14:30 |
| fungi | the only mailman site we haven't migrated yet is https://lists.openstack.org/ but that should be done in a few weeks, just wanted to wait until after the openstack release | 14:33 |
| ildikov | ah, ok, got it | 14:35 |
| ildikov | so I will have one account across all domains, but I can't access the MLs on different domains in one place | 14:36 |
| ildikov | is that an accurate way to sum it up? | 14:36 |
| fungi | correct | 14:37 |
| fungi | we had a configuration error early on where the zuul and opendev mailing lists all showed up on each others management interfaces, but that's been fixed | 14:37 |
| ildikov | well, I would be happy to still have that error, lol | 15:15 |
| ildikov | fungi: thank you for the pointer and clarifications! | 15:15 |
| fungi | yw | 15:18 |
| slittle | please add me as first member for starlingx-app-node-interface-metrics-exporter-core. As usual I'll add the rest. Thanks | 15:19 |
| fungi | slittle: done | 15:29 |
| clarkb | I'm working on local software updates and after that I' | 15:53 |
| clarkb | er I'll recycle my gerrit hold to grab the latest patchset I just pushed upstream to the replication plugin and test that | 15:53 |
| clarkb | decided i didn't need to test code i knew would be replaced almost immediately | 15:53 |
| clarkb | infra-root: etherpad made a new release if anyone has time to whip up a change for that. Also, gitea has new RCs for the next release we can start looking at | 16:04 |
| fungi | i can work on the etherpad update in a sec | 16:14 |
| clarkb | corvus: the zuul deployment on Friday should've pulled in the token fallback for github? So now we can create a token using an account (do we have one, maybe the one that created the zuul app?) and add it to the github config? | 16:31 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM testing replication plugin https://review.opendev.org/c/opendev/system-config/+/896290 | 17:33 |
| clarkb | our explicit checkouts for plugin tags override my upstream changesi n master that I was trying to test. I was really confused why nothing seemed to work the way I expected it on my test machine until I sorted that out | 17:34 |
| clarkb | but now I have a working replication config for local replication that was able to reproduce the issues we have in production and I should hopefully see those go away with that latest patchset | 17:34 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Upgrade Etherpad to 1.9.3 https://review.opendev.org/c/opendev/system-config/+/896454 | 17:40 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add nebulous/activemq repo https://review.opendev.org/c/openstack/project-config/+/896455 | 17:40 |
| fungi | guilhermesp: do you happen to know if anyone ever looked into the ipv6 routing loop in ca-ymq-1? i'm still seeing it, fwiw | 17:42 |
| corvus | clarkb: that's all generally correct except that we already have that api_token in the config file, so no action should be needed | 17:46 |
| clarkb | corvus: oh nice! I didn't realize it was already handled | 17:49 |
| opendevreview | Merged openstack/project-config master: Add nebulous/activemq repo https://review.opendev.org/c/openstack/project-config/+/896455 | 18:03 |
| fungi | infra-root: the etherpad 1.9.3 change seems to be passing tests. changelog is minimal seems to only affect deployment choices other than the ones we make. any preference for whether we hold and manually exercise a job node for this one? | 18:12 |
| clarkb | I tend to prefer doing a hold mostly because it is hard to test what etherpad does automatically given its highly interactive nature | 18:15 |
| frickler | I won't have time to check the hold and looking at the changelog would be fine to proceed without one, but up to you | 18:17 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 18:29 |
| clarkb | fwiw I just looked at the gitea rc release and there is no changelog yet so not super urgent | 18:30 |
| frickler | infra-root: https://ptg.opendev.org/ seems down, nothing listening on localhost:8000, maybe related to new image deployed this morning? | 18:38 |
| clarkb | frickler: whcih image was that? | 18:39 |
| frickler | clarkb: ptgbot? or is something else doing the web backend? | 18:41 |
| clarkb | https://review.opendev.org/c/openstack/ptgbot/+/885501 that one? | 18:41 |
| clarkb | ya I think ptgbot does it. I just wanted aware of changes but that does appear to be a likely candidate | 18:41 |
| clarkb | assumiong we have to apply similar to the dockerfile? | 18:41 |
| frickler | that's only a change to the readme, so the real change behind that must be older | 18:43 |
| frickler | but then maybe the page has been down for longer | 18:44 |
| clarkb | diablo_rojo: ^ was the website checked after the python3.11 update? | 18:45 |
| clarkb | fwiw there don't appear to be any logs indicating a crash but I agree the web server process does not seem to be working | 18:45 |
| clarkb | the ircbot side is running and logging | 18:45 |
| diablo_rojo | clarkb: ehhh I am not sure. I looked Friday. When was the py3.11 update? | 18:46 |
| clarkb | diablo_rojo: wednesday ish | 18:47 |
| fungi | there were two ptgbot changes merged yesterday, btw | 18:47 |
| clarkb | both of those changes don't appear to affect hte web server though | 18:48 |
| clarkb | but they would have triggered rebuilds which may have pulled in newer deps of something | 18:48 |
| fungi | yeah, i approved them but didn't expect them to break anything given they weren't touching any actual code, so it's possible something external influenced the build | 18:48 |
| fungi | my point was we got a new image in the past 24 hours | 18:49 |
| clarkb | https://zuul.opendev.org/t/openstack/build/ff51ffdb08dc496b8066bc87c12ab42b/log/job-output.txt logs for most recent build | 18:49 |
| clarkb | https://zuul.opendev.org/t/openstack/build/8a196d8f1f2f4f03bec3dd69746a257e/log/job-output.txt logs for the python3.11 update on wednesday | 18:49 |
| diablo_rojo | clarkb: I am pretty sure I saw the website up and fine since then. 80% sure I was looking at it Friday. | 18:50 |
| fungi | Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8000 (localhost) failed | 18:50 |
| fungi | i'm able to connect to localhost:8000 and get content back from cli | 18:51 |
| clarkb | fungi: the process is running now but was not before | 18:52 |
| clarkb | the docker container has not restarted. Maybe frickler started it manually in the container? | 18:52 |
| fungi | i think i may have just crashed it | 18:52 |
| fungi | i can't connect any more | 18:53 |
| clarkb | ya it isn't runnin anymore | 18:53 |
| frickler | I tried running it manually, yes, that worked | 18:53 |
| clarkb | maybe something else crashed it like fungi did | 18:53 |
| clarkb | ? | 18:53 |
| frickler | but I stopped it again so we can check why it didn't run automatically or stopped in between | 18:53 |
| clarkb | I see | 18:53 |
| clarkb | it doesn't look like the web server logs a log file like the ircbot does | 18:55 |
| fungi | it may log to the container output | 18:55 |
| clarkb | fungi: ya it does | 18:56 |
| fungi | docker-compose logs has some content | 18:56 |
| clarkb | but nothing useful for an indication of why the process stopped | 18:56 |
| fungi | nothing useful in syslog or dmesg either | 18:57 |
| frickler | my next idea would be to restart the docker container and see what happens | 18:58 |
| fungi | yeah, i think that's probably where we're at. and look at adding some logging to the process | 18:59 |
| frickler | well, we already run with --debug, I don't think there is much more we can do? "usage: ptgbot-web [-h] [-d] [-p PORT] [--debug] configfile" | 19:01 |
| clarkb | frickler: someone (maybe diablo_rojo?) would need to improve the webserver to have better logging | 19:02 |
| frickler | ok, so that's longer term then. any objection to just restarting now? | 19:04 |
| fungi | at a minimum, it looks like it should be logging the string 'Debugging on' at start if the debugging option is working, and i don't see that | 19:04 |
| diablo_rojo | I think there are a lot of changes that need to happen to the bot in general to accommodate pivoting to making meetpad the default (which I unfortunately think will have to happen after this PTG; I really don't have the time to get that done and tested etc before the PTG). | 19:04 |
| fungi | also its RequestHandler class is supposed to be logging every single request it receivs | 19:05 |
| diablo_rojo | And in super meta form I thought about having a ptgbot session towards the end of the week to plan some of that out, but I haven't planned that. | 19:05 |
| diablo_rojo | frickler: no objection from me | 19:05 |
| fungi | so i agree there seems to be the intent of debug logging but it doesn't appear to actually be happening? | 19:06 |
| fungi | oh, actually that is getting logged | 19:09 |
| fungi | ptgbot_1 | 2023-09-25T01:18:08.201924174Z DEBUG:root:Debugging on | 19:09 |
| frickler | fungi: I see "DEBUG:root:Debugging on" in the docker log | 19:09 |
| fungi | so maybe it's just that RequestHandler.log_message() isn't being called by anything | 19:09 |
| frickler | just nothing after that, even when it is serving requests | 19:09 |
| frickler | diablo_rojo: making meetpad the default is something I would very much like to see happen, so I'd volunteer to join any effort supporting that | 19:11 |
| fungi | also it's not clear to me whether what's there is sufficient to capture exceptions/tracebacks | 19:11 |
| diablo_rojo | frickler: that would be amazing. I definitely think it needs to happen too. Especially now that people have more familiarity and experience and the scale of the meetings is smaller so meetpad can definitely handle it. | 19:12 |
| fungi | frickler: yeah, i had a think about meetpad-as-default and basically instead of requiring the room urls to be supplied in the default config blob, it shouldn't be too hard to infer them from the etherpads for the teams as they're booked (just call the same thing that the url command does whenever the etherpad url is set or changed) | 19:13 |
| fungi | most of what would be needed is already there, because the url command overrides the default room link on any room the team is booked into with one that's specific to the team | 19:14 |
| frickler | alright so I see you're way ahead of me, I didn't look at any of the implementation things so far | 19:17 |
| fungi | as for logging, the reason i thought it wasn't working is that i don't see 'Debugging on' in syslog, even though the compose file sets logging to syslog | 19:17 |
| fungi | so whatever logging is being captured by docker/docker-compose doesn't seem to be replicating into syslog | 19:18 |
| frickler | anyway, it's late here, so I'll step out for now and leave that debugging to you, will check back tomorrow | 19:19 |
| fungi | unless the "syslog" logging driver doesn't do what i expect it to | 19:19 |
| fungi | thanks frickler! | 19:19 |
| fungi | okay, i'll go ahead and start the container back up and see what happens | 19:20 |
| fungi | and yeah, it's up and responding but still doesn't log individual requests | 19:22 |
| clarkb | fungi: syslog means write logs using the syslog protocol but not necessarily to the syslog file. In this case those contents go to /var/log/containers files using syslogd tagging rules iirc | 19:43 |
| clarkb | the infrastructure logging is fine as far as I can tell. The problem is htep rocess isn't logging much | 19:43 |
| fungi | ohhh | 19:43 |
| clarkb | I've just opushed what I hope is a reviewable version of the replication plugin fix. I rechecked and recycled my hold goign to eat lunch while I wait for that to finish | 19:44 |
| fungi | yeah, i can confirm /var/log/containers/docker-ptgbot.log has the same things that docker-compose logs outputs | 19:44 |
| fungi | held etherpad 1.9.3 has ip address 173.231.255.77 | 19:47 |
| fungi | i need to go grab early dinner, but can try it out when i get back | 19:47 |
| clarkb | I've just marked https://gerrit-review.googlesource.com/c/plugins/replication/+/387314 ready for review and left notes about the testing I did there if anyone is curious | 21:06 |
| clarkb | fungi: I'm using the held etherpad in the clarkb-test pad | 21:47 |
| clarkb | fungi: it works in firefox but breaks in chrome | 21:48 |
| clarkb | ReferenceError: require is not defined | 21:49 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Deprecate mirror-workspace-git-repos https://review.opendev.org/c/zuul/zuul-jobs/+/888366 | 21:49 |
| clarkb | if I create a random new pad it seems to not break? | 21:49 |
| clarkb | and now clarkb-test works. I wonder if I had old js cached | 21:50 |
| clarkb | hrm then it started reconnecting when I tried to use chat but now works. Weird | 21:50 |
| clarkb | but ya best guess is caches. Maybe you want to clear all your caches for etherpad.opendev.org before you test and see if you can reproduce? if not we are probably fine | 21:51 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Deprecate mirror-workspace-git-repos https://review.opendev.org/c/zuul/zuul-jobs/+/888366 | 21:51 |
| clarkb | I suppose it is possible that it is trying to hit the prod server because chrome isn't respecting /etc/hosts properly | 21:52 |
| clarkb | wow reconnectiong is very persistent :/ | 21:52 |
| clarkb | ya icognito seems more stable so almost certainly a local state problem. Of course we risk having problems on upgrade when everyone's local state is wonly | 21:53 |
| fungi | i'll test in an ff container | 21:55 |
| fungi | seems to work fine for me. no disconnects | 21:58 |
| clarkb | can you try chromium? | 21:58 |
| clarkb | my main concern is that maybe chrom* have some problem with evicting cached state so when we upgrade people will be sad for a short time. In FF it definitely seems to just work with no problems | 21:59 |
| clarkb | as well as in incognito mode | 21:59 |
| fungi | i don't often use chromium, so probably nothing to clear in my case | 22:00 |
| fungi | but it seems fine (not incognito, but it's the only tab i have open) | 22:00 |
| clarkb | in my case I only use it for testing this stuff | 22:00 |
| clarkb | which I guess means it is possible I've cached some development type stuff for etherpad in chrome | 22:00 |
| fungi | i'll test from qutebrowser on my netbook, it tends to struggle with etherpad's widget layouts for some reason | 22:01 |
| clarkb | so it seems like it is probably ok to upgrade. maybe if we can find another person to test that might be good but otherwise don't worry too much about it. | 22:09 |
| fungi | yeah, i see no problems | 22:09 |
| clarkb | related to the gerrit replication plugin testing `gerrit logging set-level all` is super useful | 22:09 |
| fungi | other than my netbook's networking is fighting me, but i don't think that's the test server's problem | 22:10 |
| fungi | oh, because i can't type. wrong ip address | 22:16 |
| fungi | yeah etherpad's still not great in qutebrowser but this version is no worse than the last | 22:16 |
| clarkb | time to edit the meeting agenda. I can put etherpad and gitea and gerrit things on there. Anything else we want to discuss? The openstack release I suppose as well as ptgbot web server crash? | 22:35 |
| fungi | sounds good. i don't have anything else other than mm3 recap and planning | 22:40 |
| clarkb | fungi: how does the updated agenda look? | 22:51 |
| fungi | sorry, looking | 23:20 |
| clarkb | no worries I just sent it out | 23:20 |
| clarkb | we can always add stuff in during the meeting if necessary | 23:20 |
| fungi | clarkb: looks good to me | 23:22 |
| clarkb | fungi: one thought re bindep and mysql/mariadb woes. The client is almost certainly there for test environment setup not direct integration with the software | 23:22 |
| fungi | i'm interested to dig deeper on the zuul registry topic and find out won't work with openssl v3 | 23:22 |
| fungi | yeah, so the metapackage is probably fine | 23:23 |
| clarkb | fungi: the thing that doesn't work with openssl v3 is python's rehash library that allows for pickling/saving hash function state on data so that you don't have to recompute hashes from scratch as you add more data to files. This is improtant to the registry due to how you add in data over time to images | 23:23 |
| fungi | is rehash part of stdlib or external? | 23:23 |
| clarkb | external. | 23:23 |
| clarkb | it does mangling of C things as well as memory so is sensitive to the openssl apis and their changes | 23:24 |
| fungi | https://pypi.org/project/rehash/ that thing | 23:24 |
| fungi | https://github.com/kislyuk/rehash/issues/6 | 23:24 |
| clarkb | yup | 23:24 |
| fungi | got it | 23:26 |
| fungi | clarkb: so, on debian sid where there is no longer any libssl1.1 and hasn | 23:32 |
| fungi | 't been for a while, i build openssl from source and link my python source builds to that | 23:32 |
| fungi | in theory we could build old openssl in those images? | 23:32 |
| clarkb | in this case we would need to link rehash to openssl not stdlib ? or maybe it is both? that gets complicated with tehse images beacuse we're consuming a prebuild python from upstream but is theoretically doable | 23:32 |
| fungi | yeah, i dunno if rehash built for libssl1.1 would work with a stdlib linked against libssl3 | 23:33 |
| clarkb | it checks the version using from ssl import OPENSSL_VERSION so ya we'd need to rebuild python on the image | 23:33 |
| fungi | makes sense | 23:34 |
| clarkb | butthen we'd stop using the upstream python and potentially have differences. Thats a lot of work for a single app | 23:35 |
| clarkb | probably easier to just keep using bullseye for now | 23:35 |
| clarkb | then trim all the bullseye builds for python that aren't that specific one | 23:35 |
| fungi | fwiw, cpython itself doesn't even touch openssl, the only thing from stdlib that links it is the ssl module | 23:36 |
| fungi | looks like i can probably drop my custom openssl linking for cpython 3.9 and later now | 23:37 |
| clarkb | rereading the issue in rehash the problem is more fundamental than simply updating the use of C apis | 23:37 |
| fungi | right | 23:38 |
| fungi | looks like it relies on internal implementation details of old openssl | 23:38 |
| clarkb | that seems like a big flaw in openssl 3.0 since hashing is expensive being able to resume from a function state is useful a number of reasons | 23:38 |
| clarkb | fungi: well it seems old openssl had a specific interface for capturing this state and now openssl does not | 23:38 |
| fungi | there are probably other cryptographic primitives that are preferable to incremental hashing, but i don't know enough about the problem domain to say what exactly | 23:39 |
| clarkb | we need to dangle "resumable sha256sum + rust" in front of the internt and wait for takers. Shouldn't take long :) | 23:39 |
| clarkb | fungi: we're limited by the docker image protocol. I'm not sure we have alternatives | 23:39 |
| fungi | oh, right if docker cares about a specific hash construction then that's that | 23:40 |
| clarkb | docker image protocol uses sha256 for everything and different objects get uploaded at different times and "combined" into a single thing that needs hashing | 23:40 |
| fungi | https://www.openssl.org/docs/man3.0/man7/migration_guide.html#Providers-are-a-replacement-for-engines-and-low-level-method-overrides is disappointingly brief | 23:44 |
| clarkb | looks like golang stdlib has resumable hashes built in | 23:44 |
| clarkb | we could do a hacky export of Go method to C and call it from python that way assuming we can serialize the necessary state in a meaningful way | 23:45 |
| fungi | If a library context is needed then all EVP_* digest functions that return a const EVP_MD * such as EVP_sha256() should be replaced with a call to EVP_MD_fetch(3). See "ALGORITHM FETCHING" in crypto(7). | 23:48 |
| fungi | The algorithm implementation that is fetched can then be used with other diverse functions that use them. For example the EVP_DigestInit_ex(3) function takes as a parameter an EVP_MD object which may have been returned from an earlier call to EVP_MD_fetch(3). | 23:49 |
| fungi | so that looks like maybe you can resume it by passing the structure along from iteration to iteration? | 23:50 |
| fungi | but this is pretty far out of my wheelhouse | 23:50 |
| fungi | i guess the underlying problem is that while you can repeatedly call EVP_DigestUpdate() on a context to add more data, it's not clear that there's a way to export the internal state of that context so that a later process can continue adding data | 23:54 |
| fungi | so you're stuck holding that context in process memory | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!