| opendevreview | Jeremy Stanley proposed openstack/project-config master: Farewell limestone https://review.opendev.org/c/openstack/project-config/+/873475 | 00:07 |
|---|---|---|
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Remove empty limestone nodepool providers https://review.opendev.org/c/openstack/project-config/+/873476 | 00:07 |
| Clark[m] | fungi: small thing wiht the first chngae in that stack | 00:15 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Farewell limestone https://review.opendev.org/c/openstack/project-config/+/873475 | 00:30 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Remove empty limestone nodepool providers https://review.opendev.org/c/openstack/project-config/+/873476 | 00:30 |
| fungi | oops, thanks! | 00:30 |
| fungi | and yeah, i spotted the git repo after i had deleted files that mentioned limestone in their name, forgot to go back and double-check those weren't related, readded the one now | 00:31 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit: increase ssh channel debugging https://review.opendev.org/c/opendev/system-config/+/873214 | 03:23 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit: send logs to syslog https://review.opendev.org/c/opendev/system-config/+/873480 | 04:29 |
| opendevreview | Ian Wienand proposed opendev/system-config master: logrotate: don't use filename to generate config file https://review.opendev.org/c/opendev/system-config/+/873481 | 04:29 |
| opendevreview | Ian Wienand proposed opendev/system-config master: logrotate: remove old log workaround https://review.opendev.org/c/opendev/system-config/+/873482 | 04:29 |
| *** yadnesh|away is now known as yadnesh | 04:31 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit: increase ssh channel debugging https://review.opendev.org/c/opendev/system-config/+/873214 | 04:47 |
| *** ysandeep is now known as ysandeep|lunch | 08:04 | |
| *** ralonsoh_ is now known as ralonsoh | 08:37 | |
| *** jpena|off is now known as jpena | 08:54 | |
| ianw | clarkb: something weird is going on | 09:16 |
| ianw | https://review.opendev.org/c/opendev/system-config/+/873214/9/playbooks/zuul/bootstrap-test-review.yaml sets " gerrit logging set-level DEBUG com.google.gerrit.sshd" | 09:17 |
| ianw | i can see in https://878773ae5f0379c9d548-8605f807f0b1cdc197533279c15edd0d.ssl.cf1.rackcdn.com/873214/9/check/system-config-run-review-3.6/2d81c9f/bridge99.opendev.org/ara-report/results/389.html this has applied | 09:18 |
| ianw | when we ssh, stuff should come out on stderr | 09:18 |
| ianw | and end up in https://878773ae5f0379c9d548-8605f807f0b1cdc197533279c15edd0d.ssl.cf1.rackcdn.com/873214/9/check/system-config-run-review-3.6/2d81c9f/review99.opendev.org/docker/gerrit-compose_gerrit_1.txt | 09:19 |
| ianw | we do not see anything | 09:19 |
| ianw | when i tried this quickly on review02, i saw all sorts of debug stuff coming out | 09:20 |
| ianw | it looks something like | 09:20 |
| ianw | [2023-02-13 03:31:07,893] [sshd-SshDaemon[4fc41cba](port=22)-nio2-thread-17] DEBUG org.apache.sshd.server.session.ServerSessionImpl : encode(ServerSessionImpl[null@/127.0.0.1:56142]) packet #6 sending command=60[60] len=300 | 09:21 |
| *** ysandeep|lunch is now known as ysandeep | 09:23 | |
| ianw | https://opendev.org/zuul/zuul-jobs/src/commit/ff1836691e00bc79a70d55f94e9aa5584f0c2c67/roles/collect-container-logs/tasks/main.yaml#L18 collects the logs; that doesn't seem like it could be missing anything | 09:32 |
| ianw | OOOOHHHHHHH | 09:37 |
| ianw | we stop it! | 09:37 |
| ianw | https://878773ae5f0379c9d548-8605f807f0b1cdc197533279c15edd0d.ssl.cf1.rackcdn.com/873214/9/check/system-config-run-review-3.6/2d81c9f/bridge99.opendev.org/ara-report/playbooks/7.html | 09:37 |
| ianw | when we rename repos | 09:38 |
| ianw | https://opendev.org/opendev/system-config/src/branch/master/playbooks/rename_repos.yaml#L8 | 09:38 |
| ianw | ... therefore, when we dump the logs at the end of the testing run -- that instance hasn't had debugging turned up. this makes sense, i thikn | 09:39 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gerrit: increase ssh channel debugging https://review.opendev.org/c/opendev/system-config/+/873214 | 09:42 |
| ianw | ^ that rebases it on 873480 -- which saves the gerrit container logs to a file on disk. that should capture both (and for upgrades, etc, all other stop/starts) | 09:43 |
| opendevreview | Merged openstack/project-config master: Update StarlingX docs promote job for R8 release https://review.opendev.org/c/openstack/project-config/+/873266 | 09:50 |
| *** soniya is now known as soniya|afk | 10:36 | |
| *** tweining_ is now known as tweining | 11:16 | |
| *** priteau_ is now known as priteau | 12:25 | |
| *** dasm|off is now known as dasm | 13:35 | |
| *** ysandeep is now known as ysandeep|out | 15:15 | |
| clarkb | fungi: the limestone project-config changes lgtm now. Did we put the limestone mirror in the emergency file so that base can run before these land? | 17:15 |
| clarkb | I'm guessing not since we still got emails early this morning relative to me | 17:15 |
| fungi | i hadn't, no, figured it wasn't urgent | 17:20 |
| clarkb | I mostly wanted to make sure we hadn't uncovered whatever the next issue is by doing so | 17:20 |
| fungi | and this way we know the mirror is still dead | 17:20 |
| clarkb | wfm and the stack lgtm too. Just need someone else to double check (though maybe that isn't super necessary for these changes) | 17:22 |
| *** yadnesh is now known as yadnesh|away | 17:52 | |
| clarkb | ianw: the gerrit changes still don't seem to collect the logs we expect to see (not in error_log, sshd_log or the containers log via syslog). Maybe we aren't doing ssh things after set level? | 17:54 |
| clarkb | infra-root: I've been thinking about replacing gitea servers recently. It looks like our haproxy configs for gitea and gerrit replication configs for gitea are separated from our gitea group membership. This means we should be able to deploy a new server and check that it configured gitea properly, then update the gerrit configuration to replicate to the server, trigger complete | 18:03 |
| clarkb | replication for that server, wait, then add the server to the haproxy config | 18:03 |
| clarkb | Then probably remove an old server before adding a second new server to keep total resource consumption down | 18:03 |
| clarkb | The last big questions in my head are do we want to continue to use boot from volume for the new servers? | 18:04 |
| clarkb | Oh and whether or not the current rough sizing is still working | 18:04 |
| clarkb | I think the current sizing is probably fine, just a matter of sorting out how we want to supply disks | 18:04 |
| *** jpena is now known as jpena|off | 18:05 | |
| *** jgwentworth is now known as melwitt | 18:33 | |
| fungi | what's the main driver for replacement? new distro version i guess? | 19:00 |
| clarkb | yes that | 19:01 |
| clarkb | I haven't been able to find a lot of time to devote to that so I'm trying to prioritize things. I think giteas, nameservers, and ehtherpad are high on the list if anyone is able/willing to help | 19:02 |
| clarkb | oh! we need the redirects | 19:21 |
| clarkb | yay for talking about this out loud :) What I'm not sure about is whether or not we need all of the other project data from the database or if we can just move the redirects over from that table in gitea | 19:22 |
| clarkb | almost makes me want to handle redirects outside of gitea... | 19:22 |
| clarkb | I think the process might be more like deploy new server (this will create empty repos but not redirects), sort out redirects, add server to gerrit replication, add server to haproxy | 19:23 |
| clarkb | I'm going to set up a test node with a hold since adding redirects to a test node is almost an identical problem to solve | 19:23 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/848181 has been rechecked and a hold is in place | 19:25 |
| *** dviroel is now known as dviroel|out | 19:26 | |
| fungi | managing redirects in gitea seems like it more gracefully handles the magic of git interactions rather than just the browseable ui (though presumably we could do both in apache configuration with some effort, but we didn't have them fronted by apache when we first started doing it) | 19:45 |
| ianw | clarkb: yeah this has me stumped. i see messages in error_log, but not on stderr | 19:51 |
| opendevreview | Ian Wienand proposed opendev/system-config master: logrotate: don't use filename to generate config file https://review.opendev.org/c/opendev/system-config/+/873481 | 20:03 |
| opendevreview | Ian Wienand proposed opendev/system-config master: logrotate: remove old log workaround https://review.opendev.org/c/opendev/system-config/+/873482 | 20:03 |
| *** Tengu7 is now known as Tengu | 20:19 | |
| clarkb | looking at the gitea99 database we do have a rename in there (woo testing) which shows that I don't think we can safely copy the db from one gitea to another to copy the redirects. Instead we need to construct the local owner ids and target repo ids against the current database. Its possible that dumping the entire db and using the source from an old server would work though as the | 20:50 |
| clarkb | owner ids and stuff would match. I just don't know if that would introduce other incompatibilities | 20:50 |
| clarkb | This has me thinking the best thing might be to dump the redirects from an existing server then write a script to lookup the corresponding values in the new server database and insert the correct rows? | 20:51 |
| ianw | since linaro is all turned off; i'll merge https://review.opendev.org/c/opendev/system-config/+/871672 and then remerge those limestone, etc. patches | 21:02 |
| clarkb | `SELECT user.id AS source_org_id, user.name AS source_org_name, repo_redirect.lower_name AS source_repo, repository.id AS target_repo_id, repository.lower_name AS target_repo_name FROM user INNER JOIN repo_redirect ON user.id = repo_redirect.owner_id INNER JOIN repository ON repo_redirect.redirect_repo_id = repository.id;` that seems to grab the info we need to reconstruct in another | 21:02 |
| clarkb | gitea install | 21:02 |
| ianw | although it's not saying they conflict | 21:03 |
| clarkb | basically we need to use source_org_name to determine the new gitea's source_org_id and target_repo_name to determine target_repo_id on the new gitea. Then we can insert into repo_redirect (new_org_id, source_repo, new_target_id) | 21:04 |
| clarkb | infra-root ^ do you think building it that way from either the old gitea db or our records is better than just trying to replace the gitea db on the new install with the gitea db from an old install? | 21:05 |
| clarkb | I'm mostly worried about things that may be different somehow through the intsallation processes. Like maybe hashes of files because they incorporate an id or something then when we change the ids around the hashes will be invalid? | 21:05 |
| clarkb | I kinda like the idea of building it from our records just because that was always the DR idea anyway | 21:06 |
| clarkb | and we can cross check that against what we get from an existing install too. But I don't want to do a bunch of that effort if we think it is overkill | 21:06 |
| fungi | ianw: thanks! i didn't do that one because i thought i remembered you having one up already | 21:07 |
| clarkb | one thing that will complicate this is the redirects table doesn't record a target org, just the target repo id. THis means that we may have to dedup or flatten the redirects ourselves? | 21:08 |
| clarkb | the held node's table definitely seems to have flattended things | 21:08 |
| ianw | clarkb: i have to admit this component of gitea configuration has completely passed me by. i'm guessing because i didn't help set it up. we mention copying the db in the docs, this is why? | 21:09 |
| ianw | (system-config docs)_ | 21:09 |
| clarkb | (on the test node we do this series of renames openstack/diskimage-builder -> opendev/diskimage-builder -> opendev/disk-image-builder, but we end up with two directs that both point at the final destination) | 21:09 |
| clarkb | ianw: ya the db copy is purely for the redirects | 21:09 |
| clarkb | but now that image handling is different I wonder if it is still safe. I guess we can just try it and see what happens too | 21:10 |
| fungi | ianw: if you're good with 873475 i'll progressively un-wip the others as that merges/deploys | 21:10 |
| clarkb | we also backup the gitea db only for the redirects. Everything else should be generateable from configs. THis makes me think investing some time in making the redirects work from config is a good idea too | 21:10 |
| ianw | fungi: yeah was just double checking on the pool quiestion | 21:11 |
| clarkb | I'm definitely not going to get around to testing either today though so we can think this through a bit | 21:12 |
| clarkb | I'll make note of it on the meeting agenda when I get around to writing that | 21:12 |
| clarkb | to generate it from our configs I think for each old/new name we query the gitea db for old_org id and then check if the new org/repo is present in repository. If so make note of the repository id and now we have enough info to insert to the redirects table. If not then skip until all renames are processed. At the end of that check our redirects and see if we've redirected multiple | 21:14 |
| clarkb | times and then flatten? | 21:14 |
| clarkb | should be straightforwardish, just a matter of interacting with their db out of band and hoping they don't change the schema on us | 21:14 |
| clarkb | ooh and we can grab a gitea db dump and check the scripting against that to ensure we don't miss anything etc | 21:15 |
| *** iurygregory_ is now known as iurygregory | 21:22 | |
| opendevreview | Ian Wienand proposed openstack/project-config master: Farewell limestone https://review.opendev.org/c/openstack/project-config/+/873475 | 21:22 |
| opendevreview | Ian Wienand proposed openstack/project-config master: Remove empty limestone nodepool providers https://review.opendev.org/c/openstack/project-config/+/873476 | 21:23 |
| ianw | ^ i just added the empty pool here, as that was what i did with linaro and it worked | 21:23 |
| fungi | oh, we do need an empty pool even though max-servers was already 0? | 21:24 |
| clarkb | ianw: typo in there. | 21:24 |
| clarkb | fungi: I suspect pool maybe a required nodepool configuration | 21:24 |
| fungi | oh, so nodepool itself breaks if no pool (even an empty one) is defined for a provider? | 21:24 |
| fungi | i guess we don't have functional testing of those configs | 21:25 |
| clarkb | maybe. I wasn't sure. I don't think we've done it before which was why I left a comment with that concern | 21:25 |
| opendevreview | Ian Wienand proposed openstack/project-config master: Farewell limestone https://review.opendev.org/c/openstack/project-config/+/873475 | 21:25 |
| opendevreview | Ian Wienand proposed openstack/project-config master: Remove empty limestone nodepool providers https://review.opendev.org/c/openstack/project-config/+/873476 | 21:25 |
| ianw | doh. i'm also not sure; i guess it passes the syntax check | 21:26 |
| clarkb | for pool in self.provider.get('pools', []): | 21:26 |
| clarkb | so maybe it isn't required | 21:26 |
| clarkb | anyway doesn't hurt to have an explicit empty pool | 21:26 |
| clarkb | I've approved that first change | 21:26 |
| clarkb | the second one should only land after the images are cleaed up | 21:27 |
| clarkb | (assuming that can happen gracefully) | 21:27 |
| ianw | ++ | 21:27 |
| clarkb | here's another gotcha for redirects: old orgs that are no longer in our config. For example osf got renamed to openinfra | 21:29 |
| clarkb | that may make it easier to just grab the db and copy that way | 21:30 |
| clarkb | we could create the org via the api as an alternative without repos though | 21:30 |
| clarkb | thinking out loud on that a bit more maybe our ansible for gitea should already be doing that (it doesn't today but we could add it) | 21:31 |
| clarkb | "here are the list of orgs you need to have even if they don't show up in a repo) | 21:31 |
| clarkb | I'm beginning to think that we should try just copying an old db to start since that is relatively quick to try. And we can stick it on a held node too first to see if there are any glaringly obvious issues | 21:38 |
| clarkb | the more I pull on the redirects the more I'm convinced doing it programmatically is possible but also that it would likely take a bit of time to get right | 21:39 |
| fungi | and could be fragile if they change how they store those | 21:49 |
| opendevreview | Merged openstack/project-config master: Farewell limestone https://review.opendev.org/c/openstack/project-config/+/873475 | 22:05 |
| opendevreview | Merged opendev/system-config master: Remove linaro-us cloud https://review.opendev.org/c/opendev/system-config/+/871672 | 22:15 |
| ianw | clarkb: as the official #opendev java developer ... am i reading https://gerrit.googlesource.com/gerrit/+/refs/tags/v3.6.3/java/com/google/gerrit/pgm/util/ErrorLogFile.java#71 right ... | 22:33 |
| ianw | ... it looks to me that it removes all logging appenders setup, and then sets up info logs to stderr, and looks for configs that then setup error_log and optionally json output? | 22:34 |
| fungi | he probably wishes shrews would come back and resume his post as the official java scapegoat | 22:34 |
| ianw | it seems to me gerrits starts up and looks at https://gerrit.googlesource.com/gerrit/+/refs/tags/v3.6.3/resources/log4j.properties | 22:35 |
| ianw | and then at some early point, removes that appender setup there and puts in the new one in initLogSystem() | 22:36 |
| clarkb | ianw: at first glance that looks right. It isn't clear to me how that text log there becomes the error_log file though | 22:36 |
| ianw | LOG_NAME @ https://gerrit.googlesource.com/gerrit/+/refs/tags/v3.6.3/java/com/google/gerrit/pgm/util/ErrorLogFile.java#93 is error_log, so that's where it starts adding things to errorlog | 22:37 |
| clarkb | aha | 22:37 |
| clarkb | I guess it is possibly running that setup after you've configured the ssh loggers nuking them? | 22:37 |
| ianw | i don't think it changes the level, but it might explain why some things we see on the stderr output and some things in error_log | 22:38 |
| clarkb | it almost looks like they are twodifferent streams though and not one a superset of another due to log level | 22:39 |
| clarkb | but maybe they are a super/sub set of each other and I just didn't compare accurately | 22:40 |
| ianw | i'm starting to wonder if public void channelInitialized(Channel channel) { | 22:43 |
| ianw | in https://gerrit-review.googlesource.com/c/gerrit/+/357694/6/java/com/google/gerrit/sshd/ChannelIdTrackingUnknownChannelReferenceHandler.java is actually ever being called | 22:43 |
| clarkb | ianw: did we try a patchset where we log at error level or whatever? | 22:44 |
| clarkb | so that we're not relying on set-level? | 22:44 |
| ianw | i did, still didn't see anything | 22:44 |
| ianw | it doesn't help that none of the level names match up | 22:44 |
| clarkb | heh ya thats a really fun design choice in java land | 22:45 |
| clarkb | took a bit of googling to figure out what the levels are. | 22:45 |
| clarkb | My hunch if logging at error/higher level isn't showing up is that that call isn't being made | 22:45 |
| clarkb | ianw: terrible idea but probably easier than attaching a debugger: add an assert False equivalent to the methods | 22:46 |
| clarkb | then see if that raises exceptions indicating they are called | 22:46 |
| ianw | that is a good idea | 22:47 |
| ianw | https://95663730722e429c421d-5cb3754a5639f7be0289e9ab84888d65.ssl.cf2.rackcdn.com/873214/10/check/system-config-run-review-3.6/290079b/review99.opendev.org/logs/error_log | 22:47 |
| ianw | has | 22:47 |
| ianw | [2023-02-13T10:40:36.902Z] [sshd-SshDaemon[3d2be0df](port=22)-nio2-thread-1] DEBUG com.google.gerrit.sshd.CachingPublicKeyAuthenticator : authenticate(admin@ServerSessionImpl[null@/127.0.0.1:53854]) use cached result=true for ssh-ed25519 key=SHA256:qy4xoVw4bvmgIt3WABlTI3zXMMsNhphibvW5DQ0h1fo | 22:47 |
| ianw | i.e. it showed a debug statement from com.google.gerrit.sshd | 22:47 |
| clarkb | oh ya that would certainly lend more evidence to the theory it isn't being called at all then | 22:48 |
| ianw | *but*, on review02, when i turned that up | 22:49 |
| ianw | i see that in the stderr output of "docker logs"! | 22:49 |
| *** dasm is now known as dasm|off | 22:54 | |
| ianw | In order to use it, the handler instance needs to be registered as **both** an `UnknownChannelReferenceHandler` and a `ChannelListener`. | 22:56 |
| ianw | https://github.com/apache/mina-sshd/commit/11b33dee37b5b9c71a40a8a98a42007e3687131e | 22:57 |
| ianw | does https://gerrit-review.googlesource.com/c/gerrit/+/238384/9/java/com/google/gerrit/sshd/SshDaemon.java register it as both? | 22:57 |
| clarkb | well we know that enabling it seems to side effect something that disabling it works around? | 22:59 |
| ianw | well perhaps it's "half" enabled | 23:00 |
| clarkb | which would imply that whatever they've done in the original change was sufficient. That said I don't see it being added as a channellistener in that file | 23:00 |
| ianw | in https://gerrit-review.googlesource.com/c/gerrit/+/238384/9/java/com/google/gerrit/sshd/ChannelIdTrackingUnknownChannelReferenceHandler.java | 23:00 |
| ianw | handleUnknownChannelCommand gets called -- we know that because we saw the error | 23:00 |
| ianw | but public void channelInitialized never gets called, because it's not registered as a "regular" handler? | 23:01 |
| clarkb | ooh that could be | 23:01 |
| clarkb | it implements the ChannelListener interface but then isn't registered as one? | 23:01 |
| clarkb | and you probably want to do that at initUnknownChannelReferenceHandler() because I'm not sure if it makes sense on its own | 23:02 |
| ianw | addChannelListener() doesn't appear in the gerrit source code afaics | 23:05 |
| clarkb | I'm trying to understand how the setUnknownChannelReferenceHandler is a valid call in that context | 23:05 |
| clarkb | I guess SshServer may include it? | 23:06 |
| clarkb | ah yup thats it | 23:06 |
| clarkb | ianw: addChannelListener is also valid on SshServer so I think you can just call it in initUnknownChannelReferenceHandler() if using the channel tracking | 23:09 |
| clarkb | I don't think it makes sense if channel tracking is disabled so that is agood place for it ? | 23:09 |
| fungi | clarkb: 873476 should be ready to go now. the parent has deployed and i've checked nodepool image-list for any remaining entries indicating that provider | 23:09 |
| clarkb | fungi: cool do you want me to approve it? | 23:10 |
| fungi | feel free. i'm around, you just hadn't +2'd teh new revision | 23:11 |
| clarkb | oh I expected my votes to carry over. Let me see | 23:14 |
| clarkb | the parent changed which I guess was enough to do it? | 23:14 |
| clarkb | everything else is the same. I didn't expect that. | 23:14 |
| fungi | part of the deletion moved into the parent change, so that removed some of the child change | 23:15 |
| clarkb | aha and its looking at the diffs being the same not just the end result | 23:15 |
| clarkb | approved | 23:15 |
| fungi | it was a net zero change, but what's being depeted move out of one change an into another | 23:15 |
| fungi | s/depeted/deleted/ | 23:15 |
| fungi | so wasn't simply a rebase | 23:16 |
| fungi | also 871672 failed in deploy, presumably because of what 873428 will fix | 23:17 |
| opendevreview | Merged openstack/project-config master: Remove empty limestone nodepool providers https://review.opendev.org/c/openstack/project-config/+/873476 | 23:25 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] gerrit: increase ssh channel debugging https://review.opendev.org/c/opendev/system-config/+/873214 | 23:31 |
| clarkb | Tomorrow's the last day for service coordinator nominations. I guess I'm it again? I'll wait util after tomorrow's meeting to make it official | 23:37 |
| clarkb | fungi and I will be out for next week's meeting too so I'm proposing we skip it on the agenda for tomorrows 'meeting | 23:41 |
| fungi | oh, good point | 23:41 |
| clarkb | ianw: did the mm3 AutoField change land? | 23:42 |
| ianw | i don't think so. they did merge a "fix" upstream too | 23:42 |
| clarkb | not yet /me sticks that in the agenda | 23:42 |
| ianw | they were specifying it in the settings.py, but we're using the one based on the docker build | 23:42 |
| clarkb | ah so maybe we need to merge those together? | 23:43 |
| clarkb | though sticking close to upstream docker is a good idea just for sanity reasons I suspect | 23:43 |
| ianw | i dunno. everyone could drop it from their settings.py now that it's in the app definition | 23:46 |
| clarkb | but only if they've released the fixed version and we are on the fixed version? I suspect we need to upgrade os we'll get there eventually | 23:46 |
| opendevreview | Merged openstack/project-config master: Remove add_master_python3_jobs.sh https://review.opendev.org/c/openstack/project-config/+/873016 | 23:48 |
| fungi | ianw: https://review.opendev.org/873428 is ready to go now the prior cleanup has merged | 23:49 |
| clarkb | ok I've done a first pass at an updated agenda. Anything else we'd like to add? If zuul was still having some struggles we could go over those but they should mostly be addressed with the weekend restart | 23:50 |
| fungi | and that should fix the base deploy | 23:50 |
| ianw | fungi: is that really in merge conflict? | 23:51 |
| fungi | oh, maybe it is since the linaro cleanup and i didn't spot it | 23:52 |
| fungi | i can rebase that series | 23:52 |
| ianw | it is the linaro stuff -- i don't mind rebasing as i caused it, lmn :) | 23:52 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Farewell limestone https://review.opendev.org/c/opendev/system-config/+/873428 | 23:57 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Finish cleaning up packethost references https://review.opendev.org/c/opendev/system-config/+/873429 | 23:57 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Final cleanup of internap/inap/iweb references https://review.opendev.org/c/opendev/system-config/+/873430 | 23:57 |
| fungi | ianw: no problem, was easy conflict to resolve, just delete things | 23:57 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!