| *** odyssey4me is now known as Guest945 | 01:22 | |
| *** odyssey4me is now known as Guest955 | 01:53 | |
| opendevreview | Merged opendev/system-config master: Remove paste01.openstack.org https://review.opendev.org/c/opendev/system-config/+/800879 | 03:03 |
|---|---|---|
| opendevreview | Merged opendev/system-config master: Enable openEuler mirroring https://review.opendev.org/c/opendev/system-config/+/784874 | 04:26 |
| *** ykarel|away is now known as ykarel | 04:31 | |
| ianw | i'm running ^ in a screen | 04:39 |
| *** chandankumar is now known as chkumar|rover | 05:21 | |
| *** iurygregory_ is now known as iurygregory | 06:38 | |
| *** amoralej|away is now known as amoralej | 07:02 | |
| *** rpittau|afk is now known as rpittau | 07:31 | |
| *** dmellado_ is now known as dmellado | 07:41 | |
| opendevreview | Vishal Manchanda proposed openstack/project-config master: Retire django-openstack-auth https://review.opendev.org/c/openstack/project-config/+/800459 | 08:07 |
| *** bhagyashris_ is now known as bhagyashris|ruck | 08:24 | |
| *** ykarel is now known as ykarel|lunch | 09:06 | |
| *** odyssey4me is now known as Guest1011 | 09:20 | |
| opendevreview | Matthias Runge proposed openstack/project-config master: Enable deleting left-overs of panko deprecation https://review.opendev.org/c/openstack/project-config/+/800241 | 09:56 |
| *** odyssey4me is now known as Guest1025 | 10:06 | |
| *** ykarel|lunch is now known as ykarel | 11:11 | |
| *** dviroel|out is now known as dviroel | 11:19 | |
| *** amoralej is now known as amoralej|lunch | 12:05 | |
| *** marios_ is now known as marios | 12:40 | |
| *** marios is now known as marios|call | 13:10 | |
| *** amoralej|lunch is now known as amoralej | 13:13 | |
| opendevreview | Ananya Banerjee proposed opendev/elastic-recheck master: Run elastic-recheck container https://review.opendev.org/c/opendev/elastic-recheck/+/729623 | 13:14 |
| opendevreview | Ananya Banerjee proposed opendev/elastic-recheck master: Run elastic-recheck container https://review.opendev.org/c/opendev/elastic-recheck/+/729623 | 13:20 |
| *** marios|call is now known as marios | 13:34 | |
| opendevreview | Tristan Cacqueray proposed openstack/project-config master: gerritbot: add matrix configuration https://review.opendev.org/c/openstack/project-config/+/801063 | 13:52 |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Run matrix-gerritbot on eavesdrop https://review.opendev.org/c/opendev/system-config/+/800506 | 13:57 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: tripleo-common-tempest-plugin - Step 2: End project Gating https://review.opendev.org/c/openstack/project-config/+/800154 | 13:59 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: tripleo-common-tempest-plugin - Step 4: Remove Project https://review.opendev.org/c/openstack/project-config/+/800157 | 13:59 |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Run matrix-gerritbot on eavesdrop https://review.opendev.org/c/opendev/system-config/+/800506 | 14:01 |
| *** rpittau is now known as rpittau|afk | 14:08 | |
| *** ykarel is now known as ykarel|away | 14:46 | |
| opendevreview | Tristan Cacqueray proposed openstack/project-config master: gerritbot: add matrix configuration https://review.opendev.org/c/openstack/project-config/+/801063 | 14:59 |
| opendevreview | Tristan Cacqueray proposed openstack/project-config master: gerritbot: add matrix configuration https://review.opendev.org/c/openstack/project-config/+/801063 | 15:22 |
| opendevreview | Tristan Cacqueray proposed openstack/project-config master: gerritbot: add matrix configuration https://review.opendev.org/c/openstack/project-config/+/801063 | 15:33 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Push a patch to try and prevent gerrit openid deletion https://review.opendev.org/c/opendev/system-config/+/800832 | 15:59 |
| clarkb | That is an update to get in sync with the current code upstream as well as add useful information to the commit message. I won't be trying to hold that and replace the currently held node with it since the code change is almost nil | 16:00 |
| opendevreview | Takashi Kajinami proposed openstack/diskimage-builder master: Replace deprecated import of ABCs from collections https://review.opendev.org/c/openstack/diskimage-builder/+/801092 | 16:02 |
| corvus | clarkb: zuul master has a sql migration we haven't run in opendev; it's a column drop/add on the build table. i think the last time we did something to the build table, it took about 30m for the migration to run. | 16:27 |
| corvus | clarkb: maybe we want to think about doing the zuul restart very late today or even tomorrow (i'm happy to do that). or we could think about trimming the build table. | 16:28 |
| clarkb | oh ya I remember we wondered why it was taking so long to start up | 16:29 |
| clarkb | corvus: by trimming you mean drop older rows ya? | 16:29 |
| corvus | we should at least have log messages telling us why it's taking so long this time :) | 16:30 |
| corvus | clarkb: yeah | 16:30 |
| corvus | of course, we'll need to trim some other tables too, so it's a little involved, but we probably have time to prepare those statements. | 16:30 |
| corvus | (we'll need to clean up the related artifact and buildset entries) | 16:31 |
| clarkb | I'm not sure I have a great idea as to which would be preferable at this point | 16:32 |
| clarkb | queues are not small right now but they also aren't huge | 16:32 |
| clarkb | corvus: do we maybe want to try this afternoon and just let the half hour migration happen? | 16:32 |
| corvus | that's sort of how i'm leaning, yes | 16:33 |
| corvus | i'd rather do the cleanup by making a tool or setting in zuul (rather than doing it now by hand) | 16:33 |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Run matrix-gerritbot on eavesdrop https://review.opendev.org/c/opendev/system-config/+/800506 | 16:34 |
| clarkb | I'm thinking that since we don't have a large backlog and it is friday we are unlikely to see queues grow | 16:34 |
| clarkb | and maybe just doing the restart as planned is easiest (but do it this afternoon as we let some backlog get handled) | 16:34 |
| corvus | clarkb: sounds good | 16:34 |
| clarkb | corvus: any concern for being able to rollback after the migration if we need to revert zuul changes? | 16:34 |
| clarkb | I guess as long as it isn't the db migration that needs to be rolled back we are fine | 16:35 |
| *** marios is now known as marios|out | 16:35 | |
| corvus | clarkb: yeah, we won't be able to drop back to the most recent tag; we'd have to revert a commit and make a new image. | 16:36 |
| clarkb | ok, just want to make sure that is understood, but seems like we have an out | 16:37 |
| opendevreview | Vishal Manchanda proposed openstack/project-config master: Retire django-openstack-auth https://review.opendev.org/c/openstack/project-config/+/800459 | 16:43 |
| opendevreview | Vishal Manchanda proposed openstack/project-config master: Retire django-openstack-auth https://review.opendev.org/c/openstack/project-config/+/800532 | 16:43 |
| *** amoralej is now known as amoralej|off | 16:45 | |
| clarkb | google followed up on my gerrit pushing issues and said it was likely an issue in their replication system | 17:00 |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Run matrix-gerritbot on eavesdrop https://review.opendev.org/c/opendev/system-config/+/800506 | 18:29 |
| corvus | clarkb: how you feeling about a restart soonish? | 20:21 |
| clarkb | corvus: let me look at the zuul satus | 20:29 |
| clarkb | oh ya queues are just about empty. I'm up for it if you are | 20:29 |
| clarkb | corvus: I let the openstack release team know and they don't seem to have any jobs running at the moment | 20:33 |
| corvus | cool, i'll open up some windows... | 20:34 |
| corvus | okay, i ran the pull playbook for good measure | 20:35 |
| corvus | i'll go ahead and restart? | 20:35 |
| clarkb | corvus: did we double check that the image published properly? | 20:35 |
| clarkb | https://zuul.opendev.org/t/zuul/build/501daeaae4d141adbd2ff80b6690921f was successful at least thats probably good enough? or do you want to check dockerhub? | 20:36 |
| corvus | clarkb: i checked that https://review.opendev.org/800112 had a green promote job | 20:36 |
| corvus | that's the extent of my checking, but i think it's adequate | 20:36 |
| clarkb | ok then ya I think we go ahead | 20:36 |
| corvus | restarting now | 20:37 |
| corvus | i think maybe while we're waiting, i'll manually install the mysqlclient package on the host? | 20:41 |
| corvus | we don't seem to have it handy | 20:41 |
| clarkb | fine with me | 20:41 |
| corvus | ok, i see "ALTER TABLE zuulbuild DROP COLUMN nodename" running | 20:43 |
| corvus | on to 'ALTER TABLE zuul_build ADD COLUMN nodeset VARCHAR(255)' | 20:53 |
| clarkb | progress :) | 20:53 |
| clarkb | the debug log shows it doing stuff now I think? | 20:55 |
| corvus | i think that's just queing up events | 20:55 |
| clarkb | maybe that is just gerrit event stream being handled. Ya | 20:55 |
| clarkb | I think it just moved on to the next step | 21:00 |
| corvus | it's done | 21:00 |
| clarkb | about 23 minutes? | 21:01 |
| corvus | that's my read | 21:01 |
| corvus | re-enqueing | 21:10 |
| clarkb | corvus: one thing I notice is that it had some errors saying unable to load build event. I suspect those are leftover from when we stopped it? | 21:10 |
| clarkb | and we had since lost the state in the scheduler to process those? | 21:11 |
| corvus | i didn't see those | 21:11 |
| clarkb | they came in right after the configs were loaded | 21:11 |
| clarkb | 2021-07-16 21:09:46,102 DEBUG zuul.Scheduler: [build: 572bdefa2a594f0f891b09afc5544ee1] Processing result event <zuul.model.BuildCompletedEvent object at 0x7f6ca7f0ff70> | 21:11 |
| clarkb | then | 21:12 |
| clarkb | 2021-07-16 21:09:46,102 ERROR zuul.Scheduler: Unable to find build 572bdefa2a594f0f891b09afc5544ee1 | 21:12 |
| corvus | clarkb: yep, that makes sense; i think they're harmless | 21:13 |
| clarkb | ok | 21:13 |
| corvus | completion events from executors shutting down after the scheduler was shut down | 21:13 |
| clarkb | ah | 21:13 |
| corvus | re-enqueue finished | 21:13 |
| clarkb | fwiw the pep8 job at the top of the check pipeline seems to be running happily so far | 21:13 |
| corvus | #status log restarted all of zuul on commit 43b7f7f22c74301e830b042357536e6b5357d6e8 | 21:13 |
| opendevstatus | corvus: finished logging | 21:14 |
| corvus | the builds and buildsets pages are less useful right now | 21:14 |
| corvus | i think we need to filter out the in-progress builds | 21:14 |
| clarkb | ++ | 21:15 |
| corvus | but they are functioning | 21:15 |
| clarkb | and that pep8 job just succeeded | 21:16 |
| corvus | on the status page, you can click on completed builds and go right to the build page now | 21:17 |
| clarkb | there is a py38 failure and looking at that job it failed due to job run content (so thats a proper fail) | 21:17 |
| corvus | meanwhile, the buildset page doesn't display the in-progress builds, but it probably should :) | 21:18 |
| corvus | so we got some things backwards :) | 21:18 |
| corvus | (sorry i mean the specific buildset page, not the list of buildsets) | 21:18 |
| clarkb | ah ok I was going to say I think the buildest list page is showing in progress items | 21:19 |
| mordred | it's certainly showing some buildsets that are less than exciting :) | 21:20 |
| mordred | or - sorry - differently exciting | 21:21 |
| clarkb | 798918,4 may be our first merge post restart | 21:22 |
| clarkb | in the openstack tenant | 21:22 |
| corvus | mordred: yeah i'm looking at https://zuul.opendev.org/t/openstack/buildset/4293ae47ea2046e083b6f2cbf8ec2822 which is basically having builds added to it as they complete, but i think we should have it show the in-progress builds too | 21:23 |
| corvus | meanwhile, https://zuul.opendev.org/t/openstack/buildsets probably shouldn't show in-progress by default | 21:23 |
| corvus | (but having an option to see them would be nice) | 21:23 |
| mordred | ++ | 21:23 |
| mordred | agree with both | 21:23 |
| mordred | on https://zuul.opendev.org/t/openstack/builds - what is the yellow ! telling me? | 21:24 |
| corvus | mordred: that we haven't written code to handle builds without results yet, so it's assumed to be some sort of anomalous result | 21:24 |
| clarkb | https://review.opendev.org/c/starlingx/ansible-playbooks/+/798918/ did merge | 21:24 |
| clarkb | and so far spot checking failures shows failures due to what ran in the job and not the zuul overlord | 21:26 |
| * corvus fires up js dev env | 21:27 | |
| clarkb | everything continues to look good. I'm going to take a break. I should be able to review any changes on the js side that come out of this | 21:35 |
| clarkb | in theory we only need to update zuul-web to pick those up too ya? | 21:35 |
| corvus | clarkb: yep | 21:35 |
| johnsom | I have a patch that has two Depends-On, I can see it checks out the correct SHA, but devstack seems to still go download and install the packages. Is there something I should check off the top of your heads? https://review.opendev.org/c/openstack/designate/+/800727 | 23:09 |
| johnsom | Direct link to the proper check out: https://zuul.opendev.org/t/openstack/build/dc7b51cbf2bd4338aa8c4f8f7cef9a71/log/job-output.txt#600 | 23:10 |
| clarkb | johnsom: does your job not disable git cloning and set error on clone? | 23:11 |
| johnsom | clarkb I don't know, it's one of the main tempest jobs for designate. | 23:12 |
| clarkb | error_on_clone is set to true | 23:12 |
| clarkb | https://zuul.opendev.org/t/openstack/build/dc7b51cbf2bd4338aa8c4f8f7cef9a71/log/controller/logs/local_conf.txt#40 that is the flag in devstack that is supposed ot prevent devstack from changing what zuul checks out | 23:13 |
| clarkb | though I guess that is only for the clone step not the checkout step. I think histoircally those two were one and the same but maybe devstack split them? | 23:13 |
| johnsom | It just seems like it's not installing out of the zuul checkout location. I'm not sure if there is a requirements re-write that should happen, or something installing it before the other parts of devstack run, or ..... | 23:15 |
| clarkb | johnsom: are you sure the problem is that it is checking out a different sha after the fact? I think it may be installing the lib from pypi instead of from source instead | 23:15 |
| clarkb | you need to set libs from git (I forget the exact form that takes in devstack) to indicate those libraries shouldbe installed fromsource and not pypi | 23:16 |
| johnsom | What I want is to use the version zuul checked out due to the depends-on. What I see is the checkout is right, but devstack still pulls the package from pypi | 23:16 |
| clarkb | ya I'm caught up now. So way back when it was decided that openstack needed to test with released versions of libs because that is what people would actually install and use rather than doing everything from source | 23:17 |
| johnsom | Ah, ok. That is probably missing here. It's so far between when I ever need to pull in special oslo versions I forget the details. | 23:17 |
| clarkb | when that change was made all your oslo libs and python clients and so on started to be installed by default from pypi. There is an escape hatch for that called libs from git that allows you to specify that some library should be installed from source instead. You need to toggle that | 23:17 |
| johnsom | Thanks! | 23:17 |
| clarkb | one thing to keep in mind is that you probably don't want to set libs from git for those repos permanently. Just do it to test that depends on, then go land the updates to the oslo libs and tag them in new releases. Then bump requirements then your applications can make use of them | 23:21 |
| johnsom | Right, this is making sure I'm doing the right things in these olso patches, so just a one-or-so-off | 23:21 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!