| *** rlandy is now known as rlandy|out | 00:36 | |
| *** mazzy509881292958085946 is now known as mazzy50988129295808594 | 02:08 | |
| *** mazzy509881292958085945 is now known as mazzy50988129295808594 | 02:26 | |
| *** soniya29 is now known as soniya29|rover | 03:41 | |
| *** ysandeep|out is now known as ysandeep | 04:26 | |
| *** marios is now known as marios|ruck | 05:06 | |
| frickler | oh, fun, new setuptools have reached our docs builds https://zuul.opendev.org/t/openstack/build/9110191bfcd049978e019bb2867a2eb5 | 05:50 |
|---|---|---|
| *** arxcruz is now known as arxcruz|out | 07:15 | |
| *** jpena|off is now known as jpena | 07:37 | |
| *** ysandeep is now known as ysandeep|lunch | 07:59 | |
| *** pojadhav is now known as pojadhav|afk | 08:08 | |
| *** pojadhav|afk is now known as pojadhav | 08:39 | |
| *** ysandeep|lunch is now known as ysandeep | 08:58 | |
| *** soniya29|rover is now known as soniya29|rover|lunch | 09:05 | |
| ChayanP | Hi everyone, I am new to open source and can you please suggest some beginner friendly issues | 09:57 |
| ChayanP | to contribute to | 09:57 |
| *** marios|ruck is now known as marios|ruck|lunch | 10:01 | |
| *** soniya29|rover|lunch is now known as soniya29|rover | 10:07 | |
| *** ysandeep is now known as ysandeep|afk | 10:09 | |
| *** rlandy|out is now known as rlandy | 10:22 | |
| *** marios|ruck|lunch is now known as marios|ruck | 10:30 | |
| fungi | ChayanP: that will depend a lot on what you want to work on | 10:35 |
| ChayanP | I was hoping todo some programming related work | 10:37 |
| fungi | but on what sort of project? is there any in particular that you use, for example? you'll usually have more luck trying to fix bugs in software you already use | 10:39 |
| ChayanP | I have not used open stack before but was hoping to learn more cloud infra I do not have any specific project in mind, but would be willing to learn the project | 10:55 |
| fungi | if it's openstack you're interested in, they have a landing page for interested contributors here: https://www.openstack.org/community/ | 10:57 |
| fungi | there's also a general openstack contributor guide for code and documentation contributions here: https://docs.openstack.org/contributors/code-and-documentation/ | 10:59 |
| *** soniya29|rover is now known as soniya29|rover|brb | 11:09 | |
| *** dviroel|out is now known as dviroel | 11:17 | |
| *** ysandeep|afk is now known as ysandeep | 11:18 | |
| *** soniya29|rover|brb is now known as soniya29|rover | 11:42 | |
| mgagne | fungi: can iweb cloud be disabled? it's not going offline right now as previously discussed but we have some issues that will impact the API so it might be better to disabled completely for now. | 11:44 |
| fungi | mgagne: we were hoping to keep it up through the next few hours until the release is finished. mass tag pushing is underway as i type this | 11:49 |
| fungi | but if it's not usable then sure, it's better we remove it from our configuration | 11:49 |
| mgagne | fungi: there are unfortunately issues outside our control right now. | 11:51 |
| mgagne | I can try to give you a workaround, issue is with iweb.com domain/DNS. | 11:51 |
| fungi | mgagne: it looks like the release activity, while resulting in a lot of builds, is going quickly and not consuming as much of our available quota as i expected, so taking iweb out of the mix shouldn't have any impact i don't think | 11:51 |
| fungi | i'll push up a change to disable booting nodes there for now | 11:52 |
| mgagne | anything already booted should work | 11:52 |
| mgagne | but might not be able to boot new nodes since API won't be reachable | 11:53 |
| mgagne | Or if you can "hack" the thing: | 11:53 |
| mgagne | 174.142.3.179 network.api.mtl01.cloud.iweb.com | 11:53 |
| mgagne | 70.38.104.65 image.api.mtl01.cloud.iweb.com | 11:53 |
| mgagne | 174.142.3.176 compute.api.mtl01.cloud.iweb.com | 11:53 |
| mgagne | 174.142.3.177 volume.api.mtl01.cloud.iweb.com | 11:53 |
| mgagne | 184.107.26.141 identity.api.cloud.iweb.com | 11:53 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Temporarily disable launching in iWeb https://review.opendev.org/c/openstack/project-config/+/835822 | 11:55 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Revert "Temporarily disable launching in iWeb" https://review.opendev.org/c/openstack/project-config/+/835823 | 11:55 |
| fungi | mgagne: no worries, i'll emergency approve 835822 and wip 835823 until we hear from you | 11:55 |
| mgagne | fungi: thanks, sorry about that. it's unfortunately currently outside our control. | 11:56 |
| fungi | no need to apologize! we appreciate the help and the heads-up | 11:57 |
| mgagne | fungi: issue seems to be fixed or was fixed a couple minutes ago. there might still be some hiccups but I think DNS TTL was less than 30 minutes so things should recover relatively fast. | 12:10 |
| fungi | mgagne: ahh, okay, so should i cancel the removal? | 12:10 |
| mgagne | if you can reach the APIs from your systems, IMO, it should be fine | 12:11 |
| fungi | yep, looks like they're resolving correctly for us. i'll abandon the removal change and its revert | 12:12 |
| fungi | thanks again! | 12:12 |
| mgagne | awesome | 12:15 |
| *** ysandeep is now known as ysandeep|afk | 12:20 | |
| fungi | clarkb: any idea if the current situation with logstash being broken could account for jobs spending a couple of minutes on "TASK [submit-logstash-jobs : Submit logstash processing jobs to log processors]"? | 12:20 |
| fungi | unfortunately, since the releasenotes jobs are intentionally serialized, that's contributing to very slow burndown in the tag pipeline | 12:33 |
| fungi | fortunately, we probably don't need to hold up the final release announcement waiting for them to complete | 12:34 |
| *** ysandeep|afk is now known as ysandeep | 12:58 | |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Drop tap-as-a-service tarballs site redirect https://review.opendev.org/c/opendev/system-config/+/835829 | 12:59 |
| Clark[m] | fungi: looks like the logstash gearman queue is quite large. I suspect that makes it slow to submit new jobs. I think you can restart the geard on logstash.o.o to reset the queue to improve that. We'll lose the backlog but we're aiming to shut it all down soon anyway | 13:29 |
| fungi | thanks! i'll give that a shot | 13:29 |
| fungi | and done. i wasn't sure if geard held that in memory or persisted it to disk across restarts | 13:30 |
| fungi | hopefully that speeds things up | 13:31 |
| Clark[m] | It is in memory | 13:31 |
| fungi | the other thing i'm noticing is that, with the releasenotes jobs serialized via semaphore, there seems to be a several minute delay between when one completes and when the next one starts. i guess that's zuul waiting to issue the node request until the semaphore is freed up | 13:32 |
| fungi | also once you're at your computer, 835829 cleans up something we noticed in cross-checking the release | 13:33 |
| Clark[m] | Looks like it is failing on the setuptools issue. I wonder if that impacts my planned gitea upgrade for later today. We can land a change similar to tripleo's workarounds to address that. | 13:36 |
| Clark[m] | fungi: are changes like that important enough to deviate from typical morning routine? Looks like the release in general is ok? | 13:37 |
| Clark[m] | Just trying to figure out if I'm taking the kids to school this morning or opening the laptop early :) | 13:42 |
| fungi | Clark[m]: not urgent, no | 13:45 |
| marios|ruck | o/ folks can someone please check this when you have some time https://review.opendev.org/c/openstack/project-config/+/834199 (updates ops for our oftc chan) thank you | 13:50 |
| fungi | marios|ruck: approved, sorry we missed that one | 13:55 |
| dpawlik | hey fungi. So according to your comment https://review.opendev.org/c/opendev/system-config/+/833264 you suggest to put the information into this document: https://docs.openstack.org/infra/ am I right? | 13:57 |
| fungi | marios|ruck: i suspect the removals will still need to be done by one of the people in your admins list talking to chanserv, since i think accessbot avoids touching entries which aren't in its configuration. if they're still in there after the new additions show up, i can help rlandy or chandankumar out with the necessary syntax | 13:59 |
| fungi | dpawlik: into https://governance.openstack.org/sigs/tact-sig.html which can be updated through edits to https://governance.openstack.org/sigs/tact-sig.html | 14:01 |
| fungi | er, through edits to https://opendev.org/openstack/governance-sigs/src/branch/master/doc/source/tact-sig.rst | 14:01 |
| opendevreview | Merged openstack/project-config master: Update channel ops for oooq (tripleo ci) channel https://review.opendev.org/c/openstack/project-config/+/834199 | 14:04 |
| dpawlik | fungi: ack. Will add it there | 14:15 |
| dpawlik | fungi: just FYI, we added more log files to be scraped, looks so far ok. Also good that the instance has been resized to flavor with more vcpus - | 14:17 |
| dpawlik | https://prometheus.monitoring.softwarefactory-project.io/grafana/d/EtRMbV1Zk/node-exporter-full?orgId=1&var-job=node&var-name=logscraper01.openstack.org.novalocal&var-node=logscraper01.openstack.org&var-port=9100 | 14:17 |
| marios|ruck | fungi: ok thank you noting to check that | 14:20 |
| *** pojadhav is now known as pojadhav|afk | 14:31 | |
| *** ysandeep is now known as ysandeep|out | 14:37 | |
| fungi | marios|ruck: looks like it got applied, if you want to double-check the access list chanserv has for it | 14:51 |
| marios|ruck | thank you fungi | 14:57 |
| *** marios|ruck is now known as marios | 15:00 | |
| clarkb | yes the setuptools + tox + docs/linters situation in system-config will prevent the gitea upgrade from landing. | 15:03 |
| clarkb | fungi: ^ any idea if we hvae a workaround change yet? If not I guess I'll push one for packages =? | 15:03 |
| *** dviroel is now known as dviroel_ | 15:04 | |
| *** dviroel_ is now known as dviroel | 15:04 | |
| fungi | i'm not aware of a workaround change yet, will check gerrit just to be sure | 15:04 |
| fungi | nothing for "is:open project:opendev/system-config file:setup.py" | 15:05 |
| fungi | so unless there's a different workaround, i think it's not been proposed yet | 15:05 |
| clarkb | ok working on it | 15:06 |
| marios | fungi: looks like you're right it has kept the previous entries (chanserv) | 15:06 |
| fungi | yeah, any of the folks with admin permission for that channel should be able to remove the old entries | 15:07 |
| marios | fungi: ack thank you will dig more into it | 15:07 |
| fungi | or i can help with it, but my dance card is already pretty full on release day | 15:07 |
| marios | fungi: ack :) thanks for help | 15:07 |
| *** marios is now known as marios|ruck | 15:10 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Workaround setuptools 61.0.0 package auto discovery problems https://review.opendev.org/c/opendev/system-config/+/835849 | 15:11 |
| clarkb | I think ^ may do what we want | 15:11 |
| fungi | thanks! | 15:11 |
| clarkb | fungi: any idea if the logstash stuff sped up after the geard restart? | 15:12 |
| fungi | i'll have to check. does ansible give us a per-task duration by any chance? | 15:13 |
| fungi | though if it's a significant speedup i can probably tell just by watching the log stream for the current build | 15:13 |
| fungi | yeah, looks like it helped! | 15:14 |
| clarkb | I think ansible gives yo ua timestamp when each task completes so yo ucan take a delta between two task completions | 15:14 |
| fungi | 2022-03-30 15:14:13.546087 | TASK [submit-logstash-jobs : Submit logstash processing jobs to log processors] | 15:16 |
| fungi | 2022-03-30 15:14:14.787645 | localhost | ok | 15:16 |
| fungi | one second instead of several minutes | 15:16 |
| clarkb | looks like 835849 is going to be happy in the gate. If we land that we can land the redirect removal | 15:24 |
| clarkb | re the redirect removal at this point nothing should really be using git.openstack.org. Do we know what is/was using it? | 15:24 |
| fungi | clarkb: where is git.openstack.org involved? | 15:28 |
| clarkb | fungi: isn't that what thos eredirects are for? | 15:28 |
| fungi | tarballs site | 15:28 |
| clarkb | oh this is docs nevermind | 15:28 |
| clarkb | ah tarballs | 15:28 |
| fungi | not docs either, no | 15:28 |
| fungi | playbooks/roles/static/files/50-tarballs.opendev.org.conf | 15:29 |
| clarkb | I didn't realize we'd ever done redirects for tarballs | 15:29 |
| clarkb | I don't think we've done those for recent renames | 15:29 |
| clarkb | maybe it has been on a case by case basis | 15:29 |
| fungi | yeah, all the stuff which had been publishing tarballs to tarballs.openstack.org which wasn't an official openstack project got moved to tarballs.opendev.org/x/ or other namespaces | 15:29 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/835849 has a +1 now | 15:29 |
| fungi | someone (me? i don't remember now) scripted up all the file moves in afs | 15:30 |
| clarkb | ah it was for the openstack org trimming | 15:30 |
| clarkb | not general renaming | 15:30 |
| clarkb | if we land 835849 we can land the redirect change and then probably upgrade gitea as well | 15:34 |
| clarkb | Was going to wait on a complete all clear for the relesae before doing that bu I think we are close if not already there | 15:34 |
| fungi | it's all clear. we're just waiting for the last of the releasenotes jobs to finish but they're going to be hours still and aren't all that critical (merging another change to any branch of an affected project will update them anyway) | 15:38 |
| opendevreview | Merged opendev/system-config master: Workaround setuptools 61.0.0 package auto discovery problems https://review.opendev.org/c/opendev/system-config/+/835849 | 15:38 |
| fungi | okay, i'm headed out to an appointment but should be back in an hour or so | 15:40 |
| clarkb | cool I just +a'd the redirect chagne and can approve the gitea change when you get back | 15:41 |
| fungi | yep, will jump back in here as soon as i can | 15:42 |
| clarkb | no rush | 15:42 |
| *** dviroel is now known as dviroel|lunch | 15:44 | |
| *** marios|ruck is now known as marios|out | 16:20 | |
| clarkb | nb01 and nb02 have filled their /opt. I'm going to see about cleaning them up | 16:26 |
| *** jpena is now known as jpena|off | 16:28 | |
| clarkb | there wer ea number of leaked intermediate vhd files on both hosts. I've cleared those out. I'll stop the daemons one at a time and clear out /opt/dib_tmp next | 16:31 |
| corvus | infra-root: i got an error from gerrit pushing a zuul tag: https://paste.opendev.org/show/bWTs38BxgPCq7XXCUtM0/ | 16:32 |
| corvus | i checked and the tag i pushed is on a commit that is in a fresh clone of zuul from gitea | 16:33 |
| clarkb | corvus: that looks a lot like the thing we put the no thin git review flag in place for | 16:33 |
| clarkb | let me double check what that does | 16:33 |
| clarkb | corvus: it sets git push --no-thin | 16:34 |
| clarkb | if it is the same issue then it has to do with the negotiation between cgit and jgit not agreeing on whether sufficient info was provded during the push | 16:34 |
| corvus | ah. shall i try again with that? | 16:34 |
| clarkb | ya I think you can try with --no-thin and that forces your C client to send jgit a lot more info and hopefully that is enough to sort it out | 16:35 |
| corvus | done; success | 16:35 |
| corvus | thanks! | 16:35 |
| clarkb | there has been a long open bug upstream about it and it sound slike some people maybe understand it but it isn't easy to fix? | 16:35 |
| clarkb | we have an lp bug from like 2013 about it on our side too | 16:36 |
| corvus | should be `git push --harder` ;) | 16:36 |
| clarkb | I see the tag and it is on master | 16:37 |
| clarkb | so ya I suspect it was just the negotiation | 16:37 |
| clarkb | I've got nb01's daemon stopped while I clear out /opt/dib_tmp then I'll reboot to clear any monts that may have been stale and start services again | 16:41 |
| clarkb | and then I'll repeat with nb02 | 16:41 |
| opendevreview | Merged opendev/system-config master: Drop tap-as-a-service tarballs site redirect https://review.opendev.org/c/opendev/system-config/+/835829 | 16:46 |
| clarkb | I went ahead and put nb01 and nb02 in the emergency file tooso that ansible won't restart things while I wait for the disk cleanup to complete | 16:52 |
| fungi | okay, i'm caught up and it doesn't seem like there are any new fires | 16:56 |
| clarkb | nothing too hot anyway. | 16:56 |
| fungi | also i should start a backup prune today, we've been getting warnings since late last week that i've been putting off dealing with | 16:56 |
| clarkb | ah I haven't set up the infra-root inbox on the laptop yet so missed that | 16:56 |
| clarkb | Once nodepool builders are cleaned up I can approve the gitea upgrade unless you want to get it started sooner | 16:57 |
| fungi | i'm happy to approve the upgrade change now if you're good with that | 16:58 |
| clarkb | yup I think so. There are a few release note jobs for openstack to complete but this shouldn't impact that | 16:59 |
| *** dviroel|lunch is now known as dviroel | 17:00 | |
| clarkb | nb01 is rebooting now | 17:05 |
| fungi | borg pruning is in progress in a root screen session on backup02.ca-ymq-1.vexxhost | 17:06 |
| clarkb | is up and running services again. I pulled it out of the emergecny file. On to nb02 now | 17:07 |
| clarkb | corvus: the zuul status dashboard for 835307 shows an estimated completion time despite one of the jobs not having started yet | 17:15 |
| clarkb | I'm not sure that is accurate? the unstarted job takes about 40 minutes too iirc | 17:16 |
| clarkb | I guess it shows time to completion for the running jobs which may be helpful | 17:18 |
| corvus | perhaps it's too simplistic and needs to be updated to account for unstarted jobs. i'm not sure. | 17:19 |
| fungi | yeah, all the queued changes in the tag pipeline also show a projected completion time even though they don't have any builds in progress yet | 17:21 |
| clarkb | nb02 is done and out of the emergency file again | 17:36 |
| opendevreview | Merged opendev/system-config master: Update gitea to v1.16.5 https://review.opendev.org/c/opendev/system-config/+/835307 | 18:14 |
| opendevreview | Merged opendev/system-config master: Clarify the purpose of this repository https://review.opendev.org/c/opendev/system-config/+/834877 | 18:14 |
| clarkb | Will be a few minutes before the gitea deploy happens due to the hourly jobs still in progress. I'll moitor it | 18:17 |
| clarkb | Chrome version 100 is out now. There was some concern that various websites wouldn't work due to user agent parsing and behavior setting | 18:21 |
| clarkb | keep an eye out for any reports that our applications have trouble I guess | 18:21 |
| fungi | will do, thanks | 18:31 |
| clarkb | gitea01 has updated https://gitea01.opendev.org:3081/opendev/system-config if anyone wants to take a look, but my quick check indicates it is happy | 18:43 |
| fungi | yep, saw that just hit, working fine for me so far | 18:46 |
| fungi | cloning from it and browsing | 18:46 |
| clarkb | all 8 are done now | 19:03 |
| clarkb | haven't seen anything unexpected | 19:03 |
| fungi | same | 19:09 |
| fungi | borg prune completed | 19:19 |
| fungi | #status log Pruned backups on backup02.ca-ymq-1.vexxhost in order to free up some storage, info in /opt/backups/prune-2022-03-30-17-05-53.log | 19:20 |
| opendevstatus | fungi: finished logging | 19:20 |
| *** dviroel is now known as dviroel|out | 21:16 | |
| *** rlandy is now known as rlandy|bbl | 22:11 | |
| *** ianw_pto is now known as ianw | 22:24 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!