Monday, 2022-04-18

*** pojadhav is now known as pojadhav|ruck01:53
*** lajoskatona_ is now known as lajoskatona03:17
*** ysandeep- is now known as ysandeep03:26
opendevreviewchandan kumar proposed zuul/zuul-jobs master: Setup RDO gpg keys and repo for CentOS distros only  https://review.opendev.org/c/zuul/zuul-jobs/+/83785003:58
*** pojadhav is now known as pojadhav|ruck04:18
*** ysandeep is now known as ysandeep|afk05:36
*** ysandeep|afk is now known as ysandeep06:08
*** marios_ is now known as marios07:04
*** pojadhav|ruck is now known as pojadhav|lunch07:33
*** pojadhav|lunch is now known as pojadhav|ruck08:30
*** chandankumar is now known as chkumar|ruck08:33
*** ysandeep is now known as ysandeep|lunch08:37
*** ysandeep|lunch is now known as ysandeep08:54
*** ysandeep is now known as ysandeep|coffee10:54
*** dviroel__ is now known as dviroej11:26
*** ysandeep|coffee is now known as ysandeep11:33
*** dviroej is now known as dviroel11:39
*** ysandeep is now known as ysandeep|afk12:04
*** rcastillo is now known as rcastillo|ruck13:01
*** ysandeep|afk is now known as ysandeep13:02
*** chkumar|ruck is now known as chandankumar14:32
*** dviroel is now known as dviroel|lunch15:20
*** marios is now known as marios|out15:34
*** ysandeep is now known as ysandeep|out15:36
clarkbits looking like tomorrow might not have a very large crowd for our meeting. I'm tempted to cancel it and we can use that time for less formal discussions if anyone is around and has something to bring up15:38
clarkbfungi: any thoughts on bindep and git review releases today/tomorrow/soon? I'm happy to push them up and keep an eye out for trouble if others are at least around to double check the tag inputs.15:40
clarkbbindep: 5d6a6c58b0eda84615d8d57fd2fb3c58768cc61b tag as 2.11.0 due to new functionaltiy supporting alma linux15:40
clarkblooks like for bindep we're still waiting on https://review.opendev.org/c/opendev/git-review/+/82331815:41
clarkber for git-review15:42
fungiyeah, i'm happy to push tags and send release announcements today, last week was just a bit crazy with the git excitement and i didn't want to push a release on friday of a holiday weekend15:42
fungiwe could remove frickler's workflow -1 from 823318 and assume ianw's -1 is a +2 with the followup change already approved15:43
opendevreviewClark Boylan proposed opendev/system-config master: Remove health and subunit works from inventory  https://review.opendev.org/c/opendev/system-config/+/83832015:45
clarkbgmann: fungi ^ fyi noticed that when I started looking at cleaning up the subunit workers. I think landing that soon would be good but not necessary to proceed on the other health cleanups15:46
clarkbfungi: yes I think that ianw's followup is an implicit method of addressing that concern15:47
opendevreviewClark Boylan proposed opendev/system-config master: Remove config management for subunit2sql workers  https://review.opendev.org/c/opendev/system-config/+/83832215:54
clarkbfungi: gmann ^ and that is the subunit2sql worker removal from config management15:55
opendevreviewMerged opendev/system-config master: Retire opendev/puppet-openstack_health: remove from system-config  https://review.opendev.org/c/opendev/system-config/+/83687915:55
chandankumarHello infra, can we get this https://review.opendev.org/c/zuul/zuul-jobs/+/837850 merged? please have a look, when free, thanks!16:04
clarkbchandankumar: one note: zuul-jobs has its own core team and they hang out in the zuul room on matrix. That said I did have a question about that, what about rocky/alma/openeuler which are not RHEL but would still trip the old condition. Are they not expected to work with the old code so the new code isn't a regression for them?16:06
opendevreviewClark Boylan proposed opendev/system-config master: Remove configuration management for ELK stack  https://review.opendev.org/c/opendev/system-config/+/83832416:06
clarkbfungi: gmann ^ and that can land once we stop zuul sneding gearman events to the geard16:06
clarkbI set the depends on for that so we should be good to land those as dependencies land16:07
chandankumarclarkb: RHEL ships openvswitch via different repos, earlier we were mixing RHEL and CentOS content, It fixes the issue and installs it on CentOS so it might work for rocky/alma/openeuler if CentOS repos get enabled or others adds conditions for enabling these repos on those distros16:09
clarkbchandankumar: right thats my concern. You are disabling it for all of those distros too16:10
clarkbit might be better to exclude rhel instaed16:10
clarkband then those distros can update if they are broken16:10
opendevreviewMerged opendev/system-config master: Remove health and subunit works from inventory  https://review.opendev.org/c/opendev/system-config/+/83832016:11
chandankumarok let me update this condition16:13
opendevreviewchandan kumar proposed zuul/zuul-jobs master: Setup RDO gpg keys and repo for CentOS distros only  https://review.opendev.org/c/zuul/zuul-jobs/+/83785016:14
chandankumarclarkb: ^^16:15
clarkbchandankumar: I think the condition in https://review.opendev.org/c/zuul/zuul-jobs/+/837850/3/roles/multi-node-bridge/templates/zuul-multi-node-bridge-ovs.repo.j2 needs to update too?16:16
opendevreviewchandan kumar proposed zuul/zuul-jobs master: Donot Setup RDO gpg keys and repo for RHEL and Fedora  https://review.opendev.org/c/zuul/zuul-jobs/+/83785016:21
chandankumarclarkb: ^^ removed centos conditionals from repo16:21
gmannclarkb: so for ELK, we are stopping the old server now ? or I think we said to keep for 1-2 month and give people time to move to new dashbaord and feedback if any ?16:34
clarkbgmann: we said after yoga we would remove it16:39
clarkband it broke during the release and slowed it down16:39
clarkbthe problem ultimately is that no one is around trying to make it work anymore and it is causing problems for things like the release16:39
clarkbI think if it hadn't impacted the release then I would be fine trying to keep it around longer but it did impact the reelase and if no one is workign to fix it we should remove it?16:40
clarkbit was always a "if it doesn't cause problems we'll keep it up" statement. It has been causing problems16:40
gmannbecause we have not send on ML about the new dashbaord and documentation patch of url and its credentials to use is not merged. 16:41
gmannmay be we can wait for an month and finish ^^ work. I am not sure if dpawlik is online/back to work so that we can merge those things soon and send on ML.16:43
clarkbok I don't think the existing system ahs been working is the problem16:44
clarkbit was definitely not working during the release and fungi had to restart some stuff16:44
clarkbwhether or not it is happy now I don't know16:44
gmannclarkb: ok, it be good if we can wait, at least until dpawlik  patches merged and he announce the new dashboard on ML. 16:49
clarkbok even if the dat isn't populating and jobs like releases are impacted?16:50
clarkbthats my concern. We can say this is up but if it isn't actually functional and things that we expect to be functional like release jobs are impacted then maybe we need to prioritize16:51
funginote it's not just release jobs, it's basically slowing down all jobs because of the gearman submission taking minutes16:51
fungiso consuming a non-negligible proportion of our test resource quota by making all jobs incur that additional time penalty16:52
clarkbbasically what we aleays said was we'd try to make it last until yoga but it was dependent on it not causing problems16:52
clarkbit is causing problems now16:52
clarkbit is after yoga16:52
clarkbI think we should turn it off16:52
clarkbwe've asked for help to manage this system in opendev and not got it. We've done a fair bit of assistance to help with the migration to opensearch. I don't think we should pretend the existing service is happy and functional if it is not even if the openasrch stuff isn't quite ready16:53
clarkbif the release was not impacted then I wouldn't mind leaving it up a bit16:54
opendevreviewClark Boylan proposed opendev/system-config master: Remove config management for subunit2sql workers  https://review.opendev.org/c/opendev/system-config/+/83832217:04
opendevreviewClark Boylan proposed opendev/system-config master: Remove configuration management for ELK stack  https://review.opendev.org/c/opendev/system-config/+/83832417:04
clarkbtesting caught that logstash puppetry depends on subunit2sql puppet module so can't remove that puppet module from stuff until we remove logstash17:04
clarkbhrm related ti seems liek we've got another backlog of tag jobs17:12
clarkbon the whole not very busy though and our node usage is fairly low17:12
*** dviroel|lunch is now known as dviroel17:23
clarkbhttps://review.opendev.org/c/opendev/system-config/+/838322 is hapyp now which removes the subunit workers17:29
opendevreviewMerged openstack/project-config master: Retire openstack-health projects: remove project from infra  https://review.opendev.org/c/openstack/project-config/+/83670918:11
opendevreviewMerged zuul/zuul-jobs master: Donot Setup RDO gpg keys and repo for RHEL and Fedora  https://review.opendev.org/c/zuul/zuul-jobs/+/83785018:29
opendevreviewMerged opendev/git-review master: Fix submitting signed patches  https://review.opendev.org/c/opendev/git-review/+/82331818:36
fungii guess we can tag git-review today with those in18:39
fungii'll get to work on both that and bindep releases shortly18:39
clarkbfungi: the second git-review change hasn't landed yet though18:40
clarkbbut yes once that lands I think we can ship them with tags18:40
fungiyeah, still waiting on it18:40
opendevreviewMerged opendev/git-review master: Enforce minimum git version  https://review.opendev.org/c/opendev/git-review/+/83745118:45
clarkbthere it is18:45
clarkbI'm going to single core approve https://review.opendev.org/c/opendev/system-config/+/838322 to remove the subunit2sql workers since that seems to be not controversial and I don't know that I'll get another set of eyeballs for a few days18:46
fungithanks18:46
clarkbI'll leave the ELK removal change up for a bit longer18:46
clarkband others can chime in if they want to figure out what is going on with that to help gmann out.18:46
clarkbfungi: any objection to cnacelling tomorrow's meeting?18:47
funginone, since i don't think i'll be able to make it myself18:47
fungiwould be sort of silly for me to insist there be a meeting i'm unable to attend ;)18:47
fungibindep commit 5d6a6c58b0eda84615d8d57fd2fb3c58768cc61b (HEAD, tag: 2.11.0, origin/master, origin/HEAD, gerrit/master, refs/changes/18/837118/5)18:49
fungigit-review commit 82a2c8df2e6deada5770d1ea48e5d0ce0eaeab36 (HEAD, tag: 2.3.0, origin/master, origin/HEAD, gerrit/master)18:49
fungithose look right?18:49
clarkblooking18:50
clarkb5d6a6c58b0eda84615d8d57fd2fb3c58768cc61b and 2.11.0 look right to me for bindep18:51
clarkbgit review's log is a bit more extensive let me read it over really quickly18:52
clarkbyes 82a2c8df2e6deada5770d1ea48e5d0ce0eaeab36 as 2.3.0 looks good on git-review18:53
fungicool, pushing both now in that case18:59
fungionce those are done, i'll see if i need to do anything to refresh the release notes pages for them19:00
opendevreviewMerged opendev/system-config master: Remove config management for subunit2sql workers  https://review.opendev.org/c/opendev/system-config/+/83832219:00
fungirelease jobs completed, though the release notes predictably still list the unreleased state as docs were not rebuilt with the tags present19:14
fungilooks like we're relying on opendev-promote-docs in the promote pipeline, and reenqueuing the most recent changes into promote is probably not advisable?19:15
fungiyeah, it would just republish the same docs already built in the gate pipeline for those changes19:18
funginot build new docs19:18
Clark[m]We can land some noopy changes just to build new docs19:20
fungii've got one better19:20
opendevreviewJeremy Stanley proposed opendev/bindep master: Run opendev-publish-tox-docs on release  https://review.opendev.org/c/opendev/bindep/+/83834019:25
opendevreviewJeremy Stanley proposed opendev/git-review master: Run opendev-publish-tox-docs on release  https://review.opendev.org/c/opendev/git-review/+/83834119:25
fungimmm, though i wonder if that's going to publish under something other than latest19:26
fungitarget_dict: "{{ afs.targets.tag }}"19:28
fungiyeah, we probably want one which relies on zuul's ability to guess the most appropriate branch from the tag?19:28
fungii suspect that's going to need a new job definition19:29
clarkbya I think the zuul and nodepool doc build suffer from this too19:33
clarkbthey push to the tag but then you need something on master to lupdate the latest/ path19:33
fungiright. the docs job they use has the same parent as opendev-publish-tox-docs19:33
fungi(opendev-publish-tox-docs-base)19:33
fungithough how does the branch-guessing-from-tag work again? or did that not actually end up implemented? looking at a tag-triggered build, i don't see any zuul var which reflects the likely branch for that tag19:35
fungii think openstack stable releases rely on this for their releasenotes jobs, so maybe i can track it down there19:36
fungiahh, no they don't19:40
fungievery python3-releasenotes build generates branch-specific release notes for all available branches, and republishes that entire tree19:41
fungiwhich i should have remembered, since that's what causes them to sometimes race one another when two tags are created on different branches at roughly the same moment19:41
opendevreviewClark Boylan proposed opendev/system-config master: Switch Refstack image over to python3.9  https://review.opendev.org/c/opendev/system-config/+/83834419:43
clarkbI've been thinking it would be good for us to get to python3.9 then we can add python3.10 images19:43
clarkbI'm hopeful that we can also drop the 3.7 and maybe even 3.8 base images onces we get everything on 3.919:43
clarkbthat is a first change to work towards that. I think zuul and nodepool may also by on 3.8 so some work to be done elsewhere19:44
clarkbThere is some other general zuul config cleanup related to that I'll work on next19:44
clarkbmnaser: are you using our base python images? I'm curious if you are still using the buster images or if you ahve moved everything to bullseye. Also what is your oldest python version that you are using. I'm hopeful I can delete all of the buster images and the python3.7 images19:47
opendevreviewJeremy Stanley proposed opendev/bindep master: Run opendev-publish-tox-docs on release  https://review.opendev.org/c/opendev/bindep/+/83834019:48
opendevreviewJeremy Stanley proposed opendev/git-review master: Run opendev-publish-tox-docs on release  https://review.opendev.org/c/opendev/git-review/+/83834119:48
fungioh, it looks like we an override the "tag" jobvar for the parent. clarkb ^ is that how you do it?19:48
opendevreviewClark Boylan proposed opendev/system-config master: Fixup some image dependencies in our zuul configs  https://review.opendev.org/c/opendev/system-config/+/83834519:49
*** dviroel is now known as dviroel|biab19:49
clarkbfungi: I think you may need to indent vars: an extra level?19:50
fungiobviously not a solution for multi-branch projects, but for single-branch projects like bindep and git-review it should suffice, i think19:50
fungioh, yep19:50
opendevreviewJeremy Stanley proposed opendev/git-review master: Run opendev-publish-tox-docs on release  https://review.opendev.org/c/opendev/git-review/+/83834119:50
opendevreviewJeremy Stanley proposed opendev/bindep master: Run opendev-publish-tox-docs on release  https://review.opendev.org/c/opendev/bindep/+/83834019:50
fungibut you can override job vars like that in a variant, right? even a variant of a job inheriting those vars from an abstract parent?19:51
clarkbI think so19:51
clarkbit should merge the vars together with the parents with children overriding iirc19:52
fungii guess we can find out the next time we tag one of those19:52
clarkbfungi: https://review.opendev.org/c/openstack/project-config/+/837621 is an easy one that I think we can land even while the logstash geard remains up for the next little bit19:55
opendevreviewClark Boylan proposed opendev/system-config master: Remove our buster python images  https://review.opendev.org/c/opendev/system-config/+/83834720:00
clarkbI'm hoping ^ will shake out things that are still using those images if they exist20:00
opendevreviewClark Boylan proposed opendev/system-config master: Remove python3.7-bullseye docker images  https://review.opendev.org/c/opendev/system-config/+/83834820:03
clarkbfungi: https://review.opendev.org/c/opendev/system-config/+/831441 another straightforward chagne if you have time (this cleans up our clouds.yaml to remove the airship cloud stuff)20:11
opendevreviewMerged openstack/project-config master: Remove geard graphing from zuul-status dashboard  https://review.opendev.org/c/openstack/project-config/+/83762120:11
clarkboh heh a number of those chagnes have piled up behind the one that updated the inventory. I guess that isn't a major problem. Just slow20:21
opendevreviewClark Boylan proposed opendev/system-config master: Switch Refstack image over to python3.9  https://review.opendev.org/c/opendev/system-config/+/83834420:23
fungiinfra-prod-remote-puppet-else failed deploy for 83832020:26
fungii can probably take a look after dinner to see why20:27
clarkbfungi: it looks like ti failed on logstash01.o.o trying to do stuff in hte kibana repo20:28
clarkbmore reason to turn things off ?20:28
clarkberror: some local refs could not be updated; try running 'git remote prune origin' to remove any old, conflicting branches20:28
clarkbbut double check me on that to make sure nothing else is unhappy20:28
clarkbmy kids get out of school early tomorrow and that conflicts with the meeting time. More reason to cancel. I'll wait on sending an email about that until ianw's theoretical morning would start if he ends up workign today to chime in though20:30
clarkbwell https://review.opendev.org/c/opendev/system-config/+/838347 didn't explode. Maybe this is safe afterall. /me checks codesearch20:32
clarkbhrm codesearch immediately finds that gear and storyboard rely on that maybe20:33
opendevreviewClark Boylan proposed opendev/gear master: Update the docker image to python3.9 and buster  https://review.opendev.org/c/opendev/gear/+/83840220:36
opendevreviewClark Boylan proposed opendev/storyboard master: Update container image to bullseye and python3.9  https://review.opendev.org/c/opendev/storyboard/+/83840320:39
clarkbI'm not finding any other uses which means we should be good once those two changes land. Let me set a depends on20:40
*** dviroel|biab is now known as dviroel20:42
opendevreviewClark Boylan proposed opendev/system-config master: Remove our buster python images  https://review.opendev.org/c/opendev/system-config/+/83834720:42
opendevreviewClark Boylan proposed opendev/system-config master: Remove python3.7-bullseye docker images  https://review.opendev.org/c/opendev/system-config/+/83834820:42
clarkblooks like storyboard is broken due to a jinja2 thing. I'll see if I have time after school run to look at it more closely20:55
opendevreviewMerged opendev/system-config master: Remove airship-kna1 from bridge clouds.yaml  https://review.opendev.org/c/opendev/system-config/+/83144121:06
*** dviroel is now known as dviroel|afk21:37
fungii think i looked into the storyboard problem already... checking my swiss cheese memory21:40
fungidoesn't look like i pushed any fix if so21:40
fungihttps://review.opendev.org/715229 seems related to container base images though21:41
fungiclarkb: 838340 and 838341 are both passing if you're up for approving. their merging should update docs as a side effect, probably? then i can send release announcements without worrying about people thinking there haven't been releases21:50
clarkbfungi: in this case the storyboard issue are all outside teh containers21:59
clarkbfungi: ya let me take a look21:59
clarkbianw: wondering if you think we should have a meeting tomorrow. I think it may just be the two of us and I've got kids that need picking up from school early tomorrow during the meeting time22:00
*** prometheanfire is now known as Guest224622:00
ianwsorry just reading backwards.  skipping is fine if nobody has anything.  i'm assuming there was some fallout of git things while I was Easter-ing to catch up on :)22:01
clarkbianw: there was some and its a bit all over as to what appraoch peopel took. But I think the issue is understood and people know how to address it22:01
ianwthanks for the fix to source-repositories22:05
clarkbfungi: left a comment on https://review.opendev.org/c/opendev/bindep/+/838340 I didn't realize that was part of an afs secrets dict and thought i t was a normal var. I'm not longer certain this will do anything22:11
clarkbI think the comment applies to both changes22:11
fungioh. hrm22:13
clarkbfungi: ya and if you look at opendev/base-jobs/playbooks/tox-docs/publish.yaml it does target_dir: "{{ target_dict.path.format(zuul=zuul) }}"22:15
fungiyeah, i missed the indentation in the description, i guess it's something like afs[targets][tag]?22:15
clarkbso its taking the value out of zuul.stuff22:15
clarkband when you set vars on a job I think they are just normal inventory vars not part of the zuul dict either22:15
clarkbfungi: ya and sorry its literally tag: path: /some/path/here22:16
fungioh, right: "This is expected to be a dict with a key value pair: `path: the full docs publication path to use if the job is run on a tag.`"22:17
clarkblooking at storyboard more closely it apepars there are linter issues, sqlalchemy issues, and also the jinja2 docs thing :/22:22
clarkbI'll start on a likely incimplete change but would be good to see some things happier22:22
fungii'd also be fine forcing the next storyboard change to work with newer container base images22:22
clarkbthe problem isn't the container images its storyboard itself22:26
fungiso anyway, to do docs republish on tag... we need a different secret i guess (reencoding the same kerberos ticket)?22:26
clarkbthe jobs related to container buidls seem fine22:26
fungiclarkb: right, i mean i'd be fine merging a change elsewhere which leaves storyboard jobs broken until we update the container builds to suit22:27
clarkboh I suee22:28
fungii guess i could work through tag-triggered docs builds by pushing tags to opendev/sandbox22:30
clarkbfungi: ya I think we need a different secret that sets a different path value that just hardcodes to /latest rather than /{zuul.tag} or whatever it does now22:32
fungiokay cool, i'll try to figure that out22:32
fungii guess i can just duplicate the encrypted token since it'll be for the same project, no need to reencode22:33
opendevreviewClark Boylan proposed opendev/storyboard master: Omnibus fixups for storyboard testing and CI  https://review.opendev.org/c/opendev/storyboard/+/83840922:41
clarkbok I'm going to send email skipping tomorrows meeting22:41
clarkbinfra-root https://review.opendev.org/c/opendev/system-config/+/838345 and https://review.opendev.org/c/opendev/system-config/+/838344 should be easy reviews. I'm going to WIP https://review.opendev.org/c/opendev/base-jobs/+/837619 since there is some contention over whether or not we need to keep running that now ...22:46
ianwhrm, i'm getting a 502 from paste trying to publish some of the long query logs we've seen22:54
clarkbthere is a size limit. IN the past it would just truncate and silently fail returning success but maybe now it doesn't do that?22:55
ianwpymysql.err.DataError: (1406, "Data too long for column 'code' at row 1")22:55
ianwi wonder if this is a result of the switch to our own hosted db22:55
ianw| code       | text        | YES  |     | NULL    |                |22:57
ianwso that would be 64k i guess22:58
fungii'd be fine increasing the size limit for that column if it's easy23:00
opendevreviewClark Boylan proposed opendev/storyboard master: Omnibus fixups for storyboard testing and CI  https://review.opendev.org/c/opendev/storyboard/+/83840923:02
ianwi think it's probably a decent size, it would just be better if it didn't 502 over the limit23:02
ianwnext size up is 16mb which seems too big23:02
clarkbI have to pop out for kids swim calss now but will check oin that storyboard change after. I think it is actually pretty close now23:09
*** dviroel|afk is now known as dviroel|out23:51

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!