| *** zxiiro has quit IRC | 00:30 | |
| *** tosky has quit IRC | 00:52 | |
| *** mlavalle has quit IRC | 01:22 | |
| *** tetsuro has joined #openstack-release | 01:24 | |
| *** diablo_rojo has quit IRC | 02:01 | |
| *** diablo_rojo has joined #openstack-release | 03:10 | |
| *** tetsuro has quit IRC | 03:46 | |
| *** tetsuro has joined #openstack-release | 03:49 | |
| *** ykarel|away is now known as ykarel | 04:08 | |
| *** udesale has joined #openstack-release | 04:47 | |
| *** tonyb[m] has quit IRC | 04:49 | |
| *** tonyb[m] has joined #openstack-release | 04:49 | |
| *** udesale has quit IRC | 05:10 | |
| *** udesale has joined #openstack-release | 05:12 | |
| *** evrardjp has quit IRC | 05:36 | |
| *** evrardjp has joined #openstack-release | 05:36 | |
| *** diablo_rojo has quit IRC | 05:39 | |
| *** armax has quit IRC | 05:53 | |
| *** ykarel is now known as ykarel|meeting | 06:01 | |
| *** ykarel|meeting is now known as ykarel | 06:49 | |
| *** jbadiapa has joined #openstack-release | 07:01 | |
| *** vishalmanchanda has joined #openstack-release | 07:56 | |
| *** evrardjp has quit IRC | 07:57 | |
| *** evrardjp has joined #openstack-release | 08:01 | |
| *** slaweq has joined #openstack-release | 08:01 | |
| *** amoralej|off is now known as amoralej | 08:02 | |
| *** guilhermesp has quit IRC | 08:06 | |
| *** dmellado has quit IRC | 08:06 | |
| *** dmellado has joined #openstack-release | 08:10 | |
| *** rpittau|afk is now known as rpittau | 08:26 | |
| *** toabctl has quit IRC | 08:30 | |
| *** e0ne has joined #openstack-release | 08:51 | |
| *** tosky has joined #openstack-release | 09:00 | |
| *** toabctl has joined #openstack-release | 09:14 | |
| *** e0ne_ has joined #openstack-release | 09:25 | |
| *** e0ne has quit IRC | 09:25 | |
| *** ykarel is now known as ykarel|lunch | 09:27 | |
| *** e0ne_ has quit IRC | 09:50 | |
| *** e0ne has joined #openstack-release | 09:51 | |
| *** ykarel|lunch is now known as ykarel | 10:14 | |
| *** tetsuro has quit IRC | 10:34 | |
| *** dtantsur|afk is now known as dtantsur | 10:48 | |
| *** e0ne has quit IRC | 10:59 | |
| *** e0ne has joined #openstack-release | 10:59 | |
| *** e0ne has quit IRC | 11:00 | |
| *** e0ne has joined #openstack-release | 11:00 | |
| ttx | beh | 11:37 |
|---|---|---|
| ttx | All releases failing with <ProtocolError for pypi.org/pypi: 503 Backend is unhealthy> | 11:38 |
| ttx | although Pypi reports all systems operational and no unusual error rate | 11:41 |
| ttx | we are not the only ones seeing issues though | 11:45 |
| *** rpittau is now known as rpittau|bbl | 11:45 | |
| *** e0ne_ has joined #openstack-release | 11:46 | |
| *** e0ne has quit IRC | 11:47 | |
| *** e0ne has joined #openstack-release | 11:50 | |
| *** e0ne_ has quit IRC | 11:50 | |
| *** e0ne has quit IRC | 11:53 | |
| *** e0ne has joined #openstack-release | 11:54 | |
| openstackgerrit | Merged openstack/releases master: Add Victoria cycle schedule for manila https://review.opendev.org/715083 | 11:57 |
| *** e0ne has quit IRC | 12:14 | |
| *** e0ne has joined #openstack-release | 12:14 | |
| *** udesale_ has joined #openstack-release | 12:21 | |
| openstackgerrit | Merged openstack/releases master: Release manila 8.1.1 off of stable/stein https://review.opendev.org/715035 | 12:23 |
| openstackgerrit | Merged openstack/releases master: Release manila 9.1.1 off of stable/train https://review.opendev.org/715032 | 12:23 |
| *** udesale has quit IRC | 12:24 | |
| openstackgerrit | Merged openstack/releases master: Release 5.2.0 of python-openstackclient https://review.opendev.org/715091 | 12:27 |
| *** e0ne has quit IRC | 12:28 | |
| *** e0ne has joined #openstack-release | 12:28 | |
| ttx | Looks like it's back to normal now | 12:32 |
| ttx | https://status.python.org/incidents/1yd6sfczzfmz | 12:32 |
| openstackgerrit | Merged openstack/releases master: Release 0.44 of openstacksdk https://review.opendev.org/715090 | 12:34 |
| fungi | ttx: does anything need to be reenqueued | 12:49 |
| fungi | ? | 12:49 |
| smcginnis | AFS error on the manila release notes, but I think we can ignore that. | 12:50 |
| smcginnis | I haven't seen any other failures. Did I miss something ttx? | 12:50 |
| smcginnis | fungi: In case it's of interest: https://zuul.opendev.org/t/openstack/build/04088a620e904d3d8c7785a7c9095279/log/job-output.txt#1620 | 12:50 |
| fungi | rsync: stat "/afs/.openstack.org/docs/releasenotes/manila/es/_static/js/.jquery-3.2.1.js.JXahrd" failed: No such file or directory (2) | 12:52 |
| fungi | that looks like another rsync race | 12:52 |
| fungi | like a different job had written that temporary location and the current job saw it but then when it tried to delete it was already gone | 12:53 |
| ttx | fungi: no, they failed validation | 12:54 |
| ttx | and now all passed | 12:54 |
| fungi | there's a .root-marker in /afs/.openstack.org/docs/releasenotes/manila/ so there would have to have been more than one job trying to upload manila release notes at the same time | 12:55 |
| smcginnis | fungi: Ah, yep. There was a stein and train release. | 12:55 |
| smcginnis | fungi: We may need some help figuring out why https://review.opendev.org/#/c/714682/ is different locally. | 12:56 |
| smcginnis | fungi: validate job passes locally. Fails to import the project in the gate even though the code appears fine from manual inspection. | 12:56 |
| fungi | ERROR: Could not find 2.4.1.0: Command '['git', 'show', '2.4.1.0']' returned non-zero exit status 128. | 12:59 |
| fungi | there's several of those for different tags | 13:00 |
| fungi | is that expected? | 13:00 |
| ttx | yeah that's expected | 13:00 |
| fungi | okay so this is the first obvious sign of trouble? ModuleNotFoundError: No module named 'xstatic.pkg.angular_gettext' | 13:01 |
| *** amoralej is now known as amoralej|lunch | 13:01 | |
| ttx | The weird thing there is that looking at the code it runs, it clearly finds the "angular_gettext" directory while looking into the xstatic/pkg directory | 13:02 |
| ttx | but then fails to load the module | 13:02 |
| ttx | like it finds angular_gettext/ but not angular_gettext/__init__,py | 13:02 |
| ttx | I can't reproduce that locally | 13:02 |
| ttx | so I was wondering if there was not some borked local checkout | 13:02 |
| fungi | you're testing locally with python3.6? | 13:03 |
| ttx | Python 3.6.9 yes | 13:03 |
| fungi | just want to rule out the possibility that one wheel is broken and others for different interpreter versions are okay | 13:04 |
| ttx | https://opendev.org/openstack/releases/src/branch/master/openstack_releases/xstaticutils.py shows the code that fails | 13:04 |
| ttx | It basically looks into xstatic/pkg | 13:04 |
| ttx | then tries to load modules for anything that has not __ in the name | 13:04 |
| ttx | so it definitely finds "angular_gettext" in there | 13:05 |
| ttx | which shows that the checkout is mostly there | 13:05 |
| fungi | xstatic.pkg.angular_gettext seems like it might be something vendored | 13:06 |
| fungi | or is that just how all the xstatic packages are organized? | 13:06 |
| ttx | yes it's normal | 13:07 |
| fungi | yeah, i guess that's normal since it's python packaging of javascript | 13:07 |
| ttx | xstatic things are python packages wrapped around JS libs | 13:07 |
| ttx | https://opendev.org/openstack/xstatic-angular-gettext/src/branch/master/xstatic/pkg | 13:07 |
| ttx | so here you can find where it gets that "angular_gettext" (it fails loading) from | 13:07 |
| fungi | i can play around with the package in ~30 minutes and see if i spot obvious problems | 13:08 |
| ttx | I don;t think it's urgent. it's just puzzling | 13:08 |
| fungi | is this installing it from source or from a release? | 13:08 |
| ttx | should be a git checkout | 13:08 |
| *** rpittau|bbl is now known as rpittau | 13:10 | |
| *** zxiiro has joined #openstack-release | 13:29 | |
| *** e0ne_ has joined #openstack-release | 13:41 | |
| *** e0ne has quit IRC | 13:41 | |
| fungi | i guess the job is doing its own cloning of that repo, since it doesn't appear in the list or projects zuul provided for the build | 13:48 |
| ttx | probably | 13:49 |
| fungi | yeah, okay, found in the log where it does that | 13:49 |
| fungi | i can definitely importlib.import_module('xstatic.pkg.angular_gettext') in a python3.6 venv after installing 6b7ba0f | 13:54 |
| fungi | and it's failing consistently, i take it | 13:55 |
| fungi | judging from prior zuul comments on that change | 13:56 |
| ttx | yeah, there were a bunch of rechecks already | 13:56 |
| ttx | hence the mystery | 13:56 |
| ttx | also I reviewed the changes since last time it worked and they are pretty minor | 13:57 |
| ttx | at least nothing justifying why that one would fail while the others work | 13:58 |
| * ttx factchecks that | 13:59 | |
| ttx | yeah so xstatic-angular-bootstrap passes that test alright | 14:00 |
| ttx | https://zuul.opendev.org/t/openstack/build/a2536a5362e94c71b880168ae72447c5/log/job-output.txt#3822 | 14:00 |
| fungi | i can set an autohold for that job and 714682, just a sec | 14:00 |
| *** amoralej|lunch is now known as amoralej | 14:02 | |
| fungi | okay, once the next failure comes back i can jump on there and see what's different. can add any of your ssh keys too if you like | 14:03 |
| ttx | I would probably not know my way around that node and where to look :) | 14:05 |
| fungi | we got one, digging around in it now | 14:16 |
| smcginnis | fungi: Let me know if I can help. Not sure what I could add, but following along at home at least. | 14:21 |
| *** chandan_kumar has joined #openstack-release | 14:29 | |
| *** chandankumar has quit IRC | 14:29 | |
| *** dulek has quit IRC | 14:30 | |
| *** corvus has quit IRC | 14:30 | |
| *** corvus has joined #openstack-release | 14:31 | |
| *** dulek has joined #openstack-release | 14:33 | |
| *** armax has joined #openstack-release | 14:43 | |
| smcginnis | fungi: Oh, not to pile on on things, but election related, we need to get https://review.opendev.org/715088 through if you have a chance to take a look at that one. | 15:05 |
| *** ykarel is now known as ykarel|away | 15:06 | |
| fungi | smcginnis: i am no longer an election official | 15:10 |
| fungi | but also it seems to have been approved moments ago | 15:10 |
| smcginnis | OK, Andy got it. I thought you still had core rights there. | 15:10 |
| fungi | if i do, it's just because they haven't been cleaned up yet | 15:11 |
| *** KeithMnemonic has quit IRC | 15:13 | |
| *** armstrong has joined #openstack-release | 15:19 | |
| fungi | okay, back to trying to dissect that xstatic job failure, sorry, was stretched between other discussions and security sig meeting | 15:33 |
| *** priteau has joined #openstack-release | 15:45 | |
| *** udesale_ has quit IRC | 15:50 | |
| *** chandan_kumar is now known as chandankumar | 15:51 | |
| fungi | oh! so the reason i'm not finding anything useful on the held node is that it failed on a different problem: https://zuul.opendev.org/t/opendev/build/b873ebadfe78421c868c4bc2009b855a/ | 15:54 |
| fungi | we've identified this as bad data coming from some nodes in pypi's cdn network | 15:54 |
| smcginnis | Oh, interesting. | 15:55 |
| smcginnis | Ah, the pygments issue just landed in upper-constraints too. | 15:55 |
| fungi | well, i don't think it accounts for the earlier failures though, because pypi's struggles just started today and the verify failures are going back as far as tuesday | 15:55 |
| ttx | yeah, different issue | 15:56 |
| smcginnis | #startmeeting releaseteam | 16:00 |
| openstack | Meeting started Thu Mar 26 16:00:01 2020 UTC and is due to finish in 60 minutes. The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: releaseteam)" | 16:00 | |
| openstack | The meeting name has been set to 'releaseteam' | 16:00 |
| *** mlavalle has joined #openstack-release | 16:00 | |
| smcginnis | Ping list: ttx armstrong diablo_rojo, diablo_rojo_phon | 16:00 |
| smcginnis | #link https://etherpad.openstack.org/p/ussuri-relmgt-tracking | 16:00 |
| diablo_rojo_phon | o/ | 16:00 |
| armstrong | o/ | 16:00 |
| smcginnis | Line 486 | 16:00 |
| elod | o/ | 16:00 |
| hberaud | o/ | 16:00 |
| ttx | o/ | 16:01 |
| smcginnis | #topic This weeks tasks | 16:01 |
| *** openstack changes topic to "This weeks tasks (Meeting topic: releaseteam)" | 16:01 | |
| smcginnis | ttx got the countdown email updated. We'll cover that on its own. | 16:02 |
| smcginnis | Other task was just to notify infra to be prepared for getting the next cycle's signing key ready. | 16:02 |
| smcginnis | That has been done. Nothing changes yet, but just being prepared. | 16:02 |
| smcginnis | So we're in good shape for this week. | 16:02 |
| smcginnis | #topic Next week's tasks | 16:03 |
| *** openstack changes topic to "Next week's tasks (Meeting topic: releaseteam)" | 16:03 | |
| smcginnis | First one is the autoreleases for c-w-i. | 16:03 |
| smcginnis | I will take that one. | 16:03 |
| smcginnis | There's still some opportunity for better automation on that. | 16:03 |
| smcginnis | I have a few fragments of things locally. I'd like to use this as a reason to keep going with that. | 16:03 |
| smcginnis | "Update the feature list and allowed stable branch names in devstack-gate for the new stable branch. For example, https://review.opendev.org/362435 and https://review.opendev.org/363084" | 16:04 |
| smcginnis | This is a good one to "delegate". :) | 16:04 |
| ttx | I can pick that up if nobody wants it | 16:04 |
| ttx | Although I have a task below it already so happy to leave it | 16:04 |
| smcginnis | Any other takers? | 16:04 |
| ttx | note: we finally went full circle and now have all weekly emails drafted | 16:05 |
| smcginnis | \o/ | 16:05 |
| ttx | Once I post taht process update they will all be living in doc | 16:05 |
| diablo_rojo_phon | Nice! | 16:05 |
| smcginnis | ttx: I guess we can put you down for that task, but anyone else feel free to speak up and take it instead. | 16:06 |
| ttx | ++ | 16:06 |
| smcginnis | Next task, sending countdown. | 16:06 |
| smcginnis | Not sure we have to have that called out as a separate task, but good reminder that we are now back to weekly emails. | 16:07 |
| smcginnis | I will get that sent tomorrow. | 16:07 |
| smcginnis | And last task for next week i the process update that ttx has taken. | 16:07 |
| ttx | ++ | 16:08 |
| smcginnis | Our process doc is pretty solid now. But if anyone notices something we just do because we just do it, feel free to point out or add to the process guide to capture anything missing there. | 16:08 |
| smcginnis | #topic Stuck reviews | 16:08 |
| *** openstack changes topic to "Stuck reviews (Meeting topic: releaseteam)" | 16:08 | |
| smcginnis | #link https://review.opendev.org/#/c/714682/ xstatic issue | 16:08 |
| smcginnis | We have one lib that is getting an odd failure. | 16:09 |
| smcginnis | fungi is helping us out by looking into it. | 16:09 |
| smcginnis | Not clear yet what is going on there. | 16:09 |
| fungi | yeah, i still need a clean run with the failure | 16:09 |
| fungi | going to set another autohold and hope it too doesn't fall victim to the temporary pypi cdn issue | 16:09 |
| smcginnis | Great, thanks for working on that. | 16:10 |
| smcginnis | Really hard to tell what's happening with gate behaving differently than local runs. | 16:10 |
| fungi | will report in this channel once i have ideas | 16:10 |
| smcginnis | Just let us know even if you just need someone to bounce ideas off of. | 16:10 |
| smcginnis | And thanks again. | 16:10 |
| fungi | it's my pleasure | 16:11 |
| smcginnis | #link https://review.opendev.org/#/q/status:open+project:openstack/releases+topic:rocky-em Rocky-EM reviews | 16:11 |
| smcginnis | First one - OSA. Waiting on a final release from them I believe. | 16:11 |
| smcginnis | That's a cycle-trailing one, so they wanted to wait for others to go first before wrapping up. | 16:11 |
| smcginnis | I think there was just one patch they had mentioned that they were still waiting on. | 16:11 |
| smcginnis | Though last I looked, it hasn't even merged to master yet I think. | 16:12 |
| ttx | maybe noonedeadpunk or evrardjp can enlighten us | 16:12 |
| noonedeadpunk | yeah that's true actually | 16:12 |
| noonedeadpunk | Actually all functional things are merged now | 16:12 |
| noonedeadpunk | I've pinged patch owner yestarday regarding this doc patch but had no feedback | 16:13 |
| smcginnis | noonedeadpunk: Ready for a final release? Or need to wait a little longer on that? | 16:13 |
| noonedeadpunk | So I'm kinda about just to merge new bump https://review.opendev.org/#/c/714926/ and do final release on that. | 16:13 |
| noonedeadpunk | I guess docs will be rendered even without being tagged? | 16:14 |
| smcginnis | noonedeadpunk: Awesome, thanks. We'll watch for that, then get the rocky-em patch updated once we have the new commit shas. | 16:14 |
| smcginnis | Oh, that's true. They still get published, so no need to wait for that for the final release. | 16:14 |
| noonedeadpunk | smcginnis: does it require to do another release to set EM on top of it? | 16:15 |
| smcginnis | Really the only thing would be if someone wanted to build the docs locally from the last official release. | 16:15 |
| smcginnis | noonedeadpunk: Yeah, we'll need a new patch doing the release, then the em patch can get rebased on top of that. | 16:15 |
| noonedeadpunk | ok, so I'm monitoring it as well, so can place it asap | 16:16 |
| smcginnis | Thanks! | 16:16 |
| smcginnis | Other outstanding on is heat. | 16:16 |
| smcginnis | I tried pinging ricolin, but no response. | 16:16 |
| elod | maybe we can just reset the patch to its original state and merge? :/ | 16:17 |
| ttx | yeah I would reset to original and set the countdown | 16:17 |
| smcginnis | OK, that sounds reasonable. I will get that updated. | 16:18 |
| smcginnis | Last item - | 16:18 |
| smcginnis | #link https://review.opendev.org/#/c/709748/ | 16:18 |
| ttx | it's a classic case where something proposed by one of us just lingers longer | 16:19 |
| ttx | since we feel self-approving would be inappropriate | 16:19 |
| smcginnis | I've had the issue that if it's not in the sections for each series in the review inbox, it's invisible. :) | 16:19 |
| smcginnis | I can take a look at that after the meeting. | 16:20 |
| smcginnis | Quick looks, looks like it should be good. | 16:20 |
| ttx | hmm, it shows on my review inbox | 16:20 |
| ttx | (I have a "other" section in there :) | 16:20 |
| smcginnis | I tend to ignore the All Releases section since it's been a lot of -W and other patches. | 16:20 |
| smcginnis | I know I shouldn't, but that's what ends up happening for me. :) | 16:21 |
| ttx | I use a custom one | 16:21 |
| smcginnis | Different than https://tiny.cc/ReleaseInbox | 16:21 |
| ttx | https://bit.ly/2JeILIb | 16:21 |
| smcginnis | I'll see about adding an Other section to that ReleaseInbox one. | 16:21 |
| smcginnis | Anyway... | 16:22 |
| smcginnis | #topic Validate countdown email | 16:22 |
| *** openstack changes topic to "Validate countdown email (Meeting topic: releaseteam)" | 16:22 | |
| smcginnis | #link https://etherpad.openstack.org/p/relmgmt-weekly-emails | 16:22 |
| smcginnis | Line 246? | 16:22 |
| ttx | down to ~246 | 16:22 |
| smcginnis | All looks good to me. | 16:24 |
| smcginnis | Added the cycle highlights in there too. | 16:24 |
| smcginnis | I've heard a few teams talking about those already, so that's good. | 16:25 |
| diablo_rojo_phon | Thanks smcginnis was about to do that myself :) | 16:25 |
| diablo_rojo_phon | Yeah same. | 16:25 |
| smcginnis | Hopefully we will get a little more in by the time they are needed this time. | 16:25 |
| smcginnis | OK, I'll send that out tomorrow, my morning. | 16:26 |
| smcginnis | If anyone has any feedback on the draft, please update it before then. | 16:26 |
| smcginnis | #topic AOB | 16:26 |
| *** openstack changes topic to "AOB (Meeting topic: releaseteam)" | 16:26 | |
| smcginnis | Anything else to discuss in-meeting? | 16:26 |
| smcginnis | Folks are also encouraged to bring up things at any point in the channel too. | 16:27 |
| smcginnis | So if there's nothing else for now, I think we can wrap early. | 16:28 |
| smcginnis | Thanks everyone. Appreciate the time spent on reviews. Stay safe. | 16:28 |
| fungi | thanks smcginnis! | 16:28 |
| ttx | thanks! | 16:28 |
| smcginnis | #endmeeting | 16:28 |
| *** openstack changes topic to "OpenStack Release Managers office - Come here to discuss how to release OpenStack components - Logged at http://eavesdrop.openstack.org/irclogs/%23openstack-release/" | 16:28 | |
| openstack | Meeting ended Thu Mar 26 16:28:23 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:28 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/releaseteam/2020/releaseteam.2020-03-26-16.00.html | 16:28 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/releaseteam/2020/releaseteam.2020-03-26-16.00.txt | 16:28 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/releaseteam/2020/releaseteam.2020-03-26-16.00.log.html | 16:28 |
| diablo_rojo_phon | Thanks! | 16:28 |
| smcginnis | fungi: Looks like that time hit the error we wanted. | 16:30 |
| fungi | yeah, i'm jumping into it now | 16:31 |
| elod | ttx: i can take over the 'Update the feature list and allowed stable branch names in devstack-gate' task. seems trivial :) | 16:31 |
| ttx | elod: it is, and you'll learn tricks! | 16:31 |
| ttx | feel free to take it over | 16:31 |
| elod | ttx: ok :) | 16:32 |
| smcginnis | ttx: Do you have a gerrit dashboard creator file for your dashboard? | 16:32 |
| ttx | no, old-school manm | 16:32 |
| smcginnis | Cool, I can pull that out of the URL. | 16:33 |
| smcginnis | gerrit-dashboard-decompile | 16:33 |
| fungi | looks like tox clears out the contents of .tox/validate/tmp/ at some point | 16:36 |
| openstackgerrit | Merged openstack/releases master: Use governance_consistency.py for MembershipFreeze https://review.opendev.org/709748 | 16:44 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: [heat] Transition Rocky to EM https://review.opendev.org/709889 | 16:53 |
| smcginnis | fungi: So no signs of what might have happened? | 16:57 |
| smcginnis | I wonder if we could manually walk through the script process to recreate it. | 16:57 |
| fungi | i'm still trying to understand the job | 16:57 |
| fungi | it's running python globally i guess, not from within a virtualenv? | 16:57 |
| fungi | the traceback cites /usr/lib/python3.6/importlib/__init__.py | 16:57 |
| smcginnis | Hmm, that doesn't seem right. | 17:00 |
| smcginnis | If it installed the package into a venv, but then tries to import it using system packages, that could explain it. | 17:00 |
| smcginnis | Not why others work and only this one fails, but it would at least be something. | 17:01 |
| fungi | to be fair, i can't find the venv it might have installed it in either | 17:01 |
| fungi | `src/opendev.org/openstack/releases/.tox/validate/bin/pip list|grep xstatic` returns nothing | 17:01 |
| smcginnis | So to run the validate script, it needs to have install the releases repo. Can you see where and how that is being done? | 17:03 |
| smcginnis | Oh, running tox -e validate I guess, based on that path. | 17:04 |
| fungi | yeah, it's installed in the virtualenv there | 17:04 |
| smcginnis | So the validate job probably needs to do a 'pip install -e xstaticpackage' somewhere in there I guess. | 17:05 |
| fungi | well, openstack_releases/xstaticutils.py works around that | 17:06 |
| fungi | it adds the repo_dir to sys.path and then uses importlib.import_module() to pull it from there | 17:06 |
| ttx | remember that in the same change angular-bootstrap is just fine | 17:07 |
| smcginnis | ttx: OpenDev+PTG wasn't important enough for the "Upcoming Community Events"? :) | 17:07 |
| fungi | right, for now i'm trying to dissect enough of the tooling to understand what it's doing so i can isolate the failing call | 17:07 |
| ttx | smcginnis: we can't put a date on it right now | 17:07 |
| *** sean-k-mooney has joined #openstack-release | 17:07 | |
| sean-k-mooney | hi o/ | 17:07 |
| sean-k-mooney | just want to check something. os-vif has not merged any patches since m2 so can we just use the m2 release for the non client lib freeze | 17:08 |
| smcginnis | sean-k-mooney: Yep. That should be fine then. | 17:09 |
| sean-k-mooney | cool if we have to merge something ill request a release but i dont think that will happen | 17:09 |
| sean-k-mooney | thanks | 17:09 |
| smcginnis | sean-k-mooney: Sounds like a plan. Thanks! | 17:10 |
| smcginnis | fungi, ttx: Could it be an issue that there are multiple xstatic.pkg's, so one is overwriting the other? | 17:10 |
| ttx | smcginnis: not totally impossible | 17:11 |
| smcginnis | Not sure about what they are doing in the __init__.py files: https://opendev.org/openstack/xstatic-angular-gettext/src/branch/master/xstatic/pkg/__init__.py | 17:11 |
| smcginnis | But would think that would get overwritten to declare a different namespace. Maybe? | 17:12 |
| smcginnis | Then resulting in the import failure. | 17:12 |
| *** e0ne_ has quit IRC | 17:12 | |
| smcginnis | Could be easy to test by doing a patch separating those to just release one at a time. | 17:12 |
| *** rpittau is now known as rpittau|afk | 17:12 | |
| *** e0ne has joined #openstack-release | 17:12 | |
| fungi | smcginnis: ttx: we've seen that in the past with packages sharing a package namespace | 17:13 |
| fungi | like is it creating a xstatic/pkg/__init__.py which is overwriting the other? | 17:14 |
| smcginnis | If I read that right. | 17:14 |
| smcginnis | But the content should be the same. But what gets loaded may change. | 17:14 |
| fungi | i'm getting pulled away by local chores for 5-10 minutes, brb | 17:15 |
| smcginnis | Thanks fungi | 17:15 |
| smcginnis | I will put up some test patches separating those releases. | 17:16 |
| elod | ttx: should I -W this patch until next week or is it OK to land it sooner? https://review.opendev.org/#/c/715272/ | 17:17 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Release xstatic-bootstrap-scss 3.4.1.0 https://review.opendev.org/715274 | 17:20 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Release xstatic-angular-gettext 2.4.1.0 https://review.opendev.org/715275 | 17:21 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Release xstatic-angular-bootstrap 2.5.0.0 https://review.opendev.org/715276 | 17:21 |
| ttx | elod: I think it;s ok this way, we won;t approve it just now anyway | 17:24 |
| elod | ttx: cool, thx | 17:26 |
| ricolin | smcginnis, regrading https://review.opendev.org/#/c/709889, can I create a new rocky version now (with the hash I put in in #4) so we can release the newest rocky commits? | 17:29 |
| smcginnis | ricolin: Yes, that would be great. Then we could update the rocky-em patch on that. | 17:30 |
| *** dtantsur is now known as dtantsur|afk | 17:32 | |
| fungi | on my held test node i've made a tarball snapshot of the entire ~zuul/src tree so i don't contaminate it, and am rerunning `tox -e validate` to see if the error is reproducible when running manually | 17:32 |
| smcginnis | fungi: Looks like the separate patches are passing. | 17:32 |
| smcginnis | So it's some kind of overwriting clash. | 17:33 |
| smcginnis | I think. | 17:33 |
| fungi | ModuleNotFoundError: No module named 'xstatic.pkg.angular_gettext' | 17:33 |
| fungi | yeah, so i can recreate it rerunning in here at least | 17:33 |
| smcginnis | Oh good. | 17:33 |
| smcginnis | Any ideas how that can be fixed? | 17:34 |
| fungi | no, i'm not sure what's breaking it yet, though it sounds from your experiments like a conflict between multiple xstatic packages at least | 17:41 |
| fungi | i'm installing a text editor to be able to try a few alternatives | 17:41 |
| smcginnis | My concern is we can get around this, but are our end users going to hit this too when installing these packages together. | 17:41 |
| *** evrardjp has quit IRC | 17:49 | |
| *** evrardjp has joined #openstack-release | 17:49 | |
| fungi | yes, i don't think releasing them in separate single-project batches is an answer | 17:49 |
| fungi | just a means of helping narrow down the cause | 17:49 |
| fungi | i've removed the angular-bootstrap release to see what happens | 17:49 |
| fungi | ModuleNotFoundError: No module named 'xstatic.pkg.bootstrap_scss' | 17:49 |
| *** openstack has quit IRC | 17:49 | |
| *** openstack has joined #openstack-release | 17:50 | |
| *** ChanServ sets mode: +o openstack | 17:50 | |
| fungi | yeah, okay, so repeating it correctly, it also fails | 17:52 |
| fungi | ModuleNotFoundError: No module named 'xstatic.pkg.bootstrap_scss' | 17:52 |
| fungi | so it seems that no two of these three xstatic modules can have releases validated at the same time | 17:53 |
| fungi | leading me to believe it's a problem with openstack_releases/xstaticutils.py | 17:53 |
| smcginnis | Now can they be coinstalled and used in normal operation at the same time? Or is it the way that we try to do our validation? | 17:53 |
| fungi | though why folks are unable to reproduce it locally is still a question | 17:53 |
| fungi | well, we're not coinstalling them in this job | 17:54 |
| fungi | they're technically not being "installed" at all that i can see | 17:54 |
| fungi | at least not with pip and friends | 17:54 |
| smcginnis | Right, just plonked down and added to the sys path. | 17:54 |
| fungi | which leads me to suspect this is a namespace-related issue | 17:55 |
| fungi | whichever one is taking precedence in the sys.path is probably obscuring the other | 17:55 |
| fungi | by taking over the entire xstatic.pkg namespace | 17:55 |
| smcginnis | So we probably are OK working around it for now by releasing separately. | 17:57 |
| fungi | i'll hack up the xstaticutils.py module with some debugging to see if i can tell | 17:57 |
| smcginnis | Then we have an open issue of fixing how we handle things in xstaticutils so we can do this right. | 17:57 |
| smcginnis | Awesome, thanks! | 17:57 |
| fungi | i also need to work out why tox is clearing the tempdir | 18:01 |
| fungi | i don't see an option to tell it to not do that | 18:02 |
| noonedeadpunk | rocky bump is blocked by pygments thing:( | 18:03 |
| noonedeadpunk | in osa | 18:03 |
| fungi | sys.path: ['/home/zuul/src/opendev.org/openstack/releases/.tox/validate/bin', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/zuul/src/opendev.org/openstack/releases/.tox/validate/lib/python3.6/site-packages', '/home/zuul/src/opendev.org/openstack/releases', | 18:08 |
| fungi | '/home/zuul/src/opendev.org/openstack/releases/.tox/validate/tmp/releases-6veo5lh_/openstack/xstatic-bootstrap-scss'] | 18:08 |
| fungi | that's from a run where i removed the openstack/xstatic-angular-gettext release | 18:08 |
| fungi | and it failed saying it couldn't import xstatic.pkg.bootstrap_scss | 18:09 |
| fungi | so there's no xstatic-angular-bootstrap in the sys.path | 18:09 |
| fungi | implying we're validating them independently from a sys.path perspective | 18:10 |
| openstackgerrit | Rico Lin proposed openstack/releases master: Release new rocky version for heat repos https://review.opendev.org/715282 | 18:12 |
| openstackgerrit | Rico Lin proposed openstack/releases master: [heat] Transition Rocky to EM https://review.opendev.org/709889 | 18:12 |
| *** amoralej is now known as amoralej|off | 18:30 | |
| fungi | i've tweaked it to use shutil.copytree to make copies of the unpacked modules in /tmp so i can play with them more interactively | 18:36 |
| *** armstrong has quit IRC | 18:37 | |
| fungi | http://paste.openstack.org/show/791207/ | 18:42 |
| fungi | doesn't fail for me when trying to emulate what xstaticutils.py does with a copytree() of the repo_dir from immediately prior to the call which raises that exception | 18:43 |
| fungi | i wonder if there could be some kind of caching at work | 18:44 |
| *** diablo_rojo has joined #openstack-release | 18:47 | |
| smcginnis | fungi: Did you try to import each one? | 18:50 |
| fungi | yeah, importing them in a fresh python interpreter process seems to be fine | 18:51 |
| fungi | i'm fiddling around with importlib.reload() on the namespaces first | 18:52 |
| fungi | maybe they're being cached | 18:52 |
| fungi | score!!! | 18:55 |
| smcginnis | Yeah! | 18:55 |
| fungi | now to see if i can whittle it down a bit | 18:55 |
| smcginnis | So some obscure import_lib internal behavior? | 18:56 |
| fungi | still no clue why this doesn't surface in local runs | 18:56 |
| fungi | but it's another piece of the puzzle at least | 18:56 |
| smcginnis | stephenfin: Any ideas on https://review.opendev.org/#/c/715282/ | 19:17 |
| smcginnis | stephenfin: Heat sdist build fails now. Looks like it's another side effect of https://review.opendev.org/#/c/567559/4 | 19:17 |
| smcginnis | stephenfin: Past issues have been with it scanning branches before release notes were present. This one is failing on a recent tag though, so not sure why reno complains about it. | 19:18 |
| *** e0ne has joined #openstack-release | 19:18 | |
| *** e0ne has quit IRC | 19:39 | |
| openstackgerrit | Jeremy Stanley proposed openstack/releases master: Bust import caching of xstatic package namespaces https://review.opendev.org/715301 | 20:08 |
| fungi | smcginnis: ttx: ^ that's the simplest solution i've been able to arrive at which gets `tox -e validate` working for the change in question | 20:09 |
| smcginnis | Interesting. Well, if that does it, then good enough! | 20:10 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Release xstatic deliverables https://review.opendev.org/714682 | 20:17 |
| smcginnis | ttx: So here's an interesting thing. | 20:20 |
| *** e0ne has joined #openstack-release | 20:20 | |
| smcginnis | ttx: I updated the xstatic releases: https://review.opendev.org/#/c/714682/ | 20:21 |
| smcginnis | Since Akihiro is the author, the PTL approved check left +1. | 20:21 |
| smcginnis | ttx: Thinking that should actually check the Committer and not the Owner. | 20:21 |
| smcginnis | Not a big concern, but someone could modify a patch originally submitted by the PTL to be something completely different, and it would look like the PTL approves it. | 20:22 |
| *** e0ne has quit IRC | 20:29 | |
| *** e0ne has joined #openstack-release | 20:29 | |
| *** priteau has quit IRC | 20:41 | |
| *** e0ne has quit IRC | 20:42 | |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Release xstatic deliverables https://review.opendev.org/714682 | 20:44 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Bust import caching of xstatic package namespaces https://review.opendev.org/715301 | 20:55 |
| openstackgerrit | Jeremy Stanley proposed openstack/releases master: Bust import caching of xstatic package namespaces https://review.opendev.org/715301 | 21:08 |
| fungi | smcginnis: that should fix it | 21:08 |
| fungi | oh, you already pushed that, sorry | 21:08 |
| smcginnis | No worries, I should have checked. Figured you would have gotten pulled down another rabbit hole by now. ;) | 21:11 |
| *** jbadiapa has quit IRC | 21:23 | |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Release xstatic deliverables https://review.opendev.org/714682 | 21:26 |
| openstackgerrit | Merged openstack/releases master: Bust import caching of xstatic package namespaces https://review.opendev.org/715301 | 21:42 |
| openstackgerrit | Monty Taylor proposed openstack/releases master: Release 0.45.0 of openstacksdk https://review.opendev.org/715317 | 21:49 |
| mordred | smcginnis: ^^ | 21:49 |
| fungi | smcginnis: yeah, so i'm fairly confident what was happening is that the first xstatic module to get imported left stale implicit xstatic and xstatic.pkg namespaces, which caused subsequent imports of other modules to fail because they weren't found in the already cached namespaces | 22:00 |
| fungi | only reloading one or the other of those did not seem to solve the problem | 22:00 |
| fungi | reloading both seemed to work though | 22:00 |
| fungi | and they had to be explicitly imported and then reloaded. i tried just reloading and catching nameerror exceptions but that was insufficient | 22:01 |
| fungi | and only importing the namespaces each time but not reloading them also didn't help | 22:01 |
| mordred | diablo_rojo, ttx, smcginnis: if anybody is around who can approve releases - 715317 turns out to also be blocking opendev ops changes | 22:55 |
| *** slaweq has quit IRC | 22:59 | |
| *** prometheanfire has quit IRC | 23:04 | |
| *** slaweq has joined #openstack-release | 23:10 | |
| *** slaweq has quit IRC | 23:15 | |
| *** tetsuro has joined #openstack-release | 23:33 | |
| *** tetsuro has quit IRC | 23:35 | |
| *** tetsuro has joined #openstack-release | 23:37 | |
| *** vishalmanchanda has quit IRC | 23:39 | |
| *** tetsuro has quit IRC | 23:47 | |
| *** tosky has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!