| openstackgerrit | Clark Boylan proposed opendev/system-config master: Test if gerrit v3.2.8 specifically is happy with selenium https://review.opendev.org/c/opendev/system-config/+/787655 | 00:03 |
|---|---|---|
| zigo | clarkb: fungi: https://f90bbe801532b1321206-1f2089e8a25dfa4acbcd47153d041690.ssl.cf2.rackcdn.com/786772/3/check/puppet-openstack-integration-6-scenario001-tempest-debian-bullseye/f39a80e/ | 00:03 |
| clarkb | ianw: ^ thats another thought. If that passes testing then it would strongly imply something has very recently changed in gerrit | 00:03 |
| zigo | Where can we see the ansible facts?!? | 00:04 |
| clarkb | zigo: https://f90bbe801532b1321206-1f2089e8a25dfa4acbcd47153d041690.ssl.cf2.rackcdn.com/786772/3/check/puppet-openstack-integration-6-scenario001-tempest-debian-bullseye/f39a80e/zuul-info/host-info.debian-bullseye.yaml that seems to confirm my suspicion | 00:04 |
| clarkb | zigo: there ^ | 00:04 |
| clarkb | it says ansible_distribution_release is n/a | 00:04 |
| zigo | Ah yeah ... | 00:04 |
| zigo | Saw it. | 00:04 |
| zigo | Is there a way to override this? | 00:04 |
| fungi | i wonder if newer ansible has it correct | 00:04 |
| zigo | It probably will also be an issue when running the puppet thingy ... | 00:04 |
| clarkb | fungi: it seems to say it is reading from /etc/os-release | 00:05 |
| zigo | If I'm not mistaking, there's an ansible part involved there too. | 00:05 |
| fungi | oh | 00:05 |
| clarkb | ansible_distribution_file_path: /etc/os-release I read that to say "I sourced this info from here" | 00:05 |
| fungi | we might be able to create or overwrite /etc/os-release in the debian-minimal dib element? | 00:06 |
| clarkb | fungi: not a bad idea | 00:06 |
| zigo | $ cat /etc/os-release | grep PRETTY_NAME | 00:06 |
| zigo | PRETTY_NAME="Debian GNU/Linux bullseye/sid" | 00:06 |
| zigo | That's what I have (on my laptop...). | 00:06 |
| zigo | Mr. Ansible is being nasty ... it should know better ! :) | 00:07 |
| clarkb | hrm maybe it is a parsing issue in ansible then? | 00:07 |
| ianw | 104.130.70.62 is a bullseye node | 00:07 |
| clarkb | zigo: if you run ansible against your laptop to get facts you should be able to see what it does | 00:07 |
| zigo | I believe it should be confused by the "/sid" added at the end until bullseye is released. | 00:07 |
| ianw | yes, os-release on that node is the same as zigos | 00:08 |
| zigo | clarkb: How one may run ansible's facter? | 00:08 |
| clarkb | zigo: I think it is called the setup module in ansible so you do something like ansible -i localhost, -m setup ? | 00:08 |
| clarkb | VERSION_CODENAME=buster <- that is what buster says | 00:09 |
| clarkb | ianw's example bullseye node does not have a VERSION_CODENAME netry | 00:09 |
| zigo | ansible: error: the following arguments are required: pattern | 00:09 |
| clarkb | ansible -i localhost, localhost -m setup ? | 00:09 |
| clarkb | anyway I think the reason it breaks is there is no VERSION_CODENAME | 00:10 |
| clarkb | so it gets recorded as n/a | 00:10 |
| clarkb | VERSION_ID="10" and VERSION="10 (buster)" are also in my buster file | 00:11 |
| zigo | On my laptop it does work ! | 00:11 |
| clarkb | zigo: does your laptop have VERSION_* entries in /etc/os-release? | 00:11 |
| zigo | Nope. | 00:11 |
| clarkb | huh neat | 00:11 |
| ianw | 2.9.18 fails for me, installing a later one | 00:12 |
| clarkb | ianw: re Gerrit I've just noticed that system-config-run-review-3.2 isn't running against my change to force v3.2.8 across the board. We just seem to build an image there. Interesting. | 00:13 |
| zigo | http://paste.openstack.org/show/804734/ | 00:13 |
| clarkb | I'll push a new ps to that that forces the job to run too | 00:13 |
| zigo | That's what is on my laptop ... | 00:13 |
| *** hamalq has quit IRC | 00:13 | |
| ianw | ok i still get n/a with 2.10.8 | 00:14 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Test if gerrit v3.2.8 specifically is happy with selenium https://review.opendev.org/c/opendev/system-config/+/787655 | 00:14 |
| clarkb | now we should run the run job as well as the build | 00:14 |
| zigo | Ansible on Bullseye is 2.9.16+dfsg-1.1 | 00:14 |
| zigo | No Debian specific patches (I just checked). | 00:15 |
| zigo | Well, none relevant to that issue we're discussing ... | 00:15 |
| ianw | still fails with devel for me | 00:16 |
| * zigo got to go sleep | 00:18 | |
| zigo | 2:20 am here ... | 00:18 |
| zigo | Bye. | 00:18 |
| ianw | zigo: no worries, i'll have a poke and see what i can find | 00:19 |
| zigo | Thanks. | 00:21 |
| ianw | # lsb_release | 00:31 |
| ianw | No LSB modules are available. | 00:31 |
| ianw | that is i think the root cause here | 00:31 |
| fungi | interestingly, `lsb_release -a` probably works? | 00:52 |
| fungi | on my systems, `lsb_release` gives the same "No LSB modules are available." message but `lsb_release -a` reports (among other things) "Codename: bullseye" | 00:52 |
| ianw | fungi: indeed -- Codename:n/a | 01:23 |
| ianw | print('Codename:\t%s' % distinfo.get('CODENAME', 'n/a')) | 01:24 |
| ianw | i feel like there might be a recursive loop here :) | 01:24 |
| *** brinzhang0 has joined #opendev | 01:28 | |
| *** brinzhang_ has quit IRC | 01:31 | |
| *** slittle1 has quit IRC | 01:33 | |
| ianw | clarkb: well, https://review.opendev.org/c/opendev/system-config/+/787652 passed everything ... | 01:37 |
| ianw | i'll run a recheck and see | 01:37 |
| *** xinliang has joined #opendev | 01:48 | |
| fungi | interestingly, i have VERSION_CODENAME=bullseye in /etc/os-release but technically i'm on sid | 02:00 |
| fungi | base-files (11.1) unstable; urgency=medium | 02:02 |
| fungi | Add VERSION_CODENAME to os-release. | 02:03 |
| fungi | from /usr/share/doc/base-files/changelog.gz | 02:03 |
| fungi | looks like 11 is what's in bullseye right now, not 11.1 yet | 02:04 |
| fungi | https://tracker.debian.org/pkg/base-files | 02:05 |
| fungi | Migration status for base-files (11 to 11.1): BLOCKED: Needs an approval (either due to a freeze, the source suite or a manual hint) | 02:05 |
| fungi | so i think it's likely to work once base-files 11.1 migrates from sid to bullseye | 02:06 |
| ianw | i think i see what's going on here, it is a recursive problem | 02:53 |
| ianw | # apt-cache policy | 02:55 |
| ianw | 500 https://mirror.dfw.rax.opendev.org/debian bullseye/main amd64 Packages | 02:56 |
| ianw | release o=Debian,n=bullseye,c=main,b=amd64 | 02:56 |
| ianw | origin mirror.dfw.rax.opendev.org | 02:56 |
| ianw | i don't know why, but this does not have a "label" field | 02:56 |
| ianw | http://httpredir.debian.org/debian/dists/bullseye/Release v https://mirror.regionone.limestone.opendev.org/debian/dists/bullseye/Release | 02:59 |
| ianw | this bug -> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=845651 explains why lsb_release won't look at /etc/debian_version for lsb_release -c | 02:59 |
| openstack | Debian bug 845651 in lsb-release "lsb-release: --codename returns n/a on stretch without apt sources configured" [Important,Open] | 02:59 |
| ianw | it falls through to looking at "apt-cache policy" | 03:00 |
| ianw | *however*, it will not parse the line "release o=Debian,n=bullseye,c=main,b=amd64" because it doesn't have "l=Debian" | 03:00 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: reprepro : add labels https://review.opendev.org/c/opendev/system-config/+/787661 | 03:24 |
| *** ykarel|away has joined #opendev | 04:18 | |
| openstackgerrit | Xinliang Liu proposed opendev/system-config master: Enable openEuler mirroring https://review.opendev.org/c/opendev/system-config/+/784874 | 04:29 |
| xinliang | ianw, please help to review openEuler mirroring patch thanks, have updated it : https://review.opendev.org/c/opendev/system-config/+/784874 | 04:34 |
| *** vishalmanchanda has joined #opendev | 04:36 | |
| *** xinliang has quit IRC | 04:55 | |
| *** ykarel_ has joined #opendev | 04:55 | |
| *** ysandeep|away is now known as ysandeep | 04:58 | |
| *** ykarel|away has quit IRC | 04:58 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Set bullseye https://review.opendev.org/c/openstack/diskimage-builder/+/787665 | 05:07 |
| *** snapdeal has joined #opendev | 05:11 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: debian-minimal: Set bullseye version https://review.opendev.org/c/openstack/diskimage-builder/+/787665 | 05:12 |
| ianw | fungi / zigo : an examination of what i think is going on http://lists.opendev.org/pipermail/service-discuss/2021-April/000222.html ... :/ | 05:12 |
| *** ykarel__ has joined #opendev | 05:23 | |
| *** ykarel_ has quit IRC | 05:23 | |
| *** priteau has joined #opendev | 05:29 | |
| *** ykarel_ has joined #opendev | 05:41 | |
| *** ykarel_ is now known as ykarel | 05:43 | |
| *** ykarel__ has quit IRC | 05:43 | |
| *** brinzhang_ has joined #opendev | 06:12 | |
| *** brinzhang0 has quit IRC | 06:15 | |
| *** slaweq has joined #opendev | 06:20 | |
| *** ralonsoh has joined #opendev | 06:26 | |
| *** sboyron has joined #opendev | 06:30 | |
| *** amoralej|off is now known as amoralej | 06:43 | |
| ianw | i think the dib gate is hung up on something to do with xenial+glean+pbr but i don't have time to diagnose it now | 06:44 |
| *** eolivare has joined #opendev | 07:00 | |
| *** ykarel has quit IRC | 07:07 | |
| *** ykarel has joined #opendev | 07:10 | |
| *** rpittau|afk is now known as rpittau | 07:12 | |
| *** ykarel_ has joined #opendev | 07:12 | |
| *** ykarel has quit IRC | 07:15 | |
| *** whoami-rajat_ has joined #opendev | 07:17 | |
| *** whoami-rajat_ is now known as whoami-rajat | 07:25 | |
| zigo | ianw: Thanks, that's a very interesting read. | 07:32 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Roles to create, cleanup and promote snapshots in ec2 https://review.opendev.org/c/zuul/zuul-jobs/+/787677 | 07:41 |
| *** snapdeal has quit IRC | 07:42 | |
| *** hrw has joined #opendev | 07:43 | |
| *** ykarel_ has quit IRC | 07:45 | |
| *** andrewbonney has joined #opendev | 07:49 | |
| *** tosky has joined #opendev | 07:53 | |
| *** jpena|off is now known as jpena | 07:54 | |
| *** ykarel has joined #opendev | 07:55 | |
| *** ykarel has quit IRC | 08:04 | |
| *** ykarel has joined #opendev | 08:06 | |
| *** hemanth_n has joined #opendev | 08:16 | |
| *** dtantsur|afk is now known as dtantsur | 08:17 | |
| frickler | infra-root: I received feedback from the neutron team that they moved back from meetpad to zoom because of repeated audio problems http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/latest.log.html#t2021-04-23T08:15:01 I also saw some note in the nova channel earlier that chinese participants still seem to have issues | 08:56 |
| frickler | http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2021-04-19.log.html#t2021-04-19T15:51:24 | 08:57 |
| frickler | so maybe we can try to do some more testing and see how to improve this for the next event | 08:58 |
| *** ysandeep is now known as ysandeep|lunch | 09:11 | |
| *** dtantsur is now known as dtantsur|bbl | 09:30 | |
| *** ykarel_ has joined #opendev | 09:39 | |
| *** ykarel has quit IRC | 09:41 | |
| *** ykarel_ is now known as ykarel | 09:42 | |
| *** ysandeep|lunch is now known as ysandeep | 09:55 | |
| *** ykarel_ has joined #opendev | 10:15 | |
| *** ykarel has quit IRC | 10:18 | |
| *** ykarel__ has joined #opendev | 10:31 | |
| *** ykarel_ has quit IRC | 10:33 | |
| *** ykarel_ has joined #opendev | 10:36 | |
| *** ykarel_ is now known as ykarel | 10:37 | |
| *** ykarel__ has quit IRC | 10:39 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| fungi | we did have multiple chinese participants in the opendev session | 11:33 |
| *** snapdeal has joined #opendev | 11:45 | |
| openstackgerrit | Ananya proposed opendev/elastic-recheck master: WIP: Run elastic-recheck container https://review.opendev.org/c/opendev/elastic-recheck/+/729623 | 12:10 |
| *** hemanth_n has quit IRC | 12:25 | |
| *** dtantsur|bbl is now known as dtantsur | 12:29 | |
| *** amoralej is now known as amoralej|lunch | 12:33 | |
| *** jpena|lunch is now known as jpena | 12:33 | |
| openstackgerrit | Merged opendev/system-config master: reprepro : add labels https://review.opendev.org/c/opendev/system-config/+/787661 | 12:47 |
| *** priteau has quit IRC | 12:48 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Roles to create, cleanup and promote snapshots in ec2 https://review.opendev.org/c/zuul/zuul-jobs/+/787677 | 12:48 |
| *** slaweq has quit IRC | 12:53 | |
| *** slaweq has joined #opendev | 12:53 | |
| *** priteau has joined #opendev | 12:54 | |
| *** slaweq has quit IRC | 12:56 | |
| *** slaweq has joined #opendev | 12:56 | |
| *** amoralej|lunch is now known as amoralej | 13:08 | |
| openstackgerrit | daniel.pawlik proposed opendev/lodgeit master: Fixed issue related to newer Pillow lib https://review.opendev.org/c/opendev/lodgeit/+/787734 | 13:09 |
| *** dhellmann has quit IRC | 13:19 | |
| *** dhellmann has joined #opendev | 13:25 | |
| *** vishalmanchanda has quit IRC | 13:26 | |
| *** ykarel_ has joined #opendev | 13:36 | |
| *** ykarel has quit IRC | 13:39 | |
| *** priteau has quit IRC | 13:43 | |
| *** mfixtex has joined #opendev | 13:43 | |
| *** priteau has joined #opendev | 13:46 | |
| *** rpittau is now known as rpittau|afk | 14:15 | |
| *** ykarel_ is now known as ykarel | 14:26 | |
| clarkb | I've rechecked https://review.opendev.org/c/opendev/system-config/+/787629 to keep moving on the inmotion mirror swap | 14:33 |
| clarkb | ianw's investigation was unable to reproduce the failed selenium runs against review so that should pass | 14:34 |
| clarkb | if it doesn't I guess I hold the jobs on that change | 14:34 |
| *** rsubrama has joined #opendev | 14:35 | |
| clarkb | frickler: fungi: re meetpad it seems that the experience depends a lot on the client side. Performance, browser, etc | 14:37 |
| clarkb | I think we still recommend chrom* and you may need to ensure you aren't trying to compile software or do other cpu intensive work at the same time | 14:37 |
| clarkb | if users start having problems then you can improve things by disabling videos | 14:37 |
| clarkb | (but as far as we've been able to tell the bottlenecks haven't been the server side as much as the client side) | 14:38 |
| *** lbragstad_ has joined #opendev | 14:38 | |
| *** ysandeep is now known as ysandeep|away | 14:40 | |
| frickler | clarkb: can a single user disable receiving videos? or would every participant have to turn their camera off | 14:42 |
| *** lbragstad has quit IRC | 14:42 | |
| clarkb | that is a good question. I don't know the answer to that | 14:43 |
| clarkb | I think the moderator can mute audio and turn off video for other participants globally within a call | 14:44 |
| frickler | also I'm not sure whether that would really be an option for teams if they can work just fine from their pov with zoom | 14:44 |
| frickler | maybe there is a server option to limit video bandwidth used by limiting resolution and/or frame rate? | 14:45 |
| clarkb | ya I think this is a major reason why zoom doesn't use webrtc and ships a webasm bundle that does all the conferencing instead of webrtc in the browser | 14:45 |
| clarkb | frickler: looks like you can limit video resolution in the config | 14:45 |
| clarkb | not sure what that looks like with the upstream images, but jitsi meet appears to support it at least | 14:46 |
| *** ykarel is now known as ykarel|away | 14:50 | |
| *** snapdeal has quit IRC | 14:52 | |
| *** mfixtex has quit IRC | 14:55 | |
| *** ykarel|away has quit IRC | 14:59 | |
| *** hashar has joined #opendev | 15:03 | |
| clarkb | fungi: if you get a chance https://review.opendev.org/c/opendev/git-review/+/787297 is one approach to fixing python3.5 testing for pbr | 15:07 |
| clarkb | I wonder if that breaks self testing in the unittests though | 15:07 |
| clarkb | it mustn't if we were relying on easy_install previously though | 15:08 |
| clarkb | the --no-thin option is blocking on that | 15:08 |
| clarkb | another option would be to stop testing 3.5 and just try to avoid adding specific >=3.6 behavior (made easier by testing 2.7) | 15:08 |
| *** hashar has quit IRC | 15:10 | |
| *** hashar has joined #opendev | 15:10 | |
| *** hashar has quit IRC | 15:10 | |
| *** hashar_ has joined #opendev | 15:10 | |
| fungi | clarkb: frickler: the lowest "performance" setting in jitsi-meet is audio-only sending/receiving, it's just not clearly labelled that way | 15:11 |
| fungi | individual participants can set it | 15:11 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: intercept-job -- self-service SSH access https://review.opendev.org/c/zuul/zuul-jobs/+/679306 | 15:14 |
| fungi | clarkb: on 787297 how is the requirements.txt addition working around things? is it because of the siblings install phase explicitly installing requirements without installing the project itself? | 15:19 |
| clarkb | fungi: it is because tox in general (including siblings) installs requirements before installing the project | 15:19 |
| clarkb | at least the way we typically use it | 15:19 |
| fungi | got it. so explicit project installation in a virtualenv on that platform would still fail, but tox is doing an end-run around setup_requires | 15:20 |
| clarkb | yup | 15:20 |
| *** rsubrama has quit IRC | 15:29 | |
| clarkb | fungi: actuall I'm going to WIP that change | 15:33 |
| clarkb | or no nevermind | 15:33 |
| clarkb | suddenly I decided it really should be a test-requirement, but I don't think there is much harm in the change as is so nevermind | 15:33 |
| clarkb | but if you think it should be a test-requirement I can move it | 15:34 |
| *** amoralej is now known as amoralej|off | 15:37 | |
| fungi | well, is it also a runtime requirement? | 15:39 |
| fungi | looks like it's not | 15:39 |
| fungi | so yeah, maybe test-requirements.txt would make more sense | 15:39 |
| fungi | clarkb: thanks, i unapproved it for now | 15:40 |
| clarkb | ok I'll update it momentarily | 15:41 |
| openstackgerrit | Merged zuul/zuul-jobs master: intercept-job -- self-service SSH access https://review.opendev.org/c/zuul/zuul-jobs/+/679306 | 15:42 |
| openstackgerrit | Clark Boylan proposed opendev/git-review master: Install PBR explicitly https://review.opendev.org/c/opendev/git-review/+/787297 | 15:43 |
| clarkb | fungi: ^ that should still function but be a bit more clear | 15:43 |
| fungi | cool, let's see if it still fixes the tests | 15:43 |
| clarkb | btw not sure I'll get to the zk stuff today since I want to finish up the inmotion mirror things and clean up networking there since that is in a half done state | 15:45 |
| clarkb | may be for the best to avoid having zk in a half cluster state over the weekend | 15:46 |
| clarkb | fungi: the gerrit testing has been looking happy though so I suspect we're still good ot do that gerrit update today | 15:46 |
| clarkb | that will get us 3.2.8 as well as your jeepyb fixes | 15:46 |
| fungi | awesome | 15:47 |
| *** mlavalle has joined #opendev | 15:47 | |
| *** hashar_ has quit IRC | 15:53 | |
| corvus | i'd like to restart zuul today to pick up the global repo state patches | 15:55 |
| corvus | last time we saw a behavior change with this (the fast-forward stuff); we think that's addressed | 15:55 |
| clarkb | corvus: that requires a full restart too as it impacts the client side of the mergers/executors too? | 15:56 |
| clarkb | (it definitely requires the executors and mergers at least) | 15:56 |
| corvus | clarkb: yes -- but i've been doing full restarts recently anyway (since everything we've done recently touches the scheduler) | 15:56 |
| corvus | things look generally quiet on the third column of the status page | 15:57 |
| corvus | so i think i'll do that now | 15:57 |
| fungi | sounds good. thanks | 15:58 |
| corvus | ran the pull playbook; restarting now | 15:59 |
| fungi | we're also planning to do a gerrit restart later today | 15:59 |
| fungi | once ptg sessions are finished | 16:00 |
| clarkb | yup and I'm trying to get the new inmotion cloud reconfigured networking wise to get as many IPs out of it as we can | 16:01 |
| corvus | re-enqueing | 16:05 |
| corvus | #status log restarted zuul on commit d4c7d293609a151f0c58b2d8a6fe4ac2817ee501 to pick up global repo state changes | 16:05 |
| openstackstatus | corvus: finished logging | 16:05 |
| clarkb | jobs are running. Don't see any that have managed to finish already | 16:08 |
| clarkb | in another minute or two we should have some results though | 16:09 |
| clarkb | the first failure appears to be a legit linting error so thats good | 16:12 |
| clarkb | we have a number of successes now too | 16:12 |
| corvus | and failures; a good healthy mix! | 16:13 |
| corvus | all done | 16:19 |
| clarkb | fungi: huh the py35 job failed with pbr in test-requirements | 16:22 |
| clarkb | I really didn't expect that /me looks more closely at the tox config | 16:22 |
| corvus | clarkb: that's pre zuul restart right? | 16:22 |
| clarkb | corvus: no, after | 16:22 |
| clarkb | corvus: https://zuul.opendev.org/t/opendev/build/d46f18619895406f92702c0af745d20c | 16:23 |
| clarkb | fwiw it seems to have checked out the correct commit based on the log so I don't think the failure is due to the restart | 16:24 |
| clarkb | likely the failure is due to the move from requirements.txt to test-requirements.txt and I don't undersatnd some subtle detail in why that makes a difference here | 16:24 |
| corvus | passed for all the other vers | 16:24 |
| clarkb | ya its a python3.5 specific issue we are trying to fix | 16:24 |
| clarkb | tl;dr is pypi.org is requiring sni, for some reason xenial python 3.5 doesn't do sni even though pypi.org claims it should | 16:25 |
| clarkb | this causes easy_install to fail when trying to install setup_requires for git-review (specifically pbr) | 16:25 |
| *** hamalq has joined #opendev | 16:27 | |
| clarkb | fungi: it looks like tox runs sdist-make before installing deps. I'm beginning to wonder if the earlier successes were simply lucky and hit pypi.org when they were not doing a brownout | 16:28 |
| *** hamalq_ has joined #opendev | 16:33 | |
| openstackgerrit | Clark Boylan proposed opendev/git-review master: Install PBR explicitly https://review.opendev.org/c/opendev/git-review/+/787297 | 16:33 |
| clarkb | maybe that will work (using tox's skipsdist flag | 16:34 |
| *** hamalq has quit IRC | 16:36 | |
| fungi | could it be that we preinstall requirements.txt but maybe not test.requirements.txt? | 16:40 |
| clarkb | fungi: no, at least from local testing it is always the sdist-make step first then it installs requirements and test-requirements in one command | 16:41 |
| clarkb | I'm beginning to think the issue is the brownouts not being a 100% failure | 16:41 |
| fungi | could be, i'm also a fan of using skipsdist | 16:41 |
| clarkb | if I'm reading this correctly the only reason to not skipsdist is the ensure that you can make an sdist? | 16:45 |
| clarkb | it doesn't provide any functional benefit to the installation of the project later which can be done via what i think is the make a wheel and install that step | 16:46 |
| fungi | there was a time when we explicitly did sdist build in pep8 jobs to cover that case | 16:48 |
| fungi | heh, looks like we meant to do it in bindep: https://opendev.org/opendev/bindep/src/branch/master/tox.ini#L4 | 16:51 |
| fungi | typing is hard | 16:51 |
| fungi | meant to use skipsdist=true i mean | 16:51 |
| clarkb | I made the same typo when testing things for git-review not that long ago. noticed it didn't work and remembered the 's' at that point | 16:53 |
| *** jpena is now known as jpena|off | 16:53 | |
| *** eolivare has quit IRC | 16:55 | |
| fungi | #status log Removed pypi.org address override on mirror01.dfw.rax.opendev.org now that v6 routing between them works again | 16:58 |
| openstackstatus | fungi: finished logging | 16:58 |
| *** andrewbonney has quit IRC | 17:00 | |
| openstackgerrit | Jeremy Stanley proposed opendev/bindep master: Correctly skip sdist building in tox.ini https://review.opendev.org/c/opendev/bindep/+/787781 | 17:00 |
| clarkb | fungi: the failures now appear to be related to actual test failures | 17:10 |
| fungi | cool | 17:10 |
| clarkb | makes me wonder if not sdisting breaks the test suite some how | 17:11 |
| clarkb | which git-review is failing? so ya maybe | 17:12 |
| clarkb | in my local testing with py38 it installed git-review==2.0.1.dev3 | 17:13 |
| clarkb | but in the test job it appears to have not installed git-review at all | 17:13 |
| * clarkb git cleans and tries again | 17:15 | |
| clarkb | huh ya if I clean out .tox (I used -r anyway) and dist/ then I don't get a git-review installed | 17:17 |
| openstackgerrit | Clark Boylan proposed opendev/git-review master: Install PBR explicitly https://review.opendev.org/c/opendev/git-review/+/787297 | 17:18 |
| clarkb | that installs . as a deps | 17:18 |
| clarkb | not sure if that will work, but lets find out | 17:18 |
| fungi | i use skipsdist in other projects and don't need to explicitly install "." | 17:22 |
| fungi | also usedevelop factors into that | 17:22 |
| *** dtantsur is now known as dtantsur|afk | 17:22 | |
| fungi | i turn on usedevelop=True in the default testenv and then set it false in testenvs which don't need the project installed | 17:23 |
| *** ralonsoh has quit IRC | 17:23 | |
| *** lbragstad_ is now known as lbragstad | 17:23 | |
| fungi | e.g. linters generally don't need the project package installed into the testenv | 17:24 |
| clarkb | ah usedevelop would probably work for this, let me test that | 17:25 |
| *** mfixtex has joined #opendev | 17:34 | |
| fungi | though in my own projects i also have an actual [testenv:dist] where i explicitly `python setup.py bdist_wheel sdist` and then `twine check --strict {toxinidir}/dist/*` | 17:36 |
| fungi | so i'm directly testing dist creation, and that doubles as part of my release process too | 17:36 |
| *** slaweq has quit IRC | 17:38 | |
| clarkb | ok usedevelop has to go in a testenv not a tox section | 17:42 |
| clarkb | that took far too much time to understand | 17:42 |
| openstackgerrit | Clark Boylan proposed opendev/git-review master: Install PBR explicitly https://review.opendev.org/c/opendev/git-review/+/787297 | 17:42 |
| clarkb | I think ^ that should work | 17:42 |
| hrw | https://marcin.juszkiewicz.com.pl/2021/04/22/sometimes-one-tweet-is-enough/ - thanks to ianw, kevinz, clarkb | 17:43 |
| fungi | hrw: awesome! | 17:44 |
| hrw | fungi: I was surprised. | 17:45 |
| hrw | fungi: as all I did was just writing frustration and then Cc: proper people | 17:46 |
| *** cloudnull has quit IRC | 18:11 | |
| openstackgerrit | Merged opendev/system-config master: Add mirror02 to inmotion https://review.opendev.org/c/opendev/system-config/+/787629 | 18:12 |
| clarkb | fungi: the latest ps on the git-review change passed on py35 | 18:15 |
| fungi | right on, reapproving | 18:19 |
| *** auristor has quit IRC | 18:40 | |
| *** auristor has joined #opendev | 18:43 | |
| clarkb | looks like LE job on the new mirror failed again. This time I was careful to add the handler though :/ I'll see why that happened but then likely rerun the service-mirror.yaml playbook against it like I did with the last one | 18:51 |
| clarkb | it failed with The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'letsencrypt_certcheck_domains' trying to update the domain list for certcheck. It also seems to have failed doing LE on nb03 which amybe triggered teh cacti issue for certcheck? | 18:53 |
| clarkb | running the mirror playbook now | 18:53 |
| fungi | that's a confusing error condition | 18:54 |
| openstackgerrit | Merged opendev/git-review master: Install PBR explicitly https://review.opendev.org/c/opendev/git-review/+/787297 | 18:55 |
| clarkb | I've rechecked the --no-thin change | 18:56 |
| fungi | looks like we have 16GB free for the rootfs on review.o.o, and the docker image list has a opendevorg/gerrit 3.2 created 19 hours ago. current image we've been running is df1c7ac8dcae (no tag) from 2 months ago | 18:57 |
| fungi | i did a `docker-compose pull` which doesn't seem to have updated the image list at all | 18:58 |
| fungi | i guess we can restart the gerrit container at 19:00 utc | 18:58 |
| clarkb | 57df55aec1eb is the new image | 18:58 |
| clarkb | df1c7ac8dcae seems to be what we have been running on | 18:59 |
| fungi | yep | 18:59 |
| fungi | status notice The Gerrit service on review.openstack.org is being restarted to pick up some updates, and should be available again momentarily | 18:59 |
| fungi | does that look good? | 18:59 |
| clarkb | lgtm | 19:00 |
| fungi | #status notice The Gerrit service on review.openstack.org is being restarted to pick up some updates, and should be available again momentarily | 19:00 |
| openstackstatus | fungi: sending notice | 19:00 |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted to pick up some updates, and should be available again momentarily | 19:00 | |
| fungi | it's starting back up now | 19:00 |
| clarkb | some complaints about not being able to compress jvm logs | 19:01 |
| clarkb | and the plugin manager complaint that we expect | 19:01 |
| fungi | yeah, i've seen both on previous restarts | 19:01 |
| clarkb | seems to be an issue renaming the jvm logs from .log to .log.gz, weird | 19:02 |
| clarkb | but also unlikely to be a significant issue | 19:02 |
| fungi | might be a file conflict | 19:02 |
| clarkb | I can load my dashboard and a change | 19:03 |
| fungi | maybe old compressed file with different permissions? | 19:03 |
| clarkb | ah ya that could be | 19:03 |
| fungi | yeah, it's loading for me now too | 19:03 |
| clarkb | the zuul summary still works | 19:03 |
| clarkb | and the reported version is about what I expected so yay | 19:03 |
| openstackstatus | fungi: finished sending notice | 19:03 |
| clarkb | https://mirror02.iad3.inmotion.opendev.org/ also looks good now after running service-mirror against it | 19:04 |
| clarkb | fungi: can you review https://review.opendev.org/c/opendev/zone-opendev.org/+/787630 as the next change in that series when you get a moment? | 19:04 |
| * fungi just realized he wrote "review.openstack.org instead of opendev in the status notice. oh well | 19:04 | |
| clarkb | then I can tail apache logs on the other mirror and wait for it to idle, delete it, then delete the network stuff we don't need anymore then bump the quota back to 8 | 19:05 |
| clarkb | fungi: oh I didn't catch it either, the .o.o's all run together sometimes | 19:05 |
| *** yoctozepto has left #opendev | 19:09 | |
| *** yoctozepto has joined #opendev | 19:11 | |
| clarkb | I guess I shouldn't delete the old host until the cleanup change lands to remove it from the inventory. But we should be able to safely approve the cleanup change, then wait for the server to idle before deleting it (since the mriror configs won't chagne in the interim) | 19:13 |
| fungi | yup | 19:13 |
| clarkb | fungi: that means https://review.opendev.org/c/opendev/system-config/+/787631 is also ready for review now | 19:13 |
| fungi | also gives nodes which were hitting the old mirror some time to notice the dns update | 19:13 |
| corvus | review.openstack.org liiiiiives! | 19:14 |
| fungi | in our memories | 19:14 |
| fungi | including my finger memory apparently | 19:14 |
| clarkb | fungi: skimming my todo list the other thing I have been trying to not let fall behind is the openstack project reparenting. Any idea if we are ready to move forward on those? | 19:15 |
| clarkb | I have suspect not as people were distracted by the ptg this week | 19:16 |
| clarkb | s/have/half/ | 19:16 |
| fungi | waiting to hear back from elod that things were good on the first batch we approved | 19:16 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Flip mirror.iad3.inmotion to mirror02.iad3.inmotion https://review.opendev.org/c/opendev/zone-opendev.org/+/787630 | 19:16 |
| fungi | testing of those has probably been delayed by ptg activity | 19:16 |
| clarkb | I'm tailing the mirror_443_access.log on each of the mirrors to get a sense for when load has shifted | 19:17 |
| clarkb | I wonder remove the old server until that happens and https://review.opendev.org/c/opendev/system-config/+/787631 has merged | 19:17 |
| openstackgerrit | Merged opendev/git-review master: Add option for disabling thin pushes https://review.opendev.org/c/opendev/git-review/+/785723 | 19:18 |
| clarkb | I wonder remove? | 19:18 |
| clarkb | I won't remove | 19:18 |
| clarkb | fungi: ^ want to do a git-review release on a friday ? | 19:18 |
| clarkb | --no-thin is the only consequential change (manpage was updated as was the pbr test fix) | 19:19 |
| fungi | i suppose we can, or we could wait for monday | 19:21 |
| *** Eighth_Doctor is now known as Conan_Kudo | 19:27 | |
| clarkb | I'm not actually in a rush | 19:28 |
| clarkb | but having it out before we next have to help troubleshoot that issue would be godo so sooner than later | 19:28 |
| corvus | i feel like git-review is an ok thing to release on fri | 19:28 |
| *** Conan_Kudo is now known as Eighth_Doctor | 19:29 | |
| clarkb | probably https://review.opendev.org/c/opendev/system-config/+/787631 is more urgent if I can get reviews on that to make the old mirror deletable | 19:29 |
| clarkb | I see both mirrors serving requests now fwiw | 19:29 |
| clarkb | neat I see requests to both from the same client IP at roughly the same times | 19:32 |
| fungi | i can work up a git-review release in a few minutes | 19:36 |
| clarkb | thanks for the reviews! | 19:41 |
| clarkb | fungi: is https://review.opendev.org/c/opendev/system-config/+/787661 expected to be sufficient for fixing the ansible facts issue on bullseye? your suggesting to install base-files from unstable implies no but ianw's email seems to imply that will make lsb_release work and then ansible will be happy? | 19:44 |
| clarkb | oh wait I see its the chicken and egg issue between update and knowing what your repos are | 19:46 |
| *** cloudnull has joined #opendev | 19:47 | |
| fungi | yeah, the reprepro change was worth a shot | 19:51 |
| fungi | and arguably more correct than what we had | 19:51 |
| clarkb | https://review.opendev.org/c/openstack/diskimage-builder/+/787665 is the other related fix which failed in testing (my comment calls out how). I'm not quite sure why the test failed though | 19:52 |
| clarkb | aha that also runs outside of the chroot | 19:53 |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder master: debian-minimal: Set bullseye version https://review.opendev.org/c/openstack/diskimage-builder/+/787665 | 19:54 |
| clarkb | lets see if that is happier | 19:54 |
| *** stevebaker has quit IRC | 19:55 | |
| *** mfixtex has quit IRC | 20:23 | |
| openstackgerrit | Merged opendev/system-config master: Cleanup mirror01.iad3.inmotion https://review.opendev.org/c/opendev/system-config/+/787631 | 20:35 |
| *** elod has quit IRC | 20:40 | |
| clarkb | The last request to ^ was almost an hour ago. I'll give it to the hour mark then proceed wtih cleanup | 20:40 |
| *** elod has joined #opendev | 20:42 | |
| *** sboyron has quit IRC | 20:45 | |
| clarkb | alright only new requests are from crawlers. I'm going to clean up the mirror now | 20:50 |
| clarkb | ok mirror has been cleaned up as has the extra network/subnet/router that consumed and IP and the floating IP that the mirror was using. I've bumped the quota back up and the max-servers: 8 value should be used now | 20:58 |
| *** whoami-rajat has quit IRC | 20:59 | |
| fungi | clarkb: git-review 2.1.0 because --no-thin is technically a new feature, or 2.0.1 because it's an off-by-default bug fix/workaround? | 21:01 |
| clarkb | I think 2.1.0 is more correct | 21:01 |
| fungi | i guess we did call it a feature in the release notes too | 21:02 |
| fungi | cool, working on that now | 21:02 |
| clarkb | yup we should do 2.1.0 for sure given that | 21:04 |
| fungi | okay, tag pushed, i'll send out a release announcement in a bit once that's uploaded | 21:07 |
| clarkb | sounds good, thanks! | 21:08 |
| clarkb | https://pypi.org/project/git-review/#files pypi has the new release | 21:47 |
| fungi | yup, will write up something for service-announce shortly | 21:55 |
| fungi | juggling with evening chores | 21:55 |
| zigo | clarkb: fungi: Do you guys know why https://review.opendev.org/c/openstack/diskimage-builder/+/787665 is failing? | 21:55 |
| clarkb | zigo: I don't know why the latest patchset is failing, but I think I figured out why the first patchset failed and that is why I pushed the new patchset | 21:56 |
| zigo | Oh ok. :) | 21:56 |
| fungi | also we're setting the debian label on the mirrors in reprepro now | 21:57 |
| zigo | fungi: So that should fix it if I retry tomorrow, with a new fresh image? | 21:57 |
| clarkb | zigo: I think the assumption is we need the dib hack | 21:58 |
| clarkb | and seems unlikely we'll get that deployed by tomorrow | 21:58 |
| fungi | http://mirror.dfw.rax.opendev.org/debian/dists/bullseye/Release | 21:58 |
| zigo | ok | 21:58 |
| fungi | yeah, the one thing which dawned on me might solve the chicken-and-egg problem from the mirror end is that we build images using one of our mirrors | 21:59 |
| fungi | so it should be picking up that label at chroot build time now | 21:59 |
| fungi | so it might work with the next image update | 21:59 |
| fungi | zigo: also, i don't suppose you have any idea how long it will take base-files 11.1 to migrate to bullseye? | 22:00 |
| fungi | that will make workarounds like 787665 moot, i think | 22:00 |
| clarkb | the fedora-32 functioanl build job hit a nova api error trying to build an unmanaged VM outside of nodepool. I think we can recheck for this one. I'm checking the xenial job now | 22:00 |
| zigo | fungi: Likely, it will migrate the day of the release, no? | 22:01 |
| fungi | i honestly can't remember if it gets held until release or just into the freeze | 22:01 |
| zigo | On the release list, they talked about release Bullseye in one week-end of may, but I'm not sure if a day is decided. | 22:01 |
| fungi | yeah, last i saw there still wasn't a day set | 22:02 |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder master: Install pbr before glean to address SNI issues https://review.opendev.org/c/openstack/diskimage-builder/+/787827 | 22:05 |
| clarkb | zigo: ^ I think that is the fix for the bullseye fix | 22:05 |
| clarkb | I'm guessing that when pypi said only 1% of requests don't use SNI they didn't look at the nature of those requests. Likely all easy_install setup_requires requests | 22:06 |
| fungi | which is a vanishingly small number since newer setuptools calls out to pip | 22:07 |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder master: debian-minimal: Set bullseye version https://review.opendev.org/c/openstack/diskimage-builder/+/787665 | 22:07 |
| clarkb | and now rebased to be double sure | 22:07 |
| zigo | :) | 22:07 |
| * zigo crosses fingers | 22:07 | |
| *** slaweq has joined #opendev | 22:09 | |
| *** slaweq has quit IRC | 22:20 | |
| zigo | Looking at the debian-release list, it looks like 22 of may could happen *IF* kibi can fix the debian-installer in time. If not, then June ... | 22:51 |
| zigo | The major blocker being black screen on AMD GPUs ... | 22:51 |
| openstackgerrit | Merged opendev/bindep master: Correctly skip sdist building in tox.ini https://review.opendev.org/c/opendev/bindep/+/787781 | 23:03 |
| clarkb | the unexpected nova API error hit the centos8 job on https://review.opendev.org/c/openstack/diskimage-builder/+/787827 but the xenial build there did pass | 23:04 |
| clarkb | I'm beginning to think we may need to grab nova logs from those jobs at least temporarily to debug furhter | 23:04 |
| clarkb | btu I'm running out of day | 23:04 |
| fungi | git-review 2.1.0 release announcement is sent | 23:05 |
| clarkb | in any case looks like my fix did address the pbr thing properly since that job now passes (and it does a full boot test which will exercise the glean install) | 23:10 |
| clarkb | fungi: thanks | 23:10 |
| ianw | clarkb: thanks for looking at the xenial issue for dib | 23:58 |
| *** tosky has quit IRC | 23:58 | |
| ianw | Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. | 23:59 |
| ianw | <class 'neutronclient.common.exceptions.Unauthorized'> (HTTP 500) (Request-ID: req-2a6715af-ced6-4884-bd7c-18554d8b7947) | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!