Wednesday, 2020-09-16

openstackgerritIan Wienand proposed zuul/zuul-jobs master: Retire Fedora 31 for 32  https://review.opendev.org/75197501:31
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-docker: only run docker-setup.yaml when installed  https://review.opendev.org/74706201:31
openstackgerritIan Wienand proposed zuul/zuul-jobs master: update-json-file: add role to combine values into a .json  https://review.opendev.org/74683401:31
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-docker: Linaro MTU workaround  https://review.opendev.org/74706301:31
*** Topner has joined #opendev02:33
*** Topner has quit IRC02:37
openstackgerritwu.shiming proposed openstack/diskimage-builder master: Update imp module to importlib  https://review.opendev.org/75201102:46
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: workaround slow account creation  https://review.opendev.org/75218403:52
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: workaround slow account creation  https://review.opendev.org/75218404:11
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Retire Fedora 31 for 32  https://review.opendev.org/75197504:11
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-docker: only run docker-setup.yaml when installed  https://review.opendev.org/74706204:11
openstackgerritIan Wienand proposed zuul/zuul-jobs master: update-json-file: add role to combine values into a .json  https://review.opendev.org/74683404:11
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-docker: Linaro MTU workaround  https://review.opendev.org/74706304:11
*** Topner has joined #opendev04:34
*** Topner has quit IRC04:38
*** ykarel|away has joined #opendev04:43
*** diablo_rojo has quit IRC04:44
*** ykarel|away is now known as ykarel04:46
*** fressi has joined #opendev04:50
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-docker: Linaro MTU workaround  https://review.opendev.org/74706304:59
*** fressi has quit IRC05:17
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-docker: Linaro MTU workaround  https://review.opendev.org/74706305:42
openstackgerritwu.shiming proposed openstack/diskimage-builder master: Update imp module to importlib  https://review.opendev.org/75201105:48
*** mrunge_ is now known as mrunge06:00
*** lpetrut has joined #opendev06:15
*** qchris has quit IRC06:21
openstackgerritwu.shiming proposed openstack/diskimage-builder master: Update imp module to importlib  https://review.opendev.org/75201106:27
*** qchris has joined #opendev06:33
openstackgerritwu.shiming proposed openstack/diskimage-builder master: Update imp module to importlib  https://review.opendev.org/75201106:42
*** slaweq has joined #opendev06:48
*** fressi has joined #opendev06:58
*** ysandeep|away is now known as ysandeep07:05
*** hashar has joined #opendev07:07
*** andrewbonney has joined #opendev07:35
*** tosky has joined #opendev07:43
*** DSpider has joined #opendev07:47
*** guillaumec has quit IRC07:51
*** guillaumec has joined #opendev07:52
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:01
openstackgerritMerged openstack/diskimage-builder master: Make iscsi-boot element support centos 8  https://review.opendev.org/74787808:11
*** dtantsur|afk is now known as dtantsur08:33
openstackgerritMerged openstack/diskimage-builder master: update various gentoo bits  https://review.opendev.org/74750208:34
openstackgerritMerged openstack/diskimage-builder master: Copy apt gpg keys directly into trusted.gpg.d  https://review.opendev.org/74781008:34
*** ykarel_ has joined #opendev08:43
*** ykarel has quit IRC08:46
*** ykarel_ is now known as ykarel08:47
openstackgerritMerged openstack/diskimage-builder master: Move centos python3 installation after RHEL subscription  https://review.opendev.org/74977308:50
openstackgerritIan Wienand proposed zuul/zuul-jobs master: update-json-file: add role to combine values into a .json  https://review.opendev.org/74683408:59
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-docker: Linaro MTU workaround  https://review.opendev.org/74706308:59
*** hashar has quit IRC09:51
openstackgerritWitold Bedyk proposed openstack/project-config master: Revert "Use only noop jobs for openstack/monasca-transform"  https://review.opendev.org/75220709:52
*** fressi has quit IRC10:25
*** Topner has joined #opendev10:36
openstackgerritzbr proposed zuul/zuul-jobs master: Partial address ansible-lint E208  https://review.opendev.org/74848010:36
*** Topner has quit IRC10:41
*** slaweq has quit IRC10:41
openstackgerritzbr proposed zuul/zuul-jobs master: Add mode to packer  https://review.opendev.org/75221310:50
*** slaweq has joined #opendev10:54
*** elod has quit IRC11:18
*** elod has joined #opendev11:18
*** fressi has joined #opendev11:25
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: ensure-haskell-stack: ensure all requirements are installed  https://review.opendev.org/75222512:00
*** slaweq has quit IRC12:01
*** slaweq has joined #opendev12:30
openstackgerritLon Hohberger proposed openstack/diskimage-builder master: rhel-common: Provide method to select module streams  https://review.opendev.org/74773212:31
openstackgerritLon Hohberger proposed openstack/diskimage-builder master: rhel-common: Provide method to select module streams  https://review.opendev.org/74773212:40
openstackgerritLon Hohberger proposed openstack/diskimage-builder master: rhel-common: Provide method to select module streams  https://review.opendev.org/74773212:43
*** ricolin has quit IRC13:20
*** sshnaidm is now known as sshnaidm|afk13:32
*** ykarel_ has joined #opendev13:35
*** ykarel has quit IRC13:37
smcginnisclarkb, ianw, fungi: Any update on the pypi index issues?13:59
fricklersmcginnis: I assumed pypi fixed things earlier tonight, do you still see issues after that?14:00
smcginnisfrickler: Thanks. Just coming online now, so I haven't seen anything. Wanted to make sure before I started approving any patches.14:02
*** ykarel_ is now known as ykarel|14:04
*** ykarel| is now known as ykarel14:04
*** owalsh has quit IRC14:12
fungismcginnis: last word was that they took the problem bandersnatch mirror offline by 22:3014:15
ildikovhi all14:16
fungibut that fastly can continue to serve "stale" content for up to 24 hours14:16
*** ricolin has joined #opendev14:16
ildikovI saw this error on a StarlingX governance repo patch: "AttributeError: 'FlakesChecker' object has no attribute 'CONSTANT'"14:16
smcginnisGreat. Nice work all tracking that down. That was a really odd one.14:16
ildikovand wanted to check if this is something that others have seen to?14:16
smcginnisildikov: I think that's a side effect of moving to running on focal nodes, and therefore running under py3.8.14:17
smcginnisildikov: Likely the hacking version needs to be raised.14:17
fungiyeah, i think the solution has usually been to switch to newer pyflakes (maybe via hacking upgrade if they're also using hacking)14:17
ildikovI only saw hacking in the test-requirements.txt: hacking!=0.13.0,<0.14,>=0.12.0 # Apache-2.014:18
ildikovso I figured I come here and ask for suggestions on what to bump :)14:18
fungiildikov: can you link the build result page? i'm happy to take a look and see what's dragging it in where14:19
ildikovfungi: https://zuul.opendev.org/t/openstack/build/69abb77fb34e4159893176bd48d3fad914:19
fricklerildikov: using hacking >= 3.2.0 is the easiest solution I think, did that for neutron-dynamic-routing https://review.opendev.org/75077914:22
AJaegerildikov: that hacking is ancient, so going with 3.2.0 as proposed by frickler is the right way forward14:24
smcginnisildikov: hacking locks down the version of the various other linters it pulls in, so hopefully you should just need to update that version and that will take care of it.14:24
ildikovcool, thanks!!14:25
ildikovI will put a patch and check14:25
smcginnisildikov: Just a warning, going from that very old version of hacking to something newer will likely pick up a lot of hacking issues.14:25
ildikovI will follow the approach in frickler's patch14:25
smcginnisSo there will probably be some cleanup work to do along with the requirements change.14:25
smcginnisAJaeger had a lot of patches taking care of issues like that, so you can probably find some good examples out there if you need it.14:26
fungiyeah, my preferred workflow, if it helps, is to raise to the latest version of your static analyzers, run them to see what they start complaining about, then put all those new rules into an ignore list. that'll get you onto the new versions with minimal pain. after that you can peel back any new rules you like one at a time fixing just the problems they expose14:28
*** priteau has joined #opendev14:30
ildikovthe repo is very plain simple so I would hope there won't be a massive amount of things to fix, while I'm sure there are skeletons in there14:35
*** fdegir has quit IRC14:36
*** lpetrut has quit IRC14:38
fungiildikov: yeah, i mean we just bumped hacking in openstack/election yesterday and all we needed to change to make it happy was this: https://review.opendev.org/#/c/734854/2/openstack_election/cmds/ci_check_all_candidate_files.py14:57
fungigranted we were only updating from 2.0.x to 3.1.x14:57
ildikovnice!14:59
*** ykarel is now known as ykarel|away15:12
*** fressi has quit IRC15:12
*** mlavalle has joined #opendev15:16
openstackgerritzbr proposed zuul/zuul-jobs master: Add mode to packer  https://review.opendev.org/75221315:18
*** ysandeep is now known as ysandeep|away15:22
*** owalsh has joined #opendev15:24
openstackgerritzbr proposed zuul/zuul-jobs master: Add managed jobs to periodic  https://review.opendev.org/74868215:24
clarkbsmcginnis: frickler fungi If we see more issues after the 24 hour mark we will probably want to track those down but until then I think we did what we could15:26
openstackgerritzbr proposed zuul/zuul-jobs master: Add managed jobs to periodic-weekly  https://review.opendev.org/74868215:31
*** ykarel|away has quit IRC15:37
zbrclarkb:  can we create the weekly pipeline now? https://review.opendev.org/#/c/748706/15:39
*** slaweq has quit IRC15:48
*** slaweq has joined #opendev15:50
clarkbzbr: I think that is a better question in #zuul15:58
*** Tengu has quit IRC16:05
*** Tengu has joined #opendev16:06
*** tosky has quit IRC16:11
clarkbdoes anyone know if pip has a force rebuild wheel flag on install? there is --no-binary and --force-reinstall but what about when you have a wheel linked to an old so and want to rebuild the wheel?16:16
clarkbI've addressed this so far by rm'ing the wheels from my wheel cache16:17
clarkbI double checked the post_failure in the gate and it appears the job may be breaking host networking (its a job that includes dynamic-routing in the name). pip install fails due to no route to host to our mirror when installing stackviz16:21
clarkbI don't think that is related to our post failure problems yesterday. I notified neutrons irc channel too16:21
clarkbinfra-root now that nb03.opendev.org is doing zk tls I think we can alnd https://review.opendev.org/#/c/750833/ to block port 2181 on the zk servers. We don't disable 2181 entirely as it is useful for local debugging16:25
openstackgerritMerged openstack/project-config master: Use only noop jobs for openstack/monasca-transform  https://review.opendev.org/75198316:40
openstackgerritMerged openstack/project-config master: End project gating for openstack/monasca-analytics  https://review.opendev.org/75198716:40
*** ykarel|away has joined #opendev16:43
openstackgerritMerged openstack/project-config master: Retired x/osops-* repos  https://review.opendev.org/75113016:49
*** ykarel|away has quit IRC16:50
johnsomHappy dance, the pypa issue seems to be resolved. Thanks for all of the effort that went into tracking down that bad pypa mirror.16:53
fungiclarkb: not sure about forcing wheel rebuilds. i too just blow away my wheel cache and my venvs regularly and then recreate them all (basically any time i build a new default python3 point release)16:53
fungiwhich reminds me, looks like v3.5.10 and v3.8.6rc1 were tagged since i last refreshed16:54
fungiand i guess 3.5 is technically eol now too16:56
smcginnisOh wow, as of this week.16:57
clarkbjohnsom: I think my favorite part of the whole thing is we stopped running bandersnatch when mirroring pypi hit 2TB of disk use. They just hit 12TB :)16:57
fungihttps://www.python.org/dev/peps/pep-0478/ "There are no specific plans for the next release of Python 3.5. Note that Python 3.5 will no longer be supported after September 2020. Therefore, 3.5.10 may be the final release of Python 3.5."16:58
smcginnis2020-09-13 according to https://devguide.python.org/#branchstatus16:58
fungicool. i can start planning to drop 3.5 testing for one of my personal projects now16:59
fungiafter i make sure it's still working with 3.5.10 that is16:59
*** fdegir has joined #opendev17:01
openstackgerritMerged openstack/project-config master: Update branch checkout for octavia-lib DIB element  https://review.opendev.org/74587717:03
*** dtantsur is now known as dtantsur|afk17:13
*** ysandeep|away is now known as ysandeep17:27
*** ysandeep is now known as ysandeep|away17:32
*** tosky has joined #opendev18:51
*** priteau has quit IRC19:12
*** Tengu has quit IRC19:15
*** Tengu has joined #opendev19:17
clarkbfungi: https://review.opendev.org/#/c/751919/ I think that is the last piece needed in dib ebfore we can do a release which will enable us to delete nb0419:18
clarkbnote that the tumbleweed job passes there but not in the parent19:18
fungii should probably subscribe my gertty to openstack/diskimage-builder changes19:20
fungisidetracked by a kernel upgrade, the openafs dkms build is currently failing with linux 5.819:22
openstackgerritMerged openstack/diskimage-builder master: Install grep before busybox on suse distros  https://review.opendev.org/75188019:22
clarkbI'm not sure its a rush but seeing ^ was about to merge reminded me :)19:23
clarkband I'm trying to clear my backlog today so that I can get back to new things :)19:23
*** andrewbonney has quit IRC19:28
*** owalsh has quit IRC20:11
*** owalsh has joined #opendev20:43
*** owalsh has quit IRC20:56
clarkbianw: I'm surprised that rhel is shipping VM images that are EFI based since I think a lot of clouds don't support that yet?21:08
clarkbI guess that is where utilities like dib come in as they can "fix" the image for you if necessary21:08
*** slaweq has quit IRC21:14
*** slaweq has joined #opendev21:15
*** owalsh has joined #opendev21:18
*** slaweq has quit IRC21:27
*** slaweq has joined #opendev21:33
*** slaweq has quit IRC21:54
fungifor the big proprietary public clouds, they have their own tweaked images provided for you so you don't have to worry about it. for everything else, red hat sells cloud software21:56
*** tosky_ has joined #opendev21:56
fungino doubt whatever red hat is covering with support contracts supports efi boot21:57
*** tosky has quit IRC21:59
*** tosky_ is now known as tosky22:00
*** slaweq has joined #opendev22:01
openstackgerritGoutham Pacha Ravi proposed openstack/project-config master: Cleanup x/devstack-plugin-glusterfs jobs  https://review.opendev.org/75233522:06
*** slaweq_ has joined #opendev22:10
*** slaweq has quit IRC22:13
*** slaweq_ has quit IRC22:19
ianwclarkb: i think the images are "dual boot"22:46
ianwin theory, dib can create dual-boot efi/bios images too22:46
ianwactually s/can/does/ after a recent change22:46
ianw... but ... practice i'm not so sure22:47
clarkbah22:47
ianwagain in theory we should be able to try our nodepool tests with efi, and then, in theory, switch on the dual-boot efi and see if that boots both bios and efi22:50
clarkbianw: if we get https://review.opendev.org/#/c/751919/ in I think that is all we need for tumbleweed fixes22:50
clarkbnot sure if there are other changes (like the rhel one) htat you may also want to get into a dib release22:50
ianwno problem; i sent a bunch back through the gate after the recent breakages, havent' checked if it all committed yet :)22:51
ianwa lot of green ticks!22:51
clarkbthe parent to that change merged today22:51
clarkbpypi is much happier :)22:51
ianwi double check the queue ,but i think 751919 would be the last outstanding thing for a release now22:52
ianwhttps://p.datadoghq.com/sb/7dc8b3250-85dcf667bd?from_ts=1600273495941&to_ts=1600277095941&live=true is an interesting page22:53
ianwpypi stats22:53
ianw942,000 requests/min22:54
clarkbI can't seem to pull that up22:54
fungiahh, yeah, debian has multiarch iso images which can boot efi/mbr and multiple processor architectures for example, definitely doable22:55
clarkbnot getting a tcp connection22:55
clarkbbut I believe it :)22:55
fungiit's too busy to give you a tcp connection22:56
TheJuliaout of curiosity, are there more log storage issues?22:56
fungiTheJulia: i haven't seen any, but that means nothing. do you have a suspect failure you can link?22:57
TheJuliahttps://4317fd6ef51d22a4668e-8daa7927094e2ef08ab8f0f95a49fc40.ssl.cf5.rackcdn.com/748315/23/check/metalsmith-integration-glance-localboot-centos8-uefi/79e4b9b/22:57
clarkbTheJulia: not that I've seen. I tracked down one post failure earlier today to broken host netowrking in a job that tests dynamic routing22:57
TheJuliaclick controller22:57
TheJuliaempty folder :\22:57
TheJuliaor at least that is what I see in two browsers22:57
fungithat's unlikely to be quota related22:57
TheJuliathat is kind of what I figured, but I wanted to ask22:57
clarkbfungi: TheJulia thats not the oe cloud either22:58
fungithough our log uploads try to generate indices, right?22:58
clarkbso if it is a swift problem it is a different problem22:58
clarkbfungi: yes we generate our own indices22:58
TheJuliaI just don't want to blindly rehceck it22:59
fungimaybe we created an empty index for that (what does our tooling do if the dir is empty?)22:59
ianware there files there in the manifest?22:59
fungigreat point22:59
*** tosky has quit IRC22:59
* TheJulia opens the main job log file22:59
fungiyes, looks like it22:59
fungi"controller" has children listed at least23:00
fungifor example 1d22a4668e-8daa7927094e2ef08ab8f0f95a49fc40.ssl.cf5.rackcdn.com/748315/23/check/metalsmith-integration-glance-localboot-centos8-uefi/79e4b9b/controller/ironic-bm-logs/23:00
TheJuliahttps://4317fd6ef51d22a4668e-8daa7927094e2ef08ab8f0f95a49fc40.ssl.cf5.rackcdn.com/748315/23/check/metalsmith-integration-glance-localboot-centos8-uefi/79e4b9b/controller/ironic-bm-logs/ works23:00
TheJulialol23:00
TheJuliajinx23:00
fungiyour cut-n-paste outshined mine23:00
fungirecent firefox likes to highlight stuff in the url bar without copying it to the paste buffer, so i have to unhighlight and rehighlight it. very aggravating23:01
ianwTheJulia: you could grab everything with the download script if you like23:01
ianwwe should definitely look into why that index wasn't generated/uploaded23:01
fungiso for some reason we failed to populate the inde23:01
fungix23:01
TheJuliawell, I managed to grab what I need :\23:01
TheJuliaJust went "well this is weird"23:02
ianwso index.html must be there, but it's blank i guess23:02
clarkbya it doesn't 40423:03
clarkbyou just get back an empty file23:03
clarkbMakes me wonder if this could be related to how our swift backed docker registry would end up with empty blob layers23:03
clarkbmaybe there is some sublte behavior thing there?23:03
ianwthere was23:06
fungiyeah, given what i've heard about the "swiftness" of rackspace's swift (cloudfiles) i doubt it suffers an identical bug to real swift, which leads me to suspect the problem could be client side23:07
ianwthere was https://review.opendev.org/#/c/751973/ which is a bad file on rax23:07
ianwhttps://58aca06c0a1750aa55ba-25dc1a1096eaf82a51392114c0a4a973.ssl.cf2.rackcdn.com/732435/7/check/heat-functional/852a5bf/controller/logs/screen-barbican-keystone-listener.txt23:07
clarkbfwiw `grep 79e4b9bb79ab429e98b4511227cfb11d /var/log/zuul/executor-debug.log` on ze06 to get hte ansible log for this job23:09
clarkbdoesn't show anything abnormal for swift log upload23:09
clarkband ya if you wget the file its a completely empty empty file but not a 40423:12
clarkblooking at the zuul-jobs upload-logs-swift role we seem to write html even if there is no other content23:13
clarkbso an empty dir should still have something there23:13
clarkball that to say it is really weird to have an empty file23:15
clarkbI'd expect a 404 (because upload failed) or something with html in it if not23:15
ianwhttp://1d22a4668e-8daa7927094e2ef08ab8f0f95a49fc40.ssl.cf5.rackcdn.com/748315/23/check/metalsmith-integration-glance-localboot-centos8-uefi/79e4b9b/controller/ironic-bm-logs/23:15
ianwi just go access denied on that? ^23:16
ianwoh, maybe it's the http i've mis-pasted23:16
ianwyep23:17
clarkbits a different "hostname"23:17
clarkbits related to how cdn works with swift aiui23:17
clarkboh interest fungi and TheJulia have different hostnames in their links23:18
clarkbwhere did the 1d22a466... come from?23:18
TheJuliaRedirect maybe?23:18
TheJuliaI got to it in a web browser based on the initial link I copied from the job23:19
clarkbmaybe different cdn nodes and only the content after the - is used to hash and verify23:19
ianwhttps://7598b949b48f30e69b3e-9945c6e94a1b9feee2753a6244e7fd57.ssl.cf5.rackcdn.com/748315/19/check/metalsmith-integration-glance-localboot-centos8-uefi/7084429/controller/23:40
ianwis the same job, on rax, with a correct index23:40
ianwso i don't know, i guess we just monitor for more and see if we can find a pattern23:40
*** DSpider has quit IRC23:42
clarkbwe could have the upload role fetch back content and check the first few bytes23:42
clarkbprobably overkill23:42
fungialso lots of extra bandwidth consumption23:49

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!