| *** sgw has quit IRC | 00:31 | |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Add official-openstack-repo-jobs for openstack/osops https://review.opendev.org/751124 | 00:45 |
|---|---|---|
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Publish structured data listing our ML archives https://review.opendev.org/751125 | 00:47 |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Add official-openstack-repo-jobs for openstack/osops https://review.opendev.org/751124 | 00:48 |
| kevinz | ianw: clarkb: Hey guys, anything I can help? | 00:48 |
| ianw | kevinz: we were just trying to remember the extact differences between the networks availble to the ci control plane tenant | 00:49 |
| ianw | now i think about it ... i think maybe we put nb03 on a ipv4 only tenant because it was having a lot of trouble uploading images to linaro-london, maybe? | 00:49 |
| ianw | (nb03.openstack.org) | 00:50 |
| ianw | kevinz: in case you haven't been following, nb03.openstack.org is replaced with nb03.opendev.org, which uses nodepool-builder from a container and is more 2020 ready :) | 00:50 |
| ianw | the biggest thing is that it allows us to enable tls on zookeeper connections | 00:51 |
| ianw | ... well, disable non-tls access is maybe more accurate | 00:51 |
| fungi | a container whose arm64 images were built in that cloud, too! | 00:51 |
| kevinz | ianw: I remember before, we have trouble uploading the image to linaro London | 00:51 |
| kevinz | so now all the context is in Linaro London side? | 00:52 |
| fungi | builder deployed with container images built on virtual machines booted from vm images created and uploaded by that same builder. it basically builds itself | 00:52 |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Move x/osops-* to openstack/osops https://review.opendev.org/751130 | 00:52 |
| ianw | kevinz: we've completely dropped linaro-london | 00:54 |
| kevinz | all right | 00:55 |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Complete retirement of x/osops-* repos https://review.opendev.org/751141 | 01:15 |
| *** rosmaita has left #opendev | 01:57 | |
| openstackgerrit | likui proposed openstack/diskimage-builder master: Stop install unnecessary packages when run tox -edocs https://review.opendev.org/751150 | 02:15 |
| *** AnonKapes has joined #opendev | 03:04 | |
| *** AnonKapes has quit IRC | 03:08 | |
| *** ykarel|away has joined #opendev | 03:54 | |
| *** ysandeep|away is now known as ysandeep | 04:24 | |
| *** ykarel|away is now known as ykarel | 04:38 | |
| *** cloudnull has quit IRC | 04:51 | |
| *** cloudnull has joined #opendev | 04:52 | |
| chandankumar | Hello #infra | 05:54 |
| chandankumar | we are seeing https://review.opendev.org/#/c/750119/ patch in merge conflict multiple times | 05:55 |
| chandankumar | we rebased the patch | 05:55 |
| chandankumar | but still seeing | 05:55 |
| chandankumar | is there something wrong with gerrit? | 05:55 |
| chandankumar | other people are also seeing, please have a look | 05:55 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/751163 | 06:09 |
| AJaeger | config-core, please review https://review.opendev.org/#/c/751124 | 06:17 |
| *** qchris has quit IRC | 06:20 | |
| AJaeger | chandankumar: indeed strange, hope an infra-root can help | 06:26 |
| AJaeger | chandankumar: I see nothing obvious | 06:26 |
| *** qchris has joined #opendev | 06:34 | |
| *** fressi has joined #opendev | 06:37 | |
| chandankumar | AJaeger: thanks ! | 06:41 |
| *** fressi has quit IRC | 06:44 | |
| *** fressi has joined #opendev | 06:55 | |
| *** fressi has quit IRC | 07:08 | |
| *** DSpider has joined #opendev | 07:22 | |
| *** ysandeep is now known as ysandeep|lunch | 07:34 | |
| *** hashar has joined #opendev | 07:39 | |
| *** fressi has joined #opendev | 07:45 | |
| *** fressi has quit IRC | 07:49 | |
| *** fressi has joined #opendev | 07:55 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:02 | |
| *** fressi has quit IRC | 08:24 | |
| *** fressi has joined #opendev | 08:29 | |
| *** fressi has quit IRC | 08:29 | |
| *** fressi has joined #opendev | 08:29 | |
| *** fressi has quit IRC | 08:34 | |
| *** ysandeep|lunch is now known as ysandeep | 08:36 | |
| *** ssbarnea has joined #opendev | 08:38 | |
| *** andrewbonney has joined #opendev | 08:44 | |
| *** tosky has joined #opendev | 08:47 | |
| *** fressi has joined #opendev | 09:01 | |
| *** ssbarnea has quit IRC | 09:06 | |
| *** ykarel is now known as ykarel|lunch | 09:36 | |
| *** hillpd has quit IRC | 10:11 | |
| *** mnaser has quit IRC | 10:11 | |
| *** aannuusshhkkaa has quit IRC | 10:11 | |
| *** dviroel has quit IRC | 10:11 | |
| *** donnyd has quit IRC | 10:11 | |
| *** ttx has quit IRC | 10:11 | |
| *** jrosser has quit IRC | 10:11 | |
| *** andreykurilin has quit IRC | 10:11 | |
| *** corvus has quit IRC | 10:11 | |
| *** andreykurilin has joined #opendev | 10:11 | |
| *** corvus has joined #opendev | 10:11 | |
| *** dviroel has joined #opendev | 10:11 | |
| *** hillpd has joined #opendev | 10:11 | |
| *** donnyd has joined #opendev | 10:11 | |
| *** jrosser has joined #opendev | 10:11 | |
| *** fressi has quit IRC | 10:14 | |
| *** ykarel|lunch is now known as ykarel | 10:16 | |
| *** mnaser has joined #opendev | 10:16 | |
| *** aannuusshhkkaa has joined #opendev | 10:16 | |
| *** ttx has joined #opendev | 10:16 | |
| *** aannuusshhkkaa has quit IRC | 10:17 | |
| *** aannuusshhkkaa has joined #opendev | 10:17 | |
| *** mnaser has quit IRC | 10:17 | |
| *** mnaser has joined #opendev | 10:17 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: pylint: 4 more https://review.opendev.org/751021 | 10:20 |
| *** tkajinam has quit IRC | 10:22 | |
| openstackgerrit | likui proposed openstack/diskimage-builder master: replace imp module https://review.opendev.org/751236 | 10:31 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Create elastic-recheck container image https://review.opendev.org/750958 | 11:18 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Unpin webob and hacking to fix compatibility issues https://review.opendev.org/751117 | 11:28 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Optimise the Story browsing query https://review.opendev.org/742046 | 11:28 |
| *** hashar has quit IRC | 11:34 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/bindep master: Document combine tag https://review.opendev.org/751253 | 11:38 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/bindep master: Document compile tag https://review.opendev.org/751253 | 11:38 |
| *** ttx has quit IRC | 11:39 | |
| *** ttx has joined #opendev | 11:39 | |
| *** ttx has quit IRC | 11:40 | |
| *** ttx has joined #opendev | 11:40 | |
| *** Goneri has joined #opendev | 12:16 | |
| *** hashar has joined #opendev | 12:37 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/751163 | 12:58 |
| fungi | chandankumar: did you figure it out already? zuul seems to have been able to test it | 13:02 |
| fungi | and gerrit isn't reporting any merge conflict on that change as far as i can see | 13:02 |
| AJaeger | fungi: looking at history it got rebased for sure | 13:06 |
| openstackgerrit | Merged openstack/project-config master: Add official-openstack-repo-jobs for openstack/osops https://review.opendev.org/751124 | 13:16 |
| openstackgerrit | Merged opendev/storyboard master: Unpin webob and hacking to fix compatibility issues https://review.opendev.org/751117 | 13:36 |
| *** dmsimard has quit IRC | 13:44 | |
| *** dmsimard has joined #opendev | 13:45 | |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Retired x/osops-* repos https://review.opendev.org/751130 | 13:55 |
| *** gmann is now known as gmann_afk | 14:05 | |
| *** hashar has quit IRC | 14:18 | |
| *** sgw has joined #opendev | 14:44 | |
| openstackgerrit | Merged opendev/elastic-recheck master: pylint: 4 more https://review.opendev.org/751021 | 14:49 |
| zbr | clarkb: what can we do to improve maintenance of tools like bindep or pbr? | 14:53 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/bindep master: Improve tox.ini setup https://review.opendev.org/605613 | 14:56 |
| *** mlavalle has joined #opendev | 15:01 | |
| *** Topner has joined #opendev | 15:01 | |
| fungi | zbr: did they break? | 15:02 |
| zbr | obviously, everything is affected by bitrot | 15:03 |
| fungi | do you actually mean maintenance, or do you mean adding features/improvements? | 15:03 |
| fungi | yep, i'm asking if we've got some bitrot we need to address in those | 15:03 |
| zbr | fungi: i care a lot about the both, i am not interested in new features, only the boring maintenance bit | 15:03 |
| fungi | happy to put down what i'm doing now and take a look | 15:04 |
| fungi | we just recently wrangled pbr back into shape, and bindep is important too so if either of those isn't testable at the moment that's a priority for me | 15:04 |
| fungi | pbr is *technically* part of openstack oslo, but we've helped substantially with it from time to time | 15:05 |
| fungi | though i do my best to have pbr discussions in #openstack-oslo as a result | 15:05 |
| zbr | regarding pbr i stopped using it about an year ago on my own projects, since setuptools-scm does work really nice. | 15:06 |
| zbr | but i can see that it would take a very long time for others to do the same switch | 15:07 |
| fungi | setuptools-scm will generate authors and changelog files? | 15:07 |
| fungi | i'll have to take a look at it again | 15:07 |
| zbr | i did not had this issue, this being addressed differently. | 15:07 |
| zbr | one reason to keep it, or maybe to transform it to do less. | 15:08 |
| fungi | times were that setuptools-scm didn't have many features, and we wanted something integrated with distutils2 which was getting abandoned. that's basically where pbr came from | 15:08 |
| zbr | fungi: don't get me wrong, pbr proved to be one of the most useful libraries so far. i still have nigtmares about code in setup.py | 15:09 |
| fungi | but yeah, in my personal projects i use pbr only at build time to create package metadata (including versions from tags) which i know setuptools-scm can do, but also to handle changelog and authors file generation from git history so i can incorporate them into my documentation and license info | 15:09 |
| fungi | for the record, this is the entirety of the setup.py in my personal projects: | 15:10 |
| clarkb | are we trying to maintain pbr or delete it? | 15:10 |
| fungi | import setuptools | 15:10 |
| fungi | setuptools.setup(setup_requires=['pbr>=5'], pbr=True) | 15:10 |
| zbr | python packaging is still not a happy dream (watching developments closely on discuss) | 15:11 |
| clarkb | because we've spent a fair bit of time getting pbr testing up to speed and fungi made some fixes | 15:11 |
| fungi | so really the *only* think i have a setup.py for is adding pbr>=5 to the dist's setup_requires list and telling pbr to enable itself | 15:12 |
| * zbr wondering if mordred will look now at https://review.opendev.org/#/c/536355/ with different eyes | 15:12 | |
| fungi | zbr: likely not, he's not working on oracle linux, he's working on oracle cloud ;) | 15:13 |
| *** ysandeep is now known as ysandeep|away | 15:15 | |
| clarkb | as a side note before pbr existed we used setuptools-scm or maybe it was a git specific variant | 15:18 |
| fungi | i think it was a predecessor | 15:19 |
| * zbr working on a fix for bindep.... | 15:19 | |
| fungi | i think it may have been https://pypi.org/project/setuptools-git we used pre-pbr | 15:20 |
| fungi | setuptools-scm didn't exist until 2015 | 15:20 |
| fungi | (at least not on pypi anyway) | 15:20 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/bindep master: Fixed pep8 job https://review.opendev.org/751322 | 15:22 |
| fungi | oh, hah, setuptools-scm started out as hgdistver (so was mercurial specific) | 15:22 |
| zbr | at least they picked a less obscure name :D | 15:23 |
| mnaser | infra-root: i must have done something wrong with my addition of octopus ceph .. http://mirror.ca-ymq-1.vexxhost.opendev.org/ceph-deb-octopus/ | 15:23 |
| mnaser | its been 16 hours since that landed | 15:23 |
| clarkb | chandankumar: fungi: it could be an issue ina merger specific repo? | 15:23 |
| fungi | mnaser: i can check logs | 15:24 |
| clarkb | we should probably dig through merger logs and see if we can find the traceback to double check that | 15:24 |
| fungi | clarkb: yeah, possible if we can spot which merger that got checked on we could see if there's a corrupt clone | 15:24 |
| fungi | mnaser: "Disk quota exceeded!" | 15:25 |
| fungi | i'll see where we need to increase quota | 15:26 |
| mnaser | oopsies | 15:26 |
| fungi | looks like it topped out at 5mb so i probably misread the initial quota on that volume and thought it was enough | 15:27 |
| fungi | hah, yep. the nautilus volume has a quota of 50000000, octopus has a quota of 5000 | 15:31 |
| fungi | okay, now it's consistent | 15:32 |
| fungi | on the next mirror pulse (in roughly 25 minutes) we should hopefully see it work as intended | 15:33 |
| fungi | #status log corrected quota on mirror.deb-octopus afs volume from 5mb to 50gb | 15:34 |
| openstackstatus | fungi: finished logging | 15:34 |
| zbr | open zuul build pages seems quite slow today, considerably slower than on average day | 15:35 |
| fungi | zbr: yes, this came up yesterday, we think the recent zuul-web restart may have either made it less efficient or broken our apache caching somehow | 15:37 |
| fungi | we're still trying to work out how to get it caching correctly | 15:37 |
| fungi | at this point i'm wondering if the api responses are including some sort of header which apache takes as a sign not to cache them | 15:38 |
| fungi | though it doesn't even log them as a cache miss, so that would also be weird | 15:38 |
| zbr | that is the usual thing happening | 15:39 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/bindep master: Fixed pep8 job https://review.opendev.org/751322 | 15:40 |
| chandankumar | clarkb: fungi thanks for looking into that, after multiple iteration, it somehow vanished | 15:41 |
| chandankumar | https://review.opendev.org/750119 | 15:41 |
| *** ykarel is now known as ykarel|away | 15:42 | |
| fungi | clarkb: zbr: corvus: maybe the webclient changed how the status api is being requested, and is passing something like an auth header which causes apache to not even try the cache? that might explain what we see | 15:46 |
| fungi | in a bit i can try the browser devtools and see if i can spot something, though i'm not overly familiar with debugging client-side javascript initiated requests | 15:47 |
| *** elod has quit IRC | 15:51 | |
| *** elod has joined #opendev | 15:53 | |
| zbr | fungi: clarkb https://review.opendev.org/#/c/751322/ - fix for bindep | 15:59 |
| fungi | zbr: thanks! i went ahead and approved it. looks good | 16:00 |
| clarkb | fungi: oh ya could be like the docker image api where we're always shipping an authorization header even if anonymous | 16:02 |
| zbr | single page load, ~17 requests, all >5.0-7.5s, looks very weird, reminds me of broken dns, and fallback. | 16:03 |
| zbr | clearly has nothing to do with the size of the files served. | 16:04 |
| clarkb | I think it may be apache limiting the number of requests to zuul-web | 16:04 |
| clarkb | and those back up because zuul-web is very busy | 16:04 |
| clarkb | (dns was suggested yesterday as well but looking at dns it seems fine) | 16:04 |
| fungi | zbr: yeah, but we've noticed the zuul-web process is pretty busy (consuming an entire core's worth of cycles constantly), and even local api requests over loopback from a shell on the server take seconds to complete | 16:04 |
| *** ykarel|away has quit IRC | 16:06 | |
| zbr | Cache-Control: max-age=0 does not look good to me | 16:06 |
| openstackgerrit | Merged opendev/bindep master: Fixed pep8 job https://review.opendev.org/751322 | 16:11 |
| zbr | it has nothing to do with caching because it takes as much time to return 304 too. | 16:13 |
| *** ajya is now known as ajya|afk | 16:13 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/bindep master: Improve tox.ini setup https://review.opendev.org/605613 | 16:13 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/bindep master: Document compile tag https://review.opendev.org/751253 | 16:13 |
| fungi | zbr: our theory is that the lack of caching is putting the zuul-web process under significant additional load causing it to take a long time to respond to any request, whether or not that request would have been cached | 16:14 |
| zbr | i need some comments on ^ -- which documents the fact that we need "compile" tag on bindep instead of test to make it work with python-builder. | 16:14 |
| zbr | as that is cherrypy, maybe webknjaz could help but | 16:16 |
| zbr | not sure who has permissions on docker hub, but https://registry.hub.docker.com/r/opendevorg/python-builder could really need some love, at least a link that sends user back the the code that builds that image. | 16:19 |
| clarkb | fungi: chandankumar: I've found what I think is the failing merge requset but if I grep the gear job id or the merge unique id on all the executors and mergers I don't find it | 16:30 |
| clarkb | wondering if I'm just searching wrong | 16:36 |
| clarkb | maybe logs rotated /me double checks that | 16:36 |
| clarkb | ok I was grepping the wrong thing. The mergers log the buildset_uuid not the job unique or merger unique values | 16:41 |
| clarkb | ze04 processed it | 16:41 |
| clarkb | http://paste.openstack.org/show/797786/ | 16:43 |
| clarkb | looking at file timestamps and uptime I think what happened is the server crashed and leaked that file | 16:48 |
| clarkb | I'm running a find now to see if there are others | 16:48 |
| mnaser | fungi: http://mirror.ca-ymq-1.vexxhost.opendev.org/ceph-deb-octopus/ looks good, maybe worth rm foo might be worth (i think you used it to test) | 16:49 |
| chandankumar | clarkb: another instance was this one https://review.opendev.org/#/c/737847/ | 16:51 |
| fungi | mnaser: oops, i meant to delete that when i was in there earlier fixing the quota too | 16:51 |
| *** fdegir has quit IRC | 16:53 | |
| clarkb | chandankumar: its the asme repo so likely the same executor in this case. I'm sorting out a find query to check then we can run this on all mergers | 16:53 |
| clarkb | `find /var/lib/zuul/executor-git/ -maxdepth 5 -mindepth 5 -type f -name index.lock -mtime +10` <- thats the sort of find that seems to work best. Uptime is 17 days on ze04 so mtime +10 should be plenty safe. But now that I think of it +1 even should be fine as those locks should be short lived | 16:59 |
| clarkb | the path will have to be adjusted for mergers too | 17:00 |
| *** Topner has quit IRC | 17:02 | |
| clarkb | of the executors only 04 has this problem and only with that single lock file. Checking mergers next then will do cleanups | 17:03 |
| clarkb | chandankumar: ok the index.lock is removed and that was the only place I found it. That repo should be much happier now. Thank you for reporting | 17:05 |
| clarkb | now to go write a change to zuul to automate that cleanup | 17:05 |
| chandankumar | ++ | 17:05 |
| chandankumar | clarkb: thank you :-) | 17:05 |
| *** andrewbonney has quit IRC | 17:13 | |
| clarkb | fungi: zbr: I see max-age=1 and cache-control= no-cache | 17:42 |
| clarkb | I expect it is the cache-control that is breaking us | 17:42 |
| clarkb | and maybe that is new | 17:42 |
| *** gmann_afk is now known as gmann | 17:42 | |
| fungi | ahh, yep, that could i guess have come in a new cherrypy version or something | 17:42 |
| clarkb | no-cache is from the js I think so from pf? | 17:42 |
| clarkb | there is also a pragma: no-cache | 17:43 |
| fungi | er, yep right if it's in the request then i guess it's from js | 17:43 |
| fungi | i guess this is something to bring up in #zuul in that case | 17:43 |
| *** Topner has joined #opendev | 17:54 | |
| *** stephenfin has quit IRC | 17:56 | |
| *** fdegir has joined #opendev | 18:02 | |
| clarkb | fungi: https://github.com/pypa/pip/issues/8368 re bdist_wheel not working and the current fallback behavior | 18:10 |
| clarkb | I wonder if we shouldn't suggest that venv needs updating (and since its stdlib glhf) | 18:11 |
| fungi | yeah, venv has an option to not install pip, but doesn't seem to have any options to tell it to include a package by default | 18:13 |
| fungi | if you're calling venv in a python script you can at least preinstall packages with its post_setup() method | 18:14 |
| clarkb | I guess you can always do python3 -m venv fooenv && fooenv/bin/pip install wheel | 18:15 |
| clarkb | its just weird to require wheel then not include it | 18:15 |
| clarkb | (maybe pip should vendor it) | 18:15 |
| clarkb | oh wait its specifically when you don'y have a pyproject.toml? | 18:31 |
| clarkb | amazing | 18:31 |
| clarkb | the description and title don't seem to agree on that | 18:32 |
| clarkb | maybe there are saying pyproject.toml needs to install wheel? | 18:32 |
| clarkb | of course I can't even comment on the issue because it is locked | 18:33 |
| fungi | i guess pbr could vendor wheel ;) | 18:52 |
| fungi | and yeah, wheel needs to be there before other packages are installed, so the way they want you to declare that is with pyproject.toml or in your build backend | 18:53 |
| clarkb | seems weird if its a strict dependency of pip to not just include it with pip | 18:53 |
| clarkb | but again I can't say anythin because they've decided they don't want communioty feedback after all | 18:54 |
| fungi | don't worry, there are plenty of python packages much cruftier than ours whose maintainers will bring out the torches and pitchforks as soon as pip stops working for their packages | 18:55 |
| fungi | i mean, sucks for the pip maintainers, but i suspect they don't really want to hear that they're looking at mutiny over changes like that | 18:56 |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add openstack-helm-images in vexxhost https://review.opendev.org/751387 | 18:56 |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add openstack-helm-images and loci in vexxhost https://review.opendev.org/751387 | 18:59 |
| * clarkb finds lunch | 18:59 | |
| Open10K8S | Hi team. Please check thi PS. https://review.opendev.org/751387 Add openstack-helm-images and loci in vexxhost | 19:00 |
| Open10K8S | thank you | 19:00 |
| openstackgerrit | Merged openstack/project-config master: Add openstack-helm-images and loci in vexxhost https://review.opendev.org/751387 | 19:19 |
| *** hashar has joined #opendev | 19:42 | |
| *** Topner has quit IRC | 20:15 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Run multiple zuul-web processes https://review.opendev.org/751426 | 20:46 |
| clarkb | corvus: fungi ^ fyi using testinfra to see if that is doable | 20:46 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Run multiple zuul-web processes https://review.opendev.org/751426 | 21:30 |
| clarkb | it didn't work, this patchset tries to get logs | 21:31 |
| *** hashar has quit IRC | 21:59 | |
| *** mlavalle has quit IRC | 22:36 | |
| *** tosky has quit IRC | 23:20 | |
| *** DSpider has quit IRC | 23:40 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!