| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add pip-and-virtualenv to arm64 images https://review.opendev.org/713152 | 00:10 |
|---|---|---|
| openstackgerrit | Merged openstack/project-config master: Add pip-and-virtualenv to arm64 images https://review.opendev.org/713152 | 00:31 |
| openstackgerrit | Merged opendev/system-config master: nodepool-builder: add /opt/dib_cache https://review.opendev.org/712824 | 00:36 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: [dnm] Use diskimages-gloabls section https://review.opendev.org/713158 | 02:55 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: nodepool config : add ZUUL_USER_SSH_KEY https://review.opendev.org/713160 | 03:08 |
| *** DSpider has joined #opendev | 05:41 | |
| *** DSpider has quit IRC | 05:42 | |
| *** DSpider has joined #opendev | 05:43 | |
| *** DSpider has quit IRC | 05:43 | |
| *** DSpider has joined #opendev | 06:15 | |
| openstackgerrit | Daniel Pawlik proposed opendev/system-config master: Added updates dir for Fedora 31 release https://review.opendev.org/713169 | 08:08 |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: Remove Fedora 29 mirroring https://review.opendev.org/713177 | 09:05 |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: Remove Fedora 29 mirroring https://review.opendev.org/713177 | 09:11 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Replace occurences of '/' in nodename to something more path friendly https://review.opendev.org/713182 | 10:08 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Replace occurences of '/' in nodename to something more path friendly https://review.opendev.org/713182 | 10:16 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Replace occurences of '/' in nodename to something more path friendly https://review.opendev.org/713182 | 10:18 |
| openstackgerrit | Daniel Pawlik proposed zuul/zuul-jobs master: DNM - checking new images https://review.opendev.org/713183 | 10:22 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Replace occurences of '/' in nodename to something more path friendly https://review.opendev.org/713182 | 10:24 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Replace occurences of '/' in nodename to something more path friendly https://review.opendev.org/713182 | 10:28 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Replace occurences of '/' in nodename to something more path friendly https://review.opendev.org/713182 | 10:30 |
| openstackgerrit | sebastian marcet proposed opendev/puppet-openstackid master: Fixed python packages installation https://review.opendev.org/713225 | 13:15 |
| openstackgerrit | Merged opendev/puppet-openstackid master: Fixed python packages installation https://review.opendev.org/713225 | 13:57 |
| mnaser | https://review.opendev.org/#/c/713123/ -- just wondering how everyone feels about this :) | 15:16 |
| AJaeger | mnaser, change LGTM but I like to have clarkb review a new tenant ^ | 15:25 |
| mordred | clarkb: how does apply a reverse of https://salsa.debian.org/debian/openssl/-/blob/debian/unstable/debian/patches/Set-systemwide-default-settings-for-libssl-users.patch sound? | 15:53 |
| clarkb | I'll be able to answer if the patch ever loads :) | 15:54 |
| clarkb | mordred: ya that looks clean and is pretty easy to understand. | 15:56 |
| openstackgerrit | Merged openstack/diskimage-builder master: bindep: remove lsb-release https://review.opendev.org/713150 | 15:59 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Undo debian changes to openssl.cnf for python-base https://review.opendev.org/713278 | 16:00 |
| mordred | clarkb: ^^ | 16:01 |
| clarkb | mordred: so I understand you applied the patch to the .cnf file then vendored the result right? rather than apply the patch each build? | 16:03 |
| mordred | yeah. because python-base itself doesn't have patch installed | 16:03 |
| clarkb | wow | 16:04 |
| mordred | it's a base image :) | 16:04 |
| Shrews | i suspect it might also be less error prone to do it ahead of time | 16:07 |
| clarkb | mnaser: AJaeger I don't have any objects, but am still trying to get on top of the fires since friday | 16:08 |
| clarkb | Shrews: thats a good point (if the underlying config moves it could break our patch) | 16:08 |
| AJaeger | clarkb: want me to review mnaser's and +2A - or wait until the fires are out? either works for me... | 16:08 |
| Shrews | mordred: i only +2'd in case someone else wants to review | 16:08 |
| clarkb | AJaeger: maybe just double check frickler's comments then approve? It should be safe to add a new tenant | 16:09 |
| AJaeger | clarkb: ok | 16:10 |
| mordred | corvus: ^^ https://review.opendev.org/713278 if you have a sec | 16:12 |
| clarkb | thinking about the zuul ssh key thing, maybe we should just copy pasta it for each image in the env section while we sort out the best way to express it in nodepool conf for the future? | 16:22 |
| clarkb | mordred: I'm reviewing https://review.opendev.org/#/c/713101/ but won't approve it since it needs some review-dev attentin | 16:24 |
| mordred | clarkb: cool. if we wanna do it - I can emergency review-dev and clean it up - then we can land this and unemergency and things should just come back up | 16:25 |
| clarkb | I think I've come around to the idea of that consistency. Maybe we want to see what frickler Shrews fungi and corvus think? ianw already +2'd it | 16:27 |
| mordred | clarkb: agree | 16:28 |
| AJaeger | frickler, mnaser, please have a look again at comments at https://review.opendev.org/#/c/713123 | 16:29 |
| corvus | docker +2 | 16:29 |
| fungi | clarkb: i still think putting the public keys we trust into our service configuration is preferable to relying on system configuration management, especially for the builder case where it really only uses the public key and only for baking into the node images | 16:30 |
| corvus | win 12 | 16:30 |
| mnaser | AJaeger, frickler: left a comment, dont think i would have been able to actually get a list of all the repos so my idea was to start with it and fixup afterwards | 16:31 |
| fungi | boxcars for a win! | 16:31 |
| AJaeger | mnaser: works for me, let's wait for frickler to respond. Or do you need it directly? | 16:31 |
| mnaser | AJaeger: it's blocking some work i'm doing, and it'd be at least nice to see this land so i can do the round 2 of adding other projects so i can merge that today too (hopefully) | 16:32 |
| AJaeger | mnaser: ok, let's give frickler an hour to review and then I'll +2A. | 16:33 |
| clarkb | mordred: you are probably the right person to doulble check my -1 on https://review.opendev.org/#/c/713148/2 (docker mounts thing) | 16:41 |
| clarkb | and with that I think I've caught up on the changes that have been written so far | 16:41 |
| clarkb | fungi: re ssh keys, ya thats where we are heading based on ianw's change stack but they currently depend on new features in nodepool that are being discussed. I think we can do the early step of being extra verbose in our service configs then convert to the new nodepool configs if/when that happens | 16:42 |
| fungi | absolutely | 16:44 |
| mordred | clarkb: your comment is correct | 16:51 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: [dnm] Use diskimages-gloabls section https://review.opendev.org/713158 | 16:57 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: nodepool config : add ZUUL_USER_SSH_KEY https://review.opendev.org/713160 | 16:57 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Verbosely apply Zuul ssh key in nodepool config https://review.opendev.org/713289 | 16:57 |
| clarkb | fungi: mordred ianw ^ fyi I went ahead and stacked the verbose version under the cleanup version so that we don't have to wait on nodepool design stuff to make the functional change | 16:58 |
| clarkb | Next on the list is figuring out where we ended up with dnf installations being slow | 17:04 |
| fungi | didn't ianw work out that it was lvm scanning during kernel package installation? | 17:06 |
| clarkb | ya reading scrollback in -infra seems it doesn't do it on a fresh host but does repreoduce listing vgs on our existing nb01.opendev builder | 17:07 |
| clarkb | I guess we don't have changes for that yet, and probably debug that further after building nb04? | 17:07 |
| fungi | right, but also it may be something we run into over time. maybe there are a ton of orphaned cruft loop block devices it's trying and timing out on? | 17:10 |
| clarkb | do loop block devices affect lvm vgs listing? I guess it scans the devices for lvm stuff? | 17:12 |
| mnaser | clarkb: i can abandon https://review.opendev.org/#/c/713136/ in favour of 713160 ? | 17:18 |
| clarkb | mnaser: yes, I think after debugging that we decided being explicit was desireable (to avoid similar confusion in the future) | 17:18 |
| mnaser | cool, i'll abandon with a note | 17:19 |
| fungi | clarkb: vgscan checks all your block devices, i believe | 17:19 |
| fungi | depends on what exactly is happening, of course | 17:19 |
| openstackgerrit | Merged opendev/system-config master: Undo debian changes to openssl.cnf for python-base https://review.opendev.org/713278 | 17:36 |
| mnaser | AJaeger: can I get the +2 now? :) | 17:48 |
| AJaeger | mnaser: +2A ;) | 17:55 |
| openstackgerrit | Merged openstack/project-config master: opendev: move vexxhost to seperate tenant https://review.opendev.org/713123 | 18:08 |
| mnaser | thanks! | 18:20 |
| AJaeger | mnaser: have fun: http://zuul.opendev.org/t/vexxhost/config-errors | 18:53 |
| mnaser | AJaeger: nice. | 18:55 |
| mordred | mnaser: as you work on that a) it would be great to fix places where there's too tight of a coupling and b) it would be great to have a doc of "here's what you need to reconsume devstack" | 19:10 |
| mnaser | mordred: it seems like its' mostly required-projects and im not sure how to decouple that | 19:12 |
| mnaser | so far i had to add openstack/requirements and https://github.com/openstack/devstack/blob/master/.zuul.yaml#L366-L373 | 19:12 |
| mordred | mnaser: nod | 19:13 |
| mnaser | mordred: having said that, the "bug" i think zuul has is | 19:13 |
| mnaser | it seems to fail after the first missing dependency | 19:13 |
| mnaser | so it only talks about openstack/cinder missing in all those, not the others | 19:14 |
| mordred | nod | 19:14 |
| mnaser | so there's probably a "return" in there when there shouldn't, but i don't have the time right now to look more into it | 19:14 |
| mnaser | i'll post in #zuul about it for now.. | 19:14 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: vexxhost: add repos needed for required-projects for devstack https://review.opendev.org/713317 | 19:18 |
| corvus | we maybe should have talked about the vexxhost tenant a bit more | 19:18 |
| mnaser | corvus: happily can revert/discuss if it's an issue | 19:18 |
| corvus | i'm sorry i didn't have time earlier, it seems i'm swamped | 19:18 |
| mnaser | i was trying to not pollute the opendev namespace by adding more required-projects for required-projects | 19:19 |
| corvus | mnaser: yeah, it's reversible, and just to be clear, it's not upsetting or anything; just more of i don't think we totally worked out what that would mean :) | 19:19 |
| mnaser | corvus: yeah. i mean to me it's not anything that wouldn't live under any other tenant, the only purpose was just not polluting the tenant with tons of extra projects which end up openstack-ifying the opendev tenant | 19:19 |
| mnaser | happy to discuss though. i just figured it's probably easier for these projects where we know other people might be contributing to live here rather than in our own gerrit (because i sure as hell not going to go through the pain of github) | 19:20 |
| corvus | agreed; we should have an answer for that, and i don't know what it is. maybe this is the right answer. but (aside from the interminable tedium of figuring out what the required projects are), other gotchas i can think of are multiple merge or config error reports, and we can't add a dedicated config project to this tenant (for security reasons) | 19:21 |
| corvus | mnaser: btw, if you do figure out what the appropriate set of required projects is, can you please document it somewhere? because every third-party ci system goes through the exact same thing, and when they're done, they disappear and don't tell the next person what the config is supposed to be. | 19:22 |
| mnaser | corvus: happily. i just don't know where is the best place, but i can gladly do that, i am thinking it might take one or two iterations | 19:23 |
| corvus | well, i guess opendev's tenant config might be that documentation :) | 19:23 |
| mnaser | so hopefully y'all are a bit more patient with me taking a few patches to get it right as i did as much research as i could | 19:23 |
| mnaser | yeah, fwiw i have no interest in maintaining a config project or wanting to have one | 19:24 |
| AJaeger | mnaser: we trade you some random reviews ;) | 19:24 |
| corvus | mnaser: anyway, about the config-project: hopefully you can live without one (i think probably so). and about the multiple merge/config errors -- i'm not sure about how to address that. | 19:24 |
| mnaser | i'm totally ok with using the opendev ones, i dont need any custom pipelines or my own base jobs | 19:25 |
| mordred | the multiple config errors one is the most troubling one | 19:25 |
| mnaser | AJaeger: i did my first attempt there ^ and then hopefully that's it, then maybe somewhere in openstack/devstack might be a good place to put it.. | 19:26 |
| mnaser | i've topic'd them 'devstack-in-zuulv3' and if any follow ups come ill put them there | 19:26 |
| mordred | largely because it can be confusing when you get a config error on an openstack repo fora. config issue in a different tenant - there is no error message - but that's maybe an experience we should improve | 19:26 |
| mnaser | so if i dont end up pushing a patch to document it, at least there will be a series of patches to look at :p | 19:27 |
| mnaser | AJaeger: i'll take a trade for one of those random reviews now :) | 19:36 |
| AJaeger | mnaser: just pick something anytime - no urgency ;) | 19:37 |
| AJaeger | mnaser: +2A | 19:37 |
| openstackgerrit | Merged openstack/project-config master: vexxhost: add repos needed for required-projects for devstack https://review.opendev.org/713317 | 19:45 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool-builder: put container configs in /etc https://review.opendev.org/713148 | 20:37 |
| ianw | mordred/clarkb: ^ thanks, yes i forgot to map it in. i can look at some "docker exec" type tests to ensure things in the container i guess; it doesn't map directly to testinfra's nice api | 20:38 |
| mordred | ianw: lgtm | 20:38 |
| mordred | ianw: related: https://review.opendev.org/#/c/713060/ | 20:39 |
| mordred | ianw: and we landed https://review.opendev.org/#/c/713278/ | 20:39 |
| mordred | ianw: so once we build a new nodepool-builder image the ssl thing should be taken care of too | 20:39 |
| ianw | mordred: yep, cool. i don't know if we want to tell rax about it | 20:40 |
| ianw | i think it has something to do with SNI and webserver configs and sha1 | 20:40 |
| ianw | that's my bug report :) | 20:40 |
| ianw | corvus / mordred / clarkb : with https://review.opendev.org/#/c/713057/ do we no longer need to think about making a nb04.opendev.org for a redeploy? | 20:41 |
| ianw | i.e. we can keep nb01.opendev.org? and the name won't conflict now? | 20:42 |
| ianw | or is there more to do? | 20:42 |
| mordred | ianw: not sure - Shrews? | 20:42 |
| ianw | i wasn't clear on that bit from the other day; https://storyboard.openstack.org/#!/story/2007407 task #39063 | 20:42 |
| ianw | <corvus> ianw: i think we're a little fuzzy on the contribution of the short-names -- the current system is using a unique short name but also completely duplicated file with all the images | 20:43 |
| mordred | ianw: oh - so I'm going to clean up review-dev manually by doing a podman-compose down, apt-get remove podman, pip uninstall podman-compose and remove the podman ppa config - I think that should do it | 20:46 |
| mordred | I'm guessing when we're ready on nb01 we can do similar - or can just wait til we land the switch-to-docker patch before spinning up nb04 if we're gonna go that route | 20:47 |
| clarkb | ianw: my understanding is we could use nb01 as the name but have to upgrade all builders forst | 20:48 |
| ianw | sounds good -- yeah i think we want to start the nb host again | 20:48 |
| clarkb | nb04 might be less confusing overall though | 20:48 |
| ianw | yeah, it also didn't seem to update what you see in "nodepool dib-image-list", or maybe it did | 20:49 |
| ianw | perhaps i should restart all the builders anyway today, but still move forward with nb04? | 20:49 |
| ianw | when the config change and the ssl change are fully merged, i think we have what we need to try again | 20:50 |
| clarkb | ya I think lets do nb04 and restarting anyway sounds good | 20:50 |
| Shrews | mordred: ianw: the nodepool fix would fix it, but we'd need to restart nodepool first. probably using nb04 is a good idea anyway? | 20:50 |
| clarkb | ianw: we may need to build new nodepool.images | 20:50 |
| clarkb | the dockerfile update went into the parent image | 20:50 |
| clarkb | (so we dont have to fix ssl for each child) | 20:51 |
| ianw | yeah, good point, do we actually need a release to upload them? | 20:51 |
| clarkb | no just a change to merge | 20:52 |
| clarkb | Shrews: ^ know of any changes wemight be able to land in order to get new images? | 20:58 |
| ianw | i think there's mordred's dockerfile change in the queu? | 21:00 |
| ianw | https://review.opendev.org/#/c/713060/ | 21:01 |
| Shrews | fungi's py38 change might be easiest for now | 21:01 |
| Shrews | i also have a cli change up that's pretty safe | 21:02 |
| clarkb | corvus: can you rereview 713060 I think you understand that one best | 21:03 |
| * fungi strives to make easy changes | 21:03 | |
| clarkb | though I think I'm ok to approve it | 21:03 |
| openstackgerrit | Merged openstack/project-config master: Verbosely apply Zuul ssh key in nodepool config https://review.opendev.org/713289 | 21:04 |
| Shrews | https://review.opendev.org/712539 is straight forward | 21:04 |
| clarkb | actually https://review.opendev.org/#/c/713060/ seems to still remove the sudo stuff corvus asked for morded to add back in? | 21:08 |
| clarkb | mordred: ^ | 21:08 |
| corvus | agreed, left comment | 21:09 |
| ianw | NPM is joining github ... top voted HN comment "this seems like a good outcome overall" ... we live in interesting times | 21:30 |
| * fungi read the announcement as "npm is now a microsoft product" | 21:31 | |
| fungi | they'll have some very useful additional insight into what' | 21:32 |
| fungi | s deployed to a lot more servers on the internet | 21:32 |
| ianw | well i guess i work for IBM, so stones and glass houses | 21:37 |
| mnaser | ah damn | 22:17 |
| mnaser | i almost got everything right | 22:17 |
| mnaser | for the devstack stuff | 22:17 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: vexxhost: load openstack/tempest jobs https://review.opendev.org/713339 | 22:19 |
| mnaser | AJaeger, mordred: that might be the last one i hope ^ | 22:20 |
| mnaser | welcome other reviews as i go through my 1-2 hour timeouts to land and see if it works or not :P | 22:20 |
| mordred | ianw: https://review.opendev.org/#/c/713101/ is good to go on my end now - I have stopped and uinstalled podman stuff on review-dev | 22:22 |
| mordred | ianw: I think it should be safe to land on your end too, yes? | 22:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove the Rackspace Cloud repo https://review.opendev.org/713341 | 22:30 |
| mordred | clarkb: ^^ I found something while cleaning up review-dev | 22:31 |
| clarkb | cool my power is out so I'm gonna do something else for a bit | 22:31 |
| mordred | clarkb: great choice! | 22:31 |
| clarkb | power company says I've got 3 hours of no power | 22:34 |
| clarkb | so that may be it for me today. I've not yet sent an agenda put but will try to if things are back before bed | 22:34 |
| ianw | mordred: yes, nb01.opendev in emergency so no issues there | 23:28 |
| *** DSpider has quit IRC | 23:33 | |
| mordred | ianw: ok. I hit +A on it | 23:42 |
| ianw | i'll work on an nb04 a bit later | 23:42 |
| ianw | i am *really* hoping that the fedora LVM/udev/uevent/??? madness does not reappear :/ | 23:43 |
| fungi | my money's on vgscan iterating over a bunch of dead loop devices | 23:49 |
| ianw | fungi: i think those it's getting some bad feedback -- the process of probing is somehow making some other even that starts a new probe or similar | 23:50 |
| ianw | event | 23:50 |
| fungi | ahh | 23:50 |
| fungi | chain reaction | 23:51 |
| ianw | yeah, difficult to debug chain reactions seem to be the theme of 2020 | 23:58 |
| fungi | maybe if we keep the processes at least 2 meters apart | 23:58 |
| ianw | :) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!