| ianw | clarkb: ^ i think that's what i'm thinking. so you'd just add an argument "zuul_data" to tests and zuul_data['inventory'] will be the inventory, and zuul_data['extra'] can be arbitrary things we might want to pass from run-base.yaml (specific test variables was mentioned by corvus) | 00:00 |
|---|---|---|
| *** tosky has quit IRC | 00:01 | |
| clarkb | ianw: why do you assert it is none in test_bridge | 00:03 |
| clarkb | are you just forcing that to fail to debug the contents? | 00:03 |
| openstackgerrit | Douglas Mendizábal proposed openstack/project-config master: Configure ansible-role-lunasa-hsm for release https://review.opendev.org/729334 | 00:05 |
| ianw | clarkb: yeah, just for wip to see if things are making it through | 00:07 |
| *** mlavalle has quit IRC | 00:15 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] testinfra: create a fixture of data from zuul https://review.opendev.org/729418 | 00:19 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] testinfra: create a fixture of data from zuul https://review.opendev.org/729418 | 01:00 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] testinfra: create a fixture of data from zuul https://review.opendev.org/729418 | 01:35 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] testinfra: create a fixture of data from zuul https://review.opendev.org/729418 | 02:02 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 02:35 |
| ianw | mordred: ^ yeah ... so because we use openstack requirements the master branch really has to be compatible with victoria, ergo 3.6 | 02:35 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: testinfra: pass inventory and zuul data https://review.opendev.org/729418 | 02:54 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 02:54 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: testinfra: pass inventory and zuul data https://review.opendev.org/729418 | 03:19 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 03:19 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 03:38 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: testinfra: pass inventory and zuul data https://review.opendev.org/729418 | 03:41 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 03:41 |
| ianw | hrw: ok, you can call it frankenstein's monster but i fiddled with the dib on the nb03 host and got a focal image out https://nb03.openstack.org/ubuntu-focal-arm64-0000000885.log | 04:09 |
| ianw | basically forced an apt fix and manually incoprorated https://review.opendev.org/#/c/726996/4 | 04:10 |
| ianw | this is probably enough until we get the containerised builder | 04:10 |
| *** ykarel|away is now known as ykarel | 04:24 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 04:28 |
| ianw | clarkb / fungi : ^ that should be ready for review. clarkb you might be interested in 729418 stacked under it for the fixture stuff. | 04:29 |
| *** diablo_rojo has quit IRC | 04:30 | |
| *** raukadah is now known as chandankumar | 04:36 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 05:01 |
| *** hashar has joined #opendev | 05:09 | |
| *** calcmandan has quit IRC | 05:16 | |
| *** calcmandan has joined #opendev | 05:17 | |
| *** dpawlik has joined #opendev | 05:57 | |
| *** hashar has quit IRC | 05:59 | |
| openstackgerrit | fuzihao proposed opendev/ansible-role-cloud-launcher master: Fix pygments style https://review.opendev.org/729512 | 06:06 |
| *** ysandeep|away is now known as ysandeep | 06:39 | |
| *** slaweq has joined #opendev | 06:46 | |
| openstackgerrit | fuzihao proposed opendev/ansible-role-cloud-launcher master: Fix pygments style https://review.opendev.org/729512 | 06:49 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Do not interpolate values from tox --showconfig https://review.opendev.org/729520 | 07:09 |
| *** tosky has joined #opendev | 07:30 | |
| ianw | ... Timeout on http://mirror.kna1.airship-citycloud.opendev.org/centos/8/AppStream/ | 07:46 |
| ianw | https://zuul.opendev.org/t/openstack/build/f9013b96e1b74565a2ddf5d859029758/log/logs/centos-minimal_8-build-succeeds.FAIL.log | 07:46 |
| ianw | seems ok to me, but one to keep an eye on | 07:46 |
| ianw | actually : [Thu May 7 18:00:29 2020] INFO: task apache2:29677 blocked for more than 120 seconds. | 07:48 |
| ianw | that's obviously a while ago | 07:48 |
| ianw | it might be worth a reboot, just in case something old is hanging around | 07:49 |
| ianw | A start job is running for OpenAFS client (10s / 1min 35s) | 07:57 |
| ianw | this host is not happy | 07:57 |
| ianw | alright, maybe i was hasty. it's back, it seems to be serving. one to keep an eye on | 08:00 |
| ianw | #status log rebooted mirror.kna1.airship-citycloud.opendev.org ; it was refusing a few connection and had some old hung processes lying around | 08:00 |
| openstackstatus | ianw: finished logging | 08:00 |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| hrw | ianw: whatever works | 08:16 |
| hrw | ianw: recheck of https://review.opendev.org/#/c/728798 started | 08:17 |
| *** tkajinam has quit IRC | 08:23 | |
| *** DSpider has joined #opendev | 08:27 | |
| hrw | ianw: and failed: https://zuul.openstack.org/stream/c1147813864b47e1858dcd116a4a60f4?logfile=console.log - unable to install packages due to hold packages | 08:39 |
| hrw | failed: E: Unable to correct problems, you have held broken packages.\n" | 08:40 |
| hrw | ianw: and python3-wheel needs to be installed: error: invalid command 'bdist_wheel' | 08:44 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 08:45 |
| hrw | ianw: are all needed components to reproduce build from nb03 included in dib repo? I would look later into it | 08:45 |
| *** ykarel is now known as ykarel|lunch | 08:57 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/system-config master: Switch prep-apply to use python3 https://review.opendev.org/729543 | 08:58 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/system-config master: WIP: Experiment newer ubuntu with puppet https://review.opendev.org/729544 | 09:07 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/system-config master: WIP: Experiment newer ubuntu with puppet https://review.opendev.org/729544 | 09:11 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/system-config master: WIP: Experiment newer ubuntu with puppet https://review.opendev.org/729544 | 09:18 |
| *** ysandeep is now known as ysandeep|lunch | 09:29 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/system-config master: WIP: Switch prep-apply to use python3 https://review.opendev.org/729543 | 10:00 |
| *** ysandeep|lunch is now known as ysandeep | 10:09 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Do not interpolate values from tox --showconfig https://review.opendev.org/729520 | 10:27 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Do not interpolate values from tox --showconfig https://review.opendev.org/729520 | 10:37 |
| donnyd | johnsom: If there is anything I can help do to make it easier please lmk. If IPv6 was busted, well pretty much all of OE wouldn't really work | 10:40 |
| *** ykarel|lunch is now known as ykarel | 10:46 | |
| *** ysandeep is now known as ysandeep|brb | 11:00 | |
| *** ysandeep|brb is now known as ysandeep | 11:13 | |
| *** sshnaidm is now known as sshnaidm|afk | 11:42 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/system-config master: WIP: Switch prep-apply to use python3 https://review.opendev.org/729543 | 11:45 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/system-config master: WIP: Switch prep-apply to use python3 https://review.opendev.org/729543 | 12:06 |
| *** hashar has joined #opendev | 12:26 | |
| *** lpetrut has joined #opendev | 12:32 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/system-config master: Switch prep-apply.sh to use python3 https://review.opendev.org/729543 | 12:35 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/system-config master: WIP: Experiment newer ubuntu with puppet https://review.opendev.org/729544 | 12:35 |
| zbr | any chance to get ubuntu-focal images? | 12:36 |
| AJaeger | zbr: we should have them already... | 12:40 |
| zbr | AJaeger: right, my mistake. I am already seeing them. | 12:41 |
| zbr | but install-puppet role does not support it, i wonder if puppet-5 is compatible with 4, or not really. | 12:41 |
| AJaeger | no idea about puppet ;( | 12:42 |
| *** sshnaidm|afk is now known as sshnaidm | 12:53 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/system-config master: WIP: Add focal support to install-puppet https://review.opendev.org/729586 | 12:53 |
| hrw | hm. focal... | 12:54 |
| hrw | have to check backlog to see was there something about arm image builder upgrades | 12:54 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/system-config master: WIP: Experiment newer ubuntu with puppet https://review.opendev.org/729544 | 12:56 |
| mordred | zbr: oh - I wouldn't bother with that - if we need a focal image, we should transition to ansible rather tahn to puppet 5 | 12:56 |
| zbr | mordred: yeah.... i was starting to learn that the hard way. | 12:57 |
| zbr | or puppet6 ! | 12:57 |
| mordred | yah! | 12:57 |
| zbr | but maybe switching from xenial with py27 to focal with py38 is a too big leap. | 12:58 |
| mordred | might not be - which thing are you looking in to? | 12:58 |
| mordred | zbr: py3.5 is also on xenial | 12:58 |
| zbr | yep but py35 is going away very soon | 12:58 |
| zbr | and many projects already dropped support for it | 12:59 |
| zbr | i also did the same with molecule, ansible-lint, and few other projects. minimal viable python is 3.6 | 12:59 |
| mordred | yeah - zuul's min is 3.5 at the moment though - and that's the python xenial - so we still support it in opendev | 13:00 |
| hrw | zbr: 3.6 == bionic ;D | 13:00 |
| mordred | that said ... | 13:00 |
| hrw | zbr: so you can be in a middle | 13:00 |
| zbr | yep, that is the logic move. | 13:00 |
| mordred | zbr: I also want to make prep-apply go away | 13:00 |
| zbr | sweet/safe range is 36-38 | 13:00 |
| hrw | mordred: is there a way to download raw image used by zuul node? | 13:01 |
| zbr | hrw++ i asked the same in the past. | 13:01 |
| mordred | zbr: we have better jobs we can run now in system-config that actually run the production ansible->puppet and don't need to do the job that does prep-apply - but that's also a long tail to pull on :) | 13:02 |
| zbr | being able to download them could prove very useful for development purposes. | 13:02 |
| mordred | hrw: there are links to the images from the nodepool builder nodes | 13:02 |
| hrw | mordred: thanks | 13:02 |
| mordred | oh - wait | 13:02 |
| mordred | no, that's just the logs | 13:02 |
| mordred | no, I don't think we have the images themselves published - although they are VERY LARGE so it would be unpleasant to download them | 13:03 |
| zbr | we should publish them, as a developer I would find it very useful. curl has good resume support. | 13:04 |
| hrw | ELEMENTS_PATH is: /usr/local/lib/python3.5/dist-packages/diskimage_builder/elements:/etc/nodepool/elements - where can I get /etc/nodepool/elements? | 13:04 |
| hrw | so I can rebuild exactly same image | 13:04 |
| mordred | hrw: https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements | 13:05 |
| hrw | 'docker run --rm -it -u root ubuntu:xenial bash' should still work to even have xenial | 13:05 |
| hrw | mordred: thanks | 13:05 |
| mordred | hrw: yes - you'll want to add --privileged to your docker run - dib needs to mount things | 13:06 |
| hrw | thx | 13:06 |
| zbr | mordred: clarkb: https://review.opendev.org/#/c/729543/5 please. | 13:06 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/puppet-elastic_recheck master: Use py3 with elastic-recheck https://review.opendev.org/729336 | 13:08 |
| mordred | zbr: ah - I now know what you're chasing :) | 13:10 |
| zbr | yep, reviving er. | 13:10 |
| zbr | avass: you want a new release of tox with the fix? gabor was asking me if he should tag one. | 13:15 |
| frickler | infra-root: seems we missed changing the mirror config for dib from openstack ro opendev somewhere, see e.g. https://nb02.opendev.org/ubuntu-bionic-plain-0000022490.log | 13:16 |
| frickler | s/ro/to/ | 13:16 |
| avass | zbr: I think that would be good | 13:21 |
| hrw | looks like it is time for xenial VM. container is not enough ;D | 13:32 |
| zbr | what do I need to do to assure that uploaded .sh files are viable in browsera instead of downloaded? | 13:39 |
| *** rosmaita has joined #opendev | 13:39 | |
| rosmaita | hello opendevvers -- the tag indexing on the web git browser interface seems to be out of date -- compare https://opendev.org/openstack/cinder/commit/7e98d14a5724efaa8b02d8dc1c5d28cde7ce0ea6 to https://github.com/openstack/cinder/commit/7e98d14a5724efaa8b02d8dc1c5d28cde7ce0ea6 --ours is showing only tags/16.0.0.0rc1 whereas theirs is showing all the RCs plus 16.0.0 | 13:40 |
| *** ykarel is now known as ykarel|afk | 13:46 | |
| *** olaph has quit IRC | 13:49 | |
| zbr | mordred: i am trying to find why apply-test.sh failed but fail to spot it, https://zuul.opendev.org/t/openstack/build/fb80492306a249e188eedb5c96c819ac | 13:49 |
| zbr | the output is far from friendly, so much spam from cloning part | 13:50 |
| mordred | zbr: yeah - that's one of the reasons I want to make that whole thing go away | 13:50 |
| zbr | i only spotted that "puppet run" task failed, but not idea why or with what output. | 13:50 |
| mordred | zbr: https://zuul.opendev.org/t/openstack/build/fb80492306a249e188eedb5c96c819ac/log/applytest/puppetapplytest20.final.out.FAILED | 13:51 |
| mordred | zbr: check the applytest dir in the logs and look for one with FAILED in the name | 13:51 |
| mordred | https://zuul.opendev.org/t/openstack/build/fb80492306a249e188eedb5c96c819ac/log/applytest/puppetapplytest20.final.out.FAILED#94 | 13:51 |
| mordred | is going to be your issue | 13:51 |
| zbr | ok thanks. i wonder how hard it would to reimplement er deployment in ansible. | 13:52 |
| mordred | zbr: not super hard. the general pattern we'd be looking at would be first to add a Dockefile to the e-r repo using the python-builder/python-base images to make an e-r image - then make ansible in system-config to deploy it using docker-compose - we have several examples of this now | 13:53 |
| mordred | we'd use ansible instead of puppet to write the config files onto disk, and then set up appropriate mounts in the compose file | 13:54 |
| zbr | ok, show me one example and I will go this path as I am confident with these, opposed to puppet. | 13:55 |
| zbr | this should also make easy to run er locally for testing. | 13:55 |
| cloudnull | To build upon what zbr asked, re: .sh files are browse-able instead of automatically downloaded -- https://pasted.tech/pastes/9431514256fdccc0fe9d5f76cdc5c7187d8ff026 | 13:58 |
| cloudnull | It looks like we're letting swift set the content-type based on file extensions, is there a way we can control that without appending something like ".txt" to every file? | 13:58 |
| mordred | zbr: zuul-registry is actually a good example. look at https://opendev.org/zuul/zuul-registry to see what the Dockefile should look like. the gerritbot dockerfile is actually a good example of a dockerfile with the right opendev zuul jobs for the container publication: https://opendev.org/opendev/gerritbot/src/branch/master/Dockerfile | 13:59 |
| mordred | (but I haven't finished deploying gerritbot, so it's not a good example to follow end to end) | 13:59 |
| mordred | zbr: then in system-config, see playbooks/roles/registry, the system-config-run-docker-registry job (for testing) https://opendev.org/opendev/system-config/src/branch/master/zuul.d/system-config-run.yaml#L412 - and infra-prod-service-registry for deployment: https://opendev.org/opendev/system-config/src/branch/master/zuul.d/infra-prod.yaml#L291 | 14:00 |
| mordred | zbr: as well as https://opendev.org/opendev/system-config/src/branch/master/playbooks/service-registry.yaml | 14:01 |
| corvus | mordred: do we have a focal node running anywhere? | 14:03 |
| *** ykarel|afk is now known as ykarel | 14:03 | |
| corvus | the apt_key module isn't working on focal; it's reporting | 14:04 |
| corvus | mv: cannot stat '/etc/apt/trusted.gpg.d/openafs.asc': No such file or directory | 14:04 |
| mordred | corvus: "awesome" | 14:04 |
| mordred | corvus: well - fwiw, on focal you can just put an .asc file into /etc/apt/trusted.gpg.d without using apt_key | 14:04 |
| corvus | mordred: yeah, but if the directory doesn't exist... :/ | 14:04 |
| mordred | corvus: other thing to try - create the directory | 14:04 |
| corvus | that's why i wanted to poke at a focal node | 14:04 |
| mordred | yea - I don't think we have one yet | 14:05 |
| corvus | k | 14:05 |
| zbr | apparently install-from-bindep does install all reqs from bindep, so I endup with both. python versions, which I do not want. | 14:07 |
| *** hashar has quit IRC | 14:10 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Vendor the apt repo gpg keys used for Zuul https://review.opendev.org/729401 | 14:19 |
| corvus | mordred: enjoy the patchset diff on that ^ | 14:20 |
| mordred | corvus: I ... uhm | 14:23 |
| mordred | corvus: no, I will choose to not enjoy that patchset diff | 14:23 |
| mordred | :) | 14:23 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/project-config master: Fix outdated mirror references https://review.opendev.org/729618 | 14:24 |
| frickler | infra-root: ^^ this should fix the build failure I quoted earlier | 14:25 |
| frickler | also, tomorrow is a bank holiday here, so I'll be mostly offline | 14:26 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Save zuul and nodepool logs from gate test jobs https://review.opendev.org/729619 | 14:27 |
| corvus | mordred: i think https://review.opendev.org/726475 is ready for +3 and https://review.opendev.org/726958 for +2 -- neither get everything working yet, but they're steps. | 14:30 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: WIP: Create elastic-recheck docker image https://review.opendev.org/729623 | 14:42 |
| *** lpetrut has quit IRC | 14:46 | |
| *** mlavalle has joined #opendev | 14:49 | |
| openstackgerrit | Merged openstack/project-config master: Fix outdated mirror references https://review.opendev.org/729618 | 14:51 |
| *** priteau has joined #opendev | 14:51 | |
| *** sgw has joined #opendev | 14:52 | |
| clarkb | hrw mordred zbr we do publish images (and have for years now) https://nb01.opendev.org/images and https://nb02.opendev.org/images | 15:01 |
| clarkb | hrw is probably interested in those on nb03.openstack.org which uses the same url path | 15:02 |
| clarkb | and yes they are large and not always pleasant to deal w ith | 15:02 |
| hrw | clarkb: thanks | 15:03 |
| hrw | fetching | 15:05 |
| fungi | rosmaita: that's intriguing... i've not paid close attention to the found in tags display on commits in gitea, but it seems to only display the earliest tag which incorporated the commit, as opposed to github which shows a tag range (which could be misleading as not all tags are guaranteed sequential) | 15:16 |
| rosmaita | fungi: yeah, i had not really paid attention either until i needed to look at a particular commit today | 15:16 |
| hrw | fetched. will play with it | 15:17 |
| clarkb | fungi: ya I think they are showing two different pieces of info | 15:17 |
| clarkb | With gitea I think it is showing the oldest tag reachable | 15:19 |
| clarkb | https://opendev.org/openstack/cinder/commit/cc9014ab4298a4509fd6139fe5b9ca4c7c69120d illustrates that a bit better as it shows 16.0.0.0rc3^0 as the tag there | 15:20 |
| clarkb | but 16.0.0 is also tagged on that commit | 15:20 |
| clarkb | rc3 is older so gets displayed | 15:20 |
| fungi | out of curiosity, do you happen to know what the ^0 on the end of that tag signifies? | 15:21 |
| clarkb | fungi: its a git delta. | 15:21 |
| clarkb | I think | 15:21 |
| clarkb | eg HEAD^0 | 15:22 |
| clarkb | eg HEAD^1 is more useful | 15:22 |
| clarkb | basically thats them saying the tag is on this commit not simply included by the tag? | 15:22 |
| fungi | is ^0 just a self-reference then? | 15:22 |
| fungi | oh! | 15:22 |
| fungi | got it | 15:22 |
| fungi | so saying this is the actual tagged commit | 15:22 |
| clarkb | ya I think so | 15:22 |
| fungi | makes sense then | 15:23 |
| rosmaita | clarkb: fungi: ok, thanks, that's good to know -- they are supposed to be displaying different information, so everything is fine | 15:23 |
| clarkb | https://opendev.org/openstack/cinder/commits/branch/stable/ussuri and https://github.com/openstack/cinder/commits/stable/ussuri both fail to show the tag data too which is fun | 15:23 |
| fungi | https://opendev.org/zuul/zuul/commit/3acc00a30eb967556e7e6484f17e1b9019d51c85 | 15:23 |
| fungi | also has the ^0 | 15:24 |
| fungi | so i think that confirms it | 15:24 |
| fungi | rosmaita: yes, what gitea is displaying there should be the earliest reachable tag in which that commit appears | 15:24 |
| fungi | which is usually fine so long as you know which release tag came after it | 15:25 |
| clarkb | fungi: have a quick moment for https://review.opendev.org/#/c/729416/ once that lands I can restart apache on mirrors and we'll have working https for all the mirror things | 15:26 |
| fungi | sure | 15:26 |
| clarkb | fungi: the child would be good to get in too so quay fix can be caught up in the same restart | 15:28 |
| clarkb | rosmaita: fwiw my hunch is that gitea is actaully trying to mimic github behavior there, but hasn't quite gotten it right. Chances are gitea would treat that as a bug if you filed it with them, though I can't say that with absolute certainty | 15:31 |
| clarkb | infra-root gitea v1.12.0-rc1 has been tagged. That will include the performance boost to scanning git repos for last updated info | 15:31 |
| clarkb | reading https://github.com/go-gitea/gitea/blob/v1.12.0-rc1/CHANGELOG.md I expect it will be a straightforward upgrade for us to 1.12 | 15:32 |
| *** priteau has quit IRC | 15:33 | |
| clarkb | separately I've realized there is a flaw with our /etc/hosts munging in jobs. We're wanting things in containers to use those host /etc/hosts values and that isn't working | 15:35 |
| clarkb | mordred: ^ fyi. I'll have to have a think on that | 15:35 |
| mordred | clarkb: oh - poo | 15:41 |
| mordred | clarkb: the internet tells me that net: host should cause the host's network stack, including /etc/hosts to be used | 15:42 |
| clarkb | hrm then why is the job still failing | 15:43 |
| corvus | mordred, clarkb: indeed, --network=host does show my /etc/hosts file in the container | 15:43 |
| clarkb | https://zuul.opendev.org/t/openstack/build/ecf5b2653f324965be7a63f1d7775a4e/log/jvb01.opendev.org/docker/jitsi-meet-docker_jvb_1.txt#59 that should've been using the hosts change | 15:44 |
| clarkb | but the ip there is for prod meetpad01 | 15:44 |
| clarkb | maybe its a java thing? | 15:44 |
| clarkb | I guess I'll have to look closer | 15:44 |
| corvus | clarkb: oh yeah, java famously does its own dns resolution | 15:44 |
| corvus | it's really helpful | 15:45 |
| mordred | I think we're probably discovering why k8s runs an internal dns service | 15:45 |
| fungi | what little i know about java, i would expect it to look for an equivalent of a hosts file inside its jvm as an override | 15:45 |
| openstackgerrit | Merged opendev/system-config master: Use ensure-nodejs in Gerrit deployment testing https://review.opendev.org/729362 | 15:45 |
| corvus | it ignores ttls too | 15:46 |
| corvus | (that's how i learned about it) | 15:46 |
| mordred | corvus: it's doing that to be "helpful | 15:46 |
| *** priteau has joined #opendev | 15:46 | |
| fungi | "JNDI DNS service provider settings" at the bottom of https://docs.oracle.com/javase/1.5.0/docs/guide/net/properties.html | 15:48 |
| clarkb | fungi: that says it should use the system default by default | 15:48 |
| mordred | https://cr.openjdk.java.net/~iris/se/11/latestSpec/api/java.base/java/net/doc-files/net-properties.html | 15:50 |
| mordred | I don't see similar settings for recent openjdk | 15:50 |
| fungi | yeah, so i guess that's specific to oracle's jvm | 15:50 |
| fungi | and maybe blackdown if that's still around | 15:50 |
| clarkb | I think I see the issue | 15:51 |
| clarkb | https://zuul.opendev.org/t/openstack/build/ecf5b2653f324965be7a63f1d7775a4e/console#1/0/24/jvb01.opendev.org | 15:51 |
| clarkb | its using the actual servername not the logical node name in the nodeset | 15:51 |
| clarkb | also ist only doing it for ipv4 (and not ipv6) but that is less of a concern | 15:51 |
| corvus | it seems that role is written with some different assumptions | 15:52 |
| corvus | we could add the logical node name to the list of names | 15:52 |
| corvus | that seems like it should be non-disruptive. i'm less sure about adding v6 addresses though | 15:53 |
| clarkb | ++ and agreed ipv6 may be a bit more disruptive since things will try to talk over ipv6 by default | 15:54 |
| corvus | could add a feature flag | 15:54 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: WIP add support for multiple jvbs behind meetpad https://review.opendev.org/729008 | 15:56 |
| clarkb | ^ is a new patchset that attempts to use the ansible inventory to look this up | 15:56 |
| clarkb | can always switch back to the name if/when we update the hosts file role | 15:56 |
| clarkb | I think the hosts file role also updates ssh known hosts, we'll want to ensure the alias names end up in there as well | 15:57 |
| clarkb | fungi: re the ipv4 in ipv6 addressing method, I don't think that helps the testing of that change. We'd still need to distinguish between v4 and v6 now instead of doing it for use of [] it will be for use of ipv6 prefix | 15:58 |
| fungi | yeah, what would be convenient is if there was a list which included all the addresses for both families but prefixed the v4 addresses into v6 notation automatically | 16:00 |
| fungi | otherwise you still need some sort of conditional either inside or outside the loop | 16:00 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Revert "Reject messages to starlingx-discuss-owner" https://review.opendev.org/729649 | 16:17 |
| fungi | infra-root: ^ worth noting, the @qq.com flood to .*-owner addresses for our mailman lists seems to have finally ceased in recent months (not sure when exactly) | 16:18 |
| fungi | whatever botnet was responsible for that must have realized there was very little value in it | 16:19 |
| clarkb | I'm working on a gitea 1.12 change in system-config fwiw | 16:21 |
| clarkb | trying to be careful wtih the template updates | 16:21 |
| *** ykarel is now known as ykarel|away | 16:23 | |
| mordred | clarkb: the template updates are always the fun part | 16:23 |
| clarkb | mordred: ya I think I may try and update our templates to carry the bits we don't want commented out so its clearly we are intentionally removing them | 16:27 |
| clarkb | mordred: beacuse otherwise its really hard to know what is just missing from previous versions and what we don't want to carry anymore | 16:28 |
| hrw | good news: ubuntu-focal-arm64 works ;) | 16:29 |
| hrw | my CI job may time out but node itself works fine | 16:30 |
| mordred | clarkb: I think that's a good idea | 16:33 |
| mordred | hrw: ++ | 16:33 |
| mordred | clarkb: we should really investigate how many of the template sections we delete can be disabled completely by config - or adding config options to allow us to do that so we can not carry those diffs | 16:34 |
| mordred | clarkb: you know - next time we have a bunch of spare time | 16:35 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update to gitea 1.12.0-rc1 https://review.opendev.org/729659 | 16:42 |
| clarkb | mordred: ++ fwiw ^ and reivew of that should be done carefully | 16:42 |
| *** ysandeep is now known as ysandeep|afk | 16:42 | |
| clarkb | I've not tried to run the docker buid locally | 16:42 |
| clarkb | though I realize I maybe should've | 16:42 |
| * hrw out | 16:46 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Added missing retry_delay instance variable https://review.opendev.org/729699 | 16:50 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Update to gitea 1.12.0-rc1 https://review.opendev.org/729659 | 16:50 |
| clarkb | zbr: one thing on https://review.opendev.org/#/c/729543/5 | 16:52 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Switch to ensure-docker role https://review.opendev.org/720145 | 16:52 |
| clarkb | mordred: gitea's changelog also says "Change default charset for MySQL on install to utf8mb4 (#10989)" https://github.com/go-gitea/gitea/blob/v1.12.0-rc1/CHANGELOG.md. Earlier they mention that you can upgrade from like 1.6 to 1.12 so I don't expect htat is an issue for us but thought I'd mention it since it caused so many problems for us with etherpad in the past too | 16:54 |
| mordred | clarkb: "neat" | 16:54 |
| zbr | clarkb: since yesterday few things changed, i realised that it will be easier to switch to ansible/docker and ditch puppet. | 16:55 |
| zbr | bad part is that I keep finding bugs while doing the work and local changes grow very fast. | 16:56 |
| zbr | for example I found https://review.opendev.org/#/c/729699/ while trying to run locally. | 16:57 |
| zbr | so before doing anything to er, i need to refresh gerritlib and assure we test it better, include py36+ testing,... | 16:58 |
| clarkb | zbr: I believe there is python3.6 testing already, but with the narrow use case of manage-projects? | 16:59 |
| zbr | nope, is only py35 which does not make me happy at all. | 16:59 |
| zbr | i use py38 locally.... | 16:59 |
| clarkb | zbr: https://zuul.opendev.org/t/openstack/build/1c00dcb08c974c4abfd5803e73ab4616/log/job-output.txt#542 that is using python3.6 | 17:00 |
| clarkb | you are correct that the unittests aren't doing a 3.6 interpreter run though | 17:00 |
| zbr | anyway, i will take care of this and ping you when CRs are ready | 17:00 |
| clarkb | ya I have 3.8 locally too. I haven o problem with 3.5... | 17:01 |
| clarkb | one thing I've learned about python is that with the quick iteration on v3 releases we've had all sorts of bugs including those to garbage collecting. The upside to 3.5 is they solved a lot of them and it got stable | 17:02 |
| clarkb | granted it was super painful before that got fixed | 17:02 |
| clarkb | so I'm always sort of waiting for everything to break as my rolling relaeses give me new pythons | 17:02 |
| zbr | clarkb: i already said earlier today, py35 is going EOL in September, if I remember well. Many package maintainers already added min_python>=3.6. | 17:03 |
| zbr | i am not proposing to drop it, but to test it. | 17:03 |
| clarkb | yes, I think we should get off of 3.5 I have no problem with that. I just don't understand all the 3.5 hate | 17:03 |
| clarkb | really I think it boils down to rhel dragging their feet then when tehy finally moved they chose 3.6 so now everyone assumes 3.5 is terrible or something | 17:04 |
| clarkb | and soon enough 3.6 will EOL too, but we will have to keep support for it around beacuse rhel | 17:04 |
| clarkb | (so the EOL argument doesn't seem like a great one, really its more what are python libraries converging around in practice and I expect 3.6 will be that thing due to rhel despite a relatively early eol) | 17:04 |
| clarkb | 3.6 EOL is ~end of year 2021 fwiw | 17:05 |
| clarkb | well before rhel8 EOL | 17:05 |
| zbr | what I seen so far is py36-py38 (less of the last) as being best bet. | 17:05 |
| clarkb | fungi: yes, my point was the python EOL matters less than what people are dev'ing libraries against due to distros like rhel8 | 17:07 |
| clarkb | fungi: 3.5 "lost out" because rhel didn't switch to python3 until 3.6 | 17:07 |
| clarkb | which means things are likely to converage around 3.6 in the library community | 17:07 |
| clarkb | for the long term even well beyond 3.6's EOL | 17:07 |
| fungi | sure, that goes back to the "what are we testing" argument (upstream python, or python as packaged by distros) | 17:07 |
| zbr | for start lets do https://review.opendev.org/#/c/729699/ | 17:08 |
| clarkb | even if 3.5's EOL was 10 years from now it probably wouldn't matter because the convergnce would still be around 3.6 | 17:08 |
| zbr | i am surprised that the linter did not spot this one. | 17:08 |
| clarkb | zbr: there has been bit of related fallout due to the refactor electrofelix did. I think we improperly assumed that was better tested | 17:09 |
| zbr | there is another very important aspect, py36 introduced optional static typing, something very useful. | 17:09 |
| clarkb | zbr: that actually works with 3.5 too | 17:11 |
| clarkb | zbr: left a note on 729699, I don't know if it is worht a new patchset but thought I'd let you check it before approving | 17:11 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Added missing retry_delay instance variable https://review.opendev.org/729699 | 17:13 |
| zbr | fixed, lets be consistent. | 17:13 |
| zbr | which job templates should I add to enable at least py36, preferably py38 too to gerritlib? | 17:14 |
| clarkb | zbr: I don't know if there is a template for that. YOu can just add the jobs | 17:14 |
| zbr | okey | 17:15 |
| *** dpawlik has quit IRC | 17:15 | |
| clarkb | openstack has its branch specific sets which don't really may sense there | 17:15 |
| clarkb | fungi: re what python are we testing, you saw the reactions to the suggestion that openstack build python base image containers :) | 17:17 |
| clarkb | fungi: I expect that attitude will be common "in the real world" basically the expectation is that rhel and debian work | 17:17 |
| zbr | ouch... the library has no tests at all, only one fake one. | 17:19 |
| clarkb | zbr: ya I think that may have been why we assumed poorly now that my memroy is coming back | 17:19 |
| clarkb | and that is what prompted the integration test since really what mattered was our ability to interact with gerrit | 17:19 |
| clarkb | the integration test doesn't test the stream watcher though, just other bits of the api used to create and modify gerrit projects | 17:20 |
| *** ysandeep|afk is now known as ysandeep | 17:22 | |
| clarkb | https://zuul.opendev.org/t/openstack/build/922e6566918c478a85c17e1185d845cb/log/jvb01.opendev.org/docker/jitsi-meet-docker_jvb_1.txt#64 <- I think that shows the extra jvb is mostly working? | 17:25 |
| clarkb | it also reports successful healthchecks. I'm not sure if that is different than the other stats things | 17:25 |
| corvus | clarkb: lgtm, now seems like a great time to spin it up and see :) | 17:26 |
| clarkb | corvus: ya why don't I remove my WIP prefix then we can properly review it and if happy land it then spin up a jvb node | 17:26 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add support for multiple jvbs behind meetpad https://review.opendev.org/729008 | 17:27 |
| clarkb | infra-root ^ I think we can safely review that now (as well as its parent for the firewall stuff) | 17:27 |
| clarkb | the way that change is written it should leave the existing meetpad install alone and the only chagnes happen when we create a jvb01 instance | 17:27 |
| corvus | mordred: do you understand this? https://84cacd386095a03eb330-320386062a8fef96051148fe5e7af6b1.ssl.cf2.rackcdn.com/729401/5/check/system-config-run-zuul/c605bdd/bridge.openstack.org/ara-report/result/c5bf39b7-9a23-4917-aeb4-fe0263795f43/ | 17:28 |
| fungi | clarkb: yes, on what we're testing, i think part of the misconception comes from naming the jobs after python interpreter versions but actually testing distro-mangled python interpreters without saying which distro (which is arguably as important as which interpreter version) | 17:28 |
| *** priteau has quit IRC | 17:28 | |
| fungi | so the distro used is (probably in most cases) buried in the job definition or a parent, while the interpreter version is out front in the job name | 17:29 |
| zbr | on ansible zuul, i already started to include disto in name, like https://dashboard.zuul.ansible.com/t/ansible/job/molecule-tox-py37-fedora-30 | 17:33 |
| zbr | in order to make it clear that I test pyXY-OS-version blends. | 17:33 |
| zbr | and the logic was to test cross platforms only with official distro python versions | 17:33 |
| zbr | this means that py38 should be tested via either py38-fedora-32 or py38-ubuntu-folcal | 17:34 |
| hrw | meh. | 17:34 |
| zbr | i find this setup much closer to what user is. expected to use in production/real-life | 17:35 |
| hrw | I checked wrong and turned out that even focal is too old for openstack requirements ;( | 17:35 |
| hrw | adding debian-bullseye would be too much imho | 17:36 |
| hrw | it is not yet released | 17:36 |
| clarkb | hrw: can you clarify on that? openstack requirements are based on what distros are shipping | 17:36 |
| clarkb | for everything but python deps | 17:36 |
| hrw | no they are not | 17:36 |
| zbr | i could try to write some tests for gerritlib, does any of you have something against pytest? | 17:36 |
| clarkb | and openstack doesn't even support focal yet | 17:36 |
| hrw | confluent_kafka python package on x86 is taken as wheel | 17:36 |
| hrw | on other archs it needs to be built | 17:37 |
| hrw | and it requires librdkafka 1.4.0+ which even focal does not have. | 17:37 |
| clarkb | hrw: the wheel is still going to link against the library right? so how does that work on older distros like bionic? | 17:37 |
| openstackgerrit | Merged opendev/gerritlib master: Added missing retry_delay instance variable https://review.opendev.org/729699 | 17:37 |
| hrw | clarkb: binary wheel from pypi | 17:38 |
| clarkb | hrw: is the binary wheel vendoring the non python bits too? | 17:38 |
| hrw | clarkb: manylinux1 wheel works on centos6+ so even xenial works | 17:38 |
| clarkb | hrw: usually wheels expect the library to be present extenally | 17:38 |
| clarkb | zbr: I tend not to be a fan because I see it repeating many of nose's mistakes | 17:38 |
| clarkb | zbr: reliance on non standard constructs as well as non standard behaviors means you are stuck using that single tool to run all the tests. That limits the sorts of reporting you can produce as well as in many cases being very slow (no parallel testing) | 17:39 |
| clarkb | zbr: I think it is better to run using standards compliant tools in CI as much as possible. That allows developers to use whatever test runner they want locally including pytest | 17:39 |
| hrw | clarkb: -rw-r--r--. 1 hrw hrw 23M 04-18 20:55 librdkafka-186e713e.so.1 | 17:40 |
| hrw | clarkb: linked against minimal set of libs | 17:40 |
| clarkb | hrw: from the wheel? | 17:40 |
| hrw | yes | 17:40 |
| hrw | https://files.pythonhosted.org/packages/47/f5/6a7c2c3a2a880f8e6a3bbf71c3ba685a2761d96925e3c900fce11bfc4478/confluent_kafka-1.4.1-cp36-cp36m-manylinux1_x86_64.whl one | 17:40 |
| fungi | undeed the wheel vendors a confluent_kafka.libs/librdkafka-186e713e.so.1 build | 17:41 |
| hrw | fully complaint with manylinux1 PEP | 17:41 |
| hrw | arch/os specific wheels can contain libraries as long they link to minimal set of libs defined by manylinuxXYZ PEP | 17:41 |
| clarkb | https://github.com/confluentinc/confluent-kafka-python/blob/master/setup.py#L44-L51 is where that is built | 17:42 |
| clarkb | I think that means you can build the package without a wheel too and its fine | 17:42 |
| hrw | clarkb: nope | 17:42 |
| clarkb | hrw: why? if it is vendoring it the system version shouldn't matter | 17:42 |
| fungi | and yeah, openstack/requirements doesn't insist that the package versions it tracks are what distros are providing, only that the packages it tracks are (or can be) included in major distros | 17:42 |
| hrw | clarkb: debian:bullseye is first distro release which has new enough librdkafka | 17:42 |
| clarkb | hrw: but distro release librdkafka doesn't matter as they vendor it | 17:42 |
| clarkb | hrw: so when you install it from source it should build the library on demand | 17:43 |
| fungi | clarkb: the lib is vendored in the *wheel* | 17:43 |
| hrw | clarkb: debian:stretch/buster, ubuntu:xenial/bionic, centos7/8 cannot build it | 17:43 |
| clarkb | the difference with a wheel is they've done the build ahead of time | 17:43 |
| fungi | you need a platform capable of building that lib to create the wheel | 17:43 |
| clarkb | fungi: https://github.com/confluentinc/confluent-kafka-python/blob/master/setup.py#L44-L51 <- it appears to be in setup.py for anything | 17:43 |
| hrw | clarkb: they do not vendor it. they expect it to be installed | 17:43 |
| hrw | 2020-05-20 17:29:56.064306 | ubuntu-focal-arm64 | In file included from /tmp/pip-install-cxp19dop/confluent-kafka/confluent_kafka/src/confluent_kafka.c:17: | 17:43 |
| hrw | 2020-05-20 17:29:56.064323 | ubuntu-focal-arm64 | /tmp/pip-install-cxp19dop/confluent-kafka/confluent_kafka/src/confluent_kafka.h:65:2: error: #error "confluent-kafka-python requires librdkafka v1.4.0 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html" | 17:43 |
| fungi | once the wheel is created, it can be installed on platforms lacking that library | 17:43 |
| clarkb | https://github.com/confluentinc/confluent-kafka-python/tree/master/confluent_kafka/src ? | 17:43 |
| clarkb | ah that src is what binds against the lib | 17:44 |
| clarkb | where are they including the lib then ? some hacked up whl build process? | 17:44 |
| hrw | on non-x86 we use https://files.pythonhosted.org/packages/28/51/710ab74d6d9435870bed97af21dea04a5828e1e3b2b8ca80fd79bae01b99/confluent-kafka-1.4.1.tar.gz | 17:44 |
| hrw | and then it collapse | 17:44 |
| fungi | manylinux1 wheel builds automatically embed the libs linked from their c extensions | 17:45 |
| fungi | unless those libs are in the set manylinux1 expects to be present on supported systems (libc et cetera) | 17:45 |
| hrw | anyway. time to get to them and improve their .travis.xml | 17:45 |
| fungi | so the upshot is that the current confluent-kafka sdist is not compatible with platforms as old as ubuntu-focal (at least not unless you install some libs from elsewhere to build against), but their manylinux1 wheels of confluent-kafka are compatible with ubuntu-focal and even much older platforms | 17:47 |
| fungi | but requires them to have built them for the architecture you're using | 17:47 |
| clarkb | yup they run https://github.com/confluentinc/confluent-kafka-python/blob/master/tools/bootstrap-librdkafka.sh | 17:48 |
| clarkb | that seems like a really bad way to distribute software | 17:48 |
| * fungi points at the comment box conveniently placed directly above the wastebin | 17:48 | |
| clarkb | hrw: I think one workaroudn for you is to run that script | 17:49 |
| clarkb | then in theory you'll have a librdkafka available for the sdist install | 17:49 |
| hrw | clarkb: and then do it in EACH openstack project which uses confluent_kafka? :D | 17:50 |
| hrw | clarkb: easier would be to sed it out ;P | 17:50 |
| clarkb | hrw: or part of a base arm64 job | 17:50 |
| clarkb | hrw: ya I guess thats another option. What is even using kafka? | 17:50 |
| hrw | clarkb: as 'add it into every node image'? | 17:50 |
| hrw | clarkb: requirements | 17:50 |
| hrw | and would need to check in kolla for others | 17:51 |
| clarkb | I'm pretty sure devstack for example has zero kafka support | 17:51 |
| fungi | oslo.messaging | 17:51 |
| clarkb | fungi: neat, I don't think we test that naywhere :) | 17:51 |
| hrw | monasca uses it also | 17:51 |
| fungi | yeah, looks like kolla installs it for monasca | 17:52 |
| clarkb | https://github.com/Parsely/pykafka may be a useable alternative too? | 17:52 |
| fungi | codesearch indicates oslo.messaging and monasca are the two relevant uses | 17:52 |
| fungi | all other mentions seem to be in support of those | 17:53 |
| clarkb | pykafka doesn't publish wheels so in theory its a bit easier to install. Though also appears to be less actively developed | 17:53 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Enable py36-py38 testing https://review.opendev.org/729734 | 17:54 |
| corvus | mordred: found it. we're currently installing the wrong ppa on xenial. | 17:55 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Vendor the apt repo gpg keys used for Zuul https://review.opendev.org/729401 | 17:56 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Enable py36-py38 testing https://review.opendev.org/729734 | 17:56 |
| *** hashar has joined #opendev | 17:57 | |
| * hrw off | 18:04 | |
| fungi | corvus: see inline comment on 729401 but shouldn't that be using .asc files? | 18:06 |
| *** chandankumar is now known as raukadah | 18:06 | |
| *** ysandeep is now known as ysandeep|away | 18:10 | |
| mordred | corvus: oh - that would do it | 18:26 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Enable py36-py38 testing https://review.opendev.org/729734 | 18:31 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Allow getPlugins to accept a format https://review.opendev.org/238159 | 18:33 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Initialize the logger https://review.opendev.org/277654 | 18:34 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Add / remove reviewer to / from patch set https://review.opendev.org/235884 | 18:34 |
| AJaeger | infra-prod-run-cloud-launcher job failed in hourly pipeline, see https://1262f01504466b9808a6-9172420545ad5f9c0765fd2855e5aee9.ssl.cf1.rackcdn.com/opendev-prod-hourly/opendev.org/opendev/system-config/master/infra-prod-run-cloud-launcher/c1ded9e/ | 18:35 |
| *** hashar is now known as hasharAway | 18:35 | |
| openstackgerrit | Merged opendev/system-config master: Open mirror ssl ports externally https://review.opendev.org/729416 | 18:37 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Enable py36-py38 testing https://review.opendev.org/729734 | 18:38 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Replace testrepository with stestr https://review.opendev.org/729742 | 18:49 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/gerritlib master: Replace testrepository with stestr https://review.opendev.org/729742 | 19:07 |
| zbr | clarkb: https://review.opendev.org/#/c/729734/ ready. | 19:08 |
| clarkb | infra-root https://review.opendev.org/#/c/729659/ which bumps gitea to 1.12.0-rc1 passed testing. I think its worth review now even if we don't want to upgrade until 1.12.0 is released | 19:12 |
| clarkb | infra-root that way we can be ready when the release is made | 19:13 |
| openstackgerrit | Merged opendev/system-config master: Listen on Quay Registry Mirror Ports https://review.opendev.org/729315 | 19:17 |
| clarkb | infra-root re ^ mirror ports I'll plan to restart apache on the mirrors after my bike ride today (which should happen soonish) | 19:19 |
| clarkb | also I think the reason it needs a restart is the listen directives, not the new vhosts | 19:19 |
| clarkb | fungi: ianw ^ I think there was some question about why that was necessary yesterday | 19:20 |
| zbr | clarkb: i managed to build the elastic-recheck image, but i need to find a solution for the ssh keys. | 19:43 |
| zbr | mc | 19:43 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add option to prefer https/ssl in configure-mirrors https://review.opendev.org/729407 | 19:43 |
| clarkb | zbr: the way we've been doing content like configs and secrets is to bind mount all of that into the container. The container itself should be largely stateless | 19:43 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Add Engagement Statistics to docs index https://review.opendev.org/724892 | 19:44 |
| zbr | yep, but I want a solution that makes it easy to run locally for development. | 19:44 |
| zbr | for other secrets I used ENV vars, so they are not included in the image. | 19:45 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: WIP: Create elastic-recheck docker image https://review.opendev.org/729623 | 19:45 |
| fungi | i think what we've done elsewhere is bindmount the keys from a host path into the running container using entries in a docker-compose file? | 19:46 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Save zuul and nodepool logs from gate test jobs https://review.opendev.org/729619 | 19:46 |
| zbr | is too late for me, need to go, but take a look at ^ and comment. | 19:46 |
| *** roman_g has quit IRC | 19:47 | |
| clarkb | fungi: yup exactly | 19:47 |
| mordred | zbr: awesome! | 19:50 |
| clarkb | zbr: mordred fungi the neat thing about a docker image too is it may allow us to bind mount over other query sets for other projects | 19:50 |
| clarkb | kind of an end around e-r not directly supporting configurable configs in the first place (though fixing that properly like still best) | 19:50 |
| *** hasharAway has quit IRC | 19:51 | |
| mordred | zbr: I have an idea ... | 19:51 |
| mordred | clarkb, zbr: I'm going to push up a followup with it - it's based on something I saw from the mapbox folks in their tippecanoe image - but I think it will accomplish the goal zbr has as well as what we're doing in opendev | 19:52 |
| clarkb | I'm going to pop out for that bike ride now. I'll be back in a bit to do the mirror apache restarts | 19:54 |
| openstackgerrit | Monty Taylor proposed opendev/elastic-recheck master: WIP Make a localdev image stage that isn't built in CI https://review.opendev.org/729748 | 19:59 |
| mordred | clarkb, zbr: somethign liek that - since we specify the target in the zuul config, the final stage will just get ignore dthere. BUT - since it's the final stage, if someone just pulls the repo and runs "docker build . " - they'll get that extra stuff copied in | 20:00 |
| *** roman_g has joined #opendev | 20:08 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Save zuul and nodepool logs from gate test jobs https://review.opendev.org/729619 | 20:18 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Vendor the apt repo gpg keys used for Zuul https://review.opendev.org/729401 | 20:18 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Run Zuul as the zuuld user https://review.opendev.org/726958 | 20:18 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add iptables_extra_allowed_groups https://review.opendev.org/726475 | 20:18 |
| corvus | the iptables thing failed in gate, so i went ahead and restacked, hopefully this series will be more stable | 20:20 |
| *** hashar has joined #opendev | 20:37 | |
| *** larainema has quit IRC | 20:38 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add support for multiple jvbs behind meetpad https://review.opendev.org/729008 | 20:41 |
| corvus | rebased ^ | 20:41 |
| corvus | if we change it again, we may just want to do a depends-on there | 20:42 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: add simple test runner https://review.opendev.org/728684 | 20:49 |
| *** sgw has quit IRC | 21:00 | |
| openstackgerrit | Merged opendev/system-config master: Document the need to use sudo in order to access OSC https://review.opendev.org/729196 | 21:03 |
| mordred | corvus: SOOOOO | 21:33 |
| mordred | corvus: wait - let me try something (I was about to tell you about an intractable issue) | 21:33 |
| clarkb | ok back now | 21:34 |
| corvus | #status log added gearman certs to private hostvars for ease of management, and moved gearman client certs and keys to the zuul group (in privatate hostvars) | 21:36 |
| openstackstatus | corvus: finished logging | 21:36 |
| corvus | mordred: ^ fyi | 21:36 |
| clarkb | I'm going to run `sudo ansible mirror -m service -a 'name=apache2 state=restarted'` now to restart all of the mirror apache processes | 21:36 |
| clarkb | * on bridge | 21:38 |
| clarkb | thats done, we should have ssl things now | 21:39 |
| clarkb | https://mirror.ord.rax.opendev.org:4443/pypi/simple/setuptools/ \o/ it works | 21:40 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add option to prefer https/ssl in configure-mirrors https://review.opendev.org/729407 | 21:40 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add option to prefer https/ssl in configure-mirrors https://review.opendev.org/729407 | 21:52 |
| *** mlavalle has quit IRC | 21:53 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Correct the test gearman certs https://review.opendev.org/729771 | 21:56 |
| corvus | okay, that's now the tip of the make-zuul-tests-work stack; we'll see how that goes | 21:56 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add option to prefer https/ssl in configure-mirrors https://review.opendev.org/729407 | 21:59 |
| *** sgw has joined #opendev | 21:59 | |
| *** mlavalle has joined #opendev | 22:07 | |
| ianw | hrw: https://zuul.opendev.org/t/openstack/build/892b08e8842042a8a6d9948e9c778717 seems to have failed due to probably missing libraries i guess, which is good (for me, the node boots) | 22:09 |
| ianw | Unable to correct problems, you have held broken packages.\n is worrying but i couldn't see that. we definitely shouldn't be holding packages | 22:10 |
| ianw | lmn if that's coming up and we'll look into it | 22:11 |
| ianw | clarkb / mordred / (but anyone else too) : can i get your eyes on dropping py2 from dib -> https://review.opendev.org/#/c/728889/ | 22:11 |
| clarkb | ianw: did you see the thread on that in openstack-discuss? | 22:12 |
| ianw | oh, not yet, let me go to mail | 22:12 |
| clarkb | I was kind of hoping for a bit more resolution there before I acked the cahnge on dib | 22:12 |
| clarkb | I think we do want ot drop python2 in dib, it may require a small amount of coordination | 22:12 |
| *** slaweq has quit IRC | 22:13 | |
| ianw | ahh, i see ... yeah. also by keeping up with openstack requirements, it becomes 3.6 only, due to transitivie dependencies that end up in >=3.6 libraries only | 22:14 |
| hrw | ianw: node works fine. thanks a lot for help | 22:17 |
| * hrw -> bed | 22:17 | |
| clarkb | I personally think stickign to openstack requirements is sort of the wrong thing for dib, but more for the networkx stuff than for python version selections | 22:17 |
| clarkb | and I Think we addressed networkx so meh | 22:17 |
| ianw | i'm not sure i disagree, i'd have to dig back through history to see what the thinking was at the time | 22:19 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add option to prefer https/ssl in configure-mirrors https://review.opendev.org/729407 | 22:20 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Remove failovermethod from fedora dnf repo configs https://review.opendev.org/729774 | 22:20 |
| ianw | clarkb: Fri Jul 26 22:27:45 2013 - As a first step to OpenStack alignment | 22:26 |
| ianw | so basically, it's always been like that :) | 22:26 |
| ianw | i've suggested on list we tag the switch to python3 as dib 3, giving us the options of a 2.x branch, if somebody absolutely requires it and wants to maintain it | 22:27 |
| clarkb | ianw: that sounds like a reasonable idea | 22:28 |
| *** hashar has quit IRC | 22:37 | |
| *** tkajinam has joined #opendev | 22:37 | |
| ianw | clarkb: i'm tempted to force merge the focal support to https://review.opendev.org/#/c/725752/7 and then tag from that | 22:46 |
| ianw | i don't see anything else in the queue that is particularly "pre 3" stuff | 22:46 |
| mordred | ianw, clarkb: I agree re: 3.x and a possible 2.x branch if someone wants it | 22:48 |
| mordred | I think we align to openstack requirements largely because _several_ of the openstack projects use it - but maybe we don't have to in our own gate jobs | 22:49 |
| mordred | like - what if we made a tox py35 job (to make sure 3.5 still works for nodepool while it still supports 3.5) | 22:49 |
| clarkb | mordred: well I think the issue is we use openstack libs and they are 3.6 only now? | 22:50 |
| mordred | and then also had an openstack-py36 job or something which _does_ run with openstack constraints - so we make sure we don't land anything that would break openstack installs | 22:50 |
| clarkb | I don't mind dropping 3.5 | 22:50 |
| ianw | mordred: the zipp library has gone to 3.6 only, and gets pulled in by networkx | 22:50 |
| mordred | it should be fine if those have done their python-requires metadata properly | 22:50 |
| mordred | our 3.5 installs will get a slightly older zipp | 22:50 |
| mordred | if they _haven't_ - we can put in version specifiers in our requirements | 22:51 |
| mordred | mostly saying- zuul is still supporting 3.5 - and dib is a big piece of nodepool - so I think just dropping 3.5 from dib puts us in a shaky situation | 22:51 |
| clarkb | thats fair and ya if packaged properly it should work | 22:51 |
| clarkb | unfortunately a lot of that depends on the packages themselves | 22:51 |
| mordred | that said - the discussion about dropping 3.5 from zuul is also out there and it seems like people are reasonably ok with it for v4 | 22:52 |
| clarkb | (we can always use a python3.5 dep in requirements though) | 22:52 |
| mordred | yeah | 22:52 |
| mordred | I think we shoudl at least make a stab - hopefully we won't have to carry 3.5 ourselves for too long | 22:52 |
| clarkb | mordred: note the latest ps on your change (I think ianw may have updated it) does drop 3.5 | 22:52 |
| clarkb | so worth a comment probably | 22:52 |
| mordred | but I worry that we're going to hit a bug in something that nodepool will need a dib patch for and we won't be able to deliver one | 22:52 |
| ianw | mordred: so basically run the openstack victoria jobs, but *also* add our own py35 job? | 22:52 |
| mordred | ianw: yes | 22:52 |
| clarkb | ianw: or maybe even drop the openstack-victoria template and list all teh things we care about too | 22:53 |
| mordred | ianw: I did this for openstacksdk | 22:53 |
| mordred | which is still supporting 3.5 because nodepool | 22:53 |
| mordred | and it's a legit openstack deliverable even :) | 22:53 |
| mordred | so far it's working fine | 22:53 |
| clarkb | mordred: well and sdk is intended to be used far and wide so being very conservative there makes sense | 22:53 |
| mordred | yah - but dib is actually used further than we probably realize | 22:54 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Patch CoreDNS corefile https://review.opendev.org/727868 | 22:54 |
| ianw | ok, i can rework the change to do something like that | 22:54 |
| clarkb | sweet https://review.opendev.org/#/c/729407/6 and its parent pass testing. Thats the first step in updating our jobs to use ssl for mirrors if people want to look at that | 22:54 |
| mordred | cool. (don't get me wrong - I want 3.5 out of here as much as anybody... I just want to make sure we don't hurt ourselves in the process) | 22:54 |
| clarkb | basically on newer debuntu and everywhere else use the https urls instead of http for distro and pypi mirrors | 22:55 |
| ianw | it looks to me bifrost is installing dib from git in the middle of playbooks | 22:55 |
| ianw | https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-ironic-install/tasks/install.yml#L63 | 22:57 |
| ianw | so essentially an override-checkout would seem to work for them | 22:58 |
| ianw | maybe johnsom could tell me more about octavia before i have to dig :) | 22:59 |
| johnsom | o/ | 22:59 |
| johnsom | ianw What would you like to know? | 23:00 |
| ianw | johnsom: discussing tagging dib basically now as the last 2.x release and upping to v3 with py2 dropped | 23:01 |
| johnsom | It slices, it dices, it even julienne fries... | 23:01 |
| mordred | (with a stable branch for the v2 series potentially for bugfixes as needed) | 23:01 |
| ianw | leaving us the option of a v2 branch, but only if we find we need it | 23:01 |
| ianw | heh, dib, stable, that made me giggle :) | 23:02 |
| johnsom | Yeah, I have been a bit of an advocate for DIB stable branches myself. This is perfectly workable for us as long as stable policy/requirements folks don't get too worked up at us for pinning the requirements in our stable branches. | 23:03 |
| johnsom | So even if we don't go full branching, a v2 seems very workable | 23:03 |
| ianw | https://review.opendev.org/#/c/729508/ also has to make it in to drop triple-o jobs | 23:07 |
| *** DSpider has quit IRC | 23:07 | |
| corvus | mordred, clarkb: i don't think zuul is in a rush to drop 3.5. there's nothing we need in a later version. so i think it would be a really friendly thing if our dependencies like dib and sdk continued supporting it as long as feasible. | 23:11 |
| mordred | corvus: yeah - I think it should be easy enough for dib to. I know it's easy enough for sdk to | 23:12 |
| corvus | oops, i stacked the gearman change on the jvb change, but that's fine for now :) | 23:13 |
| clarkb | I think the jvb change can land too as it won't affect the existing server until we add a new jvb server | 23:15 |
| corvus | clarkb, mordred: yeah, i think we can/should land everything up through https://review.opendev.org/729771 | 23:15 |
| corvus | we're not at a working zuul yet, but i'm pretty sure the logs are indicating incremental improvement and we're not breaking anything | 23:15 |
| clarkb | k I'll start reviewing the bottom of that stack now | 23:16 |
| corvus | mordred, clarkb: the end of that stack is indicating that our sql connection is lacking a sql database, which, in fact is the case. | 23:17 |
| corvus | so what should we do for a sql in the gate? | 23:17 |
| corvus | we could do like gerrit and move it to a mysql container on the scheduler host | 23:17 |
| clarkb | ya I think for testing that pattern is a good one | 23:18 |
| clarkb | and even in production we've started doing that (gitea and ehterpad) | 23:19 |
| corvus | and prod too? | 23:19 |
| corvus | oh i thought we did that for gerrit; my mistake | 23:19 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 23:19 |
| mordred | corvus: I think we haven't moved gerrit yet because it'll go away with notedb | 23:20 |
| mordred | corvus: that said - in the gate we run gerrit with h2 | 23:20 |
| corvus | ack | 23:20 |
| corvus | we could try sqlite with zuul, but it's not a supported option | 23:20 |
| corvus | but if that works with 5m of effort, maybe we should start with that :) | 23:21 |
| clarkb | corvus: couple of things on https://review.opendev.org/#/c/729401/7 the extra whitespace thing may be worth another patchset? I'm not sure if apt is happy with that | 23:21 |
| clarkb | corvus: feel free to approve if we think its fine as is | 23:21 |
| corvus | clarkb: done; i'll fix in followup | 23:22 |
| ianw | does anyone have context on if we deliberately don't have a openstack-python38-jobs template? | 23:23 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Fix whitespace in zuul-executor PPAs https://review.opendev.org/729785 | 23:25 |
| clarkb | ianw: openstack hasnt gotten that far yet I think | 23:26 |
| clarkb | focal transition talk has just started | 23:26 |
| clarkb | and before that only a few things were testing eith 3.8 but it was opy in and not global iirc | 23:27 |
| ianw | ok, it's used in the victoria job template, but there's no openstack-python38-jobs equivalent | 23:27 |
| clarkb | ianw: my guess is it was just missed | 23:29 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use sqlite with Zuul in the gate https://review.opendev.org/729786 | 23:33 |
| corvus | mordred, clarkb: ^ spaghetti thrown, we'll see if it sticks. | 23:33 |
| ianw | similar to using 3.5 on the bottom end, since zuul uses 3.8 on the top end i guess dib should at least run tox under that too | 23:33 |
| fungi | is zuul usable with 3.8 yet? i thought current ansible versions still didn't work with it | 23:34 |
| clarkb | fungi: zuul itself is, but you have to use different python for ansible or only newer ansible versions | 23:35 |
| clarkb | ansible is installed into virtualenvs so is in theory able to be run as different python version than zuul | 23:35 |
| clarkb | all of our installation tooling assumes they are the same though iirc | 23:35 |
| fungi | heh, that's an interesting definition of "usable" but sure, i get it | 23:35 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Drop support for python2 https://review.opendev.org/728889 | 23:35 |
| fungi | in theory we could make future ansible venvs with 3.8 but use 3.7 or older to make other ansible vencs | 23:36 |
| fungi | venvs | 23:36 |
| clarkb | yup | 23:39 |
| ianw | fungi: not sure if you saw https://review.opendev.org/#/c/728743/ but you might like that, to autogen the ssl check list | 23:39 |
| fungi | i saw, just been trying to keep afloat all day. can probably take a look now, thanks for the reminder! | 23:42 |
| *** mlavalle has quit IRC | 23:43 | |
| fungi | ianw: longer term, bridge seems like a perfectly reasonable place to run that in production too... it's not like it needs special privileges anyway | 23:45 |
| ianw | did i see something about containerising cacti anyway? | 23:45 |
| fungi | though it does necessitate installation of an additional distro package | 23:45 |
| fungi | if there was talk of containerizing cacti, i missed it, but i've been pretty distracted so maybe... it will happen eventually for sure | 23:46 |
| ianw | yeah just mailx i think | 23:46 |
| fungi | well, and the certcheck utility | 23:46 |
| *** tosky has quit IRC | 23:48 | |
| fungi | ianw: do you have any idea how we would go about reusing the same cert on multiple tcp sockets for the same dns name, given your proposed extensions to the letsencrypt_certs data structure? | 23:51 |
| fungi | separate entry per socket? | 23:51 |
| ianw | fungi: i figured you don't need to check it's freshness on multiple ports? | 23:56 |
| fungi | ahh, maybe. so we consider the port number listed in the letsencrypt_certs entries to be their canary ports | 23:57 |
| fungi | i can't remember if we're still mixing static certs and le on the same hosts, but if we are we could see this as a reason to just not | 23:58 |
| ianw | yeah, that was my thinking; however, we could look at *all* entries in the list, and then run a uniq on the final output maybe? | 23:58 |
| ianw | like the gitea case | 23:58 |
| fungi | and to be clear, our le automation doesn't generate multiple certs for a single host, so it's safe to only check one socket+name? | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!