| *** tosky has quit IRC | 00:06 | |
| ianw | it looks like codesearch testing is failing repeatedly | 01:10 |
|---|---|---|
| *** mlavalle has quit IRC | 01:10 | |
| ianw | Error: /Stage[main]/Hound/Exec[build_hound]: Failed to call refresh: 'go install github.com/hound-search/hound/cmds/...' returned 1 instead of one of [0]", | 01:12 |
| ianw | i can't actually seem to see what it's saying | 01:12 |
| *** iurygregory has quit IRC | 01:32 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: Use --password-stdin for upload-container-image https://review.opendev.org/762939 | 01:46 |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: Use --password-stdin for upload-container-image https://review.opendev.org/762939 | 01:47 |
| ianw | i can probably get this containerised quicker than sorting it out in puppet ... | 02:00 |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: Use --password-stdin for upload-container-image https://review.opendev.org/762939 | 02:00 |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: Use --password-stdin for upload-container-image https://review.opendev.org/762939 | 02:04 |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: Use --password-stdin for upload-container-image https://review.opendev.org/762939 | 02:09 |
| *** stevebaker has quit IRC | 02:39 | |
| *** stevebaker has joined #opendev | 03:47 | |
| *** auristor has quit IRC | 03:49 | |
| *** donnyd has quit IRC | 04:07 | |
| *** donnyd has joined #opendev | 04:08 | |
| *** stevebaker has quit IRC | 04:11 | |
| *** cmurphy has quit IRC | 04:21 | |
| *** cmurphy has joined #opendev | 04:22 | |
| *** johnsom has quit IRC | 04:46 | |
| *** johnsom has joined #opendev | 04:49 | |
| *** raukadah is now known as chandankumar | 05:01 | |
| *** ykarel has joined #opendev | 05:13 | |
| *** johnsom has quit IRC | 05:18 | |
| *** johnsom has joined #opendev | 05:19 | |
| *** aprice has quit IRC | 05:27 | |
| *** aprice has joined #opendev | 05:27 | |
| *** clayg has quit IRC | 05:30 | |
| *** clayg has joined #opendev | 05:32 | |
| *** ykarel_ has joined #opendev | 06:03 | |
| *** ykarel_ has quit IRC | 06:05 | |
| *** ykarel_ has joined #opendev | 06:05 | |
| *** ykarel has quit IRC | 06:06 | |
| *** marios has joined #opendev | 06:09 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [WIP] codesearch container https://review.opendev.org/762960 | 06:21 |
| ianw | clarkb/fungi: ^ i know we've talked about deprecating this for gitea, but i think the agreement is it's not quite there yet. i'll work tomorrow to plug the container into system-config, but that works locally for me | 06:22 |
| *** johnsom has quit IRC | 06:27 | |
| *** johnsom has joined #opendev | 06:27 | |
| *** marios is now known as marios|ruck | 06:38 | |
| *** ykarel_ has quit IRC | 06:43 | |
| *** auristor has joined #opendev | 06:44 | |
| *** ykarel has joined #opendev | 07:11 | |
| *** sboyron has joined #opendev | 07:22 | |
| *** rm_work has quit IRC | 07:26 | |
| *** rm_work has joined #opendev | 07:28 | |
| openstackgerrit | Merged openstack/project-config master: Build newer gerrit images when jeepyb updates https://review.opendev.org/762196 | 07:36 |
| *** eolivare has joined #opendev | 07:42 | |
| *** ralonsoh has joined #opendev | 07:43 | |
| *** DSpider has joined #opendev | 07:52 | |
| *** slaweq has joined #opendev | 07:58 | |
| *** rpittau|afk is now known as rpittau | 08:05 | |
| *** ykarel_ has joined #opendev | 08:09 | |
| *** hashar has joined #opendev | 08:11 | |
| *** ykarel has quit IRC | 08:11 | |
| *** guillaumec has quit IRC | 08:14 | |
| *** guillaumec has joined #opendev | 08:15 | |
| *** tobiash has quit IRC | 08:15 | |
| *** tobiash has joined #opendev | 08:16 | |
| *** andrewbonney has joined #opendev | 08:17 | |
| *** guillaumec has quit IRC | 08:23 | |
| *** guillaumec has joined #opendev | 08:24 | |
| *** ykarel_ is now known as ykarel | 08:51 | |
| *** mgoddard has joined #opendev | 08:52 | |
| *** iurygregory has joined #opendev | 08:52 | |
| *** tosky has joined #opendev | 08:56 | |
| *** dtantsur|afk is now known as dtantsur | 09:04 | |
| openstackgerrit | Merged opendev/irc-meetings master: Update TC to meet weekly https://review.opendev.org/762845 | 09:07 |
| *** ykarel has quit IRC | 09:12 | |
| *** ykarel has joined #opendev | 09:12 | |
| openstackgerrit | Merged opendev/irc-meetings master: Update Airship meeting time and week https://review.opendev.org/762922 | 09:13 |
| openstackgerrit | Merged opendev/irc-meetings master: move heat meeting time to Wednesday 1400 UTC https://review.opendev.org/760286 | 09:13 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/diskimage-builder master: Fix python-stow-versions https://review.opendev.org/751610 | 09:23 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/diskimage-builder master: Run autoremove on post-install step https://review.opendev.org/751614 | 09:23 |
| *** sean-k-mooney has quit IRC | 09:27 | |
| *** sean-k-mooney has joined #opendev | 09:27 | |
| *** mgoddard has quit IRC | 09:29 | |
| *** odyssey4me has quit IRC | 09:50 | |
| *** odyssey4me has joined #opendev | 09:50 | |
| *** whoami-rajat__ has joined #opendev | 10:12 | |
| *** mgoddard has joined #opendev | 10:19 | |
| *** kwazar is now known as quasar_ | 10:55 | |
| *** quasar_ is now known as quasar | 10:55 | |
| *** quasar is now known as quasar` | 10:56 | |
| *** brinzhang0 has joined #opendev | 11:50 | |
| *** brinzhang_ has quit IRC | 11:53 | |
| *** brinzhang_ has joined #opendev | 11:55 | |
| *** slaweq has quit IRC | 11:56 | |
| *** slaweq has joined #opendev | 11:57 | |
| *** brinzhang0 has quit IRC | 11:58 | |
| *** mgoddard has quit IRC | 12:01 | |
| *** mgoddard has joined #opendev | 12:15 | |
| *** eolivare_ has joined #opendev | 12:23 | |
| *** lpetrut has joined #opendev | 12:24 | |
| *** eolivare has quit IRC | 12:26 | |
| *** roman_g has joined #opendev | 12:47 | |
| yoctozepto | infra-core http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018817.html - I want your thoughts on this | 13:30 |
| yoctozepto | and morning! | 13:31 |
| frickler | yoctozepto: use either "infra-root" or "config-core" to trigger some attention ;) | 13:36 |
| frickler | we discussed that topic two weeks ago http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-11-03-19.01.log.html#l-22 | 13:36 |
| frickler | but no feedback since then, I was about to answer docker that we are not interested in their proposal | 13:37 |
| frickler | if you want to take over on behalf of the kolla namespace, I think you could either fill out that form for yourself or I can answer docker and then hand over to you | 13:38 |
| frickler | this is the reply I received from docker https://etherpad.opendev.org/p/EAfLWowNY8N96APS1XXM | 13:38 |
| yoctozepto | frickler: hah, thanks; I think someone has already told me that but I keep forgetting :D | 13:51 |
| yoctozepto | frickler: thanks, interesting; though the mail looks very similar to https://www.docker.com/blog/expanded-support-for-open-source-software-projects/ | 13:52 |
| yoctozepto | frickler: so I am assuming they gave up on the weird parts about 'requiring docker' to be used | 13:52 |
| yoctozepto | frickler: we can generally agree to all the terms except the last one (as we are looking towards using podman heh) | 13:52 |
| *** mlavalle has joined #opendev | 13:56 | |
| frickler | well not sure whether that is considered negotiable, I'd tend to assume that it is rather not | 13:58 |
| *** roman_g has quit IRC | 13:59 | |
| openstackgerrit | Slawek Kaplonski proposed zuul/zuul-jobs master: [multi-node-bridge] Add script to configure connectivity https://review.opendev.org/762650 | 14:00 |
| frickler | yoctozepto: would you have time to join the opendev meeting tonight? not sure we'd have much time to discuss docker with all the late gerrit upgrade prep going on, but might be a chance to get feedback from most people involved. though likely I won't attend myself, either | 14:00 |
| yoctozepto | frickler: well, that's a very encouraging invitation, would not you say? :D | 14:02 |
| yoctozepto | anyhow, just tried filling out the form template for docker exemption and, oh dear, they are asking many too many questions; seems they are focused on company-owned open source projects | 14:04 |
| yoctozepto | hence all the bs about marketing things | 14:05 |
| yoctozepto | oh well | 14:05 |
| frickler | yoctozepto: I half guessed most of the answers, not sure how relevant those are anyway | 14:05 |
| yoctozepto | not sure either | 14:06 |
| yoctozepto | I wonder whether foundation should step in | 14:06 |
| yoctozepto | but then again | 14:06 |
| yoctozepto | most projects are not docker-focused | 14:07 |
| yoctozepto | and we are looking at the podman alternative too | 14:07 |
| frickler | yoctozepto: IIRC clarkb wanted to ask them about their view on the docker offer, but I didn't see any response to that, either | 14:07 |
| yoctozepto | yeah, saw | 14:07 |
| frickler | yoctozepto: does podman have any image store of their own? | 14:08 |
| yoctozepto | frickler: none that I know of, just sucking docker's blood by default | 14:08 |
| yoctozepto | let's say I am not looking for a final solution at the moment, just trying to enumerate our options | 14:09 |
| yoctozepto | any idea what tripleo decided on doing finally? | 14:09 |
| frickler | no idea. setting up our own persistent image store might be an option, but that would likely only work with enough contributions from the consuming projects | 14:11 |
| mgoddard | yoctozepto: tripleo switched to a content provider job model | 14:13 |
| mgoddard | yoctozepto: and publishes quay.io | 14:14 |
| mgoddard | yoctozepto: quay.io being RH owned | 14:14 |
| mgoddard | they do seem to have free public image repos, with no tiering | 14:14 |
| mgoddard | sorry, no rate limiting | 14:14 |
| mgoddard | https://quay.io/plans/ | 14:14 |
| mgoddard | whether that will stay forever I don't know | 14:15 |
| yoctozepto | mgoddard: but 19:21:24 <clarkb> yes I seem to recall tripleo ruled out quay as a quic fix because they have rate limits too | 14:15 |
| yoctozepto | odd | 14:15 |
| yoctozepto | well, if it does not | 14:16 |
| yoctozepto | then we can migrate there too | 14:16 |
| yoctozepto | no regrets, mr dockerhub | 14:16 |
| mgoddard | https://hackmd.io/ermQSlQ-Q-mDtZkNN2oihQ | 14:18 |
| mgoddard | different rate limit: https://bugs.launchpad.net/tripleo/+bug/1889122 | 14:19 |
| openstack | Launchpad bug 1889122 in tripleo "mirror timeouts in upstream causing undercloud and standalone failures" [Critical,In progress] - Assigned to Ronelle Landy (rlandy) | 14:19 |
| mgoddard | http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016116.html | 14:21 |
| *** d34dh0r53 has quit IRC | 14:22 | |
| openstackgerrit | Merged openstack/project-config master: Clean up pti-python-tarball python-branch-tarball https://review.opendev.org/762699 | 14:25 |
| *** brinzhang_ has quit IRC | 14:26 | |
| mgoddard | yoctozepto: are you making notes somewhere? | 14:27 |
| frickler | https://docs.quay.io/issues/429.html still doesn't mention any concrete limit. it might get mitigated by not using our mirrors for that, increasing total load on them in turn. someone inside redhat might want to talk to the proper people to get some reasonable solution there | 14:27 |
| frickler | like not limiting our mirrors | 14:27 |
| yoctozepto | frickler: yeah, that sounds feasible | 14:28 |
| yoctozepto | mgoddard: I can start | 14:28 |
| yoctozepto | https://etherpad.opendev.org/p/docker-pull-limits | 14:28 |
| mgoddard | this sort of links up with switching to more of a build pipeline, in which case we would use the opendev intermediate registry | 14:29 |
| *** Vivek has joined #opendev | 14:30 | |
| *** d34dh0r53 has joined #opendev | 14:30 | |
| yoctozepto | mgoddard: yes, changing the approach is fine for other usage but it will not help with kolla-ansible jobs that do not wish to build | 14:33 |
| yoctozepto | and do note these are the only ones affected | 14:33 |
| yoctozepto | tripleo seems to have it a bit more complicated though | 14:33 |
| mgoddard | yoctozepto: build jobs still need to pull base images, I have seen them hit | 14:33 |
| yoctozepto | mgoddard: but base distro images should be already exempted | 14:34 |
| mgoddard | seems not | 14:34 |
| yoctozepto | duh | 14:34 |
| yoctozepto | well then we can do nada about it | 14:34 |
| mgoddard | I saw it with ubuntu:20.04 | 14:34 |
| yoctozepto | eh, life | 14:36 |
| fungi | yeah, canonical would presumably have to ask dockerhub to grant their namespace an exemption | 14:36 |
| yoctozepto | though it could be related to the fact that we pull all the others | 14:36 |
| yoctozepto | frickler, fungi: I am not really getting that mirror/proxy thing that has been mentioned | 14:37 |
| yoctozepto | I thought dockerhub/quay.io gets contacted directly no matter what | 14:37 |
| yoctozepto | or is it related to how tripleo did it (i.e. they used infra mirroring for quay) | 14:37 |
| yoctozepto | mgoddard: we are not using a proxy/mirror either, are we? | 14:38 |
| mgoddard | yoctozepto: we use the registry mirror | 14:38 |
| mgoddard | which hurts in some ways and helps in others | 14:39 |
| yoctozepto | ah, right | 14:39 |
| yoctozepto | I must get my mindset fixed that we mean pulls and not pushes | 14:40 |
| yoctozepto | well, it depends on how cdn-like dockerhub is | 14:40 |
| yoctozepto | the closer to a real cdn, the better to skip the mirror | 14:41 |
| yoctozepto | though the internet pipe is unhappy | 14:41 |
| yoctozepto | decisions, decisions... | 14:41 |
| mgoddard | well I think there are different mirrors for different clouds | 14:42 |
| yoctozepto | yeah, but then it's a reason why the failures were random (different clouds) | 14:43 |
| yoctozepto | rather than any consolation | 14:43 |
| yoctozepto | well | 14:48 |
| yoctozepto | so the quay.io issue might also be amended by making the mirrors artificially rate-limit themselves | 14:48 |
| yoctozepto | since they know how often they can contact upstream | 14:49 |
| yoctozepto | I think nginx can do that | 14:49 |
| *** lpetrut has quit IRC | 15:16 | |
| frickler | infra-root: since we don't seem to be able to get any fix or feedback regarding limestone, I'm going to propose disabling the cloud now, despite increasing our resource issue, but maybe the reduction in retries needed can make up for it | 15:44 |
| frickler | see also https://52409a2182703e644968-eb284aa94e3a202614740e20d7b20329.ssl.cf5.rackcdn.com/762650/5/check/zuul-jobs-test-multinode-roles-opensuse-15/ffc91be/job-output.txt for yet another failure mode | 15:44 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/project-config master: Disable limestone provider due to IPv4-less nodes https://review.opendev.org/763038 | 15:47 |
| clarkb | the way our docker mirrors work is they always talk to docker hub for the manifests because manifest fetches are authenticated even when pulling anonymously. This makes apache not cache those responses even though headers are set allowing that | 15:48 |
| clarkb | then from the manifest it pulls the layer blobs from object storage without auth and those are cached | 15:49 |
| clarkb | previously docker rate limited layer pulls becausethey are the actual data transfer | 15:49 |
| clarkb | we did the right thing in that setup | 15:49 |
| clarkb | noe they rate limit manifests | 15:49 |
| clarkb | people found the old limits too confusing or something so they changed it | 15:50 |
| *** ykarel has quit IRC | 15:56 | |
| clarkb | mgoddard: yoctozepto is there a tldr on the quay issue? fwiw I reached out to some people at red hat/ibm who checked and there aren't supposed to be any existing quay limits aiui. But understanding what tripleo hit would be good for understanding this general problem space better | 16:00 |
| fungi | early in the discussion about dockerhub rate limits, there was speculation that quay *might* impose similar limits in the future if they get overrun | 16:08 |
| fungi | which prompted the idea that relying on one global registry is a risk on its own | 16:08 |
| clarkb | yup | 16:09 |
| *** iurygregory has quit IRC | 16:29 | |
| *** hamalq has quit IRC | 16:56 | |
| *** hamalq has joined #opendev | 16:57 | |
| *** marios|ruck is now known as marios|out | 17:01 | |
| *** Vivek has quit IRC | 17:01 | |
| yoctozepto | clarkb: only https://docs.quay.io/issues/429.html | 17:09 |
| fungi | i guess we should be on the lookout for http 429 errors there | 17:11 |
| frickler | fungi: clarkb: wdyt regarding disabling limestone? https://review.opendev.org/763038 | 17:18 |
| *** mlavalle has quit IRC | 17:18 | |
| clarkb | I think wecan do that uf we dont want ti try the ipv4 connectivity test in the basejob when we check dns resolution and similar | 17:19 |
| *** iurygregory has joined #opendev | 17:19 | |
| *** pabelanger has joined #opendev | 17:21 | |
| pabelanger | clarkb: fungi: do you happen to know the spec on insecure-registry? | 17:22 |
| pabelanger | I cannot see it in cacti | 17:22 |
| *** marios|out has quit IRC | 17:25 | |
| *** rpittau is now known as rpittau|afk | 17:27 | |
| clarkb | pabelanger: 4vcpu and 4GB of ram | 17:27 |
| clarkb | most of the memory is used by caches | 17:27 |
| *** eolivare_ has quit IRC | 17:27 | |
| clarkb | storage is all remote in swift | 17:27 |
| pabelanger | clarkb: is it zuul-registry that needs the RAM or something else? | 17:29 |
| pabelanger | I am curious if 1vcpu / 1gb would fligh | 17:29 |
| pabelanger | fly* | 17:29 |
| clarkb | its actively using about 400MB of memory but then also using the rest for caches | 17:29 |
| clarkb | I think having a bit more memroy for caches is worthwhile since it should speed up data transfers I expect | 17:30 |
| pabelanger | k | 17:30 |
| pabelanger | thanks! | 17:30 |
| fungi | virt:1575340, res:95808, shared:3916 | 17:30 |
| fungi | per top | 17:30 |
| fungi | for the zuul-registry process | 17:30 |
| pabelanger | k, so it will cache layer in RAM when possible, then hit swift? | 17:30 |
| *** mlavalle has joined #opendev | 17:31 | |
| fungi | looks that way | 17:31 |
| pabelanger | TIL, thanks | 17:31 |
| fungi | dockred is using more vmem than the zuul-registry process | 17:32 |
| fungi | er, dockerd | 17:32 |
| fungi | but i guess it's the parent | 17:32 |
| pabelanger | ah, right. it runs as a container | 17:33 |
| clarkb | frickler: validate-host is the role I'm thinking of: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/validate-host | 17:33 |
| fungi | mmm, actually containerd is the parent (indirectly, via containerd-shim) according to the process list | 17:34 |
| pabelanger | still, that is helpful. thanks | 17:34 |
| fungi | dockerd may just be an inefficient memory hog | 17:34 |
| clarkb | frickler: maybe we can add a flag to make https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/validate-host/library/zuul_debug_info.py#L66-L90 require ipv4? | 17:34 |
| fungi | could insert a "passed = False" around line 85 | 17:36 |
| clarkb | fungi: ya but I don't think we want to require working ipv4 in all cases since this is in zuul-jobs | 17:37 |
| clarkb | but if we can make that an option it should force those jobs to retry up to 3 times if they don't have working ipv4 which should work for our purposes | 17:37 |
| fungi | right, a conditional there i suppose | 17:38 |
| fungi | though best would be to refactor that routine to set two results and then use conditionals on rolevars to decide what's required | 17:39 |
| frickler | not sure which jobs all would need that check | 17:39 |
| clarkb | frickler: for us that runs on every job | 17:39 |
| clarkb | which is possibly overkill but maybe that is ok | 17:40 |
| clarkb | (that == the validate-host role) | 17:40 |
| frickler | also not sure how big the chances are for hitting three limestone nodes in a row | 17:40 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: DNM: generate ansi log for benchmark purpose https://review.opendev.org/763054 | 17:40 |
| clarkb | frickler: well also most limestone nodes seem to work | 17:40 |
| frickler | but feel free to try that if you want, /me doesn't have time for it currently | 17:41 |
| clarkb | I think we can disable the cloud now since it is a quick and effective solution | 17:41 |
| clarkb | then if we find time we can update the validate-host role and reenable limestone | 17:41 |
| frickler | mnaser: did you see the vexxhost ci failures due to a missing master branch? | 17:41 |
| clarkb | (I hvae +2'd the limestone disable change) | 17:41 |
| mnaser | frickler: yes, spent a whole bunch of time trying to fix them to no avail, override-branch is set to `main` and i have dug through the zuul code a bunch, still trying to figure out the root cause | 17:42 |
| mnaser | :\ | 17:42 |
| clarkb | pabelanger: ^ may have insight since they use non master branches too | 17:42 |
| pabelanger | yah, we've migrated some projects in ansible to main | 17:43 |
| pabelanger | and deleted master | 17:43 |
| pabelanger | seems to work | 17:43 |
| clarkb | pabelanger: are any of them config projects? | 17:43 |
| clarkb | I seem to recall that we thought that may be related to mnaser's situation | 17:44 |
| pabelanger | clarkb: no, just untrusted right now | 17:44 |
| pabelanger | clarkb: do you mind adding me to https://review.opendev.org/#/admin/groups/2067,members and https://review.opendev.org/#/admin/groups/2068,members https://review.opendev.org/685791/ was the change to create the groups | 17:45 |
| fungi | i can do it if clarkb is busy | 17:45 |
| clarkb | fungi: pleaes do, I don't have that ssh key loaded right now | 17:45 |
| *** roman_g has joined #opendev | 17:45 | |
| pabelanger | fungi: wfm, thank you | 17:46 |
| fungi | #status log added pabelanger an initial member of new ansible-role-zuul-registry-core and ansible-role-zuul-registry-release groups in gerrit | 17:47 |
| openstackstatus | fungi: finished logging | 17:47 |
| pabelanger | I have likely asked before, for non-openstack project is the plan to move them out of that tenant into say the opendev tenant? | 17:54 |
| clarkb | we've already been putting new things in their own tenants or opendev/ so yes | 17:55 |
| clarkb | but I have no idea when that will get prioritized just looking at the list of todos | 17:55 |
| pabelanger | how is it decided to get a new tenant or go into opendev? | 17:55 |
| clarkb | I don't think there has been any hard rule on it. But more and more we're prefering a separate tenant | 17:57 |
| pabelanger | k | 17:57 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: validate-host: Options to require v4 and v6 routes https://review.opendev.org/763065 | 18:01 |
| *** mgoddard has quit IRC | 18:02 | |
| *** andrewbonney has quit IRC | 18:05 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: validate-host: Options to require v4 and v6 routes https://review.opendev.org/763065 | 18:20 |
| *** cloudnull is now known as kecarter | 18:36 | |
| *** kecarter is now known as cloudnull | 18:36 | |
| openstackgerrit | Paul Belanger proposed openstack/project-config master: Add zuul/zuul-regsitry to openstack tenant https://review.opendev.org/763071 | 18:40 |
| pabelanger | clarkb: fungi: corvus: do you mind commenting on ^, zuul-registry isn't in the openstack tenant, which is something I'd like to be able to reference from ansible-role-zuul-registry role | 18:41 |
| corvus | wfm | 18:42 |
| clarkb | the key is not loading configs from it which you've got so +3 | 18:43 |
| pabelanger | tyty | 18:43 |
| *** d34dh0r53 has quit IRC | 18:43 | |
| *** d34dh0r53 has joined #opendev | 18:46 | |
| fungi | yep, seems fine to me | 18:49 |
| fungi | pabelanger: wrt the openstack tenant, the main reason unofficial projects might still get added there is if they plan to make use of job configuration from openstack projects, that quickly gets complicated to do cross-tenant with things like devstack/tempest | 18:51 |
| fungi | it's doable, but you're talking probably 20+ required projects to pick up services, plugins, et cetera | 18:52 |
| fungi | and as soon as the official jobs add another project into the mix you're broken until you add that to your tenant | 18:53 |
| pabelanger | fungi: makes sense | 18:53 |
| fungi | unfortunately there's no magic "i want to run this job, figure out what repos i need and include them in my tenant" functionality | 18:54 |
| fungi | so it can be a fair about of trial and error to get working, and then a bit of a treadmill to keep it working over time | 18:55 |
| fungi | s/about/amount/ | 18:55 |
| openstackgerrit | Merged openstack/project-config master: Add zuul/zuul-regsitry to openstack tenant https://review.opendev.org/763071 | 19:02 |
| *** mgoddard has joined #opendev | 19:21 | |
| *** stevebaker has joined #opendev | 19:22 | |
| clarkb | fungi: I just approved https://review.opendev.org/#/c/762712/. Maybe you can take a look at https://review.opendev.org/#/c/757153/ and https://review.opendev.org/#/c/762191/ and approve those if they look good to you? | 19:39 |
| clarkb | thats all pre upgrade prep (tehre are other changes under that topic too if you have time) | 19:39 |
| fungi | yup, was on my slate for after the meeting | 19:39 |
| clarkb | thanks! | 19:39 |
| clarkb | fungi: I'll be finding lunch next but will monitor irc and be around to help with those changes as they land if necessary (so feel free to approve if they look good) | 19:49 |
| fungi | sounds good, reviewing them in a few | 19:50 |
| *** roman_g has quit IRC | 20:16 | |
| *** dtantsur is now known as dtantsur|afk | 20:17 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [WIP] codesearch container https://review.opendev.org/762960 | 20:17 |
| *** roman_g has joined #opendev | 20:18 | |
| *** smcginnis has quit IRC | 20:24 | |
| *** mgoddard has quit IRC | 20:31 | |
| openstackgerrit | Merged opendev/system-config master: Add example maintenance message for Gerrit server https://review.opendev.org/762712 | 20:36 |
| clarkb | theres one | 20:41 |
| fungi | most have been approved, i'm waiting for the jeepyb change to merge before approving its dependency though | 20:44 |
| clarkb | thanks! | 20:44 |
| fungi | er, reverse-dependency i guess | 20:46 |
| openstackgerrit | Merged zuul/zuul-jobs master: Use --password-stdin for upload-container-image https://review.opendev.org/762939 | 20:53 |
| *** roman_g has quit IRC | 20:56 | |
| *** roman_g has joined #opendev | 20:57 | |
| *** roman_g has quit IRC | 20:57 | |
| *** roman_g has joined #opendev | 20:58 | |
| *** roman_g has quit IRC | 20:58 | |
| *** roman_g has joined #opendev | 20:58 | |
| *** roman_g has quit IRC | 20:59 | |
| clarkb | fungi: for the maintenance notice, should we just go ahead and hand edit that file on the server now? then we can do the comment removals in apache config day of? | 20:59 |
| *** roman_g has joined #opendev | 20:59 | |
| *** roman_g has quit IRC | 20:59 | |
| clarkb | etherpad is coming together I think | 21:00 |
| *** roman_g has joined #opendev | 21:00 | |
| *** roman_g has quit IRC | 21:00 | |
| *** roman_g has joined #opendev | 21:01 | |
| *** roman_g has quit IRC | 21:01 | |
| *** roman_g has joined #opendev | 21:01 | |
| *** roman_g has quit IRC | 21:02 | |
| *** roman_g has joined #opendev | 21:02 | |
| *** roman_g has quit IRC | 21:02 | |
| *** roman_g has joined #opendev | 21:03 | |
| *** roman_g has quit IRC | 21:03 | |
| *** roman_g has joined #opendev | 21:04 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [WIP] codesearch container https://review.opendev.org/762960 | 21:04 |
| *** roman_g has quit IRC | 21:04 | |
| *** roman_g has joined #opendev | 21:05 | |
| *** roman_g has quit IRC | 21:05 | |
| *** hashar has quit IRC | 21:06 | |
| *** hashar_ has joined #opendev | 21:06 | |
| clarkb | fungi: looks like the apache config applied as expected on prod | 21:09 |
| clarkb | I'm going to put review-dev in the emergency file once these other changes have applied as they should be the last set of safe things for it' | 21:09 |
| fungi | yeah, we can draft the maintenance message ahead of time. happy to do that in a bit | 21:11 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [WIP] codesearch container https://review.opendev.org/762960 | 21:13 |
| openstackgerrit | Merged opendev/jeepyb master: Make local git dir creation optional https://review.opendev.org/758597 | 21:15 |
| *** hashar_ is now known as hashar | 21:20 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [WIP] codesearch container https://review.opendev.org/762960 | 21:20 |
| openstackgerrit | Merged opendev/system-config master: Disable change.move and enableSignedPush in gerrit https://review.opendev.org/757153 | 21:22 |
| openstackgerrit | Merged opendev/system-config master: Set gerrit.serverId in gerrit.config https://review.opendev.org/762191 | 21:22 |
| pabelanger | greetings, is build-python-release jobs broken? https://zuul.opendev.org/t/zuul/build/225b52058c8e454ea4152f9008a0b5a3 | 21:26 |
| pabelanger | python setuptools is missing is seems | 21:26 |
| clarkb | hrm no that job is in zuul-jobs and hasn't changed in a long time | 21:27 |
| clarkb | I thought it may be related to changes fungi had made to release jobs but those must've been openstack psecific | 21:27 |
| pabelanger | okay. Trying to see if setuptools has been removed from images at all | 21:28 |
| pabelanger | updating here, for people following. I missed setting release_python: python3 | 21:30 |
| fungi | sounds like it was that we install the package toolchain for python3 in our jobs by default, you have to tell it if you want python2 libs added | 21:30 |
| clarkb | fungi: I think those next changes have applied too | 21:33 |
| clarkb | fungi: also fyi you responded to frickler but not in the way gerrit expected at https://review.opendev.org/#/c/758598/1/playbooks/roles/gerrit/tasks/main.yaml | 21:35 |
| clarkb | took me a minute to figure that out :) | 21:36 |
| fungi | oh, yep, i keep forgetting gertty picks the old side by default if you comment on an unchanged line in the unified diff view | 21:37 |
| clarkb | fungi: also I'm thinking that since we're doing these config changes and also docker volume mount changes we should maybe restart gerrit in the near future just to be sure everything is happy before we go to the upgrade? | 21:37 |
| clarkb | maybe that is a tomorrow morning task | 21:38 |
| fungi | yeah, just so there's no nasty surprises if we need to roll back | 21:38 |
| fungi | i agree | 21:38 |
| clarkb | fungi: ianw https://github.com/go-gitea/gitea/milestone/67 looks like there are still two open issues on the gitea 1.13 milestone | 21:51 |
| hashar | clarkb: hi ! For your Gerrit upgrade, we had a lot of fix up for the zuul plugin. The series end at https://gerrit-review.googlesource.com/c/plugins/zuul/+/275055 | 21:54 |
| clarkb | hashar: thanks. We aren't going to add that plugin to start. Idea is to upgrade, get settled in then start investigating plugins and features | 21:55 |
| hashar | clarkb: I am not sure whether you are using it at opendev, but if so, you will definitely require those. That makes it compatible ;) | 21:55 |
| hashar | ah ok :) | 21:55 |
| clarkb | hashar: I think we'd like to, we've just decided that we need to do one thing at a time to make progress | 21:55 |
| hashar | sounds wise | 21:55 |
| hashar | I have updated our doc tonight, and we have some basic procedure to fetch the proper plugins from upstream https://wikitech.wikimedia.org/wiki/Gerrit#Fetch_our_additional_plugins | 21:56 |
| hashar | maybe that will be of some help when you reach that point :] | 21:56 |
| clarkb | docs are always helpful, thank you for the link | 21:57 |
| hashar | else you can probably borrow them from upstream CI : https://gerrit-ci.gerritforge.com/view/Plugins-stable-3.2/ | 21:57 |
| ianw | clarkb: huh, they look different to the blockers i saw before, so, progress? :) | 21:58 |
| clarkb | ianw: I think that list may be what I had last seen a couple weeks ago? it does seem very close | 21:58 |
| *** sboyron has quit IRC | 21:58 | |
| *** smcginnis has joined #opendev | 21:59 | |
| ianw | sigh, so the devel job uses tox and installs ansible, and then ansible-base can't install over it | 22:02 |
| *** ralonsoh has quit IRC | 22:03 | |
| pabelanger | ianw: yup, it isn't fun | 22:03 |
| ianw | does anyone know why we run "run tox without tests" in the tox role first, then install siblings? | 22:03 |
| clarkb | ianw: its to install the sibling via pypi then overwrite it iirc | 22:03 |
| clarkb | ianw: otherwise there isn't really a way to split out the package ebcause pip doesn't let you exclude things iirc | 22:03 |
| clarkb | (so a dep of a dep can pull it in at a different version and now you don't test the sibling) | 22:04 |
| pabelanger | ianw: one thing we are going to do, is create a fake 'ansible' package when installing from ansible/ansible devel to handle requirments files | 22:05 |
| pabelanger | but haven't done that work yet | 22:05 |
| ianw | sigh yeah. it breaks in this case though because the initial testinfra install pulls in ansible. i wonder if it could uninstalling siblings first, then reinstall | 22:05 |
| pabelanger | but also, ansible/ansible devel is not ACD | 22:05 |
| ianw | pabelanger: you know, i feel like we already did that somewhere ... | 22:05 |
| ianw | yep @ https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/install-ansible/tasks/install_ansible_stub.yaml | 22:06 |
| ianw | problem is the testinfra tox install | 22:07 |
| ianw | this did work, briefly. the "will not install" thing must be newish | 22:07 |
| ianw | part of the point was to test testinfra against ansible-devel ... | 22:08 |
| pabelanger | ianw: testinfra on devel may just no work any more | 22:10 |
| pabelanger | given there might be modules now missing | 22:10 |
| pabelanger | what you should test against is ansible (ACD) devel | 22:10 |
| pabelanger | which is different then ansible/ansible devel | 22:10 |
| pabelanger | ansible/ansible is becoming ansible-core | 22:10 |
| ianw | pabelanger: so ... is there something I can clone? :) i'm around the whole ansible/ansible-base split but i'm not sure how acd is built | 22:12 |
| pabelanger | git clone, I don't believe so | 22:12 |
| pabelanger | the acd package is build via scripts of collections from galaxy | 22:13 |
| pabelanger | let me see if I can find where the snapshot is stored | 22:13 |
| openstackgerrit | Merged opendev/system-config master: Stop managing gerrit's local git mirror dir https://review.opendev.org/758598 | 22:13 |
| ianw | pabelanger: hrm, that's still going to be hard to integrate into a tox environment using zuul-jobs tox runner anyway. it will install the siblings from required-projects:, but not a tarball/egg | 22:14 |
| pabelanger | ianw: yah, there is no single git source now for 'released' ansible | 22:15 |
| pabelanger | it is pull in from all over the place | 22:15 |
| ianw | i'm thinking maybe the zuul-jobs tox role should get something like "pre_unisntall_packages" or something; then it could remove "ansible" and then it would correctly install ansible-base | 22:15 |
| ianw | although, i'm not sure if the actual dependency resolver would then get unhappy and try removing testinfra and everything from the environment :/ | 22:15 |
| pabelanger | ianw: I am being pointed to https://github.com/ansible-community/antsibull/pull/195 | 22:16 |
| pabelanger | which should create nightly builds | 22:16 |
| pabelanger | what would be neat, is if we turned that into a zuul-jobs | 22:17 |
| pabelanger | then we both could use it | 22:17 |
| ianw | dmsimard is @redhat again? | 22:19 |
| pabelanger | yup | 22:19 |
| ianw | cool | 22:19 |
| pabelanger | re-joined last week I think | 22:19 |
| pabelanger | I have to run, good luck on ansible-base issue | 22:20 |
| ianw | pabelanger: hrm, yes nightly would be ok, but i'm still not sure how to convince tox to install it | 22:21 |
| ianw | we could hack maybe. have a requirements.txt that usually just specifies "ansible" but for the devel tests is modified to install from local checkout | 22:25 |
| *** hashar has quit IRC | 22:26 | |
| fungi | yeah, any hacks you're considering around selectively removing dependencies, even temporarily, are likely break horribly when the next pip release happens and it switches to statefully considering the installed dependency set | 22:28 |
| fungi | right now you can say "uninstall this package (which is a dependency of something i have installed)" and pip will be like, okay whatevs | 22:31 |
| fungi | soon pip will likely refuse, "the package you're trying to remove is needed by another package" ...or maybe remove the packages which depend on that too | 22:32 |
| fungi | anyway, i wouldn't assume pip's behavior will remain consistent there. if you're going to try a clever trick like that, be sure to also test it with the experimental dep solver flipped on | 22:33 |
| clarkb | I wonder if that will break install siblings | 22:34 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [WIP] codesearch container https://review.opendev.org/762960 | 22:34 |
| clarkb | fungi: looks like the jeeypb update may have happened? | 22:34 |
| clarkb | basing that on the contents of the docker compose file | 22:34 |
| ianw | fungi: yeah, i'm not sure it can be done. if testinfra says it depends on "ansible" and "ansible-base" will not install when "ansible" exists ... it's just broken :/ | 22:35 |
| ianw | there's no way to say "ansible | ansible-base" | 22:35 |
| * clarkb gonna take a break for a bit now for some exercise since those changes all seem to be landing without major issue | 22:35 | |
| *** stevebaker has quit IRC | 22:37 | |
| ianw | wait, it's "ansible-core" ans "ansible-base" ... but same general issue | 22:37 |
| *** stevebaker has joined #opendev | 22:38 | |
| fungi | it used to be possible before the expectation of static dep metadata for python packages | 22:42 |
| fungi | in short, you could create a package which dynamically adjusts its install_requires and then depend on that | 22:43 |
| fungi | granted, the install_requires become chiseled into stone at install time, so you'd have to reinstall to change them, which probably doesn't solve much | 22:45 |
| ianw | of course there's "set the ANSIBLE_SKIP_CONFLICT_CHECK environment variable", the change that adds that https://github.com/ansible/ansible/commit/54b002e1acad1e8d88e81965323d47ddb8c234fb tells me about how it fixed newlines, but doesn't describe something like *why* this necessary so it could be worked around | 22:51 |
| ianw | ohhh, it's because it's renaming itself ansible-core; this is why it used to work, when it was ansible base i think | 22:55 |
| fungi | so can we just directly use ansible-core there? | 23:00 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: test if ansible-core install works https://review.opendev.org/763099 | 23:03 |
| *** quasar` is now known as parallax | 23:03 | |
| *** parallax is now known as kwazar | 23:07 | |
| *** kwazar is now known as parallac | 23:13 | |
| *** parallac is now known as parallax | 23:13 | |
| *** parallax has joined #opendev | 23:14 | |
| *** slaweq has quit IRC | 23:21 | |
| dmsimard | ianw: o/ I'm indeed back at red hat in the ansible community team | 23:22 |
| ianw | dmsimard: cool, they're lucky to have you :) | 23:24 |
| dmsimard | still learning about the different bits for ansible >=2.10, collections and I see you've already found out about ansible-base/core rename | 23:24 |
| dmsimard | Transition from 2.9 to 2.10 is a bit... hard. There's a lot of work to do to improve it due to all the underlying changes from splitting out what isn't "core" | 23:25 |
| ianw | yes; https://review.opendev.org/#/c/763099/ ... i can either have the job working against -base or -core, but not both :/ | 23:27 |
| dmsimard | I wasn't even aware that core was a thing yet, only heard about plans for the rename | 23:27 |
| dmsimard | doesn't seem like it's a thing yet, at least according to https://pypi.org/project/ansible-core/#history | 23:29 |
| ianw | yeah, that's the problem; the base job tries to install that blank package | 23:31 |
| ianw | not ansible-base, but the system-config base | 23:31 |
| ianw | heh, what a mess | 23:31 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [WIP] codesearch container https://review.opendev.org/762960 | 23:32 |
| dmsimard | yeah no comments on the rename :/ | 23:33 |
| ianw | is there actually any problem installing ansible-core when ansible-base is installed? i feel like the check might actually be wrong | 23:33 |
| ianw | dmsimard: yeah, if you could get ansible changelog comments to be more openstacky, that would be great :) i think it's something we generally do well at | 23:34 |
| dmsimard | they actually adopted something reno-ish for changelog fragments | 23:35 |
| dmsimard | and I say reno-ish because it's the same general concept but not reno | 23:36 |
| dmsimard | I don't know about ansible-core vs ansible-base (yet), sorry :( | 23:37 |
| ianw | that's ok, testing against bleeding edge is probably prety niche | 23:38 |
| dmsimard | ah, so reading my backscroll | 23:39 |
| dmsimard | git source devel is now "ansible-core" | 23:39 |
| dmsimard | but it's not released yet | 23:39 |
| ianw | dmsimard: right, to sumarise our issue: | 23:41 |
| ianw | we install ansible-base in the tox environment for testinfra @ https://opendev.org/opendev/system-config/src/branch/master/tox.ini#L41 | 23:41 |
| ianw | for the system-config-base-devel-job, we have ansible as a required-project and so the tox role tries to install it as a sibling, this is to test our stuff against bleeding edge | 23:41 |
| ianw | however, that now fails, as ansible-core refuses to install over/alongside ansible-base | 23:42 |
| dmsimard | makes sense | 23:42 |
| dmsimard | I don't have a solution for you but let me take that feedback to the team tomorrow | 23:42 |
| ianw | https://zuul.opendev.org/t/openstack/build/8111670f053a46a6939e425949061d01/log/job-output.txt#19844 | 23:43 |
| ianw | i think we need to install ansible-base in the "release" job, and ansible-core in the -devel job. not quite sure how to do that from our side | 23:44 |
| dmsimard | the error message is misleading too | 23:44 |
| *** DSpider has quit IRC | 23:52 | |
| clarkb | ianw: in the release job we should just install "ansible" iirc | 23:53 |
| clarkb | since "ansible" is the compiled package? | 23:54 |
| ianw | clarkb: yeah, i think it's working ATM because what we need from testinfra can survive with "ansible-base" | 23:54 |
| ianw | basically testinfra is just using ansible to ssh in, and it's convenient because we just pass it the already written inventory file | 23:55 |
| *** zbr has quit IRC | 23:57 | |
| *** zbr4 has joined #opendev | 23:57 | |
| *** tosky has quit IRC | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!