| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 00:41 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 00:48 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 00:52 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 01:01 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 01:06 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 01:14 |
| *** ysandeep|away is now known as ysandeep|rover | 01:30 | |
| *** ysandeep|rover is now known as ysandeep|away | 01:31 | |
| *** diablo_rojo has joined #opendev | 01:32 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 01:33 |
| *** diablo_rojo has quit IRC | 01:47 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 01:57 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 02:44 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 02:52 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 03:03 |
| *** ralonsoh has joined #opendev | 03:04 | |
| *** ysandeep|away is now known as ysandeep|rover | 03:33 | |
| *** ykarel|away is now known as ykarel | 04:12 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 04:27 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add bionic-plain testing https://review.opendev.org/717655 | 04:27 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: test jobs: fixup autogeneration header https://review.opendev.org/717656 | 04:27 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 04:29 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add bionic-plain non-voting testing https://review.opendev.org/717655 | 05:44 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: test jobs: fixup autogeneration header https://review.opendev.org/717656 | 05:44 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 05:44 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 05:44 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 05:44 |
| AJaeger | ianw: I don't think 717656 is ok, could you double check that headers are in the files afterwards, please? | 06:27 |
| *** dpawlik has joined #opendev | 06:28 | |
| openstackgerrit | Merged opendev/base-jobs master: base: add bulk log download script https://review.opendev.org/716813 | 06:31 |
| *** rpittau|afk is now known as rpittau | 06:35 | |
| *** DSpider has joined #opendev | 06:53 | |
| ianw | AJaeger: arggh, yeah ... i think runuamel's claim to keep comments where they are in files is ... maybe not always true | 06:54 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: test jobs: fixup autogeneration header https://review.opendev.org/717656 | 06:57 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role https://review.opendev.org/717639 | 06:57 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip https://review.opendev.org/717663 | 06:57 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 06:58 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-tox: update Python version documentation https://review.opendev.org/717669 | 07:05 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-tox: update Python version documentation https://review.opendev.org/717669 | 07:24 |
| *** tosky has joined #opendev | 07:58 | |
| *** ralonsoh has quit IRC | 07:58 | |
| *** ralonsoh has joined #opendev | 08:11 | |
| *** dpawlik has quit IRC | 08:29 | |
| *** ysandeep|rover is now known as ysandeep|lunch | 08:36 | |
| *** dpawlik has joined #opendev | 08:38 | |
| *** ykarel is now known as ykarel|lunch | 08:50 | |
| *** ysandeep|lunch is now known as ysandeep | 09:18 | |
| *** ysandeep is now known as ysandeep|rover | 09:19 | |
| *** ykarel|lunch is now known as ykarel | 09:36 | |
| openstackgerrit | Riccardo Pittau proposed openstack/diskimage-builder master: Make ipa centos8 dib job voting https://review.opendev.org/717700 | 09:57 |
| AJaeger | infra-root, is ze05 broken? It shows a straight line at http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1&from=1586145483660&to=1586167083660 | 09:58 |
| AJaeger | Looks like we have changes waiting forever for a new node - that might be related ^ | 10:00 |
| openstackgerrit | Jan Kundrát proposed zuul/zuul-jobs master: docs: fix a typo in `run-test-command` https://review.opendev.org/717713 | 11:18 |
| *** roman_g has joined #opendev | 11:18 | |
| frickler | infra-root: AJaeger: failing to connect to ze05 via ssh, trying restart via API | 11:22 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Retire repo_build and pip_install roles https://review.opendev.org/717715 | 11:27 |
| frickler | #status log restarted ze05 which seems to have been dead since 2020-04-02-02:05Z | 11:31 |
| openstackstatus | frickler: finished logging | 11:31 |
| frickler | AJaeger: seems to have been in that state for a couple of days | 11:32 |
| frickler | infra-root: would we want to keep systemd logs from previous boots by creating /var/log/journal? I'm not sure whether we've discussed that question before or whether my brain is mixing contexts | 11:36 |
| AJaeger | thanks, frickler | 12:10 |
| AJaeger | infra-root, there are 7 hold nodes - are those all needed? | 12:13 |
| *** hashar has joined #opendev | 13:06 | |
| fungi | corvus: ianw: frickler: those 7 held nodes have your nicks in the comments | 13:08 |
| fungi | (all for over a week) | 13:09 |
| *** ykarel is now known as ykarel|afk | 13:11 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run Zuul using Ansible and Containers https://review.opendev.org/717620 | 13:26 |
| openstackgerrit | Merged openstack/diskimage-builder master: opensuse: fix python 2.x install https://review.opendev.org/716437 | 14:01 |
| *** ykarel|afk is now known as ykarel | 14:06 | |
| mordred | fungi: the databases tab in the rackspace web ui will not load for me | 14:23 |
| mordred | fungi: will it load for you? | 14:23 |
| AJaeger | mordred: want to +A your own changes https://review.opendev.org/717136 and https://review.opendev.org/717004 ? Or are they not ready yet? | 14:23 |
| mordred | fungi: nevermind | 14:23 |
| mordred | fungi: asking the question seems to have fixed it | 14:23 |
| mordred | AJaeger: thanks - first one done. second one is waiting on https://review.opendev.org/#/c/717057/ | 14:24 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency https://review.opendev.org/716663 | 14:30 |
| *** ysandeep|rover is now known as ysandeep|afk | 14:31 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: docker-install: workaround for centos-8 conflicts https://review.opendev.org/703053 | 14:36 |
| corvus | fungi: i removed my autohold | 14:37 |
| openstackgerrit | Merged openstack/project-config master: Run service-nameserver playbook on zone file changes https://review.opendev.org/717004 | 14:37 |
| fungi | mordred: the rackspace dashboard is sometimes inexplicably slow to do things for me too | 14:39 |
| mordred | fungi: I have achieved my goal - which is finding out how big the etherpad db is | 14:40 |
| mordred | fungi: (I realized over the weekend that if we're going to run the etherpad db directly, we might want a volume to put it on :) ) | 14:40 |
| fungi | corvus: thanks! looks like frickler took his out as well, so that just leaves ianw next time he's awake | 14:41 |
| corvus | mordred: we want abiword on etherpad? i thought we didn't want it | 14:41 |
| corvus | huh, apparently we do want it | 14:42 |
| mordred | corvus: yeah - I mean I don't know about "want - but we're currently installing it at least | 14:43 |
| mordred | you know - we should probably drop in some my.cnf settings - like for collation and character set | 14:45 |
| corvus | mordred: should we disable abiword? | 14:46 |
| corvus | and use upstream images? | 14:46 |
| mordred | corvus: I'd be fine with that myself | 14:47 |
| corvus | mordred: +2 https://review.opendev.org/716442 | 14:48 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Retire repo_build and pip_install roles https://review.opendev.org/717715 | 14:48 |
| mordred | corvus: cool. want me to make the abiword change first? | 14:50 |
| mordred | corvus: (otherwise we might publish one etherpad image and then stop) | 14:50 |
| corvus | mordred: i'm ambivalent; let's see what other folks think | 14:51 |
| corvus | clarkb: ^? | 14:51 |
| frickler | fungi: there are some more autoholds without held nodes in zuul it seems, maybe clean these up, too? not directly obvious how old these are, though | 14:53 |
| *** sgw has quit IRC | 14:53 | |
| mordred | corvus: I need to put new patchset up to add db settings anyway (we have 3 settings in the cloud db) | 14:54 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency https://review.opendev.org/716667 | 14:56 |
| fungi | frickler: thanks, looks like i had a few in there for some reason | 14:56 |
| fungi | does deleting autoheld nodes with `nodepool delete` not remove the autoholds? thought those went away as soon as the node entered a held state | 14:57 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Retire OSA repo_build and pip_install projects https://review.opendev.org/717775 | 14:59 |
| mordred | fungi: they stick around now - you can just manage it all from the zuul cli now - but I don't know that I've done a direct nodepool delete since that has happened | 14:59 |
| mordred | fungi, corvus, clarkb : https://review.opendev.org/#/c/717491/ is ready regardless of abiword - and the "trigger deploy on zone file changes" patch has landed, so we should get to see zuul run the infra-prod playbook for it when we land it :) | 15:00 |
| fungi | oh, neat. so the autohold persists even if the count is reached, and you can clean up the nodes by deleting the hold itself? | 15:00 |
| fungi | that'll save me some lookup steps | 15:00 |
| frickler | fungi: mordred: yeah, the only thing missing is to get the node IP from zuul, either need to go via nodepool or look up on the GUI via the build id | 15:00 |
| mordred | fungi: that is my current understanding of the system | 15:00 |
| mordred | frickler: ++ | 15:00 |
| frickler | at least I haven't found an easier path | 15:01 |
| *** roman_g has quit IRC | 15:05 | |
| clarkb | corvus: mordred does the upstream image not have abiword? so using it implies disabling that integration? | 15:08 |
| corvus | clarkb: that's my understanding | 15:10 |
| *** sgw has joined #opendev | 15:12 | |
| *** ysandeep|afk is now known as ysandeep | 15:13 | |
| fungi | excellent | 15:13 |
| fungi | it used to be horribly broken if we didn't at least provide abiword | 15:14 |
| clarkb | fungi: well we toggle the option to enable that integration | 15:15 |
| clarkb | I personally never use that integration and would be fine with disabling it | 15:15 |
| corvus | fungi: looking at the git history, i think we got into to trouble because we had a config file that said to use abiword but did not have it installed | 15:15 |
| clarkb | normal copy and pase tends to work for me | 15:15 |
| corvus | but i'm not sure we ever had it disabled and not installed; it seems like that's probably the default and i would hope would work fine | 15:15 |
| corvus | and i think it's worth asking, because sometimes we have stuck abiword 100% cpu processes? | 15:16 |
| fungi | yeah, for some reason i didn't realize it was possible to configure it to turn that integration off | 15:16 |
| fungi | maybe it wasn't possible 6-7 years ago and we never revisited | 15:17 |
| fungi | though if we have explicit configuration there enabling it, that implies it must have been possible to not enable it even back then | 15:17 |
| fungi | and yeah, i don't think i've ever used the ms-word compatible doc exporting feature | 15:18 |
| corvus | yeah, i don't know if/why we never tried that :) | 15:18 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename install-openshift to ensure-openshift for consistency https://review.opendev.org/716675 | 15:20 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename install-podman to ensure-podman for consistency https://review.opendev.org/716682 | 15:20 |
| *** ykarel is now known as ykarel|away | 15:28 | |
| *** ysandeep is now known as ysandeep|away | 15:39 | |
| mordred | corvus, clarkb, fungi : ok - I'll remove the abiword and resubmit the patch | 15:44 |
| mordred | we can always add it back later if we want | 15:44 |
| fungi | awesome, thanks! | 15:46 |
| fungi | also we can easily test that on the current etherpad-dev | 15:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org https://review.opendev.org/716442 | 15:47 |
| mordred | corvus, clarkb, fungi : ^^ that removes abiword, our own images, and adds a my.cnf file with the settings that were in rax | 15:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org https://review.opendev.org/716442 | 15:55 |
| mordred | added depends-on to the dns change | 15:55 |
| *** rpittau is now known as rpittau|afk | 16:08 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename install-devstack to ensure-devstack for consistency https://review.opendev.org/716685 | 16:08 |
| corvus | clarkb: https://review.opendev.org/717491 and https://review.opendev.org/716442 are ready to go i think | 16:14 |
| clarkb | corvus: will review as soon as meeting is over | 16:15 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename install-javascript-packages to ensure-javascript-packages for consistency https://review.opendev.org/716687 | 16:17 |
| clarkb | mordred: your editor eats tabs :P | 16:21 |
| mordred | clarkb: uhoh | 16:21 |
| mordred | clarkb: oh - I should fix that maybe | 16:21 |
| clarkb | bind won't care | 16:21 |
| *** diablo_rojo has joined #opendev | 16:22 | |
| mordred | cool | 16:23 |
| mordred | then I will not fix that | 16:23 |
| mordred | clarkb: we get to see the nameserver playbook fire when that merges! | 16:24 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add dns entries for new etherpad01.opendev.org https://review.opendev.org/717491 | 16:25 |
| fungi | bind *will* care in conffiles, but not in zonefiles (unless they eventually changed that), but keeping consistent is nice so that other people with different tabstops don't see wacky layouts | 16:25 |
| openstackgerrit | Monty Taylor proposed opendev/zone-opendev.org master: Fix whitespace because tabs and spaces aren't the same https://review.opendev.org/717794 | 16:27 |
| mordred | fungi, clarkb : ^^ | 16:27 |
| fungi | thanks! | 16:29 |
| sgw | f | 16:31 |
| sgw | Morning folks! | 16:31 |
| mordred | clarkb: hrm. it doesn't look like the job fired | 16:32 |
| sgw | fungi: I am back looking at the semver changes for starlingx and still getting confused | 16:32 |
| fungi | sgw: did you test things out? | 16:32 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Use deploy pipeline for zone file updates https://review.opendev.org/717797 | 16:33 |
| clarkb | mordred: should we land ^ then land the whitespace fix to test it? | 16:34 |
| sgw | I tried to tag with v4.0.0.0b0 and setup -q rpm_version returned 3.9999.9999.b0 which then caused a nomralized version of 3.9999.9999b0 and a mismatch | 16:34 |
| sgw | /usr/lib/python2.7/site-packages/setuptools/dist.py:355: UserWarning: Normalizing '3.9999.9999.b0' to '3.9999.9999b0' | 16:38 |
| sgw | and then later: error: File not found: /builddir/build/BUILDROOT/starlingx-dashboard-3.9999.9999.b0-0.tis.x86_64/usr/lib/python2.7/site-packages/starlingx_dashboard-3.9999.9999.b0*.egg-info | 16:38 |
| clarkb | mordred: can you see comment on https://review.opendev.org/#/c/716442/17 particularly the utf8 one, but the other needs to be handled if not yet already | 16:38 |
| AJaeger | config-core, please review https://review.opendev.org/716261 - and two zuul-base-jobs changes: https://review.opendev.org/#/c/708871/ https://review.opendev.org/#/c/716298/ | 16:39 |
| AJaeger | and two zuul-jobs changes for review, please: https://review.opendev.org/717713 and https://review.opendev.org/717655 | 16:40 |
| sgw | fungi: not sure what the next steps should be there, the difference with and without adding the Sem_Ver footer was just 3.0.9999 vs 3.9999.9999 | 16:45 |
| clarkb | sgw: the problem there being the rpm_version emitted is not normalized? I think that may be expected because rpm versions aren't necessarily pep440 compliant | 16:46 |
| clarkb | sgw: the rpm_version is a version that rpm can handle and the normalized version is a version pip/pep440 can handle | 16:47 |
| clarkb | sgw: are you setting PBR_VERSION during your build? | 16:48 |
| clarkb | I wonder if you need to explicitly override the version so that it doesn't normalize | 16:48 |
| fungi | sgw: yeah, pep-440 -> rpm -> pep-440 isn't intended to be idempotent, if that was the expectation | 16:49 |
| fungi | what is renormalizing the rpm version number? | 16:49 |
| sgw | No, not setting PBR_VERSION, working from the tags/Sem_Ver footer | 16:49 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename install-nodejs to ensure-nodejs for consistency https://review.opendev.org/716692 | 16:50 |
| fungi | pbr says the equivalent rpm version for 4.0.0.0b0 (a pep-440 normalized version number) should be 3.9999.9999.b0 (which is not pep-440 normalized and not intended to be) | 16:51 |
| fungi | the idea there is that the python version returned inside your packaging ought to be 4.0.0.0b0 | 16:52 |
| sgw | Or better yet how do I get the right pep-440 version into an rpm specfile for packaging it in an RPM? | 16:52 |
| fungi | but that the rpm containing it should declare package version 3.9999.9999.b0 | 16:52 |
| fungi | so when creating the rpm package containing 4.0.0.0b0 of some python module, you would use something like `setup -q rpm_version` to generate the version number to embed in the specfile | 16:53 |
| fungi | i think | 16:53 |
| fungi | mordred: ^ does that jive with your understanding of the use of pbr's rpm version feature? | 16:54 |
| sgw | Right which is what we are doing, but when trying to package via %files and this directive: %{python2_sitelib}/%{py_pkg_name}-%{version}*.egg-info | 16:54 |
| sgw | It fails because the version is the PBR version not the pep-440 version | 16:54 |
| clarkb | sgw: fungi I think you need to set PBR_VERSION env var to the rpm_version output | 16:55 |
| sgw | thus the above error of file not found. | 16:55 |
| fungi | the %{version} part is the problem there i guess, and is taking the rpm package version from the specfile? | 16:55 |
| clarkb | otherwise pbr is going to default to producing a pep440 version | 16:55 |
| clarkb | you need to force it to use the rpm version isntead | 16:55 |
| fungi | er, shouldn't it be the other way around? | 16:55 |
| clarkb | fungi: not if you are building an rpm | 16:56 |
| clarkb | if you are building an rpm you want the version to be rpm compliant | 16:56 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename install-yarn to ensure-yarn for consistency https://review.opendev.org/716693 | 16:56 |
| fungi | you want the pep-440 version in the python egg-info because of mixing distro packaging and pip install | 16:56 |
| clarkb | oh hrm, might want to ask rdo packagers what they do | 16:57 |
| fungi | the version number you get from `pip install foo===4.0.0.0b0` should be the same exact module as you got from `rpm -i ...` or whatever of the 3.9999.9999.b0 rpm | 16:57 |
| fungi | so instead of using the rpm version string in the egg-info template, you need the python package version string | 16:58 |
| openstackgerrit | Merged openstack/project-config master: Use deploy pipeline for zone file updates https://review.opendev.org/717797 | 16:58 |
| *** hashar is now known as hasharAway | 16:58 | |
| clarkb | fungi: sgw in that acse you'd need to instruct your spec file to look for the egg with pep440 version name? | 16:59 |
| fungi | yes, it sounds like whatever is being used to template the specfile assumes a 1:1 match between the python package version string and the rpm version string, which is probably viable for normal releases but there is no good 1:1 correlation between pep-440 prerelease version strings and rpm version strings which retain linear versioning | 17:00 |
| mordred | clarkb: I don't thnk the deploy pipeline thing is why it didn't run - I'm honestly not sure why it didn't run | 17:00 |
| clarkb | mordred: should we land the whitespace fix and see what it does ro check logs on the first change first? | 17:01 |
| mordred | clarkb: I think we should check the logs on the first change | 17:01 |
| clarkb | k | 17:01 |
| sgw | I think we are getting closer! Yes, I need a way to get the pep440 version name inside the spec file for the egg | 17:01 |
| mordred | corvus: unless you have any ideas of why the nameserver job didn't fire on the zone repo merge off the top of your hear | 17:02 |
| mordred | head | 17:02 |
| AJaeger | sgw: you might want to ask on #openstack-rpm-packaging - I guess the Packaging-RPM team run into those issue already | 17:02 |
| corvus | mordred: sorry, i'm in a debug pit right now; i can join you in a few mins | 17:02 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: docker-install: workaround for centos-8 conflicts https://review.opendev.org/703053 | 17:03 |
| fungi | AJaeger: sgw: also they probably have specfiles in public git repos they can point you to for precise examples | 17:03 |
| sgw | AJaeger: I will take a look at that repo also and ask there. | 17:03 |
| mordred | corvus: no worries - enjoy your pit | 17:03 |
| mordred | clarkb: but yeah - let's definitely hold the whitespace change for testing :) | 17:03 |
| clarkb | mordred: `grep 8981101b71c044d3adbf6b768ce6df8e /var/log/zuul/debug.log | grep promote` is what I'm looking at | 17:04 |
| clarkb | it seems to say "I matched the promote pipeline" but then its not yet clear to me why it took so little action after that | 17:05 |
| sgw | thanks all, biab | 17:05 |
| mordred | clarkb: 2020-04-06 16:26:06,260 DEBUG zuul.layout: [e: 8981101b71c044d3adbf6b768ce6df8e] Job <Job infra-prod-service-nameserver branches: None source: opendev/base-jobs/zuul.d/jobs.yaml@master#25> did not match files in <Change 0x7f3bc4458320 opendev/zone-opendev.org 717491,1> | 17:05 |
| mordred | clarkb: I thought files matchers didn't apply in promote. or do we need to do files: [] in the pipeline def? | 17:06 |
| clarkb | mordred: fiels matchers work in promote because its a change merged event | 17:07 |
| clarkb | (same thing with deploy) | 17:07 |
| clarkb | its ref-updated that files won't work so ya I think we need to clear that out | 17:07 |
| clarkb | mordred: also that explains why my | grep promote wasn't helping :) | 17:07 |
| clarkb | it wasn't logged as part of promote pipeline processing | 17:07 |
| clarkb | and ya I think if any files merge to those repos we run the job | 17:08 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Retire OSA repo_build and pip_install projects https://review.opendev.org/717775 | 17:08 |
| clarkb | so don't need explicit matchers unless we need to say .* | 17:08 |
| mordred | clarkb: ah - so ... actually, what about this. ... | 17:08 |
| mordred | clarkb: 2 things - a) on the other jobs in system-config where we trigger them from other repos too - we do the file matchers in the pipeline def instead of in the job | 17:09 |
| mordred | maybe we shoud do that here too | 17:09 |
| mordred | but also - we can match on zones/.* - since that's the only files in the zone repo that matter | 17:10 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Change zone file job file matchers https://review.opendev.org/717808 | 17:10 |
| mordred | so like that | 17:10 |
| clarkb | ya that should work | 17:10 |
| AJaeger | let's try ;) | 17:10 |
| clarkb | once that lands we can land the whitespace fix | 17:11 |
| mordred | we're gonna need one more thing ... | 17:13 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Blank out the dependencies for zone repos https://review.opendev.org/717809 | 17:14 |
| mordred | clarkb, AJaeger : ^^ | 17:14 |
| mordred | (we'll get a job graph freeze error otherwise) | 17:14 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on ensure-podman https://review.opendev.org/716578 | 17:14 |
| clarkb | mordred: why will we get a graph freeze error? those repos are all in the same tenant | 17:15 |
| mordred | because infra-prod-service-nameserver depends on 4 other jobs, none of which are run in the pipeline for zone-opendev.org | 17:15 |
| clarkb | mordred: and they are nto all soft deps? | 17:16 |
| mordred | (we hit this with the project-config patch) | 17:16 |
| mordred | they are not | 17:16 |
| clarkb | I see | 17:16 |
| mordred | but even if they are - zuul still gets grump | 17:16 |
| mordred | because they're still depends | 17:16 |
| mordred | soft lets it work if the job was skipped due to matchers - but it still needs to be there to ski | 17:16 |
| mordred | skip | 17:16 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 17:18 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename install-pdk-dependencies to ensure-pdk-dependencies for consistency https://review.opendev.org/716695 | 17:20 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on ensure-podman https://review.opendev.org/716578 | 17:20 |
| corvus | mordred: i've been looking into afs for zuul executors in containers. results are mixed. on the one hand, the simple case works -- you can get a ticket and a token inside a container and use bind-mounted afs. but zuul-bwrap gets a new pag for each process it spawns (to keep zuul jobs separate), and when it does that, it doesn't seem like that pag is accessible inside the container anymore. so | 17:24 |
| corvus | everything succeeds, but the tokens aren't there and authenticated afs accesses fail. | 17:24 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on ensure-podman https://review.opendev.org/716578 | 17:24 |
| corvus | mordred: i'm trying to think of other things to try, but i'm not coming up with anything | 17:24 |
| mordred | corvus: hrm | 17:25 |
| clarkb | corvus: pag? | 17:25 |
| clarkb | (but that almost seems like you might need to not namespace processes? | 17:25 |
| corvus | clarkb: process authentication group; a kernel construct afs uses to keep accesses scoped to users | 17:25 |
| mordred | corvus: so to use auth'd afs the ticket has to be gotten inside of the same bwrap process? | 17:26 |
| corvus | mordred: no, even that doesn't work | 17:26 |
| clarkb | corvus: I would try using the host process namespace | 17:26 |
| mordred | oh - is that becaues it's a different pag than the afs client is running in? | 17:26 |
| clarkb | (since in theory that is how it works without docker today) | 17:26 |
| corvus | once bwrap does the setpag ioctl, it seems like the pag that aklog sees is scoped in a way that is different than how the filessystem accesses appear to the kernel | 17:26 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on ensure-podman https://review.opendev.org/716578 | 17:27 |
| mordred | corvus: and docker itself isn't doing a setpag | 17:27 |
| corvus | clarkb: you mean --pid=host ? | 17:28 |
| corvus | mordred: yeah, without zuul-bwrap/setpag, all docker containers get the same pag | 17:28 |
| corvus | (so an aklog in one container allows access in another) | 17:28 |
| clarkb | corvus: yes, that appears to be the flag | 17:29 |
| mordred | corvus: I mean - it seems like we _want_ the pag separation - otherwise we'd be leaky | 17:29 |
| corvus | mordred: yeah, i don't think we need to care about separating from docker containers, but i think we need the zuul-bwrap pag separation to work | 17:30 |
| mordred | corvus: actually - wait a sec ... how is this working today? | 17:30 |
| corvus | i mention the docker pag sharing just as information to maybe help understand what's going on | 17:30 |
| corvus | mordred: we don't run in docker? | 17:30 |
| mordred | but we do run in bwrap | 17:30 |
| mordred | and it's bwrap doing the pag call | 17:30 |
| corvus | mordred: sure, but that's not docker; it's the addition of docker that's screwing it up here. i don't know why. | 17:31 |
| mordred | if docker isn't doing anything with the pag entries, one would imagine we'd see the same problem using afs in bwrap whether ... not | 17:31 |
| mordred | nod | 17:31 |
| mordred | so - docker is doing _something_ that is causing the pag separation to be weird | 17:31 |
| clarkb | it could be the nesting of namespaces (one of them I don't know which) is the problem | 17:31 |
| clarkb | its hierarchical and supposed to just work, but there may be assumptions in somethign that break that | 17:32 |
| corvus | mordred: yeah; i'm working on trying clarkb's thing | 17:32 |
| mordred | cool | 17:32 |
| clarkb | infra-root I think now is a good time to send https://etherpad.openstack.org/p/IBLWO1WBBc email about opendev comms channels | 17:33 |
| clarkb | want to give that one last look over? I'll be sending it to -infra and all the project-discuss lists | 17:33 |
| openstackgerrit | Merged openstack/project-config master: Change zone file job file matchers https://review.opendev.org/717808 | 17:34 |
| mordred | corvus: btw - we figured the other thing out | 17:34 |
| corvus | mordred, clarkb: no joy | 17:34 |
| mordred | corvus: boo | 17:34 |
| corvus | mordred: i saw, thanks :) | 17:34 |
| clarkb | corvus: is the container running privileged? if not that might have desireable side effects | 17:35 |
| corvus | mordred, clarkb: this is basically my test process: http://paste.openstack.org/show/791692/ | 17:36 |
| corvus | clarkb: it is (that's necessary for zuul-bwrap) | 17:36 |
| mordred | corvus: https://github.com/pagarme/runc/tree/afs - last 2 commits - is someone taking a stab at adding afs support to runc - it doesn't look like any new info - but including it in case there's something helpful in there | 17:37 |
| mordred | (it's basically adding runc calling setpag() - so I think new new info) | 17:38 |
| openstackgerrit | Merged openstack/project-config master: Blank out the dependencies for zone repos https://review.opendev.org/717809 | 17:39 |
| clarkb | mordred: ^ want to approve your whitespace fix now? | 17:39 |
| fungi | clarkb: assuming it's the same as yesterday, already lgtm. otherwise i can reread once i'm done cooking lunch | 17:39 |
| corvus | mordred: this is fascinating: https://github.com/ist-dsi/docker-afs/blob/master/afs-client/init-script.sh | 17:39 |
| clarkb | fungi: I haven't chagned it | 17:39 |
| corvus | mordred: thatloads the openafs module from within a container | 17:40 |
| corvus | (i don't think that's a solution for us, but i find it very interesting) | 17:40 |
| mordred | corvus: yeah | 17:41 |
| clarkb | the kernel is shared though right? so loading a kernel module within a container or without has the same underlying effect? | 17:41 |
| corvus | clarkb: i would think so :) | 17:42 |
| mordred | clarkb: the comments say "The afs kernel module cannot be used simultaneously by two (or more) containers." | 17:42 |
| clarkb | ok so afs isn't namespaced | 17:43 |
| clarkb | so bwrap must not be namespacing somethign that docker is namespacing | 17:43 |
| clarkb | ? | 17:43 |
| mordred | corvus: oh - I wonder if kafs woudl be any better here? | 17:43 |
| corvus | i don't know that i understand enough of the situation to say that | 17:43 |
| corvus | clarkb: ^ | 17:43 |
| mordred | (also - I don't know enough yet either) | 17:43 |
| clarkb | mordred: https://review.opendev.org/#/c/717794/ is the chagne I think we can land now re dns zones | 17:44 |
| corvus | clarkb: for example, some of this behavior may be related to docker's daemon process | 17:44 |
| mordred | clarkb: agree. done | 17:44 |
| clarkb | corvus: oh hrm | 17:44 |
| mordred | which is actually one of the reasons we were originally interested in using podman instead of docker for things | 17:44 |
| corvus | mordred: re kafs, perhaps? i wonder if the same ioctl is available? | 17:45 |
| *** gadzhet has joined #opendev | 17:45 | |
| corvus | to perform setpag, we call the ioctl at /proc/fs/openafs/afs_ioctl | 17:45 |
| clarkb | alright I'm sending comms channel emails now | 17:46 |
| corvus | it does seem like both a kafs and a podman experiment (possibly with both docker afs kafs) might produce additional data | 17:46 |
| mordred | corvus: https://www.infradead.org/~dhowells/kafs/user_interface.html says no | 17:46 |
| corvus | it does seem like both a kafs and a podman experiment (possibly with both openafs and kafs) might produce additional data | 17:46 |
| mordred | corvus: yes - I think those experiments are worthwhile | 17:46 |
| mordred | I don't know how up to date that web page is | 17:46 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Fix whitespace because tabs and spaces aren't the same https://review.opendev.org/717794 | 17:47 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 17:48 |
| corvus | mordred: i think kafs uses keyrings instead of pags? | 17:49 |
| corvus | so things could be very different | 17:49 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Use ensure-* roles https://review.opendev.org/717813 | 17:50 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 17:50 |
| mordred | corvus: they could - although that might be nicer for controlling things? | 17:50 |
| corvus | mordred, clarkb: did ianw's experiments with kafs result in learning that it wasn't as stable as openafs and we saw crashes on the mirror servers? | 17:50 |
| * mordred does not 100% remember - but thinks that might be true | 17:51 | |
| *** dpawlik has quit IRC | 17:51 | |
| clarkb | yes we saw crashes on the mirrors | 17:51 |
| clarkb | we had to reboot them to recover iirc | 17:51 |
| corvus | i can't recall how new of a kernel those used -- maybe that was just whatever was in xenial? | 17:51 |
| clarkb | it was bionic | 17:51 |
| corvus | :( | 17:52 |
| corvus | well, how about i try my current experiment with openafs+podman. that shouldn't be hard to do. | 17:52 |
| mordred | ++ | 17:52 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on ensure-podman https://review.opendev.org/716578 | 17:52 |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Use ensure-* roles https://review.opendev.org/717815 | 17:54 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-yarn: run ensure-nodejs before https://review.opendev.org/717817 | 17:55 |
| corvus | clarkb, mordred: same behavior with podman | 17:59 |
| mordred | corvus: well - at least it's consistent | 18:00 |
| mordred | and probably rules out the daemon | 18:00 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename install-if-python to ensure-if-python for consistency https://review.opendev.org/716698 | 18:02 |
| gadzhet | Hey! Who can help with the zuul job problem? | 18:02 |
| gadzhet | This job https://zuul.opendev.org/t/openstack/build/1198d3e829504ce0a222c0547a693f52 always crashes with the error "ImportError: No module named configparser” project x/stackalytics | 18:02 |
| *** ralonsoh has quit IRC | 18:03 | |
| mordred | gadzhet: configparser is a python3 module - stestr doesn't support python2 anymore | 18:04 |
| mordred | so stackalytics might need to pin stestr in its requirements - at least for python2 | 18:05 |
| AJaeger | gadzhet: the stackalytics team is using github nowadays AFAIK, so I'm not sure that you submit to the right place | 18:05 |
| mordred | https://github.com/stackalytics/default_data | 18:06 |
| mordred | AJaeger: should we retire that repo? | 18:06 |
| AJaeger | mordred: last merge was three weeks ago - which surprises me. But if they really use github, we should retire... | 18:06 |
| mordred | yeah | 18:06 |
| mordred | the wiki pages points to the github repo above for making data changes | 18:07 |
| gadzhet | Oh. thanks for the information. | 18:09 |
| *** prometheanfire has quit IRC | 18:10 | |
| AJaeger | mordred: let me mail the stackalytics folks.. | 18:11 |
| fungi | i think they've been manually combining those somehow, because they want to track cncf project activity and don't want cncf contributors to have to submit changes to opendev (granted at the time they switched that up, the review site still said "openstack") | 18:12 |
| clarkb | mordred: etherpad01.opendev.org seems to resolve. I think those changes to project-config fixed it | 18:12 |
| mordred | clarkb: \o/ https://zuul.opendev.org/t/openstack/build/b43bb6bb01734be28f2ba59b8f2f06eb | 18:14 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-repositories: Add role https://review.opendev.org/717507 | 18:17 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-nodejs: refactor to use ensure-repositories https://review.opendev.org/717508 | 18:17 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-yarn: refactor to use ensure-repositories https://review.opendev.org/717509 | 18:17 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-yarn: run ensure-nodejs before https://review.opendev.org/717817 | 18:17 |
| *** mlavalle has joined #opendev | 18:18 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 18:19 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Disable log collection pending further auditing https://review.opendev.org/717823 | 18:20 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Mention new mailing lists https://review.opendev.org/717824 | 18:21 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Disable log collection pending further auditing https://review.opendev.org/717823 | 18:23 |
| *** prometheanfire has joined #opendev | 18:28 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 18:38 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Retire OSA repo_build and pip_install projects https://review.opendev.org/717775 | 18:44 |
| openstackgerrit | Albin Vass proposed opendev/system-config master: Use ensure-* roles https://review.opendev.org/717833 | 18:45 |
| openstackgerrit | Merged opendev/system-config master: Disable log collection pending further auditing https://review.opendev.org/717823 | 19:04 |
| *** hasharAway is now known as hashar | 19:08 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Rename install-go to ensure-go for consistency https://review.opendev.org/716689 | 19:10 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/project-config master: Retire OSA repo_build and pip_install projects https://review.opendev.org/717775 | 19:37 |
| openstackgerrit | Monty Taylor proposed opendev/zone-opendev.org master: Rotate DS record for acme.opendev.org https://review.opendev.org/717855 | 20:55 |
| mordred | clarkb, corvus, fungi : ^^ | 20:55 |
| openstackgerrit | Monty Taylor proposed opendev/zone-opendev.org master: Rotate DS record for acme.opendev.org https://review.opendev.org/717855 | 20:57 |
| mordred | clarkb: ^^ fixed | 20:57 |
| *** jkt has quit IRC | 21:04 | |
| *** jkt has joined #opendev | 21:04 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Rotate DS record for acme.opendev.org https://review.opendev.org/717855 | 21:08 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-yarn: run ensure-nodejs before https://review.opendev.org/717817 | 21:10 |
| *** hashar has quit IRC | 21:33 | |
| fungi | clarkb: related to the new ml announcement, https://review.opendev.org/717824 updates some places we refered to the old ml in system-config (so updates the opendev.org main page and a bit in our docs) | 21:50 |
| fungi | i intentionally skipped updating some bits we have in there about certificate authority/server keys for zuul, and gerrit accounts as those likely need changing outside the repo | 21:51 |
| clarkb | infra-root ^ anyone else able to ack that one in? | 21:51 |
| fungi | i also checked infra-manual but we don't seem to mention the ml in it (maybe we should, but there's at least nothing we need to update to stop referring to the old one) | 21:51 |
| fungi | there are tons of matches in our repos for the old ml in, like, setup.cfg files but i figure those are somewhere between non-urgent and pointless to fix, depending on whether or not we publish packages for them on pypi | 21:52 |
| fungi | i updated the setup.cfg in system-config because i just happened to be in there changing it in other files anyway | 21:53 |
| clarkb | ya I expect the vast majority of those things never end up on pypi | 21:54 |
| fungi | i have a staged change for the other references, i can push that up wip in case folks want to consider some or all of it | 21:54 |
| fungi | (for system-config) | 21:54 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Update some more uses of the old ML https://review.opendev.org/717867 | 21:55 |
| fungi | for completeness ^ | 21:56 |
| fungi | and now i've set that wip with inline comments about why those are in the wip patch | 22:00 |
| openstackgerrit | Merged zuul/zuul-jobs master: Rename install-go to ensure-go for consistency https://review.opendev.org/716689 | 22:04 |
| ianw | corvus: we actualy mostly hit issues with cachefilesd, which unfortunately seems ... undermaintained. although there were certainly kafs issues but we got a lot of them sorted out quickly | 22:08 |
| ianw | corvus: we were using bleeding edge kernels (at the time) | 22:08 |
| ianw | corvus: the statck is all still at https://review.opendev.org/#/c/665057/ waiting for the right time and interest to revisit ... | 22:09 |
| clarkb | #status log restarted all of logstash geard, workers and logstash itself to reset after nova fixed its n-api log files | 22:11 |
| openstackstatus | clarkb: finished logging | 22:11 |
| mordred | ianw: if you didn't see all the way back int eh scrollback, the investigation is "what do we need to do to use AFS inside of bubblewrap inside of docker" - which is a scenario which will become real if we start deploying zuul from containers | 22:23 |
| mordred | ianw: seems like it's going to be a fun issue to solve :) | 22:24 |
| mordred | ianw: one of the hypotheses (why kafs came up) was that maybe kafs would work better for this case - either by being a newer impl - or perhaps its use of keytabs instead of PAG will avoid the issue somehow | 22:25 |
| mordred | but we haven't dug in to that at all | 22:25 |
| *** hashar has joined #opendev | 22:26 | |
| ianw | mordred: it's R/O ... don't we mostly do write operations on nodes? i may be misremembring | 22:31 |
| *** gadzhet has quit IRC | 22:31 | |
| *** DSpider has quit IRC | 22:32 | |
| fungi | we write from executors | 22:34 |
| ianw | yeah, refreshed my memory ... i'm thinking of just the wheel jobs :) | 22:36 |
| ianw | some patches flew by a while ago for the kafs client tools debian package, before that i think it was rpm only | 22:41 |
| ianw | looks like that's still in progress across a few github trees | 22:43 |
| ianw | thread starts http://lists.infradead.org/pipermail/linux-afs/2020-February/003611.html | 22:43 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove some unnecessary debug: statements https://review.opendev.org/717874 | 22:48 |
| *** tosky has quit IRC | 23:28 | |
| *** hashar has quit IRC | 23:32 | |
| openstackgerrit | Merged opendev/infra-specs master: POC Xwiki farm at wiki.opendev.org https://review.opendev.org/710057 | 23:53 |
| clarkb | meeting agenda is out. I sent it to service-discuss then sent a pointer to that thread to openstack-infra@lists.o.o | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!