| *** mlavalle has quit IRC | 00:00 | |
| *** factor has quit IRC | 00:19 | |
| *** factor has joined #opendev | 00:20 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Switch linaro-us cloud to ipv6 only https://review.opendev.org/716112 | 00:31 |
|---|---|---|
| *** kevinz has joined #opendev | 00:34 | |
| ianw | clarkb / kevinz : ^ i think that's right. their private ipv4 address should still be doing nat to the outside world? | 00:35 |
| ianw | it's just they won't be assigned a floating ip? | 00:36 |
| clarkb | ya thats usually how it works | 00:36 |
| kevinz | sounds good | 00:37 |
| clarkb | the nat_destination value differs from OE | 00:38 |
| clarkb | I think we still want that ti be true but then disable FIPs by default? | 00:38 |
| ianw | i guess it could still have a floating ip, just we don't want one | 00:40 |
| ianw | clarkb: so you think it should be true? | 00:40 |
| clarkb | ya | 00:43 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Switch linaro-us cloud to ipv6 only https://review.opendev.org/716112 | 00:44 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove inventory references to <static|files>.openstack.org https://review.opendev.org/716116 | 01:24 |
| ianw | kevinz: i'll approve the ipv6 change so we can test it soon | 01:36 |
| *** ysandeep|away is now known as ysandeep|rover | 01:55 | |
| openstackgerrit | Merged opendev/system-config master: Switch linaro-us cloud to ipv6 only https://review.opendev.org/716112 | 01:58 |
| *** dangtrinhnt has joined #opendev | 02:03 | |
| *** diablo_rojo has quit IRC | 02:24 | |
| *** diablo_rojo has joined #opendev | 02:24 | |
| kevinz | ianw: Thanks | 02:27 |
| ianw | kevinz: looks like it's deployed ... let's see if nodes deploy | 02:54 |
| kevinz | ianw: OK, will check | 02:55 |
| ianw | one should be building ... | 02:55 |
| ianw | 2c701ac8-2fb3-4414-82c6-49b93643ea83 | 02:56 |
| *** dangtrinhnt has quit IRC | 03:03 | |
| ianw | well it's running it would seem ... http://zuul.openstack.org/stream/efdb93a189194b6b9df912d07c59e726?logfile=console.log | 03:06 |
| ianw | hopefully it's that easy! :) | 03:06 |
| kevinz | ianw: I tested it can pingable, hope it can works :-) | 03:12 |
| kevinz | Thanks | 03:12 |
| ianw | kevinz: sorry, just to confirm, no point in keeping linaro-london any more? it's not going to come back in some way? | 03:19 |
| kevinz | ianw: it may be come back, if Linaro US don't have enough resources in the future | 03:20 |
| kevinz | ianw: I guess we can still keep it for a longer time? | 03:20 |
| ianw | yes, let's then ... better than merge conflicts trying to add/remove it | 03:20 |
| kevinz | ianw: +1 | 03:21 |
| *** dangtrinhnt has joined #opendev | 03:24 | |
| *** dangtrinhnt has quit IRC | 03:48 | |
| *** dangtrinhnt has joined #opendev | 03:48 | |
| *** dangtrinhnt_ has joined #opendev | 04:21 | |
| *** dangtrinhnt has quit IRC | 04:25 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add Fedora 31 builds to nb04 https://review.opendev.org/716127 | 04:31 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: nodepool: use job inheritance https://review.opendev.org/713158 | 04:38 |
| *** ykarel|away is now known as ykarel | 04:45 | |
| *** diablo_rojo has quit IRC | 04:53 | |
| openstackgerrit | Merged openstack/diskimage-builder master: run_functests: handle build without tar https://review.opendev.org/715098 | 05:03 |
| openstackgerrit | Merged openstack/diskimage-builder master: centos 8 image build: fix mirror https://review.opendev.org/714836 | 05:03 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/716159 | 06:08 |
| *** dpawlik has joined #opendev | 06:20 | |
| *** dpawlik has quit IRC | 06:28 | |
| *** dpawlik has joined #opendev | 06:47 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 06:47 |
| *** dangtrinhnt_ has quit IRC | 06:50 | |
| *** dangtrinhnt has joined #opendev | 06:50 | |
| AJaeger | infra-root, looks like the upstream import broke for new repo creation. | 06:57 |
| AJaeger | Compare https://github.com/keuko/xstatic-dagre and https://opendev.org/openstack/xstatic-dagre - and gerrit does not list the new repo, check https://review.opendev.org/#/admin/projects/?filter=xstatic-d | 06:58 |
| *** rpittau|afk is now known as rpittau | 07:17 | |
| *** tosky has joined #opendev | 07:29 | |
| *** diablo_rojo has joined #opendev | 07:32 | |
| openstackgerrit | Merged opendev/system-config master: Remove files from letsencrypt group https://review.opendev.org/715185 | 07:39 |
| *** ysandeep|rover is now known as ysandeep|rover|l | 07:47 | |
| *** DSpider has joined #opendev | 07:56 | |
| *** ralonsoh has joined #opendev | 08:02 | |
| *** ysandeep|rover|l is now known as ysandeep|rover | 08:34 | |
| zbr | fungi, clark: creation of pre-commit-hooks is not cloning of pre-commit tool, is just a collection of small hooks that is usually used with pre-commit, is not the tool. | 08:35 |
| *** lpetrut has joined #opendev | 08:53 | |
| openstackgerrit | Merged opendev/irc-meetings master: Update FC SIG Meeting https://review.opendev.org/716098 | 09:09 |
| *** hashar has joined #opendev | 09:29 | |
| *** diablo_rojo has quit IRC | 09:37 | |
| *** dangtrinhnt_ has joined #opendev | 09:59 | |
| *** dangtrinhnt has quit IRC | 10:03 | |
| openstackgerrit | Rajesh Tailor proposed openstack/project-config master: Add new project and repository for tripleo-compute-extras https://review.opendev.org/715734 | 10:08 |
| *** ysandeep|rover is now known as ysandeep|afk | 10:09 | |
| *** ykarel is now known as ykarel|lunch | 10:11 | |
| *** dangtrinhnt_ has quit IRC | 10:16 | |
| *** dangtrinhnt has joined #opendev | 10:17 | |
| *** dangtrinhnt has quit IRC | 10:22 | |
| *** dangtrinhnt has joined #opendev | 10:29 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/diskimage-builder master: Validate virtualenv and pip https://review.opendev.org/707104 | 10:37 |
| *** ysandeep|afk is now known as ysandeep|rover | 10:39 | |
| *** ykarel|lunch is now known as ykarel | 10:52 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 10:57 |
| *** rpittau is now known as rpittau|bbl | 11:05 | |
| *** hashar has quit IRC | 11:43 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Generalize parse tox output https://review.opendev.org/716263 | 11:45 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments https://review.opendev.org/716264 | 11:45 |
| *** ralonsoh has quit IRC | 11:59 | |
| *** ralonsoh has joined #opendev | 12:00 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 12:08 |
| *** rpittau|bbl is now known as rpittau | 12:15 | |
| *** hashar has joined #opendev | 12:16 | |
| *** hashar has quit IRC | 12:28 | |
| *** hashar has joined #opendev | 12:29 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 13:11 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 13:14 |
| *** ykarel is now known as ykarel|mtg | 13:22 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 13:27 |
| *** ykarel|mtg is now known as ykarel | 13:35 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Control log archive and user preservation with vars https://review.opendev.org/701381 | 13:45 |
| *** lpetrut has quit IRC | 14:01 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 14:04 |
| *** lpetrut has joined #opendev | 14:08 | |
| *** dangtrinhnt has quit IRC | 14:22 | |
| *** dangtrinhnt has joined #opendev | 14:22 | |
| *** dangtrinhnt has quit IRC | 14:28 | |
| *** slittle1 has joined #opendev | 14:35 | |
| slittle1 | We are having issues wit the newly created starlingx/rook-ceph | 14:35 |
| slittle1 | it doesn't look like group starlingx-rook-ceph-core was created | 14:36 |
| slittle1 | Also the first step is usually for me to set up a .gitreview and .zuul.yaml for the new repo, but it's not finding/accepting my credentials from my ~/.gitconfig, and prompting for a user/pass | 14:37 |
| fungi | AJaeger: agreed, it looks like the README.md may be something gerrit adds when it initializes a reopsitory, and i guess we expect jeepyb to push --force initial content over top of that | 14:37 |
| fungi | slittle1: yes, it doesn't seem like starlingx/rook-ceph got created at all in gerrit, for that matter | 14:38 |
| slittle1 | ok, don't know what a jeepyb ... glad your on it. | 14:38 |
| mordred | fungi, slittle1 I'm lokoing now | 14:38 |
| *** ysandeep|rover is now known as ysandeep|away | 14:39 | |
| fungi | slittle1: it's our gerrit integration/automation library. we've been in the midst of a switch to how we're deploying and managing gerrit since friday and trying to iron out inconsistencies with how the project creation scripting used to work before containers vs how it's working in a docker contanier now | 14:41 |
| *** ykarel is now known as ykarel|away | 14:41 | |
| slittle1 | ah, going containerized are we? That does tend to throw a few curve balls. | 14:42 |
| fungi | looks like https://review.opendev.org/713650 merged to create starlingx/rook-ceph yesterday at 21:13:56z so i'll take a look around that timeframe in the project creation log | 14:43 |
| fungi | slittle1: yeah, we're thinking this is going to give us the ability to make more rapid progress on gerrit upgrades going forward, but was a big hurdle to clear | 14:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Set python3 for review.o.o https://review.opendev.org/716317 | 14:44 |
| mordred | fungi: ^^ | 14:45 |
| openstackgerrit | sebastian marcet proposed opendev/system-config master: OpenstackId production deployment 3.0.8 https://review.opendev.org/715655 | 14:45 |
| fungi | 2020-03-31 14:44:01,971: jeepyb.utils - INFO - Executing command: git --git-dir=/opt/lib/jeepyb/starlingx/rook-ceph/.git --work-tree=/opt/lib/jeepyb/starlingx/rook-ceph add .gitreview | 14:45 |
| mordred | fungi: I ran manage-projects playbook by hand to see what was going on - noticed the python3 thing - fixed it - and it worked | 14:45 |
| fungi | mordred: oh wow | 14:45 |
| *** sgw has joined #opendev | 14:46 | |
| mordred | fungi: I'm very confused as to how anything worked before now | 14:46 |
| mordred | but - the recent change was adding the clone directly to the playbook, so maybe there's a different var that got picked up elsewhere | 14:46 |
| mordred | who knows | 14:46 |
| clarkb | what in our ansible is python3 specific? | 14:47 |
| mordred | nothing. but there's no python2 on review.o.o | 14:47 |
| mordred | fatal: [review01.openstack.org]: FAILED! => {"changed": false, "module_stderr": "/bin/sh: 1: /usr/bin/python: not found\n", "module_stdout": "", "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127} | 14:47 |
| fungi | well, not in the container anyway | 14:47 |
| mordred | maybe unattended-upgrades caused python2 to go away recently - or something else we're installing or something? | 14:48 |
| mordred | well - these commands wouldn't be running in the container | 14:48 |
| mordred | that as an error for the ansible clone of project-config | 14:48 |
| fungi | fungi@review01:~$ /usr/bin/python --version | 14:48 |
| fungi | Python 2.7.12 | 14:48 |
| mordred | I agree | 14:48 |
| mordred | wtf? | 14:48 |
| fungi | also i don't think this explains why gerrit isn't pushing content into repos? | 14:49 |
| fungi | even ones with no upstream to clone | 14:49 |
| fungi | er, why jeepyb isn't pushing content into gerrit repos | 14:49 |
| clarkb | fungi: well if it never ran on gerrot to create the repo but did run on gitea you'd get that iirc | 14:49 |
| mordred | yeah | 14:49 |
| fungi | oh, i see, the README.md is a gitea default | 14:50 |
| fungi | and we were failing to create these repos in gerrit entirely | 14:50 |
| mordred | and the recent run with python3 set did create and push rook-ceph | 14:50 |
| mordred | yup | 14:50 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Remove lint skip in tox iniline comment parser https://review.opendev.org/716321 | 14:50 |
| mordred | I think the gerrit tasks were skipped completely | 14:50 |
| fungi | yeah, https://review.opendev.org/#/admin/projects/starlingx/rook-ceph exists now | 14:50 |
| mordred | because there was no log entry for rook-ceph in manage-projects.log before I did the manual python3 run | 14:50 |
| fungi | i concur | 14:51 |
| clarkb | did that fix the repo AJaeger pointed out too? | 14:51 |
| mordred | I reverted the manual setting of ansible_python_interpreter | 14:51 |
| clarkb | xstatic/dagre | 14:51 |
| mordred | and the ansible playbook error is consistent | 14:51 |
| fungi | in fact there were no entries at all logged in manage_projects.log between 2020-03-30 20:34:51 and 2020-03-31 14:43:36 | 14:51 |
| mordred | yes | 14:51 |
| clarkb | - not / | 14:52 |
| fungi | sp maybe let's see what we merged around 20:34:51z | 14:52 |
| mordred | fungi: so - I think the addition of the git clone command | 14:52 |
| mordred | OH!!!!!!!!! | 14:52 |
| mordred | hang on - I see the real bug | 14:52 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Don't run the review clone on connection local https://review.opendev.org/716329 | 14:54 |
| mordred | clarkb, fungi : please to insta-merge taht ^^ | 14:54 |
| mordred | it can't find /usr/bin/python in that play because there is no /usr/bin/python on bridge :) | 14:55 |
| *** dangtrinhnt has joined #opendev | 14:55 | |
| fungi | mordred: did you mean review:review-dev:!disabled there? | 14:56 |
| mordred | fungi: aroo? | 14:57 |
| fungi | oh, i get it now | 14:57 |
| slittle1 | how will we know when things are fixed ? | 15:08 |
| *** dangtrinhnt has quit IRC | 15:08 | |
| clarkb | slittle1: I think they are for your specific case | 15:09 |
| clarkb | (from mordred's explicit run) | 15:09 |
| *** dangtrinhnt has joined #opendev | 15:09 | |
| *** dangtrinhnt has quit IRC | 15:09 | |
| *** dangtrinhnt has joined #opendev | 15:09 | |
| fungi | yes, should be all set now, just need to know what initial member to add to the new groups from your acls | 15:10 |
| mordred | yes - should be fixed - and once that patch lands we should be re-fixed for the general case again | 15:12 |
| *** diablo_rojo has joined #opendev | 15:15 | |
| clarkb | mordred: https://review.opendev.org/#/c/715932/1 is being approved as another thing to test it | 15:15 |
| AJaeger | thanks for fixing! Could you double check all the yesterday newly created repos, please? | 15:15 |
| AJaeger | clarkb: want to have 716329 merged first? | 15:16 |
| clarkb | AJaeger: I think its fine to wait? since it will just fail until fixed? | 15:17 |
| clarkb | er to not wait | 15:17 |
| clarkb | mordred: ^ please -W if that isn't the case | 15:17 |
| AJaeger | clarkb: I see | 15:17 |
| slittle1 | no members in starlingx-nginx-ingress-controller-armada-app-core ... don't you usually add the request author is the initial member? Or do we request that here ? | 15:20 |
| mordred | AJaeger: actually - let's hold that real quick | 15:21 |
| mordred | I'm confident in this fix - but we've got a few more patches to land to move manage-projects forward and it would be nice to have project-config patches ready to go to test those after we do | 15:21 |
| clarkb | gotcha | 15:22 |
| clarkb | slittle1: typically thats a separate request | 15:22 |
| clarkb | slittle1: the automated tooling doesn't do it | 15:22 |
| clarkb | slittle1: who should be in the rook-ceph group and the ingress controller group as first seed member? | 15:22 |
| slittle1 | guess it'l be me for now | 15:28 |
| slittle1 | I'll track down the proper cores | 15:29 |
| clarkb | slittle1: done! | 15:29 |
| slittle1 | and will the infrastructure be setting up the .gitreview for us in future? I see one has been created, but that it was automatically done for us previously | 15:31 |
| fungi | slittle1: it should be there now | 15:32 |
| clarkb | slittle1: if you start with an empty repo we add the .gitreview but if there is a seed repo we don't | 15:32 |
| fungi | slittle1: https://opendev.org/starlingx/rook-ceph/src/branch/master/.gitreview | 15:33 |
| slittle1 | ok, good to know | 15:36 |
| corvus | mordred: so about that registry swift object thing -- afaict, we uploaded the parts without error, but one of them is short. | 15:50 |
| corvus | mordred: how should we debug this further? | 15:50 |
| *** diablo_rojo has quit IRC | 15:50 | |
| corvus | mordred: one idea might be to perform some HEAD requests after the upload, and then the copy to see if the sizes are as expected? | 15:51 |
| corvus | mordred: also, am i right in understanding that the Etag and X-Object-Meta-X-Sdk-Md5 should always be equal? | 15:51 |
| corvus | (they are, in fact, not equal on the segment that is short) | 15:52 |
| mordred | corvus: yeah - I would expect them to be equal | 15:55 |
| mordred | corvus: etag is going to be calculated by swift on the object it receives - meta-x-sdk-md5 is the md5 we calculate on the data we're uploading | 15:56 |
| *** dangtrinhnt has quit IRC | 15:56 | |
| corvus | so that would be a good way to find out when things go wrong in the copy phase | 15:56 |
| corvus | er | 15:56 |
| corvus | so that would be a good way to find out if things are going wrong in the copy phase | 15:56 |
| mordred | yeah | 15:56 |
| mordred | corvus: I'm pulling straws out of thin-air here - but is it possible there's an issue with binary/text and that maybe something is trying to treat binary data as text and mistakenly reencoding something? | 15:57 |
| mordred | but yeah - maybe starting with seeing what we can isloate phase-wise | 15:57 |
| mordred | corvus: is this reproducible? | 15:57 |
| corvus | mordred: no | 15:57 |
| mordred | blast | 15:57 |
| corvus | i think it happens every few weeks | 15:57 |
| corvus | mordred: this utilizes the sdk "do whatever you need to do to upload an image from a file" path | 15:58 |
| corvus | mordred: so there's a lot of black magic in that method | 15:58 |
| clarkb | zbr: I'm not sure opendev should be carrying linter tools either fwiw | 15:58 |
| *** dangtrinhnt has joined #opendev | 15:58 | |
| mordred | corvus: well - maybe adding some heads in places would let us know if we're getting corruption in upload or in copy | 16:00 |
| zbr | clarkb: that is an interesting question, mainly my proposal was because at some point in the part infra raised the issue of cloning linters from outside as a scalability issue. | 16:01 |
| corvus | mordred: i think the encoding issue is unlikely (or if it exists, that would be in the upload method). the registry server writes the data to disk, and sdk logs the sha256sum before uploading -- that sha256sum is correct for the full data. | 16:01 |
| mordred | corvus: incidentally - we also now support taking data as input without needing it to be in a file first | 16:01 |
| mordred | corvus: although I agree - I doubt that's the issue | 16:01 |
| zbr | and i am sure someone mentioned making mirrors of these repos would be a possible option (long time ago) | 16:02 |
| mordred | (the data as input won't work with autosplitting though - which I think you're using) | 16:02 |
| mordred | corvus: actually - don't listen to me - I'm thinking about glance images ... | 16:02 |
| corvus | mordred: my gut says that either 1) sdk is masking an upload error (maybe a bug in retry logic or something to do with the autosplitting). 2) swift is failing to store data. 3) swift munges the data on the copy | 16:02 |
| mordred | corvus: so - in general - I think what you're doing should be fine - especially if the sha256 sdk is logging is correct | 16:03 |
| mordred | corvus: yes. I think those are the three most likely things | 16:03 |
| corvus | mordred: okay, so i'll try to add some heads in after the upload and the copy | 16:03 |
| mordred | cool | 16:03 |
| *** dangtrinhnt has quit IRC | 16:05 | |
| slittle1 | please add me as core to the new group starlingx-cert-manager-armada-app-core as well | 16:05 |
| clarkb | mordred: do we need to set an allowed list of projects for infra-prod-playbook based jobs? | 16:06 |
| *** rpittau is now known as rpittau|afk | 16:07 | |
| clarkb | we don't have an external locking mechanism and probably don't want multiple cases of that running at once so being careful with where we apply it seems important? | 16:07 |
| mordred | yeah - I think we only want to run it on system-config and project-config | 16:07 |
| mordred | how about I add those real quick | 16:07 |
| clarkb | I've +2'd the two changes needed to actually start doing things, I'll let you decide if the restrictions should be in followups or new patchsets | 16:08 |
| clarkb | (I think either way is safe enough at this point) | 16:08 |
| mordred | clarkb: yeah - I think it's safe as well | 16:09 |
| mordred | I'm +Aing them | 16:09 |
| mordred | I'll land the project add once they land and watch things | 16:09 |
| clarkb | mordred: we could also add a file based lock system to the base job playbook. use a lockfilename based on playbook name | 16:10 |
| clarkb | that might eb good belts and suspenders to avoid overlapping runs of the same playbook | 16:10 |
| clarkb | (let me know if I should help with any of that) | 16:11 |
| slittle1 | please add me as core to the new group starlingx-cert-manager-armada-app-core as well | 16:13 |
| slittle1 | again, I'll track down the proper starlingX cores | 16:13 |
| clarkb | slittle1: done | 16:14 |
| mordred | clarkb: I'm not sure we need to - these are running in promote which is already supercedent - so we should only ever be running one copy of a given job | 16:16 |
| clarkb | mordred: but we may trigger them from jeepyb, system-config, or project-config which have different queues? or are you thinking only trigger from project-config and avoid the problem entirely? | 16:17 |
| mordred | clarkb: oh - is the supercedent deduplication per queue? hrm. | 16:18 |
| mordred | clarkb: I think in that case maybe we should put a semaphore on the job itself? | 16:18 |
| clarkb | mordred: yes its per queue iirc. Do the semaphores cross queues too? | 16:18 |
| clarkb | s/too// | 16:18 |
| mordred | good question | 16:19 |
| mordred | corvus: ^^ ? | 16:19 |
| *** mlavalle has joined #opendev | 16:20 | |
| mordred | clarkb: the docs say a semaphore is unique to a tenant and defines the max number of jobs that can use the semaphore at the same time | 16:21 |
| mordred | clarkb: so I'd take that to mean it doesn't matter what queue the job is in - if we defined a semaphore of 1, only one thing woudl be able to use it | 16:22 |
| clarkb | ++ | 16:22 |
| fungi | yeah, i think that should do what we need | 16:22 |
| fungi | it might result in some unnecessary rerunning of the job, but it's idempotent | 16:23 |
| mordred | yeah | 16:23 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add some belts and suspenders to the manage-projects job https://review.opendev.org/716359 | 16:24 |
| corvus | mordred: yep, semaphore works here. supercedent alone won't because the uniquifier is pipeline+project+branch | 16:24 |
| mordred | corvus: nod | 16:24 |
| *** lpetrut has quit IRC | 16:24 | |
| mordred | so we really only need to semaphore infra-prod jobs that we expect to also have a non-system-config trigger, like manage-projects | 16:24 |
| fungi | yeah, we might also need to if we started branching system-config, but that's unlikely | 16:27 |
| openstackgerrit | Merged zuul/zuul-jobs master: install-docker: allow removal of conflicting packages https://review.opendev.org/702304 | 16:38 |
| openstackgerrit | Merged opendev/system-config master: Don't run the review clone on connection local https://review.opendev.org/716329 | 16:40 |
| openstackgerrit | Merged opendev/system-config master: Add job to run manage-projects in zuul https://review.opendev.org/715944 | 16:40 |
| mordred | AJaeger: once https://review.opendev.org/#/c/715945/ lands we'll be trying for the first time having zuul run manage-projects based on landing a patch in project-config :) | 16:44 |
| mordred | (no logs in zuul yet - one step at at time here) | 16:44 |
| openstackgerrit | Merged openstack/project-config master: Run manage-projects on gerrit related changes https://review.opendev.org/715945 | 16:56 |
| mordred | woot! | 17:03 |
| mordred | ok. I'm gonna land that change | 17:03 |
| mordred | corvus, clarkb, fungi, AJaeger: https://review.opendev.org/#/c/715932/ is going in to gate- will be first change to run manage-projects triggered by zuul if all goes well | 17:04 |
| mordred | mnaser: ^^ | 17:04 |
| mnaser | mordred: neat | 17:14 |
| openstackgerrit | Merged openstack/project-config master: Add ability to push signed tags to tripleo-ipa https://review.opendev.org/715932 | 17:16 |
| mordred | https://zuul.opendev.org/t/openstack/stream/e699708ff9c947a5a14bbfe65f67cc44?logfile=console.log | 17:16 |
| mordred | promote job | 17:16 |
| mordred | the playbook is running on bridge | 17:18 |
| mordred | ZOMG IT ALL WORKED | 17:18 |
| mordred | infra-root: \o/ !!! | 17:18 |
| clarkb | success | 17:19 |
| mordred | ok. so the next steps are switching output to be to stdout and capturing the bridge log file | 17:20 |
| mordred | (which, btw, was written and had what we expected) | 17:20 |
| clarkb | mordred: nothing extra either I take it | 17:20 |
| mordred | nopd | 17:20 |
| clarkb | mordred: the belts and suspenders change lgtm | 17:22 |
| mordred | cool. also - this should be good to go now: https://review.opendev.org/#/c/715964/6 | 17:23 |
| mordred | clarkb: if we don't watch out - this is about to actually be a thing! | 17:23 |
| mordred | with that - we should see stdout lines in /var/log/ansible/manage-projects.yaml.log on bridge next time we land a project-config update | 17:24 |
| clarkb | mordred: thats gonna put manage-projects output on stdout rather than into /var/log/manage-projects on review.o.o? | 17:24 |
| mordred | that's right | 17:24 |
| clarkb | k making sure I understand all the levels of nesting here :0 | 17:24 |
| clarkb | er :) | 17:24 |
| mordred | yeah - it'll be simpler once it's all done - but definitely a few layers to get there :) | 17:25 |
| corvus | mordred: cheers! here, have a golden globe award in recognition of your work :) | 17:30 |
| mordred | clarkb: once https://review.opendev.org/#/c/715964/ lands, I think we can land https://review.opendev.org/#/c/716159/ as a test case | 17:36 |
| clarkb | k | 17:37 |
| openstackgerrit | Merged opendev/system-config master: OpenstackId production deployment 3.0.8 https://review.opendev.org/715655 | 17:40 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Change manage-projects default args https://review.opendev.org/716372 | 17:43 |
| mnaser | mordred, clarkb: would it be okay to get a config-project that loads jobs/secrets only in the vexxhost tenant? | 17:49 |
| clarkb | I think so | 17:49 |
| mordred | mnaser: you don't need it for your use case | 17:50 |
| clarkb | you may need to limit what we load out of the repo | 17:50 |
| clarkb | (to avoid too much trust) | 17:50 |
| clarkb | but ya I don't think you need to for your use case | 17:50 |
| mordred | (I don't think it's a problem to make one - but you don't need it for the image thing you're talking about) | 17:50 |
| mnaser | oh because i just need a base job with allowed-projects | 17:51 |
| mordred | yup | 17:51 |
| mnaser | so really what i need is vexxhost/base-jobs | 17:51 |
| mordred | yeah | 17:51 |
| mordred | that's exactly what you need | 17:51 |
| mordred | (and a patch to amke that will give me more test patches for the manage-project updates :) ) | 17:51 |
| clarkb | actually we probably couldn't load jobs or pipelines etc. It would have to be just secrets? | 17:52 |
| clarkb | if we made it trusted | 17:52 |
| *** hashar is now known as hasharAway | 17:52 | |
| clarkb | and you dno't need a trusted repo for secrets so ya | 17:52 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: vexxhost: add vexxhost/base-jobs https://review.opendev.org/716375 | 17:53 |
| mnaser | clarkb, mordred ^ that's some testing for ya, thank you! | 17:54 |
| AJaeger | mordred: great, congratulations! \o/ | 17:54 |
| * mnaser goes and hides in tears as he has to go and write php.. | 17:55 | |
| mordred | AJaeger: couple of more patches and we'll even have logs in zuul for you | 17:55 |
| clarkb | mordred: I'm still not sure we'll hvae them in zuul | 17:55 |
| AJaeger | mordred: do I really want these? ;) | 17:55 |
| clarkb | or you mean from the log file? ya that would work | 17:55 |
| clarkb | (sorry I had the -v change paged in) | 17:55 |
| mordred | clarkb: yeah. not this patch - but the one after it | 17:56 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Collect logs on manage-projects runs https://review.opendev.org/716376 | 17:57 |
| mordred | clarkb: actually - that ^^ once we're happy that the logs on bridge are suitable | 17:58 |
| mordred | clarkb, AJaeger : xstatic-moment-timezone and xstatic-dagre-d3 didn't seem to import properly - but the rest of the xstatic's did | 18:12 |
| AJaeger | mordred: so, 3? imported, 2 failed? Why did they fail? | 18:14 |
| mordred | unsure. it looks like the "didn't get created properly in gerrit" symptom | 18:15 |
| mordred | so I've removed them from the jeepyb cache and we'll see if they get properly created next time (I don't see any errors related to them) | 18:16 |
| clarkb | I just lost home networking am debugging | 18:17 |
| clarkb | (sorry cant lool at the xstatic stuff right now) | 18:18 |
| clarkb | it almodt looks like my isp has stopped routing packets :/ | 18:18 |
| mordred | clarkb: "awesome" | 18:18 |
| clarkb | this message comes to you via google fi | 18:18 |
| *** osmanlicilegi has quit IRC | 18:19 | |
| mordred | clarkb: I've been considering getting a Netgear LB1120 and putting a Fi sim in it and using it as a backup WAN connection | 18:19 |
| openstackgerrit | Merged opendev/system-config master: Log manage-projects to stdout https://review.opendev.org/715964 | 18:20 |
| mordred | woot | 18:20 |
| fungi | i've got an interface configuration for plugging my wireless modem into my openbsd-based home firewall as a backup internet route | 18:20 |
| fungi | and i've streamed netflix et cetera over it without issue | 18:20 |
| mordred | yeah. I just connected my tv to my phone hotspot last time that happened - but I thought a more solid mechanism would be better | 18:21 |
| fungi | i wind up with broadband outages often enough here that it was worthwhile to set up a semi-permanent configuration | 18:22 |
| clarkb | reboot the router seems to have fixed it. not sure if bug in freebsd or if isp wanted to issue a new lease | 18:23 |
| clarkb | it should be noted that my isp is being acquired and transformed into a new isp so not entirely surprising to have networking glitches | 18:23 |
| mordred | clarkb: I love it when that happens | 18:24 |
| clarkb | everything on internal side was fine though so I doubt it was freebsd | 18:24 |
| clarkb | while I was on fi I noticed that network was basically unusable too | 18:27 |
| clarkb | I bet the mobile isps are dealing with bw issues way more than traditional ISPs | 18:28 |
| openstackgerrit | Merged openstack/project-config master: vexxhost: add vexxhost/base-jobs https://review.opendev.org/716375 | 18:36 |
| *** ralonsoh has quit IRC | 18:40 | |
| *** osmanlicilegi has joined #opendev | 18:41 | |
| fungi | 2020-03-31 14:43:37,791: jeepyb.utils - INFO - Executing command: git --git-dir=/opt/lib/jeepyb/openstack/xstatic-moment-timezone/.git --work-tree=/opt/lib/jeepyb/openstack/xstatic-moment-timezone remote add -f upstream https://github.com/keuko/xstatic-moment-timezone.git | 18:42 |
| fungi | it went on to remote update | 18:42 |
| fungi | and checkout master | 18:43 |
| fungi | it didn't push that though? | 18:44 |
| fungi | looks like it did the same again when manually re-run at 18:39z | 18:45 |
| clarkb | no errors from fsck or trying to push the repo? | 18:47 |
| clarkb | I beleive we added the fsck step after some repo (novnc?) was invalid | 18:47 |
| fungi | no errors recorded | 18:48 |
| clarkb | a single branch and a single tag on that upstream repo | 18:48 |
| fungi | the last step after checkout master was branch -D config | 18:49 |
| clarkb | fsck --full is clean too | 18:49 |
| fungi | and then it moved on to the next repo | 18:49 |
| fungi | at least according to what was logged | 18:50 |
| clarkb | that comes from process_acls finally block | 18:50 |
| clarkb | we only push the repo if "created" flag is set | 18:51 |
| clarkb | so we might need to go back to the first failure and we're just seeing it be noopy afterwards? | 18:51 |
| fungi | yeah, maybe we need to remove created from the cache file? | 18:53 |
| clarkb | oh sorry thats specific to github, whee do we do the gerrit push | 18:53 |
| clarkb | ah ya its pushed-to-gerrit flag in the cache | 18:54 |
| fungi | "pushed-to-gerrit": true | 18:56 |
| *** diablo_rojo has joined #opendev | 18:56 | |
| clarkb | if you look in utils:make_local_copy its a bit more complicated too | 18:56 |
| fungi | it's there for openstack/xstatic-moment-timezone in /opt/lib/jeepyb/project.cache | 18:56 |
| clarkb | because if the projcet is in gerrit's projcet list we clone from there not the upstream | 18:56 |
| clarkb | so ya I think we need to identify the original failure then maybe just force push out of band? | 18:56 |
| fungi | problem is the "original failure" looks the same in the log | 18:57 |
| clarkb | fungi: are we sure that wasn't a subsequent run too? | 18:57 |
| * fungi checks to see when the change merged | 18:57 | |
| clarkb | (I don't think jeepyb can automatically recover in the case where we've crated the project in gerrit but not pushed the content successfully) | 18:58 |
| fungi | if this was in the batch pushed late yesterday though, it didn't get processed until 14:43z | 18:58 |
| fungi | once the local connection bug was resolved | 18:58 |
| clarkb | could it have been caught by one of those issues but successfully created in gerrit? | 19:00 |
| fungi | maybe when mordred ran it with ansible python set to python3 but still using local connection? | 19:01 |
| fungi | but yeah, seems like we'll need to push --force those in | 19:02 |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "Revert "Extract pep8 messages for inline comments"" https://review.opendev.org/715727 | 19:05 |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove lint skip in tox iniline comment parser https://review.opendev.org/716321 | 19:05 |
| mordred | o/ | 19:08 |
| mordred | corvus, mnaser: ^^ might want to send an email to the openstack list letting people know that pep8 is going to start leaving inline comments :) | 19:10 |
| tosky | oh, nice | 19:13 |
| mordred | infra-root: https://review.opendev.org/#/c/709253/ - we had another test failure in instlling kubectl snap - so maybe we should land that | 19:24 |
| mnaser | mordred: it would be nice if there was a way to use the roles from zuul-jobs :P | 19:34 |
| mordred | mnaser: yeah - there is definitely a re-use that we haven't sorted out here yet :) | 19:35 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Remove upstream from xstatic-dagre https://review.opendev.org/716411 | 19:55 |
| mordred | clarkb, AJaeger: I triggered mnaser's latest project patch too early before (I forgot I had to wait for ansible to run to pick up the system-config patch first) | 19:56 |
| mordred | so that's a safe simple patch to test the manage-projects log collection | 19:56 |
| mordred | (this issue won't be there once we're triggering bridge in the next patch - but one thing at at time ...) | 19:57 |
| fungi | [2020-03-31 20:07:22.109] [ERROR] console - Running Etherpad on Node v6.16.0 is not supported. Please upgrade at least to Node 8.9.0 | 20:07 |
| fungi | that's why etherpad-dev.o.o isn't working | 20:07 |
| clarkb | fungi: there are node 8 and 10 packages available for xenial via nodesource iirc | 20:08 |
| clarkb | and It hink we already use nodesource | 20:08 |
| clarkb | (may just need to bump the version in puppet) | 20:08 |
| fungi | yeah, looking | 20:08 |
| mordred | clarkb: should we start working on / thinking about containers here? or too much work for now? | 20:08 |
| clarkb | mordred: it would probably be good to get the whole migration to LE and opendev.org done there too. I'm not sure what the best approach is. The installation is actually pretty simple so maybe diving into containers is worthwhile | 20:09 |
| clarkb | mordred: fungi though to start just getting etherpad-dev up with latest etherpad gives us the option to switch meetpad to etherpad-dev and test if that works better | 20:10 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: local-log-download : role with script to download all log files https://review.opendev.org/715756 | 20:10 |
| corvus | clarkb: ++ | 20:10 |
| corvus | none of that would be wasted work | 20:10 |
| corvus | it's a single config line change to switch etherpad servers for meetpad | 20:10 |
| mnaser | mordred, clarkb: not shooting myself in the foot here? https://review.opendev.org/716418 Add Docker base jobs | 20:11 |
| clarkb | ya and I think a single line of puppet to have it switch nodejs versions | 20:11 |
| clarkb | mnaser: I think that is correct. The pass-to-parent bits are the important ones for functionality | 20:13 |
| mnaser | cool, i'll land and then try to "revise" projects to use that parent without a secret | 20:14 |
| fungi | confirmed, https://deb.nodesource.com/node_10.x/dists/xenial/ exists | 20:14 |
| fungi | will give that a try | 20:14 |
| mordred | clarkb: yeah - let's do that to move that forward - and we can work up some containers in parallel | 20:14 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Upgrade etherpad-dev server to Node 10.x https://review.opendev.org/716419 | 20:15 |
| openstackgerrit | Merged openstack/project-config master: Remove upstream from xstatic-dagre https://review.opendev.org/716411 | 20:16 |
| ianw | corvus: i think it might work to reference "upload_results" @ https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml#L24 because it was registered by the upload-logs-swift role? however, is that considered an "api" of the upload roles (they'll all register their url in that?) | 20:17 |
| mordred | clarkb: ok - logging stdout worked and wound up in the bridge log and it's clean | 20:18 |
| mordred | HOWEVER - **MAN** is is not very readable because it's all in the stdout_lines thing | 20:19 |
| fungi | which is "the bridge log"? ansible.log? | 20:19 |
| mordred | fungi: /var/log/ansible/manage-projects.yaml.log | 20:20 |
| mordred | fungi: if you want to look at that to make sure you thnk that's safe to copy to zuul at the end of the job | 20:20 |
| fungi | thanks | 20:20 |
| mordred | then we can land https://review.opendev.org/#/c/716376/ (and https://review.opendev.org/#/c/716372/1) | 20:20 |
| mordred | we'll probably eventually want to make that less ugly - but it's better than nothing for now I suppose | 20:21 |
| clarkb | mordred: we could tee it into a log file too maybe? that way if its really hard to parse you can fallback to the server copy without the extra annotation | 20:22 |
| mordred | clarkb: yeah. or - we could jst have it log to a file on the server and copy that file back to bridge and then to zuul | 20:22 |
| fungi | it looks safe enough to include, but yeah not especially readable and so hard for me to really be sure what's in there (though i've been looking at the manage-projects log up to now and we don't seem to leak anything risky in it either) | 20:23 |
| mordred | ++ | 20:23 |
| mordred | let me see about making a change to pull a copy of the m-p log back directly | 20:23 |
| fungi | well, also the logs are overly verbose for the 99.9% no-op repos | 20:24 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: local-log-download : role with script to download all log files https://review.opendev.org/715756 | 20:25 |
| fungi | which even if made readable will still mean pinpointing the spot where it changed anything a (keyword search) needle in a haystack | 20:26 |
| clarkb | ^F $PROJECT_NAME | 20:28 |
| mordred | yeah - looking at it for a sec ... how about this ... | 20:28 |
| mordred | let's live with this for now | 20:28 |
| mordred | get everything transitioned out of run_all and into infra-prod jobs - then change up the callback plugin so that we get readable stdout/stderr from things | 20:29 |
| mordred | but do it holistically for everything | 20:29 |
| fungi | yep, that sounds fine to me | 20:31 |
| openstackgerrit | Merged opendev/system-config master: Correct jitsi-meet config file https://review.opendev.org/715572 | 20:32 |
| clarkb | mordred: that seems reasonable | 20:35 |
| mordred | clarkb, fungi: cool - in that case, https://review.opendev.org/#/c/716359/ https://review.opendev.org/#/c/716372/ and https://review.opendev.org/#/c/716376/ are all our friends and ready to land now I think | 20:36 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: Test local log download script generation https://review.opendev.org/716426 | 20:37 |
| clarkb | cool. I've reviewed the set. | 20:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run manage-projects/base/bridge on system-config changes https://review.opendev.org/715957 | 20:38 |
| mordred | woot | 20:39 |
| clarkb | I'm due for my every other day bike ride so should probably try and get out the door | 20:39 |
| mordred | clarkb: I updated that based on an ianw comment | 20:39 |
| mordred | ianw: ^^ you have a sec to re-review? | 20:39 |
| ianw | mordred: heh, well yeah i'm all for it but it seemed like a lot to debug at once! :) | 20:40 |
| clarkb | mordred: did you also need to remove infra-prod-bridge job? | 20:41 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: local-log-download : role with script to download all log files https://review.opendev.org/715756 | 20:41 |
| clarkb | oh service-bridge.yaml != bridge.yaml | 20:41 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: local-log-download : role with script to download all log files https://review.opendev.org/715756 | 20:41 |
| mordred | clarkb: yeah - terrible naming | 20:41 |
| mordred | ianw: what's to debug - it's all going to work perfectly ;) | 20:42 |
| clarkb | the other thing we should be careful of is job timeouts | 20:42 |
| clarkb | I think splitting it up as job per playbook helps avoid those issues though | 20:42 |
| mordred | yeah. | 20:42 |
| fungi | also zuul says 716419 to update etherpad-dev to node 10.x passes if anyone's interested | 20:43 |
| fungi | then we can hopefully see what newer etherpad looks like, at least | 20:43 |
| clarkb | fungi: done | 20:44 |
| fungi | i'm good on the manage-projects logging plan, but am starting to prep dinner and will review any remaining changes once done (but don't want for me if you're ready to approve some/all of them) | 20:45 |
| mordred | the minimal callback plugin is almost what we need | 20:45 |
| mordred | it's actually better for this usecase | 20:45 |
| mordred | but it doesn't print task titles | 20:45 |
| *** hasharAway has quit IRC | 20:46 | |
| mordred | but it prints stdout cleanly | 20:47 |
| *** hashar has joined #opendev | 20:48 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove inventory references to <static|files>.openstack.org https://review.opendev.org/716116 | 20:49 |
| ianw | infra-root: ^ modulo any more typos, that should be a pretty easy cleanup of even more bits of static/files | 20:49 |
| ianw | mordred: should we restore https://github.com/ansible/ansible/pull/45265 ... i got some cryptic comments on that with no further response, and there it has sat | 20:51 |
| ianw | i think this is the same noise you're talking about | 20:51 |
| ianw | ansible output noise | 20:51 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use ansible debug callback plugin https://review.opendev.org/716433 | 20:55 |
| mordred | infra-root: you can see what that looks like by looing in /root/manage-projects.yaml.log ^^ - I think it's infinitely more readable | 20:56 |
| mordred | ianw: looking | 20:57 |
| mordred | ianw: maybe - but we'd need to find the new location - profile_tasks isn't in devel and is likley in a collection somewhere now | 20:57 |
| fungi | yep, lgtm | 20:58 |
| mordred | ianw: and - no, the noise I was talking about is that the output we're getting is just putting stdout all on line line in the json ... the debug plugin patch there fixes it | 21:01 |
| mordred | ianw: if you have a sec, https://review.opendev.org/#/c/716433/ https://review.opendev.org/#/c/716359/ https://review.opendev.org/#/c/716433/ and https://review.opendev.org/#/c/716376/ - and then I can push up another 1 line project-config cleanup and we can see the whole thing | 21:03 |
| ianw | looking | 21:05 |
| openstackgerrit | Merged opendev/system-config master: Upgrade etherpad-dev server to Node 10.x https://review.opendev.org/716419 | 21:05 |
| clarkb | it started raining so I'm hiding inside now | 21:13 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: opensuse: switch to python 3.x for tumbleweed https://review.opendev.org/716437 | 21:19 |
| ianw | mordred: i'm still looking around, but per your 2.10 comment in https://review.opendev.org/#/c/716433/1/playbooks/bridge.yaml how come the -devel job still works? | 21:27 |
| *** hashar has quit IRC | 21:32 | |
| openstackgerrit | Merged opendev/system-config master: Add some belts and suspenders to the manage-projects job https://review.opendev.org/716359 | 21:34 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: WIP Dockerize etherpad https://review.opendev.org/716442 | 21:45 |
| mordred | clarkb: ^^ there's 75% of it I thnik | 21:46 |
| mordred | ianw: ansible will totally still work with a missing callback plugin - it just won't have the output format we want | 21:46 |
| mordred | ianw: so it's really a note that we might not notice if we update to 2.10 - but the output _might_ go back to being poopy. it also might very well not, because ACD is supposed to take care of backwards compat | 21:47 |
| openstackgerrit | Merged opendev/system-config master: Remove inventory references to <static|files>.openstack.org https://review.opendev.org/716116 | 21:47 |
| ianw | mordred: yeah, https://ad94df435a366c083a4e-893f9315e5179a0462ce52bc0ed28dd9.ssl.cf5.rackcdn.com/716433/1/check/system-config-run-base-ansible-devel/5000cd6/ looks right to me? | 21:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: WIP Dockerize etherpad https://review.opendev.org/716442 | 21:48 |
| mordred | ianw: yeah - or, at least it's good enough | 21:52 |
| ianw | mordred: i have no idea *how* the collection is getting installed, but it seems it is | 21:52 |
| mordred | ianw: I agree: if you search for "base-server : Install base packages" in the log_output.txt ... you'll see what the output looks like with the debug plugin | 21:56 |
| mordred | ianw: compared to https://zuul.opendev.org/t/openstack/build/6362917fbde6427d8e36cfcd724c9c0a/log/job-output.txt#2336 | 21:57 |
| mordred | which is without it | 21:57 |
| mordred | ianw: so I agree - debug seems to work on devel - which is a good sign | 21:57 |
| mordred | but we shodl still be careful when upgrading to 2.10 anyway :) | 21:57 |
| *** DSpider has quit IRC | 21:57 | |
| clarkb | I think the next run in about an hour will update nodejs on etherpad-dev | 21:58 |
| mordred | clarkb: cool | 21:58 |
| clarkb | and now sun is out and things are dry ish so I'm actually gonna get on the bike | 21:58 |
| clarkb | before the rain and hail return and make me chicken out | 21:58 |
| mordred | clarkb: I got dockerfiles and a good deal of ansible done for docker-etherpad ... but didn't get to apache or LE or setting up a db for testing | 21:59 |
| mordred | clarkb: have a good biking! | 21:59 |
| fungi | did stuff get merged for the manage-projects logging, or is there more? | 22:00 |
| mordred | fungi: relevant patches are https://review.opendev.org/#/c/716433/ https://review.opendev.org/#/c/716359/ https://review.opendev.org/#/c/716433/ and https://review.opendev.org/#/c/716376/ | 22:02 |
| mordred | fungi: rather, https://review.opendev.org/#/c/716433/ is really the one outstanding | 22:04 |
| mordred | fungi: and https://zuul.opendev.org/t/openstack/build/1923612b7e604d46b188a911743815a4/log/job-output.txt#3894 is output with debug callback vs. https://zuul.opendev.org/t/openstack/build/6362917fbde6427d8e36cfcd724c9c0a/log/job-output.txt#2336 without | 22:04 |
| mordred | (for the same content) | 22:05 |
| fungi | are we going to remember we need to revisit this in the 2.10 ansible upgrade? | 22:06 |
| fungi | i guess worst case we'll notice it when we go to try to debug a manage-projects job failure | 22:06 |
| mordred | fungi: that's why I put the comment in - because we won't upgrade without updating that value. however - it seems like it does work in 2.10 | 22:06 |
| mordred | but yeah | 22:06 |
| mordred | also - 2.10 is a REALLY BIG change, so we're gonna want to be careful with upgrading to it anyway | 22:07 |
| ianw | i'm just trying to determine if a) 2.10 automatically fixes this up -- if it does, it's rather obscure how it does -- or b) we are not testing what we think we're testing with that job | 22:07 |
| ianw | this is *exactly* why i wanted the devel job to be our canary for such things, but if it's not actually testing correctly, it's a bit of a dud | 22:08 |
| mordred | ianw: https://zuul.opendev.org/t/openstack/build/5000cd6e327640f8beb5feeccab498f3/console | 22:09 |
| mordred | it looks like it is install ansible | 22:10 |
| mordred | from ansible/ansible | 22:10 |
| ianw | yeah, and when i install locally from devel into a virtualenv, i don't get the posix callback somehow dragged in | 22:10 |
| mordred | ianw: I dunno :) | 22:13 |
| ianw | ERROR! Invalid callback for stdout specified: thisisnotaplugin | 22:15 |
| ianw | i would say it doesn't like missing plugins | 22:15 |
| fungi | thisisnotaspoon | 22:15 |
| ianw | nice ... it must be leaking to the local ansible install | 22:17 |
| ianw | 252704 stat("/usr/lib/python3.7/site-packages/ansible/plugins/callback/debug.py", {st_mode=S_IFREG|0644, st_size=1773, ...}) = 0 | 22:17 |
| ianw | that's running ansible devel from a virtualenv | 22:18 |
| mordred | *fascinating* | 22:18 |
| ianw | so this is mostly b) -devel job is not actually testing what we think it is | 22:19 |
| fungi | so it's getting ansible 2.9 (or whatever) plugins showing up in the 2.10 beta? | 22:19 |
| fungi | (from the system ansible) | 22:19 |
| ianw | i haven't confirmed directly on a ci host, but that's what appears to be happening locally | 22:20 |
| fungi | neat | 22:20 |
| fungi | so this could in theory be happening in other places where we test a different version of ansible than we're running from? | 22:22 |
| ianw | i uninstalled local ansible, and now it doesn't even run from a fresh virtualenv -> http://paste.openstack.org/show/791423/ | 22:24 |
| ianw | this could be something screwed up locally, i guess ... | 22:24 |
| ianw | ohhh hang on wrong path, my fault | 22:25 |
| ianw | ok i'm a dolt, i wasn't *running* ansible out of my virtualenv, but out of the source directory | 22:27 |
| ianw | now i see it fail "ERROR! Invalid callback for stdout specified: debug" | 22:27 |
| ianw | which *still* leads me to b) we're not testing what we think we're testing | 22:28 |
| mordred | yeah. | 22:29 |
| mordred | ianw: but I do not know / cannot see why | 22:31 |
| ianw | install-ansible: Install ARA ... i bet this is dragging 2.9.6 back in | 22:32 |
| ianw | Downloading http://mirror.us-east.openedge.opendev.org/pypifiles/packages/ae/b7/c717363f767f7af33d90af9458d5f1e0960db9c2393a6c221c2ce97ad1aa/ansible-2.9.6.tar.gz (14.2 MB) | 22:32 |
| ianw | yeah, i think that's it. so our -devel test is really a ara test :/ | 22:34 |
| mordred | ianw: wow | 22:35 |
| mordred | ianw: maybe we should just remove ara - since we don't use it in prod anyway | 22:36 |
| mordred | or - we could install ara then install ansible :) | 22:36 |
| fungi | and the more stuff we move to zuul, the more we can just use the zuul console breakdown anyway | 22:36 |
| mordred | ianw: the debug callback plugin moved to https://galaxy.ansible.com/ansible/posix | 22:40 |
| mordred | and it's in routing.yaml - so by the time the 2.10 Ansible Community Distro is built (which is what will be on pypi) - the debug plugin should still be accessible | 22:40 |
| ianw | yeah, i still think there's value in a -devel job, rather that just waiting to be surprised by releases | 22:43 |
| mordred | oh - totally | 22:43 |
| ianw | anyway, so that's asynchronous to your change, but i'm glad we figured out it's not working | 22:43 |
| mordred | mostly just followups about what things we can do about it and/or what we can expect around 2.10 time | 22:43 |
| mordred | ++ | 22:44 |
| ianw | ara is nice in these nested jobs for the fake-bridge runs | 22:44 |
| ianw | i feel like ara re-added their static generation? | 22:45 |
| mordred | maybe? but yeah - is is nice in those nested jobs | 22:48 |
| fungi | oh, good point, i guess the zuul console doesn't help with jobs like manage-projects where we proxy ansible through bridge | 22:50 |
| ianw | it seems like static generation isn't quite there yet https://github.com/ansible-community/ara-web/issues/11 | 22:53 |
| *** tosky has quit IRC | 22:54 | |
| ianw | ./test-env/bin/pip install ara does not seem to touch ansible, maybe it's somethign we're doing | 22:56 |
| mordred | ianw: uhm | 22:58 |
| mordred | we're not installing devel of ara either | 22:58 |
| ianw | yeah, we're at https://github.com/ansible-community/ara/blob/0.16.6/requirements.txt | 22:58 |
| mordred | ianw: we're installing opendev.org/recordsansible/ara | 22:58 |
| ianw | ... why does that link pop up a "authorization requested" login box? | 22:59 |
| mordred | oh - wait | 22:59 |
| mordred | nevermind. their github is a mirror of gerrit | 22:59 |
| mordred | so that's still fine | 22:59 |
| ianw | pip install ara===0.16.4 does pull in ansible 2.9 ... weird | 23:01 |
| ianw | pip bug? something to do with it being a dev release? | 23:03 |
| mordred | its how pip works | 23:08 |
| mordred | it's sequential - the dep solver hasn't landed yet | 23:09 |
| mordred | so we install ansilble - then we install ara which reinstalls ansible | 23:09 |
| ianw | but why doesn't it notice there's an already installed later version? | 23:09 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Move ara install before ansible https://review.opendev.org/716449 | 23:11 |
| fungi | maybe ansible's not well versioned when installed from source? | 23:11 |
| fungi | (from a package metadata perspective) | 23:11 |
| mordred | ianw: because pip doesn't work that way :) | 23:12 |
| mordred | at least - not until this summer | 23:12 |
| mordred | oh - wait | 23:12 |
| fungi | well, pip does avoid reinstalling packages if the installed version matches the version requested | 23:12 |
| mordred | I think I understand your question better | 23:12 |
| mordred | yeah | 23:12 |
| mordred | it should be version 2.10.0.dev0 from ansible/ansible | 23:13 |
| ianw | ahhhh | 23:13 |
| ianw | ansible-base 2.10.0.dev0 | 23:13 |
| ianw | they swapped the name | 23:14 |
| mordred | oh right | 23:14 |
| mordred | this is in a lead up to ansible-community-distribution | 23:14 |
| mordred | ansible _without_ the compat collections pulled in is going to be called ansible-base | 23:14 |
| fungi | that explains it | 23:14 |
| mordred | and it will include many fewer things | 23:14 |
| ianw | i guess if "ansible" is their mega-package then that will be ok | 23:14 |
| mordred | yup | 23:14 |
| mordred | getting from ansible-base -> ACD is still in progress | 23:15 |
| ianw | ok, so even moving the install after isn't really right; it's still going to pull in "ansible" | 23:15 |
| mordred | yup | 23:15 |
| *** xavinux has joined #opendev | 23:15 | |
| mordred | we could install ara, uninstall ansible, then install ansible from source | 23:16 |
| ianw | hrmmm | 23:16 |
| mordred | but honestly - I'm not sure that's worth it - because that's not goign to be testing a future state we're going to be using | 23:16 |
| mordred | it might be that ansible-devel is in enough of a state of flux right now that there is no valid way for us to build this job | 23:16 |
| mordred | and we need to wait until they have the mechanism to build ACD | 23:16 |
| mordred | so that we can install _that_ | 23:17 |
| mordred | (although also we might choose to switch to ansible-base + a set of collections) | 23:17 |
| ianw | right, it would be kind of nice if we could also even speculatvely test collections | 23:17 |
| ianw | like we can for ansible changes now | 23:18 |
| mordred | well, we can | 23:18 |
| mordred | the openstack collection is now fully functional | 23:18 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 23:18 |
| mordred | and if we wanted to we could start installing collections with 2.0 | 23:18 |
| mordred | 2.9 | 23:18 |
| mordred | but I think we might still want to wait for the dust to settle a little bit before we start touching them on bridge | 23:19 |
| openstackgerrit | Merged opendev/system-config master: Change manage-projects default args https://review.opendev.org/716372 | 23:19 |
| openstackgerrit | Merged opendev/system-config master: Collect logs on manage-projects runs https://review.opendev.org/716376 | 23:19 |
| ianw | i feel like it's really just ara bringing in ansible that stops this working | 23:19 |
| mordred | yah | 23:19 |
| mordred | we could remove ara from the devel job | 23:19 |
| mordred | because it's nice - but it's also not a thing we install on bridge | 23:19 |
| ianw | does requirements.txt handle "ansible<=2.9|ansible-base>2.10"? | 23:20 |
| fungi | i don't believe so | 23:21 |
| ianw | i don't think it has any way to express that | 23:21 |
| fungi | not to my knowledge | 23:21 |
| ianw | i forsee this causing a lot of people a lot of problems | 23:23 |
| fungi | you could create a stub package named "ansible" which installs nothing | 23:24 |
| fungi | just metadata | 23:24 |
| ianw | yeah, an eqivs ... that ... might just work :) | 23:27 |
| fungi | really it could just be an sdist with an egg-info file containing the package name and version number | 23:28 |
| fungi | pretty sure that's all pip really needs | 23:29 |
| clarkb | ok I'm back in front of a keyboard | 23:30 |
| clarkb | anything I should catch up on? | 23:30 |
| fungi | not much aside from ansible changing it's name ;) | 23:30 |
| fungi | er, its | 23:30 |
| mnaser | hmm... so i feel bad for all the help i'm asking but... i have https://review.opendev.org/#/c/716448/3/.zuul.yaml which contains build/upload/promote jobs.. i also have this project inside allowed projects (though just added)... https://opendev.org/vexxhost/base-jobs/src/branch/master/zuul.d/jobs.yaml yet zuul says the job is not allowed to run | 23:30 |
| *** xavinux has quit IRC | 23:30 | |
| mordred | mnaser: that seems weird to me ... corvus ^^ ? what am I missing? | 23:35 |
| mnaser | the only thing to explain it is like the change to add libvirtd_exporter to allowed was more "recent" | 23:35 |
| mnaser | but reconfigs dont take that long, i think. | 23:36 |
| mnaser | the other thing is that the fact that the parent job actually has secrets so maybe that 3 layer cake is making it unhappy | 23:36 |
| mordred | ianw, clarkb : to be clear - ansible has not changed the name of the thing you install from pypi - they have changed the name of the thing you get when you install from their git repo | 23:38 |
| mordred | but - you know - same thing for us in this case ;) | 23:38 |
| mordred | clarkb: notable tl;dr - our ansible-devel job is _not_ testing ansible-devel | 23:38 |
| clarkb | mordred: becaus of the split for collections ya? | 23:39 |
| mordred | yes. there will be a thing called ansible that is uploaded to pypi that contains ansible-base + collections | 23:39 |
| mordred | that thing is referred to colloquially as "ACD" for "ansible community distribution" - and does not yet exist but is in work | 23:40 |
| mordred | what we do have is ansible-base in ansible/ansible - and collections in galaxy and their respective git repos - and we can install collections with 2.9 if we want | 23:40 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job https://review.opendev.org/716452 | 23:41 |
| corvus | mnaser, mordred: see paragraph 2 of https://zuul-ci.org/docs/zuul/reference/job_def.html#attr-job.allowed-projects | 23:42 |
| mnaser | corvus: ahh, so it _does_ need to be a config project after all | 23:43 |
| corvus | mnaser, mordred: in other tenants, we could solve that by attaching the job to the project inside of a config-project (that's what we just did for mordred's cross-repo image building jobs); but we don't by policy have a vexxhost-controlled config-project in the vexxhost tenant | 23:43 |
| corvus | mnaser: yep | 23:43 |
| mordred | ah. poop. so we gave mnaser bad advice earlier today | 23:44 |
| mnaser | corvus: i was thinking of a config-project that only loaded secret/project/job and nothing more | 23:44 |
| clarkb | you just need to set the job to run in the config project right? | 23:44 |
| corvus | mnaser, mordred: we have also discussed adding a new type of project, call it a "tenant-config-project" which has, among other things, the ability to do this, but not run jobs in the secure conteext | 23:44 |
| clarkb | mnaser: I don't think project or job is safe | 23:44 |
| corvus | clarkb: yes but there is no config project. | 23:44 |
| mordred | yeah | 23:44 |
| ianw | infra-root: i mentioned this before, but now i'm seeing it on two browsers --> https://opendev.org/recordsansible/ara throws me a "Authentication Required" box? am i the only one? | 23:45 |
| corvus | mnaser: yes, the suggestion you just made is very similar. it's possible, but it's a non-trivial change, and just keeps getting pushed behind other work for me. | 23:45 |
| corvus | ianw: i see that | 23:45 |
| * mordred does not | 23:45 | |
| mordred | that loads for me | 23:45 |
| ianw | ok, at least i'm not going totally crazy! | 23:46 |
| clarkb | I see it too | 23:46 |
| corvus | https://opendev.org/recordsansible/doc/source/_static/ara-with-icon.png is the problem | 23:46 |
| clarkb | it happens in FF but not chrome for me | 23:46 |
| corvus | ff here | 23:46 |
| mordred | yeah - chrome does not throw the error | 23:46 |
| mordred | I see a broken image box | 23:46 |
| mnaser | corvus: right, but i meant if i added vexxhost/base-jobs under here https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L1526-L1532 ? | 23:47 |
| mnaser | sorry, i may be oblivious to some sort of obvious security concern | 23:47 |
| mnaser | i am getting the auth required too | 23:47 |
| mordred | mnaser: it's those jobs give you root on our executors | 23:47 |
| mnaser | let me help figuring otu why | 23:47 |
| corvus | mnaser: technically that would fix it, but yes, that would effitively give you root on all systems, so it's awkward :) | 23:47 |
| corvus | effectively even | 23:47 |
| mnaser | ok, i see | 23:48 |
| mordred | yeah. this is why having a construct like the tenant-config-project corvus just mentioned would be quite valiable | 23:48 |
| mordred | because the desired use-case here is obviously valuable | 23:48 |
| ianw | clarkb: i see the signin with crhome 80.0.3987.149 | 23:49 |
| mnaser | curl https://opendev.org/recordsansible/doc/source/_static/ara-with-icon.png returns 401 | 23:49 |
| clarkb | ianw: hrm I'm on old chrome | 23:49 |
| clarkb | I should proably look into why chrome isn't updating | 23:49 |
| mnaser | i mean, the "real" path is https://opendev.org/recordsansible/ara/raw/branch/master/doc/source/_static/ara-with-icon.png | 23:49 |
| ianw | let the yak shaving begin! | 23:50 |
| mnaser | why gitea has that behaviour, i dunno | 23:50 |
| mnaser | okay so finally i cant really do this without `tenant-config-project` so i'll need to continue to store dockerhub credentials as a secret in each repo for now | 23:51 |
| clarkb | it sounds like maybe we need to bump the priority of that zuul feature? | 23:51 |
| clarkb | I doubt I have time to work on that in the immediate future but maybe I can carve time next week to look into it | 23:51 |
| mnaser | yeah ... i'm deep into a bunch of other things right now, i can try helping but i can't promise much, plus this is still one of those parts of zuul that are harder for me to understand | 23:52 |
| clarkb | ya I'm reasonably familiar with the config bits of zuul. I'll pencil that in for next week | 23:52 |
| mordred | mnaser, clarkb : i have a suggestion | 23:55 |
| mordred | what if we make a config project (or turn vexxhost/base-jobs into one) - but make the core team on it project-config-core | 23:56 |
| clarkb | as an intermediate step? | 23:56 |
| mordred | mnaser is already project-config-core, so he should be able to approve the changes to it he needs | 23:56 |
| mordred | yeah | 23:56 |
| mordred | I think we can trust mnaser to not use that to root the system -s ince we already trust him to not root the normal systems :) | 23:56 |
| clarkb | I guess we just have ot make it clear that we can't add additional cores outside the opendev core | 23:56 |
| mordred | and once we grow tenant-config projects we change it to that | 23:56 |
| clarkb | but ya that would work | 23:56 |
| mordred | yes. stricly project-config-core on that repo | 23:57 |
| mordred | but without an expectation that mnaser needs to get another pcc to +2 something in there when what he's doing is stuff like this docker thing | 23:57 |
| mordred | (obviously if a job was added that was doing secure things on the executor I think we'd all ask the he ping us first, yes?) | 23:57 |
| corvus | clarkb: oh, i accidentally sent something to the wrong channel: i did enough work on this to know it's not a 2-hour programming job. it's not hugely complicated, but it would probably take me the better part of a week with few distractions. | 23:58 |
| * mordred needs to run - but will leave that suggestion here - and register his own personal support for it | 23:58 | |
| corvus | (that was in #zuul, not here) | 23:58 |
| clarkb | ah ok so it isn't as simple as modifying the config loading to have a subset of loading things | 23:59 |
| corvus | clarkb: nope, this isn't a loading problem (we can already do that) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
