| ianw | we could add a role to the base jobs to dump a config file | 00:02 |
|---|---|---|
| ianw | config file /home/zuul/.config/virtualenv/virtualenv.ini missing (change via env var VIRTUALENV_CONFIG_FILE) | 00:03 |
| clarkb | will virtualenv read a global config? | 00:03 |
| clarkb | ah ya that could work | 00:03 |
| mordred | ooh. I like the idea of putting things in to a config | 00:20 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: virtualenv-override : add download for virtualenv seeder https://review.opendev.org/715332 | 00:26 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: virtualenv-override : add download for virtualenv seeder https://review.opendev.org/715332 | 00:36 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: virtualenv-override: add to base pre playbook https://review.opendev.org/715333 | 00:36 |
| ianw | clarkb / fungi / mordred : ^ i think that is the series to add to base-test, then i can test it, then add to base | 00:36 |
| clarkb | ianw: it failed linting due to a proper error (I noted it inline if you don't want ot look at log files) | 00:52 |
| ianw | clarkb: argh, thanks | 00:55 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: virtualenv-config : add download for virtualenv seeder https://review.opendev.org/715332 | 00:55 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: virtualenv-config: add to base pre playbook https://review.opendev.org/715333 | 00:55 |
| ianw | i did run the role locally, but screwed up the name | 00:55 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Revert "upload-logs-swift: Create a download script" https://review.opendev.org/715325 | 00:57 |
| ianw | corvus: i still think the pypi issues are causing havoc, but i went and looked closely and only removed the parts of the unicode change that were related to the download script; that still seems generically useful to test | 00:58 |
| *** diablo_rojo has quit IRC | 01:01 | |
| *** diablo_rojo has joined #opendev | 01:02 | |
| *** diablo_rojo has quit IRC | 01:06 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/project-config master: Replace python-charm-jobs to py3 job https://review.opendev.org/714796 | 03:09 |
| openstackgerrit | Merged opendev/base-jobs master: virtualenv-config : add download for virtualenv seeder https://review.opendev.org/715332 | 03:40 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Add Fedora 31 support and test jobs https://review.opendev.org/708416 | 03:45 |
| ianw | AJaeger / frickler : as the next people to maybe see this, for your consideration -> https://review.opendev.org/715333 will fix a range of virtualenv problems due to it vendoring a bad version of setuptools (see https://review.opendev.org/#/c/715332/3 for links) | 04:53 |
| ianw | i've linked to a test run there too | 04:53 |
| ianw | if anyone turns up talking about script permissions that's likely it | 04:53 |
| openstackgerrit | Merged opendev/base-jobs master: virtualenv-config: add to base pre playbook https://review.opendev.org/715333 | 06:33 |
| *** ralonsoh has joined #opendev | 07:08 | |
| *** dpawlik has joined #opendev | 07:11 | |
| *** factor has joined #opendev | 07:31 | |
| *** ykarel has joined #opendev | 07:42 | |
| *** ysandeep|rover has joined #opendev | 07:43 | |
| *** lpetrut has joined #opendev | 08:06 | |
| *** elod has quit IRC | 08:13 | |
| *** rpittau|afk is now known as rpittau | 08:23 | |
| *** ykarel is now known as ykarel|lunch | 08:24 | |
| *** ysandeep|rover is now known as ysandeep|rover|l | 08:33 | |
| *** elod has joined #opendev | 08:57 | |
| *** ysandeep|rover|l is now known as ysandeep|rover | 08:59 | |
| *** ykarel|lunch is now known as ykarel | 09:00 | |
| *** tosky has joined #opendev | 09:29 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack/project-config master: Add Shrews to alumni https://review.opendev.org/715373 | 09:40 |
| *** roman_g has joined #opendev | 10:20 | |
| openstackgerrit | Thierry Carrez proposed opendev/infra-specs master: POC Xwiki farm at wiki.opendev.org https://review.opendev.org/710057 | 10:28 |
| ttx | (rebased on top of the jitsi-meet spec merge) | 10:29 |
| *** ysandeep|rover is now known as ysandeep|rov|brb | 10:41 | |
| AJaeger | infra-root, cliff and openstacksdk have been tagged for new release - thanks, ttx | 10:52 |
| *** ysandeep|rov|brb is now known as ysandeep|rover | 11:03 | |
| AJaeger | So, on zuul-jobs: py35 failure is fixed, py27 is failing differently now (stestr 3.0.0 problem?) | 11:11 |
| openstackgerrit | Merged opendev/irc-meetings master: Move Large Scale SIG meeting hour https://review.opendev.org/714922 | 11:12 |
| frickler | AJaeger: oh, I think mtreinish discussed that in -qa yesterday. 3.0.0 got published w/o proper python-require markers | 11:16 |
| frickler | so as a workaround we would have to add an explicit cap | 11:16 |
| frickler | until he does the same juggling that sdk just finished | 11:17 |
| mtreinish | frickler: https://github.com/mtreinish/stestr/issues/292 | 11:18 |
| mtreinish | I'm waiting for gmann to respond with the results of his test to see if requires-python still works, if not I'll push out a 3.0.1 with the metadata switched to python-requires | 11:19 |
| mtreinish | we probably should also upgrade pbr in that case too (since it uses requires-python) | 11:20 |
| *** rpittau is now known as rpittau|bbl | 11:32 | |
| openstackgerrit | Merged opendev/system-config master: Add IRC bots to OpenInfra Labs channels https://review.opendev.org/712588 | 11:58 |
| *** ysandeep|rover is now known as ysandeep|rov|mtg | 12:00 | |
| openstackgerrit | Merged opendev/system-config master: Build gerrit 2.14 as well https://review.opendev.org/715284 | 12:07 |
| *** elod_ has joined #opendev | 12:20 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Add opendev/pre-commit-hooks https://review.opendev.org/704411 | 12:21 |
| *** ysandeep|rov|mtg is now known as ysandeep|rover | 12:37 | |
| AJaeger | frickler: let me push a cap for now... | 12:38 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Cap stestr version https://review.opendev.org/715415 | 12:41 |
| *** hashar has joined #opendev | 12:44 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Cap stestr version https://review.opendev.org/715415 | 12:45 |
| *** elod has quit IRC | 12:53 | |
| *** elod_ is now known as elod | 12:53 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: install-kubernetes: fix missing package https://review.opendev.org/715418 | 12:59 |
| *** openstackstatus has quit IRC | 13:01 | |
| *** openstack has joined #opendev | 13:04 | |
| *** ChanServ sets mode: +o openstack | 13:04 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Cap stestr version https://review.opendev.org/715415 | 13:07 |
| *** rpittau|bbl is now known as rpittau | 13:10 | |
| openstackgerrit | Merged opendev/system-config master: Add links to dev docs on gitea https://review.opendev.org/715264 | 13:10 |
| *** roman_g has quit IRC | 13:18 | |
| *** hashar_ has joined #opendev | 13:20 | |
| *** hashar has quit IRC | 13:20 | |
| *** hashar_ is now known as hsahar | 13:26 | |
| *** hsahar is now known as hashar | 13:26 | |
| *** ysandeep|rover is now known as ysandeep|away | 13:39 | |
| openstackgerrit | Merged opendev/system-config master: Remove gitea doc links https://review.opendev.org/715267 | 13:46 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add python-stow-versions element https://review.opendev.org/713692 | 13:49 |
| *** roman_g has joined #opendev | 13:54 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Open Edge has proved to be stable - bumping to 40 https://review.opendev.org/715423 | 13:54 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Remove bashate from test-requirements https://review.opendev.org/715328 | 14:00 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Revert "upload-logs-swift: Create a download script" https://review.opendev.org/715325 | 14:00 |
| openstackgerrit | Merged zuul/zuul-jobs master: Cap stestr version https://review.opendev.org/715415 | 14:09 |
| AJaeger | infra-root, system-config and zuul-jobs repos are fixed, changes merge again... | 14:15 |
| mordred | woot! | 14:15 |
| AJaeger | Until the next fire ;) | 14:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Base run jobs needing containers on containers base https://review.opendev.org/715063 | 14:16 |
| openstackgerrit | Merged openstack/project-config master: Open Edge has proved to be stable - bumping to 40 https://review.opendev.org/715423 | 14:24 |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove bashate from test-requirements https://review.opendev.org/715328 | 14:26 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-tox: use python3 by default https://review.opendev.org/712804 | 14:26 |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "upload-logs-swift: Create a download script" https://review.opendev.org/715325 | 14:26 |
| *** ykarel is now known as ykarel|afk | 14:34 | |
| *** mlavalle has quit IRC | 14:43 | |
| openstackgerrit | Merged opendev/system-config master: Add meetpad server https://review.opendev.org/714238 | 14:44 |
| openstackgerrit | Merged opendev/system-config master: Build jitsi-meet images https://review.opendev.org/714494 | 14:48 |
| openstackgerrit | Merged opendev/system-config master: jitsi-meet: open etherpad on join https://review.opendev.org/714505 | 14:48 |
| *** mlavalle has joined #opendev | 15:03 | |
| *** Open10K8S has joined #opendev | 15:05 | |
| *** _mlavalle_1 has joined #opendev | 15:22 | |
| *** _mlavalle_1 has quit IRC | 15:22 | |
| *** mlavalle has quit IRC | 15:25 | |
| clarkb | re opendev docs links I think the next step in that process is to publish the infra manual without the openstack theme | 15:30 |
| clarkb | we can probably use a preset sphinx theme unless someone thinks they may be able to opendev a theme (I doubt I'd make much progress on that) | 15:30 |
| fungi | alabaster is probably fine for now. it will let us specify a logo image in the sphinx config as a stop-gap | 15:33 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: install-kubernetes: fix missing package, fix the client.key file path https://review.opendev.org/715418 | 15:34 |
| *** ykarel|afk is now known as ykarel | 15:36 | |
| clarkb | cool I'll try that | 15:37 |
| mnaser | infra-root: i just caught this watching logs on submit-logstash-jobs (which won't be seen in logs cause it happens after swift upload) | 15:49 |
| mnaser | Traceback (most recent call last):\n File \"/var/lib/zuul/builds/c3565c6c4d91424fa052629cb5021d2e/work/tmp/ansible_submit_log_processor_jobs_payload_6mhcqyrl/ansible_submit_log_processor_jobs_payload.zip/ansible/modules/submit_log_processor_jobs.py\", line 201, in main\n File | 15:49 |
| mnaser | \"/var/lib/zuul/builds/c3565c6c4d91424fa052629cb5021d2e/work/tmp/ansible_submit_log_processor_jobs_payload_6mhcqyrl/ansible_submit_log_processor_jobs_payload.zip/ansible/modules/submit_log_processor_jobs.py\", line 108, in submitJobs\n File \"/usr/lib/zuul/ansible/2.9/lib/python3.5/site-packages/gear/__init__.py\", line 1485, in submitJob\n conn = self.getConnection()\n File | 15:49 |
| mnaser | \"/usr/lib/zuul/ansible/2.9/lib/python3.5/site-packages/gear/__init__.py\", line 1284, in getConnection\n raise NoConnectedServersError(\"No connected Gearman servers\")\ngear.NoConnectedServersError: No connected Gearman servers\n | 15:49 |
| mnaser | Executor: ze01.openstack.org | 15:50 |
| fungi | are we seeing the same on any other executors? | 15:50 |
| corvus | well, there's a growing backlog of logstash jobs | 15:51 |
| corvus | so something's getting there | 15:51 |
| corvus | maybe this is tenant-specific? | 15:52 |
| corvus | mnaser: was that in the zuul tenant rather than openstack? | 15:52 |
| mnaser | corvus: oh yes, that was a job in zuul (zuul/zuul-jobs change) | 15:52 |
| mordred | I'm seeing a ton of red across jobs that shouldn't have any red in openstack tenant - I'm just now starting to look | 15:52 |
| mordred | "nova-compute: no process found" and similar seem to be common across them | 15:53 |
| corvus | mnaser: i don't know if we expect that to work outside of openstack (i mean, maybe we expect it to work and it accidentally broke? or maybe we don't expect it to work and we never cared to clean up the error? i'll try to figure out) | 15:53 |
| clarkb | corvus: mnaser the gearman server is up but running 'status' against it isn't returning very quickly | 15:53 |
| clarkb | corvus: it should work outside of openstack thee are no acls on it | 15:53 |
| corvus | clarkb: that's probably not surprising if there's 800k jobs | 15:53 |
| clarkb | corvus: ya I'm guessing that downstream something broke backing up the pipelines and node gearman has fallen over | 15:54 |
| clarkb | we'll probably need to restart geard and then look at logstash and elasticsearch | 15:54 |
| clarkb | I can do that | 15:54 |
| corvus | clarkb: recent messages are in logstash | 15:54 |
| corvus | so it's not completely stopped | 15:54 |
| clarkb | there is a good chances that elasticsearch ran out of disk again | 15:54 |
| clarkb | no disk utilization seems fine | 15:55 |
| mnaser | sorry i've opened a can of worms on friday :( | 15:55 |
| clarkb | and cluster reports greeness | 15:55 |
| fungi | maybe something started logging massive amounts of data and that's increased the volume beyond our capacity to process it? | 15:57 |
| clarkb | 8/20 logstash processees are running. Possible something went through and OOMed the rest. I'll start by ensuring those logstash side processes are running then we can restart geard | 15:58 |
| clarkb | fungi: ya its looking like something filled the pipe ^ | 15:58 |
| fungi | yeek | 16:00 |
| mordred | hahaha. the devstack errors I'm seeing are my fault - I added that pip check run and now it's throwing me "hacking 1.1.0 has requirement flake8<2.7.0,>=2.6.0, but you have flake8 2.5.5." | 16:02 |
| mordred | because that's important for devstack | 16:02 |
| * mordred moves to requirements | 16:02 | |
| clarkb | mnaser: thanks for the report. All logstash pipeline processors should be up and running now | 16:14 |
| clarkb | infra-root remote: https://review.opendev.org/715472 Use alabaster theme | 16:29 |
| clarkb | (gerritbot not respoinding as it doesn't have the new name yet ) | 16:29 |
| clarkb | AJaeger: ^ your thoughts much appreciated too :) | 16:30 |
| *** ykarel is now known as ykarel|away | 16:31 | |
| AJaeger | clarkb: I miss the "next page" button - but for now, that's a simple and clean change, so let's do it. | 16:38 |
| fungi | https://cbba045a24c406d10b4d-992fbb76369d370eec805eb398c9de6e.ssl.cf1.rackcdn.com/715472/1/check/opendev-tox-docs/31ebe5b/docs/ | 16:39 |
| fungi | lgtm! | 16:39 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Drop Python 2.7 support https://review.opendev.org/715476 | 16:40 |
| *** dpawlik has quit IRC | 16:41 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use our jitsi-meet image for meetpad https://review.opendev.org/714510 | 16:43 |
| corvus | clarkb, fungi: ^ could you do a speedy re-review of that? i think it hit a startup race in the gate | 16:44 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Correct launch readme link https://review.opendev.org/715478 | 16:46 |
| fungi | corvus: sure thing, i was just about to the point where i could take a look at those failures in case you hadn't yet | 16:46 |
| corvus | meanwhile, i'll spin up the server | 16:49 |
| clarkb | mordred and fungi beat me to it | 16:49 |
| *** smcginnis has quit IRC | 16:50 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: install-kubernetes: fix missing package, fix the client.key file path https://review.opendev.org/715418 | 16:54 |
| openstackgerrit | James E. Blair proposed opendev/zone-opendev.org master: Add meetpad hostnames https://review.opendev.org/715485 | 17:01 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add meetpad01 to inventory https://review.opendev.org/715486 | 17:02 |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: install-kubernetes: fix missing package, fix the client.key file path https://review.opendev.org/715418 | 17:05 |
| clarkb | I'm only 10 minutes late and already the changes are approved :) | 17:13 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add meetpad hostnames https://review.opendev.org/715485 | 17:16 |
| *** hashar has quit IRC | 17:17 | |
| mnaser | so.. about the repo freeze | 17:20 |
| mordred | oh - I think we're ready to move forward with that | 17:20 |
| mnaser | https://review.opendev.org/#/c/714965/ can gladly be a testbed if you want | 17:21 |
| mordred | clarkb: what do you think about manually running latest ansible against review - and then manually running the manage-projects from the container (should still work even with gerrit running directly) | 17:22 |
| clarkb | mordred: not sure I understand that last bit | 17:22 |
| clarkb | you mean without restarting gerrit on new image it should work? | 17:23 |
| mordred | yeah | 17:23 |
| clarkb | that seems like a good plan | 17:23 |
| mordred | we don't exec inside the image - we just docker run | 17:23 |
| corvus | and everything is bind mounted and host networking, so should be the same. ++ | 17:23 |
| mordred | so - I think the steps are a) manual run of service-review with disabled removed to run it against review and get config files up to the latest and wrapper scripts and whatnot | 17:25 |
| mordred | then log in to review, do a docker pull and then run manage-projects and watch the logs for errors | 17:25 |
| mordred | (it should be a no-op run right now) | 17:26 |
| clarkb | will it not dockerpull in ansible already? | 17:27 |
| clarkb | but ya that soundsgood | 17:27 |
| clarkb | especially the noop run first | 17:27 |
| mordred | not currently - we should probalby add that - right now we're counting on docker-compose to pull for us | 17:27 |
| mordred | but doing a pull in manage-projects playbook is likely a good idea | 17:27 |
| fungi | assuming the docker run doesn't fetch new project-config, it ought to no-op | 17:27 |
| mordred | I've made a copy of service-review.yaml as service-review-mt.yaml - removed the disabled - and I'm going to run it now | 17:29 |
| mordred | unless y'all want me to screen and run it | 17:29 |
| fungi | up to you | 17:32 |
| mordred | you know - I'm in screen - if anyone wants to watch | 17:32 |
| mordred | I'm teed up - but will wait a sec in case anyone want to watch along | 17:33 |
| fungi | i'm connected to the currently established root screen session, but i'm guessing that's not yours | 17:33 |
| fungi | looks like it was fiddling with firewall rules | 17:33 |
| fungi | oh, or... you're on bridge | 17:33 |
| fungi | not review.o.o | 17:33 |
| mordred | yeah | 17:33 |
| mordred | root@bridge:/opt/system-config# ansible-playbook playbooks/service-review-mt.yaml | 17:33 |
| mordred | should be the last thing there | 17:34 |
| mordred | although | 17:34 |
| mordred | let me _also_ get in a screen on review for the next bit | 17:34 |
| fungi | okay, found it | 17:34 |
| fungi | i also closed out the root screen session on review.o.o since it seemed to have been abandoned there for smoe time | 17:34 |
| fungi | some time | 17:34 |
| mordred | I've made a new one on review :) | 17:36 |
| mordred | k. I will now run the ansible on bridge | 17:36 |
| fungi | connected to both now | 17:37 |
| corvus | mordred: will the ansible run on bridge run manage-projects because of recent config changes? | 17:37 |
| mordred | corvus: it shouldn't | 17:38 |
| * clarkb finds root screen | 17:38 | |
| corvus | why not? because there were no changes or it's not configured to? | 17:38 |
| mordred | corvus: manage-projects is not in the gerrit role - it's a separate playbook | 17:38 |
| corvus | ok | 17:38 |
| mordred | also - we haven't landed that change yet | 17:38 |
| corvus | k. sounds like it's been thought through. :) | 17:38 |
| mordred | I'm going to go to review.o.o root screen and run manage-projects | 17:39 |
| mordred | doing the docker-compose pull first | 17:39 |
| fungi | right, didn't want to incorporate it until it saw a dry-run under close observation | 17:40 |
| mordred | that's missing --host=net | 17:40 |
| mordred | bother - those changes are in teh run-manage-projects change | 17:40 |
| mordred | https://review.opendev.org/#/c/714970 | 17:41 |
| mordred | since review is still in the emergency file - that should be safe to land | 17:41 |
| mordred | and then we can try this again | 17:41 |
| clarkb | looking at that list of mounts where is the cache? | 17:41 |
| *** dpawlik has joined #opendev | 17:41 | |
| mordred | /opt/lib/jeepyb I think | 17:41 |
| clarkb | ya ls'ing there looks like that is it | 17:42 |
| clarkb | thanks | 17:42 |
| mordred | jeepyb-cache-dir=/opt/lib/jeepyb | 17:42 |
| mordred | yeah | 17:42 |
| clarkb | mordred: I've approved that change | 17:43 |
| fungi | heh, double-tap | 17:43 |
| clarkb | mordred: note, I think we want to only run manage-projects when the file changes? or at least that was the old behavior | 17:43 |
| clarkb | with the caching its probably fast enough to run often though | 17:43 |
| mordred | clarkb: yeah - let's implement that before we take it out of emergency | 17:44 |
| mordred | of course, this would be a great candidate to run from zuul in post on project-config changes eventually | 17:45 |
| *** diablo_rojo has joined #opendev | 17:46 | |
| openstackgerrit | Merged opendev/system-config master: Use our jitsi-meet image for meetpad https://review.opendev.org/714510 | 17:47 |
| fungi | i think we ran it when gerrit/projects.yaml or anything under gerrit/acls changed, right? | 17:47 |
| clarkb | yup | 17:48 |
| mordred | yeah - but probably safe to run when anything in project-config changes? | 17:48 |
| clarkb | or anything in project-config/gerrit | 17:49 |
| fungi | probably safe, but maybe wasteful if it takes a long time to comlpete | 17:49 |
| fungi | yeah, anything under the gerrit tree is a good compromise | 17:49 |
| mordred | yeah. | 17:49 |
| fungi | the only other thing in there is the notify_impact.yaml file right now | 17:49 |
| mordred | well - I might have to re-add the puppet logic of copying files from /opt/project-config to /home/gerrit2 | 17:49 |
| mordred | because right now it's just bind-mounting in /opt/project-config | 17:49 |
| mordred | but - you know - that's not that horrible :) | 17:50 |
| clarkb | I wasn't around last friday, is the version of project-config on review.o.o up to date with the two project renames? | 17:50 |
| openstackgerrit | Merged opendev/system-config master: Add meetpad01 to inventory https://review.opendev.org/715486 | 17:50 |
| clarkb | we'll need it to be in order to properly noop | 17:50 |
| mordred | yeah - it should be. let's check | 17:52 |
| mordred | yes | 17:52 |
| *** dpawlik has quit IRC | 17:52 | |
| fungi | i thought it wasn't because of the emergency file entry | 17:56 |
| fungi | looks up to date in /opt though | 17:56 |
| mordred | yeah - cause I just re-ran the service playbook which pulled the latest (because of notify-impact.yaml) | 17:57 |
| fungi | ahh | 17:58 |
| fungi | okay | 17:58 |
| fungi | that 'splains it | 17:58 |
| *** smcginnis has joined #opendev | 18:15 | |
| noonedeadpunk | hey everyone. Do you have any info when diskimage-builder is going to be tagged? As we're kinda waiting for opportinuty to make use of recently merged python-stow-versions element:) | 18:22 |
| clarkb | noonedeadpunk: ianw typically makes the dib releases and it is a weekend right now in australia. Maybe ping ianw and see if a release can be made on monday australia time? | 18:26 |
| noonedeadpunk | clarkb: ok thanks! I guess he has just been pinged :p | 18:28 |
| *** nmickus has joined #opendev | 18:30 | |
| rm_work | hey, wanted to make a job from a template non-voting, but looks like "voting: false" doesn't work in the templates section -- what is the strategy to resolve this? can we provide the job name directly in a jobs section and have it pull the right job from the template automatically or something? | 18:32 |
| *** lpetrut has quit IRC | 18:33 | |
| clarkb | rm_work: you need to remove the template and list the jobs normally | 18:33 |
| rm_work | ok so yeah, that does work? | 18:33 |
| clarkb | ya if you remove the template | 18:33 |
| rm_work | just take the job name and list it in the jobs section instead of using the template? | 18:33 |
| rm_work | k | 18:33 |
| rm_work | do we lose anything by removing the template? | 18:33 |
| clarkb | rm_work: all the jobs in the template. You can list them directly though | 18:33 |
| rm_work | kk | 18:33 |
| noonedeadpunk | clarkb: I guess it's possible to leave template but just metnion single job as non-voting among jobs? | 18:33 |
| rm_work | I THINK in this case it's only one? | 18:34 |
| rm_work | yeah I wondered if overriding just that job was an option | 18:34 |
| noonedeadpunk | and that's will override behaviour of template? | 18:34 |
| openstackgerrit | Merged opendev/system-config master: Run manage-projects on gerrit servers https://review.opendev.org/714970 | 18:34 |
| openstackgerrit | Merged opendev/puppet-openstackid master: added www. cname on vhost file https://review.opendev.org/715293 | 18:34 |
| clarkb | noonedeadpunk: I want to say that doesn't work (people have tried ti and been confused with the result) | 18:34 |
| clarkb | rm_work: you can also update the template globally fi the chagne should be global | 18:34 |
| rm_work | ah hmmmm | 18:35 |
| rm_work | we may own that template actually, now that i think about it, so possibly an option lol | 18:35 |
| rm_work | though I think it'd be preferrable to do just an override | 18:35 |
| noonedeadpunk | clarkb: ah, yes, we were disabling job inside tempalte I guess.. | 18:36 |
| noonedeadpunk | clarkb: but actually no | 18:37 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_tempest/src/commit/0e17aafba9535d86399d1512f2fc3a6891883283/zuul.d/project.yaml | 18:37 |
| noonedeadpunk | that worked for us | 18:37 |
| noonedeadpunk | and openstack-ansible-deploy-aio_distro_metal_tempestconf-centos-7 job is part of openstack-ansible-deploy-aio_metal-jobs tempalte | 18:37 |
| clarkb | noonedeadpunk: got it | 18:38 |
| noonedeadpunk | (I meant ubuntu-bionic job not tempestconf-centos-7) | 18:39 |
| fungi | yeah, it's worth trying to do that with a variant, as long as the job in the template isn't marked final | 18:44 |
| clarkb | fungi: maybe that was why I remembered it not working | 18:44 |
| fungi | (that may even work with final jobs, can't remember if that blocks variants or just parenting) | 18:44 |
| clarkb | mordred: the change langed I think that means we should've promoted a new image? | 18:48 |
| clarkb | oh no we needed new ansible in this case right? | 18:48 |
| clarkb | I'm about to find lunch but can help more after | 18:48 |
| mordred | clarkb: yeah - it's new ansible we need | 18:52 |
| mordred | clarkb: so I'll wait for the next pulse to update us | 18:53 |
| mordred | so that I'm not fighting it | 18:53 |
| *** rpittau is now known as rpittau|afk | 18:57 | |
| *** roman_g has quit IRC | 19:03 | |
| *** ralonsoh has quit IRC | 19:16 | |
| mordred | infra-root: ok. system-config is updaed - I'm going to run the playbook again | 19:19 |
| fungi | watching | 19:23 |
| AJaeger | mordred: can I remove my WIP from the new repo requests? | 19:25 |
| mordred | fungi: ok. that went well - I think it's time to switch to review and try manage-projects | 19:25 |
| mordred | AJaeger: almost | 19:25 |
| AJaeger | (those still need another core to review) | 19:25 |
| openstackgerrit | Ronelle Landy proposed zuul/zuul-jobs master: Add var to allow jobs to skip centos8 deps install https://review.opendev.org/715524 | 19:26 |
| mordred | fungi: k. that all looks ok - I'm going to run manage-projects | 19:26 |
| fungi | yep | 19:26 |
| fungi | looks like it's doing its thing | 19:27 |
| fungi | very verbosely | 19:27 |
| mordred | wow. that's doing a lot of things | 19:27 |
| mordred | yeah | 19:27 |
| mordred | it wasn't happy about opendev/infra-manual | 19:27 |
| fungi | nope | 19:27 |
| fungi | FetchConfigException | 19:28 |
| *** nmickus has left #opendev | 19:28 | |
| mordred | http://paste.openstack.org/show/791255/ | 19:28 |
| mordred | fungi: hrm | 19:29 |
| fungi | so i think it's having trouble fetching the meta/config ref maybe? | 19:30 |
| mordred | oh - look - it tried to create a new empty repo | 19:31 |
| *** diablo_rojo has quit IRC | 19:31 | |
| mordred | did we miss renaming the repo in /opt/lilb/jeepyb? | 19:31 |
| clarkb | you shouldnt need to update the cache iirc | 19:31 |
| clarkb | it should see the project is already there and populate the cache from gerrit automatically | 19:32 |
| clarkb | (I'm still eating my curry) | 19:33 |
| *** diablo_rojo has joined #opendev | 19:33 | |
| mordred | I bet you anything it's another utf-8 thing | 19:33 |
| mordred | if output.strip() != "project.config" or status != 0: | 19:33 |
| mordred | ok. that was silly, I have started a new screen | 19:35 |
| openstackgerrit | Merged zuul/zuul-jobs master: install-kubernetes: fix missing package, fix the client.key file path https://review.opendev.org/715418 | 19:35 |
| mordred | hrm. why isn't hat working ... | 19:36 |
| mordred | oh. it did | 19:36 |
| mordred | yup. it's returning bytes | 19:37 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: Return strings from run_command https://review.opendev.org/715527 | 19:39 |
| mordred | ok. that patch ^^ fixes it | 19:42 |
| clarkb | mordred: maybe double chwck we dont have similiar problems where we interface with gerritlib? | 19:42 |
| clarkb | there was semi recent changes ti gerritlib to deal with unicode and python3 | 19:43 |
| clarkb | I cant remember if we went all bytes or all strings | 19:43 |
| mordred | clarkb: we seemed to find infra-manual in the list that time - so the acls work at least worked | 19:43 |
| mordred | but looking | 19:43 |
| clarkb | I can help in afew too. finishing up lunch | 19:43 |
| *** ysandeep|away has quit IRC | 19:44 | |
| *** ysandeep has joined #opendev | 19:45 | |
| mordred | http://paste.openstack.org/show/791257/ <-- there's the log from the most recent run | 19:45 |
| *** elod has quit IRC | 19:46 | |
| *** elod has joined #opendev | 19:46 | |
| mordred | clarkb: fwiw - I edited jeepyb in place with the above patch inside of a manage-projects container and it fixed that instance | 19:46 |
| clarkb | k | 19:47 |
| *** diablo_rojo has quit IRC | 19:51 | |
| fungi | Unable to freeze job graph: Job system-config-build-image-python-builder not defined | 19:55 |
| fungi | that's blocking the jeepyb fix from getting tested | 19:56 |
| clarkb | it needs a -37 at the end | 19:56 |
| clarkb | or 3.7? | 19:56 |
| fungi | did we wedge that from another tenant? | 19:56 |
| clarkb | fungi: it got converted in opendev/system-config and not updated on this side I think | 19:57 |
| clarkb | mordred: I've double checked gerritlib and the change that was made was to go to unicode which explains the mismatch I think | 19:57 |
| clarkb | mordred: so ya your fix is proper | 19:57 |
| fungi | oh, yep, it's fixed in opendev/system-config but not yet in openstack/project-config | 19:58 |
| clarkb | fungi: I think we might need to update that job in project-config | 19:58 |
| clarkb | ya | 19:58 |
| fungi | is the fix already proposed? | 19:58 |
| fungi | i can if not | 19:58 |
| clarkb | not that I know of | 19:58 |
| fungi | on it | 19:58 |
| mordred | thanks! | 19:59 |
| fungi | do we want -3.7 or -3.8? seems we have both defined | 19:59 |
| clarkb | 3.7 in this case I htink | 20:00 |
| mordred | 3,7 | 20:00 |
| clarkb | its our default still, but we have 3.8 to transition to eventually | 20:00 |
| mordred | yah | 20:00 |
| mordred | so the 3.8 image is there to allow us to do that on a consumer-by-consumer basis | 20:00 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Update Python image builder job name https://review.opendev.org/715530 | 20:01 |
| fungi | see if that seems right | 20:02 |
| mordred | looks perfect- thanks! | 20:02 |
| *** diablo_rojo has joined #opendev | 20:07 | |
| openstackgerrit | Merged openstack/project-config master: Update Python image builder job name https://review.opendev.org/715530 | 20:24 |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add vexxhost/smokeping_prober-helm https://review.opendev.org/715544 | 20:25 |
| mordred | I have rechecked the jeepyb change | 20:25 |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add vexxhost/smokeping_prober-helm https://review.opendev.org/715544 | 20:30 |
| openstackgerrit | Monty Taylor proposed opendev/gerritlib master: Run gerritlib integration tests with python3 https://review.opendev.org/715546 | 20:34 |
| mordred | clarkb: ^^ | 20:34 |
| clarkb | ooh | 20:34 |
| clarkb | +2 do we want to get results on that before attempting production rn again? | 20:35 |
| mordred | well - I bet we'll get results on it before the images are finished being built | 20:36 |
| mordred | and - I *bet* it'll fail | 20:36 |
| mordred | since it should need the jeepyb change to work for manage-projects to work | 20:36 |
| clarkb | ah | 20:36 |
| mordred | but let's see :) | 20:36 |
| mordred | if it does, that'll be a great indication of it being a good test | 20:36 |
| mordred | clarkb: oh - did you see the results from the don't-install-test-requirements in #openstack-qa ? | 20:37 |
| clarkb | I did not | 20:37 |
| mordred | tl;dr - it worked - and actually fixed some broken nv jobs | 20:38 |
| mordred | since they were failing on issues installing things from test-reuqirements :) | 20:38 |
| clarkb | huh | 20:38 |
| mordred | things like something something nose | 20:38 |
| mordred | anywa - thought you'd enjoy that :) | 20:39 |
| fungi | results are in | 20:46 |
| fungi | gerritlib-jeepyb-integration FAILURE in 7m 52s | 20:46 |
| clarkb | same fetch config exception | 20:48 |
| clarkb | thats good and consistent :) | 20:48 |
| mordred | woot! | 20:49 |
| mordred | let me depends-on it | 20:49 |
| openstackgerrit | Monty Taylor proposed opendev/gerritlib master: Run gerritlib integration tests with python3 https://review.opendev.org/715546 | 20:49 |
| mordred | clarkb: if we don't watch out - we're going to end up with an actually tested system :) | 20:51 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add an image pull to manage-projects playbook https://review.opendev.org/715551 | 20:59 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: Fix fsck_command for python3 https://review.opendev.org/715552 | 21:00 |
| openstackgerrit | Monty Taylor proposed opendev/gerritlib master: Run gerritlib integration tests with python3 https://review.opendev.org/715546 | 21:01 |
| mordred | found another issue - there's the jeepyb patch and I updated the gerritlib dependson | 21:01 |
| fungi | getting sidelines by dinner but will try to resume reviewing once i eat, if there's still stuff in progress for this | 21:03 |
| mordred | it's almost certain there will be | 21:03 |
| clarkb | yay testing | 21:03 |
| mordred | clarkb: I'm glad you made that integration test - although I'm a little sad it's in gerritlib and not in jeepyb :) | 21:04 |
| clarkb | mordred: ya it was largely a "put this somehwere" thing | 21:04 |
| mordred | yup | 21:04 |
| clarkb | mordred: maybe we should make a copy with a new name in jeepyb then delete the old one in gerritlib | 21:04 |
| mordred | if you'd put it in jeepyb I'd be saying "I wish you'd put it in gerritlib" | 21:04 |
| clarkb | ha | 21:04 |
| mordred | nah - by the time we're done with this it'll be fine and we'll never touch it again | 21:05 |
| clarkb | mordred: more bugs https://zuul.opendev.org/t/openstack/build/2225023284744c10b33bc6f708a888a7/log/job-output.txt#629 | 21:12 |
| clarkb | are you already looking at that or should I? | 21:12 |
| mordred | clarkb: oh fun | 21:13 |
| mordred | wat? TypeError: 'filter' object is not subscriptable | 21:14 |
| clarkb | mordred: ya we filter in gerritlib | 21:15 |
| clarkb | and I'm guessing that returns an iterable not a list in python3 | 21:15 |
| clarkb | so we can either do list(filter()) in gerritlib or list() the return from gerritlib | 21:15 |
| clarkb | changing gerritlib will requires a release | 21:15 |
| clarkb | and ya filter() returns an iterator | 21:15 |
| clarkb | maybe we should do both things? and then we can remove the jeepyb fix later when gerritlib is released? | 21:16 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: More python3 fixes https://review.opendev.org/715552 | 21:17 |
| mordred | clarkb: oh - sorry - I didn't see that - I stuck in a next | 21:17 |
| mordred | I think we shoudl do the list() wrap | 21:18 |
| fungi | is there a common topic for these? i've lost track of the outstanding patches | 21:18 |
| openstackgerrit | Clark Boylan proposed opendev/gerritlib master: Return lists from listing functions https://review.opendev.org/715555 | 21:18 |
| mordred | next doesnt' work on a list | 21:18 |
| mordred | ++ | 21:18 |
| clarkb | mordred: ^ there is the list() wrap on the gerritlib side | 21:18 |
| mordred | let me remove the jeepyb next() | 21:18 |
| clarkb | mordred: maybe do it in jeepyb too for now? | 21:18 |
| mordred | yup | 21:18 |
| clarkb | mordred: we consume it from releases so will want to make a release | 21:18 |
| clarkb | it being gerritlib | 21:18 |
| clarkb | fungi: I didn't use one | 21:18 |
| clarkb | mordred: is there a topic I should use? | 21:18 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: More python3 fixes https://review.opendev.org/715552 | 21:18 |
| mordred | clarkb: gerrit-docker maybe? | 21:19 |
| clarkb | mordred: there are several other places in jeepyb where we call gerrit\.list.* | 21:19 |
| mordred | why did we get unable to freeze job graph? | 21:19 |
| clarkb | mordred: I think you want to do the same update on those | 21:20 |
| mordred | ++ | 21:20 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: More python3 fixes https://review.opendev.org/715552 | 21:20 |
| clarkb | fungi: topic:gerrit-docker set | 21:20 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Use versioned jobs in the gate too https://review.opendev.org/715556 | 21:21 |
| mordred | clarkb, fungi: ^^ we missed the gate jobs before. doh | 21:21 |
| mordred | I'm going to squash the jeepyb patches now since we have to run it again anyway | 21:22 |
| clarkb | mordred: k | 21:22 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: More python3 fixes https://review.opendev.org/715527 | 21:22 |
| openstackgerrit | Monty Taylor proposed opendev/gerritlib master: Run gerritlib integration tests with python3 https://review.opendev.org/715546 | 21:23 |
| mordred | ok. squashed, abandoned the other and redid the depends-on | 21:24 |
| mordred | let's see how far we get this time :) | 21:24 |
| mordred | clarkb: oh - did you see noonedeadpunk asking about a dib release earlier? | 21:25 |
| mordred | we've got the new stow-python element in there | 21:25 |
| clarkb | mordred: I did, ianw typically manages those so asked that we ping ianw for a inw time zone monday release | 21:25 |
| mordred | cool | 21:25 |
| clarkb | kids preschool is going to try distance learning via zoom | 21:27 |
| clarkb | I wish the tech world would conglomerate around open tools a bit more so that by the time they trickle down into schools we don't end up using such things | 21:27 |
| clarkb | maybe I can convince them to use meetpad (though etherpad probably not super use ful for them) and they can be our load testers >_> | 21:28 |
| mordred | waiting for tempest-slow-py3 is like watching paint dry | 21:29 |
| mordred | clarkb: gerritlib integration passed! | 21:30 |
| clarkb | yay | 21:30 |
| clarkb | I think everything I didn't write has my +2, fungi if you have time to review looks like some stuff is happy now | 21:31 |
| mordred | clarkb: oh - also this https://review.opendev.org/#/c/715551/1 | 21:32 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Base run jobs needing containers on containers base https://review.opendev.org/715063 | 21:32 |
| clarkb | mordred: check comment on that one, I think its fine but someting to consider | 21:33 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove unused gerrit puppet things https://review.opendev.org/714001 | 21:34 |
| mordred | clarkb: yeah - I *think* it should still work liek if we update that image name in a host_var | 21:35 |
| clarkb | mordred: ya but not if at role include time for main role include? | 21:35 |
| mordred | but I agree - for now we should almost certainly be fine | 21:35 |
| clarkb | its probably fine in most cases | 21:35 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add openstacksdk and python-openstackclient to integrated https://review.opendev.org/715559 | 21:50 |
| mordred | clarkb: I'm torn on that ^^ | 21:50 |
| mordred | it's maybe the wrong idea | 21:50 |
| mordred | but - liek right now I'm having to wait for a devstack change to land so I can click +A because I can't depends-on and +A | 21:51 |
| mordred | BUT | 21:51 |
| clarkb | hrm a depends on would enforce order but not do the fancy testing I guess | 21:51 |
| clarkb | being able to test things together like that seems desireable | 21:51 |
| mordred | well - it's that it won't enqueue it into the gate | 21:51 |
| mordred | testing thinngs together totally works | 21:52 |
| mordred | but only at check | 21:52 |
| clarkb | ya then you hvae to wait for things to merge on the other side first | 21:52 |
| mordred | for gate - brcause devstack is in a shared queue, a depends-on from openstacksdk won't enqueue openstacksdk into the gate | 21:52 |
| mordred | yeah | 21:52 |
| mordred | so - today that patch seems liek a good idea | 21:52 |
| mordred | but - maybe most days it's a bad idea | 21:52 |
| mordred | because it is fairly rare that openstacksdk depends on one of the other projects | 21:53 |
| clarkb | other way around though | 21:53 |
| mordred | yeah. it might just be the correct thing to do regardless of whether it might be annoying sometimes | 21:53 |
| openstackgerrit | Jeremy Stanley proposed opendev/puppet-openstackid master: Add SSL options in www subdomain redirect vhost https://review.opendev.org/715562 | 21:57 |
| mordred | fungi, corvus: https://review.opendev.org/#/c/715527/ is ready to go is tested with https://review.opendev.org/#/c/715546/ | 22:02 |
| fungi | okay, i think i'm caught up on other fires and questions... looking now to see what needs reviewing for the gerrit stuff | 22:02 |
| mordred | actually - https://review.opendev.org/#/c/715556/ should go first | 22:02 |
| mordred | fungi: so - actually - hold off on the others - the gate won't work for jeepyb until that's in | 22:03 |
| mordred | we fixed check earlier and not gate :) | 22:03 |
| fungi | did i not get all the occurrences of that job name? i thought i had searched for them | 22:03 |
| mordred | I blame trolls | 22:04 |
| fungi | oh! | 22:06 |
| fungi | i fixed the name of the build job, that's fixing the name of the upload job | 22:06 |
| fungi | now i get it | 22:06 |
| fungi | we stopped at the first error, so it didn't complain about the other job name | 22:06 |
| mordred | yeah | 22:07 |
| mordred | fun right? | 22:07 |
| fungi | and we can recheck 715527 once 715556 merges | 22:07 |
| fungi | (since it's in a config repo) | 22:07 |
| mordred | yah | 22:08 |
| openstackgerrit | Merged openstack/project-config master: Use versioned jobs in the gate too https://review.opendev.org/715556 | 22:15 |
| clarkb | I rechecked 715527 | 22:17 |
| mordred | https://review.opendev.org/#/c/715546/ is ready for +A | 22:17 |
| fungi | as did mordred | 22:17 |
| mordred | me goes for bikeride - hopefully patches will be landed by the time I'm back and we can try this all again :) | 22:19 |
| clarkb | enjoy. I'm not bike riding today because it is cold and wet and miserable. And indoor bike time is just not the same | 22:20 |
| corvus | anyone care to try a meetpad? https://meetpad.opendev.org/meetpadtest1 | 22:36 |
| openstackgerrit | Merged opendev/puppet-openstackid master: Add SSL options in www subdomain redirect vhost https://review.opendev.org/715562 | 22:36 |
| clarkb | I typically jitsi from my phone, but I can get headset set up and do it on real computer | 22:38 |
| clarkb | give me a couple minutes to climb under desk and plug stuff in | 22:38 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Correct jitsi-meet config file https://review.opendev.org/715572 | 22:42 |
| clarkb | I can hear you corvus | 22:42 |
| clarkb | can you hear me? that may just be local mic problems if not | 22:42 |
| clarkb | also i get warnings about etherpad cookies not being enabled and it really wnting cookies | 22:42 |
| corvus | i can not hear you | 22:43 |
| corvus | clarkb: i have gotten the cookies thing sometimes, and sometimes not | 22:43 |
| clarkb | ok let me sort out my mic | 22:43 |
| corvus | sounds like it's not just a local problem and it may need some debugging... | 22:43 |
| clarkb | corvus: ya chrome can't get my mic :) | 22:46 |
| clarkb | I got the cookie thing again fwiw in chrome | 22:46 |
| corvus | well bummer | 22:46 |
| clarkb | the url thing in chrome says it can access my mic but then jitis itself seems confused | 22:47 |
| clarkb | I'm going to see if I can determine which device it thinks it is using | 22:47 |
| corvus | i think it's set up to proxy etherpad through the nginx container. that seems a little weird to me, and potentially a source of problems, but maybe it's necessary. | 22:47 |
| clarkb | the little pop up says it can't use it for "unknown reason" | 22:47 |
| clarkb | was the feedback me? | 22:48 |
| corvus | nope, me, sorry | 22:48 |
| clarkb | corvus: WaG87Fvi2lqlcSFuzHQg | 22:57 |
| openstackgerrit | Merged opendev/system-config master: Base run jobs needing containers on containers base https://review.opendev.org/715063 | 22:59 |
| clarkb | corvus: /opt/etherpad-lite/etherpad-lite/src/static/js/pad_editbar.js | 23:03 |
| clarkb | corvus: in toggleDropdowns | 23:03 |
| johnsom | I have an oddball question about gerrit/zuul. If I wanted to setup another category similar to a third party CI, but the job is in other OpenStack repositories (maybe x/ repos) how would I go about doing that? | 23:10 |
| johnsom | We have third party drivers that would like to run non-voting jobs, live vmware, etc. that for example are under x/vmware-nsx. We would like to distinguish that these are not supported/managed by the octavia team. | 23:12 |
| johnsom | Would just need to setup a "third party" pipeline? | 23:13 |
| clarkb | johnsom: isn't that third party ci? | 23:13 |
| clarkb | (maybe I've latched onto the vmware testing which I don't think we can do upstream too much) | 23:14 |
| johnsom | The difference is it's already an opendev job, so the trigger would be our gerrit as opposed to an external system listening and sending results back. | 23:14 |
| fungi | jobs run in our ci system, non-voting but also somehow visually separate from "official" non-voting jobs | 23:14 |
| johnsom | Right | 23:15 |
| mordred | interesting | 23:15 |
| johnsom | Both cases are current/former neutron sub-projects | 23:15 |
| johnsom | They implement drivers for Octavia (among other things there) | 23:16 |
| mordred | clarkb, corvus: I cannot see or hear anything | 23:16 |
| fungi | technically it would be possible to have a separate pipeline triggered like check but which didn't vote... or which voted on a new, different gerrit review label | 23:16 |
| mordred | in case there is anything to see or hear | 23:16 |
| clarkb | mordred: your browser should ask you for permission to use your mic and or camera | 23:17 |
| mordred | yup. it did - and I granted it - and my camera light is illuminated | 23:17 |
| clarkb | mordred: if you don't allow it then it has a sad, if it didn't do that click in the icon area left of url bar and see if you can tell it to do the things that way | 23:17 |
| clarkb | ah | 23:17 |
| johnsom | fungi Yeah, that sounds like what we would like. Basically an opendev third party. lol | 23:17 |
| clarkb | neat | 23:17 |
| clarkb | fungi: ya I think thats the trick is it can't vote or we'd need a separte account | 23:18 |
| corvus | johnsom, fungi: it's possible, but to really make it work we need another gerrit account, so it's quite a bit of work. | 23:18 |
| corvus | we abandoned an earlier similar effort because it didn't seem worth it at the time | 23:18 |
| fungi | johnsom: any reason why just having them be normal non-voting jobs is a problem? | 23:18 |
| johnsom | I am fine with "can't vote" for now. I don't see that happening anytime in the near future. | 23:18 |
| clarkb | mordred: we can see you fwiw | 23:18 |
| mordred | cool | 23:19 |
| mordred | I can see myself in the settings popup | 23:19 |
| fungi | lots of projects have non-voting jobs which they consider purely advisory | 23:19 |
| mordred | but I can't see myself in my little window on the main page | 23:19 |
| mordred | there it is | 23:19 |
| johnsom | fungi Yeah, the concern is it doesn't distinguish them as third party, so doesn't give the clue that these aren't project team supported drivers. | 23:20 |
| johnsom | Since we don't own the job definition, we can't distinguish them using the name either. | 23:20 |
| fungi | corvus: i wasn't thinking as far as reporting under a separate account, we already isolate reports from separate pipelines in different comments (and group them in our javascript overlay, though no idea how the pipeline grouping translates to the checks api in 3.x) | 23:21 |
| fungi | but i agree it's still additional work to maintain | 23:21 |
| fungi | also no idea if that's visually distinct enough to address johnsom's concerns | 23:22 |
| clarkb | fungi: ya it could vote in a verified2 category | 23:27 |
| clarkb | or if we want it to vote in verified need a second account | 23:28 |
| fungi | it could just not leave a +1 or -1 on any label and only report success or failure, right? | 23:28 |
| clarkb | I think that will clear our existing votes? | 23:29 |
| mordred | clarkb, corvus: http://paste.openstack.org/show/791263/ | 23:30 |
| fungi | oh, if it doesn't vote on a label that resets the vote, right | 23:30 |
| fungi | and if it did vote on a different label, that would need to somehow preserve any existing vote on the verified label | 23:30 |
| fungi | so yeah, i don't think it's actually an option after all | 23:31 |
| fungi | zuul would need some feature to know which labels needed to remain untouched when reporting from a particular pipeline and then reinclude the prior values if they're present | 23:32 |
| clarkb | fungi: I think it may only set values for the category it is told to write to | 23:33 |
| clarkb | so it may already ignore other pipelines | 23:33 |
| fungi | other labels you mean? | 23:33 |
| clarkb | ya | 23:33 |
| fungi | hrm, would need testing | 23:33 |
| johnsom | I am assuming this isn't testing I could do in our repo on a patch. Like I can't just define my own pipeline right? | 23:35 |
| clarkb | johnsom: correct, pipelines are "trusted" zuul config so have to go into a trusted repo | 23:36 |
| clarkb | johnsom: is the concern that you won't be able to turn those jobs off if they break? because if you define them in your repos on your project then you have full control of that | 23:36 |
| johnsom | Right, there are a couple of reasons behind the idea. One is to help clarify project ownership for the job. One is to make sure we can disable it should it become a problem. | 23:38 |
| corvus | even if they're not defined in your repo, you're still attaching them to your check pipeline in your repo, right? | 23:39 |
| johnsom | We already have a lot of "it has octavia in the name, so this is the team that owns that." kind of issues. | 23:39 |
| clarkb | johnsom: isn't that strongly determined by the repo the job lives in? | 23:40 |
| johnsom | No, this is a request to not included them in our check pipeline. We could do that today. This is about having a separate pipeline like third party ci | 23:40 |
| clarkb | (not where it is used, but where it is defined) | 23:40 |
| clarkb | johnsom: right and we are saying its difficult | 23:40 |
| corvus | clarkb, mordred: i'm going to manually stick a cors header in etherpad okay? | 23:40 |
| clarkb | and not possible today | 23:40 |
| clarkb | corvus: ok | 23:40 |
| corvus | johnsom: i'm just trying to understand -- i think you said that today you can't turn off the jobs, but i don't understand why? aren't they attached to your project's check pipeline in your repo? isn't it a matter of deleting one line? | 23:41 |
| clarkb | johnsom: because for zuul to report to gerrit it has to report a vote value, and that would clear out existing values if it say voted +0 after check voted +1 | 23:41 |
| johnsom | Right, but there is no indication that it's owned by another team, where a separate visual/labeled section would help with that. | 23:41 |
| openstackgerrit | Merged opendev/jeepyb master: More python3 fixes https://review.opendev.org/715527 | 23:42 |
| fungi | i mean, technically the openstack-tox-py37 job is owned by another team too, right? | 23:42 |
| mordred | ZOMG the jeepyb patch landed | 23:43 |
| johnsom | Yes, but we can and are on the hook to support / fix that. For third party CI jobs we are not. | 23:43 |
| johnsom | Nor do we want to be. | 23:43 |
| fungi | johnsom: it should be possible to "rename" the jobs when you include them in your project-pipeline by just creating a child job parented to it | 23:43 |
| clarkb | johnsom: if I'm parsing correctly I think what you mean by support is "when this job stops successfully running reliably we fix it" | 23:43 |
| clarkb | and today that list is all of your check jobs | 23:43 |
| johnsom | clarkb Correct | 23:44 |
| corvus | fungi: yep, should be a one-liner inheritance. could name it "unsupported-third-party-plugin-test-thingy" | 23:44 |
| corvus | etherpad restarted | 23:44 |
| johnsom | Hmm, that is actually not a half bad idea. It doesn't group them like the other third party CI jobs would, but would make it clear. | 23:45 |
| mordred | also - zuul doesn't vote until all of the jobs in a pipeline are done. so I could also imagine wanting to say "jobs in check vs jobs in shrug" and the shrug jobs have no vote but have final status so you can see them - but no activity is blocked waiting on them - right? | 23:45 |
| corvus | johnsom: jobs in pipelines are ordered, so you can at least sort them together within that one group | 23:45 |
| corvus | mordred: yes, that's the other trade-off | 23:45 |
| mordred | corvus: would the existence of checks plugin in the future allow a second pipeline in opendev zuul that only left checks and didnt' leave a vote? | 23:45 |
| corvus | mordred: it's more or less the same issue, pipelines vote or don't | 23:46 |
| mordred | nod | 23:46 |
| johnsom | Right, it would be nice to go on with normal life and if these go into endless loops (Like the OVN driver did) it wouldn't stop our progress. | 23:46 |
| fungi | bonus for renaming them with child jobs: you can add an extra couple of lines to set your preferred timeouts | 23:47 |
| johnsom | lol, 10 minutes | 23:48 |
| *** ysandeep is now known as ysandeep|rover | 23:48 | |
| clarkb | corvus: I think I figured out why the monitor is the default now too. its the always on virtual device then when I plug and unplug things I don't have to change settings | 23:50 |
| clarkb | except it is muted by default so useless by default :) | 23:50 |
| clarkb | (though from a privacy perspective that is a good default, justh ave to be aware of it | 23:50 |
| johnsom | So, this is probably workable for us. Doing the parent/child thing. People might get worked up, but tough. However, I think the second pipeline might be the *right* longer term solution. Is there an RFE I can write up to track that? | 23:51 |
| mordred | clarkb, corvus, fungi I | 23:51 |
| mordred | clarkb, corvus, fungi I'm going to try manage-projects again | 23:52 |
| clarkb | johnsom: if zuul needs updates for that (I'm not sure) then a spec there may make sense? | 23:52 |
| clarkb | if zuul is good to go (probably via second gerrit connection) then an infra-spec to createa a second gerrit connection with a second user would work | 23:52 |
| johnsom | I think designate might end up in the same situation with all of the alternate drivers if they decide to move them out of repo as well. | 23:52 |
| clarkb | personally I think that you shouldn't run those jobs if they don't provide value | 23:53 |
| clarkb | and if they do provide value then fixing them when they break is in scope | 23:53 |
| mordred | clarkb, fungi, corvus: I'm in a screen on review.o.o if anybody wants to watch along | 23:53 |
| clarkb | mordred: I see the command prompt | 23:53 |
| mordred | I have already tailed the log file in the background, fwiw | 23:54 |
| clarkb | johnsom: and we do a lot of cross project testing in here. Some of it fails (ansible-devel) some of it doesnt | 23:54 |
| mordred | nothing failed | 23:54 |
| johnsom | I think the other teams just don't want to use periodic jobs to test their driver against the tip of octavia. | 23:54 |
| clarkb | mordred: looks like it did a lot of nothing as expected? | 23:55 |
| mordred | clarkb: I think that means we're good to go on re-enabling manage-projects | 23:55 |
| mordred | yup | 23:55 |
| clarkb | mordred: k, I'm starting to enter the time of day where I'm not in a good spot to help babysit as we're cooking potatos and those need a headstart | 23:55 |
| mordred | clarkb, johnsom: I think there may be a general use case here which I've thought about a few times but don't have actual thoughts on what it means yet - which is the capability for a project to "subscribe" to another project - without necessarily reporting back to them | 23:55 |
| johnsom | Really I guess the value is no different than the third party CI, it's just in this case they are semi-third party. | 23:56 |
| clarkb | johnsom: they can test the other direction, but I guess if their change volume is low they have the problem where they don't catch it until too late | 23:56 |
| clarkb | mordred: ya that is kind of what silent is/was | 23:56 |
| johnsom | Right, our velocity is much higher than theirs | 23:57 |
| mordred | so - like - it's important for octavia-vmare that vmware jobs run on octavia patches, but not to the octavia team - so if the octavia-vmware team could say "we want to run these jobs to verify our driver - and have them report to XXX" - that is a pattern that isn't really possible today but is a relationship in the human world that we have with various folks | 23:57 |
| clarkb | mordred: we ran the job on ever change but you had to go look for them | 23:57 |
| mordred | yeah - but you still even have to go ask the octavia team for permission to run the job - or the opendev team | 23:57 |
| clarkb | extending that so that results end up where you want them and aren't completely silent would be a good thing | 23:57 |
| clarkb | mordred: oh thats a good poin too | 23:57 |
| mordred | you can't say, in the octavia-vmware repo, I want to run this job on this input - partially because there's no good place to report | 23:57 |
| mordred | but if you treated it LIKE a periodic - only reported to the dashboard | 23:57 |
| mordred | it could still be a nice thing to allow people to do to track things where the care is unidirectional | 23:58 |
| mordred | *massive hand wave here* | 23:58 |
| clarkb | I think silent would actually get you halfway there | 23:58 |
| clarkb | johnsom: with the silent pipeline we'd report to zuul's db on every change but not to gerrit | 23:58 |
| mordred | clarkb, fungi, corvus: so - I think I'm going to remove review from emergency | 23:59 |
| clarkb | johnsom: then the driver folks could query zuul for those results. This is similar to periodic except it runs on every change | 23:59 |
| clarkb | mordred: ok, you'll keep an eye on it? | 23:59 |
| clarkb | I guess mostly just to see that the first pulse is happy as it was manually | 23:59 |
| clarkb | (and all the associated ansible stuff) | 23:59 |
| mordred | yeah. or - actually - I'm about to dinner - so I'll remove it from emergency next time I'm ready to watch it :) | 23:59 |
| fungi | mordred: yay! looks good | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!