| *** tosky has quit IRC | 00:00 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: chart-testing-lint-single: new job https://review.opendev.org/715665 | 00:03 |
|---|---|---|
| *** factor has joined #opendev | 00:43 | |
| mnaser | ok, so: i feel awful about the extra (small) workload as we're doing more opendev-y things but i'm wondering what's the "delete a project" story like right now | 00:53 |
| mnaser | background: i just realized that everything about helm and their charts are really meant to be a "repository" of charts, and potentially having "one chart" per repo is something that is a little weird in the entire ecosystem | 00:53 |
| mnaser | so i'm just wondering if it makes sense to just create a vexxhost/helm-charts and manage everything there in a monorepo, which absolutely sucks, but maybe it might be "the way" unfortunately | 00:54 |
| fungi | the delete story right now is to delete the content and set the acl to read-only | 00:56 |
| fungi | so that it doesn't accept change proposals | 00:56 |
| mnaser | fungi: ah, i see, so still leaves some "artifacts" at the moment. | 00:58 |
| mnaser | i am trying to look at ways at making a chart-per-repo happen right now still. it's so silly. | 00:58 |
| mnaser | just a heads-up, the acl issue is there and i don't have access, so we may not want to land any new project creations -- https://review.opendev.org/#/admin/projects/vexxhost/smokeping_prober-helm,access | 01:18 |
| fungi | yeah, i'm going to see what removing it from the jeepyb cache does | 01:36 |
| fungi | okay, i've backed up and then manually removed the entry for that project from /opt/lib/jeepyb/project.cache | 01:55 |
| fungi | it looks like it thought it had already created the project earlier and so wasn't trying to update the acl once we fixed things | 01:56 |
| fungi | which didn't seem to help, i'll try to take a closer look at the logs tomorrow | 03:57 |
| *** diablo_rojo has quit IRC | 06:40 | |
| *** DSpider has joined #opendev | 07:43 | |
| *** tosky has joined #opendev | 10:52 | |
| *** DSpider has quit IRC | 13:52 | |
| fungi | testing a theory. reviewing the acl update code in manage_projects.py it looks like a nonexistent acl sha in the cache file and an acl sha seems to result in it "matching" in this case and getting skipped, so i've manually inserted an incorrect sha for this repo to see what it does | 14:02 |
| fungi | there may be a problem with the acl cache | 14:03 |
| fungi | somehow this was matching even though there was no acl sha recorded in the project cache for that repo: https://opendev.org/opendev/jeepyb/src/branch/master/jeepyb/cmd/manage_projects.py#L552 | 14:08 |
| fungi | since there was no acl sha in the cache, project_cache[project].get('acl-sha') returns none | 14:08 |
| fungi | which means acl_cache.get(acl_config) must also be returning none | 14:08 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Really bindmount acls https://review.opendev.org/715697 | 14:10 |
| mordred | fungi: ^^ | 14:10 |
| mordred | yeah - our fix for that wasn't quite complete | 14:11 |
| fungi | aha! | 14:11 |
| mordred | so - yes - you were definitely on the right track | 14:11 |
| fungi | i'll undo my local edit to the project cache so we can run that through its paces | 14:11 |
| mordred | cool | 14:11 |
| mordred | also - fwiw - there are errors in the ansible log related to first_found and our iptables rules | 14:12 |
| fungi | we already seem to set errors='ignore' on that task | 14:19 |
| mordred | *awesome* | 14:19 |
| fungi | is it ignoring our ignore? | 14:19 |
| mordred | maybe? | 14:20 |
| mordred | maybe it's still warning us regardless | 14:20 |
| fungi | with a confusing error saying to add a parameter which we already add | 14:20 |
| mordred | yeah | 14:26 |
| *** DSpider has joined #opendev | 14:27 | |
| openstackgerrit | Merged opendev/system-config master: Really bindmount acls https://review.opendev.org/715697 | 15:38 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: helm: collect kubernetes logs in post https://review.opendev.org/715709 | 15:40 |
| AJaeger | https://review.opendev.org/#/admin/projects/vexxhost/smokeping_prober-helm,access is not updated, did ansible run with the fix? | 16:40 |
| fungi | i've been watching, last manage-projects run occurred at 16:04 utc, which may have been too soon to incorporate the new docker config | 16:40 |
| AJaeger | change merged at 15:38 utc AFAIU, so shouldn't it be there? Thanks for watching - and no urgency from my part ;) | 16:42 |
| fungi | i'm not clear on whether it needs to be merged before the start of the ansible run, and when ansible started. i'll check the cron log | 16:43 |
| fungi | --- begin run @ 2020-03-29T15:30:01+00:00 --- | 16:44 |
| fungi | --- begin run @ 2020-03-29T16:30:01+00:00 --- | 16:44 |
| fungi | so it merged after the ansible run which was responsible for the 16:04 manage-projects | 16:45 |
| fungi | expecting around 15 minutes before we'll know if this has solved things | 16:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add constraints support to python-builder https://review.opendev.org/713972 | 16:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Fix siblings support in python-builder https://review.opendev.org/715717 | 16:57 |
| mordred | mnaser: ^^ I thnik those two will be needed for the general container building case to work properly - would love double-checking my logic there | 16:58 |
| fungi | it's underway now | 17:03 |
| fungi | should know in another minute or two | 17:04 |
| mordred | woot | 17:07 |
| fungi | it seems like it's taking a good deal longer than before | 17:07 |
| mordred | maybe that's a good sign | 17:07 |
| *** elod has quit IRC | 17:08 | |
| fungi | maybe. does appear to be cloning every repo though | 17:09 |
| mordred | fungi: that seems a bit extreme - where is it cloning them to? | 17:12 |
| mordred | (wondering if we missed a bindmount) | 17:12 |
| mordred | fungi: looks like it's cloning to /opt/lib/jeepyb like it's supposed to | 17:14 |
| fungi | but should it do that every time it runs? | 17:14 |
| mordred | hrm | 17:14 |
| mordred | it does't seem like /opt/lib/jeepyb has copies of all of these repos | 17:14 |
| fungi | i thought the point of the git cache was that it would reuse those on subsequent runs and just fetch updates for refs/meta/config | 17:15 |
| mordred | yeah | 17:15 |
| mnaser | mordred: ok neat, i'm hoping to have time to get back on hacking using those images soon (hopefully). | 17:15 |
| mordred | but check out /opt/lib/jeepyb/openstack | 17:15 |
| fungi | but yes, they seem to get cleared out between each run (or at least they were all empty when i looked earlier) | 17:15 |
| mordred | yeah | 17:15 |
| mordred | mnaser: same | 17:15 |
| fungi | it's working on cloning nova now | 17:15 |
| fungi | been cloning nova for nearly two minutes, which is unsurprising | 17:16 |
| mordred | yeah. Im a little surprised at the behavior tbh | 17:17 |
| mordred | I wish we had a log of _why_ we think we need to clone it :) | 17:17 |
| mordred | ok - first step of process_acls is to do this | 17:18 |
| mordred | but this now makes me wonder if the acl_cache erroneuously has None entries for the acl_sha for a bunch of things | 17:18 |
| mordred | fungi: acl-sha seems to be mostly null in the project cache | 17:19 |
| mordred | so I think we might, for this run, be re-pulling all of the acls so that we can verify an acl match and then re-write a new sha to the cache file | 17:20 |
| fungi | quite possible if it cleared them all out because it was previously run with an empty acl condfig | 17:20 |
| mordred | yeah | 17:20 |
| mordred | so this run might take a minute | 17:21 |
| mordred | :) | 17:21 |
| fungi | https://review.opendev.org/#/admin/projects/vexxhost/smokeping_prober-helm,access | 17:35 |
| fungi | yay! | 17:35 |
| fungi | looking good | 17:35 |
| fungi | should i go ahead and approve 713809 now or is there anything else we need to check first? | 17:36 |
| fungi | (re)approve i mean | 17:37 |
| mordred | fungi: I think go for it | 17:38 |
| fungi | bam | 17:39 |
| openstackgerrit | Merged openstack/project-config master: Added new project openstack-tempest-skiplist https://review.opendev.org/713809 | 17:46 |
| mordred | fungi: it should be noted that github mirror creation is also not running | 17:50 |
| fungi | maybe this is the time to start considering in earnest migrating the openstack namespace to git ref replication jobs | 17:51 |
| fungi | or at least take a step toward it. in that scenario, the github caretakers for openstack would need to manually create the repo in gh anyway | 17:52 |
| fungi | but being able to turn off github integration in jeepyb and gerrit is becoming increasingly attractive | 17:52 |
| *** elod has joined #opendev | 17:53 | |
| fungi | analyzing the manage-projects log, typical runs take 2 seconds if a no-op and 5 seconds if there's a new project created. that last run took just over 33 minutes | 18:06 |
| fungi | hopefully the next one will be back in the 2-5 second range | 18:06 |
| mordred | yeah | 18:07 |
| mordred | (to all of the above) | 18:09 |
| mnaser | yay, i can merge my own code again | 18:20 |
| mnaser | :p | 18:20 |
| mnaser | fungi, mordred: thanks for the work on this | 18:20 |
| fungi | mnaser: thanks for the patience! | 18:34 |
| fungi | 18:34 completed in just under 11 seconds | 18:38 |
| fungi | i guess because it also pulled a remote repo for import | 18:38 |
| fungi | https://opendev.org/openstack/openstack-tempest-skiplist exists but https://review.opendev.org/#/admin/projects/openstack/openstack-tempest-skiplist,access does not | 18:39 |
| fungi | Exception: Gerrit error executing gerrit ls-groups -v -q "openstack-tempest-skiplist-core" | 18:41 |
| fungi | looks like new group creation may not be working yet | 18:41 |
| fungi | from memory, the order of operations is that manage-projects creates the group, then tries to poll multiple times for the group because gerrit returns from the api call before the group exists | 18:56 |
| fungi | and manage-projects needs to obtain the group uuid to add to the config | 18:57 |
| fungi | (in refs/meta/config of the repo) | 18:57 |
| clarkb | it may not poll anymore? | 18:57 |
| clarkb | that was something we changed ti make this testable, removed direct db access | 18:57 |
| fungi | well, that's the polling which raiswd the exception | 18:57 |
| clarkb | ah ok | 18:57 |
| fungi | oh, wait | 18:58 |
| fungi | you may be right | 18:58 |
| fungi | it was checking for group creation via db access, and this is the uuid lookup which raised an exception | 18:58 |
| fungi | so maybe we just need to retry that? | 18:58 |
| clarkb | it shouldnt do db access anymore | 18:58 |
| fungi | yeah, which may be why we're now losing this race | 18:59 |
| clarkb | oraybe thats what you meant? and ya retrying there seems reasonable | 18:59 |
| fungi | though https://review.opendev.org/#/admin/groups/?filter=openstack-tempest-skiplist-core still returns nothing | 19:00 |
| fungi | so maybe the group creation command failed, or never happened? | 19:00 |
| fungi | we don't seem to log it | 19:00 |
| fungi | ahh, this is within the retry | 19:10 |
| fungi | for x in range(retries): | 19:10 |
| fungi | group_list = list(gerrit.listGroup(group, verbose=True)) | 19:11 |
| fungi | but gerritlib is raising an exception in there | 19:11 |
| fungi | so the loop bails on the first iteration | 19:11 |
| mordred | is this one where wrapping the call in list() may have broken it? | 19:20 |
| mordred | oh - I guess not if we get that exception message | 19:21 |
| openstackgerrit | Monty Taylor proposed opendev/jeepyb master: Trap for exception in listGroup https://review.opendev.org/715723 | 19:23 |
| mordred | fungi, clarkb : what about something like that ^^? | 19:23 |
| fungi | yeah, sorry, got sidetracked by tasty thai takeout leftovers | 19:23 |
| mordred | mmm. tasty thai | 19:23 |
| fungi | mordred: but the group was never created | 19:23 |
| fungi | so i don't think we've (yet) ruled out a problem in the group creation command itself | 19:24 |
| mordred | ah - nod | 19:24 |
| fungi | also i'm frosting the chocolate orange cake i baked earlier, so will still be a few minites | 19:25 |
| fungi | minutes | 19:25 |
| mordred | nod | 19:25 |
| fungi | so this is happening in process_acls() and we call create_groups_file() where the exception is getting raised before we push_acl_config() which is, i think, what should trigger gerrit to create the groups, right? | 19:37 |
| fungi | did something change around that recently? | 19:38 |
| clarkb | no, you have to create the group before pushing the file because the file has to have the uuid in it | 19:38 |
| fungi | ahh, yeah, so there must be an api call somewhere doing that step | 19:42 |
| fungi | oh, it's in get_group_uuid() not _get_group_uuid() | 19:43 |
| fungi | i misread the backtrace and missed the wrapper | 19:43 |
| fungi | so looking in get_group_uuid() the gerritlib exception is happening before we ever get to the gerrit.createGroup() call | 19:45 |
| fungi | maybe the exception is new? the flow there seems to imply that we expect _get_group_uuid() to return a falsey value if the group doesn't exist, rather than raise an exception | 19:46 |
| clarkb | ya maybe a python3 related change? | 19:50 |
| fungi | should we fix this in gerritlib or work around it by catching the exception in jeepyb? | 20:10 |
| fungi | i guess if we do it in gerritlib we need another release, right? | 20:10 |
| clarkb | ya | 20:11 |
| clarkb | with the last thing wefixed it in both places and can remove the jeepyb fix one arelease for gerritlib happens | 20:11 |
| openstackgerrit | Jeremy Stanley proposed opendev/jeepyb master: Catch exceptions when checking for groups https://review.opendev.org/715726 | 20:45 |
| fungi | mordred: clarkb: what do you think of trying that ^ mitigation, and then we correct in gerritlib and revert if it works | 20:46 |
| clarkb | fungi: ya that seems fine. Im about to hop on the bike so in a bad spot to record it in gerrit | 20:47 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: Revert "Revert "Extract pep8 messages for inline comments"" https://review.opendev.org/715727 | 21:09 |
| *** hashar has joined #opendev | 21:32 | |
| *** hashar has quit IRC | 21:39 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add Fedora 31 support and test jobs https://review.opendev.org/708416 | 21:47 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: DNM: making sure pep8 inline comments don't break https://review.opendev.org/715733 | 21:52 |
| openstackgerrit | Piotr Kopec proposed openstack/project-config master: Add new project and repository for tripleo-compute-extras https://review.opendev.org/715734 | 21:53 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: Revert "virtualenv-config: add to base pre playbook" https://review.opendev.org/715735 | 21:56 |
| *** DSpider has quit IRC | 21:58 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: Revert "Revert "Extract pep8 messages for inline comments"" https://review.opendev.org/715727 | 22:03 |
| openstackgerrit | Piotr Kopec proposed openstack/project-config master: Add new project and repository for tripleo-compute-extras https://review.opendev.org/715734 | 22:12 |
| *** smcginnis has quit IRC | 22:19 | |
| *** smcginnis has joined #opendev | 22:19 | |
| fungi | i've proxy +2'd 715726 on clarkb's behalf and approved it | 22:24 |
| fungi | will keep an eye on manage-projects to see if that gets us the rest of the way | 22:24 |
| openstackgerrit | Piotr Kopec proposed openstack/project-config master: Add new project and repository for tripleo-compute-extras https://review.opendev.org/715734 | 22:27 |
| *** tosky has quit IRC | 22:54 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: Revert "Revert "Extract pep8 messages for inline comments"" https://review.opendev.org/715727 | 23:08 |
| openstackgerrit | Merged opendev/jeepyb master: Catch exceptions when checking for groups https://review.opendev.org/715726 | 23:14 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: Revert "Revert "Extract pep8 messages for inline comments"" https://review.opendev.org/715727 | 23:22 |
| mordred | mnaser: your dnm testing patch didn't produce any pep8 errors :) | 23:45 |
| mnaser | mordred: not sure if my message made it but it was hopefully meant to pass and catch the warning failure | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!