| *** zaro has quit IRC | 00:17 | |
| *** zaro has joined #opendev | 00:18 | |
| *** DSpider has quit IRC | 00:36 | |
| *** ysandeep|away is now known as ysandeep | 03:30 | |
| *** raukadah is now known as chandankumar | 04:07 | |
| *** ykarel has joined #opendev | 04:39 | |
| *** marios has joined #opendev | 06:11 | |
| *** hemanth_n has joined #opendev | 06:49 | |
| *** lpetrut has joined #opendev | 07:08 | |
| *** eolivare has joined #opendev | 07:11 | |
| frickler | we should recycle #openstack-chef, it has become very quiet in there otherwise | 07:24 |
|---|---|---|
| frickler | and can't be too much offtopic, either, I regulary get job offers as pizza baker or similar, just for mentioning "experience with Chef" in my resume | 07:25 |
| openstackgerrit | daniel.pawlik proposed openstack/diskimage-builder master: Install last stable version of get-pip.py script https://review.opendev.org/c/openstack/diskimage-builder/+/772254 | 07:51 |
| *** ralonsoh has joined #opendev | 07:55 | |
| dpawlik | frickler: hey, hope this https://review.opendev.org/c/openstack/diskimage-builder/+/772254 will fix issue with building new images that uses python 2 | 07:57 |
| *** fressi has joined #opendev | 07:57 | |
| *** andrewbonney has joined #opendev | 08:19 | |
| *** rpittau|afk is now known as rpittau | 08:35 | |
| *** ysandeep is now known as ysandeep|lunch | 08:39 | |
| *** sboyron has joined #opendev | 08:39 | |
| *** DSpider has joined #opendev | 08:56 | |
| *** jpena|off is now known as jpena | 08:56 | |
| *** zul has joined #opendev | 09:08 | |
| *** ysandeep|lunch is now known as ysandeep | 09:23 | |
| *** tosky has joined #opendev | 09:23 | |
| *** marios has quit IRC | 09:26 | |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Increase autogenerated comment width to avoid line wrap https://review.opendev.org/c/opendev/system-config/+/771445 | 09:50 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 09:50 |
| openstackgerrit | daniel.pawlik proposed openstack/diskimage-builder master: Install last stable version of get-pip.py script https://review.opendev.org/c/openstack/diskimage-builder/+/772254 | 09:52 |
| *** marios has joined #opendev | 10:12 | |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Increase autogenerated comment width to avoid line wrap https://review.opendev.org/c/opendev/system-config/+/771445 | 10:21 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 10:21 |
| *** sshnaidm is now known as sshnaidm|ruck | 10:39 | |
| *** hashar has joined #opendev | 11:03 | |
| openstackgerrit | daniel.pawlik proposed openstack/diskimage-builder master: Install last stable version of get-pip.py script https://review.opendev.org/c/openstack/diskimage-builder/+/772254 | 11:09 |
| *** dtantsur|afk is now known as dtantsur | 11:11 | |
| *** zbr is now known as notify | 11:41 | |
| *** notify has quit IRC | 11:46 | |
| *** notify has joined #opendev | 11:46 | |
| *** ysandeep is now known as ysandeep|afk | 11:46 | |
| *** sboyron has quit IRC | 12:16 | |
| *** sboyron has joined #opendev | 12:17 | |
| *** bodgix_ has quit IRC | 12:19 | |
| *** bodgix has joined #opendev | 12:19 | |
| *** dirk has quit IRC | 12:21 | |
| *** dirk has joined #opendev | 12:21 | |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add swift-proxy repo in vexxhost tenant https://review.opendev.org/c/openstack/project-config/+/772331 | 12:23 |
| *** jpena is now known as jpena|lunch | 12:25 | |
| frickler | did someone change colors on the gerrit diff view or is it just me suddenly seeing something like yellow/purple instead of red/green for removed/added? | 12:32 |
| frickler | oh, wait, this likely isn't new and only seems to happen when comparing different PSs ... very weird still | 12:37 |
| *** hemanth_n has quit IRC | 12:38 | |
| *** ysandeep|afk is now known as ysandeep | 12:44 | |
| *** notify is now known as zbr | 13:16 | |
| *** jpena|lunch is now known as jpena | 13:31 | |
| dpawlik | frickler: I understand that support python2 is difficult, but it is still available and images are using it so maybe we could fix the the DIB for now: https://review.opendev.org/c/openstack/diskimage-builder/+/772254 ? | 13:34 |
| dpawlik | cc fungi ^^ (thanks in advance for the review) | 13:35 |
| *** dirk has quit IRC | 13:51 | |
| *** dirk has joined #opendev | 13:51 | |
| *** iurygregory_ has joined #opendev | 13:59 | |
| *** zbr4 has joined #opendev | 13:59 | |
| *** iurygregory has quit IRC | 14:00 | |
| *** iurygregory_ is now known as iurygregory | 14:00 | |
| *** Alex_Gaynor has joined #opendev | 14:00 | |
| Alex_Gaynor | I'm seeing ARM64 jobs not starting. | 14:00 |
| *** zbr has quit IRC | 14:01 | |
| *** zbr4 is now known as zbr | 14:01 | |
| frickler | kevinz: still lots of error node attempts and next to no nodes in use https://grafana.opendev.org/d/pwrNXt2Mk/nodepool-linaro?orgId=1&from=now-7d&to=now | 14:08 |
| frickler | Alex_Gaynor: yes, seems to be an ongoing issue with linaro | 14:08 |
| *** ykarel_ has joined #opendev | 14:09 | |
| *** ykarel has quit IRC | 14:11 | |
| *** lbragstad has joined #opendev | 14:18 | |
| *** ykarel__ has joined #opendev | 14:19 | |
| *** ykarel_ has quit IRC | 14:21 | |
| *** lbragstad has quit IRC | 14:34 | |
| *** lbragstad has joined #opendev | 14:37 | |
| *** ykarel__ is now known as ykarel | 14:40 | |
| openstackgerrit | sebastian marcet proposed opendev/system-config master: OpenstackId v3.0.17 https://review.opendev.org/c/opendev/system-config/+/772348 | 14:43 |
| openstackgerrit | Maksim Malchuk proposed openstack/diskimage-builder master: Fix hooks order for CentOS/Fedora when mirror used https://review.opendev.org/c/openstack/diskimage-builder/+/772350 | 14:46 |
| *** d34dh0r53 has quit IRC | 14:48 | |
| *** d34dh0r53 has joined #opendev | 14:54 | |
| *** artom has joined #opendev | 14:57 | |
| sshnaidm|ruck | hi, is problem with retry_limits known? like https://zuul.opendev.org/t/openstack/status/change/765624,8 | 14:58 |
| sshnaidm|ruck | problem with some of clouds? | 14:58 |
| *** hashar has quit IRC | 14:59 | |
| fungi | sshnaidm|ruck: we turned inap back on again, so we should probably look into whether it's back to the same old problem we unhooked it for the last several times (ip address conflicts from rogue vms) | 15:06 |
| sshnaidm|ruck | fungi, ack | 15:07 |
| sshnaidm|ruck | fungi, is it possible to see from web interface which cloud retry limits are on? | 15:07 |
| fungi | if they uploaded logs, then yes | 15:08 |
| sshnaidm|ruck | fungi, maybe worth to publish here: https://zuul.opendev.org/t/openstack/build/b4886413c6044bbfbdcf114658f7de14/console | 15:09 |
| fungi | if they didn't, we likely have to go hunting in service debugging logs | 15:09 |
| fungi | sshnaidm|ruck: also the cloud name is embedded in the hostname | 15:09 |
| fungi | but yeah, there's no node detail recorded in the database which that dashboard pulls from, i don't think | 15:10 |
| *** lpetrut has quit IRC | 15:10 | |
| fungi | we normally rely on checking the zuul inventory which gets included with the log upload | 15:10 |
| fungi | i'll see what nodes got used by that build id | 15:11 |
| fungi | node name was centos-8-inap-mtl01-0022703164 | 15:13 |
| fungi | so yep, it ran there | 15:13 |
| fungi | terminated with "Ansible complete, result RESULT_UNREACHABLE code None" | 15:14 |
| fungi | rough estimate we've logged ~2k RESULT_UNREACHABLE states since executor logs rotated at 06:25 utc today | 15:18 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Revert "Revert "Revert "Revert "Temporarily stop booting nodes in inap-mtl01"""" https://review.opendev.org/c/openstack/project-config/+/772355 | 15:20 |
| fungi | config-core: ^ let's stop using inap again while we see what additional details we can collect from our logs this time | 15:20 |
| mnaser | fungi: +2 | 15:21 |
| *** ysandeep is now known as ysandeep|dinner | 15:24 | |
| *** ykarel has quit IRC | 15:27 | |
| Alex_Gaynor | frickler: thanks. Is there anyway to track the status of the incident? (I guess that's really a question for the folks at linaro) | 15:37 |
| fungi | Alex_Gaynor: kevinz is really our only contact there as far as i'm aware. usually we try to reach him in here and by e-mail to get him to look into whatever's going on in there. i can also try to see if we get any useful details out of their api with the boot failure messages | 15:41 |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add swift-proxy in vexxhost tenant https://review.opendev.org/c/openstack/project-config/+/772364 | 15:43 |
| fungi | infra-root: processing the executor debug logs to map build results to providers, i see that 80% of the RESULT_UNREACHABLE states logged on ze01 since 06:25 have been for inap-mtl01 | 15:43 |
| fungi | i haven't checked the other 11 executors for comparison, but expect it's probably a representative sample | 15:43 |
| clarkb | fungi: hrw as well iirc as a linaro contact (though not on the cloud side directly) | 15:44 |
| clarkb | fungi: disabling inap seems fine then and maybe we get dansmith and melwitt in contact with them? | 15:44 |
| clarkb | fungi: btw I've learned about https://gerrit-review.googlesource.com/Documentation/rest-api-config.html#check-consistency via discussion on my gerrit repo-discuss thread | 15:46 |
| clarkb | I'm going to fiddle with that on review-test, and try and get a sense for what sort of changes we need to make | 15:46 |
| fungi | oh neat | 15:46 |
| openstackgerrit | Merged openstack/project-config master: Revert "Revert "Revert "Revert "Temporarily stop booting nodes in inap-mtl01"""" https://review.opendev.org/c/openstack/project-config/+/772355 | 15:47 |
| fungi | Alex_Gaynor: kevinz: yeah, the most we get from nodepool is that it gives up waiting for the boot request to return a reachable instance and tries again, then after the third time of the same gives up and rejects the node request, and since there is no other provider to fulfil a request for that node type zuul returns NODE_ERROR | 15:53 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Install get-pip.py for python3.5/xenial with specific url https://review.opendev.org/c/opendev/system-config/+/772369 | 15:55 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: WIP Deploy refstack with ansible docker https://review.opendev.org/c/opendev/system-config/+/705258 | 15:57 |
| *** brinzhang_ has quit IRC | 16:04 | |
| Open10K8S | Hi team. Can you check this PS please? https://review.opendev.org/c/openstack/project-config/+/772364 other stuff are waiting this :) | 16:09 |
| *** mlavalle has joined #opendev | 16:10 | |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Increase autogenerated comment width to avoid line wrap https://review.opendev.org/c/opendev/system-config/+/771445 | 16:12 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: [DNM] test comment width: review without autogenerated tag https://review.opendev.org/c/opendev/system-config/+/771798 | 16:13 |
| fungi | Open10K8S: how does that ref in the commit message relate to it? i'm a bit confused | 16:14 |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add swift-proxy in vexxhost tenant https://review.opendev.org/c/openstack/project-config/+/772364 | 16:16 |
| Open10K8S | fungi: sorry, i fixed | 16:16 |
| *** brinzhang has joined #opendev | 16:20 | |
| clarkb | ok consistency results for the test server have been generated and are in my homedir there in a file called consistency.results | 16:22 |
| *** ykarel has joined #opendev | 16:23 | |
| clarkb | we've got three types of issues: accounts without an externalid for their preferred email address, conflicting email addresses in external ids, and groups that loop their membership. This last class of problem is only affecting a small number of gruops so I think I'll start digging in there to keep scope small to start | 16:23 |
| *** brinzhang has quit IRC | 16:24 | |
| *** ysandeep|dinner is now known as ysandeep|away | 16:33 | |
| jrosser | can i ask about ipv6 addresses, i've got a focal node with an ipv6 address and a buster node without one, (https://zuul.opendev.org/t/openstack/build/e32feb9b42b54733b4894a7dcb9c1971/log/logs/openstack/instance-info/host_networkd_status_11-57-08.log.txt vs. https://zuul.opendev.org/t/openstack/build/0ebc81f393ae427494b2b6cfd17e90eb/log/logs/openstack/instance-info/host_networkd_status_11-46-01.log.txt) | 16:35 |
| jrosser | what would you expect to see here? | 16:35 |
| clarkb | jrosser: they ran in different clouds. One the cloud provides ipv6 and the other doesn't | 16:37 |
| clarkb | (technically bhs1 does provide ipv6 but not via ways our VMs can consume and configure it so its not really there from the test node perspective) | 16:37 |
| jrosser | oooh ok | 16:37 |
| openstackgerrit | Akihiro Motoki proposed openstack/project-config master: Preparation for tempest-horizon retirement https://review.opendev.org/c/openstack/project-config/+/772373 | 16:38 |
| clarkb | rm_work: johnsom: it appears the octavia-core and neutron-lbaas-core include each other whcih is a gerrit config consistency warning. Looking at the two groups I think the proper fix is to remove neutron-lbaas-core from octavia-core. neutron-lbaas-core has no members so including it in octavia-core doesn't do much. The other way around does make sure octavia-core members are in neutron-lbaas-core though | 16:39 |
| clarkb | rm_work: johnsom: any chance you can take a look at that and make that change if this is correct? | 16:39 |
| * clarkb makes this change on the test server | 16:41 | |
| johnsom | clarkb Agreed, octavia core as a member of neutron-lbaas-core is the correct answer. | 16:41 |
| clarkb | thanks for checking, if you make the change on the prod side as a group manager that woudl be great. I'll sync up my testing to ensure the complaints go away | 16:41 |
| johnsom | clarkb Done. Took a minute as I haven't had to do this since the gerrit change. lol | 16:43 |
| clarkb | johnsom: thanks! | 16:43 |
| *** ykarel has quit IRC | 16:46 | |
| fungi | zbr: looking back through the git-review changelog for things which have landed since the last release, it seems like we've missed adding release notes for some important things (particularly dropping support for python 2.7). i'm happy to push a quick set of updates to release notes but what else do you think should be incorporated? | 16:50 |
| clarkb | cool the other group that is an issue is networking-ibm-release including itself. I just removed it from itself and now check_groups consistency reports no errors on the test server | 16:50 |
| zbr | release notes are the only thing i know | 16:50 |
| clarkb | I have not modified ^ on the prod side, but can do that next if people think that is a reasonable thing to do? | 16:50 |
| zbr | there was a feature i was considering: filtering output from remote | 16:51 |
| fungi | zbr: er, i meant what other release notes do you think we need to add | 16:51 |
| fungi | `git log --oneline --no-merges 1.28.0..origin/master` shows a fair amount of stuff, some of which might warrant a note | 16:51 |
| * zbr digs into git log | 16:51 | |
| fungi | so far we just have a release note for the --notify option | 16:51 |
| clarkb | the other two classes of errors are going to be more difficult to address as they involve users and their usage of gerrit in a more "personal" way (email; login openids) | 16:52 |
| *** marios is now known as marios|out | 16:53 | |
| fungi | zbr: certainly dropping python 2.7 support, but also maybe notopic configuration, newer python (up to 3.8) testing, dropping draft support, commit hooks in submodules, push --no-follow-tags | 16:53 |
| clarkb | doing a quick skim of the other issues in some cases they are related because you'll have multiple accounts that don't have external ids for their preferred email and they are all the same email | 16:54 |
| clarkb | which means if we "fix" the lack of external id naively we'll create more of the email address conflicts | 16:54 |
| zbr | fungi: are you adding one, or should do it? | 16:55 |
| fungi | zbr: well, i'm asking whether you think any/all of those things warrant release notes. i can try to add notes for them but don't want to waste time on any which are not important enough to announce, nor do i want to miss important ones we want included | 16:57 |
| fungi | so sanity checking the list of things i mentioned against the 1.28.0..origin/master log would be appreciated | 16:58 |
| clarkb | I'm going to try disabling an account to see if gerrit ignores it for further consistency checking | 16:59 |
| *** marios|out has quit IRC | 16:59 | |
| clarkb | if that works one approach we may be able to take is simply disable the accounts, send email saying if your account stopped working please talk to us? | 16:59 |
| clarkb | anyway need to test that first (chances are it won't help based on previous observations) | 16:59 |
| zbr | i am doing it now. which versioning is going be next? 1.29.0, 2.0, or 21.0 (aka pip style) ? | 16:59 |
| clarkb | huh the rest api may not even have a method for disabling an account? | 17:00 |
| fungi | zbr: i was planning to do 2.0.0 or maybe 2.0.0.0a1 if you think announcing a prerelease on the ml first is a good idea | 17:01 |
| fungi | dropping python 2.7 support seems to me to be significant enough to warrant a major version increase | 17:01 |
| clarkb | oh nevermind tehy call it "active" | 17:01 |
| fungi | clarkb: yeah, active/inactive is what it's always been | 17:01 |
| fungi | clarkb: however, be forewarned that from what i've witnessed gerrit will still check inactive accounts for conflicts, so merely making conflicting accounts inactive likely doesn't help | 17:02 |
| fungi | though filtering for inactive accounts probably gives us an easy list of conflicts we can safely delete | 17:03 |
| clarkb | fungi: yup, I just want to double check it in the "no externalid id for this preferred address" case | 17:04 |
| clarkb | we know it doesn't help in the conflicts for external ids case | 17:04 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Install get-pip.py for python3.5/xenial with specific url https://review.opendev.org/c/opendev/system-config/+/772369 | 17:05 |
| clarkb | once I've confirmed it doesn't help, I'm going to dig into a couple of example and see if I can sort out if there is a pattern or not. I don't expect this to be quick and easy but probably our only real shot and sorting it out is to dig in :( | 17:05 |
| zbr | tbh, i do find the generated ChangeLog file good enough and I find no joy into expanding it into fragments. | 17:06 |
| clarkb | confirmed setting an account to inactive does not change the issue with missing externalid for preferred email address | 17:07 |
| fungi | zbr: the reason i asked is currently we seem to only publish the release notes not the commit-level changelog: https://docs.openstack.org/infra/git-review/index.html | 17:10 |
| zbr | cannot we change that to lower the amount of reno bureaucracy needed? -- that is a small project. | 17:12 |
| fungi | zbr: i'd be okay taking the use of reno back out, though maybe we should see if anyone else has found it especially useful. looks like mordred added the first ones in late 2018: https://review.openstack.org/593670 | 17:14 |
| fungi | we've only got a total of 8 release notes in there | 17:14 |
| fungi | though the missed-notes one contains four entries actually | 17:15 |
| fungi | so maybe ~10 notes | 17:15 |
| fungi | as a quick alternative, i can add a note about dropping python 2.7 support and call it a day. the other changes are not as significant for potential impact | 17:16 |
| mordred | zbr, fungi it's a small project, but it has a LOT of users, and it is typically slow to change significantly. that's why I thought doing explicit reno notes would be a good idea - having a solid way to communicate when something actually _does_ change seems important | 17:18 |
| fungi | yeah, i think if we want to keep relying on release notes there, we need to be more diligent about asking change submitters to also include release notes in their changes | 17:19 |
| fungi | i see probably 6 things since the last release which could have release notes but are missing them | 17:20 |
| mordred | ++ ... that said - if we're not using it, then maybe it doesn't make sense to keep it. but - in general, the change rate is slow enough maybe it's a good habit to develop? | 17:21 |
| clarkb | fungi: looking at the "preferred email ahs no external id case" since there are fewer of them than the externalid conflicts first. It almost looks like this is the set of users we have mabye merged into another user in the past. The ~4 I've checked so far have no external ids at all but have a user record with a preferred email set | 17:22 |
| clarkb | and some of them are inactive | 17:22 |
| clarkb | thats the long way of saying "setting these accounts to inactive seems to be the proper fix here" | 17:23 |
| fungi | clarkb: yes, at one time our process was to just move the external ids from the old account to the new account, but then we started marking the old account inactive later | 17:24 |
| clarkb | they are account records that we don't want to just "pollute" with fake data to make the checker happy, but we also need to indicate that they are not to be used | 17:24 |
| clarkb | but gerrit isn't happy with that :/ | 17:24 |
| clarkb | I wonder if we can unset the preferred email address directly | 17:24 |
| clarkb | that keeps the account number and name present | 17:25 |
| fungi | i expect we'll need to unset preferred_email in those cases, or may need to set it to randomly generated garbage @example.org if gerrit doesn't like accounts having no address at all | 17:25 |
| clarkb | just thinking out loud here another option would be to add external ids for bogus email addrs and set that as the preffered email then set teh account inactive | 17:26 |
| fungi | right, that... but it does sound painful | 17:26 |
| clarkb | ya let me write some notes down, then I'll look into whether or not we can unset preferred email addresses I guess | 17:27 |
| *** ykarel has joined #opendev | 17:29 | |
| clarkb | using the rest api to delete an email address does not work. It is an error ebacuse the email address can't be found : | 17:33 |
| *** eolivare has quit IRC | 17:34 | |
| * clarkb looks for docs | 17:34 | |
| clarkb | "Since all data in the account.config file is optional the account.config file may be absent from some user branches." but also "When users update their account properties by pushing to the user branch, it is verified that the preferred email exists in the external IDs." | 17:35 |
| clarkb | I think that maybe we can just delete the account.config file and push that in? | 17:36 |
| clarkb | oh except that the active state goes in that file. Instead maybe we can remove the preferredEmail line and set active = false and see if it verified that ok? | 17:36 |
| clarkb | I think ^ is the next thing to try | 17:37 |
| clarkb | but I can't do that online with review test because it won't verify the other 750 errors :/ | 17:37 |
| iurygregory | hey opendev folks, does anyone know if only the owners of the patch can add hashtags to it? =) | 17:38 |
| clarkb | I'm going to take a break to check that I don't need to catch up on anything else this morning then I ugess stop gerrit, make that edit for one of the accounts I've identified, push it directly, restart gerrit, reindex accounts and clear caches, then rerun consistency checks? | 17:38 |
| clarkb | iurygregory: I think zbr said our acls are currently set up that way | 17:38 |
| clarkb | I don't know if they need to be set up that way or not, I'm not looked at the privilege implications of using hashtags | 17:38 |
| iurygregory | clarkb, in ironic we wanted to add hashtags to have a better visibility of priorities... so if we can add hashtags to other people patches that would help | 17:39 |
| clarkb | right, someone needs to look at the privilege implications of that then potentially update tehe acls | 17:39 |
| iurygregory | at least if cores could update other ppl patches.. | 17:40 |
| clarkb | it seems the default is to be a bit more restricted, we just need to udnerstand why and if it is safe to open it up further | 17:40 |
| iurygregory | clarkb, this would need a discussion on openstack-discuss or something? | 17:40 |
| clarkb | no it needs someone to read teh gerrit docs and udnerstand why gerrit defaults to restricted access to that, then propose a change if it is safe (ideally with some pointing at relevant documentation) | 17:41 |
| clarkb | we don't need discussion as much as understanding | 17:41 |
| iurygregory | by any chance you know the gerrit docs ? =) | 17:41 |
| iurygregory | I can take a look and see if I can understand =) | 17:42 |
| clarkb | iurygregory: https://review.opendev.org/Documentation/index.html there is a link to them in the top banner of our gerrit pages | 17:42 |
| iurygregory | clarkb, tks! | 17:42 |
| clarkb | https://review.opendev.org/Documentation/access-control.html is likely to have relevant info | 17:42 |
| iurygregory | The change owner, branch owners, project owners, and site administrators can always edit or remove hashtags (even without having the Edit Hashtags access right assigned). | 17:43 |
| iurygregory | we can probably edit... | 17:43 |
| clarkb | I bet the reason it isn't default enabled for all is the "remove" ability | 17:44 |
| clarkb | I guess edit is potentially destructive too if you replace tag foo with tag bar | 17:44 |
| clarkb | in that cse allowing cores to do it seems fine | 17:45 |
| clarkb | the downside to doing it that way is we'd have to update project acls one by one rather than a global edit for registered uses | 17:46 |
| iurygregory | well, we want that in ironic hehe, not sure about other projects XD | 17:47 |
| iurygregory | I can ofc push the patches (just need to know what should be added...) | 17:48 |
| iurygregory | we are trying to improve the reviews so a search using "hashtag:ironic-prio" would give the list of all patches that need attention or something =) | 17:48 |
| clarkb | iurygregory: I think you add al ink roughly at https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/openstack/ironic.config#L2 that says 'Edit Hashtags = group ironic-core" | 17:50 |
| iurygregory | clarkb, ack o/ I will push a patch and add ironic-cores in the review | 17:51 |
| *** ralonsoh has quit IRC | 17:56 | |
| *** sboyron has quit IRC | 17:56 | |
| *** ralonsoh has joined #opendev | 17:57 | |
| *** sboyron has joined #opendev | 17:57 | |
| *** ralonsoh has quit IRC | 17:58 | |
| fungi | i think adding core reviewer access to use hashtags on a per-project basis initially would be a fairly safe way to gauge whether it can just be allowed for all users | 18:00 |
| fungi | like we do with allowing +1/-1 code review votes and comments | 18:00 |
| fungi | ease into it | 18:00 |
| clarkb | ya | 18:01 |
| *** rpittau is now known as rpittau|afk | 18:10 | |
| *** ykarel has quit IRC | 18:12 | |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add swift-proxy in zuul namespace https://review.opendev.org/c/openstack/project-config/+/772364 | 18:19 |
| *** jpena is now known as jpena|off | 18:22 | |
| clarkb | on review-test I did: stopped gerrit, backed up all-users, cloned all-users, fetched refs/userse/XY/ABXY, checked out FETCH_HEAD, deleted preferredEmail in account.config, added active = false in account.config. committed then did git push origin refs/users/XY/ABXY. Ran an offline reindex of accounts and groups. Started gerrit thene flushed accounts and groups caches | 18:24 |
| clarkb | consistency check no longer complains about account ABXY | 18:24 |
| clarkb | the tricky thing about doing this in a downtime is we can't do it in a single atomic commit since all the users have different refs | 18:25 |
| clarkb | but maybe we schedule an hour or two downtime. Script up the fixes for ^ and whatever we come up with for the external id conflicts, make all those changes in one go, do an offline reindex, and then clear caches on startup? | 18:26 |
| clarkb | not ideal, but that should get us to a point where we can push these changes to gerrit online and have it verify them for us | 18:26 |
| clarkb | whcih means it should be a one time thing | 18:26 |
| *** andrewbonney has quit IRC | 18:28 | |
| clarkb | I'm going to take a break but then will look at the external id conflicts shortly | 18:28 |
| clarkb | actually I should do a better audit of the preferred email issue. I've only looked at 4 accounts so far and checking all of them shouldn't take too long if I write a script to print some stuff out | 18:29 |
| clarkb | hrm ya doing an audit I've already found an example that didn't match my first three. In this case it is an active account with a secondary email that mathces an external id | 18:45 |
| clarkb | for those we might be able to tell gerrit to use teh secondary id as primary email? | 18:46 |
| clarkb | that can be done online via the api I think | 18:46 |
| clarkb | I think no matter what we do we're going to be making compromises workign with incomplete data (and as a result some users may be left in a weird spot) | 18:47 |
| clarkb | I'm going to keep looking at this preferred id issue list and see if there are other scenarios | 18:47 |
| *** dtantsur is now known as dtantsur|afk | 18:50 | |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Install get-pip.py for python3.5/xenial with specific url https://review.opendev.org/c/opendev/system-config/+/772369 | 18:50 |
| openstackgerrit | Guillaume Chauvel proposed opendev/system-config master: Move xenial distutils check in OS specific files. https://review.opendev.org/c/opendev/system-config/+/772420 | 18:50 |
| clarkb | we can also set a bunch of these accounts to inactive nowish via online updates. Then see if there is screaming to resolve | 18:52 |
| clarkb | that way when we get to taking a downtime we'll hopefully have more complete information | 18:53 |
| * clarkb writes down more notes | 18:53 | |
| fungi | right, we also have e-mail addresses for most of them, so like i suggested last week we could just send a form letter saying that on some specific date we'll be taking action on those accounts to resolve the conflict, and asking for users to reach out to us if the action we're proposing is the wrong one for them | 18:54 |
| clarkb | yes, though I expect for many users they won't actually know | 18:55 |
| clarkb | and do we do the same thing for all 750 issues or spend all day investigating each situation to prescribe a solution? | 18:55 |
| fungi | i expect for most users those addresses are going to /dev/null and we can just do whatever we want | 18:55 |
| clarkb | I'ev just found an account that seems to have been merged except for the ssh side, so it is possible the person is always reviewing as user foo but pushing code as user bar :/ | 18:56 |
| fungi | right, those are the specific cases i worry about (users actually using two separate accounts and not realizing it) | 18:56 |
| clarkb | the bit I'm struggling with most here is we seem to have many different situations we need to resolve and so fixing this isn't going to be quick but we also need to take a downtime. | 18:58 |
| clarkb | makes scripting it more difficult as well as communicating it | 18:59 |
| clarkb | it wouldn't be so bad if we could just do all the easy ones first without a downtime | 18:59 |
| fungi | i think the first step is exactly what you're doing though: trying to classify different types of situations | 18:59 |
| clarkb | ya | 18:59 |
| clarkb | the first bucket is accounts that are already inactive: we can just remove their preferred email address and move on | 18:59 |
| fungi | it's hard to know what to do next without a fairly complete picture of the various problems | 18:59 |
| clarkb | let me do a better job of pulling all those out and setting themaside | 19:00 |
| clarkb | then the next set is active accounts that have no external ids set (we can just remove their preferred email addr and set them inactive) | 19:00 |
| clarkb | no additional external ids is functioanlly equivalent to being inactive, they cannot log in | 19:01 |
| clarkb | then we're left with the weird ones which I'm only just starting to dig into | 19:01 |
| clarkb | also, I'm still workign against review-test so at some point we'll need to generate the data from prod | 19:01 |
| clarkb | but that shouldn't be too bad once we've sorted things out on the test side | 19:01 |
| fungi | the situation with ssh keys will get hairy too, for sure, because before 2.13 ssh keys were managed in the db and we could recombine the records into other accounts easily when merging them. in 2.13 they'd already moved ssh keys into the all-users repo, so that was no longer reasonable to fix manually | 19:03 |
| fungi | so there will certainly be accounts with nothing other than ssh keys for access, because in 2.13 we left them alone | 19:04 |
| fungi | though hopefully we were consistent about marking them inactive in those cases | 19:04 |
| clarkb | also note I've not yet started looking at accounts with external id conflicts | 19:10 |
| clarkb | but as you say step by step building a bigger picture seems like what we need to do here | 19:10 |
| clarkb | also, maybe we take multiple downtimes to spread this out a bit and see progress. In particular that reduces the number of things we might be modifying which reduces the risk we'll get something wrong | 19:13 |
| clarkb | just under half seem to be inactive accounts | 19:16 |
| clarkb | now to find the set that have no externald ids | 19:16 |
| fungi | infra-root: i'm unlikely to get to it today (or probably even tomorrow), but looks like our "wheel mirrors" haven't updated in two weeks. likely something broke with the zuul jobs which update them, i haven't looked | 19:33 |
| *** brinzhang has joined #opendev | 19:37 | |
| iurygregory | clarkb, something like http://paste.openstack.org/show/801959/ ? | 19:41 |
| clarkb | iurygregory: yes I suspect that is correct, but I'm not in a great spot to verify it right now | 19:47 |
| clarkb | I'm trying to run down a bunch of consistency issues in our gerrit accounts db | 19:47 |
| clarkb | fungi: that might make for a good distraction once I've spent most of the day looking at accounts. We'll see if i find time to dig in later today (re wheels) | 19:48 |
| iurygregory | clarkb, no worries =) I will push the patch to see how it goes | 19:48 |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/project-config master: Update ACLs of Ironic Projects to allow Edit Hashtags https://review.opendev.org/c/openstack/project-config/+/772427 | 19:53 |
| corvus | fungi, mordred: did we document the "git config gitreview.*" options? i don't see them in https://docs.openstack.org/infra/git-review/ | 20:01 |
| corvus | basically, it looks like you can run "git config --global gitreview.rebase false" but i can't find a doc telling me that should work | 20:02 |
| clarkb | fungi: it just occured to me that since refs/users/XY/ABXY are their own refs we may be able to push to them individually without needing a downtime. I will have to test that | 20:02 |
| fungi | corvus: mmm, you know, i'm not entirely sure | 20:03 |
| fungi | i thought we had incorporated them in the docs but maybe not, checking | 20:03 |
| fungi | clarkb: yes for anything not involving external-ids that might be doable, but unfortunately not for external-ids as those are all in the same ref | 20:04 |
| clarkb | fungi: ya, but maybe that means we can tackle this in a couple of iterations and get to where we need to take a downtime last | 20:04 |
| clarkb | anyway I'm still classifying things under refs/users/ its not a quick process | 20:05 |
| fungi | corvus: the configuration section of the git-review(1) manpage maybe? | 20:06 |
| fungi | it does have a gitreview.rebase entry at least | 20:07 |
| fungi | i'm not sure if it covers everything, but hopefully it does | 20:07 |
| fungi | "This setting determines whether changes submitted will be rebased to the newest state of the branch." [and then delves into details of what true and false do] | 20:08 |
| fungi | so yeah, maybe the problem is that the sphinx docs and manpage are out of sync? | 20:08 |
| fungi | also it looks like the last time anyone added anything to the manpage was 4 years ago, so changes since then aren't reflected there i guess | 20:09 |
| fungi | i have a feeling we started out with the manpage as the source of truth, added very incomplete sphinx docs, and then at some point people started documenting things in sphinx and not in the manpage | 20:10 |
| fungi | probably what we need is to merge the manpage content into the sphinx source, and then use a sphinx extension to generate the manpage and run that at package build time? | 20:11 |
| fungi | zbr: ^ you've been reviewing git-review changes lately too, what do you think? | 20:12 |
| fungi | git-review also has its own --help output, which you'll only see when running `git-review --help` because `git review --help` wants to show the manpage (and if you've installed with pip, that won't exist in your manpath because pip can't install files there) | 20:13 |
| corvus | fungi: oh yep, i see it there, thanks! | 20:14 |
| fungi | corvus: opinions on how to improve that? | 20:14 |
| corvus | fungi: (to clarify, i see it in the manpage now) | 20:15 |
| corvus | and yeah, i use --help a lot, but that doesn't mention the rebase afaict | 20:15 |
| corvus | fungi: i agree that we should move docs to sphinx and generate manpage from that if possible | 20:15 |
| corvus | from the looks of it, manpage is still way ahead of sphinx docs | 20:16 |
| fungi | i think that's what we do for bindep, will double-check | 20:16 |
| corvus | the current state is not ideal because one could (as i did) find the sphinx docs, assume they are complete, and then give up | 20:16 |
| fungi | yep, totally agree | 20:17 |
| corvus | i think it'd be great if zbr is interested in taking that on :) | 20:17 |
| openstackgerrit | Matthew Treinish proposed opendev/subunit2sql master: Add release notes to prepare for release https://review.opendev.org/c/opendev/subunit2sql/+/772429 | 20:23 |
| fungi | mmm, i'm not finding generated manpages in bindep either. i know i've seen it in use somewhere in our tools | 20:24 |
| fungi | oh, i bet i know at least one place it's in use | 20:33 |
| fungi | nope, thought pbr at least was using it, since it's what does it | 20:42 |
| mordred | I believe we had generated manpages back in the day | 20:43 |
| mordred | but with the shift to wheel-based pip stuff we gave up | 20:44 |
| fungi | functionality is still in pbr, and i could swear i did something recently on a project generating them | 20:44 |
| clarkb | out of 109 accounts with preferred emails that don't have external ids 12 are in the "complicated case" where they have ssh usernames and no other similar external id bits to tie to. 16 have other email addresses that we could possibly add as externalids then set as preferred email (though this is less clear to me). The rest are either inactive or functionality inactive because they have no external ids | 20:46 |
| clarkb | that still doesn't touch any of the email address conflicts | 20:46 |
| corvus | btw, i've been testing git-review with onlykey; seems to work so far, but due to invoking ssh multiple times (for the rebase) it can be awkward. onlykey normally prompts for a pin so you push 3 buttons on the device, and you do that twice for each of the ssh connections, but it has a setting where you can hit any button, and that makes it easier (more like a yubikey or signet). or you could turn off the | 20:47 |
| corvus | rebase. | 20:47 |
| clarkb | but I think we can probably address the 109 - 12 - 16 remaining preferred email errors as step 1 | 20:47 |
| mtreinish | fungi: pbr supports man page generation? For stestr I just wrote a bash script to do man page generation from the sphinx docs: https://github.com/mtreinish/stestr#building-a-manpage | 20:47 |
| clarkb | I'm going to test if I can update those while gerrit is online next | 20:47 |
| fungi | mtreinish: heh, wfm ;) | 20:48 |
| clarkb | iirc the issue pbr ran into is that you can build the manpages no problem but you can't install them | 20:48 |
| clarkb | beacuse MANPATH is typically privilegd space and you don't want an install going to a venv to write there | 20:48 |
| fungi | well, can't install them with a python package into anything manpathish | 20:48 |
| fungi | you can still generate manpages to be used in distro packaging and the like | 20:49 |
| clarkb | yup and sphinx supports that just fine | 20:49 |
| clarkb | but pbr specifically has trouble with that beacuse it often writes to venvs without root | 20:49 |
| fungi | oh, maybe i'm thinking of an existing sphinx feature for generating manpages | 20:49 |
| corvus | ++ and that's the value i see here (git-review should mostly be installed by most people via distro packages these days) | 20:49 |
| mtreinish | heh, well for stestr I haven't been able to convince any packagers to include a manpage with it. Despite putting it in the readme | 20:49 |
| clarkb | fungi: `make manpage` or whatever the target is should just do it | 20:50 |
| mtreinish | when I used to pacakge it in arch I did it, but they deleted my aur package when someone on the core team did their own stestr package and it doesn't have a manpage... | 20:50 |
| fungi | clarkb: that needs a makefile right? | 20:50 |
| fungi | mtreinish: so many distros want to package python modules by pulling from pypi and unpacking the python packages that i can see where that would happen | 20:51 |
| fungi | for libraries it makes sense. for utilities which just happen to be written in python, less so in my opinion | 20:52 |
| clarkb | fungi: yes, sphinx should write out a make file in your sphinx dir stuff | 20:53 |
| clarkb | when you do the sphinx canned setup it writes that for you | 20:53 |
| fungi | ahh, right | 20:53 |
| clarkb | infra-root review-test:~clarkb/gerrit-consistency-notes/preferred-email-classifications has the classifications of the first 109 problems detected on review-test | 20:54 |
| fungi | mordred: okay, i was confused, the manpages support in pbr is just for packing available manpages into the dist | 20:54 |
| fungi | not generating them | 20:54 |
| clarkb | we still need to rerun consistency checking against prod, but I'm getting orientated on the test server | 20:54 |
| clarkb | fungi: correct | 20:54 |
| clarkb | and we removed that from pbr because it only caused problems, but you can still build manpages if you want them | 20:54 |
| clarkb | now to test if I can push an update to refs/users/XY/ABXY while it is running to fix a problem | 20:55 |
| fungi | clarkb: well, manpages support is still present in pbr master, maybe some of it was removed? it's still in the commands and files hooks, and the packaging module | 20:56 |
| fungi | and apparently it's `make man` in the sphinx tree | 20:58 |
| *** slaweq has quit IRC | 21:02 | |
| clarkb | fungi: oh interesting, I recall having turned ti off in a number of places because we turned it on for like debian and then broke everyone | 21:04 |
| clarkb | something like that | 21:04 |
| fungi | also some projects including git-review are still including a pbr.manpages list in their setup.cfg | 21:06 |
| clarkb | ok I think I have confirmed that for the 109 - 12 - 16 = whatever accounts we can fix those while gerrit is online pushing directly to their refs/users/XY/ABXY ref | 21:09 |
| clarkb | then we need to sort out the 12 and 16 groups of users. Then figure out conflicting external ids | 21:10 |
| clarkb | I'm going to take a break now as its not raining or snowing and I'm due for a bike ride. When I get back I'll probably look at wheel mirrors? Then we can pick up gerrit things tomorrow | 21:11 |
| clarkb | infra-root: thinking out loud here, maybe we can do paired ops to fix the bits identified as fixable so far some time tomorrow as the impact should be low? | 21:11 |
| clarkb | that will make the inconsistencies list small and make it easier to focus on the other bits as we go along | 21:12 |
| fungi | wednesday might be a little doable for me, but if someone else wants to join you on it tomorrow that's cool | 21:12 |
| fungi | er, wednesday might be a little more doable for me | 21:13 |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/project-config master: Update ACLs of Ironic Projects to allow Edit Hashtags https://review.opendev.org/c/openstack/project-config/+/772427 | 21:13 |
| clarkb | fungi: ok, we can probably plan for wednesday too | 21:13 |
| clarkb | oh and I'll get the meeting agenda sent out when I return and try and put as much of the info I've learned so far in there | 21:14 |
| clarkb | thinking out loud here, we can probably fix the external ids in aggregate too (a single large commit) and push those fixes without a downtime | 21:14 |
| clarkb | where that will potentially get complicated is rebasing to keep up with the changes happening in gerrit. If we find we can't keep up quickly enough then we can take a downtime? | 21:14 |
| fungi | yeah, *if* we can work it all out in a single commit that should be doable | 21:14 |
| clarkb | I'll continue to ponder on that aspect of the problem | 21:15 |
| clarkb | ya also that | 21:15 |
| fungi | if we want to make changes in the meantime, that would need downtime | 21:15 |
| clarkb | in any case I think I've figured out fixing about 10% of the errors so progress ! :) | 21:15 |
| fungi | progress fore sure | 21:15 |
| fungi | er, for sure | 21:15 |
| *** tosky has quit IRC | 21:35 | |
| *** Alex_Gaynor has quit IRC | 21:36 | |
| *** tosky has joined #opendev | 21:36 | |
| *** TheJulia has quit IRC | 21:36 | |
| *** TheJulia has joined #opendev | 21:37 | |
| *** rpittau|afk has quit IRC | 21:37 | |
| *** johnsom has quit IRC | 21:37 | |
| *** rpittau|afk_ has joined #opendev | 21:38 | |
| *** johnsom has joined #opendev | 21:38 | |
| *** Alex_Gaynor has joined #opendev | 21:38 | |
| *** artom has quit IRC | 21:41 | |
| *** artom has joined #opendev | 21:47 | |
| *** slaweq has joined #opendev | 21:48 | |
| *** stand has joined #opendev | 21:49 | |
| *** sboyron has quit IRC | 21:52 | |
| *** stand has quit IRC | 21:56 | |
| *** stand has joined #opendev | 21:56 | |
| *** dmsimard has quit IRC | 22:50 | |
| *** dmsimard has joined #opendev | 22:51 | |
| *** slaweq has quit IRC | 23:01 | |
| openstackgerrit | Ghanshyam proposed opendev/subunit2sql master: Add release notes to prepare for release https://review.opendev.org/c/opendev/subunit2sql/+/772429 | 23:01 |
| mordred | es | 23:11 |
| mordred | gah | 23:11 |
| clarkb | fungi: mnaser: shoudl I keep the infra-core (really config-core) discussion topic on the meeting agenda? | 23:27 |
| clarkb | I think we're pretty set with the plan we made last week and just need to get the ball rolling? | 23:27 |
| fungi | clarkb: if folks have a chance to look over the draft rfh i wrote up and linked in #openjstack-infra that would be swell | 23:30 |
| fungi | other than that i don't really have any updates for the topic | 23:30 |
| fungi | er, #openstack-infra i mean, not the java one | 23:30 |
| clarkb | cool I'll take it off of the agenda and review the draft once I send the agenda out | 23:31 |
| fungi | i can always link it during the action items topic | 23:31 |
| mordred | fungi: wow, I haven't thought about openjstack in forever years | 23:31 |
| fungi | was it even a thing? i was just making fun of my typo | 23:36 |
| fungi | oh, right, oracle has jstack | 23:37 |
| mordred | oh - no, I was also just making fun of your typo | 23:38 |
| fungi | cool, appreciated | 23:38 |
| mordred | although I do think at one point that one guy did an java reimpl sortof but then gave up | 23:38 |
| *** tosky has quit IRC | 23:47 | |
| clarkb | ok I took a quick look at the conflicting emails in external ids and I think the vast majority of these will need us to merge accounts or pick a side to "retire" | 23:56 |
| clarkb | also note that when we merge the external ids I believe we'll create a bunch of refs/users/XY/ABXY preferred email addr orphans similar to the accounts above | 23:57 |
| clarkb | so we'll have a asecond step that requires us to retire those as I've proposed for the already inactive accounts above | 23:57 |
| clarkb | thnking out loud here maybe we set all of the side we're going to retire to inactive as a second step (first step is doing an audit and deciding which side of the two is merging into the other) | 23:58 |
| clarkb | then if people notice and complain we can talk to them about the best way forward? | 23:58 |
| clarkb | I dunno this needs more thinking. | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!