Tuesday, 2020-12-01

*** tosky has quit IRC00:02
TheJuliahttps://bugs.chromium.org/p/gerrit/issues/detail?id=974000:02
funginot sure, i suppose you've got a bunch you want to clean up?00:05
TheJuliaYeah, tons orphaned from before the last upgrade00:08
TheJuliabut, that link has the command in it00:08
fungiahh, cool00:09
openstackgerritMerged opendev/system-config master: Increase some of Gerrit's in-memory cache pools  https://review.opendev.org/c/opendev/system-config/+/76477900:11
fungiTheJulia: reading through it, supposedly there was at one time also a dashboard banner which had a button to bulk delete draft comments00:13
*** sboyron has quit IRC00:23
TheJuliaThere could have been, and maybe it went poof at some point...00:24
fungiyeah, maybe after some number of releases they removed it because what are the odds someone's going to upgrade a site straight from <2.16 to whatever point that got removed in?00:31
fungiso it eventually became noise for sites which had long since been running a version which showed you all your unsubmitted draft comments00:31
TheJuliaYeah00:32
* TheJulia looks at the tons of line breaks in test results and may need to get her eyes chcked00:32
fungiit will take time to get everything back in shape, yes00:33
openstackgerritMerged opendev/system-config master: Disable Gerrit's automatic Git GC on push  https://review.opendev.org/c/opendev/system-config/+/76480701:05
openstackgerritlikui proposed openstack/diskimage-builder master: update tox  https://review.opendev.org/c/openstack/diskimage-builder/+/75938401:41
fungichangeid_project cache is now over 8192 since the restart03:09
*** sgw has quit IRC03:09
*** larainema has joined #opendev03:45
larainemahttps://review.opendev.org/c/openstack/project-config/+/763742  can someone help merge this patch?03:50
*** sgw has joined #opendev04:33
*** icey has quit IRC06:16
*** hamalq has quit IRC06:21
*** hamalq has joined #opendev06:22
*** marios has joined #opendev06:32
*** zbr has quit IRC06:52
*** sboyron has joined #opendev06:52
*** zbr has joined #opendev06:52
*** ysandeep|away is now known as ysandeep06:53
*** icey has joined #opendev07:01
*** ralonsoh has joined #opendev07:37
*** eolivare has joined #opendev07:42
openstackgerritMerged openstack/project-config master: Remove ceilometer-zvm entry  https://review.opendev.org/c/openstack/project-config/+/76374207:45
*** slaweq has joined #opendev07:52
*** fressi has joined #opendev07:57
*** rpittau|afk is now known as rpittau07:59
*** fressi has quit IRC08:02
*** fressi has joined #opendev08:02
*** andrewbonney has joined #opendev08:03
*** hashar has joined #opendev08:10
*** tosky has joined #opendev08:40
*** mgoddard has joined #opendev09:36
*** DSpider has joined #opendev09:50
*** zbr has quit IRC09:51
*** zbr has joined #opendev09:56
*** chengsheng1 has joined #opendev10:18
chengsheng1hello10:20
*** zbr has quit IRC10:58
*** zbr has joined #opendev10:58
*** zbr has quit IRC11:00
*** zbr has joined #opendev11:00
*** zbr has quit IRC11:03
*** dtantsur|afk is now known as dtantsur11:04
*** zbr has joined #opendev11:04
*** zbr has quit IRC11:07
*** zbr has joined #opendev11:17
gibiclarkb, fungi: I dropped the keybord at that time already. And so far I'm in review mode so I did not push any changes today. I will let you know if I see issues with push11:26
*** mgoddard has quit IRC11:55
*** iurygregory|pto has joined #opendev12:08
*** waverider has joined #opendev12:16
*** eolivare_ has joined #opendev12:21
*** eolivare has quit IRC12:24
gibifungi: one addition to the yesterday gerrit load issue is that I was able to push a series of patches yesterday that resulted in an internal error but the commits ended up in gerrit. Today I realized that CI was not triggere for them so I assume that the internal error was in somewhere the event emitting part of gerrit12:26
gibifungi: this is the chain https://review.opendev.org/q/topic:%22bp%252Fsupport-interface-attach-with-qos-ports%22+(status:open%20OR%20status:merged)12:26
*** brinzhang_ has joined #opendev12:46
*** eolivare_ has quit IRC12:48
*** brinzhang0 has quit IRC12:50
*** eolivare_ has joined #opendev13:23
*** TeamDevOps has joined #opendev13:27
*** TeamDevOps has quit IRC13:27
*** slaweq has quit IRC13:34
*** slaweq has joined #opendev13:36
*** sshnaidm has quit IRC13:38
*** sshnaidm has joined #opendev13:46
*** sshnaidm has quit IRC13:48
*** mlavalle has joined #opendev13:58
*** chrome0 has quit IRC14:04
*** chrome0 has joined #opendev14:06
*** fressi has left #opendev14:43
*** ysandeep is now known as ysandeep|away15:07
*** waverider has quit IRC15:16
*** hashar is now known as hasharAway15:20
fungilooking at the load graph for gerrit, i don't think expanding the caches or disabling autogc have necessarily helped15:20
fungiwe've nearly (but not quite yet) filled the new changeid_project cache, but hit ratio is still not particularly good (35%)15:22
fungino more "missing tree" errors today though15:22
*** TeamDevOps99 has joined #opendev15:23
fungiso maybe it's somewhat of an improvement15:23
*** TeamDevOps99 has quit IRC15:23
lourotHi o/ we have been having intermittently today the following error: https://zuul.opendev.org/t/openstack/build/7a2b8ffdad83412abed573f250994b1315:26
fungigibi: thanks, i tried looking into that one yesterday when you mentioned it and didn't find anything fatal in the log, but i think i saw at least one anomalous warning related to one of your pushes around that time15:27
lourotthis is not happening all the time, so we suspect it depends on what node the zuul job runs15:27
lourotalso this is not due to pip 20.3 published yesterday we think as we get other errors that we are solving already because of pip 20.315:27
lourotdoes that ring a bell?15:28
gibifungi: no worries, I could trigger the CI with a recheck.15:29
fungilourot: pip tries to avoid retrieving wheels which are marked as not supporting the version of python you're running under, but if it gets a copy of the pypi index which lacks the necessary metadata for some reason then it will fall back to just fetching the newest version of the package which satisfies your requested requirement15:30
fungiwe've seen this in the past when the fastly cdn pypi uses can't reach the primary warehouse servers for some reason and winds up retrieving and caching an index from their backup bandersnatch-built mirror which doesn't include the python version metadata15:31
fungiso my initial guess is that the fastly cdn endpoint near some of our cloud providers is having trouble getting to the primary copy of the pypi index15:32
fungithis is usually transient, but we've sometimes seen it persist for days before they fix it15:32
lourotok thanks fungi15:33
fungilourot: if you're running openstack projects under python 3.5, you really should be using requirements and constraints contemporary with the last release which supported that15:34
lourotyes I'll tweak the constraints, thanks!15:35
fungilourot: be aware that it's going to be fairly old openstack software... even stein was not tested with python 3.5: https://governance.openstack.org/tc/reference/runtimes/stein.html15:36
lourotfungi, yes we still support Ubuntu 14.04 (up to Mitaka) and 16.04 (up to Queens) and they have python 3.5, but we have other CI jobs for this and I wonder if we really need the zuul jobs to be running python 3.515:39
lourotgnuoy, maybe that's just a matter of adapting the .zuul.yaml files to not have the zuul jobs running on python 3.5? ^15:40
lourotyes indeed it looks like we have slightly outdated .zuul.yaml files pointing explicitly to py35 in the problematic repos15:41
lourotthanks for the input fungi!15:42
fungiyou're welcome!15:42
*** whoami-rajat__ has joined #opendev16:00
*** sshnaidm has joined #opendev16:05
*** TeamDevOps43 has joined #opendev16:07
*** TeamDevOps43 has quit IRC16:07
fungichangeid_project              | 32768               |         | 36%16:13
fungiwe just filled it in the last few minutes16:13
funginow to see if the cache hit rate improves or gets worse16:13
fungii'm noticing that show-caches doesn't list a couple of the caches we're overriding in our gerrit.config: accounts_byemail and accounts_byname16:18
clarkbfungi: maybe those caches have gone away? most of the caches are documented in the gerrit docs but not all16:19
clarkbcould cross check against that I guess16:19
fungialso the accounts cache is disk-backed, so i don't know that our memorylimit override for that is relevant any longer16:19
*** marios is now known as marios|out16:46
*** rpittau is now known as rpittau|afk17:01
*** marios|out has quit IRC17:13
openstackgerritGhanshyam proposed openstack/project-config master: Update retiring Qinling repo ACL to openstack/retired.config  https://review.opendev.org/c/openstack/project-config/+/76453217:28
openstackgerritGhanshyam proposed openstack/project-config master: Remove Qinling projects from infra  https://review.opendev.org/c/openstack/project-config/+/76453617:32
openstackgerritGhanshyam proposed openstack/project-config master: Update retiring Searchlight repo ACL to openstack/retired.config  https://review.opendev.org/c/openstack/project-config/+/76453417:35
openstackgerritGhanshyam proposed openstack/project-config master: Remove Searchlight projects from infra  https://review.opendev.org/c/openstack/project-config/+/76453517:39
*** dtantsur is now known as dtantsur|afk17:55
*** d34dh0r53 has quit IRC18:13
*** andrewbonney has quit IRC18:40
*** d34dh0r53 has joined #opendev18:54
*** eolivare_ has quit IRC19:04
*** sshnaidm has quit IRC19:15
*** sshnaidm has joined #opendev19:18
ianwthe cert on review test expired 11/16/2020 , can we refresh it?19:42
clarkbianw: it is an LE cert so yes, except I don't know that we can run ansible against it in its current state19:42
fungiianw: "we" can, what's the easiest way there? (noting that we don't want to blow away some of the specific gerrit configs there)19:43
clarkbianw: not sure if the best thing there is to get ansible in sync somehow, or to jsut do the LE dance manualyl (or maybe we can just run that playbook?19:43
ianwcould probably just run the le playbook quickly with review-test un-emergencied19:43
ianwthere's no real "manual" way to just run LE on the host because you need to deploy the DNS records19:43
clarkbianw: rather than unmergency it, it might be better to use a copy of the playbook without !disabled and then limit it to the relevant hosts19:43
clarkbthat way we don't have to worry about our gerrit config updates and image changes being switched on us19:44
ianwyep, that could work too19:44
clarkbalso if we want to be lazy I can go spend $8 for a cert19:44
ianwfwiw i've clicked around on review-test and everything works as expected for me20:01
ianwfungi / clarkb: I forgot to mention in the zuul discussion a couple of fixes i found https://review.opendev.org/q/topic:%22gerrit-ssh%22+(status:open%20OR%20status:merged)20:02
ianwmaybe if we look at those, then i can hold a fresh node and fungi can maybe help me with the steps to add a admin, project, change, fake zuul user and a fake zuul comment.  that would be a minimal framework we can expand on20:08
fungionce i get the jgit config pushed i need to work on cooking dinner, but can look after that20:09
*** hasharAway is now known as hashar20:14
openstackgerritJeremy Stanley proposed opendev/system-config master: Move jgit autogc disablement into correct config  https://review.opendev.org/c/opendev/system-config/+/76500420:23
fungiinfra-root: ^20:23
fungionce that's deployed we can restart and then look at openjdk 11 as our next step20:24
mordredfungi: +2 - +A at will20:36
fungithanks!20:36
*** savihou has joined #opendev20:56
openstackgerritJeremy Stanley proposed opendev/system-config master: Move jgit autogc disablement into correct config  https://review.opendev.org/c/opendev/system-config/+/76500420:56
fungimordred: clarkb: ^ updated based on feedback20:57
ianwper clarkb's comment in 764396 i'm just going to restart review-test with a db/ dir mount, to make sure it doesn't do anything with it20:57
fungiianw: sounds good21:00
*** slaweq has quit IRC21:47
*** slaweq has joined #opendev21:49
clarkbianw: on my bike ride I also realized that we should check there isn't a dir created there already (not bind mounted) that we would mask off with a bind mount22:07
clarkbif that makes sense22:07
clarkbianw: however if we don't bind mount it then that content would be ephemeral anyway? so I think we're like 99% fine22:09
ianwclarkb: don't think there is, but let me jump in container22:09
ianwdrwxr-xr-x   2 root   root    4096 Nov 17 20:47 db22:10
clarkbianw: ya I think I realized that it would have to be ephemeral content anyway after thinking about it on the bike a bit22:10
clarkbso should be safe22:10
clarkbfungi: tox linters failed on teh jgit change22:11
clarkbfungi: https://zuul.opendev.org/t/openstack/build/ac85ac8eeb544704b8dee0360b99b1a9/log/job-output.txt#757 I think it may be a legit error22:12
clarkbfungi: ya mode is listed twice22:12
clarkbI made note of the issue with a -1 on the change22:13
clarkbpabelanger: we don't know why that happens, we think possibly beacuse your browser is telling gitea to use not english as the locale22:14
pabelangeris it just me, or is https://opendev.org/zuul/zuul and https://opendev.org/openstack/project-config displaying in non english22:14
pabelangerah22:14
clarkbat the bottom right is a globe icon, click on that and you can select english again22:14
pabelangerokay, thanks22:15
pabelangersome reason pyccnn is selected22:15
*** hashar has quit IRC22:16
pabelangerI just upgraded to f33, the other day, so likely related22:16
clarkbpabelanger: in firefox you go to preferences -> general -> Language and double chekc the language tehre22:16
clarkbMine says English (United States)22:16
pabelangeron chrome, checking its default22:17
pabelangerhmm, english (canada) is default22:18
pabelangerweird22:18
openstackgerritClark Boylan proposed opendev/system-config master: Build Gerrit 3.3 images  https://review.opendev.org/c/opendev/system-config/+/76502122:36
clarkbpabelanger: ya we haven't tracked that behavior down yet22:36
clarkbpabelanger: if you figure out why it happened let us know22:36
clarkbfungi: ^ when not looking at release notes things can you check your jgit.config change? I think it needs edits22:38
fungiclarkb: thanks, sorry got busy with dinner22:39
clarkbfungi: no worries, I got busy with a two wheeled mobility device22:40
corvusfungi: (from #zuul) we don't seem to say that forge committer is required here: https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#merge-commits22:47
corvus(i'm trying to push up a merge commit, but it's being rejected with remote: ERROR: commit 713b533: email address review@openstack.org is not registered in your account, and you lack 'forge committer' permission.22:48
corvusthat commit is a pretty standard zuul merge commit, which i would expect to be on any branch in gerrit, so this should be a well-tested process for any feature branch, etc22:49
corvusis there any chance this is a gerrit version behavior change?22:49
corvusAuthor: Zuul <zuul@review.opendev.org> / Commit: Gerrit Code Review <review@openstack.org>22:50
clarkbcorvus: the docs for that permission state that this is usually only necessary when mirroring commits from an upstream repository. That makes me think the assertion fungi made is how it is expected to work (basically if gerrit is already aware of the commits then it is fine)22:51
clarkbcorvus: is it possible that gerrit isn't already aware of those commits for some reason (a rebase gone wrong?)22:51
fungiso at one point it was possible to push a history with merges into the "wrong" branch and as long as the commits were already known in gerrit they would get added. this resulted in mishaps like people with dirty histories pushing master into a stable branch and all of master coming along for a ride. gerrit adjusted some of that in later versions so make it so that commits not already on the branch would not22:51
fungibe silently added in a merge22:51
clarkbah fungi has it paged in22:52
fungithough i thought to get that we needed to add receive.rejectImplicitMerges=false, which we haven't yet22:52
corvusi found this: https://repo-discuss.narkive.com/Ps1VbayI/forge-committer-required-when-pushing-a-change-on-top-of-an-unsubmitted-merge-commit22:53
corvusi'm pretty sure the commit history is clean, git-review shows me exactly what i expect22:53
fungithinking harder, receive.rejectImplicitMerges is about non-merge changes with fast-forwardable parents already in another branch22:54
corvusbut i am pushing a locally-authored change on top of the pending merge commit, which is what that link describes22:54
corvuslet me try only pushing up to the merge commit22:54
corvusno joy, but there's still a locally authored commit *before* the merge commit22:55
fungiit does still seem likely this is a behavior change between 2.13 and 3.2 i'm just not sure which one22:56
*** slaweq has quit IRC22:56
corvusit's possible that's affecting it too, but at this point even if that did work it would be a good solution (i'd really like to see this whole thing work before we start merging temporary changes)22:56
corvusi wonder if i should just ask the opendev admins to add forge committer for this case?22:57
corvus(or invoke project bootstrappers or something)22:57
fungiit seems like forge committer would be reasonable if that's really now a requirement for adding other committers' commits to a different branch23:01
fungiwe could just update our documentation to say that's expected in combination with allowing merge commits23:02
corvusi'd probably want to test it without the 'local commit sandwich' first before doing that; this situation may still be unusual23:02
clarkbreview-test would be a good place for that sort of stuff23:03
clarkbthough I ugess we can do it on sandbox too23:03
corvusyeah, was thinking sandbox would be sufficient23:04
corvusi could also probably just push up a throwaway change to zuul with a different changeid and see23:04
*** jaicaa has quit IRC23:04
fungii wouldn't be surprised if this is the first time since the upgrade anyone's tested pushing a legitimate merge commit, and it's not something i thought to test when we were evaluating pre-upgrade23:05
corvusokay, i get the same result when pushing up only the merge commit23:06
corvusi really don't like the idea of generally adding forge committer :(23:06
clarkbthe docs do seem to imply that shouldn't be necessary? I wonder if we should file a bug?23:08
fungiif it turns out to just be that the zuul merge commit situation is a weird one-off situation, then i'm also not opposed to the opendev sysadmin whose username starts with c and rhymes with corvus temporarily adding themselves to project bootstrappers to push it on behalf of the zuul contributors, but i'll readily acknowledge that's not scalable solution if it's a commonplace merge commit23:08
corvusfungi: yeah, i'm running out of ideas of how this could be extraordinary at this point23:09
fungidoes sound likely to be a bug in that case23:10
corvusthe 3.19.1 tag i merged locally has the same sha that gerrit does (and master matches too)23:11
clarkbfungi: were you going to push a new ps up for 765004? and if so are we reaching a point in the day where we should consider manually making that change and restarted gerrit?23:12
clarkbor leave it to tomorrow I guess?23:12
fungisorry, yeah i23:14
fungi'm working on it now23:14
openstackgerritJeremy Stanley proposed opendev/system-config master: Move jgit autogc disablement into correct config  https://review.opendev.org/c/opendev/system-config/+/76500423:16
corvusi've gone through the git log and every change in 3.19.0..3.19.1 is in gerrit; so i'm doubly confident that isn't the issue23:17
fungiyeah, it's definitely sounding like a bug, so temporarily adding your account to project bootstrappers to push it seems reasonable and we can file something upstream to get clarity either in (docs or fixing the access controls)23:19
fungiparenthetical location error #3723:19
corvusokay, i'll do the bootstrap process23:20
corvuswow there's now a new merge conflict according to git-review23:22
corvusi'll redo the stack23:22
corvusremote: ERROR: commit 713b533: missing Change-Id in message footer23:24
corvussigh23:24
corvussomething is really weird here :/23:25
openstackgerritClark Boylan proposed opendev/system-config master: Build Gerrit 3.3 images  https://review.opendev.org/c/opendev/system-config/+/76502123:26
clarkbhoping ^ fixes an error with its-base building against 3.323:26
ianwinfra-root: response about the results box per discussion i'm working on @ https://etherpad.opendev.org/p/XwWVVI9zBa8LMSZFeQoz23:26
clarkbcorvus: just so I'm clear on what you are attempting to do: you have created a new zuul branch and are now attempting to push a merge commit to it as well as a couple of other changes?23:27
clarkbcorvus: but all of those commits have previously merged on other zuul branches?23:28
corvusclarkb: not quite; i'm running 'git checkout master; git merge 3.19.1; git review'  (essentially)23:28
corvus3.19.1 is a tag made from a no-longer-extant stable/3.x branch23:28
clarkbI see23:28
corvusclarkb: i wonder if the branch is required23:29
clarkboh that is a good qusetion23:29
corvuslike, maybe we need to resuscitate stable/3.x @ 3.19.1 then do this, then delete23:29
corvusianw: good summary i think23:29
clarkbianw: I like the point about using the checks plugin as a reference23:31
clarkband ++ on the testing plan23:31
fungicorvus: great point. re-adding the stable/3.x might be the ticket, now that you mention it i wonder if that's the difference. some of the commits parented to that tag don't belong to a branch gerrit has23:31
ianwthat's about the most fleshed out example i can find23:31
fungi(even though they are represented by gerrit changes)23:31
corvusfungi, clarkb: shall i recreate the branch then?23:32
fungicorvus: it literally can't hurt23:32
clarkbcorvus: I guess so, that idea does seem likely23:32
fungiand it's a quick test23:32
fungialso it's a good source of information in case we see this crop up again23:32
*** cgoncalves has quit IRC23:33
*** cgoncalves has joined #opendev23:34
fungion a related note, i've got a to do list entry reminding me to push up a change sometime soon for rejecting "implicit merges" now that we can... the situation where a normal (non merge commit) change proposed by someone can be parented on changes from another branch and gerrit doesn't even indicate anything's out of the ordinary23:44
fungiwe reported that problem years ago, we ought to at least make use of the fix finally23:45
corvusftr in this channel: the branch creation worked23:52
*** jaicaa has joined #opendev23:52
corvusfungi: nice23:52
*** DSpider has quit IRC23:54
*** tosky has quit IRC23:57

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!