Monday, 2021-03-29

openstackgerritIan Wienand proposed openstack/project-config master: Stop launch fedora-31 nodes nodepool  https://review.opendev.org/c/openstack/project-config/+/77651000:16
openstackgerritIan Wienand proposed openstack/project-config master: Remove fedora-31 disk image config  https://review.opendev.org/c/openstack/project-config/+/77651100:16
openstackgerritMerged openstack/project-config master: Stop launch fedora-31 nodes nodepool  https://review.opendev.org/c/openstack/project-config/+/77651000:39
openstackgerritIan Wienand proposed opendev/system-config master: Stop mirroring Fedora 31  https://review.opendev.org/c/opendev/system-config/+/78358500:55
*** hamalq has joined #opendev00:56
*** hamalq has quit IRC01:01
ianwwe appear to have acquired a "rolecall vote" column in gerrit?  not sure if that was there before ...01:08
openstackgerritMerged openstack/project-config master: Remove fedora-31 disk image config  https://review.opendev.org/c/openstack/project-config/+/77651101:16
openstackgerritMerged opendev/system-config master: Stop mirroring Fedora 31  https://review.opendev.org/c/opendev/system-config/+/78358501:47
*** brinzhang has joined #opendev01:57
*** sshnaidm|afk has quit IRC02:06
*** hamalq has joined #opendev02:57
*** hamalq has quit IRC03:01
*** ykarel__ has joined #opendev03:40
*** whoami-rajat has joined #opendev03:51
openstackgerritIan Wienand proposed opendev/system-config master: Remove gem mirroring puppet  https://review.opendev.org/c/opendev/system-config/+/78359403:59
*** raukadah is now known as chkumar|ruck04:06
ianw#status log remove mirror.gem, test.fedora and a few out-of-date backup volumes used during various transitions to free up space on afs servers04:07
openstackstatusianw: finished logging04:07
ianw /dev/mapper/main-vicepa  4.0T  3.2T  845G  80% /vicepa04:07
ianwwe started at 304G04:07
ianw#status log released all wheel mirrors, some of which appeared to be locked04:08
openstackstatusianw: finished logging04:08
*** ykarel__ is now known as ykarel04:16
*** paladox has quit IRC04:20
ianwI've been seeing periodic " VLDB: no permission access for call" doing this.  i've sent off a query @ https://lists.openafs.org/pipermail/openafs-info/2021-March/043081.html04:26
ianwi feel like it must be something to do with the afsdb servers; but they all look healthy and no log errors AFAICS04:26
*** ricolin has quit IRC04:35
*** ricolin has joined #opendev04:48
*** hamalq has joined #opendev04:58
*** hamalq has quit IRC05:03
*** marios has joined #opendev05:09
*** ysandeep|away is now known as ysandeep05:16
*** brinzhang_ has joined #opendev05:26
*** brinzhang has quit IRC05:29
openstackgerritMerged opendev/system-config master: Add Debian Bullseye to the reprepro config  https://review.opendev.org/c/opendev/system-config/+/77796805:30
openstackgerritMerged opendev/system-config master: Remove gem mirroring puppet  https://review.opendev.org/c/opendev/system-config/+/78359405:30
openstackgerritIan Wienand proposed opendev/system-config master: reprepo debian : fix line-ending  https://review.opendev.org/c/opendev/system-config/+/78359905:42
openstackgerritMerged opendev/system-config master: reprepro: add dist for Ubuntu UCA Wallaby  https://review.opendev.org/c/opendev/system-config/+/78239605:44
ianw^ i am running the bullseye update in a scree now05:49
ianwscreen even05:49
*** ralonsoh has joined #opendev06:25
*** slaweq has quit IRC06:36
*** slaweq has joined #opendev06:39
*** sboyron has joined #opendev06:50
*** hamalq has joined #opendev06:59
*** hamalq has quit IRC07:03
*** hashar has joined #opendev07:03
*** rpittau|afk is now known as rpittau07:15
openstackgerritMoshiur Rahman proposed openstack/diskimage-builder master: Fix: IPA image buidling with OpenSuse.  https://review.opendev.org/c/openstack/diskimage-builder/+/77872307:22
openstackgerritMoshiur Rahman proposed openstack/diskimage-builder master: Fix: IPA image buidling with OpenSuse.  https://review.opendev.org/c/openstack/diskimage-builder/+/77872307:31
openstackgerritDmitriy Rabotyagov proposed openstack/project-config master: Add Debian Bullseye nodepool images and wheels  https://review.opendev.org/c/openstack/project-config/+/78361307:47
*** ysandeep is now known as ysandeep|lunch08:09
*** tosky has joined #opendev08:28
*** elod has quit IRC08:34
*** elod has joined #opendev08:34
*** lpetrut has joined #opendev08:45
*** ykarel is now known as ykarel|lunch08:53
*** hamalq has joined #opendev09:00
*** hamalq has quit IRC09:04
*** noonedeadpunk_ is now known as noonedeadpunk09:08
*** ysandeep|lunch is now known as ysandeep09:26
*** ykarel|lunch is now known as ykarel09:27
*** hashar is now known as hasharLunch09:36
*** CeeMac has quit IRC09:37
*** dtantsur|afk is now known as dtantsur09:41
*** brinzhang0 has joined #opendev09:54
*** brinzhang_ has quit IRC09:58
*** brinzhang_ has joined #opendev10:15
*** sshnaidm|afk has joined #opendev10:18
*** brinzhang0 has quit IRC10:18
*** artom has joined #opendev10:24
openstackgerritDmitriy Rabotyagov proposed openstack/project-config master: Add Debian Bullseye nodepool images and wheels  https://review.opendev.org/c/openstack/project-config/+/78361310:43
openstackgerritMoshiur Rahman proposed openstack/diskimage-builder master: Fix: IPA image buidling with OpenSuse.  https://review.opendev.org/c/openstack/diskimage-builder/+/77872310:44
openstackgerritMoshiur Rahman proposed openstack/diskimage-builder master: Fix: IPA image buidling with OpenSuse.  https://review.opendev.org/c/openstack/diskimage-builder/+/77872310:46
*** CeeMac has joined #opendev10:46
openstackgerritDmitriy Rabotyagov proposed openstack/project-config master: Add Debian Bullseye nodepool images and wheels  https://review.opendev.org/c/openstack/project-config/+/78361310:51
openstackgerritBharat Kunwar proposed openstack/project-config master: [magnum] Remove Review-Priority labels to reduce noise  https://review.opendev.org/c/openstack/project-config/+/78363110:55
openstackgerritDmitriy Rabotyagov proposed openstack/project-config master: Add Debian bullseye wheel cache publish jobs  https://review.opendev.org/c/openstack/project-config/+/78363310:56
openstackgerritDmitriy Rabotyagov proposed openstack/project-config master: Add Debian bullseye wheel cache publish jobs  https://review.opendev.org/c/openstack/project-config/+/78363310:56
*** roman_g has joined #opendev11:00
*** hamalq has joined #opendev11:00
openstackgerritMerged opendev/system-config master: reprepo debian : fix line-ending  https://review.opendev.org/c/opendev/system-config/+/78359911:01
*** hamalq has quit IRC11:05
*** dtantsur is now known as dtantsur|brb11:28
ianwbullseye is still copying.  will check in morning11:28
*** sshnaidm|afk is now known as sshnaidm|off11:45
*** lpetrut has quit IRC11:47
*** roman_g has quit IRC11:53
*** roman_g has joined #opendev11:53
*** roman_g has quit IRC11:53
*** roman_g has joined #opendev11:54
*** roman_g has joined #opendev11:55
*** roman_g has quit IRC11:55
*** roman_g has joined #opendev11:56
*** roman_g has quit IRC11:56
*** roman_g has joined #opendev11:56
*** roman_g has quit IRC11:57
*** lpetrut has joined #opendev11:57
*** roman_g has joined #opendev11:57
*** roman_g has quit IRC11:57
*** paladox has joined #opendev12:32
*** cloudnull has joined #opendev12:41
*** hamalq has joined #opendev13:01
*** hamalq has quit IRC13:06
auristorianw: please check that the UserList on afsdb03.openstack.org matches the ones on afsdb01 and afsdb0213:07
*** dardelean has joined #opendev13:13
*** dtantsur|brb is now known as dtantsur13:15
*** hasharLunch is now known as hashar13:17
*** roman_g has joined #opendev13:19
openstackgerritGuillaume Chauvel proposed opendev/gear master: WIP: trying to solve gear+zuul+ssl(tls1.3)  https://review.opendev.org/c/opendev/gear/+/78123813:21
*** d34dh0r53 has joined #opendev13:32
*** ykarel_ has joined #opendev13:39
*** ykarel has quit IRC13:42
*** d34dh0r53 has quit IRC13:52
*** ysandeep is now known as ysandeep|afk13:53
*** d34dh0r53 has joined #opendev13:55
*** ysandeep|afk is now known as ysandeep13:57
fungiianw: the rollcall vote is on some projects and has been for years (mostly governance and specs type repos). gerrit will dynamically add and remove columns for labels in a review list or dashboard based on which ones are relevant for the changes being listed14:05
*** whoami-rajat has quit IRC14:09
openstackgerritMerged openstack/project-config master: [magnum] Remove Review-Priority labels to reduce noise  https://review.opendev.org/c/openstack/project-config/+/78363114:21
*** lpetrut has quit IRC14:34
*** whoami-rajat_ has joined #opendev14:44
*** marios is now known as marios|call14:51
*** brinzhang_ has quit IRC14:52
corvusfyi i'm looking at some opendev zuul performance metrics in #zuul (in the context of trying to identify if we're seeing zk-related performance changes)14:52
*** brinzhang_ has joined #opendev14:53
*** marios|call is now known as marios14:54
*** ykarel_ is now known as ykarel14:55
fungithanks, and yeah what the graph shows now looks similar enough to what i recall seeing other weeks, but it's hard to say for sure with the ebb and flow of openstack release cycles15:01
*** hamalq has joined #opendev15:02
*** hamalq has quit IRC15:07
*** diablo_rojo has joined #opendev15:10
clarkbkevinz isn't in the channel currently but usually will respond to linaro cloud problems15:11
clarkbcorvus: ^ I can send an email in a bit once I've caught up on scrollback and email and package updates15:12
corvuskeystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://us.linaro.cloud:5000/v3/auth/tokens: HTTPSConnectionPool(host='us.linaro.cloud', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1091)')))15:12
corvusclarkb: ^ that's what's in the logs; welcome back! :)15:12
clarkbthanks!15:13
* corvus acquires bagels15:13
fungiyep, seems the let's encrypt cert being used for that cloud does not renew automatically. this is at least the second time we've seen it expire and need manual intervention15:14
*** toomer has joined #opendev15:15
*** toomer has quit IRC15:16
*** toomer has joined #opendev15:16
clarkbcorvus: potentially related/colliding with the zuul metric gathering is zk and/or zuul scheduler upgrades are next on my list of todos15:16
*** toomer has quit IRC15:17
clarkbcorvus: do you have a preference on which I look at first? I think zk was my initial preference simply because we should be able to do them in a rolling fashion but could do scheduler first15:17
*** toomer has joined #opendev15:17
*** toomer has quit IRC15:24
*** toomer has joined #opendev15:25
corvusclarkb: no pref; i think as long as we note when the changes happen we can account for that when looking at data;  it would be good to do all 3 servers in rapid succesion though, so we're not spending a lot of time with some servers different15:26
fungia thought, we could add a line to our sslcheck config to start monitoring the api cert for clouds we're worried about, so we know if we're about to see them expire15:27
fungimaybe with a low (7 days?) warning time15:27
fungisince we don't know how close to the wire some of them might get before renewing15:27
*** ysandeep is now known as ysandeep|dinner15:28
*** mlavalle has joined #opendev15:31
*** ykarel is now known as ykarel|away15:33
clarkbfungi: not a bad idea15:34
*** hashar has quit IRC15:39
*** ykarel|away has quit IRC15:42
*** Guest76033 is now known as jroll15:50
clarkbI've sent email to kevin now (not fully caught up though)16:00
fungithanks!16:01
*** rpittau is now known as rpittau|afk16:08
openstackgerritMerged zuul/zuul-jobs master: Create a template for ssh-key and size  https://review.opendev.org/c/zuul/zuul-jobs/+/77347416:10
clarkbcorvus: thinking out loud here for a zk rolling upgrades I think the rough plan would be boot three new servers. Then land a change that replaces zk01.openstack.org with zk01.opendev.org. Then do the same for 02 then 03. The tricky bit is we've got a few places where we set the server IPs that also need to be updated. I guess we land additional changes for those if they are in other repos (like16:13
clarkbproject-config?)16:13
clarkbI want to say we do all the connections with IP addrs and not dns names (though the dns names would be updating too)16:13
clarkban alternative would be to make a compeltely new cluster and the cut over at once16:13
openstackgerritMerged openstack/project-config master: Add masakari specs entry  https://review.opendev.org/c/openstack/project-config/+/78355916:15
*** hamalq has joined #opendev16:15
corvusclarkb: i don't think we have any ip addrs in project-config16:17
corvusclarkb: i think the rolling upgrade should work and would be ideal16:17
clarkboh right they get overwritten out of system-config16:17
corvusclarkb: yeah, if you're thinking of nodepool.yaml, ansible templates that in16:18
clarkbya and the zuul config but that also substitutes vars in from our inventory looks like16:19
clarkbso as long as we add the new server and remove the old one it should work from that perspective16:19
clarkbwhile that also updates firewall rules existing tcp connections from the old server will remain open. I think that means we will want to manually stop the zk container on the old side just before the new one is added (to avoid quorum issues)16:20
clarkbcorvus: do you think the size of the cluster needs to change based on any of the data you've collected so far?16:21
clarkbor should I replace with the same flavors?16:21
*** rfayan has joined #opendev16:27
clarkbalso I suppose it should be fairly safe to add the first new server. Double check that things look good on the clients and the cluster. Then do the next two quickly.16:27
clarkbsince we should maintain quorum with two nodes (as if we had a normal outage)16:27
corvusclarkb: i'd keep the same flavors; they appear under-utilized right now; let's reserve that headroom (but i don't think we need more)16:28
clarkbcorvus: sounds good. I doubt I'll get to that today, but will try to start booting instances tomorrow16:29
*** hamalq has quit IRC16:31
*** hamalq has joined #opendev16:31
*** hamalq has quit IRC16:33
*** hamalq has joined #opendev16:33
*** rfayan has quit IRC16:33
*** rfayan has joined #opendev16:34
*** rfayan has joined #opendev16:34
clarkbis gearman used at all in the opendev zuul install at this point (trying to catch up)16:36
corvusclarkb: oh yes, we're quite a ways away from removing gearman16:40
corvusin fact, nothing in gearman has been moved to zk at this point, only internal scheduler queues16:41
*** marios is now known as marios|out16:42
clarkbah16:43
openstackgerritGhanshyam Mann proposed opendev/irc-meetings master: Update OpenStack TC meeting chair  https://review.opendev.org/c/opendev/irc-meetings/+/78371416:43
*** ysandeep|dinner is now known as ysandeep16:47
openstackgerritJeremy Stanley proposed openstack/project-config master: Add Debian Bullseye nodepool images and wheels  https://review.opendev.org/c/openstack/project-config/+/78361316:49
*** marios|out has quit IRC16:54
openstackgerritShturm Svetlana proposed zuul/zuul-jobs master: Fix undefined error for zuul_ssh_key_algorithm  https://review.opendev.org/c/zuul/zuul-jobs/+/78371716:54
openstackgerritMerged opendev/irc-meetings master: Update OpenStack TC meeting chair  https://review.opendev.org/c/opendev/irc-meetings/+/78371416:56
*** ysandeep is now known as ysandeep|away16:57
openstackgerritMerged zuul/zuul-jobs master: Fix undefined error for zuul_ssh_key_algorithm  https://review.opendev.org/c/zuul/zuul-jobs/+/78371717:18
*** auristor has quit IRC17:22
*** ralonsoh has quit IRC17:24
clarkbfungi: do you know if anyone has had a chance to look at the gerrit account classifications I produced near the end of last week?17:41
clarkbthats the other todo item near the top of my list for getting done this week if people are comfortable withi t17:41
*** roman_g has quit IRC17:44
fungiclarkb: ahh, no, i meant to take a look17:49
openstackgerritClark Boylan proposed opendev/system-config master: Upgrade gitea to 1.13.6  https://review.opendev.org/c/opendev/system-config/+/78092318:05
openstackgerritClark Boylan proposed opendev/system-config master: Force gitea to fail for human verification  https://review.opendev.org/c/opendev/system-config/+/78177618:05
clarkbI've set a zuul autohold on ^ which should test the right version now that we've fixed the test dependencies18:06
clarkbthe previous change state to test v1.13.4 looked fine at the time but now there is a v1.13.6 we can update to so I've bumped up the version18:07
*** dtantsur is now known as dtantsur|afk18:07
*** auristor has joined #opendev18:08
*** whoami-rajat_ is now known as whoami-rajat18:31
*** frigo has joined #opendev18:43
*** cloudnull has quit IRC18:43
*** frigo has quit IRC19:20
*** cloudnull has joined #opendev19:21
*** frigo has joined #opendev19:29
*** frigo has quit IRC19:33
clarkbhttps://162.253.55.248:3081/opendev/system-config lgtm. I think 780923 is ready for review19:53
clarkbinfra-root ^ if we can get 780923 reviewed I'm around today to ensure it lands happily19:53
*** rfayan has quit IRC19:53
fungiyep, checks out19:54
fungianother set of eyeballs might be good, but maybe i should single-core approve it while you're around to keep an eye on it19:55
clarkbthe minor releases do tend to be pretty safe19:56
fungiyeah, i've approved it now19:57
clarkbcool, I will keep an eye on it19:59
fungiclarkb: what do you think about the topic:lp-integration approach to a short-term restoration of lp bug updates for now?20:11
fungii figure people can always do manual task assignment20:12
fungiat least that restores the comments20:12
clarkbfungi: I've approved the first one and +2'd the others. Didn't approve the others as I think we needt oapprove them in sequence due to not sharing queues?20:17
clarkbI could be wrong about that though20:17
openstackgerritMerged opendev/gerritlib master: Run gerritlib-jeepyb-integration on ubuntu-focal  https://review.opendev.org/c/opendev/gerritlib/+/78260320:22
clarkbwow that ran quickly. Should I go ahead and approve the next one?20:23
fungiwhich change was that?20:28
fungioh, yeah, feel free20:28
clarkbhttps://review.opendev.org/c/opendev/jeepyb/+/78253820:29
clarkbit has been approved20:29
*** diablo_rojo has quit IRC20:35
*** whoami-rajat has quit IRC20:39
clarkbhrm the system-config-run-base job failed on the gitea upgrade20:46
clarkbI'm trying to edit the meeting agenda then will look at that20:46
ianwfungi: huh, must have somehow got a change in my default with the rollcall vote that pulled the column up then20:49
ianwclarkb: not sure if you saw the token auth work for the gitea job yet?20:49
*** sboyron has quit IRC20:49
clarkbianw: I haven't20:50
ianwclarkb: ok, maybe checkout https://review.opendev.org/c/opendev/system-config/+/782887 before too much digging on the gitea job :)20:51
clarkbianw: it was the base job that failed not gitea proper20:52
*** kopecmartin has quit IRC20:56
*** akahat has quit IRC20:56
clarkbfungi: https://review.opendev.org/c/opendev/system-config/+/781593 is a quick an easy one if you have a moment (noticed it in my catch up)20:58
ianwfwiw had a poke on that gitea server and looks good20:59
*** slaweq has quit IRC20:59
*** fbo has joined #opendev20:59
ianwauristor: yep, it looks like that is the problem.  we don't have that file under configuration control, i guess we should21:01
*** kopecmartin has joined #opendev21:01
*** slaweq has joined #opendev21:01
auristorits not supposed to be configuration controlled.  its a private database of the local bosserver process and is only supposed to be modified via bos commands21:02
fungiso we could probably orchestrate it during deployment i guess21:03
ianwauristor: well, what i mean is that our automated configuration management doesn't create it, so it's not there :)21:03
fungiyeah, we blur the lines between configuration management and orchestration21:04
auristororchestration would be fine.   so would copying the file over in the meantime.21:08
clarkbI'ev rechecked teh gitea change. The base failre was the ansible rc -13 issue when trying to install a package. I wish we could figure that one out21:09
ianwi guess if I write something to interface via bos add/list/remove at least someone else might be able to reuse it.  although i'm tempted to do much less work and just stick the file in ansible and copy it out21:11
ianwinfra-root: the one i'd like to get going (after I look at this UserList issue) is https://review.opendev.org/c/opendev/system-config/+/783183 to add review02.opendev.org ... you can log into the server if you'd like to poke at it21:13
ianwi'd love to get that installed and have a static copy of current data so we can poke at it21:14
ianwthen we can come up with a more concrete checklist for switch-over21:14
fungiianw: i think as long as we have some idea of how to recreate/update that file, it's probably fine21:16
clarkbianw: any idea if review-test is in emergency or not? wondering it it makes sense to test staging on it?21:17
ianwnot too sure, i've left review-test alone.  i feel like if staging causes some particular issue with review02 that makes it unsuitable for production, we can always start again (and fix that issue :) at this point21:18
clarkbya I'm mostly worried about it doing too much but I confirmed not replication targets in your item and thats about all I can think would cause problems21:20
clarkbleft some thoughts and hints for other reviewers but +2'd21:20
*** akahat has joined #opendev21:20
ianwthanks, yeah i expect it to be a high-touch merge21:20
ianwi have a bunch of smaller things in system-config, if you're feeling review-y, but nothing major21:21
ianwclarkb: also, if you'd like to discuss glean, we can :)  i may need a cup of tea prior :)21:21
clarkbI'm up for it21:22
ianwi have to admit i'm still not 100% clear on the "ignore dhcp changes"21:25
ianwanyway i came up with https://review.opendev.org/q/project:opendev%252Fglean+status:open+topic:exec21:25
ianwwhat this essentially does is splits things up so that there is a "early" glean service that sets the ssh keys/hostname21:25
ianwand then the udev rules *only* setup the interface they're called for21:26
ianwi don't know if maybe what ironic wants is actually a way to only run the early service, and ignore NIC's (leaving setup to something else)21:27
ianwbut, independent of that, i think that series of changes is generally correct to actually divide up what glean is doing to avoid running things multiple times21:28
clarkbianw: I think there are a cuple of competing issues in ironic which has complicated the ability to think about it.21:28
clarkbOne of them is that they would like to be able to use the dhcp-all-interfaces role and simple-init together and have simple-init basically do nothing if there is no config drive21:29
clarkbanother is making the cost of running glean lower so that their fake baremetal testing can use glean relaibly with the qemu emulation21:29
ianwthe first one is the one i find odd21:30
clarkbianw: ya I found it odd too, but apparently they added a bunch of functionality to dhcp-all-interfaces that probably should've got into simple-init/glean21:31
clarkbbut that puts then in an awkward spot where sometimes they want both things?21:31
openstackgerritMerged opendev/system-config master: Rename refstack group variables  https://review.opendev.org/c/opendev/system-config/+/78159321:33
clarkbianw: in particular the difference between what glean and dhcp-all-interfaces does is it allows you to bake in config on the image that will work whether or not you've got a config drive later21:33
clarkbianw: I suspect that the use case here is that they make a ramdisk image that all the people use and want it to work whether or not config drive is also used21:34
clarkbthe first change in the stack seems like a good refactor regardless of use case needs21:41
*** slaweq has quit IRC21:44
clarkbianw: having skimmed the changes but not given them proper review I think it makes sense from the perspective of making it easier for people to understand what the pieces are that go into this. My one concern is that I think the glean-early + udev rules make make the low resource use case (fake baremetal on qemu) worse?21:47
clarkbI definitely think the first change is safe eitehr way so will go ahead and give that a proper review now, but maybe we run the rest of the stack by dtantsur|afk ?21:48
clarkbianw: for https://review.opendev.org/c/opendev/glean/+/782010/4/setup.cfg is the files directive just not necessary because pbr is already doing the right thing for us?21:55
clarkb(we still want glean.sh to be installed, but I assume it is handling that via some other mechanism)21:56
clarkboh wait I may have misinterpreted what that directive does. That is what installs to /usr/local/bin/glean.sh21:59
clarkbbut now we're just relying on pbr to put the file in the venv under the module path and we can run it out of there21:59
fungiclarkb: huh... https://review.opendev.org/782538 failed to merge on the same gerritlib test bug which its merged depends-on fixed... looks like https://zuul.opendev.org/t/openstack/build/f0a5a52aadc746db939b93b0c2310481/log/job-output.txt#660 installed gerritlib from a release version?22:03
fungidoes that job install from source or release alternately?22:04
fungithat would be weird, but maybe we did something to integrate cross-repo dependencies which only works up to the point they get merged?22:05
clarkbI thought it was always from source. and we landed the gerritlib change first22:05
fungiexactly22:05
fungihence my confusion on the failure22:06
clarkbhttps://zuul.opendev.org/t/openstack/build/f0a5a52aadc746db939b93b0c2310481/log/job-output.txt#65222:06
fungiyou can even see it's trying to connect to localhost instead of 127.0.0.1 with ssh-keyscan, so it's not using the patched gerritlib22:06
fungioh, yep, i misread, it installed form source22:07
clarkbit does that after it installs jeepyb too to ensure jeepyb doesn't override it22:07
fungiso then did the cached source on the executor lack that commit maybe?22:07
clarkbya maybe it didn't get the newer code somehow22:08
fungiSuccessfully uninstalled gerritlib-0.10.022:08
fungiSuccessfully installed gerritlib-0.10.022:08
clarkbhttps://zuul.opendev.org/t/openstack/build/f0a5a52aadc746db939b93b0c2310481/log/zuul-info/inventory.yaml#68-75 checkout:master22:08
*** cloudnull has quit IRC22:09
fungithe executors fetch from gerrit not from gitea, right?22:10
clarkbare there logs for the workspace setup that will give us a sha?22:10
clarkbfungi: yes22:10
fungiso stalled gerrit->gitea replication wouldn't be a reason22:10
ianwsorry, back now on a fairly sub-optimal 4g connection.  no sync on my cable modem, i'm dreading the isp call "have you tried rebooting your computer"22:10
ianwclarkb: yeah, that first change relies on all the files being on disk and then us referencing them directly, rather than relying on things in /usr/local/bin22:11
fungiopendev.org/opendev/gerritlib checked out to: 99136e1e164baa7b1d9dac4f64c5fb511b813c19 integration test: run manage-projects with debug22:11
fungiclarkb: so that seems to confirm it didn't see the new commit22:12
clarkbhttps://zuul.opendev.org/t/openstack/build/f0a5a52aadc746db939b93b0c2310481/console#0/3/12/ubuntu-focal22:12
fungihttps://zuul.opendev.org/t/openstack/build/f0a5a52aadc746db939b93b0c2310481/log/job-output.txt#320-32322:13
clarkbya I think that is the clue we want22:13
clarkbcould this be related to zuul's change to make sure an entire buildset uses the same refs for all repos/22:13
fungiclarkb: i wouldn't think so... the buildset should have been consistent either way22:18
clarkbfwiw I expected that it wouldn't work with depends on which is why I waited. But really odd to see this after a merge22:18
fungiwell, it passed check using the same depends-on22:20
fungiso i would be surprised if it tested differently in check and gate in that regard22:20
clarkboh I see huh22:21
clarkbin that case ya it really seems like it didn't properly update the master checkout which is supposed to happen with a gearman job to the mergers iirc22:21
fungihence my confusion22:22
fungianyway, i rechecked to see if it repeats22:22
openstackgerritIan Wienand proposed opendev/system-config master: openafs-server-config: install UserList  https://review.opendev.org/c/opendev/system-config/+/78375522:30
fungiianw: that ^ looks incomplete. intentional?22:48
ianwfungi: ahh, yes that is a gid add failure sorry22:49
openstackgerritIan Wienand proposed opendev/system-config master: openafs-server-config: install UserList  https://review.opendev.org/c/opendev/system-config/+/78375522:50
fungicool, figured that was it22:50
ianwsorry, as predicted currently dealing with someone who wants to know the mac address of my wireless router to fix a non-syncing cable modem :/22:52
*** cloudnull has joined #opendev22:52
ianwfungi: yeah, we probably want to remove a few old entries from the PTS, and that list22:59
clarkbinternet problems are the worst23:27
openstackgerritIan Wienand proposed opendev/system-config master: gitea: switch to token auth for project creation  https://review.opendev.org/c/opendev/system-config/+/78288723:37
fungibroadband service repair people, zomg23:37
clarkbwaiting patiently for the gitea update to land and I realize there is a full deploy set of jobs in progress due to that refstack change modifying inventory stuff :)23:38
clarkbit should move along pretty quickly at this point, but still /me is impatient23:38
openstackgerritIan Wienand proposed opendev/system-config master: gitea: switch to token auth for project creation  https://review.opendev.org/c/opendev/system-config/+/78288723:39
openstackgerritMerged opendev/system-config master: Upgrade gitea to 1.13.6  https://review.opendev.org/c/opendev/system-config/+/78092323:43
openstackgerritMerged opendev/system-config master: openafs-server-config: install UserList  https://review.opendev.org/c/opendev/system-config/+/78375523:43

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!