Wednesday, 2021-03-10

openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884000:16
*** brinzhang has joined #opendev00:22
openstackgerritIan Wienand proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI  https://review.opendev.org/c/opendev/system-config/+/77629200:28
*** tosky has quit IRC00:36
*** avass has quit IRC00:41
*** yourname has joined #opendev00:42
*** mlavalle has quit IRC01:27
*** iurygregory_ has joined #opendev01:30
*** mattmceuen_ has joined #opendev01:31
*** dviroel_ has joined #opendev01:31
*** iurygregory has quit IRC01:38
*** mordred has quit IRC01:38
*** dviroel has quit IRC01:38
*** ttx has quit IRC01:38
*** mattmceuen has quit IRC01:38
*** dviroel_ is now known as dviroel01:39
*** mattmceuen_ is now known as mattmceuen01:39
*** ttx has joined #opendev01:47
*** mordred has joined #opendev01:54
*** hamalq has quit IRC02:14
*** ysandeep|away is now known as ysandeep03:04
openstackgerritIan Wienand proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI  https://review.opendev.org/c/opendev/system-config/+/77629203:09
*** ysandeep is now known as ysandeep|afk03:50
*** stevebaker has quit IRC03:58
*** stevebaker has joined #opendev04:02
*** ykarel has joined #opendev04:16
*** ysandeep|afk is now known as ysandeep05:20
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884005:29
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884005:35
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884006:00
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884006:02
fricklerseems ovh is having some major issues, but not at the sites we're using. good luck to them anyway http://travaux.ovh.net/?do=details&id=49484&06:18
*** whoami-rajat has joined #opendev06:23
*** marios has joined #opendev06:28
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884006:42
fricklerouch https://twitter.com/abonin_DNA/status/1369538028243456000 also, did they really just put up a huge stack of shipping containers there?06:57
ianwyeah :(07:02
*** ralonsoh has joined #opendev07:21
*** slaweq has quit IRC07:42
*** eolivare has joined #opendev07:46
*** slaweq has joined #opendev07:46
*** yourname is now known as avass07:48
*** slaweq has quit IRC07:52
*** sboyron has joined #opendev07:53
*** slaweq has joined #opendev07:54
*** ysandeep is now known as ysandeep|lunch07:58
*** fressi has joined #opendev08:01
*** lpetrut has joined #opendev08:06
*** hashar has joined #opendev08:15
*** brinzhang_ has joined #opendev08:20
*** gothicserpent has quit IRC08:21
*** brinzhang has quit IRC08:23
*** gothicserpent has joined #opendev08:24
*** arxcruz|off is now known as arxcruz08:24
*** rpittau|afk is now known as rpittau08:27
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884008:30
*** ysandeep|lunch is now known as ysandeep08:37
*** DSpider has joined #opendev08:43
*** tosky has joined #opendev08:46
*** hashar has quit IRC08:55
*** jpena|off is now known as jpena08:58
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884009:00
*** andrewbonney has joined #opendev09:03
*** hashar has joined #opendev09:04
openstackgerritBenjamin Schanzel proposed zuul/zuul-jobs master: ensure-helm: skip helm init for helm v3 and newer  https://review.opendev.org/c/zuul/zuul-jobs/+/77969809:39
*** ykarel is now known as ykarel|lunch10:00
openstackgerritBenjamin Schanzel proposed zuul/zuul-jobs master: ensure-helm: skip helm init for helm v3 and newer  https://review.opendev.org/c/zuul/zuul-jobs/+/77969810:03
*** DSpider has quit IRC10:07
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884010:34
*** Eighth_Doctor has quit IRC10:36
*** mordred has quit IRC10:36
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884010:39
*** klonn has joined #opendev10:41
*** Eighth_Doctor has joined #opendev10:46
*** ysandeep is now known as ysandeep|away10:50
*** hashar has quit IRC10:53
*** mordred has joined #opendev11:03
*** dtantsur|afk is now known as dtantsur11:07
*** ysandeep|away is now known as ysandeep11:09
*** toomer has joined #opendev11:15
*** ykarel|lunch is now known as ykarel11:18
*** iurygregory_ is now known as iurygregory11:33
openstackgerritMoshiur Rahman proposed openstack/diskimage-builder master: Fix: IPA image buidling with OpenSuse.  https://review.opendev.org/c/openstack/diskimage-builder/+/77872311:56
*** eolivare_ has joined #opendev12:16
*** ykarel_ has joined #opendev12:17
*** eolivare has quit IRC12:19
*** ykarel has quit IRC12:20
*** jpena is now known as jpena|lunch12:34
*** klonn has quit IRC13:05
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: Bits to keep ansible-lint happy  https://review.opendev.org/c/zuul/zuul-jobs/+/77927613:11
*** ykarel_ is now known as ykarel13:15
*** brinzhang_ has quit IRC13:23
*** brinzhang_ has joined #opendev13:24
priteauHello. FYI we just had an error in the promote-openstack-releasenotes job in Kayobe.13:38
priteaursync: close failed on "/afs/.openstack.org/docs/releasenotes/kayobe/_static/css/.font-awesome.css.wrAMwd": Input/output error (5)13:38
*** jpena|lunch is now known as jpena13:40
*** klonn has joined #opendev13:44
priteauhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_64d/775936/4/promote/promote-openstack-releasenotes/64d8728/job-output.txt13:47
*** klonn has quit IRC13:49
*** auristor has quit IRC14:03
*** auristor has joined #opendev14:06
*** jrollen is now known as jroll14:13
fungipriteau: looks like that ran from ze05, i've checked out the executor and it doesn't seem to have any persistent issues communicating with the afs infrastructure. it's more likely that build raced another build of the same job and one rsync tried to remove the tempfiles the other rsync was creating. i'll see if i can find stronger evidence14:38
priteauThere were multiple Kayobe patches merging at the same time so it's plausible14:39
fungiusually that happens if there are changes merging for different branches of the same project at the same time, since the release notes all want to write to the same unversioned place, and we only supersede promote refs when they're for the same branch14:39
priteauYep, makes sense14:39
priteauIt's very likely this then, no need to dig more14:39
fungiwell, the up side is that the next releasenotes build to run will update it all again anyway14:40
*** ysandeep is now known as ysandeep|away15:11
*** toomer has quit IRC15:37
openstackgerritMerged openstack/project-config master: [magnum] Add Backport-Candidate and Review-Priority labels  https://review.opendev.org/c/openstack/project-config/+/77862915:43
*** lpetrut has quit IRC15:44
*** klonn has joined #opendev15:48
*** dmsimard8 has joined #opendev15:52
*** dmsimard has quit IRC15:54
*** dmsimard8 is now known as dmsimard15:54
openstackgerritSofia Enriquez proposed opendev/irc-meetings master: Add Cinder bug squad meeting  https://review.opendev.org/c/opendev/irc-meetings/+/77981815:54
*** toomer has joined #opendev15:57
*** ysandeep|away is now known as ysandeep16:08
*** marios is now known as marios|out16:37
*** marios|out has quit IRC16:43
*** hashar has joined #opendev16:54
*** toomer has quit IRC17:04
*** rpittau is now known as rpittau|afk17:05
*** mlavalle has joined #opendev17:06
*** dtantsur is now known as dtantsur|afk17:15
openstackgerritAntoine Musso proposed opendev/gear master: wakeConnections: Randomize connections before scanning them  https://review.opendev.org/c/opendev/gear/+/74711917:15
hasharhi, is there still any interest in opendev/gear ?   There are a few pending changes that seems straightforward :]17:19
fungihashar: it's seemed at least close to "feature-complete" for some years, but if there are fixes or in-scope improvements then we'll try to review them17:20
fungithough also its importance to us may shrink with zuul 5.0.0 and the distributed scheduler work moving that coordination into zookeeper17:22
hasharthere are a few trivial ones though that should not take too much time.   The one I hhave rebased above  better balances the workload among worker and addressed some issue we had on our setup ;)17:22
fungii'll try to take a look after i get out of my current meeting17:22
hasharthere are a few others that sounds trivial enough, I have +1 ed anything that looked sane/straightforward17:23
hasharsome others are a bit more challenging I guess or super old. So I guess they will end up being abandoned eventually17:23
fungithanks for the heads up, and for going through the open reviews!17:24
fungithat's a big help17:24
hasharI felt like if I need my change approved, I might as well spend sometime reviewing others patches ;≠17:24
hasharhappy meeting fungi!17:24
fungisame to you!17:25
hashar(the open patches I have +1ed https://review.opendev.org/q/project:opendev/gear+is:open+label:Code-Review%252B1%252Chashar  )17:27
*** akrpan-pure has joined #opendev17:40
*** klonn has quit IRC17:45
*** ykarel has quit IRC17:50
*** jpena is now known as jpena|off17:58
*** SWAT has quit IRC17:58
*** ysandeep is now known as ysandeep|holiday18:03
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: Bits to keep ansible-lint happy  https://review.opendev.org/c/zuul/zuul-jobs/+/77927618:04
*** SWAT has joined #opendev18:11
corvusclarkb: what's your zuul component restart status?18:14
fungiall mergers and executors have been rebuilt with ubuntu focal servers, and the old ones cleaned up. he's working on nodepool launchers next, then builders i think?18:17
*** klonn has joined #opendev18:17
fungischeduler was going last18:18
fungioh, also zk servers are in there somewhere18:18
fungithey're not done yet18:18
corvusok; i'd like to restart all of zuul today if conditions warrant; i'm thinking after my lunch based on the node request graph18:19
corvuswe merged a whole pile of changes and it'd be good to exercise them in prod18:19
*** andrewbonney has quit IRC18:19
fungisounds good to me. he and i are in a meeting which is running over18:20
fungibut i'll be freed up soon18:20
clarkbya should be fine18:20
corvuscool, i'll check back around 210018:21
clarkbnext up is launchers I want to see if I can get https://review.opendev.org/c/zuul/nodepool/+/779616 in to make that easier18:21
clarkbmy plan is land that change, restart current launchers on it and make sure it is happy. Then build a new nl01.opendev.org and do a switchover18:21
clarkbbuilders are already rebuilt. After launchers is zk or scheduler. I haven't sorted out which I'd prefer there yet (though suspect it is zk)18:22
*** akrpan-pure has quit IRC18:22
corvusclarkb: +2 from me; maybe if someone +3s it soon we can roll that into a restart this afternoon18:23
clarkbcorvus: ok. I can also do it later without much trouble so don't hold up restarts on that18:23
corvusk18:24
clarkbthat user that ended up in a confused state with gerrit accounts has confirmed that the account was never properly used for anything and they deleted the corresponding ubuntu one account. That means we should be able to retire it and remove all email external ids so that they can start over again.18:27
clarkbis any other infra-root interested in working through that process? We've got all the tooling for the preexisting account cleanups ready to go and thought it might be useful to have someone else work through one too18:28
clarkbmight also help grow confidence in the tooling (then we can land that change?)18:32
*** eolivare_ has quit IRC18:36
*** tbarron has joined #opendev18:37
*** ralonsoh has quit IRC18:41
*** SWAT has quit IRC18:42
*** SWAT has joined #opendev18:45
openstackgerritMerged opendev/gear master: Modify connection timeout process  https://review.opendev.org/c/opendev/gear/+/53105919:05
clarkbre the gerrit user situation, I don't think gerrit got this entirely wrong in this case. The user deleted the source account and tried again. THis is why the email addrs didn't change but the openid did19:09
clarkbhowever, it definitely would be nice if gerrit had an easy mode way to say "this external id belongs to this account"19:10
clarkbI suppose we get half easy once the other external id conflicts are sorted and can push to gerrit for this stuff19:10
openstackgerritMerged opendev/gear master: remove unicode from code  https://review.opendev.org/c/opendev/gear/+/76967119:26
*** brinzhang0 has joined #opendev19:33
*** redrobot2 has joined #opendev19:34
*** Dmitrii-Sh4 has joined #opendev19:34
*** dpawlik6 has joined #opendev19:34
*** stevebaker_ has joined #opendev19:36
*** mugsie__ has joined #opendev19:37
*** tosky_ has joined #opendev19:37
*** whoami-rajat has quit IRC19:40
openstackgerritMerged opendev/gear master: Move handleDisconnect into BaseClientServer  https://review.opendev.org/c/opendev/gear/+/71470919:41
openstackgerritMerged opendev/gear master: wakeConnections: Randomize connections before scanning them  https://review.opendev.org/c/opendev/gear/+/74711919:41
*** paladox_ has joined #opendev19:41
*** artom has quit IRC19:42
*** SWAT has quit IRC19:42
*** dmsimard has quit IRC19:42
*** brinzhang_ has quit IRC19:42
*** tosky has quit IRC19:42
*** stevebaker has quit IRC19:42
*** dpawlik has quit IRC19:42
*** Dmitrii-Sh has quit IRC19:42
*** paladox has quit IRC19:42
*** mugsie has quit IRC19:42
*** redrobot has quit IRC19:42
*** dpawlik6 is now known as dpawlik19:42
*** Dmitrii-Sh4 is now known as Dmitrii-Sh19:42
*** dmsimard has joined #opendev19:43
*** redrobot2 is now known as redrobot19:43
*** irclogbot_0 has quit IRC19:44
*** paladox_ is now known as paladox19:45
*** mugsie__ is now known as mugsie19:47
*** SWAT has joined #opendev19:48
*** stevebaker_ is now known as stevebaker20:00
*** artom has joined #opendev20:01
*** tosky_ is now known as tosky20:16
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884020:23
openstackgerritSteve Baker proposed openstack/diskimage-builder master: Support secure-boot bootloader where possible  https://review.opendev.org/c/openstack/diskimage-builder/+/77910720:37
*** SWAT has quit IRC20:37
*** SWAT has joined #opendev20:40
openstackgerritClark Boylan proposed openstack/project-config master: Startup nl01.opendev.org idle  https://review.opendev.org/c/openstack/project-config/+/77986320:46
openstackgerritClark Boylan proposed openstack/project-config master: Flip nl01.openstack.org to nl01.opendev.org  https://review.opendev.org/c/openstack/project-config/+/77986420:46
clarkbthat second change I'll WIP now but I think that captures roughly how we can do the flip over once https://review.opendev.org/c/zuul/nodepool/+/779616 lands and a new instance is deployed20:46
clarkbthe first one should be safe to land whenever though I just have to wait on that for the new nl01.opendev.org20:47
clarkb(as well as the nodepool change)20:47
*** tosky has quit IRC20:49
*** tosky has joined #opendev20:49
*** hashar has quit IRC20:52
*** klonn has quit IRC20:57
clarkbre zuul restarts, I remembered that it is feature freeze week. Queues don't look terrible but they aren't super small either20:59
clarkbit does look like a good chunk of changes should finish gating in the next 20 minutes though and that might be a good opportunity?20:59
*** mtreinish has quit IRC21:09
fungii'm about to eat dinner, but am around to help21:10
*** mtreinish has joined #opendev21:11
fungilooks like we've still got a hefty node request backlog (>1k)21:12
fungiand yeah, looking at the openstack gate pipeline graph, it's way shorter than it was a few hours ago21:14
*** hashar has joined #opendev21:14
corvusclarkb: do you want to veto the restart until next week?21:18
corvus(or this weekend)21:20
fungiclarkb: looks like those two integrated gate changes just merged if we want to do it now21:20
clarkbcorvus: I think its probably ok based on the size of the queues and their recent resets. However, I don't know what sorts of changes we want to validate (is it your zk work?)21:20
clarkbcorvus: I think feature freeze officially ends tomorrow so we could do it on friday maybe21:21
clarkb(if we wanted to be more careful)21:21
clarkbput another way I'd probably be more concerned that a restart will be buggy and prevent changes from merging than the delay in the queues right now. I just don't have a good way of evaluating the risk for that21:24
clarkbthough I am pulling up the git log now21:24
clarkblooks like the big chagne is the cycle support?21:26
openstackgerritSteve Baker proposed openstack/diskimage-builder master: Support secure-boot bootloader where possible  https://review.opendev.org/c/openstack/diskimage-builder/+/77910721:28
clarkbfor the zk changes we can probably restart a single executor after asking it to gracefully stop and use that as a canary21:28
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884021:30
corvusclarkb: the cycle changes will be new to opendev but have seen use elsewhere; i think the global state has never been in production.  so there's definitely a risk of error, and the restart is explicitly to surface errors.21:33
clarkbcorvus: 'Make repo state buildset global' that change?21:34
corvusyep21:34
corvusclarkb: however, depending on exactly when you restarted things, that may already be in [partial] production21:34
corvusthere is an executor/merger aspect to that change21:34
corvusi'm unsure if it needs a scheduler restart to be activated though21:34
corvus(at least, that's my recollection, i haven't looked)21:35
clarkbcorvus: I believe all mergers and executors were started on code prior to the merge timestamp for that change21:36
corvusthe node request backlog is higher than i expected; i'd be fine deferring until friday21:37
clarkbthe last zuul executor would've been started on march 8th local time and gerrit says that change landed march 9 morning local time21:37
clarkbcorvus: ya I think if we're looking for errors then waiting for friday is probably best. I'll dobule check on the feature freeze timing now too21:38
clarkbI've asked the release team to confirm, but lets say friday for now21:41
*** TheJulia has quit IRC21:45
*** TheJulia has joined #opendev21:45
*** johnsom has quit IRC21:46
*** johnsom has joined #opendev21:46
openstackgerritGage Hugo proposed openstack/project-config master: Add create ref acl for osh release groups  https://review.opendev.org/c/openstack/project-config/+/77987221:47
*** SWAT has quit IRC21:49
*** SWAT has joined #opendev21:52
*** slaweq has quit IRC21:55
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884022:11
openstackgerritSteve Baker proposed openstack/diskimage-builder master: Only add rhel base repos when REG_REPOS is not set  https://review.opendev.org/c/openstack/diskimage-builder/+/77987422:15
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884022:34
*** hashar has quit IRC22:45
*** sboyron has quit IRC22:56
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884023:18
*** rpittau|afk has quit IRC23:37
*** rpittau|afk has joined #opendev23:37
*** dmellado_ has joined #opendev23:47
*** dmellado has quit IRC23:48
*** dmellado_ is now known as dmellado23:48
openstackgerritIan Wienand proposed opendev/system-config master: [wip] kerberos ansible  https://review.opendev.org/c/opendev/system-config/+/77884023:56

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!