19:01:16 #startmeeting infra 19:01:17 Meeting started Tue Oct 15 19:01:16 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:18 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:20 The meeting name has been set to 'infra' 19:01:26 #link http://lists.openstack.org/pipermail/openstack-infra/2019-October/006493.html Our Agenda 19:01:33 #topic Announcements 19:01:48 The OpenStack release is happening this week (~tomorrow) 19:02:09 I believe the only potential need for infra help is requeuing any of the openstack release teams' tagging jobs 19:02:20 if they happen to timeout (there was a change to increase the timeout on them to avoid that in the first place) 19:02:46 but I guess keep an eye out for any issues they run into that we can help with 19:02:51 tagging job > 30 min? 19:03:06 corvus: ya it serially ran through every project created a tag for it and pushed it to gerrit 19:03:21 and apparently when you do that over several hundred projects you take longer than 30 minutes (this happened with stein) 19:03:31 i see 19:03:39 the job won't retag and repush things so it can be requeued and it basically skips ahead to where it left off safely 19:04:04 i don't know that it happened with stein 19:04:15 oh it happend at some point 19:04:16 it happened at stein and train rc1 19:04:19 ah 19:04:26 because those also involve branch creation on every project 19:04:31 and pushing new changes to each of them 19:04:52 branch creation is slightly more expensive I guess? 19:05:07 with all the associated change proposals it is 19:05:20 ah - yeah - that makes sense 19:05:43 we're just being extra cautious. i plan to be at the computer by 11:30z and the button mashing commences at 12:00z i believe 19:06:16 I fully endorse everything being done by the time I wake up :) 19:06:19 i expect it will be mostly done, including the follow-on jobs triggered by the tag pushes, by 13:00z 19:06:45 other things to consider: we probably want to avoid zuul restarts tomorrow (at least until fungi gives us the all clear) 19:06:56 unless something goes sideways, and even then that leaves us hours until the osf has the usual press release planned and the associated press embargoes lift 19:07:12 which i think is usually around 15:00z or thereabouts 19:07:56 * diablo_rojo sneaks in late 19:08:16 but ya I don't anticipate problems. We've fixed or are fixing a bunch of our known infra issues recently and of those I don't think they would really affect the openstack release 19:08:24 in release-related announcements, i also created the ussuri signing key a few weeks ago: 19:08:28 #link https://sks-keyservers.net/pks/lookup?op=vindex&search=0xbba3b1e67a7303dd1769d34595bf2e4d09004514&fingerprint=on OpenStack Infra (Ussuri Cycle) artifact signing key 19:08:43 (things like NM on red hat distros and the debian buster mirror) 19:09:15 #link https://docs.openstack.org/infra/system-config/signing.html#attestation infra signing key attestation procedure 19:09:32 i'll push up changes to switch the train key out for that in a week or so 19:10:19 Anything else to announce? 19:10:32 * fungi knows of nothing else 19:10:57 #topic Actions from last meeting 19:11:03 #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-10-08-19.01.txt minutes from last meeting 19:11:20 I started drafting up a thing for opendev governance 19:11:28 #link https://etherpad.openstack.org/p/rCF58JvzbF OpenDev governance email draft 19:11:45 I think it is still a bit rough and I've tried to bring up a few of the things I expect peopel to question/bring up 19:12:22 in particular how an OpenDev PTL is chosen, whether we need tie breaker rules on the proposed council and whether we want the "else" group to be covered there too 19:12:37 I think all of that is open for discussion at this point so please feel free to leave comments on the etherpad 19:13:31 clarkb: thanks! after collecting etherpad comments, what do you think is next? 19:14:31 corvus: I think draft a final version that we can all be happy with as a group, then take that to the TC as a "look this is what we'd like to do and we'd like your input and go ahead to do this formally" 19:14:45 kk 19:14:47 I don't expect that to present any problems but would be good to make it all official like 19:16:34 we should probably also ask the other groups that will be part of the council for input 19:16:44 since we will be asking them to participate 19:16:59 fungi: ^ that may help answer the question re kata 19:17:05 "does this even interest you?" 19:17:12 makes sense 19:18:00 Probably best to dive into this on the etherpad once we've collected more thoughts and opinions 19:18:10 we can come back to this at the ned of the meeting if we have time too 19:18:18 #topic Priority Efforts 19:18:22 #topic OpenDev 19:18:30 Any other opendev topics to bring up? 19:19:25 Sounds like no 19:19:27 what's our current opendev focus? 19:19:43 for me it has become the governance topic 19:19:54 okay, cool. just making sure i know where we are in the transition 19:20:06 Based on complaints I think our git hosting is fairly stable at this point other than the performance issue on large repos 19:20:07 and i guess the gerrit upgrade 19:20:14 which is a known upstream issue with people working on it 19:20:14 and the zuul log storage switch 19:20:31 mordred's is gerrit upgrade 19:20:43 right, i meant more collectively 19:21:04 and ianw is driving static retirement 19:21:10 like what do we want to complete before we start moving more services into opendev 19:21:24 those all sound like good things to get behind us first 19:21:31 more services like? 19:21:35 like adding services, or renaming? 19:21:38 etherpad et cetera 19:21:58 well, it's never been clear to me if something is officially opendev while it's still on an openstack.org domain 19:22:02 i wouldn't put up any blockers to moving anything over at this point 19:22:06 (on static; yes very distracted with glean + centos8 this week, but will get back to updating that spec as discussed last week) 19:22:13 yah- other than available human time 19:22:18 ya I think we can start shifting services like etherpad as soon as people have time to do them 19:22:26 clarkb: ++ 19:22:43 got it. i know we were pushing back on that when the focus was still on stabilizing the gerrit/gitea migration 19:22:46 i know i said "let's not move etherpad" in the past, because i think we needed all hands on deck for gerrit migration, but i don't feel like any current effort is like that 19:22:48 it might make moving services slightly easier if we prioritize the dns proposal implementation 19:22:55 clarkb: ++ 19:22:56 awesome. thanks! 19:23:16 and yes, i agree the dns/ssl plan is a high priority for that 19:23:18 speaking of the dns proposal I did bring it up with osf staff and heard no objections 19:23:28 jbryce did say it would be this week or next before he could really read and digest it though 19:23:29 groovy 19:23:30 not that there was really anything to object to 19:24:00 and there's some possible interest in moving the openstack.org domain into our dns hosting, just not sure how much and on what timeline yet 19:24:22 but i think the etherpadded plan is a simple interim solution even so 19:24:48 yeah, it won't produce any roadblocks to future work like that 19:26:36 thanks, that covers my questions. we just hadn't done a "where are we now?" on this in a few months 19:26:57 To summarize, mordred is working on improvements to gerrit config management using containers to help enable a big gerrit upgrade, ianw is working on static.openstack.org cleanup, clarkb is working on the governance side of things. Others should feel free to pick up other service "moves" which might be made easier if we focus on the dns proposal first 19:27:07 https://etherpad.openstack.org/p/openstack-org-dns is the dns proposal 19:27:57 #topic Update Config Management 19:28:21 mordred: anything new on gerrit container images? 19:28:38 there's some patches up to update to bazel 1.0 19:28:46 which are required to land the latest changes 19:28:59 that opened up a can of worms with puppet - which I think is finally unwound 19:29:30 https://review.opendev.org/#/c/688523/ and friends 19:30:02 ya that prompted some cleanup of the centos-7 puppet jobs we were running 19:30:20 since I missed the last few meetings - the current status is that review-dev is running on the container image - so we know that bit works - the next bit is finishing the patch to actually be able to do that via config management 19:32:04 at some point it might be good to rethink our gerrit server-side hook scripts that use jeepyb to be zuul jobs - as they currently require us to add jeepyb into the gerrit image itself - but it's not urgent 19:32:25 ++ 19:32:43 also - we've got some patches up to add master-based images with checks plugin enabled that we can use for zuul testing - and eventually potentially be more ready for rollouts as we go forward 19:34:55 That sounds good. Anyone else have updates related to config management? 19:36:01 Lets move on 19:36:06 #topic Storyboard 19:36:17 fungi: diablo_rojo anything new to report here? 19:36:51 we moved the storyboard "weekly" irc meeting to 14:00z thursdays 19:37:23 in hopes we have critical mass for it to be closer to weekly in frequency again 19:38:13 nothing new from me on the team management automation front yet. i've started fleshing out the module last week but haven't gotten to pushing anything into gerrit 19:38:23 focus has mostly been elsewhere 19:38:51 though in the sb meeting last week we got consensus on the direction i'm taking it 19:40:58 and that direction is using the api to manage group membership in storyboard instead of the database 19:41:13 yep 19:41:25 via ansible 19:41:26 great. Anything else? 19:41:42 nothing i'm aware of 19:42:06 #topic General Topics 19:42:21 fungi: I noticed there were some wiki changes today, I assume we should all go and review those? 19:42:30 i've imported fresh mysqldump and image tree copy from the production wiki onto wiki-dev, and then discovered mediawiki has a number of unpackaged dependencies which need installing via a php tool called "composer" 19:42:46 #link https://review.opendev.org/#/q/project:opendev/puppet-mediawiki+is:open+topic:xenial-upgrades 19:42:49 "neat" 19:43:18 yeah, i see one has a beaker failure, need to dig into it 19:43:52 also i've basically done that stuff manually on wiki-dev just to see what breaks next 19:44:19 it's now giving me no output instead of an error in the browser, so time to go hunting in logs 19:44:38 progress at least 19:44:41 yep 19:44:41 fungi: your beaker failed is the thig I just fixed for gerrit 19:44:47 oh, neat 19:44:53 thanks mordred! 19:45:01 will look after the meeting 19:45:06 ianw for static.o.o I think we can go ahead and update the spec with the dns proposal changes involved 19:45:07 so once https://review.opendev.org/#/c/688555/ lands (has +A) - we can recheck that patch and it shoiuld be fine 19:45:27 ianw: other than waiting for new patchset to review on that is there anything we can do to help re static.o.o? 19:46:06 clarkb: yes, will do for sure 19:46:31 sorry, at this point i think it's stalled on us getting ducks in a row on spec 19:46:45 /s/on/in/ 19:46:55 just needs some duck sauce then 19:47:16 no worries, just checking 19:47:24 Next up I wanted to bring up CentOS 8. 19:47:39 I think ianw managed to track down the network problems we've had on red hat distros (including 8) and those should now be fixed 19:47:54 please keep an eye out for odd behavior on centos and fedora on limestone and fortnebula in particular but we should be good now 19:48:09 zbr and ianw have updates to fix mirror management on centos 8 in zuul-jobs 19:48:19 yep, and last night the configure-mirror changes went in, so devstack can start 19:48:19 well done ianw! 19:48:27 I want to say enough of that has now merged that we should be able to run jobs on centos 8 now and have their actual workloads run 19:48:27 ergo all the setup roles are working 19:48:32 yay! 19:48:35 excellent thank you for confirming 19:48:39 that was quite the digging for that one 19:48:55 yep, i plan to announce the images to the list, along with some of the caveats about it being python3 only (by default) today 19:49:22 tomorrow I am going to start enabling new jobs 19:49:22 ++ thank you for getting that done 19:49:39 also apparently we've managed to build centos 8 cloud images before upstream so some individuals have come to us looking for them :) 19:50:01 hilarious 19:50:04 but also sad 19:50:12 yes, i need to sync with maybe karinbar there ... there are kickstarts, and their infra is building them but not publishing for some reason 19:50:24 ianw: not sure if advertising is really needed,... i prefer to slowly add jobs, so we have time to fix the broken bits. 19:50:51 zbr: I think in the past we've not been aggressive enough in advertising updated distros which leads to peopel dragging their feet and we end up not updating as quickly as possible 19:50:52 zbr: i'd like people to go in with a heads up that they're different wrt to pip/virtualenv setup to all our other images 19:51:17 I think we should let everyone know that they can (and should) update now that we are happy with it and that will help shake out any additional problems 19:52:11 Last up on the agenda is the PTG 19:52:20 #link https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 Add your PTG topic ideas here 19:52:45 #link https://www.openstack.org/ptg/#tab_schedule PTG schedule 19:53:01 I'm told I should hear back on my visa tomorrow. I am cautiously optomistic that there will be good news 19:54:22 #topic Open Discussion 19:54:26 clarkb: I hope to see you in Shanghai 19:54:30 mordred: me too :) 19:54:41 I'll open it up to further discussion on past topics or new items 19:54:51 fwiw - I'm now in Asia until the Summit (in Japan timezone) 19:55:27 the debian mirror seems to be updated now, though seeing different build failures around the debian-security mirror now 19:55:32 hangin' in the +9 19:55:32 mordred: is there a secret goal of visiting every timezone? 19:55:39 i may need to perform some similar reprepro surgery on it 19:55:44 clarkb: not very secret :) 19:56:15 my intent is to work hours that are non-normal here so I have more overlap with folks - but I'm jetlagged at the moment and have't 100% sorted out what that looks like yet 19:57:15 mordred: cool, thanks for keeping us updated 19:57:50 Oh we are removing opensuse 42.3 images from nodepool 19:57:56 thank you to ajaeger and corvus for pushing on that 19:58:12 there is one devstack change we are waiting on before rechecking the nodeset removal to see if that will go in now 19:58:17 ++ 19:58:22 f28 should probably be next 19:58:26 and opensuse 150 19:58:35 but ya agreed 19:58:59 i'm guessing we should hold off on the git repo cache move until friday? 19:59:04 or at least post-release? 19:59:18 ++ 19:59:27 that has the potential to have widespread impact probably best to play it safe 19:59:33 i'm basically waiting for clarkb and fungi to tell me it's okay to break the debian-whatever and old suse images... 19:59:44 the ones that haven't updated in ages 20:00:09 and we are at time 20:00:09 https://review.opendev.org/682935 20:00:24 thank you everyone. Feel free to continue discussion in #openstack-infra 20:00:26 #endmeeting