19:01:04 #startmeeting infra 19:01:05 Meeting started Tue Apr 23 19:01:04 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:06 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:08 The meeting name has been set to 'infra' 19:01:18 #link http://lists.openstack.org/pipermail/openstack-infra/2019-April/006333.html 19:01:32 o/ 19:03:17 I think people are fairly preoccupied with all the things today. Myself included so we'll let people trickle in 19:03:44 * mordred trickles 19:04:01 #topic Announcements 19:04:20 The Summit and PTG are next week. That means no infra meeting on the 30th 19:04:35 Looking forward to seeing people that are attending 19:05:22 I believe we have a project update that I need to start putting content together for but I've yet to have a moment to put a schedule together 19:06:04 #topic Actions from last meeting 19:06:14 #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-04-16-19.01.txt minutes from last meeting 19:06:30 No real actions recorded other than the biggest gerrit migration we've ever done 19:06:44 so maybe we should just skip ahead a bit and start talking about how that went and what needs attention next 19:06:54 #topic Priority Efforts 19:07:00 #topic OpenDev 19:07:01 clarkb: to the best of my knowledge, everythign is working perfectly right? ) 19:07:13 mordred: all things considered things are working quite well 19:07:22 clarkb: ++ 19:07:36 is openstack/openstack fixed yet? 19:07:37 fungi and corvus are currently dealing with some fallout due to restarting zuul as part of that migration 19:07:52 corvus: no the chagne to fix it is in check right now 19:07:54 not sure it's migration related 19:08:01 or, well, it was in check :| 19:08:01 fungi: the restart was migration related 19:08:27 oh, the restart from the 16th 19:08:30 the openstack/openstack issue is the only critical regression i know about, hopefully that'll be fixed otday 19:08:42 yup we have a fix up for it and I've been trying to shepherd it through 19:08:55 yeah, the current restart is not necessarily migration related (seems unlikely but i wouldn't rule it out entirely just yet) 19:09:03 we have gotten the feedback that replicating refs/changes and refs/notes would be nice 19:09:05 we've also found there are a non zero number of repos that need to be renamed (either we failed to rename them or renamed them inappropriately) 19:09:44 that's a minor regression -- i think it's one we can live with but i think we've learned that's more important than we originally thought 19:10:02 airship/in-a-bottle should be airship/airship-in-a-bottle for example (I've asked them to fill out the rename request etc) 19:10:04 and of course, searching isn't really adequate to retire hound yet; that will require gitea improvements. 19:10:36 "airship/in-a-bottle" looks pretty cool to me :) 19:10:45 speaking of refs/changes - have we put up a change to switch gerrit to use internal gitweb links? 19:10:53 mordred: I am not aware of that change 19:10:54 airship/bring-your-own-bottle? ;) 19:10:59 ok. I'll make one of those 19:11:19 mordred: for what? 19:11:21 we also need someone to push the rename playbook fixes we had to write 19:11:33 (I volunteered to do that yseterday then got distracted by gitea git stack sizes) 19:11:36 corvus: the gerrit links to browse a given change don't wokr 19:11:42 mordred: gotcha thx 19:12:03 corvus: I think this makes replicating refs/changes slightly higher than "would be nice" - but also still there is a workaround so not *urgent* 19:12:15 we also left behind the following repos: openstack-dev/ci-sandbox openstack-dev/os-http openstack-dev/sandbox openstack-infra/beaker-localhost openstack-infra/beaker-nodepool openstack-infra/netlify-sandbox openstack-infra/opendev-website openstack-infra/puppet-docker_registry openstack-infra/puppet-odsreg openstack-infra/vinz-webclient openstack-infra/zone-opendev.org 19:12:17 openstack-infra/zone-zuul-ci.org 19:12:17 mordred: yeah 19:12:30 fungi: ++ 19:12:45 openstack-dev/ci-sandbox is a curious one 19:12:56 along with openstack-dev/sandbox 19:12:59 yeah 19:13:05 maybe both should be opendev/ ? 19:13:06 curious as in where do they belong? 19:13:20 and should zone-zuul-ci.org be opendev/ or zuul/ ? 19:13:22 i'm inclined to say opendev. they're places to test out interactions with our systems 19:13:25 ya I think they are generally applicable to the various users of our gerrit system 19:13:28 mordred: zuul i think 19:13:36 (I'm also inclined to say both in opendev/ for th sandboxes as well) 19:13:43 corvus: that's where my brain was leaning 19:13:59 in particular sandbox is used for teaching people how to gerrit 19:14:15 and ci-sandbox is for teachign ci's how to gerrit :) 19:14:33 opendev-website can be retired - although we should probably rename it to opendev/ then retire it 19:14:41 my only concern with putting the various zone repos somewhere other than opendev's namespace is the implication that they're safe for folks to update without risk of breaking our infrastructure 19:15:24 pretty sure the right change to zone-zuul-ci.org could bring down the opendev nameservers 19:15:36 yeah, if we feel like opendev should manage those, i'm game 19:15:41 ++ 19:15:59 fungi: we can have tests! 19:16:01 it's more like having consistent oversight to review and approve changes for them 19:16:03 anyway we don't have to figure out the correct mapping for everything right now. More wanted to go over known issues and things we need to fixup 19:16:10 clarkb: ++ 19:16:29 corvus: that's a very good point. i guess with functional testing we have now changes which break our nameservers will likely also break those job builds 19:16:51 Sounds like at this point the big items are git stack size fix for openstack/openstack in gitea, gerrit gitweb enablement for change browsing, and updating our rename playbook. Then start collecting data on the next round of renames to clean up after ourselves 19:17:04 so i'm maybe -0.5 on putting zone-zuul-ci.org in the zuul namespace 19:17:07 on the job front fungi's text replacement work did a really good job 19:17:22 we still had to fix jobs and scripts and stuff, but overall the transition was pretty easy 19:17:33 huzzah! a round of drinks in denver on fungi! 19:17:44 hah, sure, why not 19:17:54 s/on fungi/in fungi/ 19:18:03 i'm always getting that confused 19:18:07 there will definitely be plenty of the latter 19:18:27 on the rename playbook, should have it also start updating/correcting issues links in gitea 19:18:39 fungi: +1 19:18:49 and zuul keys? 19:19:03 do we have a hitlist of things we know are currently missing from it? 19:19:22 fungi: I think half the hitlist is diff what is on bridge against what is in system-config and go from there 19:19:34 yeah 19:19:37 (also our irc logs from friday are pretty good for this) 19:19:40 ahh, i forgot we forked it there temporarily 19:20:22 oh, at one point folks were also reporting stale search results from the gitea explore search which weren't taking the renames into account... did those eventually self-correct? anybody know? 19:21:08 searching for thingsl ike git-review and nodepool in the explore searcher shows up to date results 19:21:27 I don't know what the specific instances of this problem were off the top of my head though 19:21:57 same. i'm willing to bet gitea reindexes in the background and takes time to catch up 19:22:04 there's a list of tasks in the etherpad 19:22:24 should have the things we need to do to update the rename playbook in it 19:22:29 oh right we can cross check against that for the playbook 19:23:24 i've either closed or lost the tab for that pad in my endless sea of browser tabs... can we link it here for posterity? 19:23:31 worth saying again: thank you to everyone that helped with the migration and put time into making it happen and for putting up with the outage and transition necesary on the developer side too :) 19:23:39 #link https://etherpad.openstack.org/p/the-great-opendev-git-migration 19:23:42 fungi: ^ 19:23:46 thanks! 19:24:02 how could i ever forget such a great pad title anyway? 19:24:02 anything else or should we move on? 19:24:06 fungi: right? 19:24:22 i got nothin' 19:27:15 ok I think corvus may be distracted by zuul things right now. I'll keep moving the meeting along since it helps get info recorded and out there but understand if others are a bit distracted 19:27:22 #topic Config Management Updates 19:28:32 The biggest chunk of news here is that we believe we've addressed the docker image build's problem with ipv6 19:28:44 \o/ 19:29:05 the "fix" ended up being having socat proxy ipv4 to ipv6 so that docker can pretend that ipv6 doesn't exist 19:29:14 i'm bummed that skopeo is choosing to remain bug-compatible with docker on iths 19:29:18 (and uses /etc/hosts on the test nodes where we can edit /etc/hosts) 19:29:54 corvus: and also bummed that docker isn't landing our patch 19:30:07 mordred: there are 3000 open issues ahead of it 19:30:09 my patch is actually broken but I'm wondering if anyone will show up and tell me that 19:30:25 I haev to allow for : in the ipv6 addrs too 19:31:17 in any case we should now be able to reliably do image building and testing of those images regarless of cloud we run on 19:31:32 that's very exciting 19:32:10 Oh related to the opdnev migration we don't need to run our cgit farm anymore. We've stopped running ansible and puppet on those nodes and we should probably delete them this week 19:32:17 clarkb: ++ 19:32:20 and result of that is we can remove centos7 testing of our stuff 19:32:27 which means config mgmt things don't need to run there anymore 19:33:17 #topic Storyboard 19:33:32 diablo_rojo: fungi: as with last week I've not had much time to follow storyboard recently. 19:33:36 Anything to bring up to the group? 19:34:32 i don't know, i've been basically all opendev 19:34:37 migration 19:34:56 (and, for the record and those arriving late, we discussed some storyboard-related things earlier in the opendev meeting topic) 19:35:21 i still have an outstanding task to reproduce the renames from the maintenance on storyboard-dev so that puppet gets un-broken on it 19:35:52 fungi: another thing to add to the rename script (rename on storyboard-dev too) 19:36:35 yeah, either that or stop having manage-projects try to create projects and project-groups on stroyboard-dev 19:36:46 i could go either way on it 19:37:25 ok something to think about when updating the playbook I guess 19:37:33 #topic General Topics 19:38:14 letsencrypt progress. I don't think ianw is here today. So probably not a ton to talk about 19:38:40 That said there are certs that need renewing. git.airshipit.org, git.starlingx.io and survey.openstack.org 19:38:54 Previously we had said get longest cert term possible and let those ride out into the sunset. 19:39:00 and there will be some we want to renew for compatibility reasons for a while 19:39:13 that still seem like a reasonable plan given the difficulty to manage DNS on those omains? 19:39:30 yeah, we could do 2-year validity on them and declare that their sunset date 19:39:33 I think that means a 2 year cert for those names (which I'm still happy to do just wanted to double check now that we've got letsencrypt working) 19:40:02 if there are no objections doing ^ I will likely do that after the summit 19:40:13 it's less the difficulty of doing dns for them, and more the desire not to integrate our automation with proprietary dnsaas solutions 19:40:16 ianw_pto: ^ so when you are back from pto that might be something to weigh in on 19:40:45 Next up: Trusty upgrade progress. 19:40:51 Unsurprisingly basically no progress since the last meeting 19:40:57 This was expected 19:41:06 But I'd like us to finish this up real soon now if we can 19:41:13 fungi: we should follow up with the foundation on the groups servers 19:41:27 yes 19:42:12 And now PTG Planning 19:42:18 #link https://etherpad.openstack.org/2019-denver-ptg-infra-planning 19:42:39 as you can probably tell that has been fairly bare 19:42:57 now that the bulk of the opendev work is behind us I'd like to start filling in info there based on where we have ended up 19:43:33 But everyone elseshould feel free to add things too 19:43:59 this is likely to be a tomorrow task for me at the rate today has gone 19:44:03 yeah 19:44:46 Also I thought it would be fun to go back to lowry beer garden if weather and scheduling permits 19:45:21 that would indeed be fun and nostalgic 19:45:23 I can't keep track of when all the other events are happening 19:45:29 fungi: ^ do you happen to be more on top of that? 19:45:39 nope, not even a whit 19:46:05 ok I'll try to propose a day/evening for that trip by tomorrow so that peopel can incorporate it into their planning 19:46:51 clarkb: speaking of scheduling - the summit schedule doesn't seem to be udpating google calendars. I suppose I should probably report that to jimmy rather than you 19:47:01 mordred: or smarcet 19:47:23 things I do know about friday night is the ptg party 19:48:24 ok /me looked at stuff. Sunday night is a free night for official events (is TC or anyone meeting that night), Monday night is marketplace mixer, tuesday is the official part,y wednesday is recovery day, thursday is game night, friday is ptg party 19:48:35 so out of that list sunday, monday and wednesday are probably best 19:49:27 So ya I'll send out an email and we can pick a night :) 19:49:58 Anything else people want to know about prior to Denver or want to make sure we know about for denver? 19:50:02 monday is gertty's birthday 19:50:19 corvus: I'd be up for celebrating gertty's birthday at lowry beer garden :) 19:50:31 ++ it's the big "five" 19:51:08 long term forecast shows decent weather. On the cool side for outdoors depending on what your definition of warm is, but not looking like a lot of rain so this may just work out 19:52:15 #topic Open Discussion 19:52:48 I am likely to be afk for large portions of thursday and friday to ensure I get a bit of rest in before the marathon that starts saturday 19:53:39 i think some members of the openstack tc said something about folks non-exclusively getting together for food/drinks informally after the joint leadership meeting, but it was more like "we will probably be hungry/thirsty" 19:54:20 I arrive Saturday afternoon too if anyone wants to meetup and find food 19:55:32 i arrive some time on saturday. i helpfully forget when. 19:56:04 oh the A train (the train the openstack release is named after) runs frequently from the airport to about a mile from the convention center 19:56:27 depending on how you feel after flying that might be more exciting than a taxi :) 19:57:12 And sounds like that may be it. Thank you everyone! 19:57:23 #endmeeting