19:01:11 #startmeeting infra 19:01:11 Meeting started Tue Apr 2 19:01:11 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:14 The meeting name has been set to 'infra' 19:01:23 #link http://lists.openstack.org/pipermail/openstack-infra/2019-April/006299.html 19:01:33 \o 19:01:35 #topic Announcements 19:01:56 I don't have any announcements. Anyone else have announcements? 19:03:04 #topic Actions from last meeting 19:03:07 clarkb: i think you just made an announcement :) 19:03:29 #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-03-26-19.01.txt minutes from last meeting 19:03:41 Thank you ianw for running the meeting last week 19:03:56 frickler: had an action to look into disk monitoring of nodepool builders 19:03:59 frickler: ^ any update on that? 19:04:25 https://review.openstack.org/648365 19:04:32 #link http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66463&rra_id=all 19:04:38 that seems to be solved 19:05:05 should work for optional volumes on all of our servers 19:05:05 ++ thanks! 19:05:16 cool 19:05:33 and I know shrews has a change up to fix the underlying issue with the builders 19:05:43 we should be well covered on that item going forward 19:06:21 the usage looked pretty stable in the graphs 19:06:42 yep: https://review.openstack.org/647599 and parent 19:08:00 thank you for everonye that has helped make that better 19:08:08 #topic Priority Efforts 19:08:23 I'm skipping specs approvals as I don't think we have any to look at and we have a fairly full meeting otherwise 19:08:31 #topic Update Config Management 19:08:46 Before I took off on vacation I got a bunch more hosts upgraded to puppet-4 19:08:53 thank you to cmurphy for continuing to help and push on that 19:09:03 o7 19:09:06 hear hear 19:09:35 There are still more hosts to be updated under topic:puppet-4 if you have time to review those. I'm happy to approve as I can watch them and fix issues that arise 19:11:00 Anything new on the docker side of things? 19:11:25 Seems like we've got a pretty solid foundation for both and now its mostly a matter of pushing the conversions? Not surprising those are moving slowly as we also tackle opendev things 19:11:56 yeah, nothing new from me on that, and probably not until summit... 19:12:05 but i'm looking forward to working on that at ptg 19:12:42 Considering ^ lets talk OpenDev 19:12:46 #topic OpenDev 19:13:20 How are we looking for the transition on the 19th 19:13:55 are we down to building a script/tool/playbook to do the actual transition and collecting lists of renames? 19:14:15 #link https://storyboard.openstack.org/#!/story/2004627 19:14:42 fungi: how's the redirect stuff coming? 19:15:03 (though i haven't seen fungi in this meeting yet) 19:15:32 i started playing around with the hard-coded list of repositories idea for the non-openstack.org sites and it seems to work, i'll get something up others can test later today i hope 19:15:46 * fungi has been lurking and hacking on changes at the same time 19:16:25 the rabbit hole on the script to do the repository changes is continuing to deepen though 19:16:41 another scenario came to mind which i don't know if we need to be concerned with 19:17:18 as we're also doing project renames, i expect rather a lot of namespace changes to embedded checkout file paths in zuul job definitions 19:17:55 should this task also grow to attempt altering more than just the domain name in those? 19:18:21 i'm inclined to say 'no' because the potential for error is high 19:18:26 and is there anywhere else we should be similarly concerned about namespace changes within repo content? 19:19:20 we can make sure that devstack, grenade, etc are fixed up quickly after the switch; that should solve a lot of problems 19:19:56 fungi: is that different to what we mentioned a few meetings ago, things like the requirements files checkouts? 19:20:05 ya and the unittest type jobs should use the implicit $thisproject checkout 19:20:22 ianw: yes, insofar as it's the namespaces, not just the domains 19:20:27 but in general, trying to figure out if 'openstack/nova' is an active string constant reference or just some text is a lot of work 19:21:40 so for example if a job defniition refers to git.openstack.org/openstack-infra/zuul-jobs the difference between rewriting that to opendev.org/openstack-infra/zuul-jobs vs opendev.org/zuul/zuul-jobs 19:22:07 the former will be broken and need replacing with the latter after the maintenance 19:22:10 fungi: ahh, so you're meaning (as corvus above says) places where 'openstack/nova' is found, but without a leading "openstack.org"? 19:22:28 also possible yes, without the domain present at all 19:22:51 we could probably safely include 'git.openstack.org/openstack/foo' in the translation... 19:23:18 yes, however that means incorporating the list of namespace changes for all repositories into the routine if we do 19:23:28 which is why i raise it as a point of additional complexity 19:23:30 oh, i thought we were going to do that full rewrite in the jobs 19:23:42 we need that in order to write .gitreview anyway 19:24:36 so, yeah, i think regardless the script that generate our force-merge patches should perform the full rewrite (domain + org) 19:24:39 er, that wasn't what i was expecting, no. i thought this script was replacing review.openstack.org with review.opendev.org in .gitreview files and then the normal project renaming playbook was taking care of the namespace portion 19:25:04 since it would do that anyway 19:25:10 the normal project renaming playbook makes .gitreview changes? 19:25:42 oh, you're right, it doesn't. just moves files on disk and alters database rows 19:25:51 okay, i'll plan to include that too 19:25:57 kk 19:26:34 side question: for the openstack namespaces, do you folks want those changes in the ethercalc sheet, or would something computer-readable (and writeable!)be easier? 19:26:35 so if the hostname portion is present in a string in playbooks/roles we can assume we should also alter the namespace appearing within that string 19:27:02 #info Rename script should modify .gitreview and zuul job content to update hostnames and repo names 19:27:15 fungi: ya as the other hostname won't work (particularly in the case of gitreview and zuul jobs) 19:27:40 jroll: I believe we can export the ethercalc to csv which should be simple enough to then convert to something more structured if necessary 19:27:43 jroll: the ethercalc is more for one-off/small groups of namespace changes. for large adjustments based on structured data it's probably easier to just let us know what you want done and we can generate that set ourselves 19:27:52 jroll: i yield to fungi on that since it's likely to be input to his script, but i see no reason to use the spreadsheet if you want to arrange something else 19:28:08 fungi: corvus: excellent, thanks :) 19:28:17 fungi: I'll sync with you once our governance change merges 19:28:54 and yeah, i expect we'll just export ethercalc to csv and translate/merge with the other data to input to the script 19:28:59 thanks. less likely to end up with typos if i script up the list of openstack->whatever-catch-all namespace edits based on the governance projects.yaml 19:29:38 yep 19:29:53 given it's probably something like 1k renames 19:31:16 Alright anything else on the opendev subject? 19:31:35 just for reference 19:31:37 #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-03-26-19.01.log.html#l-81 19:31:44 was the last meeting where we talked about that rename ^ too 19:32:30 #topic Storyboard 19:32:46 fungi diablo_rojo_phon SotK how are storyboard things? 19:33:09 trove just relocated to storyboard.o.o 19:33:37 (roughly two months after we stopped using trove for storyboard.o.o, the irony) 19:34:32 i've been sort of buried under other things for the past week so not really sure what else is going on... i think the outreachy flood may have died down now that the deadline has passed 19:36:37 Ok sounds like we should moev on 19:36:41 yup 19:36:46 #topic General Topics 19:36:57 i think diablo_rojo_phon is sucked into conference travel anyway 19:36:57 First up: PTG Planning 19:37:03 #link https://www.openstack.org/ptg#tab_schedule Schedule has us Thursday and Friday 19:37:21 It is my understanding that the scheduel above is the "final" scheduel and shows us thursday and friday 19:37:32 corvus: that means you don't have an idle day and we avoid the TC conflict for fungi 19:37:42 #link https://etherpad.openstack.org/2019-denver-ptg-infra-planning Agenda brainstorming 19:38:00 As we start to really dig into the opendev stuff feel free to add items ^ there for things we should followup on after the transition 19:39:42 next up is Letsencrypt progress 19:39:48 #link https://review.openstack.org/#/c/636759/ Ready for review 19:40:13 ianw says ^ is ready for review. That will end up being a big piece of transitioning further services over to opendev so reviews much appreciated 19:40:15 yes please, and a few minor changes below that 19:41:16 Next is trusty server upgrade status 19:41:22 #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup 19:41:28 #link https://etherpad.openstack.org/p/lists.o.o-trusty-to-xenial lists.openstack.org in place upgrade notes 19:41:38 I've been picking up the work to test an in place upgrade for lists.openstack.org 19:41:58 The do-release-upgrade takes about 45 minutes which should be plenty quick for people doing smtp 19:42:18 The new xenial server seems to not break our vhosting of mailman 2 19:42:30 I've created a test mailing list and sent mail through it successfully 19:43:13 The one big issue we've found is nothing I try results in mailman preserving original body content and instead it gets base64 encoded because the default is to use utf8 and python email lib converts utf8 email to base64 19:43:30 This is only really an issue if considered in the context of dkim/dmarc 19:43:50 because email won't verify if mailman is converting body content to base64 and the dmarc policy checks the body 19:44:27 it's also.. well.. weird 19:44:38 I have confirmed that overriding /usr/lib/mailman/Mailman/Defaults.py to force the en language to ascii instead of utf8 fixes this behavior 19:44:42 i mean, even aside from dmarc, it makes messages larger than necessary 19:45:59 whether that should be a blocker for us... ? 19:46:26 ya I'm sort of running out of ideas at this point short of actually learning how the deep internals of mailman work 19:46:53 we could force the list back to ascii which has downsides too 19:47:31 other options could be to use mailman's new email munging features in the mailman on xenial to deal with dmarc though we've avoided those previously 19:48:22 At this point I think it would be good if someone else looked at it without my direct help (just to make sure I've not missed anything with pebkac) 19:48:49 yeah, we at least got a glimpse of how those work in reality since kata-dev turned it on to deal with dmarc-signed messages (particularly from microsoft employees, if memory serves) 19:49:24 and if we don't turn up any good fixes after a second person looks at it then we should probably consider alternatives to dmarc handling 19:49:48 Let me know if you can help or have ideas 19:49:53 i think deciding on an answer about whether the b64 encoding is expected and normal and okay regardless of dmarc is something we should do 19:50:02 i think the method they enabled is the one which rewrites the from address on messages from domains which claim to require dkim enforcement, and wipes the dkim signatures from the headers 19:50:25 perhaps by spinning up a new MM instance (without any of our config mgmt) and examining its behavior 19:50:38 corvus: the impression I got from mailman and python was that this is at least normal and expected from their side 19:50:40 but yes, i agree, the base64 reencoding is also a concern on its own, dkim aside 19:51:16 note you'll need to use ubuntu for that as they patch in the default to utf8 19:51:29 so I guess mailman as an upstream still avoids this when lang is set to en 19:51:35 clarkb: well, i mean... we don't know what changed? 19:52:02 corvus: correct I still haven't narrowed down why openstack-discuss for example doesn't do this 19:52:21 spinning up an unmodified mailman seems like a good next step 19:52:23 to confirm the behavior 19:52:40 i do have a mailman3 instance up and running which wasn't built with our configuration management, but it's not mailman2 obviously and also hackily upgraded to bionic rather than xenial, so likely not useful for this exercise 19:53:10 fungi: maybe you can help with the mailman2 equivalent of that? I'm assuming it isn't as simple as apt-get install mailman on xenial? or maybe it is? 19:53:21 (I can probably drive that, will just have lots of questions) 19:54:09 it basically is as simple as that, yes 19:54:17 #link https://etherpad.openstack.org/p/mm3poc 19:54:19 cool I'l take a look at that after lunch then 19:54:30 that's mostly what i did for mailman 3 (used distro packages) 19:55:04 it's also basically what our puppet-mailman module does too, it just also happens to do a lot of configuring 19:55:04 corvus: on figuring out what changed the two versions (ignoring ubuntu patches) are 2.1.16 on trusty and 2.1.20 on xenial 19:55:37 I did bzr clone and diff between those versions which produces a fairly large diff but maybe I should do moer than a quick skim 19:55:48 also do a review of the ubuntu pathcset I suppose 19:56:06 likely to not be a fast process unless I happen to get lucky 19:56:22 in any case that gives us two things to try 19:56:27 #topic Open Discussion 19:57:13 Wanted to quickly note that the i18n team seems to think we don't need to take urgent action re zanata. Would be good if we could talk to fedora about their plans though 19:57:29 anyone know current fedora people? (Maybe robyn can point us in the right direction) 19:57:45 i don't but i can see what ican find out 19:58:15 maybe give me an action item so i don't forget :) 19:58:47 #action ianw check in with fedora on post zanata plans 19:59:04 it would be swell to find out if fedora plans to continue translating things, and if so, how 19:59:25 And we are at time. Thank you everyone 19:59:40 thanks clarkb! 19:59:43 #endmeeting