19:01:18 #startmeeting infra 19:01:18 Meeting started Tue Nov 27 19:01:18 2018 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:19 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:21 The meeting name has been set to 'infra' 19:01:31 o/ 19:01:33 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:01:41 #topic Announcements 19:01:42 Hi o/ 19:02:09 The only announcement I have is that openstack-discuss is replacing a bunch of openstack mailing lists that will be retired in 6 days. Please subscribe if you haven't already 19:02:17 fungi: ^ anything else to add to that? 19:03:47 probably worth mentioning here as well that I've picked up some sort of infant/toddler plague over the weekend so may not be 100% the next few days 19:04:47 #topic Actions from last meeting 19:04:54 #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-11-20-19.01.txt minutes from last meeting 19:05:22 it was a slow week due to the US holiday so not much there. However I did want to double check on the pypi cleanup 19:05:34 ianw: ^ any update on that or help that can be provided? 19:06:27 i got the reviews on the removal of the symlink from the mirrors yesterday, and just left it to make sure there wasn't any issues popping up 19:06:43 which i'm guessing no from scrollback :) so i'll clear it out probably today 19:07:20 great, then fallout from that is fedora 29 mirroring? 19:07:57 fedora29 we merged our client-side fix to ignore the upstream permissions 19:08:28 however i'd like some eyes on https://review.openstack.org/#/c/618416/ to remove f27 which will also free up space 19:08:44 #link https://review.openstack.org/#/c/618416/ cleanup f27 mirrors to free up afs disk space 19:09:08 out of an abundance of caution i depended that on https://review.openstack.org/#/c/614375/ to remove f27 from nodepool too 19:09:15 whoops! i got distracted replying to someone's e-mail, sorry. no nothing to add on the ml merger 19:09:58 #topic Specs approval 19:10:16 I don't think there are specs ready for approval but did want to call out a couple specs that could use your review/attention 19:10:25 #link https://review.openstack.org/#/c/607377/ Storyboard attachments spec 19:10:39 ianw: i +A'd 614375 19:11:24 #link https://review.openstack.org/#/c/581214/ Anomaly detection in CI jobs. This one is still marked as a draft but tristanC and dirk demoed their work in berlin and its really neat (and I think very useful) stuff 19:11:34 if you've got a momemnt to review those it would be very much appreciated 19:11:41 #topic Priority Efforts 19:11:51 #topic Storyboard 19:12:07 Other than the attachments spec is there other work or items that the infra team should be aware of here? 19:13:07 Looks like I have a few comments to address on the spec, but if anyone else wants to pile on before I update it, feel free! 19:13:23 o/ 19:13:41 At the summit we had talked about moving the db to a different node? 19:13:51 * diablo_rojo rakes brain for other things.. 19:14:27 the db is currently a trove instance right? are we talking about a new trove instance or a self managed database server? 19:15:10 yes 19:15:38 we were talking about possibly moving to a local mysql server on the sb instance 19:15:47 to reduce round-trip latency on queries 19:15:52 gotcha 19:16:01 though some sort of query caching might also bring similar performance benefits 19:16:17 well, having the server locally would make it easier to diagnose what's going on too 19:16:33 ultimately, i expect we just need someone with competence in the field to help perform some query profiling and help make them more efficient 19:17:00 #info Storyboard could use someone with mysql/database skillz to profile queries and help make them more efficient 19:17:13 yes plz 19:17:33 i'll set up a modest shrine in my hallway dedicated to their worship 19:18:18 I will put their picture on my wall next to my top monitor 19:18:30 anything else storyboard before we move to the next item? 19:18:30 did we ever get past the testing issue? 19:18:55 i tried to pitch in, but never got any help on this: https://review.openstack.org/553102 19:19:17 no, we were hoping to get some intern/mentee vigor on rewriting the test framework to be maintainable i think 19:19:22 I have not had time to dig in yet :/ 19:19:28 #link https://review.openstack.org/#/c/553102/ corvus attempt to improve testing of storyboard subscribers. Could use help 19:19:30 * diablo_rojo opens corvus's link 19:19:31 i think it would be a lot easier (read: "possible") to make database improvements to storyboard if the subsystem were testable. but i can't make heads or tails of it. :( 19:20:49 corvus, I will add it to my todolist. 19:20:56 I'll see if I can get some SotK eyes on it too 19:21:50 diablo_rojo: thanks 19:22:08 i feel pretty sure i would be able to contribute bite-sized improvements, but i don't think i have time for a subsystem overhaul :( 19:23:23 #topic Update Config Management 19:23:24 corvus, I can say I greatly appreciate anything and everything you do for StoryBoard no matter how big or small :) 19:23:43 (sorry for moving on but seems like we have a plan forward on last item and we have other agenda items to cover) 19:23:50 diablo_rojo: :) 19:24:18 clarkb, no worries 19:24:22 yeah, i don't mean to derail, just thought i'd mention it because it may facilitate new contributions in the db area 19:24:57 ++ 19:26:07 ok updating config management 19:26:19 I think the bulk of the work happening here now is with ianw's graphite docker chain 19:26:32 * mordred needs to read it 19:26:41 last week I had become concerned that we had an iptables problem, but mordred reminded me that we plan to not network namespace 19:27:24 does that affect https://review.openstack.org/605585 ? 19:27:49 corvus: it shouldn't the network namespace disablement is a container creation argument/option aiui 19:27:56 corvus: getting a docker in place should be the same either way 19:28:04 well - I think we might not want to disable the iptables tests though 19:28:10 right, but my -1 on that was mostly because ^ 19:28:23 mordred: corvus yes the -1 is still valid. I think we should assert that our rules end up in place as expected 19:28:28 ++ 19:28:49 i guess my question is, does the new thinking on namespaces mean that most of that iptables complexity goes away and we don't really need to change anything about it? 19:28:56 that way we can also verify that non-network-namepsaced containers work properly and the iptables stuff doesn't go to the bad place 19:29:01 corvus: yah. I think so 19:29:11 sort of, docker will still try to manage the iptables rules 19:29:18 the base set happen from the daemon starting 19:29:34 however it shouldn't create any container specific rules because we use the host namespace 19:29:46 (so our testing will have to accomodate that those rules may exist) 19:29:56 agree. we still want the tests to test that docker isn't breaking what we're doing with iptables 19:30:06 ok, so we still need some iptables changes, but we should be able to modify our tests to accomodate that 19:30:13 (rather than drop them) 19:30:26 yah 19:30:31 yes to our test accomodating that. I'm not sure we need to change anything in how we iptables? 19:30:37 well, we may need some iptables changes - we may not 19:30:41 what clarkb said 19:30:57 sorry, i guess i meant iptables-testing-related changes 19:31:04 not changes to our base iptables roles 19:31:05 we _shouldn't_ need to care about docker iptables - but should is dangerous :) 19:31:22 the important thing here is our tests can ensure that docker doesn't suddenly stop being nice to our rules 19:31:27 and if that changes we should notice via the testing 19:31:31 so ++ to testing it 19:31:35 ++ 19:32:24 as far as other work I think cmurphy has a stack at topic:puppet-4 that could use second reviewers (I have reviewed a bunch last week?) 19:32:58 and the other standout item for this topic is zuul doing CD. I think my changes to add a zuul cd user to bridge are still outstanding if we want to pick that back up again 19:34:21 I feel like there was something else blocking zuul cd ... but that was just the per-project key we landed ages ago perhaps? 19:35:02 mordred: ya I think that was the major blcoker and now its getting ssh into bridge working (whcih is what my changes will allow for) 19:36:15 sweet 19:36:30 mordred: please do go review ianw's stack though I think its gets some major chunks of the docker story into place for us 19:37:06 #topic General Topics 19:37:28 ianw had some good comments on how a weekly standing meeting has been useful for cross timezone coordination 19:38:00 i think this is a good use of time 19:38:10 I proposed a slight modification of my proposal from last week where instead of cancelling the meeting if there are no agenda updates we can cover the standing topics (our priority items mostly) 19:38:27 ++ 19:38:28 but retain the 24 hour agenda update so that people can selectively skip the meeting if that helps them get dinner or sleep 19:39:06 I think this gives us a good balance between making it easiser for people to prepare and possibly skip meetings while still having the sync up regularly 19:39:15 i agree 19:39:43 ++ 19:40:24 Going forward I'm going to try and be better about sending out a 24 hour pre notice with the agenda copied in 19:40:38 then its clear to anyone following the list that 1) there is a meeting happening and 2) this is what is going to occur 19:40:47 but maybe we'll find that is just noise. Willing to try it though 19:41:09 that sounds good; i think that as i mentioned a bit more structure with dates in the wiki page might help too 19:41:40 ianw: ++ and I think it was you that suggested cleaning up the agenda as part of the meeting? I worry we won't have time for that but I can take it as a chair item to use the 10 minutes after a meeting to do that 19:42:37 either or, it can just get a bit confusing as to what's been discussed and what's upcoming on the wiki 19:42:53 that is a good point 19:43:27 next is a holdover from last week but I kept it because still relevant today. We have good news on the opendev.org DNS situation. Our dns servers are now the dns servers for this domain 19:43:44 We are waiting on them to create the DS record so that dnssec is happy but finally jamesmcarthur got through to the registrar :) 19:44:06 never make jamesmcarthur mad 19:44:14 lol 19:44:16 corvus: ^ assuming dnssec is working in the next day as we've been told is the next step there to migrate zuul-ci.org to the new servers? 19:44:24 you won't like him when he's angry! ;) 19:44:45 it takes a lot, but eventually i get pushed over the edge 19:44:49 also, he needs to stop ripping his shirts 19:45:10 clarkb: maybe set up a static website first, exercise it hosting opendev.org a bit, then move zuul-ci.org over i think 19:45:24 that seems reasonable. And after that clean up the old dns servers. 19:45:28 yep 19:45:39 and from there we can start tackling the actual migrationy bits of this process 19:45:51 yes, i think the opendev-website repo is likely the next one on the docket after zone-opendev.org 19:46:14 #info opendev-website next after zone-opendev.org and associated DNS registrar situation is sorted 19:47:14 #topic Open Discussion 19:47:38 We've got a bit of time (not surprising given the very short week last week for some of us). Feel free to bring up other topics. 19:47:46 since we were talking about fedora29, there's a surprisingly large stack of stuff related in https://review.openstack.org/#/q/topic:fedora29+(status:open+OR+status:merged) 19:48:01 One I've got is should we remove the open discussion item from the agenda? so that we don't sneak things into the agenda after someone has decided it is safe to sleep in? 19:48:04 mostly related to getting glean plugged in with networkmanager on redhat rpm platforms 19:48:23 reviews welcome, but ci looking good 19:48:52 i'm good with dropping open discussion. that's basically 95% of the #openstack-infra channel's purpose anyway 19:48:53 clarkb: IMO discussing things when there is time is fine, but avoid decision making? 19:49:50 ++, it's usually mostly a ad-hoc call for reviews? (e.g. mine :) 19:50:18 frickler: works for me 19:50:43 i wouldn't mind seeing topical entries for the "standing" items on the agenda too, but wouldn't want to make a lot of extra work for the chair 19:52:06 ianw: the comments at https://review.openstack.org/#/c/618964/7..11/glean/init/glean-nm%2540.service have me slightly confused. My local xenial install doesn't have an /etc/sysconfig/network 19:52:34 that said glean-nm should only be used by red hatty things in the current setup so this is fine I think 19:53:40 fungi: I think if the chair or others have time to add topical items to the standing items that would be much appreciated 19:53:57 but I also don't think it is necessary particularly since that is our sanity check catch up period 19:54:11 clarkb: /etc/sysconfig/network/ifcfg-* is what glean will write out on Xenial systems 19:55:09 ianw: I think that is for suse? 19:55:31 and debuntu writes to /etc/networking/interfaces 19:56:04 oh, hrm, yes maybe. what's the xenial path then? 19:56:24 it's /etc/network/interfaces(.d/something) 19:56:56 ya the per interface stuff goes into interfaces.d/something 19:56:56 yeah, sorry i mean do we not have a xenial service file? 19:57:37 ifupdown? 19:58:04 in the install, we only do "if (os.path.exists('/usr/lib/systemd/system')" then install the service ... 19:58:07 ahh, networking 19:58:28 ianw: I think that condition is broken on xenial so glean will run each time? 19:58:31 is /etc/init/networking.conf what you mean by "service file"? 19:59:00 I think the systemd unit file that glean installs 19:59:19 clarkb: hrm, yes maybe xenial is falling through ... let's follow up in infra ... 19:59:31 ianw: see you there 19:59:33 and with that we are out of time 19:59:39 thank you everyone 19:59:39 thanks clarkb! 19:59:43 #endmeeting