19:01:18 <clarkb> #startmeeting infra
19:01:18 <openstack> Meeting started Tue Nov 27 19:01:18 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:19 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:21 <openstack> The meeting name has been set to 'infra'
19:01:31 <rfolco> o/
19:01:33 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:01:41 <clarkb> #topic Announcements
19:01:42 <gary_perkins> Hi o/
19:02:09 <clarkb> The only announcement I have is that openstack-discuss is replacing a bunch of openstack mailing lists that will be retired in 6 days. Please subscribe if you haven't already
19:02:17 <clarkb> fungi: ^ anything else to add to that?
19:03:47 <clarkb> probably worth mentioning here as well that I've picked up some sort of infant/toddler plague over the weekend so may not be 100% the next few days
19:04:47 <clarkb> #topic Actions from last meeting
19:04:54 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-11-20-19.01.txt minutes from last meeting
19:05:22 <clarkb> it was a slow week due to the US holiday so not much there. However I did want to double check on the pypi cleanup
19:05:34 <clarkb> ianw: ^ any update on that or help that can be provided?
19:06:27 <ianw> i got the reviews on the removal of the symlink from the mirrors yesterday, and just left it to make sure there wasn't any issues popping up
19:06:43 <ianw> which i'm guessing no from scrollback :)  so i'll clear it out probably today
19:07:20 <clarkb> great, then fallout from that is fedora 29 mirroring?
19:07:57 <ianw> fedora29 we merged our client-side fix to ignore the upstream permissions
19:08:28 <ianw> however i'd like some eyes on https://review.openstack.org/#/c/618416/ to remove f27 which will also free up space
19:08:44 <clarkb> #link https://review.openstack.org/#/c/618416/ cleanup f27 mirrors to free up afs disk space
19:09:08 <ianw> out of an abundance of caution i depended that on https://review.openstack.org/#/c/614375/ to remove f27 from nodepool too
19:09:15 <fungi> whoops! i got distracted replying to someone's e-mail, sorry. no nothing to add on the ml merger
19:09:58 <clarkb> #topic Specs approval
19:10:16 <clarkb> I don't think there are specs ready for approval but did want to call out a couple specs that could use your review/attention
19:10:25 <clarkb> #link https://review.openstack.org/#/c/607377/ Storyboard attachments spec
19:10:39 <Shrews> ianw: i +A'd 614375
19:11:24 <clarkb> #link https://review.openstack.org/#/c/581214/ Anomaly detection in CI jobs. This one is still marked as a draft but tristanC and dirk demoed their work in berlin and its really neat (and I think very useful) stuff
19:11:34 <clarkb> if you've got a momemnt to review those it would be very much appreciated
19:11:41 <clarkb> #topic Priority Efforts
19:11:51 <clarkb> #topic Storyboard
19:12:07 <clarkb> Other than the attachments spec is there other work or items that the infra team should be aware of here?
19:13:07 <diablo_rojo> Looks like I have a few comments to address on the spec, but if anyone else wants to pile on before I update it, feel free!
19:13:23 <mordred> o/
19:13:41 <diablo_rojo> At the summit we had talked about moving the db to a different node?
19:13:51 * diablo_rojo rakes brain for other things..
19:14:27 <clarkb> the db is currently a trove instance right? are we talking about a new trove instance or a self managed database server?
19:15:10 <fungi> yes
19:15:38 <fungi> we were talking about possibly moving to a local mysql server on the sb instance
19:15:47 <fungi> to reduce round-trip latency on queries
19:15:52 <clarkb> gotcha
19:16:01 <fungi> though some sort of query caching might also bring similar performance benefits
19:16:17 <mordred> well, having the server locally would make it easier to diagnose what's going on too
19:16:33 <fungi> ultimately, i expect we just need someone with competence in the field to help perform some query profiling and help make them more efficient
19:17:00 <clarkb> #info Storyboard could use someone with mysql/database skillz to profile queries and help make them more efficient
19:17:13 <diablo_rojo> yes plz
19:17:33 <fungi> i'll set up a modest shrine in my hallway dedicated to their worship
19:18:18 <diablo_rojo> I will put their picture on my wall next to my top monitor
19:18:30 <clarkb> anything else storyboard before we move to the next item?
19:18:30 <corvus> did we ever get past the testing issue?
19:18:55 <corvus> i tried to pitch in, but never got any help on this: https://review.openstack.org/553102
19:19:17 <fungi> no, we were hoping to get some intern/mentee vigor on rewriting the test framework to be maintainable i think
19:19:22 <diablo_rojo> I have not had time to dig in yet :/
19:19:28 <clarkb> #link https://review.openstack.org/#/c/553102/ corvus attempt to improve testing of storyboard subscribers. Could use help
19:19:30 * diablo_rojo opens corvus's link
19:19:31 <corvus> i think it would be a lot easier (read: "possible") to make database improvements to storyboard if the subsystem were testable.  but i can't make heads or tails of it.  :(
19:20:49 <diablo_rojo> corvus, I will add it to my todolist.
19:20:56 <diablo_rojo> I'll see if I can get some SotK eyes on it too
19:21:50 <clarkb> diablo_rojo: thanks
19:22:08 <corvus> i feel pretty sure i would be able to contribute bite-sized improvements, but i don't think i have time for a subsystem overhaul :(
19:23:23 <clarkb> #topic Update Config Management
19:23:24 <diablo_rojo> corvus, I can say I greatly appreciate anything and everything you do for StoryBoard no matter how big or small :)
19:23:43 <clarkb> (sorry for moving on but seems like we have a plan forward on last item and we have other agenda items to cover)
19:23:50 <corvus> diablo_rojo: :)
19:24:18 <diablo_rojo> clarkb, no worries
19:24:22 <corvus> yeah, i don't mean to derail, just thought i'd mention it because it may facilitate new contributions in the db area
19:24:57 <clarkb> ++
19:26:07 <clarkb> ok updating config management
19:26:19 <clarkb> I think the bulk of the work happening here now is with ianw's graphite docker chain
19:26:32 * mordred needs to read it
19:26:41 <clarkb> last week I had become concerned that we had an iptables problem, but mordred reminded me that we plan to not network namespace
19:27:24 <corvus> does that affect https://review.openstack.org/605585 ?
19:27:49 <clarkb> corvus: it shouldn't the network namespace disablement is a container creation argument/option aiui
19:27:56 <clarkb> corvus: getting a docker in place should be the same either way
19:28:04 <mordred> well - I think we might not want to disable the iptables tests though
19:28:10 <corvus> right, but my -1 on that was mostly because ^
19:28:23 <clarkb> mordred: corvus yes the -1 is still valid. I think we should assert that our rules end up in place as expected
19:28:28 <mordred> ++
19:28:49 <corvus> i guess my question is, does the new thinking on namespaces mean that most of that iptables complexity goes away and we don't really need to change anything about it?
19:28:56 <mordred> that way we can also verify that non-network-namepsaced containers work properly and the iptables stuff doesn't go to the bad place
19:29:01 <mordred> corvus: yah. I think so
19:29:11 <clarkb> sort of, docker will still try to manage the iptables rules
19:29:18 <clarkb> the base set happen from the daemon starting
19:29:34 <clarkb> however it shouldn't create any container specific rules because we use the host namespace
19:29:46 <clarkb> (so our testing will have to accomodate that those rules may exist)
19:29:56 <mordred> agree. we still want the tests to test that docker isn't breaking what we're doing with iptables
19:30:06 <corvus> ok, so we still need some iptables changes, but we should be able to modify our tests to accomodate that
19:30:13 <corvus> (rather than drop them)
19:30:26 <mordred> yah
19:30:31 <clarkb> yes to our test accomodating that. I'm not sure we need to change anything in how we iptables?
19:30:37 <mordred> well, we may need some iptables changes - we may not
19:30:41 <mordred> what clarkb said
19:30:57 <corvus> sorry, i guess i meant iptables-testing-related changes
19:31:04 <corvus> not changes to our base iptables roles
19:31:05 <mordred> we _shouldn't_ need to care about docker iptables - but should is dangerous :)
19:31:22 <clarkb> the important thing here is our tests can ensure that docker doesn't suddenly stop being nice to our rules
19:31:27 <clarkb> and if that changes we should notice via the testing
19:31:31 <clarkb> so ++ to testing it
19:31:35 <mordred> ++
19:32:24 <clarkb> as far as other work I think cmurphy has a stack at topic:puppet-4 that could use second reviewers (I have reviewed a bunch last week?)
19:32:58 <clarkb> and the other standout item for this topic is zuul doing CD. I think my changes to add a zuul cd user to bridge are still outstanding if we want to pick that back up again
19:34:21 <mordred> I feel like there was something else blocking zuul cd ... but that was just the per-project key we landed ages ago perhaps?
19:35:02 <clarkb> mordred: ya I think that was the major blcoker and now its getting ssh into bridge working (whcih is what my changes will allow for)
19:36:15 <mordred> sweet
19:36:30 <clarkb> mordred: please do go review ianw's stack though I think its gets some major chunks of the docker story into place for us
19:37:06 <clarkb> #topic General Topics
19:37:28 <clarkb> ianw had some good comments on how a weekly standing meeting has been useful for cross timezone coordination
19:38:00 <corvus> i think this is a good use of time
19:38:10 <clarkb> I proposed a slight modification of my proposal from last week where instead of cancelling the meeting if there are no agenda updates we can cover the standing topics (our priority items mostly)
19:38:27 <ianw> ++
19:38:28 <clarkb> but retain the 24 hour agenda update so that people can selectively skip the meeting if that helps them get dinner or sleep
19:39:06 <clarkb> I think this gives us a good balance between making it easiser for people to prepare and possibly skip meetings while still having the sync up regularly
19:39:15 <fungi> i agree
19:39:43 <mordred> ++
19:40:24 <clarkb> Going forward I'm going to try and be better about sending out a 24 hour pre notice with the agenda copied in
19:40:38 <clarkb> then its clear to anyone following the list that 1) there is a meeting happening and 2) this is what is going to occur
19:40:47 <clarkb> but maybe we'll find that is just noise. Willing to try it though
19:41:09 <ianw> that sounds good; i think that as i mentioned a bit more structure with dates in the wiki page might help too
19:41:40 <clarkb> ianw: ++ and I think it was you that suggested cleaning up the agenda as part of the meeting? I worry we won't have time for that but I can take it as a chair item to use the 10 minutes after a meeting to do that
19:42:37 <ianw> either or, it can just get a bit confusing as to what's been discussed and what's upcoming on the wiki
19:42:53 <clarkb> that is a good point
19:43:27 <clarkb> next is a holdover from last week but I kept it because still relevant today. We have good news on the opendev.org DNS situation. Our dns servers are now the dns servers for this domain
19:43:44 <clarkb> We are waiting on them to create the DS record so that dnssec is happy but finally jamesmcarthur got through to the registrar :)
19:44:06 <corvus> never make jamesmcarthur mad
19:44:14 <jamesmcarthur> lol
19:44:16 <clarkb> corvus: ^ assuming dnssec is working in the next day as we've been told is the next step there to migrate zuul-ci.org to the new servers?
19:44:24 <fungi> you won't like him when he's angry! ;)
19:44:45 <jamesmcarthur> it takes a lot, but eventually i get pushed over the edge
19:44:49 <fungi> also, he needs to stop ripping his shirts
19:45:10 <corvus> clarkb: maybe set up a static website first, exercise it hosting opendev.org a bit, then move zuul-ci.org over i think
19:45:24 <clarkb> that seems reasonable. And after that clean up the old dns servers.
19:45:28 <corvus> yep
19:45:39 <clarkb> and from there we can start tackling the actual migrationy bits of this process
19:45:51 <fungi> yes, i think the opendev-website repo is likely the next one on the docket after zone-opendev.org
19:46:14 <clarkb> #info opendev-website next after zone-opendev.org and associated DNS registrar situation is sorted
19:47:14 <clarkb> #topic Open Discussion
19:47:38 <clarkb> We've got a bit of time (not surprising given the very short week last week for some of us). Feel free to bring up other topics.
19:47:46 <ianw> since we were talking about fedora29, there's a surprisingly large stack of stuff related in https://review.openstack.org/#/q/topic:fedora29+(status:open+OR+status:merged)
19:48:01 <clarkb> One I've got is should we remove the open discussion item from the agenda? so that we don't sneak things into the agenda after someone has decided it is safe to sleep in?
19:48:04 <ianw> mostly related to getting glean plugged in with networkmanager on redhat rpm platforms
19:48:23 <ianw> reviews welcome, but ci looking good
19:48:52 <fungi> i'm good with dropping open discussion. that's basically 95% of the #openstack-infra channel's purpose anyway
19:48:53 <frickler> clarkb: IMO discussing things when there is time is fine, but avoid decision making?
19:49:50 <ianw> ++, it's usually mostly a ad-hoc call for reviews?  (e.g. mine :)
19:50:18 <clarkb> frickler: works for me
19:50:43 <fungi> i wouldn't mind seeing topical entries for the "standing" items on the agenda too, but wouldn't want to make a lot of extra work for the chair
19:52:06 <clarkb> ianw: the comments at https://review.openstack.org/#/c/618964/7..11/glean/init/glean-nm%2540.service have me slightly confused. My local xenial install doesn't have an /etc/sysconfig/network
19:52:34 <clarkb> that said glean-nm should only be used by red hatty things in the current setup so this is fine I think
19:53:40 <clarkb> fungi: I think if the chair or others have time to add topical items to the standing items that would be much appreciated
19:53:57 <clarkb> but I also don't think it is necessary particularly since that is our sanity check catch up period
19:54:11 <ianw> clarkb: /etc/sysconfig/network/ifcfg-* is what glean will write out on Xenial systems
19:55:09 <clarkb> ianw: I think that is for suse?
19:55:31 <clarkb> and debuntu writes to /etc/networking/interfaces
19:56:04 <ianw> oh, hrm, yes maybe.  what's the xenial path then?
19:56:24 <fungi> it's /etc/network/interfaces(.d/something)
19:56:56 <clarkb> ya the per interface stuff goes into interfaces.d/something
19:56:56 <ianw> yeah, sorry i mean do we not have a xenial service file?
19:57:37 <fungi> ifupdown?
19:58:04 <ianw> in the install, we only do "if (os.path.exists('/usr/lib/systemd/system')" then install the service ...
19:58:07 <fungi> ahh, networking
19:58:28 <clarkb> ianw: I think that condition is broken on xenial so glean will run each time?
19:58:31 <fungi> is /etc/init/networking.conf what you mean by "service file"?
19:59:00 <clarkb> I think the systemd unit file that glean installs
19:59:19 <ianw> clarkb: hrm, yes maybe xenial is falling through ... let's follow up in infra ...
19:59:31 <clarkb> ianw: see you there
19:59:33 <clarkb> and with that we are out of time
19:59:39 <clarkb> thank you everyone
19:59:39 <fungi> thanks clarkb!
19:59:43 <clarkb> #endmeeting