19:01:07 <clarkb> #startmeeting infra
19:01:08 <openstack> Meeting started Tue Oct 16 19:01:07 2018 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:11 <openstack> The meeting name has been set to 'infra'
19:01:17 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:01:39 <clarkb> I cleaned up the meeting agenda based on what was discussed during last meeting and what we have going on this week.
19:01:45 <clarkb> #topic Announcements
19:02:06 <clarkb> I'm likely to be afk tomorrow enjoying some of the last "good" weather before winter invades. My brothers demand I go fishing with them
19:02:16 <clarkb> fungi is also afk dealing with things like sea grass aiui
19:02:33 <corvus> fungi: can you see a lot of sea grass from the sea shore?
19:02:35 <fungi> most of the sea grass is back outside again where it belongs
19:02:55 <clarkb> #link https://www.openstack.org/summit/berlin-2018/summit-schedule/#track=262 Berlin Forum schedule up for feedback
19:03:15 <clarkb> As we get closer to the summit and forum the schedule is up and probably worth looking over for any hard to manage conflicts
19:03:43 <ssbarnea_> i am gere
19:04:12 <clarkb> #topic Actions from last meeting
19:04:23 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-10-09-19.00.txt minutes from last meeting
19:05:01 <clarkb> we don't appear to have recorded any formal actions, but ssbarnea_ had some items to talk about. I wanted to follow up and ask if we need to talk about any more of those today?
19:05:15 <clarkb> ssbarnea_: ^ or is there any other input you need to make some of those actionable?
19:05:36 <fungi> worth discussing the suggestion of git-review prereleases if it's not been added to the agenda yet
19:05:46 <ssbarnea_> clarkb: nope, not for now.
19:06:01 <fungi> or can bring it up during open discussion at the end
19:06:05 <clarkb> fungi: and adding new cores. We should have time to go through that during the end of the meeting
19:06:10 <fungi> awesome
19:06:20 <clarkb> #topic Specs approval
19:06:34 <clarkb> I approved the third party ci direction spec last week and it merged
19:06:48 <clarkb> we can now point to that when interested parties want to get involved running third party ci setups
19:07:29 <clarkb> the other spec people should be aware of (but is still WIP) is the storyboard attachments spec
19:07:54 <clarkb> #link https://review.openstack.org/#/c/607377/ Storyboard Attachments Spec. Would be good toget input on that from an operational perspective. Concerns like overal disk usage and not wanting to host spam etc
19:08:57 <clarkb> #topic Priority Efforts
19:09:06 <clarkb> #topic Update Config Management
19:09:25 <clarkb> #info topic:puppet-4 and topic:update-cfg-mgmt to find changes to review
19:09:39 <chandankumar> \o/
19:09:48 <clarkb> I've got all but one of the outstanding futureparser changes merged at this point. Thanks to cmurphy for continuing to push on that
19:10:06 <clarkb> There are other outstanding puppet-4 related changes to make it possible to futureparser more things
19:10:49 <clarkb> mordred: has started looking at deploying gerrit with containers and how a gerrit 2.14/2.15 might fit into that
19:10:59 <clarkb> ianw: is looking at graphite deployment from containers too aiui
19:11:17 <clarkb> For the continuous deployment work I think we are still waiting on https://review.openstack.org/#/c/604925/7
19:11:50 <clarkb> cmurphy: mordred ianw corvus any other related work you'd like to mention?
19:13:11 <corvus> no, it's back-burner for me right now, but soon i'd like to catch up with ianw on container stuff
19:13:33 <mordred> oh look. it's the infra meeting
19:13:36 * mordred sucks at clocks
19:13:41 <corvus> since i've spent a week dockering docker dockers i may be able to docker docker more docker than i could docker before dockering
19:14:00 <clarkb> that is a lot of docker
19:14:01 <cmurphy> clarkb: i had set that aside for a bit so now that more future parser changes are in i'll try to finish the list
19:14:03 <corvus> on second thought, i may not be any help
19:14:07 <mordred> https://review.openstack.org/#/c/610395/ <-- there is first stab at building a docker container of gerrit 2.15
19:14:27 <clarkb> #link https://review.openstack.org/#/c/610395/ Docker containers for gerrit 2.15
19:14:29 <cmurphy> clarkb: i updated https://review.openstack.org/#/c/576262/ but fungi had already +2'd the version before
19:15:15 <ianw> i was looking at reusing mostly an upstream thing, it will be good to have it being looked at from a few different angles
19:15:20 <clarkb> cmurphy: thanks I think between fungi ianw and myself we should be able to get htat one in shortly
19:15:32 <clarkb> fungi: ianw can you help review https://review.openstack.org/#/c/576262/ since you've reviewed it before?
19:15:41 <fungi> you bet
19:15:52 <ianw> ok
19:15:57 <cmurphy> also wondering about the state of ask.o.o because last i looked at it ask-staging was turned off so we didn't want to experiment on ask.o.o
19:16:23 <clarkb> cmurphy: ya I think ianw had some stuff in motion around that, we should resync on that and make sure we are making progress (or at least not holding up other efforts)
19:16:34 <clarkb> that might be good for after the meeting though?
19:16:40 <mordred> it's worth noting that gerrit 2.14 does not build for  me locally - so am looking in to the idea of using a pure upstream built war for the 2.13->2.14 - and then _immediately_ upgrade to 2.15
19:16:53 <ianw> yeah, let's sync.  i had a memory of turning ask-staging into a xenial host
19:17:01 <mordred> rather than spend time fighting the outdated 2.14 build to make an artifact we don't really care about long term
19:17:43 <corvus> clarkb: what happened to the idea of getting the authorized keys from a url?  (maybe with a new ansible module?)
19:18:13 <cmurphy> and one other thing is that i think the refstack module is broken, i haven't looked into it too deeply but i think someone needs to help get https://review.openstack.org/#/c/557539/ into shape so that we can have tests for it
19:18:24 <clarkb> corvus: its done https://review.openstack.org/#/c/604932/ is child change to the other change because I figured we'd want more review on this one than the simpler first change
19:18:32 <corvus> clarkb: thx
19:19:44 <clarkb> #link https://review.openstack.org/#/c/557539/ Needs to be brought into shape so that we can test refstack and get it future parsered
19:20:52 <clarkb> To summarize there are a number of efforts in progress largely blocked on reviews. If you can find time to review those two topics that were #info'd above that would be much appreciated
19:21:06 <clarkb> Anything else before we jump to the next item?
19:21:40 <clarkb> #topic Storyboard
19:22:02 <clarkb> As mentioned the WIP spec for story attachments could use our input.
19:22:35 <clarkb> This is a feature often requested by users since paste.o.o has a paste size limit and logs.o.o rotate out logs relatively quickly
19:23:13 <clarkb> fungi: diablo_rojo anything else in storyboard land we should be aware of?
19:23:30 <diablo_rojo> clarkb, nothing right now
19:23:35 <diablo_rojo> the WIP spec was all I had
19:23:40 <diablo_rojo> and you already mentioned that :)
19:23:44 <fungi> i'm out of touch wrt sb this week
19:24:25 <fungi> we just had another request (this time from prometheanfire) for constraining the project and project group story views to base theri activity decision only on tasks for the selected project(s)
19:24:48 <fungi> that's probably a low-hanging-fruit thing for sb though i'm not sure whether anyone's opened a story for it yet
19:25:34 <diablo_rojo> fungi, not that I know of
19:25:36 <clarkb> prometheanfire: ^ you might consider opening a story for this item
19:26:18 <clarkb> #topic General Topics
19:26:22 <clarkb> #topic OpenDev
19:26:47 <clarkb> I managed to find a time for us to meet with the foundation leadership/marketing staff and that time is 1800UTC Monday October 22
19:27:09 <clarkb> I know this is likely not ideal for everyone but when trying to schedule around foundation travel it was a reasonable option that popped up in the near future
19:27:23 <clarkb> if you'd like to talk opendev messaging please join us in #openstack-infra at that time
19:27:38 <clarkb> corvus: I was also going to ask about the dns bootstrapping. There are changes that need review for that?
19:28:07 <corvus> clarkb: my most recent re-attempt failed and i haven't looked at why yet: https://review.openstack.org/605092
19:28:28 <clarkb> #link https://review.openstack.org/#/c/605092/ bootstrap opendev dns servers
19:28:29 <corvus> that is closer to the top of my list, but not quite there yet; so if someone has a minute to find/fix that problem, is okay with me
19:29:36 <clarkb> with the new etherpad servers I had considered calling them "etherpad-dev01.opendev.org" and similar but realized we need to update the system config site.pp to support that
19:29:49 <clarkb> any thoughts on maybe updating the regexes in there across the board to make it easy going forward?
19:30:24 <clarkb> we could do something like foo\.(opendev|openstack).org
19:30:49 <mordred> clarkb: foo.open.*.org :)
19:30:56 <clarkb> mordred: ya that might be simpler :)
19:30:57 <corvus> clarkb: do you think we will run into $fqdn problems?
19:31:23 <clarkb> corvus: I don't think so because most (maybe all at this point?) of our services require us to set the server name in apache due to the 01/02/03 etc hosts names
19:31:36 <corvus> k, then i think that should be fine
19:31:47 <clarkb> so we can call the server foo.opendev.org in nova then point foo.openstack.org dns at the service with apache config set to foo.openstack.org
19:32:49 <mordred> clarkb: do we want to make apache redirect .openstack.org to .opendev.org once we have both dns entries? or just let the vhost serve as either name?
19:33:22 <clarkb> mordred: I think we'll end up tackling that on a case per case basis depending on the service? for etherpad I expect we'll redirect openstack.org to opendev.org
19:34:00 <clarkb> git.openstack.org the opposite as that provides an identity to oepnstack and others
19:34:05 <mordred> clarkb: nod. good point
19:34:25 <clarkb> and we'll have to work through that over time. I do think as much as possible the hostnames should be opendev.org though then we can do the majority of dns management through the code reviewed dns system
19:34:35 <mordred> clarkb: ++
19:35:33 <corvus> i think we'll need to talk more about git.*
19:35:56 <corvus> if we're going to focus on opendev as a brand/service/hosting platform, it may be counterproductive
19:36:34 <mordred> agree (that we need to talk more about it - I don't have a strong opinion one way of the other)
19:36:46 <clarkb> ya the details there are definitely worth working out and probably a good topic for the messaging meeting as the dns records take a part in the messaging
19:36:53 <corvus> i'm comping from the pov that anything which confuses the 'canonical' repo name is bad -- this makes things confusing for zuul and other systems/folks that use golang format repo names
19:37:55 <clarkb> corvus: we may need to consider the things that k8s does too in that case (I don't think they are the only one that alias github)
19:38:01 <mordred> corvus: yah. otoh - our fine friends at k8s store their source code in a neutral branded location (github) and use weird aliasing things to make their golang paths be based on k8s.io ...
19:38:04 <mordred> clarkb: jinx
19:38:14 <corvus> maybe we need to make zuul smarter about supporting per-project (rather than per-connection) canonical names
19:38:31 <mordred> which is to say - there are a pile of use cases / requirements here that we should think carefully about
19:38:38 <clarkb> ++
19:38:39 <corvus> yep
19:39:07 <clarkb> alright anyhting else on opendev before we go to the next item?
19:39:11 <corvus> on the face of it, if we go forward with the status quo of git.opendev.org and git.openstack.org both existing without any other changes, i think we'll be in a bad spot.  so we should do something.  :)
19:40:20 <clarkb> corvus: might also help to write down the concerns and constraints? maybe as a followup to my schedulign email for the messaging meeting? (not sure thats the best venue but it is time set aside to talk about related stuff)
19:40:41 <fungi> i just assumed we're going to need to do a lot of project renames into another namespace to "move" repos from git.openstack.org to git.opendev.org
19:41:17 <fungi> perhaps map namespaces to domains 1:1
19:41:36 <fungi> but this discussion can come later
19:41:47 <clarkb> #topic Upgrade sprint
19:42:38 <clarkb> Lets keep the meeting moving. About a month ago I scribbled this week in as an upgrade sprint week. I've been focused on getting etherpad servers updated. The dev server is now running on xenial and I expect to get the prod server to xenial this afternoon
19:42:59 <clarkb> this week worked out as it is between conferences and not in a bad spot for the release cycle
19:43:13 <clarkb> I expect I'll look at logstash.o.o on thursday
19:43:33 <clarkb> If others can grab a service/server or two and work on figuring out what platform upgrades look like that would be helpful to
19:43:55 <clarkb> #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup server upgrade todo list
19:44:39 <mordred> you know - I'll take storyboard - that one seems potentially fun
19:44:47 <clarkb> mordred: thanks.
19:45:03 <clarkb> and do ping me if you need help with reviews or puppet or ansible. I actually started the week fixing ansible so that puppet would run :)
19:45:13 * mordred warns diablo_rojo that he's likely about to start screwing stuff up for everybody
19:45:44 <clarkb> #topic Zuul Queue Backlogs
19:45:52 <mordred> clarkb: oh - I have a (potentially) related question that's also potentially related to the opendev topic ...
19:46:00 <mordred> (sorry, on the previous topic)
19:46:02 <clarkb> #undo
19:46:03 <openstack> Removing item from minutes: #topic Zuul Queue Backlogs
19:46:07 <clarkb> mordred: go for it
19:46:26 <mordred> clarkb: a few weeks ago I grabbed opendevorg dockerhub namespace so we could publish images we build for our services
19:46:44 <mordred> I put the creds into the password file - but haven't added a secret for it yet ...
19:46:49 <mordred> are we good with that as a location?
19:46:52 <mordred> (opendev is taken)
19:47:01 <clarkb> works for me
19:47:17 <mordred> kk. cool. just realized we hadn't actually talked about it :)
19:47:23 <corvus> mordred: is opendev used?
19:47:35 <corvus> or just squatted?
19:47:40 <mordred> corvus: just squatted
19:47:49 <prometheanfire> https://opendev.com is taken
19:47:52 <prometheanfire> ya
19:48:11 <corvus> mordred: maybe we can also ask dockerhub about a transfer?  probably best after we've actually pushed an image up :)
19:48:22 <corvus> then we can demonstrate an actual intent to use it
19:48:32 <clarkb> corvus: ya we don't want to be squatters asking for other squatter namespaces :)
19:48:34 <mordred> corvus: totes - it's def worth asking
19:48:49 <clarkb> #topic Zuul Queue Backlogs
19:49:27 <clarkb> really quickly. Wanted to point out that we continue to be aware of this and are working on various related things to try and make it better
19:49:39 <clarkb> ianw has been looking at the port leaks in ovh with slaweg and dpawlik
19:49:52 <clarkb> ovh is aware of the problems there and is apparntly working to correct it
19:50:10 <ianw> ok, i've had to disable gra1 because it doesn't seem to boot servers
19:50:20 <corvus> clarkb: the node-accounting-in-logs change has merged; needs a scheduler restart.  that would let us more accurately get a gauge for what projects/jobs are using the system the most.
19:50:36 <clarkb> we merged a zuul gate window floor value change to 10 to see if that helps, initial data is that it doesn't but I figured it was worht a shot
19:50:54 <clarkb> I continue to push projects to fix the bugs in their tests and update e-r as necessary to better track longer term issues
19:51:04 <clarkb> Overall I think we are headed in the right direction it is just a slow turning ship
19:51:12 <clarkb> corvus: thanks, I'll keep that in mind (restart in particular)
19:51:45 <clarkb> at this point a lot of the flakyness is outside of our control I think. Looking at e-r data jobs are failing for non infra valid reasons
19:52:05 <clarkb> so other than adding capacity to reduce the impact of ^ we are largely counting on projects to identify and fix their test bugs
19:52:45 <clarkb> #topic git-review
19:53:11 <corvus> clarkb: the gate floor reduction not helping the check backlog seems to suggest supply/demand is important
19:53:22 <clarkb> corvus: yup
19:53:34 <clarkb> fungi: ssbarnea_ quickly before we run out of time. There are two git-review items. First is psuhing prerelease git review tags so people can opt in to testing changes to git-review before the rest of the world sees them
19:53:47 <fungi> this seems like a no-brainer
19:54:01 <fungi> i can push one today and will happily push another any time someone asks
19:54:20 <clarkb> the other is adding core reviewers. stephenfin and ssbarnea_ have offered to be new core reviewers on git-review. I'd like to add them in with an agreement that "stabilizing" git review is the goal. Update tests, fix known bugs, etc
19:54:24 <fungi> might make it slightly easier for people who aren't comfortable cloning the repo and doing a pip install .
19:54:36 <clarkb> then if we end up in a place where git-review is stable wtih reliable testing reconsider adding features
19:55:00 <clarkb> is there any objection to adding them if we agree to that? (I think that should be global git-review dev agrement not just for the new core reviewrs)
19:55:00 <ssbarnea_> fungi: yeah, installing with "pip install --pre git-review" is more friendly IMHO
19:55:16 <clarkb> and ya I htink pushing pre release git reviews is a great idea
19:55:19 <fungi> adding a couple more git-review-core reviewers sounds like a fine plan to me, curious what others think
19:57:04 <ianw> ++ to new reviewers
19:57:37 <clarkb> we are almost out of time. But please let me knwo if you have concerns about adding new core reviewrs to git review in this way
19:57:41 <clarkb> #topic Open Discussion
19:57:47 <clarkb> and now ~2 minutes for anything else
19:58:08 <corvus> i'm happy with new reviewers if we're all on board with it being a really stable tool that doesn't change much and not a new feature magnet
19:59:10 <corvus> and to be really clear about that, i think any change which requires us to alter our contributor docs should have a nearly impossible hill to climb for acceptance.
19:59:26 <clarkb> ++
19:59:35 <fungi> absolutely
19:59:59 <fungi> unless the alteration is "remove some unnecessary things we've automated" maybe
20:00:00 <corvus> but new features like supporting the wip plugin, etc, would be great
20:00:30 <clarkb> #endmeeting