19:01:07 #startmeeting infra 19:01:08 Meeting started Tue Oct 16 19:01:07 2018 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:11 The meeting name has been set to 'infra' 19:01:17 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:01:39 I cleaned up the meeting agenda based on what was discussed during last meeting and what we have going on this week. 19:01:45 #topic Announcements 19:02:06 I'm likely to be afk tomorrow enjoying some of the last "good" weather before winter invades. My brothers demand I go fishing with them 19:02:16 fungi is also afk dealing with things like sea grass aiui 19:02:33 fungi: can you see a lot of sea grass from the sea shore? 19:02:35 most of the sea grass is back outside again where it belongs 19:02:55 #link https://www.openstack.org/summit/berlin-2018/summit-schedule/#track=262 Berlin Forum schedule up for feedback 19:03:15 As we get closer to the summit and forum the schedule is up and probably worth looking over for any hard to manage conflicts 19:03:43 i am gere 19:04:12 #topic Actions from last meeting 19:04:23 #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-10-09-19.00.txt minutes from last meeting 19:05:01 we don't appear to have recorded any formal actions, but ssbarnea_ had some items to talk about. I wanted to follow up and ask if we need to talk about any more of those today? 19:05:15 ssbarnea_: ^ or is there any other input you need to make some of those actionable? 19:05:36 worth discussing the suggestion of git-review prereleases if it's not been added to the agenda yet 19:05:46 clarkb: nope, not for now. 19:06:01 or can bring it up during open discussion at the end 19:06:05 fungi: and adding new cores. We should have time to go through that during the end of the meeting 19:06:10 awesome 19:06:20 #topic Specs approval 19:06:34 I approved the third party ci direction spec last week and it merged 19:06:48 we can now point to that when interested parties want to get involved running third party ci setups 19:07:29 the other spec people should be aware of (but is still WIP) is the storyboard attachments spec 19:07:54 #link https://review.openstack.org/#/c/607377/ Storyboard Attachments Spec. Would be good toget input on that from an operational perspective. Concerns like overal disk usage and not wanting to host spam etc 19:08:57 #topic Priority Efforts 19:09:06 #topic Update Config Management 19:09:25 #info topic:puppet-4 and topic:update-cfg-mgmt to find changes to review 19:09:39 \o/ 19:09:48 I've got all but one of the outstanding futureparser changes merged at this point. Thanks to cmurphy for continuing to push on that 19:10:06 There are other outstanding puppet-4 related changes to make it possible to futureparser more things 19:10:49 mordred: has started looking at deploying gerrit with containers and how a gerrit 2.14/2.15 might fit into that 19:10:59 ianw: is looking at graphite deployment from containers too aiui 19:11:17 For the continuous deployment work I think we are still waiting on https://review.openstack.org/#/c/604925/7 19:11:50 cmurphy: mordred ianw corvus any other related work you'd like to mention? 19:13:11 no, it's back-burner for me right now, but soon i'd like to catch up with ianw on container stuff 19:13:33 oh look. it's the infra meeting 19:13:36 * mordred sucks at clocks 19:13:41 since i've spent a week dockering docker dockers i may be able to docker docker more docker than i could docker before dockering 19:14:00 that is a lot of docker 19:14:01 clarkb: i had set that aside for a bit so now that more future parser changes are in i'll try to finish the list 19:14:03 on second thought, i may not be any help 19:14:07 https://review.openstack.org/#/c/610395/ <-- there is first stab at building a docker container of gerrit 2.15 19:14:27 #link https://review.openstack.org/#/c/610395/ Docker containers for gerrit 2.15 19:14:29 clarkb: i updated https://review.openstack.org/#/c/576262/ but fungi had already +2'd the version before 19:15:15 i was looking at reusing mostly an upstream thing, it will be good to have it being looked at from a few different angles 19:15:20 cmurphy: thanks I think between fungi ianw and myself we should be able to get htat one in shortly 19:15:32 fungi: ianw can you help review https://review.openstack.org/#/c/576262/ since you've reviewed it before? 19:15:41 you bet 19:15:52 ok 19:15:57 also wondering about the state of ask.o.o because last i looked at it ask-staging was turned off so we didn't want to experiment on ask.o.o 19:16:23 cmurphy: ya I think ianw had some stuff in motion around that, we should resync on that and make sure we are making progress (or at least not holding up other efforts) 19:16:34 that might be good for after the meeting though? 19:16:40 it's worth noting that gerrit 2.14 does not build for me locally - so am looking in to the idea of using a pure upstream built war for the 2.13->2.14 - and then _immediately_ upgrade to 2.15 19:16:53 yeah, let's sync. i had a memory of turning ask-staging into a xenial host 19:17:01 rather than spend time fighting the outdated 2.14 build to make an artifact we don't really care about long term 19:17:43 clarkb: what happened to the idea of getting the authorized keys from a url? (maybe with a new ansible module?) 19:18:13 and one other thing is that i think the refstack module is broken, i haven't looked into it too deeply but i think someone needs to help get https://review.openstack.org/#/c/557539/ into shape so that we can have tests for it 19:18:24 corvus: its done https://review.openstack.org/#/c/604932/ is child change to the other change because I figured we'd want more review on this one than the simpler first change 19:18:32 clarkb: thx 19:19:44 #link https://review.openstack.org/#/c/557539/ Needs to be brought into shape so that we can test refstack and get it future parsered 19:20:52 To summarize there are a number of efforts in progress largely blocked on reviews. If you can find time to review those two topics that were #info'd above that would be much appreciated 19:21:06 Anything else before we jump to the next item? 19:21:40 #topic Storyboard 19:22:02 As mentioned the WIP spec for story attachments could use our input. 19:22:35 This is a feature often requested by users since paste.o.o has a paste size limit and logs.o.o rotate out logs relatively quickly 19:23:13 fungi: diablo_rojo anything else in storyboard land we should be aware of? 19:23:30 clarkb, nothing right now 19:23:35 the WIP spec was all I had 19:23:40 and you already mentioned that :) 19:23:44 i'm out of touch wrt sb this week 19:24:25 we just had another request (this time from prometheanfire) for constraining the project and project group story views to base theri activity decision only on tasks for the selected project(s) 19:24:48 that's probably a low-hanging-fruit thing for sb though i'm not sure whether anyone's opened a story for it yet 19:25:34 fungi, not that I know of 19:25:36 prometheanfire: ^ you might consider opening a story for this item 19:26:18 #topic General Topics 19:26:22 #topic OpenDev 19:26:47 I managed to find a time for us to meet with the foundation leadership/marketing staff and that time is 1800UTC Monday October 22 19:27:09 I know this is likely not ideal for everyone but when trying to schedule around foundation travel it was a reasonable option that popped up in the near future 19:27:23 if you'd like to talk opendev messaging please join us in #openstack-infra at that time 19:27:38 corvus: I was also going to ask about the dns bootstrapping. There are changes that need review for that? 19:28:07 clarkb: my most recent re-attempt failed and i haven't looked at why yet: https://review.openstack.org/605092 19:28:28 #link https://review.openstack.org/#/c/605092/ bootstrap opendev dns servers 19:28:29 that is closer to the top of my list, but not quite there yet; so if someone has a minute to find/fix that problem, is okay with me 19:29:36 with the new etherpad servers I had considered calling them "etherpad-dev01.opendev.org" and similar but realized we need to update the system config site.pp to support that 19:29:49 any thoughts on maybe updating the regexes in there across the board to make it easy going forward? 19:30:24 we could do something like foo\.(opendev|openstack).org 19:30:49 clarkb: foo.open.*.org :) 19:30:56 mordred: ya that might be simpler :) 19:30:57 clarkb: do you think we will run into $fqdn problems? 19:31:23 corvus: I don't think so because most (maybe all at this point?) of our services require us to set the server name in apache due to the 01/02/03 etc hosts names 19:31:36 k, then i think that should be fine 19:31:47 so we can call the server foo.opendev.org in nova then point foo.openstack.org dns at the service with apache config set to foo.openstack.org 19:32:49 clarkb: do we want to make apache redirect .openstack.org to .opendev.org once we have both dns entries? or just let the vhost serve as either name? 19:33:22 mordred: I think we'll end up tackling that on a case per case basis depending on the service? for etherpad I expect we'll redirect openstack.org to opendev.org 19:34:00 git.openstack.org the opposite as that provides an identity to oepnstack and others 19:34:05 clarkb: nod. good point 19:34:25 and we'll have to work through that over time. I do think as much as possible the hostnames should be opendev.org though then we can do the majority of dns management through the code reviewed dns system 19:34:35 clarkb: ++ 19:35:33 i think we'll need to talk more about git.* 19:35:56 if we're going to focus on opendev as a brand/service/hosting platform, it may be counterproductive 19:36:34 agree (that we need to talk more about it - I don't have a strong opinion one way of the other) 19:36:46 ya the details there are definitely worth working out and probably a good topic for the messaging meeting as the dns records take a part in the messaging 19:36:53 i'm comping from the pov that anything which confuses the 'canonical' repo name is bad -- this makes things confusing for zuul and other systems/folks that use golang format repo names 19:37:55 corvus: we may need to consider the things that k8s does too in that case (I don't think they are the only one that alias github) 19:38:01 corvus: yah. otoh - our fine friends at k8s store their source code in a neutral branded location (github) and use weird aliasing things to make their golang paths be based on k8s.io ... 19:38:04 clarkb: jinx 19:38:14 maybe we need to make zuul smarter about supporting per-project (rather than per-connection) canonical names 19:38:31 which is to say - there are a pile of use cases / requirements here that we should think carefully about 19:38:38 ++ 19:38:39 yep 19:39:07 alright anyhting else on opendev before we go to the next item? 19:39:11 on the face of it, if we go forward with the status quo of git.opendev.org and git.openstack.org both existing without any other changes, i think we'll be in a bad spot. so we should do something. :) 19:40:20 corvus: might also help to write down the concerns and constraints? maybe as a followup to my schedulign email for the messaging meeting? (not sure thats the best venue but it is time set aside to talk about related stuff) 19:40:41 i just assumed we're going to need to do a lot of project renames into another namespace to "move" repos from git.openstack.org to git.opendev.org 19:41:17 perhaps map namespaces to domains 1:1 19:41:36 but this discussion can come later 19:41:47 #topic Upgrade sprint 19:42:38 Lets keep the meeting moving. About a month ago I scribbled this week in as an upgrade sprint week. I've been focused on getting etherpad servers updated. The dev server is now running on xenial and I expect to get the prod server to xenial this afternoon 19:42:59 this week worked out as it is between conferences and not in a bad spot for the release cycle 19:43:13 I expect I'll look at logstash.o.o on thursday 19:43:33 If others can grab a service/server or two and work on figuring out what platform upgrades look like that would be helpful to 19:43:55 #link https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup server upgrade todo list 19:44:39 you know - I'll take storyboard - that one seems potentially fun 19:44:47 mordred: thanks. 19:45:03 and do ping me if you need help with reviews or puppet or ansible. I actually started the week fixing ansible so that puppet would run :) 19:45:13 * mordred warns diablo_rojo that he's likely about to start screwing stuff up for everybody 19:45:44 #topic Zuul Queue Backlogs 19:45:52 clarkb: oh - I have a (potentially) related question that's also potentially related to the opendev topic ... 19:46:00 (sorry, on the previous topic) 19:46:02 #undo 19:46:03 Removing item from minutes: #topic Zuul Queue Backlogs 19:46:07 mordred: go for it 19:46:26 clarkb: a few weeks ago I grabbed opendevorg dockerhub namespace so we could publish images we build for our services 19:46:44 I put the creds into the password file - but haven't added a secret for it yet ... 19:46:49 are we good with that as a location? 19:46:52 (opendev is taken) 19:47:01 works for me 19:47:17 kk. cool. just realized we hadn't actually talked about it :) 19:47:23 mordred: is opendev used? 19:47:35 or just squatted? 19:47:40 corvus: just squatted 19:47:49 https://opendev.com is taken 19:47:52 ya 19:48:11 mordred: maybe we can also ask dockerhub about a transfer? probably best after we've actually pushed an image up :) 19:48:22 then we can demonstrate an actual intent to use it 19:48:32 corvus: ya we don't want to be squatters asking for other squatter namespaces :) 19:48:34 corvus: totes - it's def worth asking 19:48:49 #topic Zuul Queue Backlogs 19:49:27 really quickly. Wanted to point out that we continue to be aware of this and are working on various related things to try and make it better 19:49:39 ianw has been looking at the port leaks in ovh with slaweg and dpawlik 19:49:52 ovh is aware of the problems there and is apparntly working to correct it 19:50:10 ok, i've had to disable gra1 because it doesn't seem to boot servers 19:50:20 clarkb: the node-accounting-in-logs change has merged; needs a scheduler restart. that would let us more accurately get a gauge for what projects/jobs are using the system the most. 19:50:36 we merged a zuul gate window floor value change to 10 to see if that helps, initial data is that it doesn't but I figured it was worht a shot 19:50:54 I continue to push projects to fix the bugs in their tests and update e-r as necessary to better track longer term issues 19:51:04 Overall I think we are headed in the right direction it is just a slow turning ship 19:51:12 corvus: thanks, I'll keep that in mind (restart in particular) 19:51:45 at this point a lot of the flakyness is outside of our control I think. Looking at e-r data jobs are failing for non infra valid reasons 19:52:05 so other than adding capacity to reduce the impact of ^ we are largely counting on projects to identify and fix their test bugs 19:52:45 #topic git-review 19:53:11 clarkb: the gate floor reduction not helping the check backlog seems to suggest supply/demand is important 19:53:22 corvus: yup 19:53:34 fungi: ssbarnea_ quickly before we run out of time. There are two git-review items. First is psuhing prerelease git review tags so people can opt in to testing changes to git-review before the rest of the world sees them 19:53:47 this seems like a no-brainer 19:54:01 i can push one today and will happily push another any time someone asks 19:54:20 the other is adding core reviewers. stephenfin and ssbarnea_ have offered to be new core reviewers on git-review. I'd like to add them in with an agreement that "stabilizing" git review is the goal. Update tests, fix known bugs, etc 19:54:24 might make it slightly easier for people who aren't comfortable cloning the repo and doing a pip install . 19:54:36 then if we end up in a place where git-review is stable wtih reliable testing reconsider adding features 19:55:00 is there any objection to adding them if we agree to that? (I think that should be global git-review dev agrement not just for the new core reviewrs) 19:55:00 fungi: yeah, installing with "pip install --pre git-review" is more friendly IMHO 19:55:16 and ya I htink pushing pre release git reviews is a great idea 19:55:19 adding a couple more git-review-core reviewers sounds like a fine plan to me, curious what others think 19:57:04 ++ to new reviewers 19:57:37 we are almost out of time. But please let me knwo if you have concerns about adding new core reviewrs to git review in this way 19:57:41 #topic Open Discussion 19:57:47 and now ~2 minutes for anything else 19:58:08 i'm happy with new reviewers if we're all on board with it being a really stable tool that doesn't change much and not a new feature magnet 19:59:10 and to be really clear about that, i think any change which requires us to alter our contributor docs should have a nearly impossible hill to climb for acceptance. 19:59:26 ++ 19:59:35 absolutely 19:59:59 unless the alteration is "remove some unnecessary things we've automated" maybe 20:00:00 but new features like supporting the wip plugin, etc, would be great 20:00:30 #endmeeting