19:03:53 <jeblair> #startmeeting infra
19:03:54 <openstack> Meeting started Tue Aug  4 19:03:53 2015 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:03:55 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:03:57 <openstack> The meeting name has been set to 'infra'
19:04:05 <jeblair> #link agenda https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:04:05 <jeblair> #link previous meeting http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-07-28-19.01.html
19:04:14 <jeblair> #topic Announcements
19:04:20 <jeblair> #topic Announcements: Project management
19:04:30 <jeblair> we've made a lot of changes with how we work recently:
19:04:42 <jeblair> we write specs for any moderately complex task so we can articulate what we want to accomplish and help reviewers understand our changes
19:04:42 <jeblair> we have priority efforts so we know where we should look first to keep our most important efforts moving
19:04:52 <jeblair> and we've radically decentralized with the formation of the council composed of core teams
19:05:02 <jeblair> however, recently i feel like it's getting harder to see where we are in progressing our work
19:05:09 <nibalizer> hi
19:05:22 <jeblair> this is important because it means an effort could be blocked on something without our realizing how to move it forward
19:05:34 <jeblair> or you might imagine some dependency on an effort that involves a transition, and it's not clear whether that's happened yet
19:05:46 <jeblair> all of this can be addressed with better project management (which when done poorly is just nagging, but when done well helps facilitate communication and progress among a distributed team)
19:05:59 <jeblair> so i've asked Clint to help us out with that
19:06:08 <jeblair> and in a fit of generosity, he accepted :)
19:06:11 <fungi> thanks Clint!!!
19:06:18 <jeblair> it's a pretty open ended problem and he'll be looking at how to actually accomplish it
19:06:20 <mrmartin> o/
19:06:29 <jeblair> but i think there's a lot of room for improvement here, and i think it could be really helpful to all of us
19:06:38 <jeblair> i've asked him to start by focusing on the priority efforts, and also maniphest (as i believe it will also be helpful in this regard)
19:06:47 <jeblair> i expect that he may have a lot of questions for folks, so please help him out
19:06:54 <Clint> i will definitely be asking questions
19:07:04 <mrmartin> where is Monty?
19:07:10 <jeblair> Clint: cool :)
19:07:25 <jeblair> mrmartin: on a pacific island
19:07:26 <jhesketh> Clint: sounds great :-)
19:07:29 <Shrews> monty is vacationing
19:07:39 <aukhan> :)
19:07:46 <jeblair> we should totally have a "where is monty" agenda item; though this isn't it :)
19:08:08 <Shrews> jeblair: i have considered a cross-platform app
19:09:16 <jeblair> any other questions or helpful suggestions for Clint?
19:10:09 <jeblair> Clint: thanks!
19:10:13 * Clint nods.
19:10:13 <jeblair> #topic Specs approval
19:10:17 <jeblair> #topic Specs approval: Zuul v3 (jeblair)
19:10:36 <jeblair> #link zuulv3 spec http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html
19:10:42 <jeblair> #info zuulv3 spec was approved
19:10:48 <pabelanger> Niiice
19:10:51 <pleia2> huzzah
19:11:06 <jeblair> some quick thoughts on that:
19:11:23 <jeblair> i'm thinking of revising the spec to suggest that we do development in branches on both zuul and nodepool
19:11:49 <jhesketh> +1
19:11:57 <jeblair> and i'm going to collect the names of folks from the summit etherpad and those who otherwise volunteered to help and see if we can have a huddle and discuss how to proceed
19:12:01 <jeblair> maybe divvy up some items
19:12:19 <jeblair> so keep an eye out for that
19:12:29 <greghaynes> There might be a lot of churn in nodepool with both v3 and builders at the same time
19:12:34 <greghaynes> maybe too much rebasing
19:12:49 <jeblair> greghaynes: with branches we can use merge commits!
19:12:49 <greghaynes> and branching will make that a bit worse
19:13:08 <greghaynes> yes, there will be a pretty big delta, its more clear to resolve but still a lot of drift
19:13:24 <jeblair> yeah, if we can get builders in soon, that will help
19:13:51 <jeblair> but if we don't, i'm sure we'll manage
19:14:15 <jeblair> #topic Schedule Project Renames
19:14:32 <jeblair> i'm going to abuse this topic since i didn't put in what we should really start talking about...
19:15:00 <jeblair> which is that the "put all new projects in openstack/" resolution was approved
19:15:02 <jeblair> #link http://governance.openstack.org/resolutions/20150615-stackforge-retirement.html
19:15:23 <jeblair> so we should work on a plan for dealing with that --
19:15:47 <greghaynes> does this mean we are also moving the existing stackforge ones over?
19:15:47 <jeblair> do we want to perform an automatic mass-migration simultaneously ourselves?
19:15:59 <greghaynes> oh wow, it does
19:16:26 <fungi> i think if we don't move everyone at once, it will be painful
19:16:52 <jeblair> or let projects do it on their own time-frame, but maybe we batch them so we do moves every 2 months or so...?
19:16:52 <clarkb> we should try to get a list of those that want to be attic'd as part of hte process too (thinking at least pecan)
19:16:59 <fungi> i mean, _more_ painful
19:17:09 <jeblair> oh, also, i would like to end the practice of moving things to the attic
19:17:29 <clarkb> jeblair: ok, what do we do about pecan then? just hav ethem update that repo with a "use github" message instead?
19:17:30 <jeblair> i previously had a week preference not to attic things
19:17:43 <fungi> yeah, attic means more pointless renaming
19:17:52 <clarkb> (I am happy either way, but we do need to accomodate their concerns)
19:18:08 <jeblair> but now that i have seen projects moved to the attic with absolutely no indication they are actually dead (complete with transparent github rewrites so their users won't even notice), i have a _strong_ conviction it's a bad idea.
19:18:42 <jeblair> clarkb: yeah, so perhaps we let defunct (for whatever reason) projects just stay in stackforge
19:18:51 <fungi> if we need a script to push up a boilerplate readme and file deletion change along with a read-only acl update, that's still a much better use of our time to semi-automate
19:18:52 <jeblair> how about this:
19:19:13 <jeblair> we ask projects to sign up to be moved to openstack.  if they don't sign up, they end up on a list of projects we will make read-only.
19:19:30 <jeblair> we publicise that list, and timetable, etc, so people can review it.
19:19:41 <jeblair> then we move them en-masse, ourselves.
19:19:54 <clarkb> that seems reasonable
19:20:03 <jeblair> (and, obviously, do not move the ones on the read-only list)
19:20:07 <greghaynes> ++
19:20:15 <pleia2> "publicise" may be tricky, but can we poll+send to cores on all the projects?
19:20:22 <pleia2> I think some stackforge folks don't read all of openstack-dev
19:20:28 <clarkb> pleia2: ya that should be easy with a gerrit api script
19:20:31 <pleia2> cool
19:20:32 <nibalizer> will 'read-only' projects be listed anywhere other than gerrit acls?
19:20:55 <fungi> should be trivial to build a contact list. oh, i already almost have that written i think
19:21:06 <nibalizer> a number of people (and me) write scripts like for repo in `gerrit ls-projects | grep infra | grep puppet`; do something; git review; done
19:21:22 <nibalizer> and inactive/readonly projects in that list screws that up
19:21:56 <fungi> #link http://git.openstack.org/cgit/openstack-infra/system-config/tree/tools/who-approves.py
19:21:57 <jhesketh> I wonder if ls-projects can be limited to non-readonly somehow?
19:22:15 <jeblair> how about this: i write a message to openstack-dev giving folks general context for governance resolution, since it still hasn't really been publicised, and let people know that we will start planning a future move. then after that, i'll start a thread for planning the actual move, and we can work out logistics and dates.
19:22:27 <pleia2> +1
19:22:54 <clarkb> jeblair: sounds good to me
19:23:15 <nibalizer> that sounds good
19:23:19 <jeblair> nibalizer: maybe we should look into whether we can do something similar with the rest api?
19:23:36 <jeblair> #action jeblair write message to openstack-dev with overall context for stackforge retirement/move
19:23:40 <fungi> sounds great. we also all need to remember to remind people who don't read closely that this is a purely cosmetic change. there's no implied governance alteration needed/accompanying it
19:24:04 <jeblair> #action jeblair start discussion thread about logistics of repo moves
19:24:15 <clarkb> its a thing that only really makes sense in the github world
19:24:23 <clarkb> and we github so :/
19:24:47 <fungi> we're just "getting rid of the word 'stackforge' in our urls"
19:25:11 <fungi> (and replacing it with another word we already use a lot)
19:25:13 <jeblair> yep.  and maybe in 6 months or a year, we have a discussion like "hey, so should we remove openstack/ from git.o.o"? :)  (would be so much less typing)  but that's getting way ahead.
19:25:49 <jeblair> okay, i think we know generally how to proceed here, thanks
19:25:53 <jeblair> #topic Priority Efforts (Migration to Zanata)
19:25:53 * fungi would (eventually) love that
19:26:02 <pleia2> hey
19:26:08 <jeblair> pleia2: howdy!
19:26:21 <pleia2> so I've got UbuConLA, some hiking in the Andes and then the Ops Mid-cycle in the next two weeks (starting tomorrow)
19:26:43 <pleia2> I don't want my absence to cause problems for the i18n team and StevenK who are still working on the Zanata stuff
19:27:08 <pleia2> on line 200 of https://etherpad.openstack.org/p/zanata-install I listed my outstanding tasks, so I'd like to do a soft hand-off to someone while I'm gone so things keep chugging along
19:27:30 <clarkb> I can help out there
19:27:40 <clarkb> I am relatively familiar with how all that fits together so happy to
19:27:55 <clarkb> #link https://etherpad.openstack.org/p/zanata-install
19:28:20 <pleia2> and as always, if I can get folks to review open zanata reviews, it would get StevenK unstuck in a few places https://review.openstack.org/#/q/status:open+topic:zanata,n,z
19:28:23 <jeblair> #info clarkb is pleia2 for next 2 weeks
19:28:46 <pleia2> https://review.openstack.org/#/c/206305/ needs another +2
19:28:48 <clarkb> I did review the jeepyb change stevenK wrote yesterday
19:28:51 <clarkb> it looks good
19:28:55 <pleia2> great
19:29:03 <clarkb> need to find the puppet change to run it but that should be straightforwrad
19:29:18 <clarkb> will review 206305 today
19:29:22 <jeblair> clarkb: https://review.openstack.org/208811
19:29:28 <jeblair> topic:zanata ftw
19:29:40 <pleia2> he also said he'll be working on 203928 this week, so that WIP will move along soon
19:29:53 <clarkb> the jeepyb change is under that topic too, perfect
19:30:16 <jeblair> clarkb, pleia2: thanks!
19:30:29 <pleia2> I think that's it, I'm around for the rest of the day if clarkb needs any more details for anything
19:30:55 <pleia2> thanks clarkb :)
19:31:02 <jeblair> #topic #topic Puppet-httpd replacement of puppet-apache (pabelanger/nibalizer)
19:31:04 <jeblair> gah
19:31:05 <jeblair> #topic Puppet-httpd replacement of puppet-apache (pabelanger/nibalizer)
19:31:18 <nibalizer> this is quite close to being done
19:31:22 <pabelanger> aye
19:31:40 <nibalizer> https://review.openstack.org/#/c/198819/ one more
19:32:06 <nibalizer> to system-config this time, not a module
19:32:43 <pabelanger> https://review.openstack.org/#/c/201759/ is for grafana
19:32:49 <pabelanger> and system-config too
19:32:58 <nibalizer> its also worth noting that the existence of a patch like this demonstrates that there is still active configuration inside system-config that hasn't been pushed out into the modules, but thats a different topic
19:33:55 <jeblair> nibalizer: yeah, some of thot is probably some low-hanging fruit
19:33:56 <nibalizer> there are a couple follow on activites such as removing if statements that paul had to hack in, as well as finally removing apache 0.0.4 from modules.env
19:34:16 <nibalizer> but I'm very pleased with how quickly and effectively this went
19:34:17 <jeblair> (static.o.o is not lhf, but many of the others are)
19:34:19 <nibalizer> thanks pabelanger
19:34:41 <pabelanger> https://review.openstack.org/#/c/205696/ is the removal of puppet-apache
19:34:49 <jeblair> pabelanger, nibalizer: indeed, thanks! :)
19:34:51 <pabelanger> nibalizer: np, was strightforward
19:35:44 <nibalizer> thats all I have on that topic
19:35:49 <jeblair> cool, thanks
19:35:56 <jeblair> #topic Restore from backup test (jeblair)
19:36:12 <jeblair> this is a thing we have talked about a bit... but haven't really done
19:36:24 <jeblair> i mean, backups are great, but you don't really have them unless you've used them
19:36:40 <jeblair> when i set some of this up, early on i did some limited restore tests
19:36:52 <jeblair> (like log into the server and restore to a dir in /tmp/)
19:37:08 <jeblair> but that was a long time ago, and not at all systematic, nor has it been done for every host
19:37:12 <clarkb> when I updated the docs I too restored specific files from backup to test the docs
19:37:21 <greghaynes> For some context - what all is this a backup of?
19:37:23 <clarkb> but has all those same issues
19:37:27 <jeblair> greghaynes: ah great question!
19:37:50 <jeblair> we perform fairly traditional filesystem backups of major hosts to an off-host location
19:37:59 <jeblair> specifically a server in a different cloud account on a different provider
19:38:07 <jeblair> er
19:38:11 <jeblair> strike different provider
19:38:25 <greghaynes> heh, different region?
19:38:27 <fungi> (eventually different provider)
19:38:31 <clarkb> yes different region
19:38:31 <jeblair> yes different region
19:38:40 <greghaynes> ok, gotcha
19:38:51 <greghaynes> I am really a fan of daily restore from backup jobs :)
19:38:54 <jeblair> we used to, but then the 'different provider' did one of the things that the diversity is supposed to protect us from
19:38:59 <jeblair> which is, they deleted the host
19:39:06 <jeblair> greghaynes: no points for guessing which
19:39:13 <greghaynes> haha
19:39:58 <jeblair> but yeah, ideally, we back up to multiple locations that are as diverse from where we actually run these as possible.  we are not currently living the dream.
19:40:33 <jeblair> at any rate, i put this on the agenda so we could start talking about it and brainstorming...
19:40:45 <fungi> though it makes a great point. prepare for the loss of your backup server ;)
19:40:59 <clarkb> we do have a third region available to us to add diversity
19:41:09 <jeblair> (we should probably work on replacing the one that was deleted, i think we can now)
19:41:31 <fungi> nameless provider seems to have gotten a little more reliable about not randomly deleting our servers
19:41:35 <jeblair> since i did most of the work designing the system as is, i think i'd like someone else to take the lead on evaluating our preparedness to restore
19:42:11 <jeblair> i was imagining we design and conduct a periodic drill, but i'm also open to greghaynes suggestion of automated continuous testing...
19:42:28 <clarkb> not to derail on specifics, but one related item I think we need to handle better is the we ran out of disk on backup server issue since our backups are append only
19:42:43 <jeblair> clarkb: agreed
19:43:22 <fungi> i like the automated continuous testing idea if it's feasible (i mean, we already do a lot of that) but would need some way to assure we pay attention to failures
19:43:23 <jeblair> #link http://docs.openstack.org/infra/system-config/sysadmin.html#backups
19:43:26 <greghaynes> Yea, im sure its a matter of how much work restoring from backup is. Also, the automatedish setup is only useful if you have a good way to notice when a fialure happens
19:44:09 <fungi> periodic drills are something we can have as early as today if we want them, modulo available root sysadmin availability
19:44:42 <greghaynes> Going through one is probably a prereq for knowing what all an automated solution would need to do, also
19:44:47 <greghaynes> at least for me
19:45:17 <jeblair> greghaynes: good point
19:45:34 <jeblair> so probably we should start with that, and then see whether we think the process is amenable to automation
19:46:13 <greghaynes> ++
19:46:40 <jeblair> at any rate, i'm looking for not-me volunteers for this... think about whether you can help out, and we'll check back in next week
19:46:52 <fungi> yep, identifying _what_ we want to check is probably step #1
19:47:18 <jeblair> #topic  Puppet apply coverage for fedora21 / centos7 (pabelanger)
19:47:25 <pabelanger> hey
19:47:43 <jeblair> pabelanger: o/
19:47:49 <pabelanger> this one is quick, I have 2 reviews up to add support for Fedora21 and Centos7 for our puppet-apply tests.
19:47:53 <pabelanger> https://review.openstack.org/#/c/203756/
19:47:57 <pabelanger> https://review.openstack.org/#/c/203759/
19:48:16 <pabelanger> they are non-voting jobs right now, but want to kick them to voting for next week
19:48:25 <pabelanger> so, just need some help reviewing and approving
19:48:33 <jeblair> oh hey, they make the jobs go green, respectively
19:49:02 <jeblair> this seems like a swell idea
19:49:10 * clarkb will add to review list today
19:49:24 <ianw> while people are thinking about puppet-apply, https://review.openstack.org/205873 & https://review.openstack.org/206319 are about running it with project-config
19:49:54 <jeblair> ianw: oh thanks for bringing that up...
19:50:30 <pabelanger> thats all I had
19:50:45 <jeblair> i've been thinking about that, and i think there's the idea that it should have an impact, though it doesn't in our current jobs
19:51:25 <nibalizer> I think it matters if project_config ever grows a puppet template
19:51:27 <nibalizer> but right now it does not
19:51:30 <jeblair> that is to say, it really is related, but our apply jobs may not get far enough in what they do to actually exercise it
19:51:33 <clarkb> ianw: we pull the ocnfig we are deploying out of project_config so it needs to cogate
19:51:50 <clarkb> ianw: for noop its less important but anything else would need it
19:52:46 <nibalizer> and on the subject of puppet: https://review.openstack.org/#/c/206779/ adds a hiera directory to system-config and starts using it
19:52:49 <clarkb> (just noting that as we do the other beaker based testing)
19:53:17 <pabelanger> nibalizer: nice, will be checking that out
19:53:33 <clarkb> pabelanger: https://review.openstack.org/#/c/203756/ doesn't pass the f21 test
19:54:15 <ianw> ok, well i'll have to think about it.  i think we could skip those jobs for the bulk of project-config changes that are adding/modifying zuul jobs
19:54:15 <pabelanger> clarkb: right, I don't know why right now.  I need to check out dib builds to see what happens. Since it should be the same code for production installs
19:54:29 <pabelanger> clarkb: I won't be gating that until I figure it out
19:55:14 <jeblair> ianw: yeah.  i think it's worth some thought.  there's definitely a relationship there.  we're probably not adequately testing it.  if we can, we should.  but if it's not possible, then yeah, maybe we should drop the jobs.
19:55:35 <jeblair> #topic Open discussion
19:55:39 <EmilienM> 1/ could we have reviews on "Add support for backport-potential commit flag" > https://review.openstack.org/#/c/175849/ and 2/ Also "jenkins: implement copy-puppet-logs publisher" (miss +A) > https://review.openstack.org/206693 - thank you
19:56:37 <zaro> Gerrit upgrade update, new info in #link https://code.google.com/p/gerrit/issues/detail?id=3424
19:56:43 <pabelanger> So, I know we only have 5mins, but was going to do this on mailing list. Could somebody explain when it is appropiate to make a job (node) reusable? https://github.com/openstack-infra/project-config/blob/master/zuul/openstack_functions.py#L34
19:56:46 <clarkb> EmilienM: my initial impression of 175849 is the submitter wrote the commit message should just push to the old branch so that conflicts can be handled
19:57:03 <zaro> was wondering if you guys can take a look and help me determine if we want to increase diff timeout?
19:57:25 <jaypipes> pabelanger: hey, what's the latest status on bringing stackalytics into openstack.org?
19:57:28 <clarkb> ericsson has independently ocnfirmed our jenkins slave plugin bug
19:58:05 <pabelanger> jaypipes: good, I rebased patches today. We're pretty close. I suspect next week I'll have a request up for reviews / approval
19:58:24 <jaypipes> pabelanger: coolio. let me know if you need any assistance from my end.
19:58:41 <EmilienM> clarkb: don't you think it's a good first iteration?
19:58:55 <clarkb> EmilienM: I worry its putting too much effort into a nintractable problem
19:59:18 <ianw> pabelanger: i think reusable if it's not running untrusted code?
19:59:27 <ianw> is any code trusted though...
19:59:29 <clarkb> its trivial to push the backport if you know you need to push it and it allows you to do the non trivial work of resolving conflicts
20:00:05 <pabelanger> ianw: so, what if a job sandboxes code into the workspace, which gets deleted after a job runs?  Could that be reusable?
20:00:20 <pabelanger> ianw: EG: not using sudo
20:00:30 <EmilienM> clarkb: it's just we want to automate it
20:00:36 <EmilienM> clarkb: the "push" part
20:00:42 <jeblair> times up; maybe finish these in #-infra
20:00:43 <clarkb> EmilienM: yes and I am saying you cannot automate it
20:00:47 <jeblair> thanks everyone!
20:00:50 <jeblair> #endmeeting