19:01:27 #startmeeting infra 19:01:28 Meeting started Tue Aug 18 19:01:27 2015 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:29 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:31 The meeting name has been set to 'infra' 19:01:39 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:01:46 o/ 19:01:48 o/ 19:01:56 #topic Announcements 19:02:00 \o 19:02:01 o/ 19:02:18 #link http://lists.openstack.org/pipermail/openstack-infra/2015-August/003069.html 19:02:44 just a reminder that we're planning to mass rename repos from the stackforge namespace into a common openstack namespace, on an opt-in basis 19:03:10 o/ 19:03:16 o/ 19:03:17 and then mark any that haven't opted in as read-only/archived after a time 19:03:25 o/ 19:03:28 follow up to the thread with further discussion on scheduling 19:03:37 Morning 19:03:43 #link http://lists.openstack.org/pipermail/openstack-infra/2015-August/003075.html 19:04:21 due to recent growth of core reviewers in neutron, they've requested that we clear job changes through mestery, armax or dougwig unless it's really urgent 19:04:58 so just bear in mind, when reviewing changes to job configuration that could potentially impact what's tested for neutron that we're looking for a +1 from at least one of them if possible 19:05:25 #topic Actions from last meeting 19:05:38 #link http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-08-11-19.02.html 19:05:49 pabelanger,jasondotstar look into restore-from-backup testing 19:06:01 i know that's ongoing, we've been discussing it in channel 19:06:11 we've captured our thoughts here: https://etherpad.openstack.org/p/infra-backup-brainstorm 19:06:21 the next step is to run through our test strategy 19:06:26 #link https://etherpad.openstack.org/p/infra-backup-brainstorm 19:06:50 jasondotstar: pabelanger: thanks! we can work from there i think 19:06:57 we'd like to also ask folks to think of any other backup/restore scenarios worth testing (manually) 19:07:09 put them there on the etherpad. 19:07:52 once we identify the manual steps and validate that they work 19:07:53 sounds good 19:08:01 Just chiming in.. there is one set of data in infra-cloud that is hard to reproduce, and that is the Ironic inventory. We have a plan to store most of it in git, with the secret bits in hiera. 19:08:02 we can think about automating. 19:08:26 SpamapS: ack. 19:08:29 So that just means any backup systems need to make sure git and hiera are backed up. 19:08:38 SpamapS: cool, it's definitely the hard-to-reproduce things we actually care about backing up 19:08:41 Also hiera backups, I assume, need to be encrypted. 19:08:48 yeah 19:08:58 which is an exercise to be completed :/ 19:09:04 the hiera data is primarily on the puppetmasters...or somewhere else? 19:09:14 yes would be on the puppetmaster(s) 19:09:21 jasondotstar: it's a static file on the puppetmaster 19:09:21 clarkb: ack. 19:10:02 so we should think of an encryption method for sensitive data such as this... 19:10:29 the tricky thing is the chicke and egg with backing up the encryption key/sharedsecret 19:10:43 * jasondotstar agrees 19:10:48 I think thats where I got distracted last time I looked into this 19:11:22 pabelanger and i will capture this on the etherpad and see what we can come up with... 19:11:27 yeah, it's turtles all the way down. we were considering encrypting it to the keys of the infra root admins or with a passphrase known only to infra root and leave it at that 19:11:59 IMO, encrypt the backups w/ GPG to all of infra-root's keys. Anyone in infra-root then can restore the backup on their machine. 19:12:07 but anyway, that's veering a bit off topic for validating our current backup mechanism 19:12:14 ohai 19:12:17 agree, just make sure it gets backed up 19:12:21 +1 19:12:21 yeah 19:13:00 #topic Specs approval 19:13:08 #link https://review.openstack.org/#/q/status:open+project:openstack-infra/infra-specs,n,z 19:13:26 we're not approving any specs in meeting, just a reminder that there are some awaiting reviews 19:13:51 also the new storyboard dev team has explicitly requested review/approval of some outstanding storyboard specs 19:14:29 I didn't add it to the agenda, but this translations site spec is ready for review: https://review.openstack.org/#/c/184559/ 19:14:44 I've been working with the i18n folks to refine it, and I think it's in a good place now 19:15:02 since openstack project infra don't have a huge stake in it other than avoiding breaking it until we switch to something else, would be nice for us to give the storyboard team relatively wide berth on their specs i guess 19:15:05 The StoryBoard spec in question is https://review.openstack.org/#/c/202989/ 19:15:22 #link https://review.openstack.org/202989 19:15:25 thanks SotK! 19:15:35 and apologies for us being somewhat absentee landlords on that 19:15:59 well didn't we say see how it goes? 19:16:11 we did 19:16:14 yep. they seem to be doing stuff 19:16:18 I am guessing this is and indication that infra isn't really interested in that? 19:16:35 potentially, yes 19:17:10 anyway, specs. we have some. i need to review them, and would be thrilled if others do to 19:17:45 #topic Schedule Project Renames 19:18:18 it was suggested in the aforementioned planning thread that we should probably also do some renames in september for projects that have already requested them 19:18:52 we don't need to decide it today, but just making sure we keep it in mind and see if we can pick a good window when september draws a little closer 19:19:52 #topic Priority Efforts 19:19:54 any initial suggestions? 19:19:56 oh too late 19:20:04 #undo 19:20:05 Removing item from minutes: 19:20:16 it's not too late! 19:20:31 well played 19:20:43 i didn't have any initial suggestions though, but my calendar is wide open for september 19:20:48 there is the long weekend for MURICA we should avoid but otherwise maybe avoid burning man? I think that is next to the long weekend 19:21:05 * fungi has done a great job at avoiding burning man so far 19:21:08 after labor day I'm around all month 19:21:39 thats basically the first week of september is no go 19:21:43 maybe do second weekend? 19:22:18 second weekend... like friday the 11th'/ 19:22:20 ? 19:22:22 ya 19:22:34 11th and 12th are good for me 19:22:54 wfm. i have some guests in town the following weekend 19:23:27 #info tentative window to clear already requested project renames on september 11 or 12 19:23:31 are we planning or have plans to move any infra projects around? like from sourceforge to openstack or openstack-infra? 19:23:53 s/ource/tack/ 19:24:02 not that i'm aware, no 19:25:18 okay, nothing else on this i guess, so moving on 19:25:24 (for real this time!) 19:25:29 #topic Priority Efforts 19:26:02 if there aren't any urgent updates for any of these, i'm going to move ahead to the backlog 19:26:09 jhesketh has a first pass at the swift index gen stuff 19:26:15 anyone have a priority effort that's blocked/on fire? 19:26:15 so swift logs continues to plod along 19:26:36 gerrit upgrade might be blocked forever. 19:26:48 zaro: oh? details appreciated... link? 19:26:53 i'd like to request more reviews for several items on downstream-puppet 19:26:57 nobody is working to fix the jgit issue #link https://groups.google.com/forum/#!msg/repo-discuss/CYYoHfDxCfA/BxkgOOx6CgAJ 19:27:05 zaro: ugh 19:27:10 I think upstream gerrit may have given up on functioning properly 19:27:25 ETOOHARD 19:27:34 Yeah swift logs are slowly going. Not as fast as I'd like sorry 19:28:22 no worries 19:28:26 okay, moving along to the backlog 19:28:35 #topic Restore from backup test (jeblair) 19:28:45 this was mostly covered in the action items 19:29:04 we should keep working through it, but i don't know that it needs any additional discussion here for the time being 19:29:23 #topic Fedora 22 snapshots and / or DIBs feedback (pabelanger) 19:29:32 looks like pabelanger's not here 19:29:42 I believe he can be summoned 19:29:56 maybe not 19:30:03 we should be close to having dib images going 19:30:13 hpcloud don't have f22 images, rax does 19:30:19 yeah, i've been semi-reviewing the proposed changes 19:30:21 opps 19:30:23 i'll link them here 19:30:25 if someone from hpcloud wants to fix that, great 19:30:26 sorry, lost track of time 19:30:32 #link https://review.openstack.org/211703 19:30:39 #link https://review.openstack.org/211294 19:30:45 #link https://review.openstack.org/186619 19:31:05 devstack is broken on f22 at the moment due to a pip issues, hope to have something on that today .au time 19:31:21 a real fix + workaround in the mean-time probably 19:31:23 okay, great 19:31:39 anything else specific you needed to cover for this in the meeting? 19:31:58 will we drop f21? 19:32:08 sort of more priority effort, but request for review of the yum/dnf package caching for dib ... getting links 19:32:38 #link https://review.openstack.org/208321 19:32:42 phrased differently, should we expect the current jobs that run on f21 to work on f22 obce we have it booting? 19:33:03 er, once 19:33:26 my suggestion is to unify apt/yum paths to minimise our need to pass different env variables 19:33:56 whatcha mean "unify apt/yum paths"? 19:34:17 sorry, i mean in that review use common environment variables to indicate "keep the package cache" 19:34:28 oh, right-o 19:34:36 +1 19:34:37 as to the jobs, yes for devstack+tempest 19:34:43 within dib's baked-in elements you mean 19:35:03 i *think* there might be some docker-ish type stuff i know little about. i imagine it can transition 19:35:16 because it definitely has some widely varied caching behaviors/implementation between the dpkg element and the yum element 19:35:36 yeah, that's what i've tried to unify with those changes 19:36:10 okay, cool 19:36:18 anything else on this? 19:36:29 anyway, i think that it's all pretty contained within the reviews, so that's it 19:36:40 f21 deprecation? 19:36:50 don't see that answered 19:37:09 ianw naswered said yes for devstack + tempet 19:37:13 which is most of where we f21 19:37:17 ahh, right-o 19:37:22 the devstack job is super-unstable. i haven't bothered fixing it because f22 is so close 19:37:35 so it's half-dead already :) 19:37:38 the puppet crowd i guess are the other primary consumers of those images? 19:37:49 and kolla/magnum 19:37:54 tripleo have their own images, so can transition at their own pace i suppose 19:38:12 yep; pabelanger & me will work with people to transition 19:38:21 alright, great 19:38:36 if anyone ever asked me, i've always told them the cost of testing on fedora is they're going to be on the upgrade cycle 19:39:07 yeah, that's why we've generally not had important jobs run on non-long-term releases 19:39:28 yep, so it's fair to expect people to put some effort into moving 19:39:28 since we can't continue to maintain them in that state for our stable release branches 19:39:55 #topic Normally there are no infra-roots in Europe, some of us want to volunteer for it (yolanda) 19:40:13 hi so the topic raised that week on some gerrit outage 19:40:42 we know we have jhesketh on Australia, and Sergey on Europe, but reality is that sometimes is difficult to have roots at certain hours 19:40:52 good idea, if you don't plan to sleep in the next 6-12 months 19:40:52 yolanda: thanks for being interested in helping more! general workflow for this is to sync up with the ptl and let them drive the conversation, so probably best to bend jeblair's ear when he's no longer in the midst of the ops mid-cycle 19:41:05 AJaeger raised the problem that there were no volunteers, so from Gozer we'd like to volunteer 19:41:23 yes, we'll contact jeblair for it 19:41:26 yolanda: i think that's awesome 19:41:29 +1 for EU infra root next to Sergey 19:41:52 we are really loosing entire work days when something fails 19:41:59 we are 3 members on Europe that are able to help, and have the expertise, so we'll be contacting jeblair for it 19:42:27 yep, anyway, lets let this conversation take its course through the proper channels when jeblair can be around 19:42:43 sure 19:42:55 i just wanted to go ahead and bring it forward since you had it on the agenda. don't want you to think i'm skipping you! 19:43:04 #topic puppet-pip vs puppet-python (rcarrillocruz) 19:43:21 this got some discussion on the mailing list, though i think it might have reached a lull 19:43:48 #link http://lists.openstack.org/pipermail/openstack-infra/2015-August/003015.html 19:44:38 looks like it was covered in last week's meeting, so we can just kick that thread and see if we can bring it to a resolution i guess 19:45:02 #topic Nodepool REST API spec (rcarrillocruz) 19:45:14 this also looks like it's left over from last week 19:45:23 yep 19:45:28 was there anything else that needed covering on this one? 19:45:28 I haven't had a chance to review the spec and leave comments 19:45:42 yep, please review open specs ;) 19:45:51 * fungi is one of the worst offenders 19:46:22 #topic Open discussion 19:46:34 i give you 13.5 minutes 19:46:41 i wanted to raise the zuul services problem, we had some discussion on the mailing list 19:46:47 with several opinions 19:47:10 i'd like that we get on some agreement 19:47:20 which list? infra or dev? 19:47:47 or was it before august? 19:47:58 infra, we had the discussion last week 19:48:00 not immediately finding the thread to gain some context 19:48:08 about if puppet modules should spin up services or not 19:48:22 oh, that 19:48:27 #link http://lists.openstack.org/pipermail/openstack-infra/2015-August/003038.html 19:48:38 ah, i didn't have the link handy 19:48:50 fungi clarkb i am good from a kolla perspective with f21 deprecation in 1-2 weeks 19:49:08 we want f22 though :) 19:49:15 sdake: awesome, well we need the f22 images working anyway, yeah ;) 19:49:59 yolanda: so i think we mostly settled on it being fine to parameterize the service ensure for running/stopped/undef 19:50:03 in 1-2 weeks I will be removing our fedora-21 gating job entirely 19:50:12 fungi, that works for me 19:50:25 as soon as my gating work finishes and merges for the new kolla build tool 19:50:41 i think someone (crinkle? nibalizer? clarkb?) said ensure=>undef will have the desired outcome for our infra 19:50:50 fungi but for f22, that gating job is furthe rout and not an immediate priority 19:51:12 ensure => undef is really annoying 19:51:12 yes 19:51:18 it behaves differently in puppet3 and puppet4 19:51:19 ya my last message on that thread is what I think we should do 19:51:26 but if thats the hack I guess ok? 19:51:41 clarkb: I don't think it will behave differently? 19:51:46 clarkb: it does? 19:51:53 crinkle: using undef as a param makes it use the dfeault 19:52:01 in puppet3 but in 4 its actually a nil value aiui 19:52:15 so if there is a default for the thing you get the default which is probably not what you want if explicitly undefing 19:52:24 there's no default for ensure in the service type, is why not specifying it has worked so far 19:52:57 oh I thought it defaulted to running 19:53:13 if it defaulted to running then we wouldn't be arguing about adding ensure => running 19:53:13 but you must be right since its just dropped 19:53:15 ya 19:53:18 and yeah, i feel relatively strongly that the way we run our services in the openstack project infrastructure should be the default behavior for the modules we publish, because anything else is hypocrisy, but it's perfectly fine to make the behavior configurable so that downstream consumers can choose to override that default to whatever behavior they prefer 19:53:42 fungi: ++ 19:54:24 puppet-lint voting jobs for system-config? What was decided? 19:54:32 re-enable or not? 19:54:45 but if the people responsible for maintaining zuul.openstack.org feel that ensure-running for the zuul services is a liability, then it feels wrong for us to make that a non-default behavior for the openstackinfra/zuul module on the forge 19:55:09 fungi: agreed we should keep the current behavior as default for all the reaons argued for on the thread 19:55:20 I am infavor of expoising something to allow people to control it, does not matter the name or defaults. 19:56:16 i agree with pabelanger 19:56:42 parameterize , be flexible and not opinionated on it, and i'm perfectly fine on having the default non running for infra 19:56:53 pabelanger: i think we have a consensus to not -1 over style concerns, but the current core reviewers for the system-config repo didn't express an interest in having that repo linted since it's not going to be published to the forge 19:57:03 well if you put it that way I think we _should_ be opinionated :P 19:57:14 we know a lot about zuul 19:57:18 we know a lot about running zuul 19:57:22 we should share that knoweldge 19:57:40 we have https://review.openstack.org/#/c/175226/1 to add linting, I think we should get that rebased and merged 19:58:44 oh, i hadn't notived that one. so we need linting to give us forewarning of things we're currently doing in system-config that might break under puppet 4? 19:58:49 er, noticed 19:59:04 #link https://review.openstack.org/175226 19:59:18 anyway, we're running out of time 19:59:24 anything else in these last few seconds? 19:59:41 no 19:59:50 thanks everybody! 19:59:55 #endmeeting