19:02:12 #startmeeting infra 19:02:13 Meeting started Tue Apr 5 19:02:12 2016 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:14 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:16 The meeting name has been set to 'infra' 19:02:17 o/ 19:02:19 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:30 #topic Announcements 19:02:52 o/ 19:02:56 o/ 19:03:06 o/ 19:03:08 #info Reminder, Gerrit server maintenance on review.openstack.org is scheduled for 20:00 UTC on Monday, April 11 19:03:17 #topic Actions from last meeting 19:03:22 #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-03-29-19.03.html 19:03:30 #link https://etherpad.openstack.org/p/gerrit_server_replacement yolanda draft a maintenance plan for the gerrit server replacement 19:03:38 #link http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html 19:03:39 yolanda send maintenance reminder announcement to the mailing list on April 4 19:03:45 er, stray newline 19:03:48 #undo 19:03:48 Removing item from minutes: 19:03:58 #link http://lists.openstack.org/pipermail/openstack-dev/2016-April/091274.html yolanda send maintenance reminder announcement to the mailing list on April 4 19:04:03 so those were both done 19:04:08 yep, done 19:04:08 thanks yolanda 19:04:09 thanks yolanda!!! 19:04:27 #topic Specs approval 19:04:33 so i'd like to get feedback about the plan in the etherpad 19:04:50 yolanda: happy to 19:04:56 thanks 19:05:17 we don't have any new specs up for final council voting on this week 19:05:41 yolanda: it looks good to me, perhaps you can add the actual commands to be executed so folks can copy and paste? 19:05:48 as a reminder, feel free to pester me if there are specs administrivia changes which need approving and aren't themselves specs or substantial changes to approved specs 19:05:49 ./ 19:05:49 fungi, one question on https://review.openstack.org/#/c/284049/ - this one is just a clarification. 19:05:58 fungi: Does it need voting? 19:06:09 It documents how the translation setup was done 19:06:35 Or should I pester you outside of the meeting for this? ;) 19:06:47 fungi, i have a question about when to update dns entries... because we need to test that zuul, git mirrors, etc.. can recognize new gerrit entry 19:06:48 AJaeger: i think we can just review that like a normal change. it seems to be documenting reality, not changing a plan 19:07:12 fungi: correct 19:07:13 AJaeger: i've approved it now 19:07:22 thanks, fungi 19:07:22 have to announce the new Gerrit IP ahead of time for third parties 19:07:32 hashar: we have 19:07:40 corporate world would need to update some firewall rules 19:07:42 anteaya: \O/ 19:07:45 hashar: http://lists.openstack.org/pipermail/openstack-dev/2016-March/088985.html 19:07:49 hashar: yeah, provided a month heads-up 19:07:52 and reminders 19:07:55 posted march 10th 19:08:17 yolanda: i'll double-check your plan but usually it needs to be changed at the start of the maintenance window and we should make sure ttls are low (300s) 19:08:18 hashar: good thinking 19:08:30 anyway, moving on 19:08:30 speaking of gerrit downtime: We have one rename request for a project. Do we want to do it at the same time? 19:08:34 thanks fungi 19:08:46 i'd rather not do the rename with the move 19:08:53 I agree 19:08:53 https://review.openstack.org/299192 - an ansible repo 19:08:57 one thing at a time 19:09:01 yeah, these things should be separate windows 19:09:09 otherwise we complicate roll-back 19:09:28 #topic Priority Efforts 19:09:31 ok, works for me 19:09:46 nothing specific called out on the agenda this week, so skipping this in the interest of available time 19:09:58 #topic Scheduling a StoryBoard bug day and low-hanging-fruit tagging (pleia2) 19:10:05 o/ 19:10:10 zaro: there you are 19:10:11 #link https://storyboard.openstack.org/#!/story/2000534 First low-hanging-fruit tag! 19:10:19 so, I don't want this to turn into a massive discussion (or ignore the great work that craige has done with maniphest) 19:10:41 but storyboard has improved a lot, email support was game-changing for me, and I was hoping we could get some others in the team to start poking at it again 19:10:42 o/ 19:10:55 I am always in favour of that! :D 19:11:05 Zara: bring your lemur! 19:11:07 pleia2: ++ for the moment at least, it's still our bug tracker and any data we put into it should end up in maniphest anywy 19:11:07 pleia2: i turned on mail but haven't gotten any :( 19:11:09 Zara: pleia2: I have tried and failed to get email working 19:11:12 fwiw the kanban board for infra-cloud has made things much easier to track 19:11:13 and as I've been working with it myself, a lot of the bugs are out-dated, the low-hanging-fruit tags we brought over from launchpad (I think?) need some review 19:11:20 Zara: maybe I should talk to you after meeting about it 19:11:26 clarkb: sure 19:11:41 crinkle: yay, have you been updating it? I have lost track myself 19:11:52 (feel free to ping me if smtp debugging is needed) 19:11:54 so my proposal is to have a bug day where we evaluate the bugs and work on our low-hanging-fruit tags 19:11:58 probably after summit 19:12:03 anteaya: i haven't lately since we've been waiting for things on the dc side 19:12:05 pleia2: sounds good to me 19:12:10 in the meantime, perhaps we try and use it more :) 19:12:16 crinkle: understood, glad you find it useful though 19:12:25 pleia2: do you have a suggested date? 19:12:31 anteaya: no 19:12:42 pleia2: i'm in favor of getting back to trying to whittle down our bug count. i think we've been mostly letting things fester for at least a year 19:12:47 I don't see why we need to wait until after summit, unless others do 19:12:51 fungi: yeah 19:13:04 anteaya: I think most folks are super busy 19:13:19 I don't hink after summit will be any different 19:13:47 i also don't see any problem with some people doing something sooner than the summit, but i definitely don't have time to pitch in over the next two weeks before i leave 19:13:53 well, no, but it gives us some time to make space in our calendars :) summit is only a couple weeks away 19:14:00 I would be for doing something next wednesday, April 13 19:14:04 would I be alone? 19:14:27 I will be around and I think its a decent idea 19:14:32 me and SotK should be around, if nothing else :) 19:14:36 we don't need 100% infra team participation to make it successful 19:14:39 go for it, but getting answers out of lots of us next week may be tough if we're needed for input 19:14:47 ok, that's fine 19:14:52 great, so in storyboard channel okay for chat? 19:14:56 wfm 19:14:59 or do we want the sprint channel? 19:15:11 still, there's nothing stopping people who haev available time to devote to bug triage from coordinateing together or even going it alone at different times 19:15:23 I'll chat with zara and SotK about our options for getting a bug list like we did for lp (if that's possible, or makes sense) 19:15:25 my typing just went to pot there 19:16:01 cool 19:16:05 pleia2: do you want to do an email saying clarkb and I and Zara and SotK will be in storyboard next wednesday? 19:16:10 i'm happy we are looking at Storyboard again 19:16:12 :-) 19:16:12 or shall I? 19:16:19 pleia2: it is your idea 19:16:19 anteaya: I'll announce it and see who wants to join in 19:16:19 anteaya: I think the sprint channel would be more appropriate if we are looking at infra bugs 19:16:26 pleia2: awesome thank you 19:16:31 clarkb: ++ sprint or infra. 19:16:34 don't want to clutter the storyboard channel with a bunch of off topic stuff but ya happy to poke at that and try to cleanup 19:16:34 did some people want to announce a small-scale/impromptu infra bug squash a week from tomorrow? 19:16:37 I'll get the sprint channel set up 19:16:38 let's do sprint 19:16:48 very good 19:16:52 * anteaya books the channel 19:16:52 i doubt the sprint channel is much in use this time of the cycle 19:17:30 yeah it is free I'll book it 19:17:42 #action pleia2 to announce small-scale/impromptu infra bug squash on apr 13 19:17:57 #info Interested bug triagers are welcome to an impromptu Infra bug sprint in #openstack-sprint on Wednesday, April 13 19:18:00 that's all, thanks fungi 19:18:12 thanks! 19:18:21 #topic Debian packaging CI (zigo) 19:18:40 (after this topic, i'll dive into summit session discussion phase of the meeting) 19:18:41 Well, what is missing from https://review.openstack.org/#/c/294022/ ? 19:18:58 What can I do to make it approved? 19:19:05 I'd really like to get it going. 19:19:13 That's first. 19:19:22 Does anyone have something to say about it? 19:19:30 o/ 19:19:40 pabelanger: Let me know, by all means! :) 19:19:48 it looks like AJaeger has already reviewed and is in favor 19:20:02 Yup, he's been very helpful. 19:20:08 zigo: it took several iterations to get it in the current state... 19:20:13 Yes. 19:20:17 fungi, I think it's worth a try... 19:20:41 Once concern that I have though, is about branching. 19:20:49 I've been told that by default, we get "master". 19:20:49 I can review today too 19:21:06 This is in opposition to the current workflow. 19:21:11 zigo: you get the change checked out that you use when submitting a change 19:21:43 Oh, so if I change something in debian/mitaka, that's going to be what will be in the repo by default? 19:21:46 Perfect ! :P 19:21:59 zigo, sure. We test what you submit ;) 19:22:06 Or better the CI tests it... 19:22:25 Ok, so let's say we get this CR merged, the next step will be to publish the built artifacts in a Debian repo. 19:22:44 I know how to do it at home, on my laptop, on a server, etc., but no clue how I'll do it on upstream infra. 19:22:55 Will we get a special VM ? 19:23:11 How is it going to work? What would you guys advise? 19:23:44 well, I think that will take more discussion honestly. 19:23:45 zigo: generally you would run a job like that either after the change merges (post pipeline) or after you push a tag for it (tag/pre-release/release pipelines) 19:23:57 especially if you are wanting openstack infra to host them 19:24:05 but first thoughts would be using the AFS mirror 19:24:21 but yes, the design would start with some mock-up of the sequence of steps you take to do it by hand now 19:24:24 AFS ? 19:24:32 afs is an implementation detail here 19:24:34 I agree with pabelanger: This needs some better understanding of CI in general and shouldn't be discussed here. 19:24:40 What means AFS ? 19:24:46 andrew file system, iir 19:24:49 c 19:24:50 http://docs.openstack.org/infra/system-config/afs.html 19:24:51 Ah... 19:25:13 distributed filesystem, to avoid doing sync scripts on mirrors and all 19:25:17 easiest would probably be to generate the packages and push them somewhere like tarballs.o.o, then have another job which indexes them to update a cohesive package repository 19:25:17 that link explains rationale 19:25:21 If I'm given a VM, I can setup something like mini-dinstall or something similar. 19:25:25 but that can be worked out in design phase 19:25:59 I don't want to install a full dak, but there's alternatives to that. 19:26:12 Is it possible to do something like this? 19:26:19 zigo: it would just be another job (or multiple jobs) run on existing workers or some similar to our existing workers, which are built via automation 19:26:22 Can I be given a virtual machine with root access? 19:26:27 zigo, let's do a spec. Write up what you want, give a proper problem statement and then we can chime in and help with the missing pieces. 19:26:52 Well, the point is, I'm not sure what I should ask for, and I need to know what's possible or not. 19:27:20 There's a million way to do things... what do you guys think should be done? 19:27:22 it wouldn't be a system people log into and run things on. it would be a system deployed from a template (either existing for the duration of only one job, or used for multiple jobs but which could be redeployed as needed) 19:27:26 well, since infra-cloud is possible to run custom packages, it might be worth talking about it at the summit 19:27:29 zigo: what do you need to do? Define your problem first and leave blanks to fill in - including questions. 19:27:31 since there is some overlap 19:27:33 there are no manual steps involved on our job workers 19:27:57 so far, reprepro has been working well for us 19:28:10 pabelanger: From my past experience, the summit doesn't solve issues, we will *all* be very busy. 19:28:37 I need to get all built packages published on each merge. 19:28:50 Available to all the new jobs. 19:29:16 Fedora and SUSE have their own build services for this - and Debian has as well AFAIK. 19:29:28 So, the .deb, .changes, .orig.tar.xz, etc. would need to be pushed to a filesystem, then we run a script to scan all packages of a given OpenStack release. 19:29:37 and other projects like delorean too 19:29:38 i think that's doable, but step #1 is getting packages built. consuming them in other jobs is a separate step. best to work on one phase of this project at a time 19:29:43 What you propose sounds like reimplementing such a service in Jenkins and I'm not sure that's the best way. 19:29:45 Ideally, you want packages published on each patch set 19:29:51 I got the scripts to generate the Packages.gz files and all already. 19:30:15 fungi: I don't want to wait, it's taking already A WAY too much time. 19:30:16 zigo: i understand. but generating the package repository indices is the _easy_ part 19:30:18 So that you can use them to run tests anywhere 19:30:51 fungi: IMO, it'd be best to think about next step already right now. 19:30:53 zigo: also your impatience is causing you to try to plan too many steps ahead, which means you get caught up in things you aren't even sure how will work yet 19:31:28 fungi: I don't agree I'm lacking patience. 19:32:03 designing something like this to work in a completely hand-off automated ci system is completely different than writing some scripts you can run locally on your own machine 19:32:11 fungi: The project entered the big tent last summer. 19:32:46 Ok, moving on, is there someone volunteering to work on the design with me? 19:32:53 And by the way, where's Monty? :) 19:33:27 on a plane 19:33:38 ok 19:33:40 or probably in a security line headed to a plane 19:34:40 fungi: What do you advise, is AFS a good idea? Do we have that already available in the current infra? 19:35:01 zigo: afs is an implementation detail, as i said. best not to get confused by that right now 19:35:28 fungi: So what should I get working on then? 19:35:42 tying this into local package mirrors in our providers is likely one of the last steps to be concerned with 19:36:33 zigo: making sure the jobs you have to test building packages do so successfully, with results you can inspect to be able to troubleshoot when things go wrong there 19:36:37 igorbelikov: What's your view? 19:37:01 zigo: then work on a job which can build packages after merge and upload them somewhere durable 19:37:22 the steps after that will depend somewhat on the design you end up with for those previous steps 19:37:32 anything else on this topic that needs the full attention of the entire infra team (minus those on internet-less airplanes)? 19:37:34 zigo: I agree with fungi here. Let’s get the first steps going 19:37:43 fungi: What I was asking is: where is this "durable somewhere" and how will it work... 19:38:04 Ok. 19:38:13 So probably we can move on... 19:38:16 Thanks fungi. 19:38:38 zigo: thanks, and let's continue this with interested participants in #openstack-infra 19:38:51 #topic Design summit session planning 19:39:24 #link https://etherpad.openstack.org/p/infra-newton-summit-planning Newton Summit Planning 19:39:53 i haven't transcribed the days/times for our slots into this yet 19:40:34 #link https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=Infrastructure: Summit Session Days/Times 19:41:21 looks like we've got at least a few fishbowls proposed, but so far only one workroom 19:41:32 you have the colon but my clicking on it lost the colon 19:41:40 weird 19:41:41 so glad I know to use the colon 19:41:53 yeah 19:42:02 ; 19:42:17 can any of the fishbowls be workrooms? 19:42:19 #undo 19:42:20 Removing item from minutes: 19:42:26 #link https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=Infrastructure%3A Summit Session Days/Times 19:42:30 better? 19:42:51 yep 19:42:53 anteaya: some less popular fishbowls can move to workrooms 19:42:59 yes 19:43:00 thank you 19:43:09 the main limitations there are space and seating arrangement 19:43:16 i could see mordred's groups fishbowl working as a workroom 19:43:37 ++ 19:44:06 that works for me, mordred's thing as a workroom 19:44:19 I am sure I can come up with other workroom ideas 19:44:31 the ansible one has frustrated me semi recently so I threw it up there 19:44:45 I also don't want to be running half the sessions :) 19:45:04 ha ha ha 19:45:36 the ansible one looks like a good workroom topic 19:45:53 also i'm going to need to squeeze in newton priorities planning somewhere. it could be a fishbowl though we're likely to get a lot of random (potentially distracting/irrelevant) input. on the other hand we could do a workroom but having it on the schedule may just draw the same crowd and see them all try to cram into a tiny space 19:46:17 last time interested core reviewers broke away and found a quiet corner during the sprint day, which worked out pretty well 19:46:23 would it make sense ot have it on the friday? 19:46:36 Having it adhoc on the friday is likely to work best 19:46:54 is anyone interested in the proposed JJB API fishbowl session? 19:47:11 i think i'm leaning that way again. though we'll be hard-pressed to find as awesome a teahouse as we did in that courtyard in tokyo 19:47:33 fungi: yes I don't think austin teahouses are nearly as awesome 19:47:58 i hear they put barbecued meat in the tea cups. so... depends on your preferences i suppose 19:48:09 yeah different 19:48:16 lol 19:48:43 waynr: I think some folks will be, though I personally don't have a lot of experience with the JJB API 19:49:46 so the item on line 40 19:49:49 there's not much API to speak of in the sense of "this is really usable outside of the JJB command line context" 19:50:00 this was proposed by someone who won't be at the summit 19:50:12 so looks like for fishbowls we have xenial upgrade planning. there's both the ci aspects of this for openstack itself, and also generally upgrading our servers. the latter can probably be deferred i think since we already mostly agreed not to plan for moving our services to xenial until ocata, if even that soon 19:50:20 so if anyone is interested in it, it will need a leader 19:50:48 fungi: the issue is we are still alrgely precise 19:50:59 fungi: so we need to either trusty first or jump straight toe xenial 19:51:06 and do it all before april next year 19:51:14 yep, so upgrading our services to trusty might be a more useful discussion to have at this point than upgrading them to xenial 19:51:25 Should we plan on a session for that? 19:51:34 I thin kwe can use the same session and just refine it a bit more 19:51:41 fungi: i agree 19:51:59 I have a thing? 19:52:10 * mordred is about to take off on a plane ... sorry for mssing things so far 19:52:11 mordred: someone signed you up for a thing 19:52:22 i worry about mixing topics in that session. planning how to lay out our ci transition to test newton changes for openstack on xenial is likely a solid slot on its own 19:52:28 mordred: the ansible groups thing 19:52:33 fungi: we should at minimal discuss upgrading our services to ubuntu-trusty 19:53:11 jeblair: ah - neat. yes. workroom 19:53:16 clarkb: do you think the job transition to xenial would be a good (maybe double) workroom topic instead? 19:53:32 we could just, you know, try to "get it done" there 19:53:48 that seems like a reasonable use of a workroom 19:53:52 fungi: maybe, the only problem with that is if things go horribly wrong its hard to set it straight when outside of the workroom 19:53:56 I supose you just revert in that case 19:53:59 well, our ubuntu xenial mirrors are online now, so that work could start anytime now :D 19:54:33 clarkb: maybe we plan to get as far as having the jobs ready for cut-over immediately posty-summit 19:54:41 er, post-summit 19:54:57 right before we get on a plane ;) 19:55:09 well, maybe not _that_ immediate 19:55:14 ha ha ha 19:55:38 my main concern with making it a workroom topic though is whether we can parallelize enough of that work to be able to engage 10+ contributors on it for an hour-ish 19:55:53 that's the tough part 19:56:11 fungi: we can probably get people testing different aspects 19:56:26 python unittests, doc builds, sdisting, and push changes for each one as they go? 19:56:34 we should go into it iwth a plan for how we handle the branch switching though 19:56:37 since that is messy 19:57:02 i'm unconvinced as to whether we need a summit session for the trusty upgrades, though i suppose it could consist of trying to reach consensus on a sequence and approximate timeline for rebuilding various servers, as well as a status check as to where we're already at with it 19:57:06 3 minutes remaining 19:57:12 thanks anteaya 19:57:42 clarkb: i suppose we could double-whammy xenial testing by doing design in a fishbowl wednesday and implementation in a workroom thursday 19:57:50 but comment still remains, we should work on upgrading precise nodes to trusty first... since they have lagged this long 19:57:54 but that might be monopolizing too much of our allotment 19:58:03 s/nodes/services 19:58:04 I think being in sync with operating system plans and expectations would be a great way to start this release 19:58:12 fungi: ya and may make us go crazy 19:58:19 fungi: we could use sprint day for that if we wanted to split 19:58:41 also, just to set expectations for today's meeting, my goal was mainly to discuss proposals and get ideas. we have until the end of next week to finalize our schedule 19:58:47 (btw, i think infra-cloud sprint is a great idea) 19:58:51 awesome 19:59:00 clarkb: i think that's an excellent idea for one sprint group 19:59:30 and yes, looks like timing will be ripe (hopefully!) to be able to infra-cloud either on sprint day or in a workroom (or both!) 19:59:31 (mostly because sprinting on operating a cloud *at the summit* with lots of openstack devs could be really cool) 19:59:46 agreed 19:59:47 as long as hp doesn't hit any delays with the remaining cabling/routing anyway 19:59:56 fingers crossed 20:00:14 okay, we're out of time today. add/refine topics on the etherpad for next week when we'll reach final decisions and i can schedule after that 20:00:19 thanks everybody! 20:00:25 #endmeeting