19:02:10 #startmeeting infra 19:02:11 Meeting started Tue Apr 12 19:02:10 2016 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:14 The meeting name has been set to 'infra' 19:02:17 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:22 #topic Announcements 19:02:27 #info Tomorrow (April 13th) the Infra team will hold a bug day 19:02:29 #link http://lists.openstack.org/pipermail/openstack-infra/2016-April/004124.html April 13th Bug Day Announcement 19:02:40 * nibalizer hands out hammers 19:02:43 #topic Actions from last meeting 19:02:48 #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-04-05-19.02.html 19:02:54 #link http://lists.openstack.org/pipermail/openstack-infra/2016-April/004124.html pleia2 to announce small-scale/impromptu infra bug squash on apr 13 19:02:59 o/ 19:03:02 thanks for organizing that, pleia2! 19:03:05 \o/ 19:03:11 #topic Specs approval 19:03:13 yes thanks 19:03:18 none new this week 19:03:26 #topic Priority Efforts 19:03:28 no critical/blocking updates this week, focus is mostly on post-release activities and upcoming summit 19:03:29 o/ 19:03:40 #topic Proposal jobs (AJaeger) 19:03:42 He states: "We currently have some reviews for extra proposal jobs, what are our polices for adding them? Those run on the proposal jobs and thus we have to review for security:" 19:03:50 #link https://review.openstack.org/301375 Implement periodic job to update Puppet OpenStack constraints 19:03:51 #link https://review.openstack.org/267941 Update stable constraints url 19:03:53 #link https://review.openstack.org/291514 Add periodic job to autogenerate grenade plugins list 19:03:55 #link https://review.openstack.org/291517 Add periodic job to autogenerate tempest plugins list 19:04:07 o/ 19:04:09 this is something we can also discuss in Austin if desired. 19:04:25 yeesh, Austin is coming up quick 19:04:27 it looks like our insistence that having jobs propose autogenerated commits back into projects being a terrible model is increasingly popular in spite our concerns 19:04:45 indeed 19:04:57 or rather, our insistence is not doing much good 19:05:02 to convince anybody else 19:05:08 I wonder what our policy should be for these. I think we merged similar jobs already - and I'd like to especially review "my" translatoion jobs with somebody, so put this on the etherpad 19:05:11 pleia2: ++ 19:05:32 when files hold data and git holds the files, then something like this is the only way to modify it programatically 19:05:40 so far I've not reviewed any of these changes and will abstain reviewing... 19:05:43 o/ 19:05:52 I'd have more head space for this chat in austin personally 19:05:54 (abstain voting) 19:05:57 well, as you adeptly noted, any scripts being run on proposal.slave.openstack.org need to be closely analyzed because they could be used to expose credentials 19:06:19 and some of these scripts are in other repositories... 19:06:21 and we have to make sure they don't run code from arbitrary locations 19:06:33 yes, especially not from other repositories 19:06:34 o/ 19:06:49 AJaeger: yeah that is a bad choice it needs to be infra-controlled repo not project controlled 19:07:02 i think requiring the code to be in the project-config repo is reasonable 19:07:18 many things will need to be changed then 19:07:37 including requirements repo setup;) 19:07:45 the credentials on that worker are... luckily not super highly sensitive since all they can do is propose changes which people might fail to look as closely at, but we should still take care not to put them at risk 19:08:10 fungi: yeah, i was thinknig the same thing -- they aren't *that* interesting; otoh, they would let you propose a "translation update" to nova 19:08:27 and it'd be pretty easy to slip something in there 19:08:28 * morgan is trying to think of a better alternative to offer folks vs many many more proposal bot things. 19:08:29 backdoor^H^H^H^H^H^H^H^Htranslation update to nova 19:08:59 o/ 19:09:08 there's the option to fan this out into multiple individual accounts so as not to cross privilege boundaries, but that's complex 19:09:20 what's a grenade plugins list? 19:09:37 fungi: ++ at least isolated from the translations and requirements update account 19:09:43 like, why is that something that needs to be kept in a git repo? 19:09:51 (it sounds more like something that could be a cron job on a web server) 19:10:02 fungi: that have a lot of history of being safe to +2/+a with little in dpeth checking 19:10:17 jeblair: http://docs.openstack.org/developer/devstack/plugin-registry.html for grenade 19:10:42 the other side of this is not exposure of creds but just the fact that this is complicated automation and it's already challenging to keep it from running haywire even when people experienced in this end of things are the ones reviewing/approving changes to it 19:10:54 mtreinish: we could generate that list also dynamically at publish time, couldn't we? 19:10:56 jeblair: it should live in tree so people can build things locally 19:11:03 that is a good depiction of the other side of this 19:11:49 AJaeger: iirc, someone was told not to do that because the proposal bot is the openstack way of doing things 19:11:59 o/ 19:12:23 a job which could propose changes to gerrit and inadvertently (due to a misplaced while instead of a for or whatever) started shoving thousands of reviews in could do cause some major headaches 19:13:08 AJaeger: it walks all opentstack/* trees and takes about 20 minutes 19:13:18 Clint: yikes. we keep trying to tell people that's a terrible solution, it just happens to have been a tolerable fit for translations and, later, requirements synchronization 19:13:34 ianw: does it use http to git.o.o? 19:13:58 jeblair: yes; it wasn't going to, but then the git trees aren't cached on the proposal slave 19:14:02 the app-catalog has a preiodic job that detects dead links 19:14:11 fungi: i should point out that this is the first day i've heard anyone say this is a terrible solution and there is plenty of prior art 19:14:12 cause that sounds like it should be about 1.2k web requests and take a couple of seconds 19:14:56 Clint: yep, the prior art is the pain point. i'm not sure where is best to document that these jobs are not a model of hos things should be done, but a model of how things ended up being done in a couple of very specific cases wher ethe alternatives were worse 19:15:01 y 19:15:41 one of the problems with documenting how something shouldn't be done is that more folks do it the way you advise not to 19:15:55 because you documented it 19:15:55 we're also self-guilty -- we have a proposal job to clean up the projects.yaml 19:16:55 (which wouldn't be a problem if people didn't keep adding things to it ;) 19:16:57 fungi: ya im with Clint in this one 19:17:06 antipattern attractive people 19:17:06 i've reccomended people set up periodic jobs that propose before 19:17:06 well, the underlying concern with the model is threefold: 1. tricky credentials handling causes us to run them in a constrained environment/conditions, 2. it's a complicated rube-goldberg-esque series of operations (moreso than our usual fare even), and 3. autogenerated content is not something you usually want in your git repositories 19:17:14 all that job does is remove the source: line from gerrit/projects.yaml, I don't think it is a very important job 19:17:20 i thought this is basically the only way to do some of this stuff 19:17:39 Should also note, EmilienM has a patch up also adding code to proposal bot too for puppet. 19:17:45 fungi: it could be worse if it were removed from the repositories (like requirements.txt) only to be generated from some submodule or worse... 19:18:04 i would also say jobs that are rube goldberg stuff in our infra are prefered to random cron jobs out in the world that propose 19:18:06 pabelanger: right https://review.openstack.org/#/c/301375/ 19:18:14 nibalizer: it may be the only way to automatically propose a change to a git repo -- i think the idea being expressed is that automatically generating content in a git repo is not the only way to do some higher level things. 19:18:26 jeblair: ya we put a lot of data in our git repos 19:18:55 i think you once said that when all you have is a hammer everything looks like a git repo 19:18:59 yeah, taking the projects.yaml cleanup job as an example, it would be better to just fix jeepyb/manage-projects to use some other mechanism than a one-time import line in an otherwise durable structured dataset 19:19:17 proposing changes to clean up those lines instead is a workaround 19:19:33 and one that causes ongoing work for people having to approve the changes from it 19:19:55 its also something that is safe to leavein the file... 19:19:55 or just leave the source: line there, it isn't hurting anything 19:20:00 anteaya: +1 19:20:03 yep 19:20:12 fungi, it's also a santiy check that the import was done correctly - which could be done in jeepyb... 19:20:34 i think the only reason the desire to clean it up is there is because it confuses people into thinking it's used indefinitely and needs to be maintained 19:20:40 it's code is also in the project-config repo so isn't our biggest problem. :) 19:21:12 iff we want to scale this, then i think more accounts is the best we can do. 19:21:23 yep, so anyway, i don't think we're going to come up with a blanket policy in the meeting today 19:21:27 it's more upfront work for us, but less ongoing work. 19:21:39 ya lets chat in austin about it 19:21:42 but it's good to get the design concerns/constraints noted so everyone is on the same page 19:21:44 nibalizer: +1 19:21:47 (in that we can say, sure, openstack-foo, go crazy with your creds) 19:21:52 oh - idea: 19:21:53 rather than people getting lots of conflicting messages about this model 19:21:58 pre-generate an account for every project 19:22:01 so to AJaeger's point, do we decline to merge that stuff until we have consensus in austin? 19:22:04 fungi: +1 19:22:10 so we don't need to spend any time doing this one-by-one 19:22:16 \o_ 19:22:26 we can automate the whole account creation/adding creds to puppet 19:22:48 we still have the slave issue, at least until zuulv3 19:22:52 jeblair: ya 19:22:57 slave per project! 19:22:58 jeblair: interesting idea, though yeah we'd need multiple workers for them 19:23:01 so zuul v3 19:23:29 okay, any last minute thoughts on this other than people can try to come up with solutions to the stated issues and/or discuss at the summit? 19:23:56 moving on! 19:23:59 #topic Virtual Machines are provided with inconsistent swap configuration (dmsimard) 19:24:02 #link https://review.openstack.org/300122 Add fix_disk_layout script and builder 19:24:32 dmsimard doesn't seem to be around 19:24:50 I can call him 19:24:58 fwiw, i'm mostly ok with it, just wants a better name 19:24:59 anybody happen to know what he wanted to discuss regarding this? ianw and pabelanger seem to have reviewed the linked change 19:25:07 o/ 19:25:08 ugh, getting memory right on these seems to be a neverending battle 19:25:13 ah, there we go 19:25:14 I have to leave in literally 2 minutes to pick up kid at school 19:25:28 fungi jobs that dont use d-g have swap.issues too 19:25:41 i support the idea that this should run on all images rather than just inside devstack-gate 19:25:43 goal is toaddress across the board 19:25:43 yep, this is true 19:25:50 jeblair: ++ 19:26:05 I think we just wanted to see if it was a dib element or JJB template 19:26:14 i have felt for a while that having a macro to add swap for a job is a good thing, and that we should not set up swap unless it's requested by a job 19:26:15 for more then devstack jobs 19:26:36 I think it is a slave script that jobs can call 19:26:44 I see value is taking that out of devstack-gate and into a more generic approach -- IMO all VMs should be provided equal 19:26:47 the flip side, is there any benefit _outside_ the ci to having the setup tooling where it is now? 19:27:10 for example, do people run devstack-gate and expect it to set up their swap? possibly today, yes 19:27:15 i don't see how we could do this in dib, without some sort of boot script which doesn't sound great. 19:27:29 Isn't the functionality already there for boot scripts? 19:27:33 ianw was suggesting a first boot service script in dib, yes. 19:27:34 bkero no 19:27:45 jeblair: I was thinking, create the swap file, then add something to crontab on @reboot to mount the swap 19:28:06 pabelanger this is what fstab is for 19:28:07 bkero: if we don't know in advance whether the job will need swap, then we can't predict whether to set it up on the worker at build/boot anyway 19:28:16 I have to step out to pick up my kid, sorry, brb. Will read backlog. 19:28:18 we could also start making full disk images complete with partition tables 19:28:20 everybody seems onboard with JJB 19:28:26 if folks want it as a boot script -- ok -- but that sounds unreasonably difficult to maintain. 19:28:26 dmsimard: thanks for bringing it up 19:28:30 dmsimard: travel safely 19:28:31 clarkb: Right 19:28:48 the real issue here is every cloud is different 19:29:03 some have swap some dont and they come in dofferent sizea 19:29:11 clarkb: some of the elements do it, but maybe there isn't a common method 19:29:16 for example growroot 19:29:18 (we have no choice with ip addresses, so we had no other choice than the 3 months we spent getting glean right) 19:29:37 yeah, writing something generic in boot is going to end up not very generic and a bear to maintain i feel 19:29:47 (but if we do this with swap, we'll spend just as long getting it right and waiting for images to build after touching each of the edge cases) 19:30:01 true 19:30:19 mordred: is this an option? 19:30:21 I would prefer jobs that need swap declare so 19:30:29 just like with package deps 19:30:35 also swap is something that _can_ be setup at runtime without needing to reboot, and shouldn't add significant delays (in theory) 19:30:36 jeblair: yes, I believe so 19:30:40 SpamapS: ^^ 19:30:41 and once we are on ext4 again the cost is basically zero 19:30:55 SpamapS: dib can make images with partition tables, right? can those be booted normally by nova? 19:30:59 then we make those jobs stop needing swap 19:31:11 and openstack is better 19:31:37 i have a slight preference for having all nodes have a sensible swap configuration all the time, because i don't believe in servers without swap. but i don't object to the jjb macros. 19:32:12 Seems starting with JJB everybody can get onboard with 19:32:15 right now the only real cost involved in this is dd'ing 8GBof zeros 19:32:26 get rid of xfs and then that goes away 19:32:59 my main concern is that we have a tradeoff on clouds with small rootfs and no additional ephemeral volume such that swap eats into already tight available filesystem space even for jobs which may need disk but don't come anywhere near filling up available ram 19:33:02 jeblair I would be fine with that too 19:33:51 though i suppose an alternative would be to invert the logic and have a macro that dows a swapoff -a and rm /swapfile or whatever before starting the job 19:34:10 fungi: we could also investigate boot-from-volume on small-disk clouds 19:34:19 not saying it's a good idea 19:34:37 but maybe it's an option that's available that we have not explored? 19:34:43 or just to declare that jobs should not expect to have more than (lowest common denominator rootfs minus default swapfile) amount of available space to use 19:34:52 fungi: s/or/and/ 19:35:07 we should at the very least publish the size 19:35:45 fungi less git cache less rootfs its something like 60GB available but easy to co fi 19:35:47 so anyway, in the short term, i don't think we have any significant concerns over 300122 (moving the swap creation script out of devstack-gate), but some think that it could be improved further? 19:35:49 *confirm 19:36:23 more specifically, moving the swap creation script out of devstack-gate and into a jjb macro/slave script 19:36:36 bikeshed over giving it a better name aside 19:36:45 that doesn't really move it out of devstack-gate -- it just makes it available to !devstack-gate 19:36:55 well, yes this step is duplicating it 19:37:18 in theory next patch would be to add that macro to all d-g based jobs and then patch after that could delete it from d-g 19:37:52 and we still have one more (hopefully quick) topic to go before we talk about summit planning 19:37:56 sure. it would be nice to hook it into the reproduce.sh script too 19:38:00 hopefully 19:38:20 all good suggestions for the review 19:38:31 but as far as the general direction sounds like this is a fine path for now? 19:38:59 i'm taking further silence as tacit agreement 19:39:16 #topic timing issues with the ansible rename patch (anteaya) 19:39:18 #link https://review.openstack.org/299192 Rename openstack-ansible-ironic to openstack-ansible-os_ironic 19:39:20 thanks 19:39:32 they're in a hurry because they'd like it renamed before they release? 19:39:47 so I was looking at this rename patch and the commit message had said they want it renamed before they ta 19:39:51 anyone around to accommodate a rename this weekend? 19:40:11 I didn't know until yesterday they want to tag friday or next week 19:40:14 on an unrelated note, i'll be entirely afk friday through sunday (mini-vacation to one of the more remote islands here) 19:40:17 not me (I am swamped with !work currently) 19:40:22 fungi: good for you 19:40:46 I can help prep if someone is available to do the root work and the reindex which we now know takes 4 hours 19:41:04 if we don't rename they are goind to do documentation gymnastics on github as a work around 19:41:09 which I don't entirely understand 19:41:17 yeah, that offline reindex is a killer, and i guess the online reindex doesn't do what we need 19:41:29 it didn't last time we tried no 19:41:48 can we downgrade to 2.8? ;) 19:41:56 :) 19:42:03 might be faster than reindexing 19:42:12 so that's the item, thanks for your time 19:42:14 maybe bigger gerrit server will reindex faster 19:42:35 okay, sounds like we don't have any takers to for a last-minute rename this weekend 19:42:40 thank you 19:42:56 Sorry for the late response. Yes dib can make partitioned images. I don't know, however, if clouds will boot them without extra finagling 19:43:20 #info No Infra root sysadmins are available to volunteer for a Gerrit project rename maintenance/outage on short notice prior to the summit 19:43:42 #topic Design summit session planning 19:43:48 #link https://etherpad.openstack.org/p/infra-newton-summit-planning Infra Newton Summit Planning 19:43:50 #link https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=Infrastructure: Infra Newton Summit Schedule 19:43:52 we have the following workrooms proposed (note we have 2 slots on wednesday, and a pair back to back on thursday which could accommodate a double-length working session): 19:44:15 NA bug fixes (clarkb) 19:44:32 oh, that's not a title ;) 19:44:37 hehe 19:44:42 Make ansible puppet more robust (clarkb) 19:44:45 i am in favor of "not applicable bug fixes" 19:44:52 ha ha ha 19:44:53 xD 19:44:55 A new landing page for contributors (ttx/thingee) 19:45:02 ya can add the ssh problems to the ansible robjstness list 19:45:08 Fix puppet apply test not using the repo local hiera file (clarkb) 19:45:23 Review of jobs on proposal slaves (AJaeger) 19:45:24 nibalizer: ^ are you already working that? 19:45:39 fungi: that is 4 right? 19:45:43 the apply test may not be needed if nibalizer has it sorted 19:45:54 clarkb: ya it works except for f23/puppet 4 19:45:57 4 slots, 4 proposals? 19:46:03 someone suggested last week that mordred's "the interaction between launch-node, ansible and puppet and how group membership works" might be a fit for a work session? 19:46:12 instead of a fishbowl? 19:46:16 jeblair: suggested that 19:46:18 fungi: i doubt that? 19:46:18 we can definitely discuss the landing page somewhere else if you don't have room for that. Just want to make sure we do it right, but we could take that off-DS 19:46:32 i think that fits a hacking session, so my vote on making it work session... 19:46:38 fungi: yes please 19:46:41 but if you do have room, why not take advantage of being there to make progress 19:46:41 rcarrillocruz: which one? 19:46:49 mordred's one 19:46:58 ttx: i'd show up for the new contributor landing page session whether it's infra or not 19:46:59 I think there are two sides 19:47:03 it also sounds like we should have a fishbowl on periodic jobs? 19:47:04 ttx: I like your proposal 19:47:15 there is work to be done on the above ansible/puppet/group topic 19:47:20 ttx: ++ i will be happy to discuss that with you in Conference Room Pub 19:47:30 but also I'm pretty sure almost nobody in infra is on the _same_ page as far as how it currently works or should work 19:47:37 so some discussion is worthwhile I think 19:47:38 or Friday contributor meetup 19:47:42 mordred: ya 19:47:48 i've put the schedule summary times/locations at the top of the etherpad too, which may help us decide what ordering could work 19:47:51 mordred: I'm not on a page with anyone else on that topic, and would like to be 19:47:57 anteaya: ++ 19:48:19 there are two fishbowls on wednesday, one of which is even before the work sessions that day, but both of which are before the work sessions on thursday 19:48:21 fungi: um, with 12 minutes left can you offer us some suggestions? 19:48:28 fungi: should that topic get a fishbowl, I will volunteer to run it 19:48:40 so if we did got for a fishbowl/work session split on something, that would be the ones to aim for 19:48:46 mordred: i'd like to know what those 'pages' are, maybe worth opening a mail thread upfront before summit? 19:48:48 er, did go for 19:48:57 rcarrillocruz: bah. that involves writing email 19:49:02 rcarrillocruz: just kidding, it's a great idea 19:49:33 we also have the option of doing three work session topics and making the back-to-back sessions on thursday a double-length session if one of these is a better fit for a long session and another is unpopular 19:49:37 mordred: i'm open to discuss that pre-work session with some beers, that works too :P 19:49:48 :) 19:50:22 i'm leaning toward taking the four work sessions as proposed, to anteaya's point 19:50:35 the number seems convenient, after all 19:50:57 anyone feel differently? 19:50:59 fungi: though the third one should be fixed beofre the summit 19:51:12 puppet apply local hiera repo? 19:51:19 i mean i have that in review the only thing needed is for pabelanger and to discuss f23 and why it is different 19:51:28 as for fishbowls I think we should talk about: openstackid, the wiki, task tracker and xenial/operating system upgrades 19:51:30 maybe let's scratch that one from the list then 19:51:43 maybe change it to be about periodic/proposals ? 19:51:46 it doesn't sound like the puppet apply testing item would have enough work for a room full of people 19:51:54 and have an active friday meetup for those sessions that didn't get a fishbowl 19:52:07 could also leave it untopic'd and let people work on what they want 19:52:42 nibalizer: well, they can do that regardless. there will be breakout areas if a topic we're hosting is uninteresting 19:52:53 so probably better to have _some_ theme 19:53:07 happy to help 19:53:16 and proposal/periodic job stuff is already one of the proposed 4 19:53:19 you can move mordred's ansible thing into that spot. 19:53:23 oh right, reading 19:53:31 and you can back-to-back it with clarkb's ansible thing :) 19:53:36 indeed we can 19:54:14 ok 19:55:17 that leaves fishbowls as follows: trusty/xenial split for jobs, something about CICD i don't grok, task tracking, jjb 2.x, authentication service, and wiki upgrade 19:55:34 fungi: the cicd has been removed by the author i think 19:55:35 looks like the cicd thing was maybe scratched off the list by the proposer? 19:55:38 yeah 19:55:43 fungi: the second one doesn't have a chair it was offered by someone who won't be in attendance 19:55:46 just not with actual removing having happened :) 19:55:55 yeah I think we can scratch the second item 19:56:07 so we have 5 options with 3 slots, and the two that remain can as anteaya said be good items for sprint breakout 19:56:12 I can remove in a sec 19:56:25 eil397: it's cool, we figured it out ;_ 19:56:27 ;) 19:56:30 eil397: I think it is a good idea for a different venue 19:56:43 eil397: we can discuss more later in channel perhaps 19:56:46 thanks : - ) 19:56:51 of course 19:57:22 can we take the extra fishbowl? 19:57:45 that would give us 4 fishbowls, right? 19:57:46 anteaya: i think they probably got snapped up 19:57:51 fungi: not really 19:57:55 i think i'd bump jjb 2.x, i'm on the fence between wiki and task tracking 19:58:06 ttx: oh? got a spare laying around? 19:58:20 lying around, that is 19:58:25 Proposed one on the etherpad somewhere 19:58:31 (NB from ttx: we can add one extra fishbowl on Thursday afternoon (conflicting with OSC) if needed) 19:58:44 i see, okay 19:58:50 3:10pm or 4:10pm 19:58:51 ttx: thanks, we'll take it 19:58:55 yay 19:58:59 \o/ 19:59:07 ttx: i can work with either of those 19:59:14 3:10 has OSC fishbowl and a QA workroom 19:59:15 though 3:10 is maybe preferred slightly 19:59:27 \o/ 19:59:28 real quick, a reminder that I added a "Other sessions of interest" section at the bottom of the etherpad, our team is getting bigger and so is openstack, if there's anything outside our track we really should be aware of and may not be obvious, it may be helpful to add it there 19:59:30 oh, yeah those might be worse conflicts 19:59:37 4:10 has a Oslo fishbowl and a OSC workroom 19:59:41 pleia2: awesome--thank you! 19:59:42 pleia2: thank you 19:59:48 ttx: let's grab the 4:10 20:00:01 yeah 4:10 sounds better 20:00:03 i think wiki and osc may not have a lot of people-overlap? 20:00:11 thanks everybody! i'll work up the schedule and try to arrange the least-conflicting solutions 20:00:17 jeblair: yep, agreed 20:00:18 thanks ,fungi 20:00:18 fungi: thank you 20:00:28 #endmeeting