19:01:44 #startmeeting infra 19:01:45 Meeting started Tue Sep 2 19:01:44 2014 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:46 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:48 The meeting name has been set to 'infra' 19:01:51 #link agenda https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting 19:01:51 #link last meeting http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-08-26-19.01.html 19:01:52 o/ 19:01:54 o/ 19:02:03 #topic Actions from last meeting 19:02:09 anteaya invite third-party email addrs from gerrit to announce list 19:02:12 done 19:02:12 anteaya: did that happen? 19:02:15 yes 19:02:15 neato 19:02:17 o/ 19:02:22 i should subscribe to those lists now :) 19:02:27 if you like 19:02:36 you are clear to post to announce 19:02:40 ooh, good idea ;) 19:02:43 as is clarkb fungi and SergeyLukjanov 19:02:51 did I miss anyone? 19:03:01 lyz is already an admin 19:03:15 anteaya: what's the uptake like on the announce list? 19:03:19 good 19:03:25 anteaya, do third-party chairs count? 19:03:25 59 I think? 19:03:35 krtaylor: you are already subscribed 19:03:48 65 now 19:03:55 I thought you were saying it was post moderated 19:04:12 much better than request, which has 18 19:04:23 pleia2: yes 19:04:34 krtaylor: let me check 19:04:38 krtaylor: oh, yes it is. i expect the use to be for broadcast messages to third-party operators, most of which would be from infra (announcing changes to the system, etc) 19:04:39 o/ 19:05:03 o/ 19:05:15 krtaylor: i think if you have something to announce, we can still use it for that 19:05:34 (meeting minutes?) 19:05:39 krtaylor: let's get it underway and then look at permissions 19:05:43 (for teh third-party meetings) 19:05:44 krtaylor: just check with one of the moderators to make sure it's appropriate, and/or go ahead and send the message and they can let it through 19:05:53 fungi: possibly 19:05:53 sounds good 19:06:02 fungi: i think low-traffic would be good 19:06:15 agreed 19:06:16 yeah 19:06:18 jeblair: the last item on today's agenda will probably the first post to -announce 19:06:18 fungi: i want something that no one has any excuse to ignore :) 19:06:31 jeblair: right 19:06:33 anteaya: yeah, i think we may be thinking along the same lines :) 19:06:40 good 19:06:45 best chance of success 19:06:46 well, a weekly link to the third-party irc meeting minutes seems relatively low-volume, and makes for a good ping 19:07:04 fungi: I'll consider the thought 19:07:12 let's get the other things in shape first 19:07:13 fungi: hrm. that's the sort of thing that causes me to create a mail filter so that they go into a folder that i never read 19:07:26 jeblair: ahh, yep. fair enough 19:07:33 well, we also post highlights to the weekly newsletter 19:07:43 krtaylor: that is a better place 19:08:05 I'd vote keep it low volume 19:08:18 cool 19:08:19 pleia2 send the email to -infra announcing the new lists/policies after 116989 merges 19:08:24 i think i saw this :) 19:08:34 it happened, yes 19:08:41 o/ 19:09:09 as a followup, are we ready to ask people to take some topics to the request list? 19:09:27 o/ im hello hello 19:09:31 i think we're finishing up some account re-enabling mails that originally went to the infra list 19:09:34 I haven't seen any new requests to redirect yet 19:09:52 and i think it's fine to not be too particular about those 19:09:59 yeah, I've been hoping the current ones just finish up on -infra 19:10:02 yes, and I don't know as any of the account requests outstanding from last week were fulfilled 19:10:05 but probably if we get too many more, we should start asking people to use the new list 19:10:11 +1 19:10:12 yup 19:10:31 I asked SergeyLukjanov in channel but didnt' get an answer 19:10:40 if someone can help me with cleanup after the meeting 19:10:48 okay, we may have another week of overlap then 19:10:53 I've not read the channel logs yet 19:10:56 sorry 19:11:06 anteaya, I'll read it now and answer 19:11:17 SergeyLukjanov: the outstanding account creation requests from last week 19:11:18 I can take a stab at helping with creating accounts if SergeyLukjanov wants help 19:11:21 SergeyLukjanov: thanks 19:11:25 clarkb: thanks 19:11:44 anteaya, oh, sure, I'll cover accounts creation tomorrow 19:11:51 cool, once that's done we can probably at least redirect new account requests 19:11:55 SergeyLukjanov: let me know if there is a group you want me to do 19:11:56 yes 19:11:57 happy to help 19:12:19 #topic Priority Specs (jeblair) 19:12:21 clarkb, thx 19:12:29 #link https://review.openstack.org/#/c/100363/ 19:12:42 this had 2x+2s and got an update 19:12:53 i think it was a minor update, so we can probably get those back pretty quickly 19:13:01 yes, minor update 19:13:19 so consider this a last call. i'll aprv it tomorrow morning assuming it has at least 2x+2s. 19:13:25 I asked for some expantions so the third party folks could see themselves in the mirror 19:13:47 yesterday I learned they don't understand how the current infra work will benefit them 19:13:58 anteaya: oh, thank you for that. 19:14:01 they think it is for infra 19:14:06 this is, i think very important. :) 19:14:12 extremely 19:14:20 well, partially because the commit message wasnt exactly clear to that point 19:14:29 so at least a few people understand now, hopefully the share 19:14:39 maybe see about getting hashar's input too 19:14:43 krtaylor: all that was missing was third-party in the commit message 19:14:47 he's been using our infra probably the longest 19:15:01 apparently if they dont' see third-party they don't see the benefit to them 19:15:24 anteaya, I always like commit messages that say why and to what benefit 19:15:27 yeah, this and the next one are all about modularizing how we run things to make the tools easier for other people to use 19:15:31 downstream consumers apparently doesn't ahve the same ring to it 19:15:53 for instance, we don't all need our own puppet modules for zuul 19:16:15 but we do need our own zuul layout 19:17:14 or, more to the point for this one, we may want to share the list of packages we're installing on build workers without also configuring my ssh key for login :) 19:17:50 moving on to https://review.openstack.org/#/c/99990/ 19:17:54 jeblair: you don't like getting root on mystery boxes? 19:18:01 yea 19:18:02 ha ha ha 19:18:17 i got focused on the tech side and not the big picture 19:18:33 99990 has a lot of positive feedback 19:18:37 I am ready to go on that one 19:18:41 does anyone have a reason i shouldn't +3 this one now? 19:18:45 +3 19:18:49 pleia2: I am on parental leave for 2 weeks but feel free to drop an email, I might be able to process it :] 19:18:54 go for it 19:18:56 jeblair: doet 19:19:00 hashar: congratulations 19:19:14 yay \o/ 19:19:21 hashar: congrats :) 19:19:21 hashar: hopefully the leave is new baby and not sick child 19:19:22 congratulations 19:19:45 hashar: yes, congratulations! :) 19:19:47 (meeting has been hijacked by a 2 days old baby) :D 19:19:57 hehheh, congrats! 19:19:58 oh! a new proto-hashar. awesome 19:19:59 that is their job 19:20:11 hijack all the things 19:20:13 hashar: and that's just the beginning :) 19:20:27 hashar: it will get worse ;) Congratulations! 19:20:40 haha 19:21:07 https://review.openstack.org/#/c/110730/ 19:21:11 jeblair: i think we need an #agreed for harshar's baby 19:22:12 wife + baby are all fine. Meanwhile if there is any change that can use a review please shout :] 19:22:21 #info Congratulations to hashar on his new baby! 19:22:23 hashar: best kind of news 19:22:25 hehe 19:22:38 hashar: dropped you an email, take your time :) 19:23:08 so on https://review.openstack.org/#/c/110730/ i pushed a new patchset this morning that adds some files and drops any reference about puppet templates 19:23:14 and thank you for your kind words everyone! 19:23:33 if the group thinks I could do it 110730 will be a great learning experience for me 19:23:43 jesusaurus pointed out that using puppet templates outside of a module context can be complicated 19:23:49 jeblair: I will rereview 19:23:59 but if the volume of dumb questions to work accomplished is too much, I understand if someone else wants to do it 19:24:10 and none of the files currently under consideration for that repo use templates, so i think it's best to just define it out of scope for now, unless we need to figure something else out later 19:24:31 honestly, most of the files i think are good candidates for the project-config repo are generally not templates anyway, so it should be okay 19:24:39 i think those become todo items for figuring out how to template and configure separately or something 19:24:41 (scripts, zuul layout, jjb, etc) 19:24:51 anteaya: if you want to learn more about puppet, it's a nice opportunity, at least on the review side 19:25:15 I think we'll really need folks reviewing the changes in a test env 19:25:18 pleia2: well I was thinking more about my comfort with jjb and layout.yaml and so on 19:25:22 we might want to put the nodepool config itself in there, which is currently a puppet template. but i think we can change nodepool to put the secrets in another file 19:25:23 ah, ok :) 19:25:46 pleia2: oh well I'm not good at setting up puppet in a test env, so perhaps I should withdraw my offer 19:26:04 I've been working with jedimike to make it easier 19:26:13 jeblair: yeah, those sorts of refactoring, which probably are a one-by-one bug list 19:26:35 pleia2: I believe you have, but whether it actually makes it easier, I will hold me evaluation for a while 19:26:38 but we'll get a huge immediate gain just with the files already in the list 19:26:41 might make for a good set of low-hanging fruit bugs 19:26:54 anteaya: maybe you can be our guinea pig :) 19:26:59 anteaya: also, becoming comfortable with that process would not be time wasted :) 19:27:05 pleia2: I'm willing to try 19:27:08 \o/ 19:27:11 jeblair: agreed the initial list is quite good 19:27:17 i'm running some stats over the entire config repo to see the change frequebcy on individual files, whichi should identify any we might be missing 19:27:30 jeblair: I was hoping for something that would give my confidence a shot in the arm 19:27:30 fungi: excellent idea! 19:27:46 jeblair: yes, I hear that I need to improve my puppety skills 19:28:08 i'll update the list in the spec with any others i find, and a potential todo list for things which are more embedded data lists in puppet manifests or templates which could stand to get split 19:28:30 some of which, i would argue, goes in hiera 19:28:33 though the todo list is probably best left to follow-on specs 19:28:38 or just bugs/stories 19:28:53 nibalizer: quite possibly 19:28:53 at wikimedia we have a jenkins job that let one compile catalog against list of nodes and do the diff of catalogs between the patchset and parent one 19:28:57 that sometime help :D 19:29:27 thats a cool idea! 19:29:42 so that one probably has at least a few days left in review 19:29:43 nibalizer: agreed, but if we cant put templates in the new project then we have to do one or the other, we cant mix hiera and these files 19:30:10 https://review.openstack.org/#/c/110793/ 19:30:14 is docs publishing 19:30:42 i think it's up to date, except we're still a little vague on what we'll do with openstack-manuals 19:31:27 i think there was some general support for doing option 1 -- build all the manuals in the openstack-manuals repo every time 19:31:32 right, we were going to see if Ajaeger1 could quantify it for us 19:32:01 jeblair: I think we can figure out some stuff, once we have the first steps done 19:32:09 clarkb: what exactly to quantify? 19:32:16 Ajaeger1: the cost of building all the docs each time 19:32:24 (in openstack-manuals) 19:32:27 from a cpu time perspective 19:32:37 clarkb: that's not too expensive - compared to what a nova jobs costs right now ;) 19:32:41 clarkb: 30 mins or so 19:33:10 okay, cool. yeah, i think we have enough options with openstack-manuals that we can go ahead and get started on the rest of this and leave a final decision on openstack-manuals until later. 19:33:32 (since at least no one is saying "none of those 3 options will work") 19:33:37 note that we build in parallel... 19:34:17 my feel is that premature optimization is best avoided for now 19:34:25 simplicity first 19:34:43 * Ajaeger1 can easily change the scripts for the simple way and we can optimize from there... 19:35:06 yeah, optimizing where we see clear performance impact later makes sense 19:35:15 cool. so i'll check in on this one again tomorrow, hopefully we can aprv before next meeting 19:35:23 (especially of "optimizing" means "afs") 19:35:28 er, if 19:35:54 fungi: indeed :) 19:35:59 #topic Manila project renaming (fungi, bswartz) 19:36:13 I'd like to have an +1 from annegentle on the docs repo and will ping her 19:36:21 Ajaeger1: okay, we can wait for that 19:36:41 this was a placeholder. bswartz doesn't seem to hang out in irc much. i reached out to vponomaryov earlier today but haven't heard anything 19:36:50 :( 19:37:12 mainly just wanting to be aware that their program application was approved last week, and they have a couple of projects currently in the stackforge namespace 19:37:15 isn't "have some folks on irc" one of the things the tc asks for in a project? 19:37:46 well, i also could have thought to pester people in #openstack-manila. i'll give that a shot after the meeting 19:37:55 fungi: okay, let me know how that goes 19:38:07 anyway, no need to belabor this topic 19:38:23 otherwise, we'll just rename their project and tell them that we totally mentioned it on irc. ;) 19:38:27 ha 19:38:34 #topic Fedora/Centos testing updates (ianw 02-09-2014) 19:38:39 vponomarov is in Russia AFAIK - should be late for her... 19:38:46 hi 19:38:54 a few updates on this 19:39:19 1) the nodepool allocation issues -- i think everyone is happy we got to the bottom of this? 19:39:33 yep, thanks! 19:39:44 there was some talk of alternative allocators, other than "round-robin-at-full-capacity", which is pretty much always 19:39:57 your original analysis was correct (and we came up with a way to reproduce the problem that verifies this) 19:40:00 i'm getting back around to rereviewing the nodepool update stack for those now 19:40:16 is that going to be a blocker, or are people happy with the current behavior? 19:40:17 the sooner we get that implemented the better, i thinkl 19:40:25 i think it's better than waiting a very long time for f20 nodes 19:40:41 (and py33 nodes, and precise nodes, and...) 19:40:50 ianw: i don't think it needs to block; at least, i think we agreed that this approach sounds better than what we have now, and i'm not a fan of changing the goal posts like that 19:41:14 i was thinking that some sort of parallel execution environment might be good longer term for testing changes 19:41:14 yup I think we explored future improvements but they don't need to be done before current improvements go in 19:41:18 ianw: so i'm happy to try this out, and barring unforseen highly negative effects, stick with it for a bit 19:41:35 that is something i will think about 19:41:45 ianw: problem is that we don't have the resources to test at scale 19:41:53 ianw: (i mean, if we had 2x nodes, we'd use them all for real :) 19:42:06 ianw: and small-scale testing is something that most of us should be able to do on our own 19:42:22 ianw: along with large-scale fake testing 19:43:09 so when i'm making a nodepool change and i want to know how it's going to look with 1000 servers, i run it locally with a fake config and tell it to spin up 1000 fake servers 19:43:09 jeblair: yeah, i hadn't thought it through too much. but it's quite scary to deploy any changes in this area, as it seems like anything other than perfection brings down all forward progress for openstack 19:43:41 and when i want to make sure it actually works against a real cloud, i use a real account and have it spin up 1 or 2 real servers 19:44:13 (also, having that config handy is really nice when i want to test a devstack-gate change -- i mean there's a daemon running that keeps making new devstack servers for me :) 19:45:04 ok, i think we can all agree that more testing won't hurt in general :) 19:45:05 ianw: but i agree that it's a problem. i think the best thing we can do is continue to enhance the in-tree testing 19:45:31 since the manual test process is slow and involved; automating that would be great 19:45:35 i also have a couple of minor nodepool changes out, in the agenda 19:45:45 oh before we move on to those 19:45:46 i know that minor reviews are not high on priority list 19:46:19 we should get your first two changes merged soon... 19:46:22 but it would be nice if things got cleared 19:46:27 fungi: mentioned he wantod to review them soon 19:46:39 assuming they merge in a small number of days from now 19:46:47 when should we restart nodepool to pick them up? 19:46:47 yep, i have them up in a tab--just trying to get through the rest of the cloner/puppet integration series first 19:47:03 yup on my list too 19:47:12 i also have reviews out for f20-bare nodes -- is there any philosophical objection to this? 19:47:30 jeblair: considering FF is happening maybe we can shoot for a friday restart? that will let it ramp up more slowly 19:47:31 it requires one change i found to pull in some more recent postgresql puppet modes 19:47:35 modules 19:47:36 i think as soon as there are at least a couple of us on hand to babysit and revert if needed. shouldn't do much more than delay the gate worst case 19:47:37 jeblair: otherwise it iwll just be full speed ahead during FF 19:48:20 clarkb: well, if we do it friday, it should be early friday; i don't want to find problems over the weekend. 19:48:21 and has the chance to significantly increase our test volume/throughput 19:48:43 or rather, other people to find problems, since i won't be around ;) 19:48:50 jeblair: :) agreed 19:49:01 or maybe we can try to get it done early tomorrow? 19:49:05 I can review the changes today 19:49:06 assuming it works, it should not actually be disruptive 19:49:13 jeblair: good point 19:49:17 only if it breaks would it be noticed 19:49:21 and it has been tested :) 19:49:35 ok I am less worried about it now 19:49:44 since we've been using the check pipeline as a barrier to entry for the gate pipeline recently-ish, starvation in check is effectively limiting the rate at which changes can get tacked onto the gate (plus it slows down iteration there in general if devs/reviewers are waiting for test results) 19:50:11 i'm a fan of early tomorrow (where early is that at least one of you are up, not just me) 19:50:24 fungi: that is early :) 19:50:26 okay, so we'll restart nodepool early morning the day after those changes merge 19:50:31 jeblair: sounds good 19:50:50 f20-bare nodes 19:50:56 i'm hoping to finish reviewing the series this evening 19:52:00 jeblair: f20-bare nodes ^ 19:52:12 i don't have a philosophical objection to them (which was your question) 19:52:45 i think it's probably just worth reminding folks that the fedora support cycle is shorter than the openstack support cycle 19:53:02 (as is ubuntu non-lts) 19:53:25 jeblair: yep, anyone i interact with who is interested in f20 i'm making sure to tell them they're signing up to move to f21 19:53:27 so we still need to focus the bulk of our testing on centos/lts, but we can opportunistically test things in master on newer distros 19:53:32 which brings me to centos7 ... 19:53:35 for long term testing 19:53:54 we have rackspace images and an experimental job 19:54:17 my request for hp cloud people to contact me about their support didn't get a response 19:54:28 oh i thought someone said they were handling it 19:54:38 but i found the main person generating upstream centos7 images, who has contacts within hp 19:54:50 yeah, that's what i was remembering 19:54:52 jeblair: sorry, yes, no response from HP as such 19:55:25 anyway, i would like to bring up centos7 there to bridge the gap before d-i-b support is fully deployed 19:55:42 i don't want to complicate the already complex d-i-b deployment with another new architecture at this stage 19:55:51 ianw: are you satisfied with karanbir's response, or would you like me to chase down people at hp for you? 19:56:13 ianw: ++ I think the easiest dib thing is to convert an existing image at a small scale 19:56:17 jeblair: karanbir's response is fine, and i'm working with him now, so all good there 19:56:24 okay good 19:56:27 ianw: rather than mixing new image with dib at least until we are completely dib 19:56:33 or nearly there 19:56:53 i am keeping up centos7 support within d-i-b in the background and as new images are released 19:57:03 https://review.openstack.org/#/c/118246/ 19:57:20 so i hope that this will ease the eventual transition there 19:57:39 cool. i think we may be ready to re-review the dib in config patches 19:57:52 so hopefully we can start exercising that soon 19:57:59 but that is my general plan. i am told hp should have images "soon", so probably expect to see changes from me in that regard soon+a few days :) 19:58:11 sounds good 19:58:23 that is all i have, thanks 19:58:28 #topic The infra-manual is somewhat blocked on the Style Guide (pleia2) 19:59:03 real quick, if anyone has opinions, please dig back into July archives and say so, otherwise I plan to chat with anne about her thoughts 19:59:10 #link http://lists.openstack.org/pipermail/openstack-infra/2014-July/thread.html#1591 19:59:28 this came from a patch which used bold frequently 19:59:35 pleia2 felt it was overused 19:59:43 far too frequently - less is more with bold ;) 19:59:47 we have no consensus on how often to use bold 19:59:50 yeah. i forget the rst syntax frequently 20:00:00 right , so we need it written down 20:00:13 and if using what docs have works, that works for me 20:00:14 so having a style guide and making sure that we start out with a good consistent style will help 20:00:19 * pleia2 nods 20:00:30 (once the doc exists and uses conventions, it's much easier for follow-on edits to keep the same style) 20:00:31 can we use a template like for specs? 20:00:44 and yeah, less is more with bold. :) 20:00:48 hard to template if you look at teh patch offered 20:01:01 and i think we're at time 20:01:07 pleia2: sure happy to give input, we have an entire tagging guide, but RST is quite the non-semantic beast 20:01:18 anteaya: maybe let's just send a reminder to third-party folks about the wiki thing for now 20:01:21 annegent_: thanks, I'll be in touch once I collect thoughts from the team :) 20:01:21 jeblair: and exclamation marks!!! :) 20:01:22 and discuss more next time 20:01:30 jeblair: will do 20:01:42 we're at (over!) time 20:01:47 thanks everyone !!!1one! 20:01:52 #endmeeting