19:02:48 #startmeeting infra 19:02:49 Meeting started Tue Feb 4 19:02:48 2014 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:50 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:52 The meeting name has been set to 'infra' 19:03:07 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:03:15 o/ 19:03:19 #topic Actions from last meeting 19:03:51 #link http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-01-28-19.01.html 19:03:52 lifeless: sure.. 19:04:22 mrmartin work with smarcet to get through infra processes 19:04:26 that's been happening 19:04:35 #link https://review.openstack.org/#/c/69636/4 19:04:49 mordred find a way to run manage-projects automagically without puppet 19:05:02 updates on that? anything newer than in the bug? 19:05:10 fungi: can you _very_ quickly update me on why that's necessary? 19:05:12 is it in the bug? 19:05:18 he has been sick, I have been talking with UtahDave about salting it 19:05:28 then UtahDave has been sick 19:05:31 no code yet 19:05:48 jeblair: seems the issue we're encountering only arises when manage-projects is called from puppet agent 19:05:59 we have a 100% success rate when run by hand 19:06:00 fungi: is it more complicated than the 10min exec timeout? 19:06:21 jeblair: mordred had suggested punting and working around the puppet call with an additional layer 19:06:30 jeblair: yes, it actually does create the projects and some of the groups but not all of the groups 19:06:41 which indicates the timeout isn't being hit, but weirdness is happening 19:06:52 o/ 19:07:10 clarkb: does it log? 19:07:52 jeblair: I think so but not with the -v flag when run by puppet iirc 19:08:41 one of my suggestions in the bug was to either figure out why the exec stdout/stderr capture in puppet isn't working as advertised, or to add a python logging config for manage-projects, but i haven't foung time to look into either of those 19:08:51 okay, well, this idea that we can't figure out what this simple program is doing is way too voodoo for me.... 19:09:11 jeblair: fungi: puppet execs are weird and voodoo like 19:09:15 they are documented as such 19:09:24 i would like manage-projects to get proper logging and log what it's doing so we can see what's actually happening 19:09:25 and the recommendation from puppetlabs is to not use them 19:09:31 but I am all for better logging 19:09:45 clarkb: do they fail at calling fork/exec? 19:09:50 jeblair: yes 19:10:08 well they blame ruby, but yes 19:10:10 clarkb: how does anything in puppet work? 19:10:31 i mean we exec stuff _all the time_ 19:10:42 are they not closing fd's properly or something? 19:10:49 no it has to do with environment stuff 19:11:47 okay, so basically we're not anywhere near being able to automatically create new projects 19:12:30 should we resume the moratorium on approving new projects? 19:12:45 right. mordred has resorted to manually running manage-projects after new project configs merge 19:13:13 and we merged a puppet config change to disable the exec temporarily 19:13:18 ok 19:13:31 we should capture more of this into the bug 19:13:35 #link https://launchpad.net/bugs/1242569 19:13:36 so mordred is in charge of semi-manual new project creation until that bug is fixed 19:14:15 #action mordred continue looking into bug 1242569 19:14:21 if that's the case, that works for me 19:15:04 i do agree that "let's just not use puppet" seems like a bit of a kluge 19:15:28 but maybe not, without a more detailed proposal it's hard to say 19:15:31 yes. i think we need proper logging regardless of how it's invoked. 19:15:35 sorry, I guess I mis-understood then, I thought that was where we were last week 19:15:52 i'm just plain embarrassed that we're like "it's broken and we don't know why because we don't know what the computer is doing". 19:16:22 i concur 19:16:44 i've dragged this on enough though; please continue 19:16:53 we have an agenda item for this, so we can pick it up again in a bit if needed 19:17:06 clarkb upgrade jenkins.o.o and jenkins01 to 1.543 and upgrade zmq plugin and scp plugin everywhere 19:17:12 that all happened, correct? 19:17:23 that is all done but the zmq bit ebcause there is an outstanding review to fix the master name reportig 19:17:33 so the action can be updated to: 19:17:48 #action clarkb upgrade zmq plugin everywhere to report master name in events 19:17:58 oh, i missed that the master name patch was buggy. have a link to the fix? 19:18:16 #link https://review.openstack.org/#/c/69763/ 19:18:42 fungi move graphite whisper files to faster volume 19:18:56 jeblair actually moved these to a cinder volume 19:19:14 fungi: oh do we want to move that from spinning to ssd? 19:19:41 we should keep tabs on performance and see whether it improves before pvmoving it to an ssd volume 19:19:51 but should be trivial now that it's lvm 19:20:05 fungi: ok. also, that host would probably benefit from a move to performance flavor 19:20:12 which should also be easier now :) 19:20:38 #action fungi see if graphite whisper files need to be moved to a faster pv or faster vm 19:20:59 (since i fell down on getting it taken care of so far, it's only fair) 19:21:15 also didn't get around to... 19:21:18 #action fungi prune obsolete whisper files automatically on graphite server 19:21:28 that's hard because some metrics don't report very often 19:21:41 is once a month sane there? 19:21:46 once a quarter? 19:21:58 how often is not very often? 19:21:59 fungi: i'm thinking some release jobs only happen twice a year 19:22:05 ahh, yep 19:22:09 ick 19:22:25 so the easy answer is prune after 1 year; more frequently becomes tricky. 19:22:30 i can start by seeing if we have any whisper files untouched for a year 19:22:44 if we've had a graphite server that long 19:22:58 anteaya assist mordred in automating manage-projects again 19:22:59 could probably do like 8 months or so 19:23:14 we can probalby also manually prune using a list generated by find 19:23:21 a lot of things will be obviously obsolete 19:23:32 i guess that ties back in with the earlier bug 1242569 discussion 19:23:35 clarkb: yeah, with some thinking we could identify metrics that zuul/nodepool don't use anymore. 19:24:08 anteaya: were you still working with mordred on bug 1242569? 19:24:26 I had it on my radar 19:24:37 as of yet, no progress 19:24:46 he was sick 19:24:48 should we keep that as an action item for next week? 19:24:55 may as well 19:25:03 #action anteaya assist mordred in automating manage-projects again 19:25:08 thanks 19:25:17 push for improved logging first to see whether it's necessary 19:25:37 i know this hasn't happened yet... 19:25:38 k 19:25:41 #action mordred to lift virtualenv 1.10.1 pin when we're ready to babysit it 19:25:52 yes, that is my name on the logging bug 19:26:02 zaro to point zuul-dev at gerrit-dev 19:26:08 zaro: that happened right? 19:26:09 done 19:26:13 awesome! 19:26:32 hey guys, I have something to discuss if possible 19:26:55 dguitarbite: is it on https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting ? 19:27:05 if not, i can try to work it in 19:27:31 zaro to review jeepyb integration with new gerrit and update gerritlib for gerrit 2.8 19:27:39 i know i saw some work on that as well 19:27:42 done. 19:27:46 that's continuing, right? 19:27:50 oh, even better 19:27:53 it's done. 19:27:57 great 19:28:12 fixes have been pushed to review.oo 19:28:45 #linkhttps://review.openstack.org/#/c/69800 19:28:56 fix gerrit hooks - #link https://review.openstack.org/#/c/69791 19:29:08 fix replicate command - #link https://review.openstack.org/#/c/69768 19:30:46 fungi: thanks 19:30:46 ok. nothing more on that testing. 19:31:36 ohh just wanted to mention that tested the following scripts with gerrit 2.8: expire_old_reviews.py, manage_projects.py, update_blueprint.py, update_bug.py, welcome_message.py, openstackwatch.py 19:32:02 anything major worth reporting on that real quick, or just want to plug the etherpad issues list? 19:32:02 #link https://etherpad.openstack.org/p/gerrit-2.8-upgrade 19:32:02 okay, so we've got half the meeting slot left. i'll try to blow through topics which we covered in the action items 19:32:02 also, my irc client's lag timer keeps creeping up, no idea whether freenode is still under a ddos, but i'll add clarkb as an optional chair in case i vanish 19:32:25 #chair clarkb 19:32:26 Current chairs: clarkb fungi 19:32:44 #topic Trove testing (mordred, hub_cap, SlickNik) 19:33:05 looks like only mordred is in channel 19:33:18 we can come back to this one at the end of they show up 19:33:26 that was the case last week. ++ to shifting 19:33:32 and several weeks 19:33:41 wow meetbot is sluggish 19:33:43 is there a bug to track it? maybe we pull from agenda for now 19:33:54 hello 19:34:04 howdy SlickNik 19:34:24 any quick updates on "Trove testing" 19:34:28 ? 19:35:06 I have a patch out to tempest (basic trove flavor API tests) waiting on reviews. 19:35:10 one sec 19:35:24 #link https://review.openstack.org/#/c/69501/ 19:35:38 I'm working on adding more tests 19:36:02 Also the gate job has been set up and is running on jenkins. 19:36:08 It's non-voting as of now. 19:36:33 okay, great 19:36:40 that's some awesome progress 19:36:50 yaaay! 19:37:17 since we use trove for some of our infrastrucutre (and soon hopefully much more) i'm glad it's being tested ;) 19:37:41 moving on quickly... 19:37:45 #topic Tripleo testing (lifeless, pleia2, fungi) 19:38:16 we've had some setbacks with nodepool bugs/tripleo-ci outages but all in all things are progressing 19:38:46 yeah, continuing to work on getting fedora in the gate too (helps tripleo and some other things we want to do) 19:39:29 anything in particular we should be aware of beyond that for now? 19:39:42 I don't think so, gritty details otherwise were covered in tripleo meeting 19:40:59 as an aside, i've got puppet agent temporarily disabled on nodepool.openstack.org until the tripleo-ci provider is back up and running, which i gather will probably be some time later today. for now it's been deleted out of nodepool.yaml since it was bringing new bugs to the surface we haven't had time to dig into yet 19:41:10 I do think we may need to consider reverting some of those nodepool changes though. Should debug the slow deletes first, but performance is worse than before 19:41:17 thanks for staying on top of that, fungi 19:41:26 which is why all the tripleo jobs are in a pending state in zuul's status 19:42:26 #topic Requested StackForge project rename (fungi, clarkb, zhiwei) 19:42:52 no update on this... it's still pending hearing back from zhiwei but we'll probably also be moving some oslo libs out of stackforge to openstack 19:43:00 so we can try to time those together 19:43:11 and savanna if they choose a new name 19:43:11 oh neato 19:43:12 also maybe savanna renames if they work out what their new name is 19:43:18 what clarkb said 19:43:26 this item is on the agenda twice, because I didn't notice that until the meeting started 19:43:38 dhellmann: no worries 19:43:38 fungi, yup, we're searching new name atm 19:43:57 and we're up to 4 libraries moving into oslo, not just the 3 mentioned on the agenda :-) 19:43:59 we'll hit it at that point on the agenda 19:44:05 or now is fine 19:44:08 SergeyLukjanov: `cat /dev/urandom | tr -dc '[:alpha:]' | head -c6` 19:44:20 dhellmann: and you have a change up for that already too, right? 19:44:20 fungi, my estimate is 2-3 weeks 19:44:22 fungi: either works, I don't have much to say unless there are questions 19:44:33 fungi: yes 19:44:35 #link https://review.openstack.org/#/c/70435/ 19:44:35 clarkb, :) 19:44:46 savanna is definitely out though? 19:44:50 #topic Oslo adopting some stackforge libs (dhellmann) 19:44:57 since we seem to be doing that 19:45:06 cliff, stevedore, taskflow and... 19:45:10 pycadf 19:45:18 excellent 19:45:39 dhellmann: is that an asap move, or do you need some lead time/. 19:45:41 ? 19:45:49 fungi: we're ready when you all are 19:46:07 there's no particular hurry, but sooner is better, so I think I'd rather not wait for savanna to pick a name if that's the only thing holding us up 19:46:57 dhellmann, agreed, it could take up to a month for savanna 19:46:57 i think i'm free this weekend, but would appreciate at least one other infra core on hand in case we get into the weeds 19:47:20 fungi: I am maybe free this weekend 19:47:37 no plans yet but the planning of possibilities has started... 19:47:52 I have a family thing this weekend, or I'd hang out with you guys -- I'm not sure I'll be much more than moral support, though :-) 19:48:32 dhellmann: well, we'll have .gitreview file patches which you'll want someone to merge 19:48:44 we could do it friday afternoon... 19:48:52 i'm cool with that 19:49:02 friday works for me 19:49:16 activity seems to be mostly dead on friday afternoons north-american time 19:49:29 cool, and it's just a few mins of downtime 19:49:43 yep 19:49:59 what do we think... 20:00 utc friday then? 19:50:19 that's noon pst i think. go later? 19:51:10 I'm in EST, so that time is great for me, but later works too 19:51:22 that time or later wfm; your pick 19:51:26 yeah, i'm in est as well 19:51:57 okay, let's do 20:00 utc. good compromise without pushing pst friday lunch too far out 19:52:35 I'll be there 19:52:44 ++ 19:52:45 #agreed will rename new oslo projects on friday 2014-02-07 at 20:00 utc 19:52:47 sounds good 19:53:12 pulling some topics out of order for the last few minutes 19:53:21 #topic Request for a new branch of openstack/operations-guide for ongoing O'Reilly work (annegentle) 19:53:31 annegentle: i can do this--just get up with me after the meeting 19:53:58 #topic Storyboard update (jeblair) 19:54:14 oh neat! 19:54:28 so we had a really good meeting; see the etherpad for what we captured 19:54:40 should have _basic_ functionality real soon 19:54:42 i would have said by wednesday 19:54:47 but krotscheck_sick is sick 19:54:56 so it could be later than that 19:55:10 said he was going on vacation too right? 19:55:22 hopefully with the basic arch in place, it'll be easier for others to pitch in and cargo-cult forms and stuff. :) 19:55:23 fungi: sounds good 19:55:34 zaro: yes, he has vacation scheduled after his illness 19:55:53 going on vacation sick is such a waste of a vacation. he should just work instead ;) 19:55:57 anyway, once that's done we can start using it to dogfood storyboard and for infra projects 19:56:13 it will lack _many_ features, but we can start adding them quickly 19:56:36 this is all under the "minimum viable product" on the etherpad 19:56:50 the lack of features will drive us to add what we need, i suspect 19:57:20 good for itch-scratching 19:57:25 fungi: that's the plan. To psychologically compel contributions with super bare features. :) 19:57:47 gothicmindfood: have the etherpad link handy? 19:58:02 https://etherpad.openstack.org/p/StoryboardMeetup 19:58:06 #link https://etherpad.openstack.org/p/StoryboardMeetup 19:58:10 https://etherpad.openstack.org/p/StoryboardMeetup 19:58:18 [eot] 19:58:19 ha. jeblair and ttx were quicker 19:58:28 years of training 19:58:28 we all owe each other cokes. 19:58:46 * fungi is so very excited 19:58:58 oh 19:59:10 ttx: sorry, I have to run a short errand mid-meeting (C->kindy) - bbs 19:59:10 and fwiw - I am currently putting together an etherpad set of stories related exclusively to Storyboard authentication 19:59:22 https://etherpad.openstack.org/p/StoryboardAuth 19:59:31 also, http://storyboard.openstack.org/#!/ is in puppet and should do things like upgrade (including db migrations) as needed automatically 19:59:31 I even have nice diagrams in powerpoint form 19:59:44 #action jeblair get ssl cert for storyboard 19:59:54 very productive 20:00:00 sarob: on the moodle app integration for training manuals thing (dguitarbite seems to have vanished), we can talk over in #openstack-infra if you like 20:00:10 we're at time 20:00:14 #endmeeting