21:00:32 <russellb> #startmeeting nova 21:00:33 <openstack> Meeting started Thu Dec 19 21:00:32 2013 UTC and is due to finish in 60 minutes. The chair is russellb. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:36 <openstack> The meeting name has been set to 'nova' 21:00:39 <beagles> o/ 21:00:43 <russellb> hello, everyone! 21:00:49 <alaski> o/ 21:00:51 <n0ano> o/ 21:00:53 <bpokorny> o/ 21:01:00 <dansmith> \o 21:01:01 <russellb> #link https://wiki.openstack.org/wiki/Meetings/Nova 21:01:03 <hartsocks> \o 21:01:14 <russellb> #topic general 21:01:17 <russellb> first, a meeting about meetings 21:01:29 <russellb> let's skip next week, i'm sure many people will be out anyway 21:01:42 <russellb> once we get back, we'll start a new schedule where we alternate each week 21:01:57 <russellb> hoping to offer more people a chance to join in every other week 21:02:01 <russellb> wiki page has the times 21:02:11 <russellb> we'll do 2100 UTC when we first start back 21:02:15 <russellb> then 1400 UTC after that, then alternate 21:02:18 <russellb> we'll see how it goes ... 21:02:44 <russellb> alternative approach as suggested by ttx is to keep this every week, and occasionally (monthly?) hold other time slots to check in with other time zones 21:02:58 <russellb> but seems a number of folks are happy to try the alternating bit first 21:03:27 <russellb> that was the only general thing, on to the real stuff 21:03:29 <russellb> #topic sub-teams 21:03:46 * hartsocks waves 21:03:48 <n0ano> o/ scheduler 21:03:59 <russellb> melwitt had a conflict, but sent this for novaclient 21:04:02 <russellb> <melwitt> open bugs, 122 !fix released, 78 !fix committed and !fix released 21:04:02 <russellb> <melwitt> 20 new bugs 21:04:02 <russellb> <melwitt> 0 high bugs 21:04:02 <russellb> <melwitt> 46 patches up, 6 WIP, increasing activity, number of patches continually increasing 21:04:20 <russellb> so, new bug count steady, but a growing review queue 21:04:34 <russellb> so, don't forget to take a look at those when doing nova reviews 21:04:49 <russellb> n0ano: you're up 21:05:21 <n0ano> no_db scheduler - got slowed down by devstack failure, that should be resolved and they can make some more progress soon... 21:06:08 <russellb> cool, haven't had a chance to look at those patches yet myself 21:06:15 <n0ano> gantt (code forklift) - new tree up, I'm in the process of trying to get it to pass Jenkins so we can actually submit patches (lots of detailed name changes to go throught) 21:06:28 <russellb> #link http://git.openstack.org/cgit/openstack/gantt 21:06:35 <russellb> n0ano: so one thing we could do is switch all jobs to non-voting 21:06:45 <russellb> while we work to land a series of changes to get jobs to pass 21:06:54 <russellb> instead of having to do one giant "make it work" commit 21:07:07 <russellb> if "make it work" is getting really big anyway 21:07:10 <n0ano> russellb, yeah but I'd at least like to get things like tox -epep8 to work first 21:07:30 <russellb> "make it pass" i mean 21:07:32 <russellb> not necessarily work 21:08:12 <n0ano> anyway, going forward my idea is I'll keep the gantt tree in sync with any scheduler changes to the nova tree, hopefully not too many changes until we can cut over 21:08:23 <russellb> related: please add openstack/gantt to your review queue queries. right now gantt-core == nova-core + lifeless 21:08:30 <russellb> since he was helping bootstrap the thing 21:08:43 <russellb> n0ano: great 21:08:49 <lifeless> russellb: ack, think it's there already but I'll double check 21:09:02 <russellb> we're going to have to revisit the deprecation plan once it's working 21:09:08 <russellb> we sort of punted that discussion on openstack-dev 21:09:11 <russellb> pending it working 21:09:21 <lifeless> as soon as gantt is passing tests I'm going to ping the list and the volunteers to do the actual work 21:09:29 <russellb> cool 21:09:38 <n0ano> my goal is to get it working as soon as possible, then we can start some serious discussions about futures 21:09:50 <russellb> though making the repo and getting all the tests to pass is quite a bit of actual work itself :) 21:09:57 <russellb> but probably mostly a one person thing for now 21:10:00 <russellb> or a small group anyway 21:10:08 <russellb> n0ano: sounds great to me 21:10:17 * n0ano is getting exhausted already :-) 21:10:28 <russellb> hope you're having some fun though 21:10:33 <russellb> it's exciting bootstrapping something new 21:10:44 <n0ano> that's about it for now, we've cancelled the next 2 meetings, start up again in Jan. 21:10:55 * n0ano fun, yeah that's it, fun 21:11:04 <russellb> alright, thanks! 21:11:09 <russellb> let me know what i can do along the way 21:11:12 <russellb> hartsocks: you're up 21:11:16 <hartsocks> hey 21:11:25 <hartsocks> Just wanted to update on the Minesweeper. 21:11:35 <russellb> ooh 21:11:39 <hartsocks> We're still on track for Jan 26th 21:11:42 <russellb> still love that name 21:11:53 <russellb> on track for? 21:12:01 <russellb> running against all changes? 21:12:02 <hartsocks> Our current plan is to have the Minesweeper vote +1 and 0 21:12:13 <hartsocks> on changes in our driver dir and above. 21:12:25 <jog0> above? 21:12:32 <hartsocks> So that's all patches except for patches affecting the other hypervisors. 21:12:46 <jog0> neato 21:13:04 <hartsocks> Yeah. So since in the tree the other hypervisors are siblings… you know… test all above and around. 21:13:05 <russellb> so ... everything except something that only changes nova/virt/someotherdriver/*.py (and tests) ? 21:13:17 <hartsocks> yeah. Not sure how we *could* test that. 21:13:29 <russellb> it's just like changing anything else 21:13:33 <dansmith> well, 21:13:41 <russellb> testing anything else i mean 21:13:43 <dansmith> yeah, it's just a patch that you run and make sure everything is cool 21:13:53 <russellb> you're just running against that state of the nova tree 21:14:02 <hartsocks> right. 21:14:23 <dansmith> we've had cross-driver dependent code before, which isn't cool, but... 21:14:42 <hartsocks> So the current logic is, if it only touches the other drivers then we're not going to be interested… or qualified to test it anyway. 21:15:10 <dansmith> and the logic should be: it's a patch against nova, so you run your tests against it 21:15:24 <russellb> another benefit ... running tests more often helps find bugs that only surface some of the time 21:15:26 <dansmith> if it only changes virt/libvirt code, then it should pass 21:15:32 <dansmith> yeah 21:15:40 <hartsocks> Well, we wouldn't be exercising a libvirt patch anyhow. 21:15:47 <dansmith> right 21:15:47 <hartsocks> It would just be a noop. 21:15:49 <dansmith> yep 21:15:53 <hartsocks> So we're ignoring them. 21:15:54 <russellb> noop in a perfect world 21:15:57 <hartsocks> But… 21:16:10 <russellb> in the gate world today, we're running "no-ops" over and over and over, but we find bugs that way 21:16:15 <hartsocks> if it touches libvirt *and* some other code in the tree outside that… then yeah we have to test. 21:16:52 <russellb> if it's a matter of you can keep up without those changes, but can keep up filtering them out, that may be OK short-term 21:17:00 <russellb> long term, running against everything seems best overall 21:17:14 <russellb> i didn't say that right 21:17:19 <hartsocks> Okay. This is why I wanted to bring it up. It didn't seem right to have Minesweeper voting on other hypervisors since it can't really say anything about them anyway. 21:17:37 <dansmith> it absolutely should be voting on every patch, IMHO 21:17:38 <russellb> sure, and i appreciate it 21:17:48 <russellb> just trying to share feedback 21:17:51 <hartsocks> I'll push the feeback back to the testing guys. 21:17:52 <dansmith> https://wiki.openstack.org/wiki/HypervisorSupportMatrix/DeprecationPlan 21:17:56 <russellb> you guys are still ahead of the pack on all of this :) 21:18:00 <jog0> ++ to vote on all patches but babysteps are good too 21:18:01 <dansmith> says: " feedback is being reported on every proposed change to nova," 21:18:05 <russellb> jog0: ++ 21:18:15 <hartsocks> our testing guys rock and I'm honored to be their receptionist in IRC. 21:18:20 <hartsocks> :-) 21:18:23 <russellb> heh 21:18:36 <hartsocks> The other bit I *might mention is* 21:18:43 <hartsocks> https://blueprints.launchpad.net/oslo/+spec/vmware-api 21:19:01 <hartsocks> But that's a longer discussion and not really necessarily for IceHouse. 21:19:24 <hartsocks> That's all I wanted to bring up this session. 21:19:26 <russellb> so here's a question 21:19:38 <russellb> would this vmware python library be useful outside of openstack? 21:19:47 <russellb> or do you really see it as openstack specific 21:20:00 <russellb> just something to think about WRT that blueprint 21:20:07 <lifeless> how many machines did you end up needing to keep up? 21:20:16 <hartsocks> That *is* a good question… *and* what we *should* be contributing is only the bits useful for OpenStack. 21:20:18 <lifeless> thats useful data for other orgs, I think 21:21:23 <hartsocks> lifeless: I don't have that stat in front of me, it changed recently. 21:21:45 <hartsocks> lifeless: I can tell you we "burned out" one set of older infrastructure and had to move to new machines. 21:21:53 <lifeless> hartsocks: I'd like to know how far off the estimates in https://etherpad.openstack.org/tripleo-test-cluster are basically 21:22:16 <russellb> 10s of servers? 100s? 21:22:18 <hartsocks> lifeless: and I'll deliver a message to the testing guys about that :-) 21:22:21 <lifeless> hartsocks: thanks! 21:22:38 <hartsocks> russellb: part of it is in an internal IaaS thing that makes that hard to answer. 21:22:40 <russellb> heh, anyway, yeah, would love to know more if you can share 21:22:42 <russellb> fair enough 21:23:00 <russellb> ah, it's like you're using a cloud for what it's good for 21:23:12 <hartsocks> russellb: yep. 21:23:13 <russellb> alright, i think that was all the sub-teams that raised hands 21:23:28 <russellb> #topic bugs 21:23:30 <lifeless> russellb: the estimates I have are ~20 concurrent vm's for an exact devstack-gate setup, which is to say 60 cores, 320GB of RAM. mumble. 21:23:36 <lifeless> oh, my fav topic 21:23:41 <lifeless> so I started the thread 21:23:42 <russellb> lifeless: cool 21:23:44 <lifeless> it seemed to peter out 21:23:53 <russellb> definition of triage? 21:24:05 <lifeless> yeah, you asked the q in there that the other thread was about 21:24:09 <lifeless> so I answered it there 21:24:26 <russellb> was there another thread too that i missed? 21:24:32 <lifeless> no 21:24:40 <russellb> ok 21:25:17 <russellb> guess i need to take another look 21:25:24 <russellb> so, we have 8 critical bugs open right now for gate issues 21:25:26 <russellb> https://bugs.launchpad.net/nova/+bugs?search=Search&field.importance=Critical&field.status=New&field.status=Incomplete&field.status=Confirmed&field.status=Triaged&field.status=In+Progress&field.status=Fix+Committed 21:25:35 <russellb> could really use some more eyes on these things 21:25:45 <russellb> we need to help improve reliability, as it's been a little rough lately 21:26:18 <russellb> notmyname came up with a nice graph that shows probability of patches to pass the gate ... http://not.mn/gate_status.html 21:26:30 <russellb> just to help demonstrate the importance of looking at these bugs 21:26:47 <jog0> russellb: https://bugs.launchpad.net/bugs/1253896 21:26:49 <uvirtbot> Launchpad bug 1253896 in neutron "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,In progress] 21:26:50 <russellb> not *all* of that is openstack bugs .... some of it is external events (sphinx API, etc) 21:26:54 <jog0> is a msassive one 21:27:09 <russellb> is it just with neutron? 21:27:20 <russellb> not trying to pass it off on the neutron team 21:27:20 <jog0> no, confirming now though 21:27:40 <jog0> AFAIK it was a nova-network issue 21:27:44 <russellb> ok 21:27:57 <jog0> hmm it my be neutron 21:28:24 <jog0> yeah its neutron. but neutron == nuetron+nova 21:28:29 <russellb> has anyone set up an env to reproduce? 21:28:32 <russellb> right 21:28:47 <russellb> i'd like a devstack VM I can log in to, run the test that triggers it over and over and see it happen 21:28:48 <sdague> the neutron team has been banging on that one a lot 21:28:50 <russellb> and then poke at the env 21:29:04 <russellb> i know i could make that, just haven't yet :) 21:29:06 <sdague> they found a bunch of races, which made it better, but there are still issues falling out 21:30:19 <russellb> sorry, looking at it 21:31:10 <russellb> well, if anyone has some time to put in, eyes on that one seems to be the most important 21:31:17 <markmcclain> yeah.. we've found and fixed a bunch of other items while digging into this one 21:31:22 <russellb> markmcclain: nice 21:31:53 <jog0> markmcclain: any nova patches from neutron folks we need to look at 21:32:08 <russellb> so one thing we talked about earlier this week is, at what point do we have to limit other dev activities to force more focus on the gate bugs 21:32:18 <russellb> i'm not sure what the answer is, and would just prefer we don't let it get to that point :) 21:32:33 <markmcclain> jog0: none that I know of 21:32:52 <russellb> markmcclain: please ping me, or #openstack-nova, if any come up 21:32:59 <markmcclain> russellb: will do 21:33:19 <russellb> jog0: any other bugs worthy of special attention today you think? 21:33:33 <jog0> https://bugs.launchpad.net/nova/+bug/1254890 21:33:35 <uvirtbot> Launchpad bug 1254890 in tempest ""Timed out waiting for thing" causes tempest-dsvm-neutron-* failures" [Low,Confirmed] 21:34:07 <jog0> https://bugs.launchpad.net/tempest/+bug/1258682 21:34:10 <uvirtbot> Launchpad bug 1258682 in tempest "timeout causing gate-tempest-dsvm-full to fail" [Undecided,Invalid] 21:34:26 <jog0> and it would be good to try to rule out nova on https://bugs.launchpad.net/tempest/+bug/1253896 21:34:30 <uvirtbot> Launchpad bug 1253896 in neutron "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,In progress] 21:35:23 <russellb> OK, cool, thanks for the pointers 21:35:28 <russellb> we can come back to these in open discussion if we have time 21:35:30 <russellb> #topic blueprints 21:35:39 <jog0> those are the top 4 bugs on http://status.openstack.org/elastic-recheck/ 21:35:46 <russellb> #link https://launchpad.net/nova/+milestone/icehouse-2 21:36:01 <russellb> i posted to the ML last week and said that today is the deadline to have blueprints approved for icehouse-2 21:36:12 <russellb> and after today, we'll be pushing the rest off to icehouse-3 21:36:27 <russellb> for any that are waiting on review (in Pending Approval or New), I'd like to review them before deferral, though 21:36:30 <russellb> so I could use some help on that part 21:36:55 <russellb> unfortunately that status doesn't show up there, so it's easier to find those on this page: https://blueprints.launchpad.net/nova/icehouse 21:37:09 <russellb> any specific blueprints anyone wants to discuss today? 21:37:42 <hartsocks> I have https://blueprints.launchpad.net/nova/+spec/autowsdl-repair I'm trying to finish tonight. It's only low priority though. 21:38:29 <russellb> OK, looks like it's probably fine 21:38:35 <russellb> but will review before deferring 21:39:04 <russellb> looks like we have 11 that need review 21:40:01 <russellb> #topic open discussion 21:40:26 <russellb> anything else? 21:41:12 <russellb> alright, thanks everyeone! we'll meet again in 2 weeks 21:41:17 <russellb> #endmeeting