21:00:52 <russellb> #startmeeting nova 21:00:53 <openstack> Meeting started Thu Jan 16 21:00:52 2014 UTC and is due to finish in 60 minutes. The chair is russellb. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:54 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:57 <openstack> The meeting name has been set to 'nova' 21:01:00 <russellb> hello, everyone! 21:01:06 <n0ano> o/ 21:01:08 <melwitt> hi 21:01:13 <dripton> hi 21:01:17 <geekinutah> ola! 21:01:20 <hartsocks> \o 21:01:37 <russellb> #topic general 21:01:45 <russellb> nova meetup coming up soon! 21:01:47 <russellb> #link https://wiki.openstack.org/wiki/Nova/IcehouseCycleMeetup 21:01:51 <russellb> details on that page 21:02:06 <russellb> a more recent addition was the etherpad where we'll collect some discussion ideas: https://etherpad.openstack.org/p/nova-icehouse-mid-cycle-meetup-items 21:02:48 <russellb> general idea for the format right now is ... reserve some time for group discussion/presentation, unconference style ... and at least half the time reserved for just working on things together 21:03:02 <russellb> whether that's hacking code, reviewing code, reviewing blueprints, talking through tough problems, etc ... 21:03:06 <russellb> should be good. 21:03:25 <russellb> #topic sub-teams 21:03:32 * hartsocks waves 21:03:37 * n0ano gantt 21:03:38 <russellb> who wants to give a report and discuss work in sub-teams? raise hand 21:03:43 <russellb> hartsocks: you're up first 21:03:45 * melwitt novaclient 21:03:54 * hartsocks waits for slow typists 21:04:00 <russellb> actually real quick 21:04:06 <russellb> i should have mentioned in general a quick schedule note ... 21:04:09 <russellb> icehosue-2 is next week 21:04:15 <russellb> code freeze for that is tueseday 21:04:24 <russellb> so you have < 1 week to get code merged for icehouse-2 21:04:25 <russellb> that is all on that 21:04:28 <russellb> hartsocks: ok now you can go 21:04:43 <hartsocks> Okay, so we're dealing with performance issues on Minesweeper. There's several BP and bugs in flight now on how to get the Minesweeper to sweep faster. 21:04:50 <hartsocks> That's our big deal. 21:05:00 <russellb> so what kind of performance 21:05:07 <russellb> how long test runs are taking basically? 21:05:51 <hartsocks> okay. 21:06:01 <hartsocks> so we're seeing 4 hours w/o parallel 21:06:15 <hartsocks> and 2 hours with 21:06:29 <russellb> parallel with what concurrency 21:06:30 <hartsocks> the parallelization stuff causes problems but we're working on it. 21:06:48 <russellb> cool, sounds like it's helping improve quality :) 21:06:50 <russellb> the goal after all 21:07:10 <hartsocks> IIRC: we're looking at 4 … but that might have changed since my last look. 21:07:18 <russellb> OK, just curious 21:07:27 <russellb> devstack-gate jobs currently run with 4 21:07:36 <russellb> (but it's a bit too high, we'll talk about that later) 21:07:59 <hartsocks> so that's the big news for us. 21:08:00 <russellb> thanks for the updates! 21:08:12 <hartsocks> I spammed the ML with more details on reveiws 21:08:32 <russellb> OK, if you #link it in the meeting itll be in the minutes 21:08:41 <russellb> but we'll move on for now 21:08:41 <hartsocks> #link https://etherpad.openstack.org/p/vmware-subteam-icehouse-2 21:08:45 <russellb> perfect 21:08:49 * hartsocks bows out 21:09:06 <russellb> will have to check later to make sure it shows up if you do it and not me ... 21:09:14 <russellb> n0ano: hi! 21:09:17 <russellb> so, scheduler stuff. 21:09:30 <n0ano> scheduler - never heard of it :-) 21:09:37 <russellb> sounds like the plan isn't too clear or agreed upon this week 21:09:56 <n0ano> flurry of emails on that but I think we can bottom out 21:10:08 <russellb> few different ways we can improve it i think ... 21:10:13 <russellb> what are your thoughts? 21:10:20 <n0ano> looks like the idea of continuous update vs. the nova tree is not in favor, which is OK 21:10:53 <n0ano> the top idea now is we use the current gantt snapshot and get it working, unit tests and full integration... 21:11:06 <russellb> OK, that sounds pretty good to me 21:11:14 <russellb> though depending on timing, integration may not work 21:11:18 <russellb> if it's too far behind nova 21:11:35 <n0ano> after that we get all the patches that were needed, re-create the gantt tree from the current nova tree and then update the gantt tree to work again 21:11:46 <russellb> ok, sounds perfect 21:11:53 <russellb> if we know we have a path to a gantt that works quite quickly 21:12:03 <russellb> then i'm much more open to freezing nova-scheduler at that point 21:12:13 <n0ano> we should be OK, the current snapshot was a working nova tree so getting it to pass integration test should be possible 21:12:32 <n0ano> agreed, I don't want to freeze anything until we have a working gantt, no point in stopping development 21:12:55 <n0ano> I'll send out an email that details this new plan and make sure everyone agrees to it 21:13:01 <russellb> sounds good 21:13:19 <russellb> depending on how that comes along, it's also worth considering planning the re-sync after feature freeze 21:13:25 <russellb> that timing would work nicely, i think 21:13:42 <russellb> then starting with juno dev, all features go to gantt instead of nova 21:13:51 <n0ano> probably a good idea, I think it's going to take longer than we wanted to get gantt working but we'll see 21:13:55 <russellb> a clean switch point 21:13:58 <russellb> yeah 21:14:02 <russellb> let's see how long it takes 21:14:11 <russellb> but if it takes until then, i think that's actually OK because the timing is nice 21:14:24 <n0ano> for sure, I can's see gantt being a default until after Icehouse 21:14:33 <russellb> agree 21:15:00 <n0ano> others - no db work is ongoing, there are patches out under review, hopefully they'll go in soon 21:15:14 <russellb> ok, don't think we have that targeted for icehouse-2 21:15:17 <russellb> must be on icehouse-3 ... 21:15:35 <russellb> indeed, icehouse-3 it is 21:15:46 <russellb> alright, thanks for the updates! 21:15:55 <n0ano> one other... 21:15:57 <russellb> sure 21:16:16 <n0ano> glikson wants reviews for https://review.openstack.org/#/q/topic:bp/multiple-scheduler-drivers,n,z if people can find the time 21:16:27 <n0ano> just a gentle reminder 21:16:40 <n0ano> that's about it for now. 21:16:48 <russellb> cool, that's one of the ones on the blueprint list for icehouse-2 21:17:03 <n0ano> codes there, just needs to be reviewed 21:17:08 <russellb> sounds good 21:17:23 <russellb> seems we've been struggling a bit to catch back up on the review queue since the holidays 21:17:29 <russellb> very slowly improving based on stats 21:18:00 <russellb> melwitt: ok, novaclient! 21:18:10 <melwitt> the weekly novaclient report 2014/1/16: 21:18:10 <melwitt> open bugs, 139 !fix released, 90 !fix released and !fix committed 21:18:10 <melwitt> 24 new bugs, 0 high bugs 21:18:10 <melwitt> 33 patches up, 2 are WIP, good activity on reviews (as usual) 21:18:10 <melwitt> I need to catch up on the bug queue, got behind on it because of holiday etc 21:18:11 <melwitt> that's about it! 21:19:00 <russellb> cool, status seems stable 21:19:14 <russellb> new bug count isn't much higher at all than before holidays, so not bad at all 21:19:25 <melwitt> yeah 21:20:01 <russellb> thanks! 21:20:12 <russellb> #topic bugs 21:20:16 <russellb> sooo, nova bugs 21:20:20 <russellb> 207 new 21:20:28 <russellb> could use plenty of triage help as usual :-) 21:20:45 <russellb> some process notes on that if you want to help triage a few bugs: 21:20:47 <russellb> #link https://wiki.openstack.org/wiki/Nova/BugTriage 21:21:02 <russellb> and 21:21:03 <russellb> #link https://wiki.openstack.org/wiki/BugTriage 21:21:22 <russellb> we've got 8 critical bugs right now 21:21:28 <russellb> all gate affecting test failures pretty sure. 21:21:37 <russellb> which i wanted to talk a bit about 21:21:41 <russellb> because there has been some good progress lately 21:21:53 <russellb> #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/024052.html 21:22:05 <russellb> that link covers some gate failure analysis from last week, and some actions as a result 21:22:19 <jog0> russellb: there is also the libvirt gate issues, which running libvirt 1.x *may* fix. 21:22:30 <russellb> jog0: yes 21:22:36 <jog0> but we don't even know of nova works with libvirt 1.x 21:22:48 <russellb> my feeling is that the biggest improvement would be seen by merging: https://review.openstack.org/#/c/65805/ 21:22:55 <russellb> but it hasn't had much attention, unfortunately 21:23:21 <jog0> so nova-network was only hurting large-ops which hasn't been faling lately 21:23:26 <russellb> that's not true 21:23:36 <jog0> link? 21:23:45 <russellb> it was causing most of the failures i was seeing in the "timed out waiting for thing" bug too 21:23:51 <russellb> which was the #2 gate bug 21:23:58 <russellb> further, it's also going to help the #1 gate bug, the ssh timeout thing 21:24:05 <russellb> because it seems even that is largely performance related 21:24:09 <russellb> (they had some bugs too) 21:24:21 <russellb> note the neutron guy's comment on the review, as well 21:24:50 <russellb> the vast majority of the top failures are performance related due to the tests overloading nova+nova-network or nova+neutron 21:24:59 <russellb> was my conclusion from last week 21:25:11 <jog0> russellb: hmm lets talk after 21:25:12 <russellb> regarding nova-network, we've landed a few small changes 21:25:28 <russellb> and we also have this patch series that dan and I have been doing this week ... https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/nova-network-objects,n,z 21:25:35 <russellb> which will speed up nova-network quite a bit 21:25:58 <russellb> we've been busting our tails on that this week, largely as a response to realizing that the performance was a root cause (and that the mitigating patch is being ignored :-( ) 21:26:21 <russellb> anyone else have some bug related topics for today? 21:26:35 <jog0> russellb: FWIW jeblair is on vacation this week and infra is barebones right now 21:26:43 <jog0> and dealing with many issues 21:27:21 <russellb> OK, guess I didn't really have insight into issues they're fighting 21:27:24 <dripton> It would be good if all the good reviews that fix real bugs got reviewed in time for Icehouse-2. The slow review queue hurts. 21:27:37 <russellb> yeah, hard to make that process go much faster 21:27:45 <russellb> we have a lot of icehouse-2 targeted bugs - https://launchpad.net/nova/+milestone/icehouse-2 21:28:11 <russellb> #topic blueprints 21:28:13 <russellb> speaking of icehouse-2 ... 21:28:17 <russellb> #link https://launchpad.net/nova/+milestone/icehouse-2 21:28:28 <russellb> we've merged 8 blueprints so far for icehouse-2 21:28:35 <russellb> our velocity is down a good bit compared to last cycle 21:28:47 <russellb> some of that is holidays, some of it is gate issues, some of it is just lower availability of the core team it seems 21:29:17 <russellb> but if the core team could help out in the next few days doing a review push for icehouse-2, that'd be great 21:29:44 <jog0> russellb: FWIW I would prefer to see a push on fixing gate issues first, that will improve general velocity 21:29:48 <russellb> i used to have an easy way to email that group directly, but don't anymore :( 21:29:58 <jog0> but thast not really an xor issue 21:30:05 <russellb> jog0: sure, been trying 21:30:16 <russellb> that's where all the nova-network performance work came from this week 21:30:21 <russellb> would love help from more people 21:30:33 <jog0> russellb: maybe this week can be libvirt 21:30:38 <jog0> russellb: ++ 21:30:41 <russellb> trying to track current status here: https://etherpad.openstack.org/p/nova-gate-issue-tracking 21:30:50 <russellb> jog0: yeah, that can be next after this nova-network series lands i think 21:30:58 <jog0> russellb: woot 21:31:23 <russellb> any blueprint topics? 21:31:58 <russellb> note that we've completed 21 blueprints so far out of about 150 icehouse blueprints, heh 21:32:08 <russellb> 118 targeted at icehouse-3 right now 21:32:20 <mikal> Yay! 21:32:23 <russellb> so if you own one or more of those ... be prepared for some sort of major adjustment soon after icehouse-2 21:32:30 <russellb> perhaps an early code proposal deadline 21:32:43 <russellb> because there's absolutely no way all of that can make it in 1 milestone 21:32:52 <russellb> so we'll have to start doing a reality check and cutting it early 21:32:58 <russellb> icehouse is flying by ... 21:33:08 <russellb> for reference: 21:33:10 <russellb> #link https://wiki.openstack.org/wiki/Icehouse_Release_Schedule 21:33:47 <russellb> #topic open discussion 21:33:56 <russellb> any other topics for today? 21:35:31 <russellb> alrighty then 21:35:40 <russellb> as usual, #openstack-nova open for discussion throughout the week 21:35:42 <russellb> thanks, everyone! 21:35:45 <russellb> #endmeeting