21:00:52 <russellb> #startmeeting nova
21:00:53 <openstack> Meeting started Thu Jan 16 21:00:52 2014 UTC and is due to finish in 60 minutes.  The chair is russellb. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:54 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:57 <openstack> The meeting name has been set to 'nova'
21:01:00 <russellb> hello, everyone!
21:01:06 <n0ano> o/
21:01:08 <melwitt> hi
21:01:13 <dripton> hi
21:01:17 <geekinutah> ola!
21:01:20 <hartsocks> \o
21:01:37 <russellb> #topic general
21:01:45 <russellb> nova meetup coming up soon!
21:01:47 <russellb> #link https://wiki.openstack.org/wiki/Nova/IcehouseCycleMeetup
21:01:51 <russellb> details on that page
21:02:06 <russellb> a more recent addition was the etherpad where we'll collect some discussion ideas: https://etherpad.openstack.org/p/nova-icehouse-mid-cycle-meetup-items
21:02:48 <russellb> general idea for the format right now is ... reserve some time for group discussion/presentation, unconference style ... and at least half the time reserved for just working on things together
21:03:02 <russellb> whether that's hacking code, reviewing code, reviewing blueprints, talking through tough problems, etc ...
21:03:06 <russellb> should be good.
21:03:25 <russellb> #topic sub-teams
21:03:32 * hartsocks waves
21:03:37 * n0ano gantt
21:03:38 <russellb> who wants to give a report and discuss work in sub-teams?  raise hand
21:03:43 <russellb> hartsocks: you're up first
21:03:45 * melwitt novaclient
21:03:54 * hartsocks waits for slow typists
21:04:00 <russellb> actually real quick
21:04:06 <russellb> i should have mentioned in general a quick schedule note ...
21:04:09 <russellb> icehosue-2 is next week
21:04:15 <russellb> code freeze for that is tueseday
21:04:24 <russellb> so you have < 1 week to get code merged for icehouse-2
21:04:25 <russellb> that is all on that
21:04:28 <russellb> hartsocks: ok now you can go
21:04:43 <hartsocks> Okay, so we're dealing with performance issues on Minesweeper. There's several BP and bugs in flight now on how to get the Minesweeper to sweep faster.
21:04:50 <hartsocks> That's our big deal.
21:05:00 <russellb> so what kind of performance
21:05:07 <russellb> how long test runs are taking basically?
21:05:51 <hartsocks> okay.
21:06:01 <hartsocks> so we're seeing 4 hours w/o parallel
21:06:15 <hartsocks> and 2 hours with
21:06:29 <russellb> parallel with what concurrency
21:06:30 <hartsocks> the parallelization stuff causes problems but we're working on it.
21:06:48 <russellb> cool, sounds like it's helping improve quality :)
21:06:50 <russellb> the goal after all
21:07:10 <hartsocks> IIRC: we're looking at 4 … but that might have changed since my last look.
21:07:18 <russellb> OK, just curious
21:07:27 <russellb> devstack-gate jobs currently run with 4
21:07:36 <russellb> (but it's a bit too high, we'll talk about that later)
21:07:59 <hartsocks> so that's the big news for us.
21:08:00 <russellb> thanks for the updates!
21:08:12 <hartsocks> I spammed the ML with more details on reveiws
21:08:32 <russellb> OK, if you #link it in the meeting itll be in the minutes
21:08:41 <russellb> but we'll move on for now
21:08:41 <hartsocks> #link https://etherpad.openstack.org/p/vmware-subteam-icehouse-2
21:08:45 <russellb> perfect
21:08:49 * hartsocks bows out
21:09:06 <russellb> will have to check later to make sure it shows up if you do it and not me ...
21:09:14 <russellb> n0ano: hi!
21:09:17 <russellb> so, scheduler stuff.
21:09:30 <n0ano> scheduler - never heard of it :-)
21:09:37 <russellb> sounds like the plan isn't too clear or agreed upon this week
21:09:56 <n0ano> flurry of emails on that but I think we can bottom out
21:10:08 <russellb> few different ways we can improve it i think ...
21:10:13 <russellb> what are your thoughts?
21:10:20 <n0ano> looks like the idea of continuous update vs. the nova tree is not in favor, which is OK
21:10:53 <n0ano> the top idea now is we use the current gantt snapshot and get it working, unit tests and full integration...
21:11:06 <russellb> OK, that sounds pretty good to me
21:11:14 <russellb> though depending on timing, integration may not work
21:11:18 <russellb> if it's too far behind nova
21:11:35 <n0ano> after that we get all the patches that were needed, re-create the gantt tree from the current nova tree and then update the gantt tree to work again
21:11:46 <russellb> ok, sounds perfect
21:11:53 <russellb> if we know we have a path to a gantt that works quite quickly
21:12:03 <russellb> then i'm much more open to freezing nova-scheduler at that point
21:12:13 <n0ano> we should be OK, the current snapshot was a working nova tree so getting it to pass integration test should be possible
21:12:32 <n0ano> agreed, I don't want to freeze anything until we have a working gantt, no point in stopping development
21:12:55 <n0ano> I'll send out an email that details this new plan and make sure everyone agrees to it
21:13:01 <russellb> sounds good
21:13:19 <russellb> depending on how that comes along, it's also worth considering planning the re-sync after feature freeze
21:13:25 <russellb> that timing would work nicely, i think
21:13:42 <russellb> then starting with juno dev, all features go to gantt instead of nova
21:13:51 <n0ano> probably a good idea, I think it's going to take longer than we wanted to get gantt working but we'll see
21:13:55 <russellb> a clean switch point
21:13:58 <russellb> yeah
21:14:02 <russellb> let's see how long it takes
21:14:11 <russellb> but if it takes until then, i think that's actually OK because the timing is nice
21:14:24 <n0ano> for sure, I can's see gantt being a default until after Icehouse
21:14:33 <russellb> agree
21:15:00 <n0ano> others - no db work is ongoing, there are patches out under review, hopefully they'll go in soon
21:15:14 <russellb> ok, don't think we have that targeted for icehouse-2
21:15:17 <russellb> must be on icehouse-3 ...
21:15:35 <russellb> indeed, icehouse-3 it is
21:15:46 <russellb> alright, thanks for the updates!
21:15:55 <n0ano> one other...
21:15:57 <russellb> sure
21:16:16 <n0ano> glikson wants reviews for https://review.openstack.org/#/q/topic:bp/multiple-scheduler-drivers,n,z if people can find the time
21:16:27 <n0ano> just a gentle reminder
21:16:40 <n0ano> that's about it for now.
21:16:48 <russellb> cool, that's one of the ones on the blueprint list for icehouse-2
21:17:03 <n0ano> codes there, just needs to be reviewed
21:17:08 <russellb> sounds good
21:17:23 <russellb> seems we've been struggling a bit to catch back up on the review queue since the holidays
21:17:29 <russellb> very slowly improving based on stats
21:18:00 <russellb> melwitt: ok, novaclient!
21:18:10 <melwitt> the weekly novaclient report 2014/1/16:
21:18:10 <melwitt> open bugs, 139 !fix released, 90 !fix released and !fix committed
21:18:10 <melwitt> 24 new bugs, 0 high bugs
21:18:10 <melwitt> 33 patches up, 2 are WIP, good activity on reviews (as usual)
21:18:10 <melwitt> I need to catch up on the bug queue, got behind on it because of holiday etc
21:18:11 <melwitt> that's about it!
21:19:00 <russellb> cool, status seems stable
21:19:14 <russellb> new bug count isn't much higher at all than before holidays, so not bad at all
21:19:25 <melwitt> yeah
21:20:01 <russellb> thanks!
21:20:12 <russellb> #topic bugs
21:20:16 <russellb> sooo, nova bugs
21:20:20 <russellb> 207 new
21:20:28 <russellb> could use plenty of triage help as usual :-)
21:20:45 <russellb> some process notes on that if you want to help triage a few bugs:
21:20:47 <russellb> #link https://wiki.openstack.org/wiki/Nova/BugTriage
21:21:02 <russellb> and
21:21:03 <russellb> #link https://wiki.openstack.org/wiki/BugTriage
21:21:22 <russellb> we've got 8 critical bugs right now
21:21:28 <russellb> all gate affecting test failures pretty sure.
21:21:37 <russellb> which i wanted to talk a bit about
21:21:41 <russellb> because there has been some good progress lately
21:21:53 <russellb> #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/024052.html
21:22:05 <russellb> that link covers some gate failure analysis from last week, and some actions as a result
21:22:19 <jog0> russellb: there is also the libvirt gate issues, which running libvirt 1.x *may* fix.
21:22:30 <russellb> jog0: yes
21:22:36 <jog0> but we don't even know of nova works with libvirt 1.x
21:22:48 <russellb> my feeling is that the biggest improvement would be seen by merging: https://review.openstack.org/#/c/65805/
21:22:55 <russellb> but it hasn't had much attention, unfortunately
21:23:21 <jog0> so nova-network was only hurting large-ops which hasn't been faling lately
21:23:26 <russellb> that's not true
21:23:36 <jog0> link?
21:23:45 <russellb> it was causing most of the failures i was seeing in the "timed out waiting for thing" bug too
21:23:51 <russellb> which was the #2 gate bug
21:23:58 <russellb> further, it's also going to help the #1 gate bug, the ssh timeout thing
21:24:05 <russellb> because it seems even that is largely performance related
21:24:09 <russellb> (they had some bugs too)
21:24:21 <russellb> note the neutron guy's comment on the review, as well
21:24:50 <russellb> the vast majority of the top failures are performance related due to the tests overloading nova+nova-network or nova+neutron
21:24:59 <russellb> was my conclusion from last week
21:25:11 <jog0> russellb: hmm lets talk after
21:25:12 <russellb> regarding nova-network, we've landed a few small changes
21:25:28 <russellb> and we also have this patch series that dan and I have been doing this week ... https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/nova-network-objects,n,z
21:25:35 <russellb> which will speed up nova-network quite a bit
21:25:58 <russellb> we've been busting our tails on that this week, largely as a response to realizing that the performance was a root cause (and that the mitigating patch is being ignored :-( )
21:26:21 <russellb> anyone else have some bug related topics for today?
21:26:35 <jog0> russellb: FWIW jeblair is on vacation this week  and infra is barebones right now
21:26:43 <jog0> and dealing with many issues
21:27:21 <russellb> OK, guess I didn't really have insight into issues they're fighting
21:27:24 <dripton> It would be good if all the good reviews that fix real bugs got reviewed in time for Icehouse-2.  The slow review queue hurts.
21:27:37 <russellb> yeah, hard to make that process go much faster
21:27:45 <russellb> we have a lot of icehouse-2 targeted bugs - https://launchpad.net/nova/+milestone/icehouse-2
21:28:11 <russellb> #topic blueprints
21:28:13 <russellb> speaking of icehouse-2 ...
21:28:17 <russellb> #link https://launchpad.net/nova/+milestone/icehouse-2
21:28:28 <russellb> we've merged 8 blueprints so far for icehouse-2
21:28:35 <russellb> our velocity is down a good bit compared to last cycle
21:28:47 <russellb> some of that is holidays, some of it is gate issues, some of it is just lower availability of the core team it seems
21:29:17 <russellb> but if the core team could help out in the next few days doing a review push for icehouse-2, that'd be great
21:29:44 <jog0> russellb: FWIW I would prefer to see a push on fixing gate issues first, that will improve general velocity
21:29:48 <russellb> i used to have an easy way to email that group directly, but don't anymore :(
21:29:58 <jog0> but thast not really an xor issue
21:30:05 <russellb> jog0: sure, been trying
21:30:16 <russellb> that's where all the nova-network performance work came from this week
21:30:21 <russellb> would love help from more people
21:30:33 <jog0> russellb: maybe this week can be libvirt
21:30:38 <jog0> russellb: ++
21:30:41 <russellb> trying to track current status here: https://etherpad.openstack.org/p/nova-gate-issue-tracking
21:30:50 <russellb> jog0: yeah, that can be next after this nova-network series lands i think
21:30:58 <jog0> russellb: woot
21:31:23 <russellb> any blueprint topics?
21:31:58 <russellb> note that we've completed 21 blueprints so far out of about 150 icehouse blueprints, heh
21:32:08 <russellb> 118 targeted at icehouse-3 right now
21:32:20 <mikal> Yay!
21:32:23 <russellb> so if you own one or more of those ... be prepared for some sort of major adjustment soon after icehouse-2
21:32:30 <russellb> perhaps an early code proposal deadline
21:32:43 <russellb> because there's absolutely no way all of that can make it in 1 milestone
21:32:52 <russellb> so we'll have to start doing a reality check and cutting it early
21:32:58 <russellb> icehouse is flying by ...
21:33:08 <russellb> for reference:
21:33:10 <russellb> #link https://wiki.openstack.org/wiki/Icehouse_Release_Schedule
21:33:47 <russellb> #topic open discussion
21:33:56 <russellb> any other topics for today?
21:35:31 <russellb> alrighty then
21:35:40 <russellb> as usual, #openstack-nova open for discussion throughout the week
21:35:42 <russellb> thanks, everyone!
21:35:45 <russellb> #endmeeting