21:01:27 <mikal> #startmeeting nova 21:01:28 <openstack> Meeting started Thu Jun 5 21:01:27 2014 UTC and is due to finish in 60 minutes. The chair is mikal. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:31 <openstack> The meeting name has been set to 'nova' 21:01:34 <dansmith> o/ 21:01:38 <jgrimm> o/ 21:01:38 <mriedem> o/ 21:01:40 <russellb> o/ 21:01:40 <mikal> So, who is around? 21:01:41 <cyeoh> hi 21:01:42 <n0ano> o/ 21:01:45 <jogo> o/ 21:01:46 <oomichi_> hi 21:02:00 <mikal> Cool! 21:02:09 <ewindisch> . 21:02:11 <mikal> The agenda for today is at https://wiki.openstack.org/wiki/Meetings/Nova as always 21:02:16 <adrian_otto> o/ 21:02:29 <mikal> #topic Juno mid-cycle meetup 21:02:45 <mikal> So while I slept last week, John asked people to fill in a survey for the mi cycle meetup 21:02:49 <mikal> mid even 21:03:05 <mikal> The winning dates were the weekdays after OSCON 21:03:13 <russellb> the weekend? 21:03:17 <russellb> oh weekdays 21:03:17 <mikal> So July 28 - 30 21:03:23 <mikal> russellb: no, Monday thru Wednesday 21:03:27 <dansmith> so my cash bribes worked 21:03:27 <russellb> great. 21:03:30 <mikal> Heh 21:03:45 <mikal> Intel has now confirmed they can host on those dates, so I think we're good to announce 21:03:54 <mikal> I'll send an email to openstack-dev after this meeting 21:04:00 <alaski> o/ 21:04:10 <mikal> I intend to try and arrange a block hotel booking, but that hasn't been done yet 21:04:19 <mikal> But at least this way people can start negotiating with their managers 21:04:47 <jogo> where will it be portland? 21:04:48 <mikal> Is there anything else we need to cover there apart from my promise of an email soon? 21:04:58 <mikal> jogo: Beaverton, at the Intel campus 21:05:02 <n0ano> jogo, technically beaverton 21:05:12 <mikal> jogo: which is a 20 minute drive from the airport IIRC 21:05:20 <jogo> neato 21:05:20 <russellb> main portland airport would be best i presume? 21:05:22 <russellb> ah. 21:05:28 <russellb> dansmith will give everyone a ride 21:05:29 <dansmith> pfft 21:05:31 <mikal> russellb: I think so, but I sahll confirm 21:05:34 <dansmith> not even 20 minutes at midnight 21:05:39 <n0ano> mikal, you drive fast :-)( 21:05:42 <mikal> I think I was told there is a train as well, but that might be a lie 21:06:06 <dansmith> n0ano: which campus? 21:06:21 <n0ano> dansmith, jones farm is the current site 21:06:28 <dansmith> okay 21:06:41 <mikal> Oh, I see 21:06:46 <mikal> I didn't realize there was more than one campus 21:06:50 <mikal> I was looking at maps for Aloha 21:07:04 <dansmith> there are lots of them 21:07:07 <dansmith> all over the damned place 21:07:08 <n0ano> there's about 5, maybe more 21:07:14 <mikal> *shrug* 21:07:14 <dansmith> five that we know about 21:07:17 <mikal> We'll work it out 21:07:23 <mikal> There are secret hidden campuses? 21:07:25 <russellb> wiki-ify it 21:07:28 <dansmith> it’s intel 21:07:33 <russellb> with the deets 21:07:33 <dansmith> everything is a secret 21:07:38 <mikal> russellb: yep, that's the plan 21:07:40 <dansmith> all the campuses have privacy berms in front 21:07:43 <n0ano> they're all relatively close together (<10 min by care) 21:07:45 <russellb> k 21:07:58 <dansmith> “look not beyond yonder wall” 21:08:04 <mikal> Heh 21:08:13 <mikal> Ok, so, we're having the meetup in a lair 21:08:17 <mikal> In other news... 21:08:23 <mikal> Are we done with this topic for now? 21:08:30 <russellb> will there be cake? 21:08:33 <russellb> yes. 21:08:37 <russellb> i mean yes, i'm done. 21:08:54 <mikal> #action mikal to wiki page up the mid cycle details 21:09:04 <mikal> #topic Gate breakage 21:09:12 <mikal> This one isn't on the agenda because its new 21:09:24 <mikal> I've woken up to emails saying everything is busted 21:09:31 <mriedem> one of the breakers was force merged 21:09:34 <mriedem> the resize one 21:09:35 <mikal> Does someone have more details on if there's things nova needs to do to fix the world? 21:09:53 <mriedem> there was a get-pip one that sdague fixed yesterday 21:10:02 <mriedem> sounds like things are still backed up, not sure if nova is the main issue now 21:10:14 <mikal> Ok, but we should hold off on approving things, right? 21:10:15 <mriedem> neutron has the ssh issue everyone knows and loves as #1 21:10:25 <mriedem> i think that's what they were asking, unless they fix race bugs 21:10:36 <mikal> Ok 21:10:44 <mikal> But we're not aware of any more nova bugs that need looking at? 21:10:57 <jogo> look at http://status.openstack.org/elastic-recheck/ 21:10:58 <mriedem> well let's see 21:10:59 <mriedem> yteah 21:11:00 <mriedem> that 21:11:08 <jogo> Bug 1254890 - "Timed out waiting for thing ... to become ACTIVE" causes tempest-dsvm-* failures 21:11:09 <uvirtbot> Launchpad bug 1254890 in nova ""Timed out waiting for thing ... to become ACTIVE" causes tempest-dsvm-* failures" [High,Confirmed] https://launchpad.net/bugs/1254890 21:11:28 <jogo> that looks like the biggest nova one 21:11:39 <mriedem> which has been around forever 21:11:46 <mikal> jogo: that's probably not a nova bug though right? 21:11:48 <jogo> yup 21:11:50 <mriedem> sure 21:11:51 <russellb> it may be 21:11:53 <mikal> jogo: that's nova waiting on other services? 21:11:54 <mriedem> compute api tests can timeout all over 21:11:58 <russellb> that bug tends ot catch a number of underlying bugs 21:11:59 <mriedem> not necessarily 21:12:00 <leifz> o/ late to the meeting. 21:12:02 <russellb> need to dig into specific failures 21:12:10 <mriedem> e.g. timeout waiting for snapshot 21:12:12 <mriedem> something like that 21:12:15 <mriedem> qemu-img taking too long 21:12:15 <jogo> yeah, so its not uncommon as we dig into bugs to find they are 5 or 6 bugs 21:12:24 <jogo> in which case we can split the bug up and add seperate e-r queries etc 21:12:30 <russellb> that one in particular ... a bunch has come out of that in the past, it's been on there a long time 21:12:33 <mikal> jogo: do you split them out into separate bugs at that point? 21:12:39 <mikal> Cool 21:12:49 <mriedem> we want this: https://review.openstack.org/#/c/97812/ 21:13:04 <mriedem> i think we need to help get better diags when things fail to help with a lot of these timeouts 21:13:17 <jogo> mikal: yeah. we just say this bug covered several underyling issues ... 21:13:27 <mikal> Ok 21:13:28 <mriedem> there are probably several different similar timeout bugs 21:13:37 <mikal> But I'm not seeing a specifical call to action here for nova, is that fair? 21:13:40 <mikal> ? 21:13:41 <mriedem> not sure if there are some that hit more than others, we haven't dug into that 21:13:49 <anteaya> except please don't approve stuff 21:13:53 <jogo> also getting rid of stacktraces in the logs really helps with these things 21:13:56 <mikal> Yeah, except that 21:14:15 <jogo> we still ahve a lot 21:14:17 <mikal> jogo: is there a list of bogus stacktraces in logs somewhere? 21:14:29 <mikal> jogo: how would someone wanting to help with that proceed? 21:14:39 <mriedem> there is a whitelist dump at the end of the runs 21:14:44 <mriedem> of all the errors that aren't whitelisted from the logs 21:14:50 <jogo> mikal: what mriedem said 21:14:51 <mriedem> used to gate on that but couldn't keep up 21:15:01 <mriedem> i think the only thing we gate on is no errors in n-cond 21:15:14 <dansmith> right, afaik 21:15:16 <mriedem> and alaski has a fix up for one of those 21:15:28 <mriedem> would be this https://review.openstack.org/#/c/96955/ 21:15:37 <mikal> #action We need to help remove bogus stack traces from our tempest logs 21:15:50 <mriedem> i thought, maybe that's not the one 21:16:01 <alaski> mriedem: 97942 maybe? 21:16:01 <mriedem> there was a bogus info cache one 21:16:25 <mriedem> alaski: doesn't look right 21:16:29 <mikal> That second one is approved 21:16:30 <mriedem> i think there is a specofic bug 21:16:40 <alaski> mriedem: oh, the one you're thinking of merged 21:16:56 <alaski> mriedem: https://review.openstack.org/#/c/96824/ 21:17:06 <mriedem> that's the one 21:17:09 <jogo> mikal: sample output http://logs.openstack.org/98/96998/1/check/check-tempest-dsvm-full/95f0c01/console.html#_2014-06-03_04_42_16_638 21:17:20 <jogo> from most recent nova patch that merged 21:17:25 <mriedem> yeah 21:17:33 <mriedem> there was a sec group list race bug too that was merged yesterday 21:17:40 <mriedem> fix merged i mean 21:18:01 <jogo> mriedem: correct me if I am wrong, but a lot of the instability this time isn't from nova 21:18:13 <mriedem> besides the resize thing reverted this morning, i think that's correct 21:18:18 <russellb> mostly neutron looks like ... 21:18:19 <mikal> Cool 21:18:19 <mriedem> lots of infra issues 21:18:22 <mriedem> and neutron yeah 21:18:25 <mikal> I didn't mean to imply it was 21:18:31 <mikal> Just making sure we're pulling in the right direction 21:18:32 <russellb> i'm sure those teams wouldn't mind help on those issues though :) 21:18:32 <mriedem> ceilometer UT is shitting the bed also 21:18:38 <mikal> It sounds like we're off the hook mostly 21:18:44 <mriedem> they copied our test timeouts and started timing out :) 21:19:07 <mikal> Is there anything else here or should we move on? 21:19:10 <mriedem> move on 21:19:18 <mikal> #topic juno-1 21:19:36 <mikal> Over the last week johnthetubaguy has been pushing things from juno-1 to juno-2 that look like they wont land in time 21:19:44 <mikal> juno-1 being 12 June, which is real soon now 21:20:06 <mikal> We should not be approving specs at the moment, but instead should be trying to review bps targetted to juno-1 (and bug fixes, more on that later) 21:20:12 <russellb> #link https://launchpad.net/nova/+milestone/juno-1 21:20:27 <mikal> Obviously the gate thing will slow us down there 21:21:01 <mikal> So this is mostly a reminder that juno-1 is just around the corner 21:21:11 <mikal> johnthetubaguy: you around? 21:21:18 <mikal> (I suspect not) 21:21:52 <mikal> Moving on 21:22:01 <mikal> #topic Bugs 21:22:04 <mriedem> i suspect compute-manager-objects-juno will be a moving target 21:22:18 <mikal> mriedem: yeah, that seems likely to me too 21:22:19 <dansmith> yeah, not much point in that being anything other than j3 I think 21:22:58 <mriedem> bugs! 21:23:01 <mikal> Ok, I changed that one 21:23:02 <mikal> Bugs 21:23:02 <mriedem> any bug day analysis? 21:23:09 <mikal> tjones ran a bug day the other day 21:23:16 <tjones> we had about 8ish bugs merged yesterday 21:23:16 <mikal> The early analysis is "it sucked" 21:23:25 <mikal> Well, that's my analysis at least 21:23:27 <tjones> people are continuing to fix and review bugs today as well 21:23:34 <mikal> Ok cool 21:23:42 <mikal> I think tjones did a good job 21:23:43 <tjones> my 1st bug day so not sure what to expect 21:23:46 <mikal> We just didn't fix enough 21:23:52 <mikal> Given that we have 1,200 bugs 21:23:54 <mriedem> closing invalid bugs == fixing 21:23:58 <mriedem> imo 21:24:00 <tjones> true 21:24:02 <mikal> mriedem: agreed 21:24:04 <mriedem> there was quite a bit of that yesterday 21:24:10 <mikal> 1,200 is so many we just don't know what's there 21:24:14 <cyeoh> a bunch set to "need more info" too 21:24:16 <jogo> why wasn't the bug day stuff done in the nova room? 21:24:19 <tjones> open bug counts went down about 30ish 21:24:20 <mikal> #link http://status.openstack.org/bugday/ 21:24:29 <mriedem> jogo: it was... 21:24:31 <dansmith> jogo: i was in there chasing bugs 21:24:42 <jogo> mriedem: ohh I thought there as an email saying it wasn't, ignore me 21:24:44 <mriedem> i was going through a lot of abandoned things 21:24:45 <tjones> as mriedem said yesterday - hard to see the granularity of this chart 21:25:15 <mikal> The bit that worries me is that it feels to me like we're ignoring our users. They try to get our attention and we don't keep up. 21:25:35 <mikal> I don't know how to fix that, except for asking people to be consistent about trying to close bugs 21:25:41 <mikal> That's really a huge list to try and burn down 21:25:54 <mikal> I'm 100% sure there's dupes etc in there, but I don't know how we find them when the list is so long 21:26:11 <mriedem> one by one 21:26:14 <jogo> have we closed bugs older then a year? 21:26:22 <jogo> to get us down to a sane list etc? 21:26:30 <mikal> jogo: no we haven't 21:26:33 <mriedem> jogo: the bug day email has a link to in progress bugs but a lot of those are old and abandoned patches 21:26:37 <leifz> We probably want to query opener for 6 months with no activity as well. 21:26:41 <mikal> jogo: I think we'd want to discuss that on the mailing list before doing it 21:26:42 <devananda> jogo: does that necessarily mean they are not valid any longer? 21:26:46 <mriedem> jogo: i went through a lot of those one by one and closed as invalid, incomplete or dupes 21:26:53 <mriedem> or moved back to triaged and removed assignee 21:26:57 <devananda> i'm sure there are bugs >1yr old for nova-bm which are still valid, for example 21:27:10 <mriedem> yes 21:27:18 <mriedem> which brings up a point i raised yesterday about nova-bm bugs 21:27:20 <mriedem> there were 40+ 21:27:21 <leifz> I did find one which had been fixed and never duplicated to closed issue. 21:27:28 <mriedem> i'm not sure those baremetal bugs are tagged for ironic 21:27:29 <jogo> not as a blanket rule but as a guideline 21:27:29 <mriedem> but they should be 21:27:33 <devananda> mriedem: ++ 21:27:38 <tjones> also our bug management tools need help badly. currently i get all bugs and stick them in excel to see what is going on 21:27:47 <devananda> mriedem: some of them i may have untagged specifically (but there'd be a coment trail) 21:27:52 <devananda> mriedem: because they didn't apply 21:28:01 <mikal> I don't expect we can solve this today 21:28:08 <mikal> But I would like people to ponder it 21:28:08 <tjones> wish we used bugzilla 21:28:19 <mikal> tjones: the foundation is writing a new thing, but its not ready 21:28:20 <mriedem> how often do we do bug days? 21:28:26 <leifz> When is the next bug day? 21:28:26 <russellb> tjones: i'm not sure i've ever heard someone say that 21:28:27 <mikal> mriedem: in general, once per release 21:28:33 <mriedem> mikal: we should do them more often then 21:28:34 <devananda> food for thought -- is there a way to make it easier for users to submit drive-by bug fixes (and for those to get accepted/merged) 21:28:36 <mriedem> once a month 21:28:39 <tjones> it would be better than what we have 21:28:39 <anteaya> tjones: pleia2 does our bugdays, do you want to chat with her about tools? 21:28:40 <mikal> Last time we tried to do a second no one showed up 21:28:41 <devananda> that might apply more systemically, not just to nova 21:28:49 <mriedem> no one showed up yesterday 21:28:51 <mriedem> imo 21:28:54 <tjones> sure thanks anteaya 21:29:00 <anteaya> np 21:29:07 <mriedem> no one == very few compared to the number of people in the room 21:29:11 <mikal> mriedem: that's kind of how I feel. I know its unfair on the people who did, but not enough people showed up. 21:29:28 <mikal> Do we think people would get bored if we did something monthly? 21:29:28 <russellb> i think counting on a a catch-up bug day is always going to fail long term 21:29:37 <russellb> and i think a regular bug team (what tjones has been trying to do) is the best bet 21:30:00 <mikal> russellb: I agree, but I also think that every dev has to help out 21:30:00 <tjones> no one is showing up to the bug meetings as well. i use that time to triage bugs 21:30:20 <russellb> mikal: right, and tjones has been trying to keep the work organized and broken up for devs to pitch in 21:30:30 <russellb> using the tagging, and a coordinated time for people to meet and triage 21:30:30 <mikal> I personally feel that some of this is because stackalytics doesn't track bug fixes, so people aren't competing based on them 21:30:38 <devananda> i doubt we can get any real numbers, but the sense i have is that companies are fixing bugs internally as they hack things into product/ion, and the benefit of that isn't often flowing upstream 21:30:57 <russellb> that and honestly, the bugs i care about most are ones that come from my customers 21:31:00 <mikal> I've filed a bug against stackalytics to ask for that to change 21:31:01 <russellb> i'm sure other people work that way too 21:31:07 <russellb> it's just how it ends up happening' 21:31:28 <jogo> so this isn't a good solution, but at the very least we should just discuss this issue more 21:31:36 <jogo> to at least raise awareness 21:31:41 <mikal> Yeah, I think this will be on the agenda for the mid cycle 21:31:46 <mikal> I can't see it being fully solved by then 21:31:52 <leifz> tjones: you keep of list of bugs which have proposed fixes in play? 21:32:09 <mikal> I feel a bit like many of the bugs with fixes out there are for bugs which were just filed 21:32:14 <pleia2> our procedure for infra is pretty simple, I just have a launchpad lib script that pulls a list that we drop into an etherpad and go to town https://github.com/pleia2/openstack-infra-scripts/blob/master/infra_bugday.py 21:32:17 <mikal> i.e. dev files bug to track work, fixes it 21:32:19 <tjones> i have a list of bugs, their age, and when last updated 21:32:36 <mikal> I don't have data on that though, perhaps its unfair 21:32:40 <devananda> also having tools to keep track of stale bugs (assigned but not working on, patch proposed and then bit rotted) 21:32:44 <pleia2> lplib could be better documented, but you can grab a fair amount from the api 21:32:45 <devananda> might help 21:33:11 <tjones> pleia2: that is what i use - grab everything and stick in excel for analysis 21:33:23 <mikal> So what I did for bug day is trolled around looking for an interesting bug, then search for similar ones. I think I found six related bugs to work on. I fixed three. 21:33:32 <mikal> So that is a technique which might work for other people as well 21:34:16 <mriedem> automatically marking a bug as no longer in progress if the patch was abandoned over 6 months ago might help 21:34:21 <mriedem> that's come up before 21:34:28 <mikal> mriedem: I think shorter than that 21:34:31 <mriedem> sure 21:34:33 <mikal> abandoned for more than a couple of weeks 21:34:34 <mriedem> but automating it 21:34:48 <mriedem> we probably have hundreds of those 21:35:03 <cyeoh> yep, it really needs to be automated 21:35:05 <tjones> what do you mark it as?? 21:35:11 <mriedem> depends 21:35:15 <mriedem> usually triaged 21:35:21 <mriedem> but depends on why it was abandoned 21:35:23 <mikal> tjones: how often do you ask people for more info on new bugs? What release they're running, stuff like that? 21:35:38 <mriedem> yeah sometimes i mark those as incomplete 21:35:47 <tjones> when i triage, if it is not clear to me i ask for more info and mark incomplete 21:35:57 <mikal> tjones: I think we're just talking about removing the assignee and reverting from "In progress" to "Confirmed" 21:36:05 <mriedem> something like that 21:36:10 <mikal> Would automating asking for more info help? 21:36:17 <mikal> Like sending a survey to everyone who files a new bug? 21:36:31 <tjones> by "triage" i mean tagging untagged bugs. I am not reading each new bug. if they come in tagged i leave them 21:36:32 <mikal> That might annoy our frequent bug filers 21:37:04 <mikal> We could also auto-close bugs which have been incomplete for more than six months 21:37:14 <cyeoh> do we have a wiki page on what info is very useful for a bug report to contain? Because lots of the API ones I see a very vague 21:37:17 <devananda> mriedem: ++ to an aotmated tool for that. would help several projects, i bet 21:37:31 <cyeoh> and I end up setting a lot to incomplete because there simply isn't enough info to replicate the issue 21:37:54 <mikal> cyeoh: I can't think of one off the top of my head 21:38:18 <devananda> cyeoh: do you require a bug report to have enough info to reproduce it? 21:38:32 <mriedem> if there was one, i'd think we'd find the template here https://wiki.openstack.org/wiki/Bugs 21:38:35 <mriedem> or linked from there 21:38:53 <cyeoh> devananda: well often its info I know they have, they just probably didn't think to include it but would have if they'd known it would be useful 21:39:14 <cyeoh> devananda: even if its just simple things like they were running against master, or icehouse or havana etc. 21:39:18 <mriedem> which release, which commit, basic setup/topology, steps, stacktrace 21:39:35 <mikal> #action Everyone to think about how to improve our bug state, there are some ideas for automation if people want a coding task 21:39:47 <mikal> cyeoh: I think making a wiki page is a good idea, can you have a go at it please? 21:40:03 <cyeoh> mikal: sure 21:40:08 <mikal> cyeoh: thanks 21:40:18 <mikal> This is really important to me, but I feel we need to move on 21:40:41 <mikal> #topic Sub team reports 21:40:50 <adrian_otto> o/ 21:40:56 <mikal> So what sub teams do we have hanging around? 21:41:02 <mikal> adrian_otto: you have a containers report? 21:41:05 * n0ano gantt 21:41:06 <adrian_otto> Containers 21:41:09 <adrian_otto> #link http://eavesdrop.openstack.org/meetings/containers/2014/containers.2014-06-03-22.00.html Containers Sub-Team Meeting Minutes 21:41:10 <adrian_otto> Top takeaway is determining how cinder support can be added and what DefCore requirements should be relaxed (if any). 21:41:10 <cyeoh> o/ 21:41:10 * devananda wonders if he qualifies as a subteam :) 21:41:35 <adrian_otto> next discussion will be about how to support operating specific needs 21:41:51 <mikal> adrian_otto: I assume that's something you guys are progressing... Do you need anything from nova itself? 21:42:01 <mikal> adrian_otto: or is it all in progress? 21:42:02 <adrian_otto> next meeting is 1600 UTC Tuesday. I will be at a conference and need a backup chair to start the meeting if I am unable to get online. 21:42:30 <adrian_otto> we will will follow up with those two topics on ML, requesting input 21:42:37 <adrian_otto> so please keep an eye out. 21:42:41 <mikal> Cool 21:42:43 <adrian_otto> /end 21:42:54 <mikal> n0ano: gantt report? 21:42:59 <adrian_otto> please let me know if you can back me up as chair (anyone) 21:43:28 <n0ano> sure, forklift effort still on-going, cleaning up the interfaces is the main job right now 21:43:47 <mikal> n0ano: what about the refactor bp that was originally in juno-1? 21:43:57 <mikal> n0ano: is that a gantt thing or being done by different people? 21:44:29 <n0ano> which refactor bp are you referring to, that's probably the same as what we are calling forlift these days 21:44:33 <mikal> #link https://blueprints.launchpad.net/nova/+spec/scheduler-lib 21:44:40 <mikal> Ahhh, I see 21:44:44 <n0ano> yeah, that's the one we're working on 21:44:57 <mikal> Ok, cool 21:45:05 <mikal> n0ano: anything else you need to raise or should we move on? 21:45:07 <n0ano> we seem to have scared the guy working on the no-db work, he's abandoned the BP for it 21:45:20 <mikal> n0ano: you mean boris-42? 21:45:39 <n0ano> actually yoriksar took over from boris-42 but it's the same work 21:45:49 <devananda> boris-42 was also on vacation for a few weeks post summit 21:45:49 <mikal> Ok 21:45:59 <devananda> i dont have the sense he's abandoning that effort, last time we talked 21:46:03 <mikal> He was around yesterday, but I suspect is very busy 21:46:16 <devananda> though i hope he's rethinking it :) 21:46:37 <n0ano> devananda, that was the message from the BP but I agree, I don't want to give up entirely 21:46:49 <mikal> We should move on, given the time 21:46:59 <n0ano> OK, that's the big things 21:47:01 <mikal> devananda: how is the ironic nova drive going? 21:47:05 <mikal> driver even 21:48:26 <devananda> mikal: aside from broken for ~36 hours, i need to get back to the specs 21:48:39 <devananda> mikal: they haen't gotten a lot of feedback 21:49:04 <devananda> mikal: shrews is spinning up some unit tests that will watch the internal APIs we're depending on 21:49:10 <mikal> devananda: fair point 21:49:14 <devananda> so that should avoid the kind of gate breakage we got two days ago 21:49:27 <mikal> #action nova-drivers please take a look at the ironic driver specs 21:49:30 <devananda> but as far as the driver itself, i'm not sure how best to proceed -- we need eyes on the specs 21:49:45 <mikal> Yep, let's see if we can improve that over the next week 21:49:51 <devananda> i'm not going to put the time into splicing up the driver itself until those are at least looking close to approved 21:49:54 <devananda> thanks :) 21:50:12 <mikal> Any other sub teams I missed? 21:50:16 <ewindisch> docker. 21:50:20 <tjones> vmwareapi 21:50:24 <cyeoh> nova api 21:50:30 <mikal> Oh, I suck 21:50:33 <mikal> ewindisch: go! 21:50:37 <tjones> LOL 21:50:47 <ewindisch> we have pause/unpause merged and we have patch to be merged for soft-deletes 21:51:10 <devananda> mikal: fwiw, here are the links for ironic specs: https://review.openstack.org/95024 and https://review.openstack.org/95025 21:51:15 <ewindisch> I have an AI to speak more with mikal and mark about our glance integration… so we can fix snapshots 21:51:18 <mikal> ewindisch: I assume you're part of this cinder conversation with adrian_otto ? 21:51:36 <ewindisch> mikal: yes. 21:51:44 <mikal> Cool 21:52:00 <mikal> Perhaps a mail thread about glance is the way to go. It might be easier than getting us all paying attention at the same time 21:52:15 <mikal> Going quickly because of time... 21:52:18 <mikal> tjones: go! 21:52:47 <tjones> ok very quick - last phase 1 refactor patch is up for review, has a +2 from mriedem. https://review.openstack.org/92691 phase 2 will be posted later on today. 21:52:48 <tjones> done 21:53:10 <mikal> Excellent! 21:53:14 <mikal> cyeoh: go! 21:53:17 <mriedem> s/has/had/ 21:53:19 <mriedem> rebase 21:53:39 <cyeoh> ok, so just mostly wanting more eyes on spec reviews - we're a bit blocked on doing anything at all until we get some approved 21:53:54 <cyeoh> https://review.openstack.org/#/c/84695/ - is the v2.1 on v3 api one 21:54:18 <cyeoh> the microversion would could do with more eyes as well - even if its just people saying they don't care which route we take (we just need to choose one!) 21:54:24 <cyeoh> https://review.openstack.org/#/c/96139/1/specs/juno/api-microversions.rst 21:54:34 <dansmith> can we be sure to talk about the microversion stuff at the meetup? 21:54:41 <mikal> Ok, so that's another thing we can try and look at over the next week then 21:54:47 <mikal> dansmith: yes, but we might not get everyone there 21:54:47 <dansmith> because I think we decided we’d punt on how that works, 21:54:49 <dansmith> but we need to do that 21:54:56 <mikal> dansmith: we can do a hangout if needed 21:54:59 <cyeoh> having policy implemented in the REST API I think is a lot less controversial: https://review.openstack.org/#/c/92005/ 21:55:07 <mikal> #action Specs for ironic and nova api need review 21:55:11 <cyeoh> yea unfortunately I won't be able to make the midcycle but happy to attend remotely 21:55:25 <mikal> #action Discuss microversions at the mid cycle 21:55:34 <mikal> cyeoh: yeah, we can work something out 21:55:42 <cyeoh> we can proceed with v2.1 on v3 quite a way without microversions bedded down 21:55:42 <dansmith> we were able to get alaski in last time 21:55:47 <dansmith> which worked okay I think 21:55:49 <russellb> we had a hangout last time, but it was unplanned and just with a cheap tablet 21:55:54 <alaski> it worked pretty well 21:55:55 <russellb> we could probably plan ahead and have a nicer setup 21:55:59 <mikal> I think its actually easier than at the summit 21:56:01 <mikal> And we did ok there 21:56:12 <mikal> Agreed we can make it a bit fancier though 21:56:12 <russellb> alaski: we heard you *really* well 21:56:20 <dansmith> yeah, kindof amazing 21:56:26 <russellb> better than people in the room 21:56:28 <russellb> it was epic 21:56:29 <mikal> #action Determine the AV facilities at intel and how that works for hangouts 21:56:33 <alaski> heh 21:56:40 <mikal> Ok, moving on though 21:56:47 <mikal> #topic Open Discussion 21:57:01 <mikal> Enjoy your three minutes of open discussion 21:57:11 <alaski> I want to bring up https://review.openstack.org/#/c/64769/ with everyone 21:57:32 <alaski> ports are apparently being leaked at times, causing issues for some CI systems 21:57:47 <alaski> I'm not a big fan of this solution, but something needs to be done I think 21:58:09 <dansmith> yeah, this sucks 21:58:11 <dansmith> not this patch, 21:58:13 <anteaya> the bug was filed by one of the hp cloud ops 21:58:25 <anteaya> who is trying to help us fix testing on hp cloud 21:58:25 <russellb> :/ 21:58:25 <dansmith> the problem of trying to maintain this synchronized state 21:58:35 <alaski> it's not clear who owns the ports in this situation 21:58:38 <alaski> so cleanup is a mess 21:58:44 <anteaya> do we need more data on the bug report? 21:58:47 <dansmith> s/mess/disaster/ 21:59:04 <russellb> can't wait until we kill nova auto creating ports 21:59:23 <dansmith> I really think we shoud just do it 21:59:24 <alaski> anteaya: the bug report is pretty comprehensive as i recall 21:59:29 <anteaya> kk 22:00:12 <jogo> if this is coming from hp cloud ops then I think we should consider it high priority 22:00:27 <jogo> the new hp 1.1 cloud (running trunk) is a big part of why things got bad in the gate 22:00:27 <anteaya> that is where it is coming from, yes 22:00:28 <mikal> So, we should all promise to review that change? 22:00:40 <alaski> that would help 22:01:01 <mikal> Ok 22:01:07 <alaski> nova not creating ports is the long term solution, but we could use a stopgap 22:01:08 <anteaya> essentially we aren't able to run much on hp cloud right now 22:01:10 <tjones> alaski: ddn't you propose a patch that would help this situation on friday? 22:01:19 <jogo> alaski: ++ 22:01:27 <mikal> That's us out of time unfortunately 22:01:41 <alaski> tjones: I did. it's getting reviews, but just failed jenkins it looks like 22:02:49 <tjones> https://review.openstack.org/#/c/96955/ 22:03:12 <mikal> Ok, we better end given we're over 22:03:16 <mikal> Thanks everyone for coming 22:03:28 <mikal> Please keep talking in openstack-nova if there's more to discuss 22:03:36 <mikal> #endmeeting