21:00:47 #startmeeting networking 21:00:48 Meeting started Mon Feb 2 21:00:47 2015 UTC and is due to finish in 60 minutes. The chair is mestery. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:49 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:52 The meeting name has been set to 'networking' 21:00:55 #link https://wiki.openstack.org/wiki/Network/Meetings Agenda 21:01:01 #chair armax 21:01:02 Current chairs: armax mestery 21:01:08 * mestery makes armax chair in case his connection dies 21:01:22 * armax sits down 21:01:23 So, we have a heavily focused Kilo-2 agenda today, so lets get at it! 21:01:31 #topic Announcements 21:01:40 Kilo-2 is this week! If things aren't in the merge queue Wednesday, they won't make Kilo-2. 21:01:46 Even if they are in the merge queue, they could miss. 21:01:48 pong 21:02:06 Note: PLEASE BE CAUTIOUS WHEN MERGING CODE THIS WEEK. 21:02:23 Be cautious every week, but especially when we're about to tag and release something. :) 21:02:47 I've been yak shaving the Kilo-2 LP page, I expect to remove more things from Kilo-2 tomorrow. 21:02:50 Who tests the tagged releases mid cycle? 21:03:02 kevinbenton_: All of our users. I hope. :) 21:03:10 kevinbenton_: Seriously, I think distros may do some testing. 21:03:37 kevinbenton_: isn't the integrated gate assurance enough? ;) 21:03:48 o_O 21:04:00 Ah. Most people I encounter are running master devstack if they want to try kilo 21:04:23 Lets move on now. 21:04:24 #topic Bugs 21:04:27 Sorry for derailing. Was just curious who runs the tagged releases of a dev version 21:04:27 enikanorov_ enikanorov__: Hi! 21:04:29 hi 21:05:07 nothing major this week. few high-priority bugs were fixed like race with subnet deletion and DVR job failures 21:05:44 the remaining important bugs are on meeting wiki page 21:05:45 enikanorov_: Excellent! 21:05:48 mostly unchanged 21:05:54 enikanorov_: Thank you. 21:06:06 rkukura: Did you want to bringup your bugs around ML2 port binding here now? 21:06:13 sure 21:06:54 Thanks to help from both armax and matrohon, we tracked down why the HPB patches were failing the DVR tempest tests. 21:07:37 rkukura: not sure I have any merit in this, but the patch it’s on my backlog 21:07:43 I filed two bugs, and https://review.openstack.org/#/c/151913/ fixes both and is also the 1st step of the DVR ML2 schema/logic cleanup. 21:08:24 I think getting this patch (if not the two remaining HPB patches which I’ll update tonight) in for kilo-2 would be ideal. 21:09:06 This patch is also critical for any attempt to use the DVR VLAN support with ML2 mechanism drivers that manage switch based on port bindings. 21:09:40 Thanks for the update rkukura. 21:09:40 Aynway, if any cores can look at it, I’d appreciate it. 21:10:11 rkukura: I was planning to look at it 21:10:20 markmcclain: thanks 21:10:32 Thanks markmcclain, armax, since this is DVR related, may be good to get you to eyeball that first patch (https://review.openstack.org/#/c/151913/) as well. 21:10:53 mestery: yup it’s on my backlog, Vivek also looked at it, which is reassuring 21:11:01 armax: ++, thanks! 21:11:16 Any other bugs the team should discuss here today? 21:11:30 there was an interesting turn of events on the DHCPNAK bug 21:11:39 #link https://review.openstack.org/#/c/152080/ 21:12:12 +1 for that review - I wonder if that also has impact on the big thread kevinbenton_ started about dhcp lease times? 21:13:09 armax: Yikes, lots to digest there. Do yo uahve a good summary for us here 21:13:33 mestery basically we have two compenting proposals 21:13:49 https://review.openstack.org/#/c/152080/ and https://review.openstack.org/#/c/108272/ 21:14:00 the latter is more complex and it’s been in review for a while 21:14:12 the former is the underdog, it’s supertrivial and it does look it fixes the problem 21:14:38 it’s almost too good to be true…so we’re being cautious, carl_baldwin may want to expand a bit, but that’s the gist of it 21:14:46 we originally made a conscious decision to not use init when we move the functionality over from nova 21:15:11 I think armax covered it. 21:15:39 markmcclain: What was the reason for that? 21:15:49 complexity 21:16:01 152080 certainly looks cleaner, looks like haleyb has an email out to the dnsmasq list for some guidance here 21:16:08 required a 2nd level of scripting hooks 21:16:09 haleyb: Any word back yet? 21:16:47 markmcclain: I think we’ve boiled a lot of complexity out of 108272. However, if there is no problem with 152080 then it will be the clear winner. 21:16:54 Regardless, sounds like we'll need to wait to hear back before making a decision here. 21:16:59 carl_baldwin: ++ 21:17:01 mestery: ++ 21:17:34 OK, one more opening for bugs today before we move on. Any other bugs to discuss today? 21:18:13 #topic Docs 21:18:18 sc68cal: Hi! Thanks for subbing for emagana. 21:18:23 np 21:18:49 For those interested, the meeting notes from the networking guide meeting are here 21:18:56 #link http://lists.openstack.org/pipermail/openstack-docs/2015-January/005776.html Networking guide meeting notes 21:19:22 We have a number of scenarios that we have put together, and it would be great for neutron folk to review, especially the DVR stuff 21:19:40 make sure all the packet flows are correct and everything is technically sound 21:20:01 #info Please review the networking guide and provide feedback 21:20:21 #link https://github.com/ionosphere80/openstack-networking-guide Networking guide scenarios 21:20:29 sc68cal: Swami is your man 21:20:45 agree - he's been attending the meetings iirc 21:20:50 armax: mestery : I have already reviewed the alpha version. 21:21:01 Awesome! 21:21:02 Swami: nice 21:21:17 that's all I have, 21:21:23 sc68cal: Thanks! 21:21:34 #topic Plugin Decomposition Status and Updates 21:21:43 This is a standing item 21:21:52 For people to discuss issues around plugin decomposition 21:21:59 And for armax to provide us with other udpates here :P 21:22:08 Last week, we spent a lot of time here. 21:22:09 I have a linked spreadsheet on the bp page 21:22:15 https://blueprints.launchpad.net/neutron/+spec/core-vendor-decomposition 21:22:20 we had a question in infra earlier today about a spec for testing 21:22:33 which I use to capture the status updates 21:22:33 we redirected the querent to -neutron 21:22:44 #link https://docs.google.com/spreadsheets/d/1OQN0VlKzKC1gYlxgalQXKDTGGWogWFRtD3S5OpepsX4/edit?pli=1#gid=0 21:23:01 This is great, thanks armax! 21:23:08 anteaya: I'm sorry I must have missed that part? was the question about testing decomposed plugins? 21:23:16 salv-orlando: yes 21:23:17 I update it as soon as I get to the related reviews that are in my pipeline 21:23:36 Looks like we ahve three completely done now (Mellanox, Midokura, and ODL). 21:23:39 so the current snapshot may not be necessarily updated to the minute 21:23:49 #status 3 Plugins completely decomposed (Mellanox, Midokura, and OpenDaylight) 21:23:53 OK 21:23:55 anteaya: what cluss of tests exactly? 21:23:56 how many to go? 21:24:01 dougwig: 200 21:24:03 #info 3 Plugins completely decomposed (Mellanox, Midokura, and Op 21:24:08 lol 21:24:13 salv-orlando: that was what we didn't know, hence the redirect 21:24:14 :) 21:24:26 mestery: there may be still loose ends 21:25:05 anteaya: I see. From our perspective I have seen different teams setting up unit tests and integration tests on 3rd party repos 21:25:21 salv-orlando: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-02-02.log 2015-02-02T11:44:09 Hello all! Sorry if I missed something. There is a blueprint https://blueprints.launchpad.net/neutron/+spec/core-vendor-decomposition What is going to happen with Third Party CIs? Will they continue to test patchsets in openstack/neutron or they should start 21:25:22 testing patchsets of appropriate repo at stackforge? 21:25:37 anteaya: the decomp should not change that 21:25:38 anteaya: they need to keep testing neutron patches 21:25:41 armax: ++ 21:25:47 they can use our testing system for stackforge repos 21:25:56 but what to test comes from neutron 21:25:59 anteaya: This is what I'm doing with vmware plugins - even if decomp is not completed - test all that can affect my plugin in openstack/neutron and report there 21:26:00 anteaya: 3rd party CI system are supposed to test whatever change impacts the solution they are meant to validate 21:26:08 plus test everything that goes in the plugin repo 21:26:23 armax: great, is there some documentation we can point them to in future? 21:26:27 salv-orlando is right, as a matter of fact when I was in charge of the VMware CI, I was even testing Tempest changes 21:26:36 not sure if salv-orlando is continuing the tradition 21:26:48 armax: meh meh meh gneh gneh gneh 21:26:58 anteaya: it’s in the blueprint spec, if I recall correctly 21:27:03 salv-orlando: is that a no? 21:27:10 okay I will look thanks 21:27:16 salv-orlando: or is it your cat typing? 21:27:24 armax: that's a "armax want to moan and whine because that's what he does best" 21:27:35 * dougwig is feeling the vmware love. 21:27:41 salv-orlando: ah, ok, thanks for claryfing this for the sake of others :) 21:27:44 tempest and devstack are still tested with the same paths you specified 21:27:56 salv-orlando: oh, ya, devstack too 21:28:03 salv-orlando: good lad 21:28:04 From the packaging perspective, does it make easier if the vendor plugin stays on stackforge? Or it should not matter! 21:28:22 rms_13: I think it doesn't, but I'm not a packager 21:28:27 rms_13: it does not matter 21:28:33 I think packagers are happy as long as they can git pull the stuff 21:28:49 Or even pull a released tarball 21:29:14 Technically yes, but not sure if we has a community have a feedback from redhats or canonical 21:29:36 rms_13: ihar provided this type of feedback on the spec 21:29:36 rms_13: asking on the ML won't hurt 21:29:37 one reminder I have for people is to co-reference the bp/core-vendor-decomposition on the changes they post to gerrit 21:29:59 so that that gets captured on the Launchpad BP page and it does not get lost 21:30:18 armax: sounds good 21:30:38 Any other questions around plugin decomposition from anyone? 21:30:42 Good or bad experiences? 21:30:46 mestery: I am not completed with the devref 21:30:47 Anyone send armax flowers yet? :) 21:30:51 armax: lol 21:30:53 whoops 21:31:03 mestery: not flower, but perhaps bullets in an envelope 21:31:07 or pigs' heads 21:31:11 lol 21:31:16 mestery: so long as I don’t see dubious packages out my front door, I am happy 21:31:34 so I had a little question 21:31:36 armax: Hint, if you open the door to a paper bag on fire, don't step on it. 21:31:46 :) 21:31:58 mestery, salv-orlando thanks for the colorful description of the packages I was alluding 21:32:02 You guys are having too much fun :-) 21:32:13 some decomposed repos have very little patch traffic compared wiht openstack/neutron 21:32:25 salv-orlando: As it should be, no? 21:32:37 and some do not implement periodic CI checks, mostly because there is no translation patch for them! 21:32:43 or requirements patch for that patter 21:32:51 this means that when the release day comes 21:33:06 we might have plugin repos which have not done a CI run in like 7 or 10 days 21:33:12 what do we do about that? 21:33:20 salv-orlando: as we mentioned earlier, the CI is supposed to test the Neutron changes 21:33:26 salv-orlando: just as it used to 21:33:37 salv-orlando: In my opinion it should be reported 21:33:41 armax: correct, sorry about that. In a perfect world that it correct. 21:33:48 in a real world, we will have such plugins. 21:33:51 what do we do? 21:33:54 deprecate anyway? 21:34:02 we need periodic jobs 21:34:13 CI should remain unchanged otherwise plugin and vendor compatibility will be out of date 21:34:16 salv-orlando: Well, we'd have to let the plugin users know the status of their plugin backend at release. 21:34:24 So what about a page listing the last known good CI run? 21:34:25 anteaya: is this is something supported by infra at this time? 21:34:27 At release? 21:34:29 marun: i don't see why. we should just enforce that they test neutron changes 21:34:31 salv-orlando: Arista code is out of neutron tree - but, our CI is testing every neutron patch - is that not sufficient? 21:34:40 Sukhdev: +1 21:34:49 a deprecation decision needs to be taken on a case by case basis, 21:34:50 Sukhdev: I am not blaming A or B or C 21:34:55 Sukhdev: +1 21:34:59 periodic jobs work from openstack/* so I'm assuming the smae machinery is in place for stackforge/* 21:34:59 marun: sweston is working on a tool called radar 21:35:01 kevinbenton_: um, race conditions? 21:35:02 salv-orlando: what about D? 21:35:06 but it is in development 21:35:11 Can't understand what those +1s are for? 21:35:17 anteaya: cool 21:35:23 so doubtful it will meet your needs in time 21:35:26 well it sound like I'm appearing stupid 21:35:32 and ignorant, which I probably am 21:35:33 anteaya: well, better late than never 21:35:34 salv-orlando: no 21:35:42 but it’s not certainly something we’re thinking of right now…this is a larger discussion that needs to happen regardless of the decomp 21:35:50 salv-orlando: just wondering why testing neutron changes wasn't good enough 21:35:58 but maruns point about a race condition is valid 21:36:04 armax: so that is not regulated by the spec? 21:36:08 markmcclain: yes stackfore can use our periodic pipeline 21:36:23 kevinbenton_: because some CIs might either fail in doing periodic jobs, or might be failing at all ;) 21:36:48 * salv-orlando hopes he's found a loophole in the spec 21:36:55 the correctness of a 4rd party CI sytem goes beyond the decomp 21:36:56 salv-orlando: right, but how can we tell where a 3rd party is doing periodic jobs? 21:36:59 *3rd 21:37:20 armax: decomp just says they have to test all neutron patches, right? 21:37:57 kevinbenton_: that is my question too. I guess we have to police those. It's probably not that different from before. 21:37:58 * armax pulls the exact wording 21:38:13 armax: that's not necessary, we're wasting precious time. 21:38:14 * mestery hands salv-orlando and kevinbenton_ their neutron CI police badges 21:38:23 3rd Party CI will continue to validate vendor integration with Neutron via Tempest (e.g. functional testing); 3rd Party CI is a communication mechanism. This objective of this mechanism is threefold: 21:38:30 * anteaya applaudes 21:38:30 … 21:38:31 mestery: emagana already has that role 21:38:38 salv-orlando: He needs some deputies 21:38:46 salv-orlando: you go read the reast 21:38:47 * anteaya nods 21:38:48 rest :) 21:38:55 mestery: conflict of interest for both salv-orlando and myself, can't do it ;) 21:38:59 http://specs.openstack.org/openstack/neutron-specs/specs/kilo/core-vendor-decomposition.html 21:39:01 lol 21:39:16 bottom line is: if a CI system goes tits up 21:39:17 by the way, while armax show again that the system is perfect and the people are the issue, I simply suggest that where in the previous versions we had full control over non-CI compliant plugins 21:39:22 that has nothing to do with the decomp 21:39:34 why? 21:39:43 now we probably need to act differently, or perhaps not? 21:40:02 we have full control because of the neutron in-tree vendor integration part 21:40:04 salv-orlando: well we can still kick the integration point out of the tree :) 21:40:05 I guess at the end of the day we just remove from tree the integration library for 21:40:16 the plugins which are not running CI or are evidently broken? 21:40:22 salv-orlando: ++ 21:40:30 ok let's say it's sorted then 21:40:31 That's the stick salv-orlando 21:40:34 if and when folks want to take everything out, they are on their own, just as it is right now 21:41:16 armax: Right, decomp doesn't change that. 21:41:17 mestery, armax: It's not like I want to be an annoying moron, but I've realized that the decompisition increases plugin breakage chances 21:41:36 I was indeed considering tweaking VMware CI to post also unit tests results on openstack/neutron 21:41:45 salv-orlando: no, it increases the change of transitionary breakages 21:41:52 salv-orlando: that’s different 21:41:54 :) 21:41:59 subtle difference 21:42:19 armax: thanks for the explanation. Again the system is perfect, humans and their terrible usage of words are the problem 21:42:31 salv-orlando: we're done nannying. 21:42:38 salv-orlando: no system is perfect 21:42:49 just like humans 21:42:54 (as an aside: if any unit test changes as a result of a different plugin, i'd think that was a broken unit test.) 21:42:57 who designed the system in the first place 21:43:06 lol 21:43:31 OK, lets move on. 21:43:33 We've beaten this horse to death now I think. 21:43:33 meh I don't anything to add 21:43:36 salv-orlando: thanks for the opportunity to reflect on this phylosofical matter 21:43:39 dougwig: not from a different plugin change, but a core change 21:43:54 salv-orlando: I appreciate your thoughts, especially as it relates to agitating armax. :) 21:44:00 right, but jenkins covers that? i'm missing the point of running those outside. 21:44:07 mestery: are you accusing me of trolling 21:44:08 i'm happy to move on, though 21:44:10 OK, lets move on to the next topic now. 21:44:13 salv-orlando: :P 21:44:19 I can bring trolling to whole different levels if you want ;) 21:44:27 #topic Status of DVR test job and update from jogo on multi-node devstack 21:44:29 salv-orlando: we can trust you on that one! 21:44:32 On the fly agenda item! 21:44:34 From jogo! 21:44:40 and clarkb 21:44:42 lol 21:44:48 We're nothing if not adaptive to change here 21:44:55 * armax has a halo 21:45:01 jogo clarkb: We can do multi-node now for DVR job? 21:45:22 so clarkb and I have been working on getting multi node devstack gate working. we have tempest smoke passing 21:45:44 yay multi-node 21:45:47 jogo clarkb: Yay! 21:45:50 so the question is as mestery said, should we switch the DVR job to multinode or is that premature being single node doesn't work 21:45:59 great!! +1 21:46:01 armax: ^^^ :) 21:46:18 jogo: I have a feeling figuring it out on single node will make things much easier to switch to multinode 21:46:42 jogo: just having wrestled with a lot of the networking going on in multinode I expect that half working will only get worse under weird network setup 21:46:50 clarkb: possible but single node DVR as you said isn't very useful and there have seen odd edge cases that only happen on single node in the past 21:46:50 jogo: single node DVR works fine 21:46:50 jogo: easier to debug these things in isolation 21:47:24 armax: define 'fine' ;) 21:47:27 can I troll here too? I think multi node might expose failure modes of which we have no idea at the moment, so it is useful regardless of single-node failures. On the other hand, if the issue is consumption of testing resource, that;'s another matter 21:47:38 lol 21:47:53 salv-orlando: no thats not the issue. The big issue so far has been lots of things break when we go to multinode 21:47:55 armax: I think we should add a multinode job non-voting once the single node is safe to allow to vote 21:48:08 and then we can replace single node with multinode when mn becomes stable 21:48:14 I have spent a few months now unbraeking them and if dvr doesn't work reliably in single node we should just make that work first to avoid even more broken 21:48:23 either way, people should just pay attention to non-voting jobs just as much as the voting ones 21:48:24 clarkb: agreed 21:48:24 marun: we can also add multinode to experimental if that helps 21:48:24 if it does work then great 21:48:26 just saying.. 21:48:30 jogo: +1 21:48:41 armax: ++ 21:48:47 armax: I'm not sure that's reasonable. 21:48:52 armax: I chased people breaking the full job for 6 months. It's your turn now ;) 21:49:13 unless the chance of false-positives is extremely low (and I'm not sure how anyone could argue that for dvr) 21:49:21 side note: tempest-full multi node fails for neutron 21:49:27 salv-orlando: that’s fine…I am just making a point 21:49:41 jogo: can I see logs for that 21:49:47 jogo: is it in the experimental queue? 21:49:51 So, looks like we should focus on making the existing job working flawlessly before going muilti-node? 21:50:00 if tempest-full mn fails for neutron it's likely to fails for DVR as well 21:50:03 * mestery tries to summarize the discussion 21:50:07 salv-orlando: http://logs.openstack.org/04/136504/14/experimental/check-tempest-dsvm-neutron-aiopcpu/d700407/ 21:50:16 mestery: +1 21:50:24 mestery: I don't think it needs to be flawless but it should have reliability that lines up with not dvr 21:50:27 jogo: do you know what aiopcu stands for? 21:50:32 mestery: for your summary, we need also to fix the full job on multi node possibly as a precondition for the DVR job 21:50:37 thanks jogo 21:50:44 all in one p*? CPU 21:50:47 plus* 21:50:54 jogo: thanks for the update 21:50:57 flawless. openstack. that's quite the bar to new tests. 21:51:04 salv-orlando: Good point 21:51:17 * mestery realizes flawless was the wrong word here 21:51:21 so sounds like the conclusion here is, hold off on even setting up a multi node DVR 21:51:23 How about equal footing to the non-DVR job? 21:51:25 Better? 21:51:34 still a lot of work to do with single node first 21:51:49 mestery: is that accurate? ^ 21:52:14 jogo: Well, there is work, not sure on the level. 21:52:19 * salv-orlando got so addicted to faulty openstack that I've installed cyanogenmod on my phone just to see it reboot from time to time 21:52:31 salv-orlando: lol 21:53:01 OK, lets move on now. 21:53:03 Tahnks all 21:53:06 mestery: cool, once you guys are ready for multi node DVR as experimental ping me or clarkb and we can set it up 21:53:08 thanks 21:53:11 #topic nova-network to neutron migration 21:53:16 anteaya markmcclain: Hi! 21:53:21 jogo: guys -> y'all 21:53:24 I think it would make sense to have multi-node on the check queue 21:53:24 mestery: hey 21:53:32 the experimental queue takes forever to execute 21:53:34 anteaya: Any updates for us today? 21:53:41 so status report 21:53:45 #link https://review.openstack.org/#/c/147723/ 21:53:51 has a new patchset 21:54:01 nice 21:54:02 that is the parent of the two part spec 21:54:16 we are arguing our way closer to something that can merge 21:54:26 reviews appreciated 21:54:35 we also have prototype code 21:54:46 #link https://review.openstack.org/#/c/150490/ 21:54:49 is the proxy 21:55:01 #link https://review.openstack.org/#/c/148260/4 21:55:06 is the db migration 21:55:34 Lots going on here it appears anteaya. 21:55:35 thanks to salv-orlando and assaf and obondarev and jlibosav here 21:55:35 that sounds good to me... btw is obondarev around? 21:55:47 He's sick 21:55:52 mestery: there is movement which is good to see yes 21:55:57 so please review 21:56:13 I would like to get part one of the spec merged soon 21:56:14 sc68cal: thanks for the info (troll me would have said: "are you his doctor?") 21:56:24 rofl 21:56:28 :) 21:56:37 and then work on the wip patches which are needed to get nova agreemtn around the second part of the spec 21:56:40 armax: it would be pretty costly, though, to add 2 dsvm nodes per check patch 21:56:40 any questions? 21:56:41 OK, I encourage folks who are interested in helping with the migration work to reach out to anteaya and attend the weekly meeting 21:56:47 thank you 21:56:47 I think we should start playing around with the migration part and see whether there is any obvious hiccup there 21:56:49 done 21:56:56 salv-orlando: yes, please 21:56:58 Thanks anteaya. 21:57:01 #topic Open Discussion 21:57:06 marun: you already know what I’d tell you…drop the non-sensical -2 mirror jobs :P 21:57:08 Feel free to let your ideas flow for 3 minutes 21:57:09 :) 21:57:10 mestery: sorry i was late, picked a bad time to shovel show, but heard back from Simon on dnsmasq option, will pass it along to people via reviews/bug 21:57:17 armax: +100 from me on that one 21:57:24 marun: and get some nodes repurposed to do something more useful 21:57:24 haleyb: ++, thanks! 21:57:41 armax: maybe that's good enough ammunication?: 21:57:45 ammunition, sorry 21:58:01 marun: for the -2 jobs I'd say now they're there more for historical reasons when we needed 4 runs to have some confidence in a neutron patch 21:58:07 marun: meaning drop the -2 clone jobs in favor of multi-host? 21:58:19 salv-orlando: now I am the one trolling... 21:58:23 those jobs are useless 21:58:29 the other side of the question is: does the rest of the community feel like they can trust a patch with only 2 jobs passing? 21:58:33 and you know it as much as I do 21:58:34 salv-orlando: lol 21:58:37 salv-orlando: still trolling, eh? 21:58:39 lol 21:58:46 salv-orlando: there are 8 jobs, anyway 21:58:56 mestery: I only have one more OVSD API left method to implement for the native OVSDB interface for ovs_lib. I plan on having something reviewable for it tonight. 21:58:58 salv-orlando: so it would be 'only 6 jobs passing' 21:59:06 otherwiseguy: Thank you sir! :) 21:59:13 frankly the -2 jobs become as useless as the non-voting ones 21:59:17 * mestery starts to worry when we get down to 5 jobs 21:59:40 OK folks, thanks for attending the meeting today! 21:59:43 marun: methinks we can remove them 21:59:48 Please, please, please focus on Kilo-2 reviews for the next two days! 21:59:53 mestery: Just wanted to point out that we are seeing issues where neutron changes break *aaS repo tests. 21:59:53 And enjoy your week! 21:59:55 salv-orlando: who do we talk to? 21:59:58 even if they do unveil races, no-one pays attention to them 22:00:01 #endmeeting