16:00:05 #startmeeting Fuel 16:00:06 Meeting started Thu Jun 19 16:00:05 2014 UTC and is due to finish in 60 minutes. The chair is vkozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:07 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:09 The meeting name has been set to 'fuel' 16:00:14 #chair vkozhukalov 16:00:15 Current chairs: vkozhukalov 16:00:28 Hi 16:00:42 hi 16:00:44 Ok guys and girls, hello everyone 16:00:51 who is here? 16:00:52 o/ 16:00:54 hello 16:01:28 hi all 16:01:30 hi 16:01:38 hi all, mihgen do you have any announcements for us? 16:01:42 and as usually agenda is here https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda 16:01:48 #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda 16:02:06 hi all 16:02:07 yep I do a bit 16:02:12 I'll try to be short this time 16:02:15 #topic announcements 16:02:23 #link https://wiki.openstack.org/wiki/Fuel/5.1_Release_Schedule 16:02:29 hi 16:02:33 we finished first iteration in 5.1 16:02:40 results are pretty good overall 16:02:47 (in spite of anything )) 16:03:05 we will go over features status 16:03:09 demo of upgrades was quite impressive :) 16:03:24 slowpokes, I've spent an hour for doing presentation :) 16:03:28 yeah it's weird to see something work on top of my code :) 16:03:29 ok, let's go through statuses 16:03:45 yep, actually I'm thinking about running public demos at the end of every iteration instead of IRC meeting 16:03:54 replacing IRC meeting on webex demo 16:04:00 +1 on that 16:04:01 once in two weeks 16:04:45 that would be great according to the fact that we have more and more people contributing to Fuel 16:05:03 ok. I'll ask in ML then and see how it's perceived 16:05:20 it breaks the IRC tradition but I believe some people could be persuaded 16:05:22 ok folks another important thing in the release is bugs status 16:05:43 #topic bug status 16:05:54 bug squashing day this week wasn't a full success 16:05:57 frankly I was expecting better results for bug squashing day 16:05:59 yep 16:06:06 we cleaned up bugs in 5.1 a little, but didn't really fix many of them 16:06:20 unfortunately python & ui teams didn't work on it, library was good and qa is best in adding more bugs) 16:06:22 drop in number is mainly due to cleaning out old incomplete bugs 16:06:34 I guess one of the things was nobody knew it was bug squashing day 16:06:43 angdraug: you are the primarily assigned to bugs in this release 16:06:50 I thing we need to loudly announce these things the day before 16:06:55 at least 16:07:01 maybe we need to change the format of bug squashing day a little bit 16:07:04 meow-nofer: it was announced multiple times 16:07:31 I'll point you to IRC logs, to openstack-dev logs and I was personally walking in the room loudly speaking about it.. 16:07:37 mihgen, first time I heard about it was that day's morning 16:07:37 I'm sorry that you didn't hear 16:07:42 mihgen, and that's not only me 16:07:48 #link http://lists.openstack.org/pipermail/openstack-dev/2014-June/037746.html 16:08:03 meow-nofer: what kind of notification we should do then? 16:08:05 mihgen, sorry, will do better next time 16:08:08 maybe sitting around the table and having kind of nano summit would be more impressive and motivating 16:08:20 mihgen, I don't know, maybe go through bugs the day before 16:08:40 bug squashing day is for bug squashing 16:08:41 meow-nofer, we organized a fuel-library bug triage at the start of the day and proceeded from there 16:08:45 mihgen, and then kick the crap out of them on next day 16:08:51 i think it is a bad idea anyway to interrupt all current tasks and go fix the bugs 16:09:03 vkramskikh: disagree completely 16:09:10 there are still 350 bugs now 16:09:14 vkramskikh, too bad. we need to do bugs at some point mid-cycle. not after feature freeze 16:09:17 if we don't squash them 16:09:22 then it means we don't release 16:09:39 so stop all of your activities when bug squash day comes, and start fixing bugs 16:09:47 fixing bugs when they are found is much easier than 3 months later 16:09:50 it's common practice, and openstack follows it strictly 16:09:59 we should not keep big backlog of high priority bugs 16:10:04 so my hard opinion on this - let's do it this way 16:10:08 vkramskikh, meow-nofer we will have another bug squash next Tuesday. Let’s do our best 16:10:09 low priority, fine, but we have over 100 high&critical bugs 16:10:20 if you don't want to do, your features will be stopped from accepting 16:10:27 mihgen, I agree, but let's start with triaging the day before or in the morning 16:10:31 breaking more and more master is unacceptable, seriously 16:10:32 ok, we'll try to do our best 16:10:42 meow-nofer: we can do it, yep 16:10:43 mihgen, I think this will do the trick and swith our brains to happiness 16:10:44 it was really unclear what it means 16:10:59 you can dedicate part of the day for bug triaging then 16:11:07 and next day for squashing, for example 16:11:21 vkramskikh: unclear what? 16:11:42 so my proposal is to do squashing on Tu 16:11:43 mihgen: unclear that we should stop all other activities 16:11:48 I think we need to have more time set aside for sustaining and take time out of every week, to work bugs 16:12:10 ok, guys, let's move on, we have plenty of other topics 16:12:22 and keep doing it every week unless we meet a better situation which I'll allow angdraug to decide whether it's good or not, as I'm taking vacation for next 2 weeks 16:12:51 vkramskikh: sorry if it was not clear in the email, but I believe it was 16:12:51 just for the note, it would be drastically less bugs for squash day (at least by its status updates and triaging part), if *every one* of us would have opened and read it and comment or even RCA by the snapshot logs - as far as he recieve an email notification 16:13:02 my primary criteria of good is below 20 high priority bugs open at any time 16:13:16 every such an update could take 2-5 min of your daily time 16:13:28 vkramskikh: if something is unclear from email, there is Reply button and you can ask question ;) 16:13:55 ok let's move on 16:13:58 moving on to code reviews? 16:14:01 #topic code reviews status 16:14:08 angdraug: you will speak? 16:14:14 we have a huge backlog of unreviewed commits 16:14:25 I have posted a link to review inbox that helps prioritize them 16:14:47 our ratio of reviews vs posting new patches is wrong and we need to find a way to do more code reviews 16:14:59 #link https://review.openstack.org/#/q/project:%255Estackforge/fuel-.*+is:open,n,z 16:15:06 dpyzhov: thx 16:15:21 #link https://wiki.openstack.org/wiki/Fuel#Development_related_links 16:15:44 the link from angdraug is really neat, you guys should try it 16:16:24 angdraug: wow 16:16:30 if you have gerrit reviews where you are reviewer, make sure you go through more of them in a day than you submit your own patches 16:17:22 especially if it's review requests from outside fuel core 16:17:45 angdraug: yep, that's a cool thing 16:17:58 angdraug: +1 for special treatment of reviews from outside 16:18:18 people from outside core have much harder time in any project, including fuel 16:18:35 so if you have an external review in your inbox, start with that 16:18:53 good karma will catch up to you when that contributor fixes your bugs for you 16:19:12 I'm done about code reviews 16:19:25 angdraug: thx 16:19:28 moving on 16:19:59 #topic Upgrade 5.0 -> 5.0.1 16:20:09 Hi 16:20:12 Here is demo of fuel upgrades + OpenStack patching features 16:20:18 #link https://docs.google.com/file/d/0B7I3b5vI7ZYXYVlXYldoc1Z0Q0U/edit 16:20:24 evgeniyl: great 16:20:46 There are several unresolved issues for upgrades, like cobbler data migration, I forget to mention that we need to run puppet on the host system to upgrade fuelcilent and dockerctl. 16:21:10 It should not take much time and looks easy 16:21:51 Our QA team is testing upgrades 5.0 -> current master 16:22:22 Also there are no patches which related to upgrades on review 16:22:51 We have only part of make system (for openstack pacthing) which is not in the master 16:22:55 evgeniyl: that's cool. we may want to go with idea of distributing upgrade as ISO.. if bundle so large anyway 16:23:56 mihgen: how large? 16:23:58 yeah, actually current version of build system builds iso and then retrieves all necessary data from upgrade 16:24:04 2gb for upgrade tarball is large 16:24:05 over 2G 16:24:32 s/from upgrade/for upgrade tar-ball/ 16:24:48 I really think we should consider after 5.1 ways to adapt this upgrade to include a manifest of files to "copy from this previous major release" 16:25:09 unless we can get the upgrades as diff's the only resonable option is having the install ISO double as upgrade media 16:25:33 anyway I like that idea that you can just pop in the newer iso and it will read the migration data 16:25:46 evgeniyl: we need to think about optimizations 16:25:50 I think we should not implement diff upgrades, it will be difficult to support 16:25:54 xarses: yep 16:25:56 seems elegant almost 16:26:01 xarses, this is doable and I know a lot about what's needed, but it's going to be a ton of customized anaconda work 16:26:44 it's better to fake a real upgrade from media and still do it in a special boot from disk 16:26:58 we can just use iso instead of tar 16:27:23 all you need is a conversion script mounts that iso, and pulls the files out of it in the same order ugprade script expects it 16:27:27 angdraug: that's what we are talking about with xarses 16:27:44 Like i said, if we need to ship a 2gb upgrade tarball, then we will get frowned at. People will want to know why its not a diff. If we have the upgrade also be a clean install method, then the issue is swept under the rug 16:28:04 ok evgeniyl let's consider this 16:28:06 and we get to avoid creating diff tarballs. Which do we prefer 16:28:53 #action evgeniyl do research using iso instead of tarball 16:29:03 moving on 16:29:17 #topic Patching of OpenStack 16:29:34 I'm not sure if Igor here. 16:29:35 who has status details about this? 16:29:45 I can provide status update 16:29:59 evgeniyl: go ahead 16:30:30 We were able to run patching process, also Igor tested rollback today 16:30:50 We have part of make system which on review now 16:31:14 As Igor told we have some problem with upgrading of centos + nova installation 16:31:24 Centos + netutron works fine 16:31:25 evgeniyl: does rallback work? 16:31:39 Yeah, it works 16:31:44 evgeniyl: thx 16:31:49 moving on 16:32:02 #topic LSI 16:32:17 #link https://blueprints.launchpad.net/fuel/+spec/lsi-raid-controllers 16:32:18 we had a meeting with LSI guys 16:32:31 anyone from LSI team around to get update? 16:32:35 on if they can integrate into Fuel as a plugin 16:32:47 meow-nofer: oh please provide update if no one is here 16:32:50 I did write some meeting notes to openstack-dev 16:33:08 yep I've seen. I didn't see code yet.. 16:33:10 meow-nofer: we have 1 iteration left before feature freeze 16:33:12 in gerrit 16:33:27 so, there is no real way for both us and them to provide fully-functional Fuel plugin on this release 16:33:37 yep risk is very high for skipping it out, design doc is huge 16:33:59 but we can at least try to simplify our future integration, I've already provided two ways this can be done 16:34:03 I think given the timing, making a full plugin out of it in 5.1 is impossible 16:34:39 I saw the nailgun agent tweaks and didn't like it - I think it should be in separate file, not changing agent with inclusions of lsi external commands over it, mixing it all 16:34:43 either they're doing it like "half-plugin", which is much easier for us to work with, but also seems llike a very ugly solution 16:35:02 ok let's move on, we need to have code first anyway to make a final decision 16:35:09 or they are just writing their code in a way it will be simple to move into plugin 16:35:09 ok, status is quite clear, very risky 16:35:14 yep 16:35:19 moving on 16:35:31 #topic Mellanox 16:35:51 Hi 16:36:01 Im from Mellanox 16:36:01 vkozhukalov, you skipped plugins, I moved them upper 16:36:08 oh, nevermind 16:36:15 nuritv: hi 16:36:20 nuritv, hello 16:36:52 so, we are progressing with our integration 16:37:07 meow-nofer: sorry, next report will be your 16:37:11 I've seen packages ready? 16:37:14 ok sorry 16:37:39 nuritv, just continue 16:37:51 mihgen: yes, we have CentOS packages ready 16:37:56 nuritv: you mentioned that all packages required for your integration are asl/bsd licensed? 16:37:59 we will have Ubuntu shortly 16:38:16 bsd/ apach2 16:38:38 nuritv: I hope we are not gonna install them by default on all systems? 16:38:42 nuritv, I saw binaries. Will you be able to submit srpms too? 16:38:50 nuritv: have you written a spec for your feature? 16:39:05 are we pulling them into our mirrors or implementing a way to pull external package repositories into fuel master? 16:39:07 mattymo: no, only when Mllanox drivers are enabled 16:39:09 only on those which have mellanox hardware, could be discovered by agent 16:39:54 angdraug, I believe we'll ship them with Fuel but will be an optional install (like Ceph) 16:40:15 vkozhukalov: we have a fully design document by aviramb 16:40:20 with ceph, we're running a 6-month old LTS version thereof 16:40:54 nuritv: is it public? could you share a link? 16:41:19 nuritv: i mean for the spec 16:41:43 are mellanox packages LTS? as in, major upgrades are not expected too often in the future? 16:41:49 #link: https://docs.google.com/document/d/10oynt0-mr_bsOuU1hvPLQZgXIjNzjkloGx1CI01XVL8/edit?disco=AAAAAJprRQI# 16:42:00 nuritv: great, thx 16:42:05 #link https://blueprints.launchpad.net/fuel/+spec/mellanox-features-support 16:42:20 nuritv: it is important to have links in irc log 16:42:23 moving on 16:42:36 #topic Nailgun plugins 16:43:01 meow-nofer: your turn 16:43:07 so, I did a little presentation on current plugins state, but it's more for a speech, not reading 16:43:10 https://drive.google.com/a/mirantis.com/file/d/0B5VvW7EEta1WbnMwREEzVlFSQzg/edit?usp=sharing 16:43:21 folks let's be prepared for your topic, pre-record stuff you gonna write here, we need to move faster - too many topics to cover 16:43:27 #link https://drive.google.com/a/mirantis.com/file/d/0B5VvW7EEta1WbnMwREEzVlFSQzg/edit?usp=sharing 16:43:57 as for now, we are together with OSCI team testing new versions of packages 16:44:09 meow-nofer: will take a look at your presentation 16:44:15 meow-nofer: fun presentation :) 16:44:25 I'm going to install Ceph from "plugin" 16:44:34 meow-nofer: can you allow comments there? 16:44:39 meow-nofer: any other comments? 16:44:42 mihgen, sure 16:44:57 mihgen, ready 16:45:05 so 16:45:17 meow-nofer: are you done? 16:45:19 meow-nofer: we need pieces merged in 5.1 16:45:22 let's move on 16:45:27 gotta make working ISO, gotta install ceph and then will work with Vitaly on UI patrt 16:45:35 meow-nofer: expecting design specs from you on this 16:45:37 also, working on specs for 5.1 16:45:40 #topic Monitoring 16:45:43 mihgen, yep, I'm on it 16:45:44 meow-nofer: ok thanks 16:45:53 #link https://blueprints.launchpad.net/fuel/+spec/monitoring-system 16:46:14 akislitsky not here 16:46:16 let's move on 16:46:19 if akislitsky is not here, I can do some update on this, too 16:46:21 looks like here is not here 16:46:26 but not so much 16:46:27 yep 16:46:44 #topic Neutron ML2 16:47:00 #link https://blueprints.launchpad.net/fuel/+spec/ml2-neutron 16:47:14 xarses: here? 16:47:17 yes 16:47:29 I've re-written the corosync resources from our old neutron plugin into a composition layer than can be cleanly applied to upstream puppet-neutron module. I'm currently stuck trying to get it to cleanly handle dependencies. As everyone who works on this notes, it sucks to get right. 16:47:41 I'd like dilyin and xenolog (and anyone else) to look over https://github.com/xarses/fuel-library/compare/bp;ml2-neutron?expand=1 and see if they can help find the problem. I will create ML topic with more information on how to test. 16:47:59 We need to agree on how to deal with sanatize_network_config. As it stands we can't use it in the upstream module with out creating a composition layer to plug it back in, or we won't be able to maintain upstream. The main issue I have with maintaining it is that it is its self a composition and config generation layer that is embedded in ruby that no one expects for and its results are only available at runtime. 16:48:22 xarses: your typing is really good 16:48:28 pre-wrote =) 16:48:34 he came prepared, see above :) 16:48:59 sanitize or how you call is really under concern 16:49:06 another problem with it is that it overrides all neutron settings from both puppet-neutron and neutron.conf 16:49:07 guys, any questions on ML2? 16:49:07 we need more people to discuss it 16:49:32 #link https://review.openstack.org/99807 16:49:33 xarses: link to design spec pls 16:49:43 angdraug: thansk 16:49:44 #link https://github.com/xarses/fuel-library/compare/bp;ml2-neutron?expand=1 16:50:08 #link https://review.openstack.org/#/c/99807/ 16:50:11 ok I don't have questions, we just need to read and decide in spec 16:50:21 moving? 16:50:26 yep 16:50:35 #topic Multiple cluster networks 16:50:38 rmoe: here? 16:50:40 I haven't started re-plugging the rest of the module into fuel because we haven't decided direction for sanitize 16:50:45 yep 16:50:47 xarses: let immediately know if you have further issues, as we may need to run another plan 16:50:49 #link https://blueprints.launchpad.net/fuel/+spec/multiple-cluster-networks 16:51:01 but once we agree it should only take a day to workout 16:51:26 rmoe: you're up 16:51:50 I've agreed after talking with Alexey Kasatkin to rework the API, but I haven't had a chance to start implementing that because I've been dealing with a criticcal customer issue 16:52:11 how much work you think is there? 16:52:12 rmoe: actually this feature is low priority 16:52:19 we really need to squash bugs 16:52:22 yeah 16:52:31 bug-squashing day also delayed my progress :) 16:52:35 so out of other tasks, preference is for bugs definitely 16:52:40 ok 16:52:44 thanks 16:52:47 #topic HA improvements 16:53:04 holser: will u provide update? 16:53:37 galera ocf is ready, it’s on review 16:54:03 I am trying to wrap up and update blueprint for other improvements 16:54:22 holser: could you share a link for review? 16:54:27 that’s about Galera HA improvements 16:54:33 sure 16:54:53 https://review.openstack.org/#/c/95764/ 16:55:03 #link https://review.openstack.org/#/c/95764/ 16:55:11 xenolog is going to remove shadow in another review 16:55:46 but for now this review has both … 16:55:55 holser: xenolog if we remove shadow from galera, should we remove it from neutron? 16:55:56 Galera and cs_shadow removal 16:56:06 will maybe fix some of my issues I have 16:56:17 xarses, yeah 16:56:25 If so, do you have a review for me to look at 16:56:36 you can use my review as it removes from all manifests 16:56:45 specificcally if we change the corosync cluster config so I can test the same 16:56:51 yep 16:57:11 ok guys, 3 minutes 16:57:19 I have finished 16:57:27 #topic image based provisioning 16:57:52 #link https://review.openstack.org/#/c/95575/ 16:57:56 here is spec 16:57:59 status is 16:58:39 i've almost finished partitioning part of fuel_agent and working on building bootstrap image with built in agent 16:59:02 agordeev2: is working on migrating cobbler snippets to cloud-init 16:59:07 my current status: about 50% of snippets replacements with cloud-init based things done for ubuntu. Expecting to finish them all just in few days later. Then will switch to centos. 16:59:29 and time 16:59:29 agordeev2: great 16:59:48 and we have just two small pieces of functionality 16:59:58 1) is downloading images 17:00:04 xarses, yes, I working on it 17:00:09 2) is prepare configdrive 17:00:16 let's move other discussion to #fuel-dev as usual 17:00:27 vkozhukalov: thanks 17:00:36 looking forward to see pieces merged 17:00:37 and another point is we need to add image based provision driver to astute 17:00:46 ok thx everyone 17:00:47 let's close meeting other folks might wait 17:00:53 thanks folks 17:00:54 #endmeeting Fuel