16:00:12 #startmeeting Fuel 16:00:13 Meeting started Thu Aug 14 16:00:12 2014 UTC and is due to finish in 60 minutes. The chair is vkozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:14 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:16 The meeting name has been set to 'fuel' 16:00:21 #chair vkozhukalov 16:00:22 Current chairs: vkozhukalov 16:00:32 agenda as usual 16:00:42 #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda 16:00:50 #topic greetings 16:00:59 hi 16:01:18 hi 16:01:28 we were so close to release 5.0.1 .. :( 16:01:30 anybody else is here? 16:01:38 hi 16:01:38 hi 16:01:40 hi 16:01:41 o/ 16:01:42 hello 16:01:53 hi 16:01:58 o/ 16:02:01 #topic 5.0.1 release status 16:02:20 angdraug: around? 16:02:41 angdraug: I can share what was found MSK day today with 5.0.1 16:02:41 hi 16:02:43 hi boys and girls 16:02:54 hi 16:02:55 mihgen: please do 16:03:05 mihgen: it would be great 16:03:09 #link https://bugs.launchpad.net/fuel/+bug/1353497 16:03:10 Launchpad bug 1353497 in fuel/5.0.x "[Fuel Upgrade] Can't re-deploy environment after upgrade - failed with ceph-deploy --overwrite-conf config pull node-16 returned 1 instead of one of [0]" [Critical,Confirmed] 16:03:25 this bug was closed as incomplete/invalid…. it appeared to be a real issue 16:03:46 when you upgrade 5.0 -> 5.0.1, your astute container gets rebuild and you lose all ssh keys of old envs 16:04:04 so when you add node to the existing env, astute has to generate new jekeys 16:04:26 so your new node is not completely operational when you use compute or ceph roles, which rely on ssh connectivity to other nodes 16:04:56 it affects only if you upgrade 5.0 -> 5.0.1, and doesn't if you use directly 5.0.1... 16:05:15 so fix is committed, but we have to spend 24 more hours to test 16:05:42 that is it. Also we need to work on mistakes to improve workflow 16:05:58 so such issues resolved earlier / treated better 16:06:03 24h means we can release tomorrow? or is it next week? 16:06:15 we can release tomorrow according to nurla 16:06:18 nurla: can we? 16:06:34 it's matter of updated upgrade scripts to preserve keys from old envs when you do upgrade. 16:06:35 i hope we can do it tomorrow 16:06:52 so we don't need to respin the whole acceptance cycle 16:07:08 ikalnitsky: actually can you fix status of https://bugs.launchpad.net/fuel/+bug/1353497 16:07:08 it 16:07:09 Launchpad bug 1353497 in fuel/5.0.x "[Fuel Upgrade] Can't re-deploy environment after upgrade - failed with ceph-deploy --overwrite-conf config pull node-16 returned 1 instead of one of [0]" [Critical,Confirmed] 16:07:17 still says Confirmed incorrectly 16:07:22 while it's actually merged 16:07:26 which build is RC2? 16:07:34 yep, we just runs systests and check related issues around bug 16:07:53 migration and adding new nodes to 5.0 envs 16:07:56 angdraug: #170 16:08:00 expected to be 16:08:17 angdraug: what's the status on release notes readiness ? 16:08:41 ywe checked Meg's review 16:08:45 aside from one entry that I've posted a question about (network verification), it's all done 16:09:05 all I need is team leads to confirm that we didn't miss any important bugs 16:09:34 after that I'll need 30 mins to 1 hour to go over most recent changes done since yesterday and it will be ready to merge 16:09:52 bookwar: merges to stable/5.0 still go straight to docs.mirantis.com, right? 16:10:10 angdraug: yes 16:10:17 angdraug: ok. So dpyzhov, aglarendil, nurla, you would need to take a look 16:10:27 ok 16:10:41 Tomas is on vacation, can't see Lukasz around 16:10:41 ok 16:10:51 do I need to give them an action? 16:10:57 rvyalov: you too for release notes check 16:11:02 vkozhukalov: pls do :) 16:11:07 also need someone from mos, probably Ruslan 16:11:19 or Dima 16:11:39 ok 16:11:46 #action rvyalov nurla aglarendil dpyzhov take a look on rc2 16:11:54 moving? 16:12:19 #topic 5.1 CI status 16:12:22 action was to review bugs in 5.0.x and confirm nothing important is missing from release notes... 16:12:34 angdraug: we need to do it for sure 16:12:58 also, nurla promised to come up with HA issues description in terms of probability 16:12:58 we had a lot of CI failures with 5.1 recently, what's the current situation? 16:13:06 #action rvyalov nurla aglarendil dpyzhov to review bugs in 5.0.x and confirm nothing important is missing from release notes 16:13:16 angdraug: what failures exactly? 16:13:23 guys do you mean Fuel CI failures or nightly tests? 16:13:32 nightlies 16:13:40 mihgen: one or the other bvt fails frequently 16:13:43 ))) let's provide status of all 16:14:03 nurla: we need your team to look into sporadical ubuntu BVT failures 16:14:07 I refuse to make distinction between gerrit triggered jobs and cron triggered jobs when saying "CI" :) 16:14:15 torrent failures were fixed thanks to teran 16:14:52 #action nurla to file bug about Ubuntu sporadic failures (BVT) 16:15:00 okay 16:15:17 #action nurla to file bug about Ubuntu sporadic failures (BVT) 16:15:30 aglarendil: if we speak about CI, can you say about astute issues you found? 16:15:39 which make Fuel CI putting -1 now ? 16:15:40 yep 16:15:53 #link https://bugs.launchpad.net/fuel/+bug/1356954 16:15:54 Launchpad bug 1356954 in fuel "successful deployment failed by astute" [Critical,Confirmed] 16:16:09 I found a bug that despite there are no puppet errors 16:16:16 astute sets deployment as failed 16:16:20 how can it affect CI though? 16:16:26 we use stable astute there, don't we? 16:16:31 CI marks good requests with -1 16:16:38 may be there was a floating bug 16:16:45 do you mean those requests in fuel-library? 16:16:48 devops guys are going to update ISO on CI nodes 16:16:55 yep, fuel-library ones 16:17:07 why do we want to update ISO, when did we update it last time? 16:17:18 if it was stable, how comes it is not anymore? 16:17:21 bookwar: ^^^ 16:17:22 is there a plan to automatically update ISO as soon as new one passes bvts? 16:18:11 andreaf: there was a plan 16:18:29 oops 16:18:34 angdraug: ^^^ 16:18:42 got it ) 16:18:48 what happened to that plan? 16:18:52 but now it takes manual work to decide if iso fits fro ci or not 16:19:03 mihgen: why would we not want to run fuel-library CI on most recent ISO? 16:19:07 bookwar: so question when did you update last time? 16:19:14 bvt doesn't check that iso is suitable for ci 16:19:25 latest iso is 423 16:19:50 aglarendil: so astute should not be a reason of failure in theory 16:19:59 as there is still old version of astute 16:20:07 ok anyway we need to triage it 16:20:08 mihgen: may be it had this bug 16:20:12 i put info about iso updates on jenkins main page, btw 16:20:21 mihgen: I saw one, but could not find it in launchpad 16:20:25 I think we should move on and someone should take it 16:20:50 ok 16:21:04 #topic 5.1 bugs status 16:21:16 mihgen: the diagnostic snapshot in this astute launchpad issue is exactly 423, so iso update might help 16:21:39 action for bookwar to update ISO? 16:21:47 please do not update 16:21:54 without proper investigation first 16:21:57 ok, lets take it to ML 16:22:24 current topic is 5.1 bugs 16:22:26 as I say, it was working fine. Now it is not. Nothing changed. Let's find root cause, what still changed 16:22:35 aglarendil: you go with the status ;) 16:22:54 okay 16:23:09 we have not yet reached the status good enough to call Hard Code Freeze 16:23:22 we have a bunch of issues unresolved 16:23:42 oslo.messaging bug is not yet completely fixed in 5.1 branch 16:23:49 we still have fuel-library fixes on review 16:24:12 dmitryme: can you comment on oslo.messaging in 5.1 ? 16:24:16 #link https://launchpad.net/bugs/856764 16:24:20 do we have the fix for 20 nodes bug? python /dev/log? rabbitmq 3.3.5? 16:24:22 Launchpad bug 856764 in oslo.messaging "RabbitMQ connections lack heartbeat or TCP keepalives" [Critical,In progress] 16:24:40 afaik it is not merged 16:24:53 dmitryme: what do we need to do it? 16:24:54 python code was merged 16:25:03 we have two fixes on review 16:25:14 #link https://review.openstack.org/#/c/112677/ 16:25:15 and 16:25:25 #link https://review.openstack.org/#/c/114282/ 16:25:41 the former depends on the latter one 16:26:00 keystone bug is here: #link https://bugs.launchpad.net/fuel/+bug/1340657 16:26:01 Launchpad bug 1340657 in fuel "[library] 'keystone' delays lead to unstable MOS operations" [High,In progress] 16:26:07 we are waiting for keystone support 16:26:13 puppet part is ready 16:26:42 also we have rsync tuning issue on review 16:26:57 #link https://bugs.launchpad.net/fuel/+bug/1322577 16:26:59 Launchpad bug 1322577 in fuel/5.0.x "rsync puppet modules partial failure breaks deployment" [High,In progress] 16:27:17 mattymo: https://bugs.launchpad.net/fuel/+bug/1355347 should be Fix Committed? 16:27:18 Launchpad bug 1355347 in fuel "20 nodes, not all are able to get provisioning info from cobbler" [High,In progress] 16:28:28 rabbitmq was mentioned here 16:28:35 as I know msemenov has it ready to go 16:28:36 looks like mattymo isn't here 16:29:07 mattymo: is here 16:29:09 sorry I was distracted 16:29:28 mattymo: can you confirm that the 20 nodes bug has all the commits that are needed merged? 16:29:32 it is not fix committted. aglarendil asked us to stall merging this fix to astute until MOS hardening team looked at it 16:29:45 I confirm it is good and so does Vladimir Sharshov, but aglarendil blocked the last bit 16:29:47 for astute 16:29:50 it was merged an hour ago 16:29:50 that's right 16:29:59 MOS team should verify it 16:30:28 aglarendil: why? 16:30:39 rabbitmq update: we built an ISO with update for ubuntu and centos. Now fuel QA is running tests against it 16:30:40 http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_master_iso/31/ 16:30:40 angdraug: because MOS team is responsible for this testing 16:30:41 I thought they were the ones that asked for this patch? 16:30:52 angdraug: nope. it is release feature 16:31:26 msemenov, you are from the hardening team. can you please take a look at my mail I sent earlier about this? 16:31:30 aglarendil: https://bugs.launchpad.net/fuel/+bug/1355347/comments/8 16:31:31 Launchpad bug 1355347 in fuel "20 nodes, not all are able to get provisioning info from cobbler" [High,In progress] 16:32:13 fix confirmed to resolve the issue for the original reporter 16:32:15 angdraug, aglarendil re 20 nodes deployment: we will do it eventually (I think soon) for 5.1 16:32:37 angdraug: I am waiting for sign off from the ones who are responsible 16:32:41 angdraug: not from aleksandr_null 16:32:56 dmitryme is responsible for MOS 16:32:57 are you saying aleksandr_null is irresponsible? :p 16:33:14 mattymo, can't found such email from you( can you please resend? 16:33:16 dmitryme: should we close this bug now? 16:33:25 or how do you want to proceed with it? 16:33:36 you can open another if you find issues during testing 16:34:07 aleksandr_null: are you here? if it's fixed for aleksandr_null , then should be fine to close.. 16:34:22 alexander is the original bug filer, if he says it is fixed, I trust him 16:34:46 Folks, we also have a number of issues, criticals basically, with patching 16:34:47 I'll recheck with him later today and reopen if it's still a probelm 16:35:05 like around 5 were reported by Tatyanka_Leontov today.. 16:35:09 ouch 16:35:16 not five 16:35:23 nurla: Tatyanka_Leontov do you think we are in really bad status there ? 16:35:27 okay, if you want to merge it - just merge 16:35:36 what are your feelings on where we are with the feature? 16:35:53 aglarendil: merge what? 16:36:01 merge 20-nodes fix 16:36:15 https://review.openstack.org/#/c/114247/1 16:37:06 aglarendil: 20-nodes fix merged, nurla - it's another bug 16:37:29 20-nodes fixes: https://review.openstack.org/113924, https://review.openstack.org/113920 16:37:32 there are several issues with 20-nodes 16:38:05 mattymo: got it. Actually we in mos-linux don't have even 3 baremetal nodes in the lab (lend it from moscow devops) 16:38:07 well let's clarify tickets / statuses / patches then. I see that bug https://bugs.launchpad.net/fuel/+bug/1355347/ should be closed, as it has merged patches 16:38:07 can we please move on to status of patching? 16:38:08 Launchpad bug 1355347 in fuel "20 nodes, not all are able to get provisioning info from cobbler" [High,In progress] 16:38:15 still https://review.openstack.org/#/c/113867/ and https://review.openstack.org/#/c/113942/ 16:38:21 both unmerged 16:38:33 yes, we still have a lot of stuff to discuss 16:38:37 moving 16:38:43 ok sorry. moving on 16:38:44 msemenov: you can easily request nodes from fuel-devops folks, teran 16:38:56 #topic 5.0.2 - taking 5.1 as a base 16:38:59 mattymo: in future we will have a bigger lab, but now this process is in conversation state 16:39:01 mihgen: patching status can be found here : https://docs.google.com/a/mirantis.com/spreadsheets/d/1njLJt_fO4zSgZf9JskJZ6JitYawObI8p_iEz5FQ9YBA/edit#gid=0 16:39:16 what this topic is about? 16:39:41 mattymo: oh thanks did not see those. Let's get aleksandr_null and others to review, I think we need those patches 16:40:06 vkozhukalov: dmitryme will provide a status on 5.0.2 16:40:18 mihgen, sure, thank you 16:40:26 Tatyanka_Leontov: share with anyone pls 16:40:40 so, today we checked and found that 5.1 does not have any migrations 16:40:49 which means that it could be used as a base for 5.0.2 16:40:51 msemenov: ok, I can try to request it from Igor S. But why it's our task instead of fuel-qa team? 16:41:10 Tatyanka_Leontov: looks mature :) with many red cells .. :( 16:41:20 the idea is to copy 5.1 to 5.0.2 and so we will have similar OpenStack in 5.0.2 and 5.1 16:41:31 I would say, almost identical 16:41:43 msemenov, I think you meant to address mihgen and aglarendil 16:41:47 which is our desire actually 16:41:49 mattymo: may be I'm not in context 16:42:05 dmitryme: so this is going to be something between 2014.1.1 & 2014.1.2 ? 16:42:21 ou, and by the way people, I am speaking of OpenStack compoents only, not Fuel 16:42:27 mihgen: right 16:42:32 and delta between 5.1 & 5.0.2 will be as minimal as possible (ideally no delta), right? 16:42:41 mihgen: exactly 16:43:09 dmitryme: excellent. Do we expect to get it done tomorrow? 16:43:19 and start testing tomorrow I hope? 16:43:20 yep, lets do it 16:43:24 https://bugs.launchpad.net/fuel/+bug/1356817 16:43:25 Launchpad bug 1356817 in fuel "[OSCI] New branches for MOS 5.0.2" [High,Won't fix] 16:43:37 rvyalov: what's the status on ^^^ ? 16:43:38 I am moving this bug to confirmed meaning OSCI team can proceed 16:44:00 mihgen: it was ‘won’t fix’ so far 16:44:08 we waiting approved from MOS team 16:44:20 15 minutes and 5 topics 16:44:21 meaning we wanted to make sure we want 5.1 in 5.0.2 16:44:31 rvyalov: dmitryme is the approval guy 16:44:43 rvyalov: I’ve moved it to confirmed 16:44:50 lets move on 16:44:51 vkozhukalov: we will be quick 16:44:51 rvyalov: please proceed 16:44:55 ok 16:44:56 thanks dmitryme 16:45:00 #topic Current master Astute issue 16:45:07 thank you guys, I have to go 16:45:13 vkozhukalov discussed actually 16:45:16 let's move ahead 16:46:10 #topic 6.0 plans 16:46:39 for 6.0, we don't have much time. We have huge thing to do is to consume Juno release and make it reliable being deployed by Fuel 16:47:08 so idea for 6.0 is to concentrate on Juno, and might be do a few minor features, not related to deployment 16:47:23 as any deployment feature would depend on Juno working 16:47:38 so it will stay on critical path, putting itself under high risk 16:47:42 when are we going to start getting juno packages? 16:47:50 ie from master? 16:47:53 it started 16:47:55 what about nailgun plugins? and image based provision? 16:48:00 rvyalov: dmitryme do you guys know about it? 16:48:15 vkozhukalov: nailgun plugins is fine, as it doesn't touch deployment 16:48:16 are those supposed to be addressed in 6.0? 16:48:24 we building 90% openstack packages from master 16:48:42 image based provisioning seems to me more risky but should be fine too if we merge only tested things, making sure they work 16:49:14 something like any neutron plugin, NSX, anything like that sounds to me unrealistic 16:49:29 the idea is to merge those review requests about image based right after 5.1 and start to test them 16:49:39 also I think we need to finally put xarses ML2 puppet code 16:49:53 and merge all left upstream puppet modules 16:49:59 vkozhukalov: let's build custom iso first, test, and fix all blockers first :) 16:50:23 10 minutes 16:50:30 If we merge all in one day without proper testing, we gonna end up with mess and sleepless nights 16:50:38 so let's do things wisely 16:50:54 moving? 16:51:06 #topic Still open blueprints on 5.1 16:51:16 #link https://blueprints.launchpad.net/fuel/+spec/test-patches 16:51:24 Tatyanka_Leontov: it's still deployment? 16:51:31 or can be closed as Implemented? 16:51:33 nurla: ^^^ 16:52:08 closed 16:52:13 excellent 16:52:17 #link https://blueprints.launchpad.net/fuel/+spec/build-packages-for-openstack-master-rpm 16:52:19 rvyalov: ^^^ 16:53:05 yes , it blueprint in progress 16:53:22 when do we expect it finished? 16:53:57 is it still good progress?) 16:54:36 yes, eta 21 aug 16:54:51 any other un-addressed BPs? 16:54:59 #link https://blueprints.launchpad.net/fuel/+spec/vcenter-documentation-update-fuel-5.1 16:55:18 anyone from that area here? 16:55:30 don't know nick of assignee 16:55:56 #link https://blueprints.launchpad.net/fuel/+spec/osci-gerrit-to-dmz 16:55:58 rvyalov: ^^ 16:56:05 what about this one? 16:56:31 3 minutes 16:56:35 regarding vcenter documentation I know that it is almost done 16:56:52 yes, new osci gerrit deploy in public. we synced projects from old gerrit 16:56:56 #link https://blueprints.launchpad.net/fuel/+spec/fuel-web-docs-dev-env-restructure -- christopheraedo, yours 16:57:05 rvyalov: so it's done? 16:57:10 or we keep the status 16:57:26 we don't have much time left, so while waiting for responses on bps 16:57:31 yes its done 16:57:36 on advanced networking in 6.0 16:57:43 frankly I don't think it's doable 16:57:46 lets move our disscussion to #fuel-dev, we still have two other topics 16:57:53 but we should address most annoying bugs 16:58:04 and work on this in parallel track up until 6.1 I think 16:58:17 we had a discussion on networking in 6.0 on cross-team syncup yesterday. agenda and results are here: https://etherpad.openstack.org/p/networking-in-6-0-agenda , https://etherpad.openstack.org/p/fuel-us-eu-sync 16:58:37 akasatkin: ok I'll take a look 16:58:42 Open question from agenda is: pluggable networking. It's moved out of 6.0. 16:59:16 thanks folks 16:59:20 I think we need to write up 16:59:34 left questions - please email to openstack-dev 16:59:56 thanx everyone 17:00:02 closing 17:00:06 #endmeeting