16:00:07 <kozhukalov> #startmeeting Fuel
16:00:07 <openstack> Meeting started Thu Oct  9 16:00:07 2014 UTC and is due to finish in 60 minutes.  The chair is kozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:10 <openstack> The meeting name has been set to 'fuel'
16:00:15 <kozhukalov> #chair kozhukalov
16:00:17 <openstack> Current chairs: kozhukalov
16:00:40 <kozhukalov> agenda as usual
16:00:49 <kozhukalov> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:01:08 <kozhukalov> who is here?
16:01:15 <mihgen> hi
16:01:17 <kozhukalov> are we ready to start?
16:01:21 <mihgen> , out of agenda, reminder on schedule
16:01:24 <mihgen> #link , out of agenda, reminder
16:01:28 <mihgen> #link https://wiki.openstack.org/wiki/Fuel/6.0_Release_Schedule
16:01:34 <mihgen> this link is better, sorry )
16:01:39 <bogdando> hi
16:01:45 <xarses> hi
16:01:48 <ikalnitsky> o/
16:01:49 <vkramskikh> hi
16:01:50 <mihgen> so we are in an active development phase of 6.0
16:02:11 <kozhukalov> mihgen: thanx for the link
16:02:15 <dpyzhov> hi
16:02:41 <mattymo> hi
16:02:47 <kozhukalov> ok, let's start first topic
16:02:59 <kozhukalov> #topic New DocImpact process for documentation (holser, bogdando)
16:03:33 <kozhukalov> holser: bogdando around?
16:03:36 <bogdando> Yes
16:03:37 <holser> yes
16:03:46 <holser> okey
16:03:56 <holser> we aligned our process to OpenStack
16:04:02 <angdraug> #link https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Documentation
16:04:26 <holser> so every reviewer should control the patchset
16:04:27 <irina_pov> hi to all
16:04:47 <kozhukalov> irina_pov: hi
16:04:48 <mihgen> holser: that's great, thanks for pushing this forward and leading this effort until it finally succeeded
16:04:52 <holser> and ask commiter to add DocImpact to Commit Message if the patchset requires the documentation update
16:05:12 <holser> that’s improtant especially for Core reviewers
16:05:38 <holser> Our Documetation team will help to prepare good User or Development documentation
16:05:40 <mihgen> yeah - let's put -1 if there is no DocImpact while changeset has to go with docs piece
16:05:52 <bogdando> and if DocImpact has been added, to ask commiter to provide explicit and clear description for the scope of changes he propose in his patch
16:05:55 <holser> if you have any questuons, ask me on #fuel-dev I’ll help
16:06:25 <holser> irina_pov ^^^
16:06:26 <angdraug> most importantly, reviewers should make sure commit message is descriptive
16:06:38 <bogdando> yes, so now the quolity controll of documentation is up to reviewers ;)
16:06:50 <mihgen> commit message or bug comments ?
16:06:58 <angdraug> commit message
16:07:00 <holser> Commit message
16:07:12 <mihgen> ok
16:07:13 <holser> comments won’t be merged
16:07:18 <bogdando> s/quolity/quality
16:07:19 <angdraug> bug comments describe the process of fixing a bug, commit message describes the outcome
16:07:31 <mihgen> agree, good
16:07:37 <mattymo> +1 angdraug
16:08:03 <mihgen> thanks. Let's move to the next topic?
16:08:09 <kozhukalov> ok
16:08:22 <kozhukalov> #topic Fuel buid system documentation (irina_pov)
16:09:22 <kozhukalov> afaiu, there is a PR
16:09:22 <irina_pov> so, I'd like guys to help me with describing fuel build system. I've used mailing list but it made little success
16:09:27 <kozhukalov> #link https://review.openstack.org/#/c/125928/
16:09:45 <kozhukalov> and i had no time last week to take a look at it
16:10:12 <kozhukalov> irina_pov: i'll take a look till the end of this week
16:10:17 <kozhukalov> irina_pov: i promise
16:10:26 <irina_pov> i see, but anyway I need guys to comment on modules for instance
16:10:56 <irina_pov> kozhukalov okay, thanks! I appreciate receiving any feedback
16:11:15 <mihgen> irina_pov: what modules?
16:11:15 <kozhukalov> yes, everyone who is aware about how our build system works please help to improve this doc
16:11:15 <tzn> guys, this is extremely important, as we have some customers watinig for this
16:11:40 <irina_pov> mihgen like docker for instance
16:11:41 <kozhukalov> mihgen: modules in terms of build system
16:11:59 <kozhukalov> mihgen: mirror, bootstrap, packages etc
16:12:09 <kozhukalov> ok let's move on then
16:12:14 <irina_pov> thanks!
16:12:28 <mihgen> kozhukalov: so will you help?
16:12:34 <dpyzhov> irina_pov: looks like we need to talk more about it
16:12:36 <mihgen> or should we find someone else to help too
16:12:41 <kozhukalov> mihgen: yes
16:13:03 <kozhukalov> #topic authx improvements (prmtl)
16:13:08 <irina_pov> dpyzhov yup, any information on modules structure is strongly required
16:13:19 <prmtl> good progress, “must have” items from bp are on review
16:13:19 <prmtl> #link https://review.openstack.org/#/q/status:open++topic:bp/access-control-master-node-improvments,n,z)
16:13:19 <prmtl> what is done: removed admin_token usage, ostf and nailgun have dedicated service users, support for token in cookie, asking for password during upgrade, generating new credentials for service users during upgrade
16:13:21 <prmtl> next step: add services with endpoints to keystone (keystone itself, nailgun, ostf) for service discovery purpose
16:15:48 <kozhukalov> prmtl: thanx, any questions?
16:15:52 <kozhukalov> on this topic
16:16:06 <xarses> prmtl: I was poking around with authx the other day, I'd like to talk to you about it later
16:16:15 <prmtl> xarses: sure
16:16:36 <kozhukalov> ok, moving on then
16:16:56 <kozhukalov> #topic Pacemaker improvements: pcs service provder for puppet and openstack-services as well (bogdando)
16:17:17 <bogdando> good progress, some patches on review, some already merged
16:17:20 <bogdando> #link https://blueprints.launchpad.net/fuel/+spec/pacemaker-improvements
16:17:48 <bogdando> monit for compute nodes implemented, and could be enabled once monit package included in ISO
16:18:13 <bogdando> OCF scripts for heat and ceilometer improved by amqp connection status tracking
16:18:44 <bogdando> on the road - make the same (OCF control plane) for openstack-api services
16:19:07 <holser> I am merging them
16:19:08 <bogdando> of cause we should have to contribute this to puppet-openstack upstream modules
16:19:43 <bogdando> and that activity requires some additional efforts for puppet upstream contribution for service provider type for pacemaker
16:19:56 <bogdando> I created an improvement in puppet JIRA
16:20:14 <bogdando> so we have to port our custom Fuel 'hax' there as well :)
16:20:22 <bogdando> #link https://tickets.puppetlabs.com/browse/PUP-3427
16:21:08 <kozhukalov> is it mandatory to contribute those patches into upstream?
16:21:10 <bogdando> that would allow us to put all stateless openstack services under this new pacemaker provider, and bring it to Fuel from upstream
16:21:32 <bogdando> it is really nice to have, otherwise we will just increase diverge significantly
16:21:40 <tzn> this is something that community was also talking about for some time
16:21:45 <tzn> We should contribute it back
16:21:46 <bogdando> and neglate the efforts done with syncing our manifests
16:21:58 <angdraug> it is mandatory to contribute back to upstream any changes we make to modules we take from upstream
16:22:02 <bogdando> so, I believe we really should
16:22:12 <angdraug> +1 bogdando
16:22:16 <holser> My status:
16:22:27 <tzn> +1. Also it will make our lifes easier
16:22:33 <holser> I am in process ofadding Openstack OCF resources
16:23:05 <holser> by our plan we’ll have all openstack related resources in Pacemaker (on controller) and monit on computes
16:23:19 <holser> Tomorrow, I’ll play with Corosync 2.0
16:23:32 <holser> Thanks to rvyalov for Corosync 2.0 packages
16:23:56 <mattymo> btw holser I'm seeing some issues with new corosync/pacemaker in an iso I just built
16:24:02 <mattymo> dependency issues, actually
16:24:05 <holser> Blueprint also requires some efforts in services providers ...
16:24:21 <angdraug> what about monit packages, any problems expected there?
16:24:35 <holser> mattymo, our crm puppet provider is not compatible with corosync 2.0
16:24:46 <holser> monit is already merged,
16:24:57 <holser> requires documenttion for Operation and Uers
16:25:01 <bogdando> andreaf_, I have it working at my PoC lab, so there shouldn't be much...
16:25:02 <holser> Users*
16:25:07 <mattymo> so fuel-library is broken. do you have a patch for it ready yet holser ?
16:25:19 <bogdando> angdraug,  I have it working at my PoC lab, so there shouldn't be much...
16:25:22 <holser> I do mattymo
16:25:31 <holser> We can work together tomorrow
16:26:14 <kozhukalov> guys, great.
16:26:39 <kozhukalov> looks like there is no any other q here
16:26:43 <kozhukalov> moving on
16:26:57 <kozhukalov> #topic Juno deployment status (mattymo)
16:27:11 <mattymo> hi!
16:27:28 <mattymo> A number of unfortuante changes in our build system led to more and more delays of Juno
16:27:31 <mattymo> unfortunate*
16:28:06 <mattymo> switching between base mirrors, package updates, etc. We encountered breakage in OSTF, bootstrap, HA, haproxy, and numerous surprise dependency issues in the past week
16:28:23 <mihgen> mattymo: do you confirm that now all changes in build system are settled?
16:28:36 <mattymo> mihgen, all except pacemaker which holser mentioned
16:29:24 <mattymo> I can't confirm our fixes to sahara or murano (which were merged 3 days ago) because I still can't get a deployment
16:29:31 <mattymo> but maybe tomorrow the story will change
16:30:20 <mattymo> we're very very close to working Juno that passes all tests
16:30:32 <mattymo> and that's it from me
16:30:35 <kozhukalov> mattymo: thanx, great
16:31:03 <kozhukalov> but today i had some problems with broken ruby json in juno mirror
16:31:13 <mattymo> me too! It's fixed now
16:31:22 <kozhukalov> great
16:31:27 <kozhukalov> moving on?
16:31:46 <kozhukalov> #topic 6.0 Juno builds on https://fuel-jenkins.mirantis.com/view/ISO/  (mihgen)
16:33:10 <kozhukalov> mihgen: do you have something on this topic?
16:33:27 <mihgen> yeah, sorry
16:33:47 <mihgen> teran has to give update actually, but I'll pass from his words
16:34:01 <mihgen> builds are ok for 6.0 there, with Juno packages
16:34:17 <mihgen> though they are red because as mattymo says not everything fixed yet
16:34:27 <mihgen> so they should become green once fixes are merged
16:34:33 <mihgen> that's it.
16:34:41 <mattymo> we need to hold off on merging features until we can get it green
16:34:48 <mattymo> if we don't, we'll never actually get there
16:35:13 <kozhukalov> and I saw a letter from rvyalov, his suggestion is to disable bvt for a while for juno
16:35:26 <kozhukalov> just to be able to merge new packages if we need
16:35:39 <kozhukalov> for example we need mdadm for ubuntu
16:35:56 <angdraug> what's the bvt status for 6.0_icehouse?
16:36:08 <kozhukalov> they work
16:36:16 <kozhukalov> and green
16:36:33 <angdraug> how come feature merges break juno but not icehouse?
16:36:35 <bookwar> kozhukalov: latest one failed
16:37:18 <kozhukalov> angdraug: rvyalov for details
16:37:23 <mihgen> bookwar: latest what?
16:37:28 <xarses> angdraug: ya, how come?
16:37:30 <mihgen> icehouse bvt?
16:37:44 <angdraug> icehouse ubuntu bvt just failed
16:37:54 <bookwar> staging bvt_2 with Ubuntu ha on 6.0-icehouse
16:38:12 <mihgen> angdraug: ok, then let's use standard process - go ahead and find a bug, revert commit which broke it
16:39:04 <mihgen> and then propose a proper fix..
16:39:13 <mihgen> anything else to discuss on this topic?
16:39:13 <kozhukalov> ok, looks like we are done on this topic
16:39:30 <kozhukalov> moving on?
16:39:38 <mihgen> kozhukalov: yep
16:39:41 <kozhukalov> #topic 100 nodes (salmon_)
16:40:35 <kozhukalov> looks like salmon is not here
16:41:17 <kozhukalov> ok, switching to the next topic
16:41:33 <kozhukalov> #topic reduce upgrade tarball size (ikalnitsky)
16:41:38 <ikalnitsky> hi
16:41:47 <ikalnitsky> This week we had an active discussion about changes in infrastructure, so please welcome to review the spec:
16:41:49 <ikalnitsky> #link https://review.openstack.org/#/c/121591/
16:41:57 <ikalnitsky> angdraug, aglarendil - please take a look
16:42:07 <ikalnitsky> Also, in last meeting I mentioned about the issue with OBS - so OBS rebuilds all packages (that are depends on some changed one) without version incrementation:
16:42:08 <ikalnitsky> #link https://bugs.launchpad.net/fuel/+bug/1376694
16:42:09 <uvirtbot> Launchpad bug 1376694 in fuel "OBS rebuilds a package, but don't increase its version number" [Critical,Confirmed]
16:42:17 <ikalnitsky> We have turned off this OBS feature (as experiment), so I hope the tarball will get smaller in size.
16:42:26 <ikalnitsky> What is done: We have "make mirror_diff" patch on review, and we're going to integrate it with Jenkins tomorrow.
16:42:35 <ikalnitsky> Also, I almost done with changes to nailgun/fuel_upgrade and I'm going to test it tomorrow.
16:42:46 <ikalnitsky> Next steps: implement "make openstack_bundle" goal in make system and create a jenkins job for it. When it's done we'll be able to build upgrade  tarballs with  diff-based repos and test them on CI.
16:42:56 <ikalnitsky> that's it. questions ?
16:43:16 <angdraug> is 1376694 now fixed?
16:43:25 <kozhukalov> ikalnitsky: but there is still  no artifact related stuff in that patch
16:43:43 <ikalnitsky> angdraug: I'm not sure it's fixed. we're going to test in practice.
16:43:50 <mihgen> ikalnitsky: sounds like good progress, thanks. when do you expect the whole feature to be done and fully tested?
16:44:13 <ikalnitsky> kozhukalov: sorry, what do you mean?
16:44:19 <mihgen> 1376694 unlikely fixed as far as I know, looks like serious issue
16:44:29 <kozhukalov> ikalnitsky: it is mentioned in spec that diff mirror should be published as an artifact so as to make it possible to use this artifact for other targets and builds
16:44:47 <ikalnitsky> mihgen: i think we need 1.5-2 weeks.
16:45:02 <angdraug> I think fixing that bug should be a requirement for this feature
16:45:17 <angdraug> right now, the spec mentions it as a fact, should be a work item
16:45:46 <mihgen> ikalnitsky: thanks, sounds great
16:45:58 <ikalnitsky> kozhukalov: oh, got it. the artifact will be provided by jenkins. don't you think so?
16:45:58 <kozhukalov> #link https://bugs.launchpad.net/fuel/+bug/1376694
16:45:59 <uvirtbot> Launchpad bug 1376694 in fuel "OBS rebuilds a package, but don't increase its version number" [Critical,Confirmed]
16:46:04 <mihgen> angdraug: I hope packaging team will find time any soon for that bug
16:46:07 <ikalnitsky> angdraug: i'll add it to work item
16:46:18 <mihgen> they really overloaded now
16:46:24 <kozhukalov> guys, please post urls instead of just bug id
16:46:56 <angdraug> kozhukalov: there's a link above
16:47:25 <kozhukalov> ikalnitsky: artifact related stuff should be implemeted in make file and then jenkins is supposed to launch those targets if necessary
16:47:47 <angdraug> kozhukalov: where are we with artifact based build system?
16:47:54 <kozhukalov> angdraug: yes, sorry, didn't see )
16:48:32 <kozhukalov> angdraug: i don't know, my view on this stuff was slightly different
16:49:11 <kozhukalov> we all agree that we need to have a good doc on this stuff first
16:49:18 <angdraug> yup
16:49:30 <ikalnitsky> kozhukalov: should we change build system only for these goals?
16:49:55 <ikalnitsky> kozhukalov: i mean, should we change goals that are not related to this bp? they still don't care about artifacts
16:50:00 <kozhukalov> i believe dpyzhov is the right person to ask about where we are about artifact based build system
16:50:36 <kozhukalov> ikalnitsky: let's discuss this stuff tomorrow
16:50:43 <angdraug> I'm just confused why are we discussing artifacts in context of ikalnitsky's work if abbs isn't even in progress yet
16:50:44 <ikalnitsky> kozhukalov: ok
16:50:50 <kozhukalov> afaiu, we have a meeting scheduled
16:51:27 <angdraug> ok, moving on?
16:51:33 <kozhukalov> ok
16:51:45 <kozhukalov> #topic image based provisioning (demo results) (kozhukalov)
16:51:54 <kozhukalov> ok, today we had a demo
16:52:23 <kozhukalov> it was a little bit broken due to broken ruby json in juno mirror
16:52:55 <kozhukalov> we still have one little topic to discuss on that feature
16:53:39 <kozhukalov> the topic is where we should put image/not image check mark
16:54:05 <kozhukalov> should it be a separate release or maybe just a little cluster attribute
16:54:24 <mihgen> kozhukalov: from UX perspective, I would say checkbox on settings tab is better
16:54:47 <mihgen> as getting 4 releases instead of 2 in very first wizard is not that beautiful to me
16:54:52 <kozhukalov> after short discussion with evgeniyl__ I think having just a check mark on the settings tab is ok
16:55:31 <mihgen> so when you want your env to be provisioned with image-based, you just go to settings tab and enable checkbox
16:55:33 <kozhukalov> yes, if nobody has other opinion on that stuff
16:55:47 <kozhukalov> let's implement this on settings tab
16:56:01 <salmon_> hi
16:56:13 <mihgen> kozhukalov: the whole feature looks pretty solid
16:56:20 <kozhukalov> ok, that is it from me
16:56:30 <mihgen> I'm wondering how many unmerged code we have on review still for it?
16:56:36 <kozhukalov> let's go back to the topic 100 nodes (salmon_)
16:56:48 <salmon_> can I start?
16:56:56 <kozhukalov> mihgen: not so much
16:57:06 <mihgen> salmon_: let's start
16:57:07 <kozhukalov> #topic 100 nodes (salmon_)
16:57:13 <mihgen> we can finish in parallel with kozhukalov )
16:57:16 <kozhukalov> salmon_: 2 minutes
16:57:25 <salmon_> ok, so we have some bugs for fuel https://bugs.launchpad.net/fuel/+bug/1376686 https://bugs.launchpad.net/fuel/+bug/1376426 https://bugs.launchpad.net/fuel/+bug/1378000  https://bugs.launchpad.net/fuel/+bug/1376680 and https://bugs.launchpad.net/fuel/+bug/1379306
16:57:26 <uvirtbot> Launchpad bug 1376686 in fuel "HA nova  cluster in error state with 504 error from keystone" [High,Triaged]
16:57:40 <salmon_> and for openstack
16:58:01 <salmon_> for nailgun everything looks good. We have stats what's slow and now we are fixing it
16:58:25 <mihgen> that's nice
16:58:26 <salmon_> for astute we didn't start yet because of debugging bugs
16:58:34 <mihgen> what about that dup-IPs issue?
16:58:44 <salmon_> looks like dnsmasq issue
16:59:01 <mihgen> is there a known fix?
16:59:21 <salmon_> see last comment https://bugs.launchpad.net/fuel/+bug/1376680
16:59:22 <uvirtbot> Launchpad bug 1376680 in fuel "Deploy cluster  with 57 nodes failed" [High,In progress]
16:59:24 <kozhukalov> mihgen: 5 PR, will try to split them into clear independent pieces
16:59:25 <mattymo> Alexander Shapshnikov suggested longer leases and switching the hashing algorithm for storing leases
16:59:26 <mihgen> what our strategy on provisioning, do we do 100 in parallel? Or we plan to do it in batches ?
16:59:41 <mihgen> kozhukalov: thanks
16:59:44 <kozhukalov> ok, guys 20 seconds
16:59:45 <salmon_> We need to check what will happen when IP is changed for node
16:59:51 <mattymo> salmon_, got the link. I'll keep quiet)
16:59:53 <salmon_> in bootstrap mode
17:00:08 <kozhukalov> let's close our meeting
17:00:14 <mihgen> salmon_: ok, thanks
17:00:15 <kozhukalov> thanx everyone
17:00:24 <kozhukalov> #endmeeting