16:02:50 <mihgen> #startmeeting fuel
16:02:51 <openstack> Meeting started Thu Jul 31 16:02:50 2014 UTC and is due to finish in 60 minutes.  The chair is mihgen. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:02:54 <openstack> The meeting name has been set to 'fuel'
16:03:39 <mihgen> hey we don't have vkozhukalov today who is usually a chairman, so I gonna run it
16:03:45 <mihgen> let's discuss where we are
16:03:57 <mihgen> we are in SCF phase and squashing bugs actuallyu
16:04:10 <mihgen> #topic 5.0.1 status
16:04:38 <mihgen> the most severe issue we have is HA bug related to oslo.messaging
16:04:46 <mihgen> nurla: can you please give update on that?
16:05:18 <nurla> we still have issue with oslo patching in 5.0.1 and now we active test it
16:05:27 <nurla> for RC preparing
16:06:09 <mihgen> dpyzhov: what are the issues with patching / upgrades - which we proposed to 5.0.1 ?
16:06:55 <mihgen> angdraug: I also see a few new bugs assigned to 5.0.1
16:07:12 <nurla> special for patching feature we've already update all our clients
16:07:28 <dpyzhov> https://bugs.launchpad.net/fuel/+bug/1348331 this bug is almost the same as other provision issue and we decided to skip it for 5.0.1
16:07:30 <uvirtbot> Launchpad bug 1348331 in fuel "upgrade: new envs uses old kerlenl images during provisioning" [High,In progress]
16:07:41 <dpyzhov> ikalnitsky: is it the same? ^^
16:07:47 <angdraug> #link https://launchpad.net/fuel/+milestone/5.0.1
16:07:52 <angdraug> no new bugs right now
16:08:10 <ikalnitsky> dpyzhov: yeah, almost the same. non-critical at all
16:08:16 <dpyzhov> And we have this issue https://bugs.launchpad.net/fuel/+bug/1349833
16:08:19 <uvirtbot> Launchpad bug 1349833 in fuel/5.0.x "[upgrade] Upgarde script restores old DB dump while upgrading second time" [High,In progress]
16:08:53 <dpyzhov> It is not trivial case, so I’m not sure if we want to fix it in 5.0.1
16:09:06 <evgeniyl> I we have a fix, and tarball with the fix, we are testing it right now
16:09:10 <dpyzhov> mihgen, your oppinion?
16:09:24 <mihgen> on 833, I think we should get core developers to take a closer look
16:09:39 <mihgen> if risks are low with possible regressions, I would get it in
16:09:49 <mihgen> anyway we are waiting for oslo fix
16:09:53 <dpyzhov> We want to merge it, but we can skip it if it really slows us down
16:10:04 <dpyzhov> ok, so we’ll merge it
16:10:14 <mihgen> dpyzhov: let's get extensive QA on that too
16:10:36 <aglarendil> #link https://bugs.launchpad.net/fuel/+bug/1346939
16:10:36 <mihgen> mattymo: are you still working on https://bugs.launchpad.net/fuel/+bug/1346939 ?
16:10:37 <nurla> asledzinskiy should test it more thoroughly
16:10:39 <uvirtbot> Launchpad bug 1346939 in fuel/5.1.x "[fuel-library] Puppet does not generate settings.yaml file correctly" [High,In progress]
16:10:54 <aglarendil> there was a fix that was reverted
16:11:05 <aglarendil> I think we will have a fix on this week
16:11:13 <mihgen> this week doesn't work
16:11:31 <mihgen> we are about to start acceptance..
16:11:42 <mihgen> also let's get better info on it's severity
16:11:45 <mihgen> it's now High
16:12:02 <mihgen> I want to skip all High's … if it's Critical, then we want to get it in
16:12:03 <aglarendil> this bug is related to dns name resolving on the master node
16:12:12 <mihgen> I can't get an impact on real user
16:12:14 <aglarendil> it is 100% not Critical
16:12:28 <mihgen> angdraug: you are release manager of 5.0.1, please take a closer look ;)
16:12:38 <angdraug> I agree with aglarendil
16:12:46 <angdraug> the fix will be more dangerous than leaving this bug in
16:12:58 <nurla> move it to 5.0.1
16:13:01 <mihgen> ok, thanks, please move
16:13:02 <angdraug> 5.0.2
16:13:07 <nurla> *5.0.2
16:13:10 <nurla> yep
16:13:12 <angdraug> done
16:13:15 <mihgen> is it all about 5.0.1 ?
16:13:33 <angdraug> the rest is oslo.messaging related
16:13:58 <angdraug> I think we're ready to move on
16:14:00 <mihgen> well there is one more
16:14:03 <mihgen> #link https://review.openstack.org/#/c/110599/
16:14:13 <aglarendil> it does not pass CI
16:14:16 <mihgen> don't know if we want to get it in…
16:14:19 <aglarendil> bogdando: what is the status now ?
16:14:21 <mihgen> angdraug: take a look too …
16:14:44 <mihgen> ok let's move ahead
16:15:18 <mihgen> #topic 5.1 bugs
16:15:37 <mihgen> folks, we are 1 week before HCF
16:15:55 <mattymo> mihgen, sorry I stepped away. https://bugs.launchpad.net/fuel/+bug/1346939refs/changes/82/110982/1 I am still working on
16:15:59 <uvirtbot> Launchpad bug 1346939 in fuel/5.1.x "[fuel-library] Puppet does not generate settings.yaml file correctly" [High,In progress]
16:16:07 <mihgen> overall I feel we are more or less Ok, however a lot of QA / DevOps bugs still there
16:16:19 <aglarendil> there is bug with galera
16:16:25 <aglarendil> but we are close to fixing it
16:16:28 <mattymo> I proposed https://review.openstack.org/#/c/110922/ because I don't believe that the reverted patch is the culprit
16:16:41 <aglarendil> #link https://bugs.launchpad.net/bugs/1350245
16:16:42 <uvirtbot> Launchpad bug 1350245 in fuel "Failed to call refresh: /usr/bin/mysql -uwsrep_sst -ppassword -Nbe "show status like 'wsrep_local_state_comment'"" [Critical,In progress]
16:16:49 <aglarendil> holser is working on the patch
16:16:55 <nurla> yep, yesterday was fixed neutron with gre for 5.1
16:17:17 <aglarendil> the actual bug is related to slow environments
16:17:35 <aglarendil> we are going to fix it with renicing of galera state transfer
16:17:36 <mihgen> well cpu should not be eaten for no reason..
16:18:16 <mihgen> among other bugs we have, I'd like to request core devs to move remaining Medium/Low to 6.0
16:18:22 <mihgen> including 6.0
16:18:37 <mihgen> oh folks I've not shared agenda today, but it's in the same location )
16:18:43 <mihgen> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:18:49 <nurla> thx!
16:18:56 <angdraug> btw we still have 46 new bugs :(
16:19:12 <mihgen> I see a lot of medium / low on OSTF, dpyzhov please help with those
16:19:18 <mihgen> as you leading python team
16:19:43 <mihgen> also, rvyalov - there are a lot of bugs on OSCI which are likely to be moved
16:20:25 <mihgen> angdraug: 46 new bugs ?
16:20:38 <xarses> 46 bugs in the new state
16:21:12 <xarses> #link https://bugs.launchpad.net/fuel/+bugs?search=Search&field.status=New
16:21:13 <mihgen> yep, there 46, many are not targeted to 5.1
16:21:26 <akasatkin> 22 for 5.1
16:21:53 <angdraug> new status means that we didn't look at them, every one of them could be a critical
16:21:54 <mihgen> in the current phase of the project, it's highly important to review New bugs on daily basis, and triage them / set milestone sooner than later
16:22:05 <mihgen> angdraug: precisely.
16:22:17 <mihgen> So all - please pay highest attention to those
16:22:49 <mattymo> We could propose a triage duty, where a person is designated to make sure all new bugs get confirmed and set priority and team
16:23:15 <angdraug> one person cannot confirm more than several bugs a day, we should all be doing it
16:23:23 <mihgen> let's try to do it without special events..
16:23:35 <mihgen> if it doesn't work, we will need to group together on Monday
16:23:40 <mattymo> "everyone" means in reality "no one" until someone gets beaten about it
16:23:44 <angdraug> yes, everybody should dedicate an hour or two per day looking only at new bugs
16:24:01 <mihgen> mattymo: I'll get stats from LP karma ;)
16:24:04 <angdraug> team leads get beaten
16:24:21 <mihgen> it must be everyone on *daily* basis!
16:25:09 <mihgen> I think we can move ahead
16:25:12 <mattymo> is it possible to find out who least of all marks bugs from new to another status?
16:25:18 <mattymo> maybe identifying our offenders is a step forward
16:26:10 <angdraug> once again, job for team leads. lets move on
16:26:13 <mihgen> you can take a look on karma
16:26:26 <mihgen> #topic patching of openstack issues
16:26:42 <mihgen> dpyzhov: please provide latest status on it
16:26:50 <dpyzhov> Ok, now we have jobs for tarballs
16:27:09 <mihgen> we know there were serious issues, including in design found..
16:27:12 <dpyzhov> tarballs have been tested manually and seem to work
16:27:21 <dpyzhov> We have pack of bug reports
16:27:27 <dpyzhov> But iso is broken =)
16:27:43 <dpyzhov> It is not urgent, but we need to fix iso on this jobs
16:28:12 <dpyzhov> It has lower priority then setting automated tests for tarballs
16:29:17 <mihgen> what is your estimate when we can get all automated for QA?
16:29:24 <mihgen> so they can start testing it from end to end?
16:29:37 <mihgen> it's piece by piece still up until now..
16:29:46 <dpyzhov> passing question to nurla
16:30:22 <nurla> today we've created all jobs for patching on jenkins
16:30:40 <mihgen> well overall we still don't get 5.1 upgrade tarball which would contain all required items
16:30:41 <dpyzhov> anyway we can not merge changes in buildsystem before 5.0.1 release
16:30:44 <nurla> and i hope its'll start with swarm
16:30:48 <mihgen> and no job which can make it
16:30:50 <dpyzhov> It is too risky for 5.0.1
16:30:56 <mihgen> is it the case?
16:32:21 <mihgen> the issue is that we will provide patching of 5.0.1 envs to 5.0.2, but 5.0.2 doesn't yet exist
16:32:22 <dpyzhov> mihgen: we do have upgrade tarball with patching from 5.0 to 5.0.1
16:32:29 <mihgen> but we have to somehow test it already
16:32:58 <nurla> exactly for this case we created jobs for testing
16:33:06 <mihgen> what's our approach here? it should be ready by 5.1… and we must ensure there will be no issues in patching 5.0.1 -> 5.0.2 , not only 5.0 -> 5.0.1
16:33:18 <mihgen> nurla: do we have repo 5.0.2 ready ?
16:33:26 <mihgen> which we can use now to emulate the process?
16:34:57 <mihgen> well I see we are still not on the same page about this (
16:35:04 <mihgen> let's take it the mailing list
16:35:08 <dpyzhov> ok
16:35:20 <mihgen> another thing about patching
16:35:29 <mihgen> #topic patching of non-openstack packages (ceph, firmware)
16:35:49 <mihgen> our current implementation will patch everything what puppet installs
16:35:50 <mihgen> right?
16:35:59 <rvyalov> mihgen:  repo for 5.0.2 ready
16:36:02 <mihgen> so when we provide updated 5.0.1 / 5.0.2 repo
16:36:12 <mihgen> then it's gonna be an issue, correct?
16:36:48 <mihgen> angdraug: don't we up ceph version in 5.0.1 ?
16:36:55 <angdraug> we do
16:37:20 <mihgen> so when we do patching of 5.0 to 5.0.1
16:37:25 <mihgen> ceph is gonna be updated
16:37:29 <angdraug> yes
16:37:32 <mihgen> which we didn't expect in patching right?
16:37:51 <mihgen> if we just restart ceph by puppet, will it be Ok?
16:37:55 <angdraug> yes
16:38:17 <mihgen> ok then it's fine
16:38:23 <mihgen> however in general
16:38:31 <mihgen> it was only about patching OpenStack
16:38:38 <angdraug> if we patch mysql or rabbitmq it will be more interesting
16:38:46 <xarses> yes, the monitors first preferred, but it should be fine anyway
16:38:48 <mihgen> so the thing is how many more packages were updated already
16:38:50 <aglarendil> no patching of mysql
16:39:18 <mihgen> rvyalov: are you still around?
16:39:26 <mihgen> we must have list of all packages updated in 5.0.1
16:39:31 <rvyalov> mihgen we have problem with patching in 5.0.2 -upstream bug https://tickets.puppetlabs.com/browse/PUP-682
16:39:37 <mihgen> to ensure we don't have something like mysql update
16:39:46 <mihgen> otherwise we are screwed up
16:40:08 <dpyzhov> rvyalov: how does it affects us?
16:40:20 <rvyalov> yes we rebuilding all openstack packages in 5.0.1
16:40:26 <mihgen> rvyalov: need time to read it up
16:40:42 <mihgen> openstack pkgs is fine, what we are rebuilding besides
16:40:59 <rvyalov> dpyzhov: version in 5.0.2 <5.0 in rpm
16:41:14 <dpyzhov> rvyalov: excellent
16:41:27 <rvyalov> yum correct updated packages, but puppet no
16:41:47 <mihgen> rvyalov: omg can that bug be fixed in puppet custom provider or some other monkey-patch?
16:42:00 <ikalnitsky> mihgen: the fix https://review.openstack.org/#/c/106997/
16:42:01 <mihgen> or we will need to deliver new puppet version with a fix?
16:42:23 <mihgen> ikalnitsky: nice :)
16:42:31 <ikalnitsky> mihgen: we need to deliver this fix fo 5.0 and 5.0.1. perhaps during fuel-upgrade
16:43:04 <mihgen> bogdando: excellent :) fix is ready before we find a bug )
16:43:49 <mihgen> ikalnitsky: we don't need to back port it to stable/5.0 before 5.0.1 comes out I think, right?
16:44:26 <rvyalov> need test update from 5.0.1 to 5.0.2
16:44:31 <mihgen> when we release 5.1, we can have this fix in stable/5.0 puppet manifests for updating 5.0 -> 5.0.1, 5.0.1 -> 5.0.2
16:45:07 <mihgen> rvyalov: yep I hope it should be possible
16:45:24 <ikalnitsky> mihgen: we need this fix for both 5.0 and 5.0.1 puppets - without it rollback feature will fail. I think we need to deliver it during fuel-upgrade script , so we can skip backporting to 5.0.1
16:45:55 <rvyalov> problem , if we updated 5.0 to 5.0.2. we can skip to 5.0.1
16:46:02 <mihgen> ikalnitsky: ok so we must merge it in stable/5.0 right after 5.0.1 goes out
16:46:06 <ikalnitsky> mihgen: one more thing : we can update that way - 5.0 -> 5.0.2 and 5.0.1 -> 5.0.2. we don't need to update 5.0 to 5.0.1 and then to 5.0.2
16:46:21 <mihgen> ikalnitsky: yep
16:46:25 <mihgen> rvyalov: wait
16:46:39 <mihgen> what do you mean skip?
16:47:01 <rvyalov> skip backport patch to 5.0.1 (stable/5.0)
16:47:10 <dpyzhov> As I see, we will not have cases when someone will update 5.0 to 5.0.1
16:47:28 <mihgen> rvyalov: patching is not available in 5.0.1
16:47:38 <mihgen> so there is gonna be no issue when 5.0.1 comes out
16:48:10 <mihgen> then we can manage it the way that we put 5.0.2 puppet manifests for updates directly to it from 5.0
16:48:33 <mihgen> actually I don't know why someone would do patching 5.0 -> 5.0.1 when there is 5.0.2 out
16:49:55 <mihgen> ok anymore on patching?
16:50:12 <mihgen> looks like no :)
16:50:22 <mihgen> we are out of topics from the agenda
16:50:31 <angdraug> open discussion?
16:50:38 <mihgen> yep
16:51:37 <mihgen> among other,
16:51:40 <mihgen> #link https://wiki.openstack.org/wiki/Fuel#Nightly_builds
16:51:50 <mihgen> it will have build verification tests soon
16:51:54 <mihgen> which will do HA tests
16:52:07 <angdraug> hurray!
16:52:07 <mihgen> so we will be able to see which night build passes HA
16:52:28 <mihgen> bookwar is working on BVT jobs
16:52:38 <mihgen> thanks teran for torrent, works very well
16:52:41 <angdraug> so it's not the same as internal iso smoke?
16:52:52 <mihgen> I don't think smoke is needed
16:52:56 <mihgen> smoke is centos simple
16:53:05 <mihgen> it's gonna be ubuntu HA & centos HA
16:53:08 <teran> mihgen: you're welcome :)
16:53:14 <mihgen> internally we need smoke for quick verification
16:53:42 <angdraug> make sense, but I'm concerned that we'll have different isos with different ci coverage
16:53:43 <xarses> like angdraug said, so it's not a copy of the internal iso?
16:53:45 <mihgen> teran: I think we need to write an announcement to openstack-dev ML [Fuel] about that we have nightly builds..
16:53:58 <mihgen> it is not
16:54:11 <mihgen> internally we build iso which is MOS'
16:54:14 <xarses> that's bad, we aready have too many build numbers
16:54:17 <mihgen> externally it's Fuel community ISO
16:54:51 <mihgen> which is at the moment different only in the way that it has enabled experimental features by default, and doesn't contain mirantis logs
16:54:54 <mihgen> logos*
16:54:59 <xarses> 1) We need to start using a global build number (or number scheme) so we can tell which job made the iso
16:55:14 <angdraug> xarses: +1
16:55:36 <angdraug> but I'm more concerned about consistent ci coverage across different builds
16:55:38 <mihgen> xarses: angdraug it might be complicated to make
16:55:38 <xarses> 2) If we can't use the same iso as the internal, then we need to make them use the same commit's so we can copy over the bvt results
16:55:58 <mihgen> xarses: agree on #2..
16:56:29 <xarses> mihgen: for 1, if we cant use a single global number then we use scheme like builds 10xxx are from job A
16:56:41 <angdraug> if single iso numbering for both jobs is too hard, we should indicate job type in the file name
16:56:45 <mihgen> I think it should be doable (#2), will talk to devops team about it
16:56:59 <angdraug> and at that rate we're gonna need a wiki page that explains iso file naming conventions :)
16:57:07 <mihgen> anyway BVT is critical priority as we need to give ISO for users to provide feedback before we release
16:57:23 <mihgen> > if single iso numbering for both jobs is too hard, we should indicate job type in the file name
16:57:25 <mihgen> it's there
16:57:29 <mihgen> in /api/version
16:57:39 <angdraug> should be in the file name
16:57:46 <angdraug> not inside the image
16:58:18 <angdraug> like fuel-gerrit-*
16:58:21 <xarses> opening the iso is pita just to find the version number
16:58:22 <nurla> you may find iso version in jenkins job output - grep 'iso version'
16:58:36 <mihgen> angdraug: http://paste.openstack.org/show/89418/
16:59:00 <angdraug> I know all that. people from outside Mirantis might not
16:59:23 <angdraug> we need a public wiki page that explains how to trace where the iso came from
16:59:28 <mihgen> angdraug: true. should be explained
16:59:37 <mihgen> angdraug: +1, I can do it
16:59:45 <mihgen> time to wrap up
16:59:53 <mihgen> thanks folks for participating !
16:59:53 <xarses> nurla: this is what our iso folder looks like http://paste.openstack.org/show/89419/
17:00:14 <mihgen> #endmeeting