Monday, 2016-02-01

*** dims_ has joined #kolla00:02
*** salv-orlando has quit IRC00:03
*** dims has quit IRC00:05
*** tpeoples has quit IRC00:08
*** tpeoples has joined #kolla00:10
*** dwalsh has joined #kolla00:41
*** neilus has joined #kolla00:48
*** salv-orlando has joined #kolla00:49
*** neilus has quit IRC00:52
*** Jeffrey4l_ has joined #kolla01:02
*** dims_ has quit IRC01:04
*** salv-orlando has quit IRC01:23
*** weiyu has joined #kolla01:26
*** dwalsh has quit IRC01:41
*** salv-orlando has joined #kolla01:45
*** iceyao has joined #kolla01:50
*** iceyao has quit IRC01:56
*** Allen_Gao has joined #kolla02:02
*** dims has joined #kolla02:02
*** Marga_ has joined #kolla02:07
*** Marga_ has joined #kolla02:08
*** dims has quit IRC02:12
*** dims has joined #kolla02:12
*** dims has quit IRC02:13
*** salv-orlando has quit IRC02:17
*** tfukushima has joined #kolla02:17
*** kejlly_ has quit IRC02:19
*** iceyao has joined #kolla02:20
*** Marga_ has quit IRC02:35
*** neilus has joined #kolla02:36
*** Marga_ has joined #kolla02:36
*** neilus has quit IRC02:41
*** salv-orlando has joined #kolla02:42
*** coolsvap|away is now known as coolsvap03:02
*** dwalsh has joined #kolla03:02
*** yuanying has quit IRC03:22
*** yuanying has joined #kolla03:22
*** salv-orlando has quit IRC03:26
*** tfukushima has quit IRC03:32
*** tfukushima has joined #kolla03:34
*** tfukushima has quit IRC03:36
*** tfukushima has joined #kolla03:42
*** klint has joined #kolla03:43
*** Allen_Gao has quit IRC03:45
*** tfukushima has quit IRC03:47
*** tfukushima has joined #kolla03:49
*** iceyao has quit IRC03:54
*** iceyao has joined #kolla03:55
*** unicell has quit IRC04:02
*** yuanying has quit IRC04:06
*** unicell has joined #kolla04:10
*** weiyu has quit IRC04:22
*** iceyao has quit IRC04:22
*** salv-orlando has joined #kolla04:24
*** neilus has joined #kolla04:24
*** neilus has quit IRC04:29
*** iceyao has joined #kolla04:30
*** tfukushima has quit IRC04:31
*** achanda has quit IRC04:44
*** iceyao has quit IRC04:45
*** iceyao has joined #kolla04:46
*** Marga_ has quit IRC04:50
*** weiyu has joined #kolla04:51
*** achanda has joined #kolla04:55
*** weiyu has quit IRC05:02
*** salv-orlando has quit IRC05:06
*** tfukushima has joined #kolla05:08
*** Marga_ has joined #kolla05:08
*** tfukushima has quit IRC05:09
*** tfukushima has joined #kolla05:11
*** salv-orlando has joined #kolla05:17
*** jasonsb has joined #kolla05:21
*** jasonsb has quit IRC05:25
*** jasonsb has joined #kolla05:29
*** salv-orlando has quit IRC05:35
*** coolsvap is now known as coolsvap|away05:40
*** tfukushima has quit IRC05:44
*** weiyu has joined #kolla05:45
*** achanda_ has joined #kolla05:47
*** iceyao has quit IRC05:49
*** dwalsh has quit IRC05:49
*** Kennan has quit IRC05:49
*** logan- has quit IRC05:49
*** cloudnull has quit IRC05:49
*** achanda has quit IRC05:49
*** coolsvap|away is now known as coolsvap05:49
*** logan- has joined #kolla05:50
*** iceyao has joined #kolla05:50
*** Allen_Gao has joined #kolla05:56
*** Kennan has joined #kolla05:58
*** Marga_ has quit IRC06:04
*** jasonsb has quit IRC06:08
*** salv-orlando has joined #kolla06:13
*** iceyao has quit IRC06:21
*** dave-mccowan has quit IRC06:21
*** iceyao has joined #kolla06:22
*** Marga_ has joined #kolla06:22
*** chandankumar has joined #kolla06:33
*** cloudnul- has joined #kolla06:39
*** jasonsb has joined #kolla06:47
*** cloudnul- is now known as cloudnull06:51
*** tfukushima has joined #kolla06:55
*** Allen_Gao has quit IRC06:56
*** salv-orlando has quit IRC06:57
*** achanda_ is now known as achanda06:57
*** neilus has joined #kolla07:07
*** jasonsb has quit IRC07:08
*** salv-orlando has joined #kolla07:09
*** Allen_Gao has joined #kolla07:09
*** neilus has quit IRC07:11
*** ppetit has joined #kolla07:25
*** ppetit has quit IRC07:27
*** ppetit has joined #kolla07:29
*** ppetit has quit IRC07:30
*** ppetit has joined #kolla07:36
*** salv-orlando has quit IRC07:43
*** RA_ has quit IRC07:44
*** mbound has joined #kolla08:04
*** Marga_ has quit IRC08:06
*** salv-orlando has joined #kolla08:06
*** fgimenez has joined #kolla08:08
*** fgimenez has quit IRC08:08
*** fgimenez has joined #kolla08:08
*** mkoderer has quit IRC08:14
*** tfukushima has quit IRC08:19
*** tfukushima has joined #kolla08:20
*** dwalsh has joined #kolla08:22
*** salv-orlando has quit IRC08:25
*** achanda has quit IRC08:31
*** coolsvap is now known as coolsvap|away08:36
*** achanda has joined #kolla08:37
*** achanda has quit IRC08:39
*** shardy has joined #kolla08:42
*** mkoderer has joined #kolla08:43
*** diga has joined #kolla08:46
*** kproskurin has joined #kolla08:47
*** kproskurin has quit IRC08:49
*** chandankumar has quit IRC08:49
*** macsz1 has joined #kolla08:50
*** kproskurin has joined #kolla08:51
*** dwalsh has quit IRC09:00
*** diga_ has joined #kolla09:00
*** diga has quit IRC09:01
*** ppetit has quit IRC09:03
*** vbel1 has quit IRC09:04
*** dwalsh has joined #kolla09:06
*** ppetit has joined #kolla09:07
*** salv-orlando has joined #kolla09:07
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510909:14
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml  https://review.openstack.org/27456009:14
*** macsz1 has quit IRC09:19
*** macsz1 has joined #kolla09:20
*** mbound has quit IRC09:20
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Use docker_remove_delay for auto-removing containers  https://review.openstack.org/27021309:22
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Remove rsyslog from all profiles  https://review.openstack.org/27015109:22
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Use private network in default settings  https://review.openstack.org/26954509:22
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml  https://review.openstack.org/27456009:22
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510909:22
*** macsz1 has quit IRC09:22
*** athomas has joined #kolla09:22
*** salv-orlando has quit IRC09:27
*** mbound has joined #kolla09:36
openstackgerritMerged openstack/kolla: Fix missing swift proxy server image in pull action  https://review.openstack.org/27426809:41
*** ppetit has quit IRC09:44
openstackgerritProskurin Kirill proposed openstack/kolla-mesos: Support deploy without configs  https://review.openstack.org/26647609:48
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Remove rsyslog from all profiles  https://review.openstack.org/27015109:50
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Use private network in default settings  https://review.openstack.org/26954509:50
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml  https://review.openstack.org/27456009:50
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510909:50
*** egonzalez90 has joined #kolla09:50
*** ppetit has joined #kolla09:51
*** egonzalez90 is now known as egonzalez09:52
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Dependencies clean up / simplification  https://review.openstack.org/27291209:52
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Add some unit tests for show_tasks  https://review.openstack.org/27384209:52
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Use a new per-service format  https://review.openstack.org/27384309:52
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Fix file_utils.find_config_file (remove kolla)  https://review.openstack.org/27457609:52
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: add a "fake" option to zk_connection()  https://review.openstack.org/27457709:52
*** akwasnie has joined #kolla09:53
*** ajafo has quit IRC09:55
*** ajafo has joined #kolla09:55
*** ppetit has quit IRC09:57
*** ppetit has joined #kolla09:58
*** openstackgerrit has quit IRC10:02
*** openstackgerrit_ has joined #kolla10:02
*** openstackgerrit_ has quit IRC10:03
*** salv-orlando has joined #kolla10:03
*** achanda has joined #kolla10:09
*** achanda has quit IRC10:14
*** dwalsh has quit IRC10:21
*** ppetit has quit IRC10:23
*** tfukushima has quit IRC10:24
*** macsz1 has joined #kolla10:25
*** tfukushima has joined #kolla10:26
*** tfukushima has quit IRC10:26
*** ppetit has joined #kolla10:28
*** ajafo has quit IRC10:34
*** egonzalez has quit IRC10:36
*** salv-orlando has quit IRC10:40
*** ajafo has joined #kolla10:41
*** akwasnie has quit IRC10:43
*** tfukushima has joined #kolla10:46
*** macsz1 has quit IRC10:46
*** salv-orlando has joined #kolla10:50
*** weiyu has quit IRC10:52
*** ppetit has quit IRC10:56
*** macsz1 has joined #kolla10:56
*** ppetit has joined #kolla10:57
*** macsz1 has quit IRC10:58
*** dims has joined #kolla11:09
*** achanda has joined #kolla11:11
*** achanda has quit IRC11:17
*** fgimenez has quit IRC11:19
*** RA_ has joined #kolla11:19
*** fgimenez has joined #kolla11:19
*** iceyao has quit IRC11:22
*** iceyao has joined #kolla11:22
*** chandankumar has joined #kolla11:28
*** jmccarthy has quit IRC11:28
*** jmccarthy has joined #kolla11:29
*** weiyu has joined #kolla11:34
*** akwasnie has joined #kolla11:41
*** macsz has joined #kolla11:45
*** ppetit has quit IRC11:53
*** openstackgerrit has joined #kolla11:53
*** openstackgerrit has quit IRC11:54
kproskurinLooks like http://trunk.rdoproject.org/centos7 is dead :-\11:54
*** openstackgerrit_ has joined #kolla11:54
*** ajafo has quit IRC11:55
*** openstackgerrit_ is now known as openstackgerrit11:55
*** chandankumar has quit IRC11:55
*** ajafo has joined #kolla11:55
*** weiyu has quit IRC11:58
*** weiyu has joined #kolla11:59
*** openstackgerrit has quit IRC11:59
*** athomas has quit IRC12:00
*** diga_ has quit IRC12:02
*** salv-orl_ has joined #kolla12:03
*** weiyu has quit IRC12:04
*** salv-orlando has quit IRC12:06
*** openstackgerrit has joined #kolla12:07
*** ppetit has joined #kolla12:08
openstackgerritAllen Gao proposed openstack/kolla: Remove early failure on install types in dockerfile  https://review.openstack.org/27462912:11
*** weiyu has joined #kolla12:12
*** chandankumar has joined #kolla12:13
*** chandankumar has quit IRC12:15
*** jtriley has joined #kolla12:15
*** fthiagogv has joined #kolla12:26
*** akwasnie has quit IRC12:32
*** chandankumar has joined #kolla12:32
*** akwasnie has joined #kolla12:35
*** fthiagogv has quit IRC12:37
*** fthiagogv has joined #kolla12:40
openstackgerritMerged openstack/kolla-mesos: Fix file_utils.find_config_file (remove kolla)  https://review.openstack.org/27457612:41
*** kproskurin has quit IRC12:42
*** fthiagogv has quit IRC12:42
*** fthiagogv has joined #kolla12:42
*** fthiagogv has quit IRC12:43
*** fthiagogv has joined #kolla12:43
*** salv-orl_ has quit IRC12:47
*** chandankumar has quit IRC12:53
openstackgerritProskurin Kirill proposed openstack/kolla-mesos: Add heat config  https://review.openstack.org/27197813:02
*** salv-orlando has joined #kolla13:02
*** kproskurin has joined #kolla13:06
*** dims has quit IRC13:06
*** shardy has quit IRC13:06
*** dims has joined #kolla13:09
*** dwalsh has joined #kolla13:14
*** weiyu has quit IRC13:17
*** dwalsh has quit IRC13:21
*** dave-mccowan has joined #kolla13:22
*** Jeffrey4l_ has quit IRC13:23
*** fgimenez has quit IRC13:28
*** fgimenez has joined #kolla13:29
*** chandankumar has joined #kolla13:32
*** fthiagogv has quit IRC13:36
*** fthiagogv has joined #kolla13:37
*** fthiagogv has quit IRC13:38
*** fthiagogv has joined #kolla13:38
*** Allen_Gao has quit IRC13:40
*** fthiagogv_ has joined #kolla13:40
*** fthiagogv_ has quit IRC13:40
*** fthiagogv has quit IRC13:40
*** fthiagogv has joined #kolla13:41
*** chandankumar has quit IRC13:46
*** Jeffrey4l has joined #kolla13:47
openstackgerritProskurin Kirill proposed openstack/kolla-mesos: Add heat config  https://review.openstack.org/27197813:48
*** dwalsh has joined #kolla13:50
*** klint has quit IRC14:00
*** jtriley has quit IRC14:01
*** diogogmt has quit IRC14:16
*** akwasnie has quit IRC14:18
*** diogogmt has joined #kolla14:24
*** RA_ has quit IRC14:40
*** diogogmt has quit IRC14:45
*** iceyao has quit IRC15:03
nihiliferSamYaple: kproskurin: guys, getting back to the mesos volume thing15:05
nihiliferi have no idea how to clean named volumes in mesos15:05
nihiliferfor 99% there is no support for it15:05
kproskurinnihilifer: Imo we just need to add to our cleanup tool docker call with wipe of ALL volumes15:06
nihilifercan we provisionaly accept the solution with host-bound columes?15:06
nihiliferkproskurin: and what will call docker on all hosts?15:06
kproskurinnihilifer: well, I didnt though about it in multynode… yet. :-)15:06
nihiliferalso, we use mesos to *not* call docker15:07
nihiliferby hand15:07
kproskurinIn what case we really should dig into mesos PS15:07
nihilifers/dig/contribute to15:07
kproskurinwell, it’s allredy in some experemntal state, so start with test15:07
nihiliferguys, we have the three options15:08
nihilifer1) accept host bound volumes temporarily and wait for named volumes support in mesos15:08
nihilifer2) accept my workaround for using named volumes where there is no clean way to clear volumes (bad solution imo)15:09
nihilifer3) complain to the end of the world and block the development...15:09
kproskurin%-)15:09
nihiliferi'm for 1st solution15:09
nihiliferplease let me know whether you have some other propositions or whether you agree with some point (1-2 of course ;) )15:10
kproskurinHonestly, Im ok with use named volumes(current patch) + create a docker call in cleanup, so it will be working in AIO without any problems. And think about fix for multinode. It will unblock dev, it will work for AIO. We dont really test multinode yet, so...15:11
nihiliferi'll be afk, getting back in ~2hrs probably15:11
nihiliferok, so SamYaple, your vote will be decisive :P15:11
*** achanda has joined #kolla15:18
*** achanda has quit IRC15:23
openstackgerritSteve Noyes proposed openstack/kolla-mesos: Add state info to commands  https://review.openstack.org/27142215:26
*** jtriley has joined #kolla15:26
*** mdnadeem has quit IRC15:29
*** mdnadeem has joined #kolla15:29
*** achanda has joined #kolla15:30
*** kproskurin has quit IRC15:33
*** kproskurin has joined #kolla15:33
*** fgimenez has quit IRC15:35
*** diogogmt has joined #kolla15:36
*** blahRus has joined #kolla15:36
*** blahRus has quit IRC15:37
*** blahRus has joined #kolla15:38
Jeffrey4lSamYaple, morning. Do you depoy the kolla in multi host recently? I try to deploy it(for the first time), but the mariadb always failed.15:38
*** fgimenez has joined #kolla15:38
*** achanda has quit IRC15:41
*** inc0 has joined #kolla15:46
*** dtturner has joined #kolla15:49
inc0good morning15:50
*** kproskurin has quit IRC15:51
*** kproskurin has joined #kolla15:51
pbourkehey inc015:52
pbourkeinc0: Im looking at your nova upgrade as a basis for swift. Wondering why you break out the services into upgrade/compute.yml, upgrade/nova.yml, etc. rather than just 'include start.yml' the way you did in keystone?15:53
inc0pbourke, mostly because nova is more complicated15:54
inc0for the sake of clarity15:54
inc0for ks it's just rebuild of containers reallyh15:54
inc0for nova there are different steps15:54
inc0and I wanted to clearly separate compute part so people would have easier time to work on it15:55
*** salv-orlando has quit IRC15:57
pbourkeinc0: also whats up with the sighup stuff why not just docker restart15:59
*** dave-mccowan has quit IRC16:03
*** ccesario has joined #kolla16:08
*** dave-mccowan has joined #kolla16:18
*** anteaya has quit IRC16:18
*** thumpba has joined #kolla16:19
inc0fuck.16:23
inc0we have problem16:23
inc0upgrades might actually be really bad experience for us16:23
inc0SamYaple, around?16:26
pbourkeinc0: ?16:30
inc0so we run vms in a container16:30
inc0and we started making use of new docker stuff16:31
inc0while being stuch with 1.8 before16:31
*** absubram has quit IRC16:35
*** akwasnie has joined #kolla16:40
*** salv-orlando has joined #kolla16:42
inc0which means restarting docker, which means restarting libvirt container16:42
*** fgimenez has quit IRC16:43
*** tfukushima has quit IRC16:43
inc0which may mean restarting vms, unless we will find out how to use pid=host to our benefit16:43
*** Marga_ has joined #kolla16:44
*** Marga_ has quit IRC16:44
*** fgimenez has joined #kolla16:45
*** Marga_ has joined #kolla16:45
*** diogogmt has quit IRC16:51
*** akwasnie has quit IRC16:51
*** akwasnie has joined #kolla16:51
SamYapleinc0: around16:52
*** diogogmt has joined #kolla16:52
SamYapleJeffrey4l: i have not be able to deploy multinode since i moved (about 2 months back). i wont be able to until right around the time of the midcycle either16:52
inc0SamYaple, restarting docker without restarting running vms16:53
inc0we need to figure it out16:53
SamYaplenihilifer: my vote is always going to be the one to unblock dev since mesos isnt 'stable' yet it doesn't matter what temporary hacks we do. However if at all possible we need to attempt to used named volumes in my opinion16:53
SamYapleinc0: as i can see it there is no solution16:54
SamYapledocker "fixed" that bug in 1.6.x16:54
inc0which means upgrades are no-go16:54
SamYapleso a kill -9 of the docker daemon is the only option to preserve the vms16:54
SamYapleim -2 on a kill -916:54
inc0I don't want upgrades to be no-go16:54
SamYaplethis isn't a killer for upgrades. dont be so dramatic inc0 :P16:55
inc0it is16:55
SamYapleit isn't16:55
inc0all it takes is one unmigrable vm16:55
SamYapleyou dont even need to migrate vms to have an upgrade16:55
inc0and we promised that we'll improve upgradability16:55
SamYaplewhat we _promised_ was upgrades with as little down time as possible16:55
inc0how so?16:55
SamYapleshutting down vms is not the end of the world no matter what you say16:56
inc0which currently means restarting vms, that's more than non-kolla deployed cloud16:56
inc0way more16:56
inc0it actually is killer for most users I know of16:56
SamYapleits not more...16:56
SamYapleI mean i am an operator here. its not ideal, but we most certainly have done it16:57
inc0may be the case for you, but I know ops who can't do stuff like that16:57
*** jasonsb has joined #kolla16:57
inc0there are many different SLAs there you know16:57
SamYapleI know. but its not a killer16:57
SamYapleif thats the way it has to be thats the way it has to be16:58
SamYaplethis is waht live migrate is for16:58
SamYaplecomplain to the docker-dev mailing list and maybe for M-N we will have the feature you want16:58
dmsimardSamYaple: live migration is painful at scale with real SLA'd customers16:59
inc0yeah...but we made decisions fuck up backward compatibility16:59
dmsimardThere are a ton of issues16:59
inc0potentially impossible to solve issues16:59
dmsimardfor instance between different versions of QEMU, you can't live migrate an instance that has both ephemeral and attached volumes, etc.16:59
dmsimardIt's such a headache16:59
pbourkei feel its not so much us that made these decisions as it is the docker team16:59
SamYapledmsimard: i agree, but saying upgrades are a no-go here is a hugely negative uneeded statement17:00
SamYaplepbourke: i agree 100% with that too17:00
dmsimardSamYaple: if it can take weeks instead of months to upgrade I'm cool with that17:00
SamYapledmsimard: no reason it can't17:01
SamYapleyou can limit per compute node upgrades17:01
SamYapleim not recommending running in that state for a long time, but thats what these methods are there for17:01
dmsimardyeah there's things to take into account with api backwards compatibility flags17:01
SamYapledmsimard: for kolla our backend we recommend is ceph for a reason. it does solve alot of those "live-migration" headaches you describ17:02
dmsimardSamYaple: is ephemeral on ceph too ?17:02
SamYapledmsimard: indeed17:02
inc0it does not solve memory-bound vms17:02
dmsimardinc0: yeah we've had customers that would dirty their memory faster than it would transfer17:03
inc0as unmigrable because too much memory get shuffled in17:03
dmsimardnuts17:03
inc0it's unsolvable problem without RDMA17:03
dmsimardwindows was notoriously worse17:03
SamYaplebut all of that is outside kollas control17:03
inc0but that's the world17:03
SamYapleif that compute node dies all the data is down. outside of kolla control17:03
SamYaplewe arent solving those issues17:03
inc0and currently we introduced something that breaks upgradability terribly17:03
SamYaplewe are making the upgrade with as little downtime as kolla can provide as painless as possible17:04
SamYapleyou are freaking otu for no reason again17:04
inc0SamYaple, and we failed to do this because it's considerably more painful with kolla than without17:04
SamYapleI disagree and the operators I know would too (though that is a subset of them for sure)17:04
SamYapleit won't work for everyone but it absolutely will work for some17:05
SamYaplebut thats all beside the point. purpose a solution17:05
kproskurinGuys, could you please review this heat trusts? https://review.openstack.org/#/c/236198 I belive it’s ready to merge.17:05
SamYaple"upgrades are no-go" is unacceptable and not helpful17:05
inc0we might leverage pid=host17:05
SamYaplewe do already17:05
SamYaplewe run with pid=host17:05
inc0even if that means moving qemu out of container after upgrade17:05
SamYaplehell no17:05
inc0I know, that's why it might be solvable problem17:05
SamYaplethats a -2 idea right there17:05
kproskurinSamYaple: I fixed your comments in heat: https://review.openstack.org/#/c/271978/ Takw alook then you have time17:05
inc0might be17:06
SamYaplekproskurin: absolutely ill look now17:06
kproskurinthank you!17:06
SamYapleinc0: what you are asking for is a change in docker. so hit up the mailing list17:06
SamYaplethis was the original behaviour of pid=host and you want it back17:06
SamYaplethats a use case17:06
SamYaplebut its not oging to happen now17:07
inc0what I'm asking is how to solve our biggest priority, even if that breaks cleanliness of solution17:07
SamYapleim -2 on anything touching the host like that at all. im positive sdake is too17:07
inc0or rollback changes that require us to upgrade docker17:07
inc0until docker is ready to solve this problem17:08
SamYaplethese are not options at all. telling operators to migrate vms is an option however17:08
SamYaplewe arent rolling back the entire mitaka cycles worth of ork17:08
SamYaplework*17:08
pbourkeSamYaple: tbh even if inc0 is being dramatic I dont think its wise to dismiss real operator concerns17:13
SamYaplejust fyi dmsimard and inc0. to pull in a new version of qemu you absolutely must migrate/restart the vm.17:13
SamYaplepbourke: im not. im acknowleging it! but its not the end of hte world17:13
inc0but we don't need to upgrade qemu every 6 months17:13
SamYaplewe will absolutely do this without stopping vms if possible17:13
pbourkeok cool17:13
SamYapleinc0: yes. yes we do17:13
inc0we should make this optional17:13
SamYapleits baked into the package repos17:13
SamYaplethere is nothing optional here17:14
inc0keep the old container17:14
pbourkewhich features are we using again that requires 1.917:14
inc0because as dmsimard said - you upgrade one node, say bye-bye to LM to or from it17:14
SamYapleinc0: that wont work....17:14
SamYapleand thats not entirely true17:14
inc0pbourke, 1.1017:14
SamYapleyou can migrate from different versions of qemu to eachother17:14
inc0named volumes, thin containers17:14
*** kproskurin has quit IRC17:15
SamYapleinc0: we _need_ thin containers for no downtime17:15
SamYaplethis was also one of your complaints17:15
dmsimardSamYaple: some version of qemu are NOT compatible17:15
dmsimardI'm not the who took care of this but there was a large issue between qemu 12.04 and 14.0417:15
dmsimardno migration path17:15
inc0and we can't be sure if new version will be compatible17:15
dmsimardwe had to compile our own version with a fix17:15
SamYaplethats fair17:16
SamYaplebut qemu is making efforts to ensure that doesnt happen in the future17:16
SamYapledmsimard: additionally we arent using the 14.04 qemu17:16
SamYaplewe are stepping with cloud-archive for ubuntu17:16
SamYapleand so far all of those versions have been compatible (probably by design)17:16
SamYaplenot saying it is a certain thing, but these are outside of kollas control17:17
dmsimardyeah, I'm not overreacting or anything. I'm just saying that most of the time live migrations are not easy at scale17:17
SamYaplei agree dmsimard. im not dismissing that. but there is only so much kolla can do to fix that at this time17:18
SamYaplewe need _any_ upgrades now, and then we refine them17:18
SamYaplethis is far from a "upgrades are a no-go" problem17:18
dmsimardsure, iterative17:18
inc0it's just - right now we're doing worse than devstack would17:18
inc0and that's....depressing17:18
SamYapledont be such a drama queen17:19
SamYaplewe arent17:19
SamYapleif qemu needs tob e upgraded it has to be17:19
SamYapleand last i checked devstacks multinode is awful17:19
inc0you should be able to run qemu 2.4 for some time17:20
inc0it's not every 6 months problem17:20
SamYapleit is when you have to follow canonical17:21
*** rhallisey has joined #kolla17:21
SamYapleand they have the newest versions every 6 months17:21
*** unicell has quit IRC17:21
SamYaplethis is whats "supported"17:21
*** akwasnie has quit IRC17:23
*** ssurana has joined #kolla17:31
*** daneyon_ has joined #kolla17:32
*** absubram has joined #kolla17:33
*** daneyon has quit IRC17:35
*** rhallisey has quit IRC17:36
*** salv-orl_ has joined #kolla17:36
*** salv-orlando has quit IRC17:39
*** fgimenez has quit IRC17:43
*** akwasnie has joined #kolla17:44
*** akwasnie1 has joined #kolla17:49
*** akwasnie has quit IRC17:51
openstackgerritPaul Bourke proposed openstack/kolla: Fix image plugin functionality for oslo.config  https://review.openstack.org/26821117:52
*** pbourke has quit IRC17:59
*** pbourke has joined #kolla18:00
openstackgerritSam Yaple proposed openstack/kolla: Rename kolla_ansible to kolla_toolbox  https://review.openstack.org/27415418:03
pbourkeinc0: if you get a chance could you address the question around sighup on the nova review? would be good to know as am hoping to attack swift tomorrow18:04
pbourkethanks :)18:04
*** unicell has joined #kolla18:04
inc0pbourke, sighup is less disruptive18:05
inc0and works well, there is no need for restart imho18:05
*** dwalsh has quit IRC18:06
openstackgerritJeffrey Zhang proposed openstack/kolla: Playbook for horizon upgrade  https://review.openstack.org/27482718:07
SamYapleinc0: i think we need a conversation around that18:07
SamYapleinc0: i want to start doing that for config updates, but tehre was... concern when i mentioned it before18:07
inc0which was..?18:07
SamYapleit was never actually stated18:07
SamYaplejust "concern"18:07
inc0I mean apart from "not everything supports it"18:07
inc0what was the concern?18:08
SamYaplethats my point18:08
SamYapleit was just stated as "concern" so we need to have the conversaion18:08
inc0ok...who has these concerns?18:09
SamYaplelast time the conversation was had i don't think you were even part of the community18:09
*** jasonsb has quit IRC18:09
SamYaplei dont recall everyone, but sdake was one18:10
inc0sdake commented in my review that he liked this solution18:10
inc0so I guess let's have a conversation when someone have "concern";) I'm happy to have it18:10
SamYaplewell well have it again in a wider audience to see if anyone sees an issue with this approach.18:11
SamYaplelast time it was suggested was around the time of the kolla-ansible spec18:11
inc0that's what gerrit is for18:13
inc0if someone has this concern, he/she is free to review this patch18:13
inc0but unless someone actually does that, I assume it's ok18:14
SamYaplegerrit is not for a wider audeience. you are thinking of the mailing list18:14
inc0as for sdake's concern, https://review.openstack.org/#/c/254395/12/ansible/roles/nova/tasks/upgrade/reload.yml he did comment on that specific one18:14
*** Jeffrey4l has quit IRC18:14
inc0I did in upgrades thread18:15
*** dtturner has quit IRC18:15
inc0I specifically said there I'm going to sighup services18:15
openstackgerritQiu Yu proposed openstack/kolla: Fix Swift for multinode P+ACO deployment  https://review.openstack.org/27421518:15
SamYapleinc0: im not in the mood to do this with you today18:15
inc0me neither, trust me18:15
SamYaplethen drop it and move on18:15
*** limamauricio has joined #kolla18:15
*** Marga_ has quit IRC18:16
inc0feel free to start ML thread if you feel there is need for conversation about that, I'm happy to have it18:16
*** akwasnie1 has quit IRC18:17
limamauricio Hello, I'm having some difficulty to give root privileges to manila access some directories. /var/lib/manila and /var/cache/manila18:17
SamYaplelimamauricio: what trouble are you talking about?18:18
SamYapleduring build or after?18:18
limamauricioAfter18:18
limamauriciobuild and deploy works18:18
SamYaplei believe the container is running as the manila user18:19
SamYapleso in the build you would have to chown those folders18:19
SamYapledo you have a patchset you are wrokging on?18:19
limamauriciopermissions are given in the build or deploy?18:21
*** dims_ has joined #kolla18:21
*** dims has quit IRC18:22
SamYaplelimamauricio: both, but mostly build18:22
limamauricioI do not think I ran the build after my last change.18:23
limamauricioShould be it18:24
limamauricioThank you18:24
limamauricioI'll check it now18:24
SamYaplelimamauricio: ok let me know!18:25
limamauricioI run build once, can I run again?18:26
limamauricioor you have to clean up for work?18:26
limamauricioI have*18:27
SamYaplelimamauricio: you can run it again18:27
limamauricioOk18:27
SamYaplethe images it puts out from a successful run _should_ be the latest18:27
limamauriciook18:27
SamYaplethough you may want to clean all images and rebuild fresh as a final test to ensure it works as desired18:27
limamauriciook]18:28
*** Marga_ has joined #kolla18:51
*** ppetit has quit IRC18:53
*** rmart04 has joined #kolla18:57
*** salv-orl_ has quit IRC19:00
*** ajafo has quit IRC19:21
*** thumpba has quit IRC19:23
*** thumpba has joined #kolla19:26
*** ajafo has joined #kolla19:32
*** dave-mccowan has quit IRC19:57
*** salv-orlando has joined #kolla20:03
*** salv-orlando has quit IRC20:07
*** dave-mccowan has joined #kolla20:16
*** nihilifer has quit IRC20:26
*** nihilifer has joined #kolla20:30
SamYaplehas anyone tried to boot a vm on a fresh build from master in the past few days?20:49
SamYapleI am getting this bug https://bugs.launchpad.net/kolla/+bug/154057920:49
openstackLaunchpad bug 1540579 in kolla "nova-api -- No sql_connection parameter is established" [Undecided,New]20:49
*** salv-orlando has joined #kolla20:53
*** thomasem is now known as thomasem-away20:53
dmsimardSamYaple: we're seeing this in RDO as well20:56
dmsimardSamYaple: I haven't found the fix but I found this https://review.openstack.org/#/c/274703/20:57
dmsimardhttp://lists.openstack.org/pipermail/openstack-dev/2016-February/085497.html20:57
dmsimardI don't know if there's a relationship, I haven't looked much at all yet20:57
SamYapledmsimard: ok cool not crazy then20:58
dmsimardSamYaple: if you find the problem, do let me know - I'll do the same, I subscribed to the bug in the meantime :)20:58
*** thumpba has quit IRC21:00
*** dims has joined #kolla21:02
SamYapledmsimard: i did a git blame on nova _and_ oslo.db right after it first happened21:03
SamYaplecould not track it down21:03
*** dims_ has quit IRC21:03
SamYaplenot that oslo.db is running master in kolla or anything, but still21:03
dmsimardyeah it's a mess, it could be almost anyone's fault21:03
SamYapleit might be oslo.config21:03
dmsimardI saw something about moving to keystoneauth1 in nova too21:03
SamYaplebut i can't see it being anything else21:03
SamYapleit doesn't seem to be an auth issue21:04
SamYaplethe object itself was missing info, so maybe oslo issue somewhere...21:04
SamYaplewhat recently bumped?21:04
SamYapledmsimard: i bet i found it https://github.com/openstack/requirements/commit/0520b3a219560c79da43c0bd27133377da1459cb21:04
SamYapleshould be a quick test21:05
dmsimardI just asked #openstack-dev to see if anyone else has seen it21:05
dmsimardSamYaple: are you getting that from source too ? on which distro ?21:05
*** unicell has quit IRC21:06
SamYapledmsimard: ubutnu source21:08
SamYaplei bet its that oslo.config change21:08
SamYapleim rebuilding now21:08
SamYapletestign ETA 15m21:08
dmsimardthat's very cool, seeing that on binary RDO21:08
dmsimardcool as in it's not isolated to one type21:08
SamYapleyea21:08
dimsSamYaple : we haven't released any libs today21:19
SamYapledims: it happened last week21:20
SamYaplethought it was isolated to me21:20
SamYaplei think it was the oslo.config bump to 3.4.021:20
SamYapleim testing that theory21:20
dimsSamYaple : ack https://pypi.python.org/pypi/oslo.config/3.4.0 was on Jan 26th21:21
SamYapleyea then globals got bumped right after dims21:21
SamYapleshouold know momentarily21:21
dimsSamYaple : oslo.db was on 1/23 as well - https://pypi.python.org/pypi/oslo.db/21:21
SamYapleits not oslo.db, tested taht last week21:22
SamYaplewell i say not, its _proabbly_ not21:22
dimsSamYaple : cool :)21:22
SamYapleits downgrading log fixes it ill git blame oslo.log and narrow it down21:22
SamYapleim thinking its a nova bug oslo brought out though, just my gut21:23
*** mbound has quit IRC21:23
dimsSamYaple : thanks! just holler on either the Nova or the Oslo channels :)21:23
SamYaplewill do dims21:23
dimsThanks!21:24
SamYaplegit blame oslo.config* dims21:25
SamYaplewrong oslo project ;)21:25
*** dwalsh has joined #kolla21:32
SamYapledims: doesn't look like older versions of oslo fixes this, going to pbr and see if i can figure it out21:47
*** tpot has joined #kolla22:01
SamYaplearrrrrggg dims dmsimard https://github.com/openstack/nova/commit/df0fca62cf5324f5b6eae0fed1f88c6c9ed61c7122:02
SamYaplespecifically https://github.com/openstack/nova/blob/df0fca62cf5324f5b6eae0fed1f88c6c9ed61c71/releasenotes/notes/request-spec-api-db-b9cc6e0624d563c5.yaml22:02
*** thumpba has joined #kolla22:02
SamYapleinc0: you too22:02
SamYaplethats goign to hit upgrades22:02
dmsimardWow, there's a nova-api database now ?22:03
SamYapleapparently22:03
SamYaplei need a drink22:03
SamYapleill push up a patch when i get back22:03
dmsimardSamYaple: I was going to open a bug to puppet-nova but someone beat me to it https://bugs.launchpad.net/puppet-nova/+bug/153979322:05
openstackLaunchpad bug 1539793 in puppet-nova "Nova Mitaka requires a new nova_api database" [High,In progress] - Assigned to Dan Prince (dan-prince)22:05
inc0omfg22:06
inc0and I mean omfg22:06
*** thomasem-away is now known as thomasem22:09
SamYapleinc0: right?22:09
dmsimardWhy don't they use the nova database :(22:09
dimsSamYaple : i had no idea :)22:09
SamYapleit was dims GET HIM!22:10
dimsLOL22:10
dimsi would have expected to have defaulted to the older config(s) if the new one was not there...22:10
SamYapledims: why need second database tho?22:10
SamYaple:(22:10
dimslet's ask alaski over on #openstack-nova22:11
*** tpot has quit IRC22:12
dmsimardI'm coming with the popcorn22:13
*** thomasem is now known as thomasem-away22:15
*** kproskurin has joined #kolla22:15
kproskurinHey guys, could you pls look into heat trusts review? https://review.openstack.org/#/c/236198/ Im really think it's ready for merge22:16
SamYaplekproskurin: looking22:16
SamYaplekproskurin: my job there is done22:17
kproskurinSam, you allready have +222:17
kproskurin:-)22:17
kproskurinSome1 else? :-)22:18
SamYapleinc0: api_database has been there since liberty22:19
SamYapleinc0: join #openstack-nova22:19
*** jtriley has quit IRC22:21
inc0it's like...we made rule to use conductor for every db write...but we can't now, so let's make new db to not use conductor22:25
*** inc0 has quit IRC22:26
*** RA_ has joined #kolla22:28
*** dwalsh has quit IRC22:42
*** yuanying has joined #kolla23:02
*** RA_ has quit IRC23:02
openstackgerritSam Yaple proposed openstack/kolla: Create nova-api database  https://review.openstack.org/27492823:04
*** kproskurin has quit IRC23:09
*** RA has joined #kolla23:19
*** RA is now known as Guest5147823:20
*** rmart04 has quit IRC23:27
*** alisonh has quit IRC23:27
*** alisonh has joined #kolla23:33
*** blahRus has quit IRC23:37
*** salv-orl_ has joined #kolla23:37
openstackgerritMerged openstack/kolla-mesos: Remove hostname-unique pidfile for mariadb  https://review.openstack.org/27458423:39
*** stvnoyes has quit IRC23:40
*** salv-orlando has quit IRC23:40
*** stvnoyes has joined #kolla23:41
*** diogogmt has quit IRC23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!