Friday, 2016-02-05

*** daneyon has quit IRC00:03
*** daneyon_ has quit IRC00:09
*** dave-mccowan has joined #kolla00:11
sdakeSamYaple is there some reason /dev is bindmounted in ansible?00:29
sdakein kolla_ansible00:29
*** neilus has joined #kolla00:32
SamYaplesdake: yup00:36
sdakewell with master ansible it doesn't work00:36
SamYapleyoull have to be more specificate00:36
sdakesee review - there is some debugging in it00:36
sdakeyou will understand when you look in review00:37
sdakehttps://review.openstack.org/#/c/276328/100:37
SamYapleI don't understand your question00:37
sdakeread the pastes please00:37
sdakeit will take you 5 minutes00:37
sdakeone with dev bindmounted00:38
sdakeone without00:38
sdakewith that patch applied00:38
SamYapleI have already read the review00:38
sdakeok, so you see how ansible fails with dev bindmounted in the first comment?00:38
SamYaplei see your paste, i dont see that behaviour, no00:39
sdakethe err 1300:39
*** dims_ has joined #kolla00:39
sdakepermission denied comes from multiprocessing library00:39
sdakebecause of some wierd interaction with not being able to read stuff in /dev/shm00:39
sdakeso the patch as is doesn't work with a dev bindmount00:40
sdakethat means new versions of ansible wont work with a dev bindmount, atleast not docker 1.8.200:40
*** dims has quit IRC00:40
SamYapleno it means you need to fix your host /dev/shm00:40
sdakethe second paste removes the dev bindmount00:40
sdakeand the original problem still exists00:40
SamYapledev is required00:40
SamYapleand i cannot reproduce the stack in the bug00:40
sdakecool i am up to fixing my /dev/shm00:40
sdakewe can document that00:41
SamYapleso youll have to take over this bug as it is centos only00:41
sdakewhat precisely needs fixing00:41
*** neilus has quit IRC00:41
SamYapleyoull have to track that down sdake00:41
SamYapleits on you00:41
sdakesecond paste00:41
sdakeshows the ordering doesn't work00:41
SamYapledoesnt happen on ubuntu00:41
sdakeon stable/liberty?00:41
sdakedid you try it00:41
SamYapleyup00:41
SamYapleyup00:41
SamYapleso did inc000:42
SamYapleno issues00:42
sdakedid he report that in channel today?00:42
SamYapleyup00:43
sdakeplease educate why /dev needs bindmounting00:43
SamYaplebecause we containerized the find_disk module00:43
sdakethanks00:44
*** aginwala has quit IRC00:44
sdakesamyaple http://eavesdrop.openstack.org/irclogs/%23kolla/%23kolla.2016-02-04.log.html#t2016-02-04T21:43:1700:52
*** farid has joined #kolla00:52
sdakemy intpretation of that log is that inc0 did not test with haproxy, and as a result was not testing the kolla-ansible change you made00:52
sdakeand hence, didn't hit the ordering error00:52
SamYaplebut thats not how kolla works00:53
sdakei understand kolla ansible  gets calleda a million times00:53
sdakebut he didn't say your patch worked00:53
sdakehe said it worked dwithout your patch00:53
SamYapleim not claiming we need my patch00:53
sdakeyour patch doesn't work00:54
SamYapleit does on ubuntu00:54
SamYaplebut ubuntu isnt the one with either of these issues00:54
sdakewith ubuntu you build everything from source00:54
sdakeso thats a good data point00:54
SamYapleso ignore my patch and figure out why centos is busted would be my suggestion00:54
sdakeyup00:54
sdakei am going to build source centos00:54
*** salv-orl_ has quit IRC00:54
sdakeif that works, then my speculation would be two packages installed at once00:55
sdakepip and rpms00:55
* sdake hates computers00:55
SamYaplewhich is what mandre stated he just never finished his patch00:55
sdakedid he write that in the bug log?00:56
sdakeor irc00:56
mandrehey00:56
sdakehey mandre00:56
SamYapleboth00:56
*** farid has quit IRC00:57
mandrei see you're talking about mysql failing in kolla-ansible container00:57
sdakemandre working on https://bugs.launchpad.net/kolla/liberty/+bug/153096200:57
openstackLaunchpad bug 1530962 in kolla liberty "six version dependency in kolla_ansible appears to be incorrect." [Critical,Confirmed] - Assigned to Sam Yaple (s8m)00:57
sdakewell i dont have my lab gear so i had to build a vm which took all damn day00:58
sdakebut yup i'm good to go now :)00:58
SamYaplemandre: yea i cant reproduce anywhere so im going to step out of this conversation00:58
sdakemandre what do you think the problem is00:58
sdakethanks for tyring SamYaple00:58
sdaketrying00:58
mandreyeah, it's been on my todo list for some time, just never got around to finish it00:58
mandreI can reproduce, I was planning to tackle the issue today actually00:59
*** farid has joined #kolla00:59
sdakegood lets both work on it00:59
sdakebecause stable is broken00:59
sdakeand that needs to be fixed asap00:59
sdakesound like a plan?01:00
mandrenot sure, what the issue is sdake, but I suspect version mismatch]01:00
sdakemandre i think its pip and rpm package installed at same time01:00
sdakespeculation01:00
sdakei am goign to build centos source now01:00
mandresdake, it's only broken for centos-binary01:00
mandrecentos source works fine01:00
sdakewell there ya go01:00
sdakethanks for that info that saved a couple hours01:01
mandrei'll submit a proper fix today hopefully01:01
*** Allen_Gao has joined #kolla01:02
sdakehow are you thinking of fixing it01:02
mandrenot sure yet01:03
mandrei'll find something01:03
mandreactually, ansible is totally busted in that container01:03
mandre'ansible --version' also generates a traceback01:04
sdakelets work on this together mandre01:04
sdakei need this quickly as well01:04
sdakewhat other data do you have01:04
mandrepip shows it's ansible 2.0.0.2, and six 1.9.001:05
sdakemy experience with six errors is its actually a different dependency01:06
sdakeor two versions of six installed at once01:06
sdake2900 commits in kolla01:08
sdakecrazy growth01:08
mandrethe "AttributeError: 'module' object has no attribute 'add_metaclass'" for six makies me think of an old (< 1.4) version of six, but I can't find where it's coming from01:08
*** daneyon has joined #kolla01:08
*** daneyon_ has joined #kolla01:09
sdakeyup i hear you01:10
*** daneyon has quit IRC01:13
openstackgerritSam Yaple proposed openstack/kolla: Convert Neutron to thin containers  https://review.openstack.org/27271501:13
sdakei have found the problem01:13
sdakerpm installs six 1.9.001:13
sdakeansible installs six 1.10.001:14
sdakewe dont need 1.9.001:14
mandresdake, have you tested?01:15
sdakemandre i looked at the typescript01:15
sdakei am testing now01:15
sdakewtb backward compatible libraries in python01:16
*** iceyao has joined #kolla01:16
*** Jeffrey4l has joined #kolla01:18
*** daneyon_ has quit IRC01:18
sdakemandre do you get a problem with the ansible container about /dev semlock?01:19
sdakeok that works01:19
sdakenow I get can't connec tt o my sql server at 10.0.3.1501:20
sdakewhich is kind of what i expect with the fact that my vm networking is fucked01:20
sdakemandre when does your work day begin?01:20
sdakemandre http://paste.fedoraproject.org/318707/54635261/01:21
sdakei'm a bit jammed up with this virtualbox networking setup01:21
sdakeI have it set to nat01:22
sdakebut I think it needs to be set to something else01:22
mandresdake, incredibly well... you're fixing the bugs assigned to me :)01:22
*** neilus has joined #kolla01:22
sdakewell i assigned it to me but you  can assign it to you if you finis htthe job01:22
sdakei just want it fixed :)01:22
mandrethe yum clean all on line 20 is redundant01:23
*** Allen_Gao has quit IRC01:24
*** Allen_Gao has joined #kolla01:28
*** neilus has quit IRC01:28
mandresdake, just like you did in kolla-ansible removing python-six worked for me01:32
*** sdake has quit IRC01:38
openstackgerritMartin AndrĂ© proposed openstack/kolla: Uninstall conflicting python-six in kolla-ansible  https://review.openstack.org/26886301:41
mandresdake, let's see if the gate agrees with us01:41
mandreinc0: if you're reading this, I'm in Japan until end of March01:45
*** alisonh has quit IRC01:45
*** jtriley has joined #kolla01:47
*** aginwala has joined #kolla01:47
*** tfukushima has joined #kolla01:50
*** iotapi322 has joined #kolla01:52
iotapi322Hi All, looking for a little help.  I'm trying to run a build of kolla.  I've tried with the tagged release of 2.000b from github, but I can't seem to get all the containers to build.01:54
iotapi322Is there any wiki pages or blog pages that point to a known good instructions on how to get a good build up and running, maybe a CI loop i can look at?01:54
*** sdake has joined #kolla01:59
sdakemandre sorry i had to take a minor emergency in the household01:59
sdakeso I may hae missed what you said01:59
sdakemandre does that deploy for you, and my virtualbox networking is just busted?02:00
sdakethe patch isn't correct, it needs a conditional around the rpm removal02:00
sdakeand /dev needs to be sorted out - probably on centos in some way02:00
sdakeSamYaple did the find_disks.py get backported into stable after it was containerized?02:01
sdakef02:01
sdake11/20/201502:03
sdakewas when that change went in02:03
sdakewe tagged 10/21/2015 the release02:03
sdakethat change may need to be reverted in stable02:03
*** alisonh has joined #kolla02:04
*** jasonsb has joined #kolla02:10
*** jasonsb has quit IRC02:12
*** tfukushima has quit IRC02:13
*** jasonsb has joined #kolla02:13
sdakef02:16
*** neilus has joined #kolla02:18
*** aginwala has quit IRC02:18
iotapi322anyone have a blog post with instructions or maybe even a bash script to share that start from scratch and build out a kolla liberty release?02:19
iotapi322or maybe someone can point me to the RDO CI loop?02:19
sdakeiotapi32202:19
sdakedocs.openstack.org/developer/kolla02:19
sdakeactually02:20
sdakethose are master docs02:20
sdakeyou will need to read the docs for the liberty release specifically which are here:02:20
iotapi322Yes I' have followed those directions and keep running into different errors such as " he command '/bin/sh -c echo 'binary not yet available for centos'     && /bin/false' returned a non-zero code: 1 ERROR:__main__:mistral-engine:Parent image error'd with message "error""02:21
iotapi322on numerous containers after several build retries02:21
iotapi322I'm just looking for someone to share a known good version...02:21
iotapi322like release tree tag 2.0.0.b with centos as the base02:22
*** aginwala has joined #kolla02:22
sdakehttps://github.com/openstack/kolla/tree/1a1958f209d69abfdc6634d687824f1ace62a626/doc02:22
*** alisonh has quit IRC02:22
sdakeiotap some containers are not available for centos in binary form because rdo has not yet released them02:23
sdakemistral is one of those02:23
sdakethose are just warnings, as long as you dont actually deploy mistral you will be good to go02:23
iotapi322i get those for murano, mistral,  and gnocchi,02:24
sdakeI would recommend stable/liberty but we are working on 1 bug that was broken in an upstream ansible change that broke all of stable/liberty for kolla02:24
sdakeif mandre appears we should have a fix in 1-2 hours02:24
sdakeit has been root caused02:24
sdakemy house is in a remodel, so my lab is busted, so I am running on vms02:25
sdakeand I can't test the change02:25
sdakeuntil I figure out how to operate virtual box ;)02:25
iotapi322been there with the house thing02:25
sdakeyup we are on room 5 of 6 for 1100 square feet of tile02:25
sdakeand then done and back in action yay :)02:26
sdakebut I am doing all the furniture moving and floor demo02:26
sdakewhich sucks ass02:26
sdakeiot is your goal to deploy kolla for eval purpose?02:26
sdakeiotapi32202:27
mandresdake, the one liner did fix kolla-ansible but we seem to have different issue now: https://jenkins05.openstack.org/job/gate-kolla-dsvm-deploy-centos-binary/244/console02:27
iotapi322been there we moved into a fixer up a few years ago and lived up stairs while the kitchen, and master was a complete demo reno... I would go to work, then work on the house at night.  I lost a few years of my life on that project02:27
sdakeiotapi322 always have to have a project :)02:27
iotapi322yes eval purposes, and home lab.  I work for Cisco, and we have mirantis doing a deployment and I'm evaluating another way to do openstack, because deploying openstack is HARD.02:28
iotapi322oh wait sdrake, you must know Victor02:28
sdakemandre that is an oslo bug02:28
mandreyou know the fix?02:28
sdakemandre there is a patch in master for it02:28
sdakesecond02:28
*** farid has quit IRC02:29
*** farid has joined #kolla02:30
*** neilus has quit IRC02:30
sdakemandre https://github.com/openstack/kolla/blob/master/docker/openstack-base/oslo-log-bug1499620.patch02:31
sdakethat is the same patch i wrote that got -202:31
sdakenot sure how it made it in the repo02:31
sdakeoh well ;)02:31
sdakehttps://github.com/openstack/kolla/blob/master/docker/openstack-base/Dockerfile.j2#L4202:32
sdakehttps://github.com/openstack/kolla/blob/master/docker/openstack-base/Dockerfile.j2#L6902:32
mandrethat patch is only applied on ubuntu binary02:32
sdakeya mine was global02:33
sdakesam didn't like my fix and -2 it02:33
sdakelet me see if i can find his solution moment02:33
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Dependencies clean up / simplification  https://review.openstack.org/27291202:33
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Add some unit tests for show_tasks  https://review.openstack.org/27384202:33
*** tfukushima has joined #kolla02:33
sdakehttps://github.com/openstack/kolla/blob/stable/liberty/docker/openstack-base/Dockerfile.j2#L8002:35
sdakethat is in stable already02:36
sdakeapparently it does not work02:36
sdakebecause it is source only02:36
sdakemandre what needs to happen is that ubuntu binary patch needs to be made to work on centos as well02:36
*** alisonh has joined #kolla02:36
*** daneyon has joined #kolla02:37
mandreor something requires the broken version of oslo.log02:38
sdakenothign requires oslo log 1.1102:38
sdakerdo is just lazy on updating dependencies02:38
mandrewell, we might have the same type of issue with install oslo-log rpm02:39
sdakenot if you apply teh patch02:40
sdakeat some point if they udpate the repo we will have to revert the patch02:40
sdakei have already filed a bug in red hat's bug tracker for this issue months ago02:40
iotapi322anyway: so the you would recommend stable/liberty, with base_distro = centos?  This is a known good configuration?02:40
sdakei'm surprised they haven't moved the version yet02:40
mandrei don't like that solution, I prefer to get a working package02:40
sdakemandre lets do both :)02:41
*** daneyon_ has joined #kolla02:41
sdake i will find the bugzilla for you so you can push appropriate buttons02:41
mandrecentos-source deploys fine in liberty btw02:42
*** achanda has joined #kolla02:42
*** daneyon has quit IRC02:43
iotapi322mandre, thanks,02:46
iotapi322alright, I'll give it a try and give some feedback02:59
*** unicell has quit IRC03:01
iotapi322Yeah tried to do a deploy with what I had: TASK: [nova | Creating nova-compute volume] ***********************************  failed: [localhost] => {"changed": true, "failed": true} msg: InvalidVersion('volumes is not available for version < 1.21',)  FATAL: all hosts have already failed -- aborting03:02
*** yuanying_ has joined #kolla03:18
*** yuanying has quit IRC03:21
*** aginwala has quit IRC03:22
sdakeiotapi322 if your using master ou need docker 1.1003:28
sdakeit was tagged about 4 hours ago :)03:28
sdakeenjoy03:28
sdakethe docs are out of date03:28
sdakelet me fix that03:28
iotapi322ok,  i was using the tag 2.0.0.b03:29
iotapi322off of the stable release03:29
sdake1.9 needed for that03:29
iotapi322ah ok03:29
sdakeit should be in the docs03:29
iotapi322i was doing the downgrade to 1.8 from the webpage03:29
sdakewe do a pretty good job of keepign the dependency requireements up to date03:29
iotapi322ok, i'm out for the night.  THanks03:30
sdakecome back tomrorow03:30
sdakewe will be happy to help ;)03:30
iotapi322will do!03:31
sdakethis channel runs 24/703:31
*** dims_ has quit IRC03:33
*** sdake has quit IRC03:37
*** sdake has joined #kolla03:38
*** yuanying has joined #kolla03:42
mdnadeemajafo, I will do, Thanks :)03:43
*** yuanying_ has quit IRC03:44
openstackgerritMartin AndrĂ© proposed openstack/kolla: Workaround for oslo.log 1.10.0 in centos-binary  https://review.openstack.org/27655503:46
*** daneyon_ has quit IRC03:57
*** achanda has quit IRC03:59
*** dave-mccowan has quit IRC04:00
sdakemandre https://review.openstack.org/#/c/268863/3/docker/kolla-ansible/Dockerfile.j204:02
mandresdake, yeah I've seen your comment, I want to get the gate results for the oslo.log patch before updating this one04:04
sdakeyour patches are in the wrong order for that mandre04:04
sdakeunless that second one is really the first one04:04
sdakelearning my way around the new gerrit interface :)04:04
*** sdake has quit IRC04:05
mandrefirst one is removing python-six, which exposed the oslo.log bug04:06
*** yuanying has quit IRC04:06
mandresdake, use gertty, seriously04:07
*** sdake has joined #kolla04:07
*** yuanying has joined #kolla04:07
sdakeback04:08
*** alisonh has quit IRC04:12
mandrei'll generate the proper diff once i have the container building04:14
*** sdake_ has joined #kolla04:16
*** sdake has quit IRC04:17
*** ssurana has joined #kolla04:27
openstackgerritMartin AndrĂ© proposed openstack/kolla: WIP Workaround for oslo.log 1.10.0 in centos-binary  https://review.openstack.org/27655504:36
openstackgerritMartin AndrĂ© proposed openstack/kolla: Uninstall conflicting python-six in kolla-ansible  https://review.openstack.org/26886304:36
*** tfukushima has quit IRC04:39
*** alisonh has joined #kolla04:49
*** achanda has joined #kolla04:58
*** aginwala has joined #kolla04:59
*** sdake_ has quit IRC05:03
*** salv-orlando has joined #kolla05:08
*** sdake has joined #kolla05:16
*** salv-orlando has quit IRC05:18
*** tfukushima has joined #kolla05:18
*** Marga_ has quit IRC05:19
*** iceyao has quit IRC05:20
*** sdake_ has joined #kolla05:31
*** v1k0d3n has joined #kolla05:32
*** sdake has quit IRC05:32
openstackgerritMartin AndrĂ© proposed openstack/kolla: Workaround for oslo.log 1.10.0 in centos-binary  https://review.openstack.org/27655505:35
mandre^ please review, this fixes centos-binary in liberty05:36
*** sdake_ has quit IRC05:44
mdnadeemajafo, ping05:45
nihiliferajafo is probably sleeping now05:46
mdnadeemnihilifer, I face an issue while writing code for Ubuntu_binary for designate05:49
mdnadeemJust for testing purpose I try to install designate api on my Ubuntu server (14.04)using  command05:50
mdnadeem$ sudo apt-get install -y --no-install-recommends designate-api05:50
*** Marga_ has joined #kolla05:50
mdnadeem>> it report E: Unable to locate package05:51
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add nova config  https://review.openstack.org/26502105:51
nihilifermdnadeem: and you're using ubuntu cloud archive as a repo?05:51
mdnadeemnihilifer, I have not create any repo, may be it use default repo05:52
*** Marga_ has quit IRC05:53
nihilifermdnadeem: i don't mean creating any repos... i'm asking whether you're using https://wiki.ubuntu.com/ServerTeam/CloudArchive or some other apt repo05:53
*** Marga_ has joined #kolla05:54
*** neilus has joined #kolla05:54
*** Allen_Gao has quit IRC05:58
mdnadeemnihilifer, I guess my server  use cloud archive05:59
*** Allen_Gao has joined #kolla05:59
*** neilus has quit IRC05:59
nihilifermdnadeem: just check your /etc/apt/sources.list.d06:05
nihiliferbut nvm06:05
nihiliferi see that there is only one "designate" package06:05
nihilifersudo add-apt-repository cloud-archive:liberty06:05
nihilifersorry, bad link06:05
nihiliferhttps://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/liberty-staging06:05
nihiliferso maybe just check whether this "designate" package has everything you need? (it means - central, api, sink, dns components of designate)06:06
nihiliferif not, then well, we should ping ubuntu packagers then06:07
mdnadeemnihilifer, ok, i will check06:07
*** yuanying_ has joined #kolla06:08
*** yuanying has quit IRC06:11
*** yuanying_ has quit IRC06:13
*** yuanying has joined #kolla06:14
*** yuanying has quit IRC06:15
*** achanda has quit IRC06:19
*** achanda has joined #kolla06:20
*** yuanying has joined #kolla06:28
*** iotapi322 has quit IRC06:29
*** weiyu has joined #kolla06:33
*** sdake has joined #kolla06:36
openstackgerritMerged openstack/kolla: Update kernel version in ubuntu-bootstrap.sh  https://review.openstack.org/27617806:39
*** tfukushi_ has joined #kolla06:42
*** tfukushima has quit IRC06:45
elemoine-hi06:45
*** ssurana has quit IRC06:49
*** weiyu has quit IRC06:49
sdakeyo elemoine-06:49
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Use a new per-service format  https://review.openstack.org/27384306:49
*** ssurana has joined #kolla06:49
sdakemandre your 2nd patch has athe oslo log syslog bug in it for some odd reason06:49
sdakeI think you need to rebase it06:49
sdakei have to ptfo06:50
sdakelater06:50
*** ssurana has quit IRC06:53
*** aginwala has quit IRC06:53
*** weiyu has joined #kolla06:54
*** sdake has quit IRC06:55
openstackgerritMartin AndrĂ© proposed openstack/kolla: Workaround for oslo.log 1.10.0 in centos-binary  https://review.openstack.org/27655506:55
openstackgerritMartin AndrĂ© proposed openstack/kolla: Uninstall conflicting python-six in kolla-ansible  https://review.openstack.org/26886306:55
*** sdake has joined #kolla06:55
*** weiyu has quit IRC06:57
mandresdake, that was my first patch you were looking at06:58
sdakeoslo.log works06:58
sdakethe second patch fails the gate06:58
mandreoslo.log workaround is the second patch, so all good06:58
*** aginwala has joined #kolla06:58
sdakeok so can yu reorder them so the gate passes :)06:58
sdakei guess the gate is busted either way06:59
sdakeok bedtime06:59
mandreyou'll always have the 1rt patch fail the gate no matter what order I put them in06:59
mandrethey both fixes different issues06:59
mandregood night sdake07:00
*** sdake has quit IRC07:15
*** reddy has joined #kolla07:16
*** unicell has joined #kolla07:22
*** weiyu has joined #kolla07:25
*** weiyu has quit IRC07:28
*** weiyu has joined #kolla07:29
*** weiyu has quit IRC07:32
*** weiyu has joined #kolla07:33
*** farid has quit IRC07:35
*** weiyu has quit IRC07:36
*** ppetit has joined #kolla07:37
*** tfukushi_ has quit IRC07:42
*** neilus has joined #kolla07:43
*** tfukushima has joined #kolla07:44
*** weiyu has joined #kolla07:47
*** neilus has quit IRC07:47
*** Jeffrey4l has quit IRC07:48
*** dwalsh has joined #kolla07:51
*** esikachev has joined #kolla07:54
openstackgerritMerged openstack/kolla: Remove unexisting data_files in setup.cfg  https://review.openstack.org/27523807:54
*** weiyu has quit IRC07:55
*** fgimenez has joined #kolla07:58
*** Jeffrey4l has joined #kolla07:58
*** weiyu has joined #kolla07:58
*** tzn has joined #kolla08:01
*** weiyu has quit IRC08:01
*** weiyu has joined #kolla08:03
openstackgerritEvgeny Sikachev proposed openstack/kolla-mesos: Add unit-tests for chronos.py  https://review.openstack.org/27629808:03
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix image plugin functionality for oslo.config  https://review.openstack.org/26821108:06
openstackgerritJeffrey Zhang proposed openstack/kolla: fix the custome profile raise exception issue  https://review.openstack.org/27202208:07
*** weiyu has quit IRC08:08
*** aginwala has quit IRC08:10
*** aginwala has joined #kolla08:12
*** weiyu has joined #kolla08:12
*** akwasnie has joined #kolla08:16
*** weiyu has quit IRC08:16
*** weiyu has joined #kolla08:17
*** weiyu has quit IRC08:21
*** weiyu has joined #kolla08:22
*** weiyu has quit IRC08:22
*** jmccarthy has quit IRC08:27
*** jmccarthy has joined #kolla08:28
openstackgerritMerged openstack/kolla: Ubuntu custom apt repos  https://review.openstack.org/27546308:28
*** athomas has joined #kolla08:37
*** kproskurin has joined #kolla08:42
*** Allen_Gao has quit IRC08:43
*** salv-orlando has joined #kolla08:45
*** mbound has joined #kolla08:47
*** akwasnie has quit IRC08:47
*** Allen_Gao has joined #kolla08:48
*** salv-orlando has quit IRC08:49
openstackgerritAllen Gao proposed openstack/kolla: Fix the ugly base directory path  https://review.openstack.org/27662808:59
ajafoo/09:00
*** salv-orlando has joined #kolla09:00
*** aginwala has quit IRC09:01
*** ppetit has quit IRC09:05
*** akwasnie has joined #kolla09:07
openstackgerritMartin AndrĂ© proposed openstack/kolla: Add a new tool to help find the latest versions  https://review.openstack.org/27663009:08
*** tzn has quit IRC09:10
*** ppetit has joined #kolla09:11
*** openstackgerrit has quit IRC09:17
*** openstackgerrit has joined #kolla09:17
*** dwalsh has quit IRC09:22
*** fgimenez has quit IRC09:23
openstackgerritEvgeny Sikachev proposed openstack/kolla-mesos: Add unit-tests for chronos.py  https://review.openstack.org/27629809:25
*** fgimenez has joined #kolla09:25
*** achanda has quit IRC09:31
*** neilus has joined #kolla09:31
*** tfukushima has quit IRC09:31
*** ppetit_ has joined #kolla09:32
*** tfukushima has joined #kolla09:33
*** ppetit has quit IRC09:33
*** ppetit_ is now known as ppetit09:33
*** tfukushima has quit IRC09:34
openstackgerritJeffrey Zhang proposed openstack/kolla: fix the custome profile raise exception issue  https://review.openstack.org/27202209:34
*** neilus has quit IRC09:36
*** esikachev has quit IRC09:37
*** aginwala has joined #kolla09:41
*** aginwala has quit IRC09:46
*** aginwala has joined #kolla09:47
*** esikachev has joined #kolla09:48
*** aginwala has quit IRC09:52
*** tzn has joined #kolla09:56
*** pbourke has quit IRC10:00
*** tzn has quit IRC10:00
*** pbourke has joined #kolla10:04
*** akwasnie has quit IRC10:16
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect RabbitMQ logs  https://review.openstack.org/27584410:21
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for OpenStack  https://review.openstack.org/27584510:21
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Nova logs  https://review.openstack.org/27584610:21
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for MariaDB  https://review.openstack.org/27584710:21
openstackgerritEric Lemoine proposed openstack/kolla: Add a Dockerfile for Heka  https://review.openstack.org/27584010:21
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka to common role  https://review.openstack.org/27584110:21
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect HAProxy logs  https://review.openstack.org/27584210:21
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Keepalived logs  https://review.openstack.org/27584310:21
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect MariaDB logs  https://review.openstack.org/27584810:21
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder infrastructure  https://review.openstack.org/27666710:21
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for RabbitMQ  https://review.openstack.org/27666810:21
*** tfukushima has joined #kolla10:24
openstackgerritMD NADEEM proposed openstack/kolla: Designate ubuntu binary container  https://review.openstack.org/27667610:31
*** akwasnie has joined #kolla10:31
*** macsz1 has quit IRC10:35
*** macsz has joined #kolla10:35
*** dave-mccowan has joined #kolla10:36
*** salv-orl_ has joined #kolla10:39
*** salv-orlando has quit IRC10:42
*** macsz has quit IRC10:51
*** tzn has joined #kolla10:54
*** macsz has joined #kolla10:56
mdnadeemajafo, Hi10:57
*** dave-mccowan has quit IRC10:58
ajafomdnadeem: hi10:59
mdnadeem>>regarding your comment on https://review.openstack.org/#/c/276676/1/docker/designate/designate-api/Dockerfile.j210:59
*** tzn has quit IRC10:59
mdnadeem>> You ask me to keep 8 space indentation, So should i change indentaion of my code only or keep 8 indentaion through out the file11:00
mdnadeemAs line 8 and 9 also have 4 space indent only11:01
mdnadeemajafo, Please confirm11:01
ajafomdnadeem: 8 yes, and 9 not11:04
ajafomdnadeem: please look as an example here https://github.com/openstack/kolla/blob/master/docker/openstack-base/Dockerfile.j211:04
mdnadeemajafo, ok, got it . Thanks a lot :)11:05
ajafomdnadeem: n/p btw. at this moment I've no possibility to check this patch working :( hope you did it11:07
mdnadeemajafo, yah, i will take care of it.11:09
ajafo:)11:10
ajafothx :)11:10
*** jmccarthy has quit IRC11:12
*** reddy has quit IRC11:12
*** dmsimard has quit IRC11:12
*** ccesario has quit IRC11:12
*** neilus has joined #kolla11:19
openstackgerritMerged openstack/kolla-mesos: Add some unit tests for show_tasks  https://review.openstack.org/27384211:21
*** jmccarthy has joined #kolla11:23
*** neilus has quit IRC11:24
*** ccesario has joined #kolla11:25
*** reddy has joined #kolla11:26
*** dmsimard has joined #kolla11:26
*** gfidente has joined #kolla11:28
*** gfidente has quit IRC11:31
*** ppetit has quit IRC11:35
*** macsz has quit IRC11:38
*** macsz1 has joined #kolla11:38
*** macsz has joined #kolla11:39
*** dims has joined #kolla11:46
*** esikachev has quit IRC11:53
*** tzn has joined #kolla11:56
*** esikachev has joined #kolla11:58
*** salv-orl_ has quit IRC12:01
*** tzn has quit IRC12:01
*** salv-orlando has joined #kolla12:29
*** fgimenez has quit IRC12:29
*** fgimenez has joined #kolla12:31
*** fgimenez has quit IRC12:31
*** fgimenez has joined #kolla12:31
*** alyson_ has quit IRC12:32
*** tfukushima has quit IRC12:36
*** tzn has joined #kolla12:36
*** akwasnie has quit IRC12:40
*** Allen_Gao has quit IRC12:41
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect RabbitMQ logs  https://review.openstack.org/27584412:59
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for OpenStack  https://review.openstack.org/27584512:59
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for RabbitMQ  https://review.openstack.org/27666812:59
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Nova logs  https://review.openstack.org/27584612:59
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for MariaDB  https://review.openstack.org/27584712:59
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect HAProxy and Keepalived logs  https://review.openstack.org/27584212:59
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect MariaDB logs  https://review.openstack.org/27584812:59
*** esikachev has quit IRC13:04
*** neilus has joined #kolla13:08
*** esikachev has joined #kolla13:09
*** Jeffrey4l has quit IRC13:09
*** Jeffrey4l has joined #kolla13:09
*** neilus has quit IRC13:12
*** salv-orlando has quit IRC13:13
*** aginwala has joined #kolla13:18
*** aginwala has quit IRC13:23
*** jtriley has quit IRC13:27
*** jtriley has joined #kolla13:29
*** ppetit has joined #kolla13:31
*** fthiagogv_ has quit IRC13:38
*** fthiagogv has joined #kolla13:39
*** sdake has joined #kolla13:40
*** sdake_ has joined #kolla13:53
*** sdake has quit IRC13:55
*** rmart04 has joined #kolla14:00
*** rmart04 has quit IRC14:01
*** jtriley has quit IRC14:01
*** dave-mccowan has joined #kolla14:06
limamauriciowhy kolla not works with the generic driver cinder?14:08
*** salv-orlando has joined #kolla14:13
*** akwasnie has joined #kolla14:17
sdake_limamauricio  you mean lvm?14:18
sdake_limamauricio because its too hard(tm)14:18
sdake_limamauricio we have pretty much made a decision to abandon iscsi+lvm in favor of ceph14:19
sdake_limamauricio at some point we will tackle plugins for third party hardware gear, the community agrees it needs to happen just a matter of priorities14:19
sdake_wont happen this cycle, not a priority14:20
limamauriciomake sense14:20
sdake_ceph works fantasticially14:20
sdake_although I know your having problems with it14:20
limamauricio:/14:21
limamauricioI have no knowledge of the ceph, I follow all the guides but does not work, at least the aio14:23
limamauricioI have always the same error, [ceph | Bootstrapping Ceph OSDs]14:23
limamauricioand the logs show me a timeout error. :/14:24
limamauricioI have to disable the firewalld?14:24
sdake_oh ya14:24
sdake_your firewall is on?14:24
sdake_openstack requires iptables not firewalld14:25
limamauriciook14:25
limamauricio:)14:25
limamauricioError connecting to cluster: TimedOut14:25
limamauricioi have always this error14:26
limamauricioError connecting to cluster: TimedOut14:26
limamauricio:/14:26
ccesariolimamauricio, are you from Brazil!? O_o14:26
*** fgimenez has quit IRC14:29
*** salv-orlando has quit IRC14:30
*** fgimenez has joined #kolla14:31
*** fgimenez has quit IRC14:31
*** fgimenez has joined #kolla14:31
sdake_pbourke around14:35
pbourkesdake_: yup14:35
sdake_need two reviews please14:36
sdake_stable is broken14:36
sdake_https://review.openstack.org/#/c/268863/14:36
sdake_the gate is expected to fail on one of those14:36
sdake_because two things are broken14:36
sdake_the last patch fixes the gate14:36
sdake_assuming i am reading gerrit correctly14:36
sdake_pbourke can you confirm on that point?14:37
sdake_basically oslo..log fails the first patch ( the bottom of the list)14:37
sdake_becuase it hasn't been patched yet14:37
*** inc0 has joined #kolla14:37
inc0good morning14:37
sdake_inc0 need revies14:37
sdake_https://review.openstack.org/#/c/268863/14:37
sdake_stable is now working witht hese patches14:37
inc0wow that was fast14:38
inc0ok, I couldn't reproduce bug you mentioned yesterday14:38
sdake_the gate is expected to be broken on one of those patches14:38
sdake_inc0 its centos binary only14:38
sdake_you were running non on centos binary i assume14:38
inc0I see, no I run ubuntu-source14:38
inc0normally14:38
inc0yeah that makes sense14:39
inc0the review14:39
*** aginwala has joined #kolla14:39
inc0I think venvs will prevent such things from happening in the future14:40
sdake_venv is not in stable14:40
sdake_but i agree14:40
inc0yeah I know14:40
sdake_also we are not backportign venv to stable ;)14:40
inc0in the future;)14:40
inc0sure, however what we should backport is version pin to everything14:41
inc0write and backport14:41
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect RabbitMQ logs  https://review.openstack.org/27584414:41
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for OpenStack  https://review.openstack.org/27584514:41
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for RabbitMQ  https://review.openstack.org/27666814:41
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Nova logs  https://review.openstack.org/27584614:41
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for MariaDB  https://review.openstack.org/27584714:41
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka to common role  https://review.openstack.org/27584114:41
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect HAProxy and Keepalived logs  https://review.openstack.org/27584214:41
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect MariaDB logs  https://review.openstack.org/27584814:41
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Glance logs  https://review.openstack.org/27676014:41
* elemoine- is sorry for the patches wave14:41
pbourkeinc0: have you come across an issue where kolla-ansible wont upgrade14:42
pbourkeinc0: I think its due to the common_run attrs that are set around it14:43
openstackgerritMerged openstack/kolla: Uninstall conflicting python-six in kolla-ansible  https://review.openstack.org/26886314:43
ajafohave you notice some problems with docker 1.10?14:44
inc0pbourke, not yet, it just rebuilds for me14:44
openstackgerritMerged openstack/kolla: Workaround for oslo.log 1.10.0 in centos-binary  https://review.openstack.org/27655514:44
pbourkeinc0: hmm ok, thanks14:45
*** aginwala has quit IRC14:45
inc0pbourke, you're reffering to docker upgrade or service upgrade?14:46
pbourkeinc0: service upgrade. I think I see the problem. basically if you do a liberty deploy, then try to upgrade a single service (-e action=upgrade), it wont run the common role so no new kolla_ansible gets deployed14:46
pbourkeinc0: which is a problem if the service you're upgrading needs a new kolla_ansible14:47
inc0well we didn't write kolla_ansible upgrade yet14:47
inc0we should14:47
pbourkeinc0: also it seems swift also recommend sighup when upgrading :)14:47
pbourkefor graceful shutdown14:48
pbourkeyou would think there'd be a way of telling docker to send that signal though (other than exec)14:48
inc0afair there was issue open for that14:49
sdake_guys i'll be out for 2-3 hours this morning, then put together the agenda14:49
sdake_that should give people a chance to erview it14:49
sdake_gotta go right now14:49
*** sdake_ has quit IRC14:49
inc0cyas14:49
*** ftcpops has joined #kolla14:50
pbourkeinc0: actually question on that area. in your upgrade.yml you run docker start tasks on each nova container before reload.yml at the end14:52
pbourkeinc0: but wont the start tasks stop/start the containers?14:52
inc0yeah14:52
pbourkeinc0: making the reload pointless14:53
inc0it will14:53
inc0it does not14:53
inc0so I run controller nodes upgrade first14:53
inc0so you will have new containers in controller14:53
inc0then run compute one at the time14:53
inc0so potentially there will be hundreds of them14:53
inc0and it will take time14:53
inc0for careful people, LMing VMs out of compute just in case, weeks even14:54
inc0upgrade_level=auto binds nova communication to oldest version it knows in cluster14:54
pbourkeright but hold on14:54
inc0so if you run them initially, that still be Liberty14:54
inc0we need to tell it "check again, now it's all Mitaka"14:54
pbourkeok maybe you're reloading for a different reason that I need to14:55
pbourkefor swift its to allow them to finish up all communication before stopping14:55
inc0that's only reason I'm reloading:)14:55
pbourkemaybe you should do the same with nova?14:56
inc0no, in nova I don't really care about graceful shutdown14:56
inc0most of operations are atomic14:56
pbourkewhy not14:56
pbourkeok14:56
*** akwasnie has quit IRC14:56
inc0there are certain cases, like Live Migration14:56
*** neilus has joined #kolla14:56
inc0but well, it's ongonig problem for nova anyway14:56
pbourkewhat if say a request to boot has made it through the api and then the scheduler gets upgraded during that request14:56
inc0however that might be good thing to do later on14:57
inc0pbourke, if api finished to post it to rabbit, msg will stay in rabbit14:57
inc0and another scheduler will take it on14:57
pbourkeok sounds good14:57
inc0if you happen to restart in EXACT moment this is processed, request might be lost14:57
inc0you can fiddle around haproxy to prevent that14:58
inc0for scheduler, it might be lost14:58
inc0but yeah, room for improvement there14:58
*** jtriley has joined #kolla14:58
pbourkeim assuming docker stop sends sigkill?14:58
pbourkewithout googling14:58
inc0yeah14:59
inc0but you can specify this in config14:59
inc0so we can send sigterm on docker rm14:59
SamYaplemorning14:59
SamYaplehey inc0 pbourke14:59
inc0hey SamYaple14:59
pbourkeSamYaple: morning14:59
inc0room for improvement;)14:59
SamYapleinc0: would you do the +W honors on the kolla-toolbox rename since you are the only one that has voiced any concern at all?15:00
*** neilus has quit IRC15:01
inc0SamYaple, ok15:01
inc0there you go man15:01
*** jtriley has quit IRC15:03
*** aginwala has joined #kolla15:04
openstackgerritMerged openstack/kolla: Rename kolla_ansible to kolla_toolbox  https://review.openstack.org/27415415:04
openstackgerritSam Yaple proposed openstack/kolla-mesos: Convert kolla_ansible container to kolla_toolbox  https://review.openstack.org/27677215:05
SamYapleinc0: thanks!15:05
SamYaplehey everyone, just FYI15:06
SamYaplegates are breaking because docker 1.10 was released15:06
SamYapleim working on a patch to fix this15:06
SamYapleplease don't recheck15:06
SamYapleits pointless15:06
elemoine-SamYaple, thanks15:06
*** athomas has quit IRC15:07
elemoine-I'm seeing errors when deploying Rsyslog15:07
SamYaplemandre inc0 pbourke Jeffrey4l ^^15:07
pbourkeSamYaple: thanks15:07
Jeffrey4lroger that. thanks15:07
*** aginwala has quit IRC15:09
*** akwasnie has joined #kolla15:11
*** aginwala has joined #kolla15:16
*** jtriley has joined #kolla15:17
*** dave-mccowan has quit IRC15:17
*** jpeeler has quit IRC15:20
*** blahRus has joined #kolla15:21
*** esikachev has quit IRC15:26
*** aginwala has quit IRC15:27
ajafoelemoine-: it's docker 1.10 problem15:28
ajafoprobably15:28
ajafoi've got the same15:28
elemoine-yeah15:28
elemoine-so let's way for SamYaple's patch15:28
elemoine-s/way/wait15:28
ajafoelemoine-: sthg like this http://paste.openstack.org/show/486098/ ?15:29
elemoine-ajafo, yep exactly that15:29
*** dave-mccowan has joined #kolla15:32
ajafohmm sudo yum downgrade docker-engine-1.9.1 docker-engine-selinux-1.9.1   and service restart didn't help15:34
openstackgerritSam Yaple proposed openstack/kolla: Convert Neutron to thin containers  https://review.openstack.org/27271515:35
openstackgerritSam Yaple proposed openstack/kolla: Correct issue with virtualenv rootwrap  https://review.openstack.org/27550415:35
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Fix gate for docker 1.10  https://review.openstack.org/27678815:35
SamYapleajafo: check your docker-py version15:35
SamYapleajafo: if you have docker-py 1.7.0 is probably causing your issue15:36
*** jtriley has quit IRC15:39
*** jtriley has joined #kolla15:42
*** reddy has quit IRC15:44
sean-k-mooneyo/15:45
sean-k-mooneySamYaple: i was wondering if i could ask you a question about how kolla currently deploy neutron with ansible somewhat related to the neutron thin contianers15:46
*** kproskurin has quit IRC15:46
elemoine-inc0, I just sent a "make central logging proposal" on the mailing list, to follow up on our discussion15:47
inc0ok, thanks, will look it up later today15:48
inc0meetings meetings :(15:48
sean-k-mooneySamYaple: currently kolla starts the network backed as part of configuring and starting the neutron agents. chould we seperate the two?15:48
elemoine-inc0, it's WE here so no rush :)15:48
inc0thanks for all the cool stuff elemoine- !15:48
elemoine-inc0, no worries, I enjoy working in Kolla15:49
inc0sean-k-mooney, with docker 1.10 we potentially could15:49
ajafoSamYaple: thx, you're right I had docker 1.7.015:49
inc0before that we had to have all the neutron agents in same container as namespave15:50
inc0as for ovs, well, I see no reason why not15:50
sean-k-mooneyinc0: so looking at https://review.openstack.org/#/c/272715/10/ansible/roles/neutron/tasks/start.yml i just find all the openvswitch vs linux bridge branching messey15:51
SamYaplesean-k-mooney: can you expand on what your request is?15:52
sean-k-mooneyit would be nice if we could first start and configure the network backend(ovs/linux bridge) then handel deploying the neutron agents independently15:52
SamYapledefine independantly?15:52
SamYapleand the agents have to start first15:52
SamYapleneutron-server needs to start first*15:53
inc0SamYaple, why neutron server has to start before ovs?15:53
inc0not agents, ovs itself15:53
sean-k-mooneySamYaple: i was thinking of a simple refactoring to have a start ovs and start linux bridge that would be called first the the start.yml would start the agents/server15:54
SamYapleinc0: ovs _itself_ can start whenever15:54
inc0I think that's the point here15:54
SamYaplebut there is no linux-bridge continer like there is an ovs one15:54
ajafodowngrade docker-py to 1.6 didn't help in my case :/ now i've docker 1.9.1 and docker-py 1.6.0 and still have the same issue :/15:54
SamYapleso im not sure what "start linux bridge" means there15:55
SamYapleajafo: the issue is with mounts15:55
inc0I actually think that would be cleaner to separate these two15:55
SamYapleajafo: check `mount` and unmount the docker container mounts15:55
SamYapleinc0: define seperate, thats been my question15:55
sean-k-mooneyon sec ill share a link to the linux bridge peice15:55
SamYapleas long as everything starts and we don't duplicate _any_ tasks, then i dont care if they are in different playbooks15:56
ajafoSamYaple: thx for hints15:56
inc0SamYaple, for cleanliness purpose, at separate file in tasks15:56
SamYapleajafo: stop the daemon first though15:56
SamYapleajafo: there were some changes around all this stuff and im still working through it as well15:56
SamYapleinc0: in fact we are going to have to break them into different playbooks anyway so as not to dup them for upgrades15:57
SamYaplethe tasks I mean15:57
sean-k-mooneyso basicaly i am just suggesting have a #include openvswitch_start.yml when neutron_plugin_agent == "openvswitch"15:57
sean-k-mooneyand remove all the direct check for openvswitch in the start.yml15:57
inc0yeah I'm ok with this, that's good change15:58
*** aginwala has joined #kolla15:58
SamYapleyea as long as funcationality doesnt change we can do includes all day long15:58
ajafoSamYaple: yes I remember to stop deamon15:58
SamYapleif its easier to read15:58
SamYaplebut sean-k-mooney dont worry about that now15:58
SamYapleive got to break all that stuff up15:58
SamYaplefor the neutron upgrades15:58
SamYaplewhen the dust settles see if you still need a change15:59
sean-k-mooneyok i might submits a patch ontop of your neutron slim containers patch to do that then. i wont change any of the task just move them so we dont have so many neutron_plugin_agent == x checks15:59
inc0afk for now15:59
SamYaplesean-k-mooney: well have conflicting patches then, because ill have upgrades too15:59
*** logan_ has joined #kolla16:00
*** logan_ is now known as Guest7865416:01
*** logan- has quit IRC16:01
*** Guest78654 is now known as logan-16:02
*** aginwala has quit IRC16:03
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect RabbitMQ logs  https://review.openstack.org/27584416:06
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for OpenStack  https://review.openstack.org/27584516:06
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for RabbitMQ  https://review.openstack.org/27666816:06
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Nova logs  https://review.openstack.org/27584616:06
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for MariaDB  https://review.openstack.org/27584716:06
openstackgerritEric Lemoine proposed openstack/kolla: Add a Dockerfile for Heka  https://review.openstack.org/27584016:06
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka to common role  https://review.openstack.org/27584116:06
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect HAProxy and Keepalived logs  https://review.openstack.org/27584216:06
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder infrastructure  https://review.openstack.org/27666716:06
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Glance logs  https://review.openstack.org/27676016:06
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect MariaDB logs  https://review.openstack.org/27584816:06
*** sean-k-m1 has joined #kolla16:07
sean-k-m1sorry i had a disconnect so i was missing lines. sure ill wait for your upgrade stuff to land and look at it later if it still need to be done16:13
*** sean-k-mooney has quit IRC16:14
*** sean-k-m1 has quit IRC16:14
SamYaplesean-k-m1: i think what you are asking for may just happen just because, unrelated to why you are asking for it16:14
*** Jeffrey4l has quit IRC16:15
SamYaplesince we have to start sharing the tasks between deploy and upgrade, they just get split up anyway16:15
*** sean-k-mooney has joined #kolla16:15
*** jpeeler has joined #kolla16:15
*** jpeeler has joined #kolla16:15
*** jtriley has quit IRC16:16
sean-k-mooneySamYaple: ya it likely will be covered by the deploy/upgrade refactoring. cool it just jumped out at me as something that would simply the start.yml when i was looking at your review16:18
*** esikachev has joined #kolla16:19
*** jtriley has joined #kolla16:28
*** esikachev has quit IRC16:30
*** esikachev has joined #kolla16:32
*** shardy has joined #kolla16:43
*** athomas has joined #kolla16:43
*** neilus has joined #kolla16:45
*** neilus has quit IRC16:49
*** esikachev has quit IRC16:49
*** esikachev has joined #kolla16:52
*** jtriley has quit IRC16:53
*** esikachev has quit IRC16:58
*** macsz has quit IRC17:01
*** aginwala has joined #kolla17:01
*** jtriley has joined #kolla17:01
*** aginwala has quit IRC17:02
*** akwasnie has quit IRC17:02
*** ppetit has quit IRC17:06
*** achanda has joined #kolla17:15
*** fgimenez has quit IRC17:19
*** farid has joined #kolla17:20
openstackgerritSam Yaple proposed openstack/kolla: Fix two missing toolbox references  https://review.openstack.org/27684117:25
SamYapleinc0: pbourke https://review.openstack.org/27684117:25
SamYapleout-of-date patch merged17:25
SamYapleour gates need to be voting :(17:25
*** ftcpops has quit IRC17:27
*** unicell has quit IRC17:37
SamYaplenihilifer: https://review.openstack.org/#/c/276772/17:39
*** ssurana has joined #kolla17:45
openstackgerritSam Yaple proposed openstack/kolla: Make a final check to validate mysql works  https://review.openstack.org/27685117:49
SamYaplegit branch17:49
openstackgerritMerged openstack/kolla: Fix the ugly base directory path  https://review.openstack.org/27662817:52
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Fix gate for docker 1.10  https://review.openstack.org/27678817:53
openstackgerritSam Yaple proposed openstack/kolla: Convert Neutron to thin containers  https://review.openstack.org/27271517:53
openstackgerritSam Yaple proposed openstack/kolla: Correct issue with virtualenv rootwrap  https://review.openstack.org/27550417:53
openstackgerritMerged openstack/kolla: Fix manila perms  https://review.openstack.org/27535917:53
*** salv-orlando has joined #kolla17:55
*** sdake has joined #kolla17:56
*** esikachev has joined #kolla17:57
openstackgerritMerged openstack/kolla-mesos: Convert kolla_ansible container to kolla_toolbox  https://review.openstack.org/27677217:58
*** sdake_ has joined #kolla17:59
openstackgerritPaul Bourke proposed openstack/kolla: Upgrade tasks for Swift  https://review.openstack.org/27685517:59
openstackgerritPaul Bourke proposed openstack/kolla: Upgrade tasks for Swift  https://review.openstack.org/27685518:00
*** sdake has quit IRC18:01
nihiliferSamYaple: can you look at this named volumes workaround? https://review.openstack.org/#/c/27523718:02
nihiliferit's dirty as hell and only for AIO18:03
*** esikachev has quit IRC18:03
nihiliferbut well, i'd like to wait for mesos to deal with named volumes18:03
nihiliferbefore trying to do such a workaroud for multinode, which would be even more ugly18:03
*** achanda has quit IRC18:06
*** pbourke has quit IRC18:06
nihiliferoh, nvm, i have over 9k merge conflicts now18:09
*** unicell has joined #kolla18:10
limamauricioSamYaple i have to create two partitions sdb1 and sdb2 to deploy ceph?18:10
*** athomas has quit IRC18:11
limamauriciodocker logs bootstrap_osd_0 -> http://pastebin.com/2fMqiKqn18:12
*** blahRus has quit IRC18:12
*** achanda has joined #kolla18:18
*** dims is now known as dimsum__18:30
*** neilus has joined #kolla18:33
*** sdake_ has quit IRC18:36
*** esikachev has joined #kolla18:36
*** neilus has quit IRC18:38
*** mbound has quit IRC18:38
*** sdake has joined #kolla18:41
*** aginwala has joined #kolla18:46
*** ayoung has joined #kolla18:49
ayoungCan anyone give me the simplest way to deploy just Keystone and MySQL on a local host using Kolla?18:50
*** esikachev has quit IRC18:50
ayoungI'm sure it is trivial, but I'm having trouble knowing where to start.  Do I need an inventory file>18:51
ayoungand then use Ansible?18:51
inc0well18:51
inc0you always need inventory18:52
inc0it tells ansible where to deploy18:52
inc0bot if you want to do local deploy, there is predefined inventory for that18:52
inc0aio18:52
inc0ayoung, so https://github.com/openstack/kolla/blob/master/doc/quickstart.rst this is quickstart guide18:53
SamYaplenihilifer: ok ill look18:53
ayounginc0, OK..I've been reading that.  Still going through all the "here;'s how you fetch docker" type lines18:54
SamYaplelimamauricio: no, check the guide18:54
inc0to deploy just keystone you can run tools/kolla-ansible deploy -t keystone18:54
*** esikachev has joined #kolla18:54
SamYaplelimamauricio: you should onlt be creating one partition named KOLLA_BOOSTRAP_OSD18:54
ayounginc0, let me try that...18:55
inc0ayoung, so tell me what do you have now and what is problem you're having18:55
inc0also18:55
inc0correct18:55
inc0correction18:55
ayounginc0, ok so I need an inventory, ...and I guess if I am going to do this on my local machine, I need sudo set up to not require a password from whatever user I connect to...18:57
ayounginc0, I'm on F23, my laptop, an I'd like to not open and glaring security holes if I can help it18:58
ayoungI'm OK with having to type my sudo password to run this...prefer it to disabling passwords, or making an key'ed log in to a trusted user.18:58
*** shardy has quit IRC18:59
SamYapleayoung: F23 wont work18:59
ayoungSamYaple, I only want Keystone18:59
ayoungSamYaple, I don't care about Nova for this18:59
ayoungI might pull in glance to have a remote app, but even that is optional19:00
SamYapleayoung: oh hey you are ayoung!19:00
SamYaplei know you19:00
ayoungYes you do19:00
SamYaplegive me a minute ill see what you need19:00
limamauricioKOLLA_CEPH_OSD_BOOTSTRAP or KOLLA_BOOSTRAP_OSD?19:00
SamYaplelimamauricio: the first one, apologies19:00
SamYaplethe guide does work, i promise19:01
limamauriciook19:01
*** aginwala has quit IRC19:01
limamauricio:)19:01
SamYapleso ayoung just keystone not mariadb or any other services?19:02
ayoungSamYaple, I need maria too19:04
ayoungSamYaple, I'm trying to mimic a live setup for Functional testing, but also want to, in the future, replace the Keystone code with -v  mounted file from the underlying FS (my git repo)19:05
SamYapleayoung: cool. ill patsebin the config you need in globals.yml for just keystone and mariadb19:05
ayoungthanks19:05
SamYapleayoung: http://paste.openstack.org/show/486128/19:12
ayoungSamYaple, anyway I can put them somewhere other than /etc19:13
ayoungI want to make this a developer friendly approach, very little Sudo/Root magic other than launching things via docker19:13
SamYapleayoung: no thats hardcoded in kolla-ansible script. youll ahve to change ./tools/kolla-ansible19:13
SamYaplebut if you pop open the bash script it should be straight forward19:14
SamYaplehowever, you wil have to adjsut other values to not use /etc19:14
SamYaplelet me look those up for you19:14
SamYaplefyi ayoung you do need to build the following images (you can use this command) `./tools/build.py keystone mariadb rsyslog kolla-toolbox`19:15
SamYaplethose are all you will need (saves alot of build time)19:15
*** esikachev has quit IRC19:16
*** iotapi322 has joined #kolla19:16
iotapi322Hi after connecting last night trying to get a build, sdrake suggested that I try with the stable/liberty release, however I can't seem to find the command "kolla-build" in this particular stable branch.19:17
iotapi322what command should i run to build the containers?19:17
inc0iotapi322, tools/build.py19:18
inc0standalone script with same command options as kolla-build19:18
ayoungSamYaple, is this tox/venv based ?19:18
inc0actually kolla-build calls it underneath;)19:19
iotapi322so for an all in one in the /etc/kolla/globals file do you suggest the openstack_release to be latest or to be liberty?19:19
SamYapleayoung: you can run it in a venv, there is no tox command to build _just_ what you have asked for though19:20
ayoungSamYaple, Nah, I meant for building the venv in the first place19:20
ayoungSamYaple, tox -r -e py34  ?19:20
ayoungor py27 ?19:21
SamYapleshoudl work for either 27 or 3419:21
inc0iotapi322, so that's probably something we have to change as it's misleading. openstack_release in global.yml is tag which you build your images with19:21
SamYaplethough 27 is the most tested19:21
ayoungthats what I was asking thanks19:21
inc0or rather what image tag it will look for19:21
inc0by default both are 2.019:21
inc0in master19:21
inc0in liberty/stable I think its still latest19:22
inc0defaults should work in any case19:22
SamYapleshoot ayoung. testing this outside of /etc/ has reileved a problem. ill have a patch up momentarily (blame inc0! .... and those that reviewed his patch)19:22
ayoungINFO:__main__:Found the docker image folder at /opt/stack/kolla/docker19:22
ayoungTraceback (most recent call last):19:22
SamYaplerevieled*19:22
ayoungrevealed?19:23
SamYaplethats the one19:23
ayoungk19:23
iotapi322Ah i understand.19:23
inc0SamYaple, what? now hold on, I have my rights19:23
ayoungso long as you were not reviling inc0 cuz he's on my good list19:23
SamYapleinc0: :P its old code19:24
SamYapleinc0: just a missed thing from syslog patch19:24
*** salv-orlando has quit IRC19:24
inc0duh19:24
inc0it's not my fault, I did most of stuff;)19:25
inc0well anyway, add patch I want to see what I'm blamed for19:25
inc0my name needs cleaning now19:25
*** mimizone has quit IRC19:26
iotapi322building these docker containers and dealing with the timeouts is terrible.  Does everyone just run their own docker repositories?  Or do you guys use proxy's on your local networks or what?19:29
*** mimizone has joined #kolla19:29
*** mimizone has quit IRC19:30
*** mimizone has joined #kolla19:31
inc0for proxy we have -i and -I options in build19:31
inc0you can inject proxy setting code19:31
inc0I build all images locally anyway19:32
iotapi322Yeah i'm trying to setup a squid proxy here at my local network for this.  i keep blowing up on :19:34
iotapi322Cannot open: http://repo.percona.com/release/7/RPMS/x86_64/percona-release-0.1-3.noarch.rpm. Skipping. INFO:__main__:base:Error: Nothing to do INFO:__main__:base:19:34
ayoungSamYaple, is kolla making assumptions about access control to DOcker?19:35
ayoungdocker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', error(13, 'Permission denied'))19:35
ayoung(py27)[ayoung@ayoung5419:35
*** aginwala has joined #kolla19:36
*** limamauricio has quit IRC19:36
*** aginwala has quit IRC19:36
iotapi322ayoung - i had to have the python-docker tools installed19:37
*** mbound has joined #kolla19:38
ayoungiotapi322, on F23?19:39
SamYapleayoung: youll have to forgive the slowness, im working at beer capacity19:40
ayoungSamYaple, heh19:40
ayoungBrno?19:40
SamYapleayoung: kolla is just assuming it can connect to the docker socket19:40
ayoungSamYaple, I figured it out:19:41
ayounghttp://askubuntu.com/questions/477551/how-can-i-use-docker-without-sudo19:41
ayoungprobably a security hole, but I can live with it for now19:41
SamYapleyep thats the one19:41
SamYapleits a docker security hole to be honest19:41
SamYaplebut it can be rememdedy with tls conenction19:41
ayoungdocker is a security hole19:41
ayoungno iot can't19:41
ayoungit19:41
SamYapleyea but docker != security19:41
ayoungyep19:42
SamYaplei mean tcp+tls19:42
SamYaplewhy can it not?19:42
ayoungif I really, really, reaaly cared what I would do is:19:42
ayoungcreatea develop user19:42
ayoungsu - developer19:42
ayoungand do everything as that user19:42
openstackgerritSam Yaple proposed openstack/kolla: Fix non-root deploys  https://review.openstack.org/27688719:42
ayoungnot as ayoung19:42
SamYapleayoung: ^^ that _should_ do it, but im testing now19:42
SamYapleadded an option to kolla-ansible script too19:42
ayoungSamYaple, cool...I'm still a couple steps behind, but I;'ll catch up19:43
*** mbound has quit IRC19:43
SamYapleayoung: youve come at a bad time too, docker 1.10 was released yesterday and our gate is AFU19:43
SamYapleprobably will hit you too, still working through the issue19:43
SamYaplethis should all work with docker 1.9 and docker-py 1.6.019:43
SamYapleif you run into any issues that is19:44
ayoungSamYaple, There is no such thing as a good time.  Good time means everything is stable, which means everyone has moved on to new projects and your code is legacy19:44
SamYapletis true19:45
ayoungSamYaple, so, here is the plan19:45
ayoungget this to work with the code in the built docker images19:45
ayoungthen createa new keystone image where I mount /opt/stack  via -v19:45
ayoungthen, inside the keystone container, to a python setup.py of the keystone code so that it points at the /opt/stack/keystone repo19:46
ayoungand run Keystone that way19:46
ayounguse that to check commits in my laptop git repo19:46
ayoungthink it will work?19:46
inc0bbl19:47
*** inc0 has quit IRC19:47
SamYapleayoung: i test patched the way i _think_ you are describing, yes19:47
SamYaplebut one of these ways it should work19:47
ayoungSamYaple, good to know.19:47
ayoungI don't want to have to do a complete rebuild each time I want to test a change from Keystone if I can help it19:47
SamYapleyea that model doesnt work for devel19:48
ayoungSamYaple, and devstack is way too invasive19:48
SamYapleyou can rerun setup.py from in the container, and you can bindmount from outside the container... so this _should_ all work19:48
SamYaplebut if it doesnt work the way you want you I gaurantee we are a patchset away from making it do so19:48
ayoungSamYaple, what version of the keystone code is going to be running when I do the ./tools/.... build you linked above?19:49
SamYapleayoung: it pulls master tarball from tarballs.openstack.org19:49
SamYapleyou can configure that to point to a git repo19:49
SamYapleand i _think_ we have local as well19:49
ayoungSo right now it will be Keystone Mitaka RC 1?19:49
ayoungEr Milestone 119:49
SamYaplewhatever master tarball is19:49
SamYapleayoung: http://tarballs.openstack.org/keystone/keystone-master.tar.gz19:50
ayoungkeystone-master.tar.gz05-Feb-2016 09:09 1.0M19:50
SamYaplebut again, this is configurable19:50
ayoungIts OK, I just want to know the delta19:50
ayoungthat should be close enough to a git checkout that I won't be jumping major versions19:50
*** aginwala has joined #kolla19:51
ayoungSamYaple, OK...I have a completed build...19:52
SamYapleayoung: im still testing my patch for non-root deploy19:52
ayoungSamYaple, I'll test with what you pasted abovce, and then try your patch after19:52
ayoungminimize the number of moving pieces.19:53
SamYapleayoung: awesome. get it working root first, then non-root will make more sense19:53
ayoungSamYaple, Ok you have kolla_base_distro: "ubuntu"19:53
ayoungcan I make that Fedora?19:53
SamYapleyou can make it centos19:53
SamYapleim not sure the fedora images build19:53
SamYaplecentos we gate against19:53
SamYapleas long as you dont run nova, youll be fine19:53
*** sdake has quit IRC19:54
ayoungOK.  Let me try with Centos First...19:55
ayoungI'll try Fedora later19:55
SamYaplebtw ayoung i did a full keystone v3 kolla build in liberty based on one of you blog posts. worked well with kolla. hoping to have a v3 only option for kolla before mitaka drops19:55
SamYaplei say worked well... the service that worked with v3 worked well ;)19:55
ayoungSamYaple, that makes me so happy19:55
*** Marga_ has quit IRC20:04
ayoungSamYaple, so it looks like it is insisting on at least an empty dict in /etc/kolla/passwords20:13
ayoungand /etc/kolla/rsyslog needs to be writible, I think20:14
*** dave-mccowan has quit IRC20:14
openstackgerritMerged openstack/kolla: Fix two missing toolbox references  https://review.openstack.org/27684120:14
ayoungah,  needs to exist and be a directory and writable20:15
SamYapleayoung: ive just done a full deploy from an unprivileged user20:17
SamYapleso should be good with my patch. let me post some steps20:18
SamYapleayoung: http://paste.openstack.org/show/486143/20:20
SamYaple/home/sam/kolla_config contains globals.yml and passwords.yml only20:20
SamYaple(the deploy populates it with more files of course)20:20
*** neilus has joined #kolla20:21
ayoungSamYaple, next up was "ERROR! ERROR! 'docker_registry_password' is undefined"20:21
SamYapleayoung: just copy the passwords.yml from the repo20:22
SamYaplethose variables need to be set20:22
SamYapleits a jinja2/ansible thing20:22
*** sdake has joined #kolla20:23
SamYapleyoull only need to tweak the mariabd and keystone password variable20:23
ayoungSamYaple, makes sense.  What do you mean by tweak?20:23
SamYapledefault password is "password"20:24
SamYapleif you want it to be different, change the value20:24
SamYapleyou dont have to change the value20:24
*** salv-orlando has joined #kolla20:24
SamYapleif you dont adjust the variables in passwords.yml youll end up with a database with root:password with keystone user keystone:password20:25
ayoungSamYaple, so, I ran this without a password.yml file, and then again with, and it errored on rsyslog20:25
ayoungAn exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 020:25
ayoungfatal: [localhost]: FAILED! => {"changed": false, "failed": true, "parsed": false}20:25
ayoungtask was20:25
ayoungTASK [common : Starting rsyslog container]20:25
SamYaplewhat version of docker and docker-py are you running?20:25
*** neilus has quit IRC20:26
ayoungdocker-1.9.1-6.git6ec29ef.fc23.x86_6420:36
ayoungdocker-py==1.7.020:36
SamYapletry 1.6.0 for docker-py20:36
ayoungSamYaple, that is in venv20:36
ayounghmmm20:36
*** dave-mccowan has joined #kolla20:37
ayoungok setting it to =1.6...20:37
SamYapleso we have an ansible module, kolla_docker. when i wrote it I did not implement the "command" part of the api and I *think* thats causing our troubles20:37
SamYapleim still investigating20:37
ayounghmmm20:38
ayoung<1.7?20:38
SamYapleyea20:38
ayoungNope20:38
ayoungdoesn't like that20:38
SamYaplehow so?20:38
SamYaplelike you cant install it?20:38
ayoungdisregard.  install ran20:39
ayoungok testing again20:39
openstackgerritSam Yaple proposed openstack/kolla: Fix non-root deploys  https://review.openstack.org/27688720:40
SamYapledocker-engine 1.9.1 + docker-py 1.6.0 works for me (and its worked for master for a month or so)20:41
*** inc0 has joined #kolla20:43
ayoungSamYaple, nope, no difference20:44
ayoungTASK [common : Starting rsyslog container] *************************************20:44
ayoungAn exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 020:44
ayoungfatal: [localhost]: FAILED! => {"changed": false, "failed": true, "parsed": false}20:44
ayounglemme try with -vvvv20:45
SamYaple... KeyError....20:45
SamYapleis this master branch?20:45
*** achanda has quit IRC20:45
dmsimardSamYaple: last time I saw something really weird you happened to have seen it too20:51
dmsimardHave you come across this on master? :P20:51
dmsimardhttp://logs.openstack.org/92/276492/6/check/gate-puppet-openstack-integration-scenario001-tempest-dsvm-centos7/78b9c32/logs/nova/nova-compute.txt.gz#_2016-02-05_20_21_53_98320:51
dmsimardtempest failing on [most|all] VM related tests20:52
SamYapledmsimard: policy.json related methings20:52
*** aginwala has quit IRC20:53
ayoungSamYaple, so, yes, this is master, and I'm looking at the code now20:58
SamYapleayoung: im in docker-dev, they are confused right now as well20:58
SamYapleworking on it20:58
ayoungSamYaple, http://git.openstack.org/cgit/openstack/kolla/tree/ansible/library/kolla_docker.py#n56420:59
ayoungIts the return from20:59
ayounghttp://git.openstack.org/cgit/openstack/kolla/tree/ansible/library/kolla_docker.py#n55320:59
*** achanda has joined #kolla20:59
SamYapleayoung: are you ansible 2.0?21:00
ayoungSamYaple, nope 1.921:01
SamYaple1.9.4?21:01
ayoungI lied21:01
ayoungansible --version21:01
ayoungansible 2.0.0.221:01
SamYapleyup21:01
SamYaplethatll do it21:01
SamYaplewe havent converted to 2.0 yet21:01
ayoungrpmquery -f `which ansible`21:01
ayoungansible-2.0.0.2-1.fc23.noarch21:01
ayoungGah...21:01
SamYaplealways another thing21:02
ayounglet me see if I can downgrade21:02
SamYaplebut good to know that error21:02
ayoungI don't need 2 yuet21:02
ayoungyet21:02
SamYapleill patch it with a better error message21:02
SamYapleso reqs shoudl be docker-engine==1.9.1 docker-py==1.6.0 ansible==1.9.421:02
ayoungdnf downgrade ftw!21:03
SamYapleit is _unlikely_ that we will switch to ansible 2.0 for mitaka21:03
SamYaplepossibly, well be talking about it at midcycle21:03
SamYaplebut if not mitaka, first thing in newton cycle21:03
ayoungNah, too soon21:03
SamYapleunfortuantely we have to21:03
SamYapleansible bugs they wont fix :/21:03
ayoungYeah, Newton is right.  I thought I was still on 1.921:03
ayoungcoulda swore I checked that21:03
ayoungAh..that was docker, not Ansible ...silly me21:04
SamYapleso many numbers21:04
*** esikachev has joined #kolla21:05
*** aginwala has joined #kolla21:06
*** achanda has quit IRC21:06
ayoungSamYaple, TASK: [common | Starting rsyslog container] ***********************************21:09
ayoungfailed: [localhost] => {"failed": true}21:09
ayoungmsg: The requested image does not exist: kollaglue/centos-source-rsyslog:2.0.021:09
ayounghmmm21:09
ayoungkollaglue/centos-binary-rsyslog          2.0.0               6b727d9e6e42        5 minutes ago       339 MB21:09
ayoungSamYaple, what's the difference?  whether it was built as part of the21:10
SamYapleayoung: binary vs source :P21:10
ayoung./tools/build.py keystone mariadb rsyslog kolla-toolbox21:10
SamYaplesorry thats my bad21:10
SamYaple./tools/build.py --type source keystone mariadb rsyslog kolla-toolbox21:10
ayoungkolla_install_type21:10
SamYapleyea21:10
SamYapleyou built rdo packages21:11
ayoungin globals, change that to binary21:11
SamYapleglobals is for ansible only21:11
SamYaplebut if you change globals to binary it will deploy yes21:11
ayoungk21:11
SamYapleyou did not build containers with the toolchain in them so you won't be able to do the setup.py dev tests21:11
SamYaplejust fyi21:11
ayoungSamYaple, ok  so the source usese the tarball, the binary builds using Deloarean packages?21:12
ayoungDelorean21:12
SamYaplecorrect21:12
*** achanda has joined #kolla21:12
ayoungSamYaple, I kindof love this....it feels right after Puppet21:12
SamYaplei also kind of love this, and once you get it setup it is _much_ _much_ easier to work with21:13
SamYaplebut the learning curve is a bit high if you dont know docker or ansible21:13
ayoungSamYaple, I did a whole Proof of concept using Ansible and Packstack last summit.  I've been battling tripleo and Puppet stuff.  The tooling is essential, and I think its taken a while to get the right vision here21:14
ayoungI've been working with Ansible for a while now, but Docker is new to me21:14
ayoungand I like that we have a whole docker ansible savvy team here21:14
SamYaplei have used docker for a few years21:15
SamYaplethen i worked at RAX during the OSAD devel (not as a dev)21:15
ayoungI'd love to replace devstack with something along these lines.  Narrow the distance between development and deployments21:15
SamYapleso that introduced me to ansible21:15
SamYaplei started a docker+ansible project, then merged it with kolla21:15
SamYapleso i know the pain of deployment21:16
SamYaplewould love to see devstack replaceemnt with kolla workflow21:16
*** sdake has quit IRC21:16
ayoungSamYaple, so, maybe we need:  ./tools/build.py --type git keystone mariadb rsyslog kolla-toolbox21:16
SamYaplewell you can deploy from git now21:16
SamYapleits a config option21:17
ayoungok...good enough21:17
ayoungI'll get there21:17
SamYaplelet me find it...21:17
SamYapleyea ayoung if you tox gen the config youll get a kolla-build.conf config21:17
SamYapleyou can change that to git instead of url and point it to a repo21:18
ayoungOK...might bug you again if I have trouble there...still going for positive thread  here on the source build21:18
SamYapleyea get _something_ going and then we can move forward21:18
SamYapleunfortuantely i think soe of this stuff is just tribal knowlege on master21:19
SamYaplesome*21:19
ayoungSamYaple, I might push to use something like this for Keystone functional tests21:19
ayoungNeed tests that run against a live MySQL and LDAP backend, Horizon with some Web SSO Identity provider and the like21:20
ayoungthis is much lower barrier to entry than devstack and full VMs21:20
SamYapleayoung: I will 100% help you with that21:21
SamYapleI did all the gate work here for kolla21:21
SamYaple(dont look its awful!)21:21
SamYaplewe are at an empasse with teh gate21:21
SamYaplewe are out of ram21:21
ayoungSamYaple, um...postgresql?21:22
ayoungah, just an RPM21:22
SamYaplei dont understand the question21:22
*** aginwala has quit IRC21:22
ayoungINFO:__main__:openstack-base:  Verifying  : postgresql-devel-9.2.14-1.el7_1.x86_64                      6/3921:22
ayoungI was looking at the log as build was running and saw a postgresql reference.21:23
*** aginwala has joined #kolla21:23
ayoungSamYaple, let me get this working, and I can then turn around and talk with the rest of Keystone Core and say "hey, this works"21:24
ayoung"lets use it to test"21:24
SamYapleayoung: yea thats for... something i think thats for gnocchie21:24
SamYaplegnocchi*21:24
ayoungand I think they will be happy with it21:24
SamYaplesomething needs the devel lib for postgresql21:24
SamYaplecool!21:24
*** esikachev has quit IRC21:24
ayoungYeah, I saw sqlite as well...few dependencies21:25
SamYaplewell we have an openstack-base container that has lots of common deps21:25
SamYaplebut not all contaienrs require all those deps21:25
inc0ayoung, I meant to write howto-use-kolla-instead-of-devstack but never got to do it21:25
ayounginc0, well, I might co-author it with you...21:25
SamYapleit greatly increases the speed of the build, and reduces the _total_ size ayoung, but its not perfect21:25
ayoungSamYaple, yeah, I grok the tradeoff21:26
inc0way I wanted to propose it is to run everything in kolla *besides* service you're working on21:26
ayoungOK,  source build completed.  deploy....21:26
inc0you can run that normally in terminal and don't have to deal with containers during development21:26
ayoungOSError: [Errno 13] Permission denied: '/etc/kolla/mariadb'21:26
SamYapleinc0: ayoung is trying to use it to test, but thats how i use it21:26
ayounglet me make that...21:26
SamYapleayoung: ok you *may* want to switch to my patch if you are unprivileged user21:27
ayoungI should probably chown /etc/kolla ,no?21:27
SamYapleotherwise youll want to run it as root hoenstly21:27
inc0or run plays as root21:27
ayoungWe'll see....21:27
SamYapleayoung: just chown /etc/kolla as your user21:27
SamYapleyoull be fine21:27
ayoungTASK: [mariadb | Copying over galera.cnf] *************************************21:28
ayoungfatal: [localhost] => {'msg': "One or more undefined variables: 'dict object' has no attribute u'ansible_dummy0'", 'failed': True}21:28
ayoungfatal: [localhost] => {'msg': 'One or more items failed.', 'failed': True, 'changed': False, 'results': [{'msg': "One or more undefined variables: 'dict object' has no attribute u'ansible_dummy0'", 'failed': True}]}21:28
inc0for CI testing, yeah that's cool...how many times we've seen broken deps that someone included...21:28
SamYapleayoung: oh right. ok so i left a comment in there21:28
SamYapledummy0 needs to be like eth0 for you21:28
inc0and didnt show up in CI because devstack and something else already installed it21:28
SamYapleand kolla_internal_interface should be your ip address on eth021:28
ayoungAh in the config21:28
*** Marga_ has joined #kolla21:28
SamYaplenormally kolla_internal_interface is a VIP that keepalive assigns, bt since you are running without keepalived you need to set it to your interface ip21:29
ayoungSamYaple, so my host has eth0 as 10.0.0.13  is that what I want for kolla_internal_address21:30
SamYapleyes21:30
SamYaple10.0.0.13 is kolla_internal_address21:30
SamYaplenetwork_interface: eth021:30
SamYaplewe dont bind to 0.0.0.0 so we need some specfics21:31
ayoungactually not eth0.  it is enp0s25.  Where did that come from?21:31
SamYaplei said eth0 before21:31
SamYaplebut youre a smart fella youll get it done!21:31
ayoungFedora is wwweierded21:31
*** achanda has quit IRC21:31
SamYaplednf install biosdevname21:32
ayoungSamYaple, OK, so , as I wait for this to deploy.  THe git option you posted before builds and image from those files, but they are not linked from the base image, right? They are copied, and if I want to test out a new change I need to regen the keystone image?21:32
SamYapleudev and there stupid consitent (inconsistent) naming rules21:32
*** aginwala has quit IRC21:32
SamYapleayoung: that is correct21:33
ayoungHmmm...no mariadb running21:33
SamYaplehas it crashed?21:33
ayoungfailed: [localhost] => {"attempts": 6, "changed": false, "cmd": ["docker", "exec", "mariadb", "ls", "/var/lib/mysql/mariadb.pid"], "delta": "0:00:00.019984", "end": "2016-02-05 16:32:42.132941", "failed": true, "rc": 1, "start": "2016-02-05 16:32:42.112957", "stdout_lines": [], "warnings": []}21:33
SamYapleah21:33
inc0didnt it got merged?21:33
inc0duh21:33
SamYapleso you should see container running21:33
SamYaple`docker ps -a`21:33
*** mbound has joined #kolla21:33
ayoungnot running21:34
SamYapledo `docker logs mariadb`21:34
ayoungrsyslog and kolla_toolbox  only21:34
SamYaplebut it exists?21:34
ayoungchown: changing ownership of '/var/lib/mysql': Permission denied21:34
ayoungum...you need to stop running as root21:34
SamYaplethats not the host21:34
SamYaplethats in the container21:34
ayounghmmm21:34
SamYaplebut like I said, use my patch at this point21:35
SamYapleyoull be happier21:35
SamYapleor run as root21:35
SamYaplethis in between will only give you headache21:35
ayoungOK,  I'll grab your patch21:35
inc0SamYaple, can you plz remind me link?21:35
SamYapleinc0: https://review.openstack.org/#/c/276887/21:35
inc0because it's ridiculus, it's there for quite some time21:35
inc0ah not this one21:35
inc0but ok, will review21:36
ayoungSamYaple, so now copy /etc/kolla to...21:36
ayoungor just rerun as is?21:36
*** aginwala has joined #kolla21:37
SamYapleayoung: http://paste.openstack.org/show/486155/21:37
SamYapleyoull need to clean your environement21:37
SamYaplewe have a clean-containers script in the tools/ folder21:37
ayoungSamYaple, images are OK?21:38
*** esikachev has joined #kolla21:38
openstackgerritSam Yaple proposed openstack/kolla: Fix non-root deploys  https://review.openstack.org/27688721:38
SamYapleayoung: yes21:38
SamYapleignre that new patch21:38
SamYapleit wont affect you21:38
ayoung./tools/kolla-ansible --configdir /home/ayoung/kolla   deploy21:39
ayoungrunning now...21:40
SamYapledid you clean all containers first?21:40
SamYapleand volumes?21:40
ayoungnot volumes21:40
ayoungjust containers21:40
SamYapleif yo u ran script it should have done both21:40
SamYapleif you cleaned up manually volumes probably stuck around21:40
ayoung./tools/cleanup-containers21:40
SamYapleyup should have gotten volulmes21:40
ayoungstill failed at mysql21:40
ayoungstderr: ls: cannot access /var/lib/mysql/mariadb.pid: No such file or directory21:40
SamYapleyea21:41
inc0docker volume rm $(docker volume ls)21:41
SamYapleayoung: yea volume stuck around :/21:41
*** achanda has joined #kolla21:41
SamYapleclean containres and do what inc0 has21:41
ayoungok  let me21:41
SamYapleyou can confirm with `docker volume ls` that volumes are removed21:41
ayoungfc7cdb24b442        kollaglue/centos-source-mariadb:2.0.0   "kolla_start"       4 minutes ago       Up 4 minutes                            mariadb21:41
ayoung  still tehre21:41
ayounglet me kill that manually21:41
inc0docker rm $(docker ps -a) && docker volume rm $(docker volume ls)21:41
SamYaplelol yea that21:41
SamYapleif you want to nuke everything21:42
inc0docker rm -f $(docker ps -a) && docker volume rm $(docker volume ls)21:42
ayoungError response from daemon: Cannot kill container fc7cdb24b442: [2] Container does not exist: container destroyed21:42
ayoungError: failed to kill containers: [fc7cdb24b442]21:42
ayounghmmm21:42
inc0thats disturbing21:42
ayoungSamYaple, that won't kill images though, right?21:42
inc0nah, images are safe21:42
ayoungOK,  I restarted the docker service, container was still there, but now I was able to kill it and ps shows it is gone21:43
SamYapleayoung: correct images are safe, controled by `docker images`21:43
*** Marga_ has quit IRC21:43
ayounghow do I list volumes?21:44
*** mbound has quit IRC21:44
inc0docker volume ls21:44
SamYaple`docker volume ls`21:44
ayoungman docker does not have volume specified21:45
inc0what do you mean?21:45
*** esikachev has quit IRC21:46
SamYapleayoung: yea docker just recently got man pages21:46
ayoungtried to open a new terminal and getpt failed: Permission denied21:46
SamYaplecheck your perms on /dev/ptmx21:47
inc0aaa you run kolla and gnome;)21:47
inc0I have no idea why that happends21:47
SamYapleinc0: nah i get the same probelm too21:47
SamYaplefix perms on /dev/ptmx21:47
ayoung$ sudo ls -la  /dev/ptmx21:47
ayounglrwxrwxrwx. 1 root root 8 Feb  5 16:45 /dev/ptmx -> pts/ptmx21:47
SamYapleok so check /dev/pts/ptmx :P21:47
inc0yeah I did too21:47
ayoungc---------. 1 root root 5, 2 Jan 30 11:42 /dev/pts/ptmx21:48
ayoungI assume 700?21:48
SamYaplefix that21:48
SamYapleyea21:48
SamYapleor 77721:48
*** aginwala has quit IRC21:48
SamYaplei dont know default21:48
SamYaplesupposedly thats fixed in docker 1.1021:48
ayoung77721:48
ayoung700 still perm denied21:48
*** Marga_ has joined #kolla21:49
*** Marga_ has joined #kolla21:49
ayoungI wonder if that is supposed to be read only...let's see21:49
*** Marga_ has joined #kolla21:50
ayoung666 worked21:50
ayoungIs that going to be happening a lot?21:50
SamYapleayoung: its a docker bug21:50
SamYaplesupposedly fixed in 1.1021:51
SamYaplei think it happens at daemon start21:51
inc0and wont happen on servers21:51
*** aginwala has joined #kolla21:51
SamYapleit happens... randomly21:51
inc0at least never experienced it21:51
SamYapleyea and inc0 is right21:51
ayoungmariadb failed again21:51
SamYapleinc0: binding /dev /for syslog broke docker!21:51
ayoungstderr: ls: cannot access /var/lib/mysql/mariadb.pid: No such file or directory21:52
SamYapleayoung: can you check logs again?21:52
SamYaplesame chown issue?21:52
ayoungthe funny thing is, I was successful getting the mariadb server to launch with the Fedora docker container21:52
ayoungSamYaple, Its running again, one sec21:52
ayoungchown: changing ownership of '/var/lib/mysql': Permission denied21:53
SamYapleugh21:53
SamYapleso we do not run as root inside the containers ayoung21:53
SamYaplebut we need _some_ root commands. and thats whats tripping you up21:53
SamYapleayoung: https://github.com/openstack/kolla/blob/master/docker/mariadb/extend_start.sh#L2321:54
SamYaplethe container itself has a sudoers entry for that, but i suspect its selinux at play here21:54
ayoungSamYaple, let me see what the Fedora one does...21:54
SamYapledo you have it set to enforce mode?21:54
ayoungSamYaple, selinux?21:55
SamYapleayoung: yep21:55
ayoung$ getenforce21:55
ayoungEnforcing21:55
SamYapleyoull have to not.... kolla + selinux wont work until someone writes the profiles21:55
SamYapleno one has written the profiles21:55
ayounghttps://github.com/fedora-cloud/Fedora-Dockerfiles/tree/master/mariadb/scripts21:56
ayoungmaridb should work ok21:56
SamYapleayoung: this is not related to mariadb21:57
SamYaplebut rather kolla21:57
SamYaplebecause we are calling sudo in the container we are violating some selinux things21:57
SamYaplewe need profiles to allow those things21:57
SamYapleno one has written those profiles21:57
SamYaplethus we need selinux turned off :(21:58
ayoungDocker runs as root, why do we need to sudo?21:58
SamYaplebecause the containers run as unprivileged users21:58
inc0we don't run root inside containers21:58
SamYapleso if anyone breaks out of a service with an exploit, they are running as unprivileged user21:58
SamYapleinc0: ive always hated that drop-root stuff :/21:59
inc0yeah I don't really get it21:59
ayoungSamYaple, so you think if I setenforce permissive it will work?21:59
SamYapleayoung: i do21:59
ayounglet me try21:59
inc0openstack services are normally run as root21:59
inc0but put this service inside container and it's security risk22:00
inc0I mean...it's good mindset to remove permissions that aren't needed22:00
inc0but it causes problems22:00
SamYaplei agree, but i wanted to get stablier first22:01
ayoungWell, I think it is possible to run mysql without needing root permissions22:01
SamYapleayoung: it absolutely is22:01
SamYaplethats not the issue22:01
ayoungif it listens on a port above 100022:01
SamYapleso we use named volumes with docker22:02
SamYaplethose mount in as root22:02
SamYaplewe can do nothing to change that as of yet22:02
ayounglooking good22:02
SamYaplebut those perms need to change for us to use the named volume22:02
ayoungI just saw the Keystone stuff fly past22:02
SamYaplehence the sudo22:02
SamYapleayoung: you are free to stop it then22:02
SamYapleits not going to do anything else22:02
ayoungHorizon failed.  No surprise.  I adde that to the ignor in globlas, right?22:03
*** aginwala has quit IRC22:03
SamYapleprobably22:03
SamYaplenext time run with just `kolla-ansible deploy --tags keystone,mariadb`22:03
SamYapleit will filter all the rest22:03
inc0so if you have mariadb and stuff deployed22:03
inc0yeah, I wanted to say about tags22:03
ayoungI might actually enable rabbit next run.  NOtifications from Keystone are sent over the mq22:04
ayoungwould like to make that part of testing22:04
*** dims has joined #kolla22:04
ayoungI think I'm good here, except for SELinux.22:04
*** dimsum__ has quit IRC22:04
ayoungOK, that was fast22:04
ayounghow to test?22:04
ayoungI want to do openstack token issue22:04
SamYapleayoung: normal openstack ocmmands22:05
SamYapleendpoint is at 10.0.0.1322:05
ayoungis there an rc file generated?22:05
inc0netstat -nalp | grep 500022:05
SamYapleayoung: yea22:05
inc0is should be there22:05
SamYaplekolla-ansible post-deploy22:05
SamYaplethat will spit out admin-openrc.sh to your config directory22:05
SamYaplenext to globals.yml22:05
SamYapleyou can source that then `openstack user list`22:05
ayoungum...I think it does not honor the conf dir, though22:06
ayounglooking in /etc/kolla, there is a lot there still22:06
ayoungincluding the dev dir...22:06
ayoung$ openstack token issue  succeeded22:07
ayoungOK, thanks guys, I'm going to take this success for today.22:07
ayoungSamYaple, inc0 you two going to Austin?22:08
inc0yeah, I actually live 70miles from Austin22:08
SamYapleayoung: indeed22:08
ayoungSamYaple, inc0, first beer is on me22:08
SamYapleayoung: i just moved ~1000 miles away!22:08
*** neilus has joined #kolla22:08
*** jtriley has quit IRC22:08
inc0ayoung, will remember that for sure22:09
ayounginc0, I hope they still let you get a hotel.22:09
inc0I'll get airbnb or sth22:09
SamYapleayoung: yea lots to be desired for non-root stuff. we do no testing on that as of yet. I was hoping we would this cycle but it was not made a priority... I will try to make it one next cycle22:09
inc0I'm staying in Austin for alcohol, that one is certain22:09
ayoungOK...so, now that I have this, let me test out the features that just landed in Keystone.22:10
SamYapleayoung: how was the speed of deploy for you?22:11
ayoungSamYaple, pretty snappy22:11
ayoungI didn;'t time it, but not long enough for me to get bored and go read the news22:12
SamYaplecool22:12
SamYaplefull openstack AIO deploy for me takes under 2 minutes22:12
SamYaplethats with prebuilt images of course22:12
*** neilus has quit IRC22:13
dmsimardayoung: curious, are you deploying source or binary ? centos or ubuntu ?22:14
SamYapledmsimard: centos source22:16
dmsimardok22:17
*** bmace has quit IRC22:18
*** bmace has joined #kolla22:19
ayoungSamYaple, OK, so what are these "profiles" that need to be written for the SELinux stuff to work?22:19
SamYaplemight be called contexts ayoung ?22:20
SamYapleprofiels are for apparmor22:20
ayoungSamYaple, you mean SELinux policy?22:21
SamYaplethat sounds right22:21
SamYaplethat is right yup22:21
ayounghmm...OK.22:21
SamYapleayoung: https://blueprints.launchpad.net/kolla/+spec/enable-selinux22:22
ayoungI'm ot 100% certain what the right thing to do there is.  Would you expect that, once the cluster is deployed, I can re-enable selinux and everythign would work?22:22
SamYaplenihilifer: you make any progress on this?22:22
SamYapleayoung: yes, once deployed22:22
ayoungOK...it hasn;t blown up yet22:22
SamYaplenot for a full stack22:22
SamYaplebut just what you deploy it will work22:22
nihiliferSamYaple: you mean selinux?22:22
SamYaplenihilifer: yea22:22
nihiliferthen nope22:22
nihiliferwell, currently the only thing we can do regarding selinux, is adding ":z" suffix to volumes22:24
SamYaplewhat does that do?22:24
nihiliferapplies svirt_sandbox_file_t flag to the volume22:24
SamYapleok guys I to go move my stuff into my new house22:25
SamYapleill be around later22:25
nihiliferin selinux, you need this flag to be applied on the directory to mount it as a volume22:25
nihiliferok22:25
SamYapleping me if you hit anymore issue ayoung, my irc is always ticking22:25
SamYaplethanks for the info nihilifer22:25
nihilifernp22:25
ayoungSamYaple, thanks,  will do.  Right now I'm going to make sure some keystone features are working.22:25
ayoungProbably all I'll get to tonight22:25
*** aginwala has joined #kolla22:26
*** jtriley has joined #kolla22:27
inc0ayoung, if that helps, I sit right next to Dolph, so if you guys run into problems I can provide quick help22:34
*** aginwala has quit IRC22:39
*** salv-orl_ has joined #kolla22:40
*** salv-orlando has quit IRC22:43
*** mbound has joined #kolla22:45
*** mbound has quit IRC22:50
*** aginwala has joined #kolla22:51
*** jtriley has quit IRC23:02
*** dims_ has joined #kolla23:06
*** neilus has joined #kolla23:08
*** dims has quit IRC23:10
*** neilus has quit IRC23:12
*** farid has quit IRC23:20
*** aginwala has quit IRC23:28
*** aginwala has joined #kolla23:31
*** aginwala has quit IRC23:34
*** inc0 has quit IRC23:37
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Dependencies clean up / simplification  https://review.openstack.org/27291223:38
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Use a new per-service format  https://review.openstack.org/27384323:38
*** stvnoyes has quit IRC23:43
*** stvnoyes has joined #kolla23:44
*** aginwala has joined #kolla23:45
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Use a new per-service format  https://review.openstack.org/27384323:45
*** aginwala has quit IRC23:46
*** aginwala has joined #kolla23:46
*** aginwala has quit IRC23:53
*** aginwala has joined #kolla23:56
*** iotapi322 has quit IRC23:59
*** jtriley has joined #kolla23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!