Thursday, 2015-11-05

SamYaplediogogmt: is your '~/kolla/.gitreview' file correct?00:03
SamYapledoes '~/.gitconfig' have a gitreview section?00:03
SamYapleare you using the global git-review.conf?00:03
diogogmtSamYaple http://pastebin.com/gnit7SR400:07
kfox1111_Any updates on the galera recovery guide thing?00:07
diogogmti even tried using https and i get the same problem, does it take a while for my account to be activated?00:07
kfox1111_want to deploy kolla in the next few days but can't without it.00:07
asalkelddiogogmt: it might be that00:07
diogogmti’ll give it a shot tomorrow then, i just register my account 30 mins ago so it might be that00:08
SamYaplekfox1111_: we talked about lightsout at the summit that wil lfix rabbitmq00:10
SamYaplekfox1111_: the galera stuff I can write, but since rabbitmq will be changing I cannot write those docs00:10
SamYaplegood news is it will not require operator intervention with the new way ive implemented it00:10
SamYapleso rabbitmq auto recovers00:11
*** jasonsb has quit IRC00:11
kfox1111_ah. cool.00:11
*** jasonsb has joined #kolla00:11
SamYaplethat change will not land until the 1.1 release of Kolla unfortunately00:11
SamYaplei think sdake plans two weeks until then00:12
SamYaplesdake_: ^00:12
kfox1111_for the first phase of deployment, I'm just do keystone/dashboard/galera for keystone.00:12
kfox1111_this will support all of our planned regions.00:12
kfox1111_so I think we can live without rabbit on that.00:12
kfox1111_but since its so central, the recovery piece is really critical.00:12
diogogmtswitching gears a bit, any ideas where this error would be coming from? http://pastebin.com/XRmzrFeZ00:13
diogogmti have all the other service images already built00:13
SamYaplesounds interesting. let me know how that works out.00:13
kfox1111_yup. :)00:13
diogogmtit just fails at the lokolla/centos-binary-openstack-base:latest00:13
SamYaplekfox1111_: after the summit discussion i will need to tweak the mariabd container so let me get on that now00:13
kfox1111_k. thx.00:13
kfox1111_I was really hoping that keystone federation would be finished. :/00:14
kfox1111_then we could do a keystone for authentication, and a keystone per region that is federated.00:14
kfox1111_but dashboard support still doesnt work!?! :(00:14
kfox1111_federation without dashboard isn't very useful. :/00:14
SamYaplein kolla?00:14
SamYaplewe havent done much with federation in kolla. where you see limitations please file bugs00:15
kfox1111_no. in openstack-dashboard. it onlyh knows how to login to one keystone and doesn't support federation at all yet.00:15
SamYapleoh ok00:15
kfox1111_it needs to know how to authenticate to one keystone, then ask that keystone for a token for the federated to keystone.00:15
SamYapleyup00:16
kfox1111_not a thing yet. :/00:16
SamYapleitll get there00:16
kfox1111_yeah. but when public clouds' are selling you "you can federate to us", its not quite true.00:16
kfox1111_its more like, if your an api developer, you can federate to us.00:16
kfox1111_if your an end user, not so much. :/00:16
kfox1111_hopefully mitaka will finish it up.00:17
openstackgerritSam Yaple proposed openstack/kolla: Add Docker in Docker container  https://review.openstack.org/24144400:31
openstackgerritSam Yaple proposed openstack/kolla: Remove unused setup_fedora.sh  https://review.openstack.org/24149800:31
openstackgerritSam Yaple proposed openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24149900:31
openstackgerritSam Yaple proposed openstack/kolla: Adjust gate btrfs setup  https://review.openstack.org/24146500:31
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150000:31
*** achanda has quit IRC00:32
SamYaplekfox1111_: so you know how to recover from a normal galera lightout event?00:41
SamYapleor is this all new?00:41
kfox1111_I do, but my management doesn.t so if you can easily copy/paste in those bits, that woudl be cool. if its much more expedient to just link to the other docs, thats cool too.00:42
*** kelkoobenoitr has quit IRC00:42
*** diogogmt has quit IRC00:42
SamYapleok ill link a few docs around and post the general idea00:42
SamYapleim going to try and write a playbook to automate the steps involved00:42
*** nihilifer has quit IRC00:44
kfox1111_awesome. thanks. :)00:44
*** kelkoobenoitr has joined #kolla00:46
kfox1111_should we be standing up our own docker registry or using stuff streight off the hub, and should we be building containers ourselves, or using the kollaglue ones?00:52
SamYapleyour call on both accounts00:52
SamYaplekolla will be providing vanilla images, but you can provide our own as well00:53
kfox1111_ok. if you were me? :)00:53
SamYaplewell pushing to dockerhub isnt an option for me, so i have a local registry00:53
kfox1111_ah.00:53
SamYaplebut if it were prod, it would be in dockerhub00:53
SamYapleit just slows down dev too much00:53
kfox1111_yeah, I could see that.00:54
kfox1111_k. thanks.00:54
SamYaplefor the company i work at I will definetely be building my own images, but i will also be applying custom patches on nova and neutron00:54
SamYapleso i have those forked00:54
kfox1111_is the repo location all or nothing?00:54
SamYapleusing Kolla-team build images makes you dependant on us to deliever updates, and that may not be acceptable to your SLAs00:55
SamYapleno, you can mix and match00:55
SamYaplebut if the base layer does match you end up with different libs and sutff in the containers00:55
kfox1111_ok. so I may just pull kollaglue ones all local, just to make it easy, then I can switch out some for others later if I need to.00:55
SamYapleyes that should work just fine00:56
kfox1111_cool. thanks. :)00:56
SamYaplebut from a purist standpoint you introduce the risk of different libs and that mamy cause problems00:56
SamYaplejust keep it in mind00:56
kfox1111_yeah. always a potential issue with docker containers in general.00:57
SamYapleno more so than bare metal00:57
SamYapleinfact less so than bare metal00:57
kfox1111_right. :)00:57
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150000:57
kfox1111_hmm....00:58
kfox1111_https://hub.docker.com/r/kollaglue/centos-rdo-base/tags/00:58
kfox1111_don't see a liberty or a latest.00:58
kfox1111_what's the tagging scheme?00:58
SamYaplethere should be a liberty-1.0.0 image, but perhaps no one pushed it. i dont use centos so i dont know who was taking up the reighs on that one00:59
SamYaplesdake should know00:59
kfox1111_k. thx.00:59
*** sdake_ has quit IRC01:12
*** sdake has joined #kolla01:14
*** ssurana has quit IRC01:14
kfox1111_sdake: not seeing a liberty-1.0.0 in kollaglue/centos-rdo-base. know when those will be pushed?01:15
sdakekofx they werednt psuehed because 1.0.0 was doa01:23
*** asalkeld is now known as asalkeld_afk01:25
*** tzn has quit IRC01:25
bmaceeh?  i know a while back there was talk about when to push images up to dockerhub and tag them, etc.  were the images non-function as of the 1.0.0 tag?01:26
SamYaplebmace: they were fine when 1.0.0 was tagged01:27
SamYaplethere was an update to xtrabackup that broke centos (only centos)01:27
SamYapleso since the images werent pushed after they were tested for the tag01:27
SamYaple1.0.0 can't be built anymore for centos01:27
SamYapleubuntu still works01:28
openstackgerritSam Yaple proposed openstack/kolla: Add Docker in Docker container  https://review.openstack.org/24144401:28
openstackgerritSam Yaple proposed openstack/kolla: Remove unused setup_fedora.sh  https://review.openstack.org/24149801:28
openstackgerritSam Yaple proposed openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24149901:28
openstackgerritSam Yaple proposed openstack/kolla: Adjust gate btrfs setup  https://review.openstack.org/24146501:28
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150001:28
*** ByPasS has quit IRC01:28
bmaceseems like it would be awfully nice to have at least some set of working images out there, for some distro.01:29
kfox1111_oh. bummer....01:29
*** ByPasS has joined #kolla01:29
kfox1111_whem will 1.1 be released?01:29
SamYaplesdake: ^01:30
bmaceno point in even having a space on dockerhub if the images are always useless :)01:30
SamYaplekfox1111_: there is a list of things needed to be backported/fixed for 1.1.0 and we have a timeframe to do so01:30
kfox1111_I need to get a keystone/galera cluster up by next week. :/01:30
SamYaplebmace: no the problem was the images were never pushed01:30
SamYaplebmace: they _would_ be useful if they were pushed when they were tested01:30
SamYaplebut you can blame dockerhub for alot of this01:31
SamYaplepushes take like 8 hours01:31
SamYaplestupid 2.001:31
bmaceyeah, that is downright silly01:31
kfox1111_yeah. shouldn't take that long. :/01:31
SamYaplefrom my house I have to let it run for 14 hours for a full push01:31
SamYapleim only pushing up 2GB of raw data :(01:32
kfox1111_wow.01:32
kfox1111_hmmm...01:32
SamYapleshould* only be pushing up 2GB01:32
kfox1111_would it be possible to just upload a bunch of generated Docker files to github repo's that are tracked by hub autobuilding repo's?01:33
kfox1111_(Bad hack....)01:33
SamYaplethe 2.0 registry has a problem in that it makes you push the same layer up multiple times01:33
kfox1111_ah.01:33
SamYaplekfox1111_: yes i was just looking into that :)01:33
SamYapleit may be possible yes01:33
*** tpot has joined #kolla01:33
*** diogogmt has joined #kolla01:33
kfox1111_the one issue I've run into with that path, is you can auto build, or you can upload tagged stuff,01:33
kfox1111_but you cant tag autobuilt stuff without uploading. :/01:34
kfox1111_so you can't do an autobuild, test it, then tag the tested thing as "stable" :/01:34
SamYapleyou can.... but you have to download it and repush it up P01:34
kfox1111_I wasn't able to push to an autobuilt repo. it was blocking it.01:34
kfox1111_you have to create a stable repo. :/01:34
kfox1111_and then you almost don't want to bother with the autobuilding one. :/01:34
SamYapleoh then nevermind01:35
SamYapleok i gotta go get food for the family01:35
SamYaplebbiab01:35
kfox1111_k. have a good one.01:35
*** dolpher has joined #kolla01:40
*** tzn has joined #kolla01:46
*** dims has joined #kolla01:49
*** tzn has quit IRC01:56
*** jasonsb has quit IRC02:02
sdakekfox i think tagging 1.0.1 and dpsuhing images is a reasonable course of faction02:03
sdakekfox1111_ ^^02:03
sdakei just woke up from 36 hours of sleep - so something clearly is busted on my end02:03
sdakeand my mom is afu in the hospital atm so i'm a bit defocused from kolla atm02:04
*** bmace_ has joined #kolla02:08
*** tfukushima has joined #kolla02:08
SamYaplehope your mom gets better02:08
*** alisonh1 has joined #kolla02:09
*** stvnoyes1 has joined #kolla02:10
*** alisonh has quit IRC02:10
*** stvnoyes has quit IRC02:11
*** bmace__ has joined #kolla02:11
*** bmace has quit IRC02:11
*** alisonh has joined #kolla02:12
*** stvnoyes has joined #kolla02:13
*** alisonh1 has quit IRC02:14
*** stvnoyes1 has quit IRC02:14
*** bmace_ has quit IRC02:14
*** tfukushima has quit IRC02:17
*** asalkeld_afk is now known as asalkeld02:18
*** tzn has joined #kolla02:23
*** tfukushima has joined #kolla02:25
sdakeSamYaple wsass the team meeting held this mroning?02:25
SamYaplei was asleep02:26
SamYaplelooking at the logs.... no02:26
SamYaple:) timezone confusion02:26
*** achanda has joined #kolla02:27
openstackgerritSam Yaple proposed openstack/kolla: Add Docker in Docker container  https://review.openstack.org/24144402:27
openstackgerritSam Yaple proposed openstack/kolla: Remove unused setup_fedora.sh  https://review.openstack.org/24149802:27
openstackgerritSam Yaple proposed openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24149902:27
openstackgerritSam Yaple proposed openstack/kolla: Adjust gate btrfs setup  https://review.openstack.org/24146502:27
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150002:27
docaedowhat's the regularly scheduled time for the meeting? I need to put it on my calendar (for weeks when there's less jet lag, time zone confusion and sicknesses of course)02:27
SamYapledocaedo: its in the topic02:27
SamYaple1630 UTC02:27
SamYaplewednesday02:27
*** tzn has quit IRC02:28
docaedoSamYaple: ahahaha, yep, right in front of me - but who looks at topics??02:28
docaedoSamYaple: thanks :)02:28
SamYaplenot me!02:28
*** tzn has joined #kolla02:28
*** tzn has quit IRC02:29
docaedoah damn, math tells me that translates to a local time which finds me in the car taking a kid to a thing every wednesday :/02:29
SamYapleuse yer phone02:29
* docaedo can't wait for robot cars to arrive, so he can join while in transit :)02:30
*** _tzn has joined #kolla02:30
*** vilobhmm11 has quit IRC02:35
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150002:47
*** tfukushima has quit IRC02:51
*** tfukushima has joined #kolla02:55
*** sdake_ has joined #kolla03:02
*** sdake has quit IRC03:03
*** cloudnautique has joined #kolla03:10
*** cloudnautique has quit IRC03:11
*** sdake_ has quit IRC03:16
*** cloudnautique has joined #kolla03:26
*** cloudnautique has quit IRC03:26
*** cloudnautique has joined #kolla03:27
*** tobe has joined #kolla03:27
*** jasonsb has joined #kolla03:27
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150003:30
openstackgerritSam Yaple proposed openstack/kolla: Allow disabling of sysctl values  https://review.openstack.org/24185603:30
*** tfukushima has quit IRC03:34
*** jasonsb has quit IRC03:34
*** _tzn has quit IRC03:35
*** cemmason has joined #kolla03:43
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150003:43
*** tfukushima has joined #kolla03:47
*** unicell has joined #kolla04:02
*** unicell1 has quit IRC04:02
*** Slower has joined #kolla04:05
*** tfukushima has quit IRC04:06
*** vilobhmm11 has joined #kolla04:14
*** achanda has quit IRC04:22
*** tzn has joined #kolla04:25
*** tzn has quit IRC04:30
*** jasonsb has joined #kolla04:38
*** tfukushima has joined #kolla05:07
openstackgerritSam Yaple proposed openstack/kolla: Add Docker in Docker container  https://review.openstack.org/24144405:11
openstackgerritSam Yaple proposed openstack/kolla: Allow disabling of sysctl values  https://review.openstack.org/24185605:11
openstackgerritSam Yaple proposed openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24149905:11
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150005:11
*** tfukushima has quit IRC05:11
*** achanda has joined #kolla05:22
*** achanda has quit IRC05:30
*** nihilifer has joined #kolla05:39
*** athomas has quit IRC05:41
*** achanda has joined #kolla05:42
*** dwalsh has joined #kolla05:55
*** tfukushima has joined #kolla05:58
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150006:06
*** tpot has quit IRC06:14
*** vilobhmm111 has joined #kolla06:20
*** vilobhmm11 has quit IRC06:23
*** tzn has joined #kolla06:27
*** tzn has quit IRC06:29
*** tzn has joined #kolla06:29
openstackgerritKuo-tung Kao proposed openstack/kolla: using registry v2 in image-building doc  https://review.openstack.org/24190006:33
*** rajalokan has joined #kolla06:37
*** rajalokan has quit IRC06:38
nihiliferKuo-tung, are you in IRC? ;)06:38
kjellynihilifer: Yes06:47
nihiliferkjelly: hi06:47
kjellynihilifer: I reproduct the problem I meet. And post error message to you.06:47
nihiliferkjelly: oh, ok06:48
nihiliferkjelly: I wanted to ask about v2 registry06:48
nihiliferdoes it work for you fine?06:48
kjellynihilifer: Yes, I use v2 and work fine.06:49
kjelly[+1]1d [user@user-GA-6PXSV3:~/github/kolla] master 1 ± sudo docker run -d -p 3000:5000 --restart=always  registry06:49
kjelly[sudo] password for user:06:49
kjelly8043877990ce455280046d48d4ab86b9270b683410d95566015280fb6876fd9206:49
kjelly[+1]1d [user@user-GA-6PXSV3:~/github/kolla] master ± sudo tools/build.py --registry 172.22.2.81:3000 -b ubuntu -t source --push data06:49
kjellyINFO:__main__:data:Building06:49
kjellyINFO:__main__:data:Step 0 : FROM ubuntu:latest06:49
kjellyINFO:__main__:data: ---> 1d073211c49806:49
kjellyINFO:__main__:data:Step 1 : MAINTAINER Kolla Project (https://launchpad.net/kolla)06:49
kjellyINFO:__main__:data: ---> Using cache06:49
kjellyINFO:__main__:data: ---> a1d87cb7dd4a06:49
kjellyINFO:__main__:data:Step 2 : CMD /bin/sleep infinity06:49
nihilifernooooo06:49
kjellyINFO:__main__:data: ---> Using cache06:49
kjellyINFO:__main__:data: ---> 09f6ae18edc606:49
kjellyINFO:__main__:data:Successfully built 09f6ae18edc606:49
kjellyINFO:__main__:data:Built06:49
kjellyERROR:__main__:unable to ping registry endpoint https://172.22.2.81:3000/v0/06:49
kjellyv2 ping attempt failed with error: Get https://172.22.2.81:3000/v2/: EOF06:49
nihiliferplease not paste it here!!!06:49
kjelly v1 ping attempt failed with error: Get https://172.22.2.81:3000/v1/_ping: EOF06:50
kjellyINFO:__main__:Images that failed to build06:50
kjellyINFO:__main__:===========================06:50
kjellyERROR:__main__:data      Failed with status: error06:50
nihiliferomg...06:50
kjellyWell, So many message.06:50
kjellynihilifer: Sorry06:50
nihiliferpaste.openstack.org ;)06:50
nihiliferplz06:50
nihiliferdo you use vagrant or you deployed it somewhere else?06:50
kjellynihilifer: http://paste.openstack.org/show/478051/06:51
nihiliferok, thx06:51
kjellynihilifer: I am not use vagrant. And the issue happen when building image then pushing.06:51
kjellynihilifer:wait06:52
nihilifersorry, I'm so confused now...06:53
kjellynihilifer: wait a moment.06:53
nihiliferyou say that registry v2 is working fine for you and in the same time you cannot ping to it or push?06:53
nihiliferor are these things separate?06:53
nihiliferok, I'll wait06:54
kjellynihilifer: I found I can't reproduce the error I meet before.07:00
kjellynihilifer: In 11/03, I have error about the docker registry too old. But I don't save the message.07:00
kjellynihilifer: I abandon the patch now. And try to reproduct the problem I meet before. If I can reproduct it, I will reopen the patch.07:01
nihiliferwell, if registry v2 will work for someone, it would be great to make such a change to kolla07:02
kjellynihilifer: Now I can push image to  registry v2. But I don't test it using ansible.07:04
kjellynihilifer: I will test it using ansible in all in one mode.07:04
bmace__that was a docker version specific issue, there is a known ansible bug related to it.07:07
bmace__i believe the problem was with docker 1.8.3 with v1 registries07:07
kjellynihilifer: I recall what the problem I meet. http://paste.openstack.org/show/478053/07:08
kjellynihilifer: I have the problem when deploying instead of building07:08
SamYaplethe problem with docker 1.8.3 is registries and ansible module07:14
SamYapleblame docker for chaning the api without chanigng the api version07:14
SamYaplei brought it to thier attention07:14
kjellySamYaple: http://paste.openstack.org/show/478053/ Is the problem  docker issue?07:17
openstackgerritSam Yaple proposed openstack/kolla: Add Docker in Docker container  https://review.openstack.org/24144407:17
openstackgerritSam Yaple proposed openstack/kolla: Allow disabling of sysctl values  https://review.openstack.org/24185607:17
openstackgerritSam Yaple proposed openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24149907:17
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150007:17
SamYaplekjelly: kinda docker changed some stuff (you seem to be rrunning >1.8.2 docker) in 1.8.307:18
SamYaplekjelly: because of this some outputs changed. the ansible docker module has to update to correct for these changes07:19
SamYaplethe issue is two fold, firstly ansible will not be releasing another 1.9.x tag07:19
SamYapleso onto ansible 2.0 we go07:19
SamYaplesecond, ansible will not be catching this return (without additional pressure) forcing you to use the v2 registry07:20
SamYapleso basically we are stuck with 1.9.4 ansible and docker 1.8.2 as a requirement until ansible 2.0 lands07:20
SamYaplethen we will have to force ansible 2.0 and regisry v2 across the board07:21
SamYaplewhen docker 1.10 drops we will also have a hard requirement on that since it has some features we will be using07:21
*** sdake has joined #kolla07:22
kjellySamYaple: Yes, I use Docker version 1.8.3. I reinstall my computer in monday.07:23
kjellySo I hit the issue.07:23
*** sdake_ has joined #kolla07:24
SamYapleKolla has a hard <=1.8.2 requitement07:25
SamYaplerequirement*07:25
SamYapleits in the documentation07:25
kjellyOk. I saw. Thank for SamYaple, nihilifer07:27
*** sdake has quit IRC07:27
SamYaplesorry about the trouble07:29
SamYapleI have been seriously considering building a docker module for us07:29
SamYapleim tired of this change all the time stuff07:29
kjellySamYaple: It seems good.07:31
sdake_what type of module would oyo ubuild samyaple07:32
bmace__i assume a replacement of the ansible docker stuff07:34
SamYaplesdake_: what bmace__ said07:34
sdake_soundsgood go forth and prospoer ;)07:34
SamYapleits overly bloated and it has to have support for all versions of all docker07:34
SamYaplei hate breaking away from the community though07:35
bmace__SamYaple except instead it has broken support for many versions? :)07:35
SamYaplestill seems like it may resolve more problems for us as time goes on07:35
SamYapleyea i know07:35
SamYapleif there was a better way.....07:35
SamYapleit feels dirty07:35
sdake_feels good to me07:35
sdake_it woudl be differenet in ansiblee guys would cut a 1.9.507:35
sdake_which they wil not do07:35
* SamYaple makes note: Never cross sdake_ 07:36
sdake_just being pragmatic here ;)07:36
SamYapleme too its why i said it07:36
SamYapleit would help us get new features quickly and fix bugs as well07:37
SamYaplethe docker module is the most volitale module we use from them... and the most critical07:37
SamYaplemaybe it is the right call07:37
sdake_imo get a review up and ssee what the cr team says07:40
SamYapleit would allow us to keep using registry v1 until they fix v207:41
sdake_boy my time is afu07:45
sdake_slept 36 hours about 6 hours ago07:45
sdake_ready to go to bbed again :(07:45
kjellyGood night, sdake.07:46
sdake_need to stay up07:46
*** vilobhmm111 has quit IRC07:47
SamYaplesdake_: ive been the same way07:50
*** slotti has joined #kolla07:56
*** zigo has joined #kolla08:00
zigoHi there!08:00
zigoAnyone awake already over here?08:00
asalkeldhi zigo08:00
zigoHi ! :)08:00
zigoasalkeld: I'm starting the initial Debian packaging of Kolla.08:00
asalkeldnice08:00
zigoasalkeld: Can you help by replying to a few questions?08:01
zigoEasy questions ...08:01
openstackgerritSam Yaple proposed openstack/kolla: Add Docker in Docker container  https://review.openstack.org/24144408:01
openstackgerritSam Yaple proposed openstack/kolla: Allow disabling of sysctl values  https://review.openstack.org/24185608:01
openstackgerritSam Yaple proposed openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24149908:01
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150008:01
asalkeldas best as i can, others here08:01
asalkeldcan help too08:01
zigoI need to know what are the names of daemons running.08:01
SamYaplezigo: awesome. i tried to make that as simple as possible08:01
zigoI haven't seen them "documented" in the setup.cfg.08:01
asalkeldzigo: no deamons08:01
zigoAh? :)08:01
*** inc0 has joined #kolla08:01
zigoSo, just python code and some config files?08:01
asalkeldmostly dockerfiles and ansible08:02
zigoHum...08:02
asalkeldwould you normally package ansible stuff?08:02
zigoI haven't done so yet, but anything can be packaged.08:02
asalkeldk08:03
zigoI guess it'd go in /usr/share/kolla/ansible, no?08:03
asalkeldi'd guess08:03
asalkeldzigo: i am truely not familar with packaging of ansible and docker files08:04
zigoDoes the setup.py install everything as it should already?08:04
inc0good morning08:04
asalkeldhi inc008:04
zigoThe data_files seems to say so.08:04
*** shardy has joined #kolla08:04
sdake_zigo yes we think so08:04
zigo(within setup.cfg, I mean)08:04
zigoOk, easy enough then.08:05
kjellyinc0: morning, inc008:05
zigoI'll try.08:05
sdake_zigo note we kind of guessed on the packaging but i think its correct08:05
zigoWould it make sense to have just 2 packages, one called python-kolla, and one called just kolla then?08:05
asalkeldi don't see an include of anisble08:05
asalkeldin manifest.ini08:05
sdake_asalkeld that is handled in setup.cfg08:05
asalkeldok, cool08:06
inc0we includes ansible in ansible modules at least08:06
zigoasalkeld: There's data_files = share/kolla/ansible = ansible/* in the setup.cfg, that's enough.08:07
sdake_zigo persoanlly i woul go with one package but if python needs to be an a separate packagey ou can ddo it that way08:07
inc0anyway, I'm out of context, what seems to be an issue?08:07
zigosdake_: It does, because we need to allow later move from python2 (ie: python-kolla) to python3 (ie: python3-kolla)08:07
asalkeldinc0: zigo is doing debian packaging of kolla08:07
SamYaplehello inc008:08
inc0you are aware that ansible has slightly different licence08:08
SamYaplehow is the life?08:08
inc0got my visa;)08:08
inc0in 2 weeks from now I'll be in US08:08
SamYaplew00t w00t08:08
inc0less than 2 week really08:08
inc0that means I'll be pretty inactive in following days, for which I'm terribly sorry08:09
inc0moving across ocean will consume all my attention08:09
SamYaplei have multinode gating stuff in the works :D08:09
inc0cool08:09
asalkeldguys could you imagine that *if* we split kolla-anisble, that the config templates would go into a common place?08:10
asalkeldi want to rely on those08:10
*** dwalsh has quit IRC08:10
SamYapleasalkeld: the jinja2 templates in the ansible/*/templates folders?08:10
inc0what kind of config do you refer to? I don't recall any overlap08:10
asalkeldyip08:10
inc0ah08:10
inc0I would expect it will...08:11
asalkeldso i am driving the jinja for those files08:11
SamYaplehmm im not sure how that would work08:11
SamYapleim sure we can get it to work out though08:11
asalkeld(partially redoing some of wat ansible does)08:11
SamYaplethats to be expected08:11
inc0we can symlink stuff...08:11
asalkeldseems a waist to totlly copy that stuff08:11
SamYapleinc0: >:(08:11
*** egonzalez__ has joined #kolla08:12
inc0just saying08:12
*** egonzalez__ has quit IRC08:12
*** egonzalez90 has joined #kolla08:12
asalkeldSamYaple: how is "hostvars" genterated by ansible? do you know?08:12
SamYapleasalkeld: ansible has a "setup" module that fetchs all that stuff08:12
inc0asalkeld, facts?08:12
SamYaplestrictly and ansible thing08:13
SamYapleasalkeld: i think the real trouble will come into is that if we split off kolla-ansible then people deploying with kolla-ansible wont be cloning kolla unless they are building images08:13
asalkeldSamYaple: is it runtime calcultated?08:13
SamYapleasalkeld: most of the time08:13
SamYaplethough ansible supports fact caching and database facts08:13
SamYapleall outside of kollas control08:13
inc0when you run ansible-playbook you'll see "gathering facts" step08:13
asalkeldok, i need to convert that into zookeeper variables i think08:13
inc0that's the place08:13
SamYapleinc0: only if not doing fact caching, its not a requirement08:14
inc0that's general idea08:14
asalkeldok, that's enough for the moment - thanks guys08:14
SamYapleasalkeld: ill have to see an implementation. my 5 minute brainstrom says it wont work out08:15
SamYaplebubt im not opposed08:15
*** openstackgerrit has quit IRC08:16
*** openstackgerrit has joined #kolla08:16
*** akwasnie has joined #kolla08:17
asalkeld:-(08:17
*** egonzalez90 has quit IRC08:18
*** egonzalez90 has joined #kolla08:18
openstackgerritSam Yaple proposed openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24149908:18
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150008:18
kjellyIf you use ubuntu 15.10, it seems that you can't downgrade the docker version to 1.8.208:20
SamYaplekjelly: whats the reason?08:20
SamYapleno that cant be right08:20
SamYapleim running 1.8.2 on 15.10 right now08:20
kjellyI use the command to list version, "sudo apt-cache policy docker-engine".  I saw "1.9.0-0~wily 0" only.08:20
SamYapleyou can still pin08:21
SamYapletry uninstalling docker entirely then `apt-get install docker-engine=1.8.208:21
kjellySamYaple: Ok, try it.08:22
SamYaplekjelly: http://apt.dockerproject.org/repo/dists/ubuntu-wily/main/binary-amd64/Packages08:22
SamYapleits there08:22
SamYaple`apt-get intall docker-engine=1.8.2-0~wily`08:22
SamYaplefyi policy only shows the installed version and most recent version available per repo08:23
kjellySamYaple: E: Version '1.8.2-0~wily' for 'docker-engine' was not found08:23
SamYaplekjelly: did you add the docker repo?08:25
*** achanda has quit IRC08:25
kjellySamYaple: I rerun the command, 'curl -sSL https://get.docker.io | bash'. Now it found the package.08:26
SamYaplewierd08:26
SamYapleglad its wokring08:27
kjellySamYaple: ok, it works.08:27
SamYaple$ cat /etc/apt/preferences.d/docker08:27
SamYaplePackage: docker-engine08:27
SamYaplePin: version 1.8.2*08:27
SamYaplePin-Priority: 55008:27
SamYaplefyi kjelly ^^ that makes the pi permanent08:27
*** mfalatic has quit IRC08:27
SamYaplepin*08:27
*** mfalatic has joined #kolla08:28
kjellySamYaple: Ok, does this should be document ? Or I just write it in my note?08:29
SamYapleif you feel its important enough to document, go for it08:30
zigoWhat's the point of empty config files in etc/kolla ?08:30
SamYaplezigo: they allow overriding of any of the openstack conf file values08:30
zigoSamYaple: So, anything that an operator would put there would override what Kolla does automatically?08:31
SamYaplein relation to the config files (glance-api.conf, nova.conf, etc) yes08:31
SamYapleit actually merges, not strictly overrides08:32
SamYapleanything in /etc/kolla gets preference, but you only need ot put in your override values08:32
SamYaple[libvirt]08:32
SamYaplevirt_type=qemu08:32
*** openstack has joined #kolla08:35
*** egonzalez90 is now known as egonzalez08:39
openstackgerritSam Yaple proposed openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24149908:42
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150008:42
openstackgerritMerged openstack/kolla: Remove unused setup_fedora.sh  https://review.openstack.org/24149808:45
*** egonzalez is now known as egonzalez9008:51
*** egonzalez90 is now known as Guest8928108:51
*** Guest89281 is now known as egonzalez08:52
*** sdake_ has quit IRC08:53
*** egonzalez is now known as egonzalez9008:57
*** dolpher has quit IRC08:57
*** egonzalez90 is now known as egonzalez08:58
openstackgerritSam Yaple proposed openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24149909:02
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150009:02
openstackgerritSam Yaple proposed openstack/kolla: Add additional ntp information around ceph  https://review.openstack.org/23675809:11
openstackgerritSam Yaple proposed openstack/kolla: Add additional ntp information around ceph  https://review.openstack.org/23675809:14
openstackgerritMerged openstack/kolla: Tidy up quickstart.rst  https://review.openstack.org/24192709:14
openstackgerritSam Yaple proposed openstack/kolla: Add additional ntp information around ceph  https://review.openstack.org/23675809:17
openstackgerritMerged openstack/kolla: Adjust package name for Ubuntu Ironic  https://review.openstack.org/23864009:20
inc0SamYaple, is there such thing as 2 space after period?;) never seen it09:21
SamYapleinc0: its an older thing and he is quite consistent about it09:21
inc0ok, seems legit09:22
inc0for obvious reasons I will not argue about proper english stuff09:22
*** mbound has joined #kolla09:22
SamYaplemost of that quickstart doc is actually two spaced09:22
*** jmccarthy has joined #kolla09:23
*** gfidente has joined #kolla09:24
*** diga has joined #kolla09:28
*** sdake has joined #kolla09:34
*** tzn has quit IRC09:40
openstackgerritMerged openstack/kolla: Fix RHEL builds  https://review.openstack.org/24146209:42
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150009:49
openstackgerritMerged openstack/kolla: Adjust gate btrfs setup  https://review.openstack.org/24146509:49
*** inc0_ has joined #kolla09:53
*** inc0 has quit IRC09:56
openstackgerritSam Yaple proposed openstack/kolla: Add additional ntp information around ceph  https://review.openstack.org/23675810:01
*** inc0_ has quit IRC10:06
*** inc0 has joined #kolla10:07
*** inc0 has quit IRC10:17
*** inc0 has joined #kolla10:17
*** tfukushima has quit IRC10:23
*** egonzalez has quit IRC10:27
*** inc0 has quit IRC10:40
*** tzn has joined #kolla10:40
*** sdake has quit IRC10:41
*** dwalsh has joined #kolla10:41
openstackgerritPaul Bourke proposed openstack/kolla: Clean up keystone httpd pid files for RPM distros  https://review.openstack.org/24198010:41
*** pbourke has quit IRC10:43
*** tzn has quit IRC10:46
*** inc0 has joined #kolla10:51
*** inc0 has quit IRC10:53
*** inc0 has joined #kolla10:54
SamYapleyay!10:55
SamYaplethe multinode gate found a bug!10:55
SamYaplea rather large one actually /10:55
SamYaple:/10:55
*** dwalsh has quit IRC10:59
*** inc0 has quit IRC11:00
openstackgerritSam Yaple proposed openstack/kolla: Add Docker in Docker container  https://review.openstack.org/24144411:02
openstackgerritSam Yaple proposed openstack/kolla: Allow disabling of sysctl values  https://review.openstack.org/24185611:02
openstackgerritSam Yaple proposed openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24149911:02
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150011:02
openstackgerritSam Yaple proposed openstack/kolla: bootstrap takes place on the wrong node  https://review.openstack.org/24198911:02
*** pbourke has joined #kolla11:15
openstackgerritSam Yaple proposed openstack/kolla: bootstrap takes place on the wrong node  https://review.openstack.org/24198911:24
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150011:24
*** dolpher has joined #kolla11:31
*** dolpher has quit IRC11:35
*** dwalsh has joined #kolla11:36
*** inc0 has joined #kolla11:44
*** inc0_ has joined #kolla11:46
*** dwalsh has quit IRC11:46
pbourkeSamYaple: so this is merged to ansible now? https://github.com/ansible/ansible-modules-core/pull/2258#event-43534300811:47
pbourkeSamYaple: can we bump our pin in kolla-ansible to remove our docker 1.8.3 restriction?11:47
SamYaplepbourke: nope11:48
SamYapleits merged into 2.0 ansible11:48
inc0_when do we move to 2.0?11:48
SamYapleits also in stable/1.9 _however_ 1.9.5 will not be tagged11:48
SamYapleso 1.9.x will never be able to go past 1.8.211:48
SamYapleinc0_: when its ever released11:48
*** inc0 has quit IRC11:49
*** ssurana has joined #kolla11:55
openstackgerritMerged openstack/kolla: Add Docker in Docker container  https://review.openstack.org/24144411:57
*** achanda has joined #kolla12:03
*** achanda has quit IRC12:05
*** akwasnie has quit IRC12:10
*** sdake has joined #kolla12:10
sdakemorning12:10
SamYaplemorning sdake12:11
SamYaplemy multinode gate caught something!12:11
sdakedamn sam you have been up for 2 days ;)12:11
SamYapledid you hear me?!12:11
SamYapleit caught somethign!12:11
sdakewhat ?12:11
SamYaplebroken bootstrap process when compute is seperate from nova-api12:11
sdakeand that is pretty cool ;)12:11
SamYaple_and_ the name of the compute host is lower in alphabetical order than that of the host running nova-api12:12
SamYapleyou know sdake... we could techinically extend this multinode to be seperate services per each node like an ultra explodd out environment12:14
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Make the database json variable more readable  https://review.openstack.org/24202412:16
openstackgerritMerged openstack/kolla: Restrucutre gate scripts  https://review.openstack.org/24149912:22
*** dims_ has joined #kolla12:22
*** athomas has joined #kolla12:24
*** athomas has quit IRC12:25
*** athomas has joined #kolla12:25
*** dims has quit IRC12:25
*** athomas has quit IRC12:27
*** athomas has joined #kolla12:27
*** tzn has joined #kolla12:43
zigoWhat've just read on http://docs.openstack.org/developer/kolla/quickstart.html horifies me...12:44
zigocurl -sSL https://get.docker.io | bash12:44
SamYaplethat is the official docker way of doing it12:45
SamYapledont look at us12:45
zigoAm I the only one believing this is unbelievably unsafe?12:45
SamYaplei never do it that way12:45
zigoWell, IMO, we shouldn't advise for that.12:45
SamYaplethat is the official stated docker way12:45
SamYaplewhen docker changes things, thats how they get changed12:45
zigo"official" as in, from upstream?12:45
SamYapleyes12:45
zigoUpstream isn't always right... :P12:46
zigoAnyway, is there a minimum requirement for Docker's version?12:46
zigoShould Kolla depend on Docker by the way?12:46
SamYapleall the docker info should be in that guide12:46
SamYaplemin and max12:47
zigo"Kolla release 1.0.0-liberty tested to work with docker 1.8.2"12:47
zigoThat's all it says ...12:47
*** tzn has quit IRC12:48
SamYaplenope12:48
zigo(appart that 1.8.3 and later wont work)12:48
zigoIn other words: I haven't seen a minimum required version.12:48
SamYapleit says more12:48
zigoOk.12:48
SamYapleyea there is a table right there12:48
SamYaple1.8.212:48
zigoAh... so in fact, Kolla works *only* with 1.8.2?12:49
SamYaplethat doesnt sound right, but thats what it says12:49
SamYaplei thought it worked with 1.6.0 and up to 1.8.212:49
SamYaplebut whoever updated the docs may have newer info12:49
zigoDebian has only 1.512:50
zigo:(12:50
SamYaplei know12:50
SamYapleand as time moves on that version will only get higher and higher as a req for kolla12:50
zigoAh no...12:50
zigodoker != docker.io ! :)12:50
SamYaplefor instance when 1.10 comes out we will be using that12:50
SamYapledocker.io was ubuntus name12:50
zigoJessie has 1.6.2, and Sid has 1.8.3.12:51
SamYapleare you sure its docker.io in debian?12:51
SamYapleill trust you since those seem like legit versions. it should work with 1.6.212:51
zigoWell, it looks like docker.io in Ubuntu derives from the package in Debian.12:51
SamYaplesdake: comments ^12:51
zigoWell, I can only upload to Sid, so I need kolla to work with 1.8.3, unfortunately.12:52
zigoWhat's the blocker?12:52
SamYapleansible12:52
zigoAh?12:52
zigoCould you expand? :)12:52
SamYaplefor the record though zigo, when Mitaka comes out it will have a req for at _least_ 1.10 docker12:53
SamYapleansible modules that are not in our control are broken12:53
zigoWho's working on these Ansible modules then?12:53
sdakeansible 1.9.z is abandoned12:53
SamYaplei already patched them12:53
SamYaplebut they wont tag another 1.9 ansible12:53
sdakethey are a 50 person company - can only focus on supporting 1 release not two12:54
SamYaplebarely even one....12:54
SamYaplehey sdake can you validate the version numbers in the quickstart guide?12:55
SamYaplethey seem off12:55
sdake1.8.2 is the max12:55
sdakenot sure on the min12:55
SamYaplei dont think the minimum has changed from 1.6.012:56
sdakeatlaewsat 1.6.012:56
SamYaplebut i dont know12:56
SamYapleyea12:56
sdake1.5.1 is broken12:56
sdake1.5.0 is also broken12:56
inc0_I don't suppose Debian will bump up version of docker when 1.10 releases12:57
inc0_and we'll want to use it right away12:57
inc0_(thin containers)12:57
SamYapleinc0_: thats only for mitaka (master)12:57
SamYaplenot liberty12:57
inc0_yeah12:58
SamYaplei dont think debian has the version of ansible needed either in packaged form12:58
inc0_by the way, can you force version of docker using this curl install thingy?12:58
inc0_if so, we should have it in our docs12:58
SamYaplethis is why ubuntu has to maintain cloud-archive so they can support openstack12:58
SamYapleinc0_: you can package pin ahead of time12:58
openstackgerritSam Yaple proposed openstack/kolla: Make the database json variable more readable  https://review.openstack.org/24202412:59
*** tobe has quit IRC13:03
openstackgerritSam Yaple proposed openstack/kolla: Make the database json variable more readable  https://review.openstack.org/24202413:03
*** sdake_ has joined #kolla13:04
*** akwasnie has joined #kolla13:06
*** dwalsh has joined #kolla13:07
*** sdake has quit IRC13:08
*** tobe has joined #kolla13:10
*** tobe has quit IRC13:14
*** tobe has joined #kolla13:14
*** tobe has quit IRC13:19
*** tobe has joined #kolla13:20
*** dwalsh has quit IRC13:21
*** sdake_ has quit IRC13:21
*** tobe has quit IRC13:25
*** sdake has joined #kolla13:27
sdakepbourke ping13:30
openstackgerritMerged openstack/kolla: Clean up keystone httpd pid files for RPM distros  https://review.openstack.org/24198013:33
openstackgerritSam Yaple proposed openstack/kolla: Make the database json variable more readable  https://review.openstack.org/24202413:33
*** diga has quit IRC13:33
*** diga has joined #kolla13:33
*** diga has quit IRC13:34
SamYaplesdake: so for the gate, how do you want to play this13:35
SamYaplei dont think its reasonable to get 3 more gates for multinode13:35
*** diga has joined #kolla13:35
sdakei prefer that13:35
SamYaplei know13:35
sdakebreaking stuff apart is easier to debug13:35
SamYaplethats what i made my coment13:35
SamYaplei dont think its reasonable to get 3 more gates for multinode13:36
SamYaplei would prefer more gates for other things13:36
SamYaplelike excersiing different options13:36
sdakethere are bo lmits on gates we can haved13:36
sdakebo/no13:36
SamYaplei am aware. my comment remains13:36
SamYaplei dont want to fight for gates13:37
sdakenova has like 60 gates13:37
SamYaplethey do not13:37
*** rhallisey has joined #kolla13:37
SamYaplethey have like 10 and then a bunch of thrid party ci13:37
SamYaplethird party has that equipment13:37
SamYapleand we are not nova13:37
sdakei see13:37
sdakewell infra doesn't mind if we have many gates, already had that discussion13:37
sdakethey care about run length not numberof gates13:38
sdakewhat is the objection to mor egates?13:38
sdakethe / your13:38
SamYapleok then I want our AIO gates to have diferent options than our multinode gates13:38
SamYaplei want to test more options13:38
SamYapleand dont want 2000 gates13:38
SamYaplefor every single permutation13:38
*** inc0_ has quit IRC13:39
sdake2k gtates is probably not goin gto fly ;)13:39
sdakehow many permutations do you want?13:39
sdakei'd be happy with basic stuff ;)13:39
SamYaplenow we have spice vs novnc, linuxbridge vs openvswitch, etc. And that is just going to grow13:39
SamYaplei want the main ones13:41
SamYapleso if we can switch up AIO and multinode options I am ok with seperate gates13:41
SamYaplebtw I am working on the logging situation13:42
SamYaplesoon we will have a logs folder and not one massive console.html13:42
SamYapleso diving into the gate will be 1000% easier13:42
SamYapleincluding actual service log output13:42
sdakemy motive for multiple gates isn't that its easy to debug its that the dashboard turns +1 / -1 eaisly13:43
sdakei expect debugging a failure will always be painful13:43
SamYaplethats why i said 'btw'13:43
sdakeroger13:43
*** tzn has joined #kolla13:44
sdakei dont understand whaty ou mean by mixing up options13:44
sdakeyou mean having aio do spice and multinode do novnc?13:44
SamYapleAIO has spice and linuxbridge as options, multinode has novnc and ovs13:44
SamYaplesomething liek that13:44
sdakeya that wfm but i dont htink we can gate linuxbridge13:45
sdakei thikn we detemrined its not possible13:45
SamYaplewhys that?13:45
SamYapleno it should be completely possible13:45
sdakenot a networking rocket scientist13:45
sdakebut you said long ago ovs was the only one we could get working13:45
SamYapleright before I was doing what im doing13:45
SamYapleshould be good now13:45
sdakei tried long and hard to get linuxbridge to get one interface going13:45
*** akwasnie has quit IRC13:45
sdakeand failed13:45
SamYapleyea we should be solid now13:46
sdakesepnt couple weeks on it atleast13:46
sdakewell gotta say that is fantastic news ;)13:46
SamYaplei think im killing the node resources :(13:46
SamYapleit gets to horizon and breaks down13:46
sdakeya your ooming13:46
sdakethis is why it hink dnd isn't viable13:46
SamYaplehttps://jenkins02.openstack.org/job/gate-kolla-deploy-images-ubuntu-source/32/console13:46
sdakewe can't get more ram in teh vms unfortunately13:46
SamYapleits no worse than anything else13:46
SamYaplewe can set the affinity for the processes13:47
SamYapleso only 2 copies of this, but 3 of that13:47
SamYapleetc13:47
SamYaplei thikn we can work it out13:47
sdakei'd like multinode to do multiple vms but ymmv :)13:47
SamYaplemultiple 128mb ciros images... sure!13:48
sdakemainy to work arund the ram problem which is unsolveable13:48
*** tzn has quit IRC13:48
SamYaplewell 'multinode' upstream is 2 vms right now13:48
SamYapleno more13:48
SamYaplethats not good enough13:48
sdakethey said they woudl give us as many as we need (within reason)13:48
sdakeafter we got 2 owrking we could geet 3 after 3 working could get 4 etc13:49
SamYapleright. multinode upstream only works as 2 vms right now13:49
SamYapleunless something has changed...13:49
sdakeinfra folks confirmed they would give us many nodes if we incrementally prove our gates are reliable as we add nodes13:49
sdakewe talked about 5 as a cap13:49
sdakei think we dcould get more like 10 or so13:49
sdakethe reason they have 2 as a cap now is nobody can get 2 vms to successfully gate correctly13:50
sdakenova basicallyl gaveu pon multinode gating13:50
SamYaplewell good luck with that13:50
sdakefrom my understanding13:50
sdakeso infra hasn't added more nodes13:50
SamYaplewe wont be able to gate 2-3-4 nodes due to the registry problem13:51
SamYaplea push in the gate even to a local registry will take ages13:51
sdakev1 registyr shouldn't be too bad13:51
SamYapleyes it will13:52
sdakeok well another option is to build on each node13:52
sdake?13:52
SamYapleyoure joking right?13:52
sdakejust brainstorming13:52
SamYapleno were not doing that. thats teh exact oposite of kolla13:52
SamYaplei can trim the fat in the gate right now13:53
sdakemaybe we can squeeeze out a 16gb vm from infra13:53
SamYapleif i can prove multinode on a single node... maybe?13:53
sdakeyup13:53
sdakelets go ask and see if its possible13:53
SamYaplei mean it is from an L2 standpoint13:53
SamYaplecompletely13:53
sdakecrickets probably too early13:54
SamYapleyea good luck with that13:55
sdakegood luck with which?13:55
SamYapleim going to trim the gate fat right now and prove that multinode works13:55
sdakeI have never asked for a 16gb vm13:55
sdakeSamYaple join #openstack-infra13:56
*** unicell1 has joined #kolla14:02
*** unicell has quit IRC14:03
*** cemmason1 has joined #kolla14:03
v1k0d3nmorning SamYaple (at least where i'm at)14:05
*** cemmason has quit IRC14:05
v1k0d3ni cleaned out my work from yesterdays issue...and i am still having problems. is there something i'm missing in the docs?14:05
v1k0d3ncheck login credentials (login_user, and login_password, which can be defined in ~/.my.cnf), check that mysql socket exists and mysql server is running14:05
v1k0d3nthat's the error  ^^14:05
v1k0d3ni looked at /etc/my.cnf which includes /etc/my.cnf.d and points to mysql-clients and client.cnf14:07
v1k0d3nbut not a lot in there...are there supposed to be changes or something added that i'm overlooking?14:08
*** diogogmt has quit IRC14:13
*** ssurana1 has joined #kolla14:14
*** ssurana has quit IRC14:15
*** cemmason1 has quit IRC14:15
*** ssurana1 has quit IRC14:15
*** diogogmt has joined #kolla14:16
SamYaplesdake: heat is the eater of ram14:21
SamYaplesdake: goodbye heat!14:21
sdakeimo add swap ;)14:24
sdakei'd like to gate the big tent rather then copute kit14:24
SamYapleits just heat...14:24
SamYaplebut well see14:24
SamYaplei want time for tests. with swap i have no time to actually test anything14:25
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150014:25
*** absubram has joined #kolla14:26
SamYaplesdake i think i can trim most services down to two nodes, only quorum ones requiring three nodes14:29
SamYapledrop horizon entirely from multinode14:29
SamYaplethen well be better14:29
sdakerecommend atleast testing more swap to see if that really has a bad performance ipact on testing14:30
SamYapleit does :) the system already has swap14:30
SamYaplethe gate doesnt crash, it slows to a crawl and times out14:30
SamYapleits swapping14:30
sdakei see14:31
sdakewell i guess big tent testing will be aio then14:31
*** diogogmt has quit IRC14:31
SamYaplemaybe14:31
SamYaplewe are probably at the limit there as well14:31
SamYapledont know how much we can test14:31
sdakewhat is problem with building on each node with multivm?14:32
SamYaplethey arent the same images and could have different packages and libraries?14:33
SamYapleaka everything Kolla is not14:33
SamYapleim -2 on that idea forever14:33
sdakewhy do you think v1 registry will be longer then 90 mins?14:34
sdakemy v1 registry takes 15 mins on my gear14:34
SamYaplei dont.14:34
SamYaplethe build will take 20-30 minutes. the push AT BEST takes 15 minutes14:34
SamYaplethen we have to deploy14:34
SamYaplehow can we test?14:34
SamYaplebtw we only have 60 minute gates14:35
SamYaplebut even at 90 we would time out14:35
sdakewe can definatelly get 90 minute gates14:35
SamYaplei know14:35
sdakemaybe able to get more time14:35
sdake120 min seem to be the max people will tolerate in infra ;)14:36
sdakeit sucks computing is all about constraints ;(14:37
SamYapleyoure welcome to attempt multinode stuff14:37
SamYaplei will not be. this dind stuff is going to excrise more code paths than what you suggest14:38
SamYapleit already caught a problem multinode vms would not have because it can simulate many more nodes14:38
*** dims has joined #kolla14:40
sdakewell gating compute kit is a good first start14:41
sdakebetter then 014:41
*** guruMeditator has quit IRC14:41
SamYaplenot gating heat is not the same thigns as just compute kit14:41
*** slotti has quit IRC14:42
*** sdake has quit IRC14:42
*** diga has quit IRC14:43
*** dims_ has quit IRC14:43
*** athomas has quit IRC14:43
*** egonzalez has joined #kolla14:44
*** kjelly has quit IRC14:44
*** athomas has joined #kolla14:45
*** tzn has joined #kolla14:45
*** sdake has joined #kolla14:47
*** tzn has quit IRC14:49
*** dims has quit IRC14:53
*** guruMeditator has joined #kolla14:56
guruMeditatorGreetings. I am attempting to deploy a Compute node with Neutron that has multiple provider networks. Each provider network is on a different interface. I see that the Neutron external interface is defined with "neutron_external_interface". Is it possible to accomplish this? Currently I have been manually changing the bridges to each neutron node14:56
*** cemason has joined #kolla14:56
*** sdake has quit IRC14:57
*** cloudnautique has quit IRC15:07
*** athomas has quit IRC15:09
openstackgerritPaul Bourke proposed openstack/kolla: Clean up keystone httpd pid files for RPM distros  https://review.openstack.org/24210615:09
rhallisey SamYaple ^^ are you able to answer the above question? Because idk the answer :)15:11
pbourkeguruMeditator: rhallisey: SamYaple: I wonder could you override neutron_external_interface per node in the inventory?15:12
pbourkewould that solve it?15:12
guruMeditatorThat would allow a single neutron node to map to disparate interfaces, but, I don't think it would allow multiple provider networks to map to different interfaces15:14
SamYapleguruMeditator: not in an automated fashion with the current playbooks15:18
SamYapleits expected to have one provider interface15:18
SamYaplehowever you could override the appropriate config options and manually attach them to openvswitch15:18
SamYaplebtw guys15:18
SamYaplemultinode gate is a goooooo!15:18
guruMeditatorYes, that is what I have been doing (manually adding the bridges via OVS on all neutron nodes). Is this the best approach? Are there plans to add multiple provider networks mapping in the future?15:20
SamYapleguruMeditator: i would love to see it, but i will not have the cycles to add that support any time soon15:20
SamYaplethat is my suggested approach for you15:21
SamYaplesee if you can add the feature to kolla yourself or bug someone to do it for you :)15:21
guruMeditatorOk, thanks SamYaple. Appreciate the suggestions.15:21
SamYaplelol 230mb memory free in the gate after multinode with no heat or horizon15:23
SamYaplehalarious15:23
*** blahRus has joined #kolla15:27
v1k0d3nare you guys aware of the broken ansible link in the main github README.md?15:28
v1k0d3nhttps://github.com/openstack/kolla/blob/master/doc/ansible-deployment.rst15:28
SamYaplev1k0d3n: probably not. feel free to correct it15:28
SamYapledoes that dox still get published?15:28
SamYaplev1k0d3n: oh wait those links only work at docs.openstack.org15:29
v1k0d3nit looks like it's gone....or maybe that needs to point to: https://github.com/openstack/kolla/blob/master/doc/quickstart.rst?15:29
SamYaplethey dont work in the get repo15:29
v1k0d3nah!15:30
v1k0d3nso you're saying they should point to: http://docs.openstack.org/developer/kolla/dev-quickstart.html15:30
*** sdake has joined #kolla15:30
*** sdake has quit IRC15:31
v1k0d3nthe title Quickstart to Deploying OpenStack using Ansible15:31
SamYaplev1k0d3n: no im saying that they will _work_ at docs.openstack.org15:31
SamYaplethey dont work in teh git repo15:31
SamYaplei dont really understand the docs publishing or linkages to be honest15:31
*** sdake has joined #kolla15:31
SamYapleKennan may be able to explain it better v1k0d3n15:31
v1k0d3noh, i think that <https://github.com/openstack/kolla/blob/master/doc/ansible-deployment.rst> should probably read like [Title](https://www.likehere.com) for markdown.15:33
v1k0d3ni'm probably missing how the links are being used though.15:33
SamYapleits not markdown though15:33
SamYapleits for the sphinx docs15:33
v1k0d3nyeah, rst files i noticed.15:33
*** sdake has quit IRC15:35
*** sdake has joined #kolla15:36
openstackgerritDavid Moreau Simard proposed openstack/kolla: Update kollaglue docker repository URL  https://review.openstack.org/24211815:36
*** diogogmt has joined #kolla15:37
*** cloudnautique has joined #kolla15:38
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150015:38
openstackgerritSam Yaple proposed openstack/kolla: Remove vip for rabbitmq  https://review.openstack.org/23577715:43
*** tzn has joined #kolla15:45
*** diogogmt has quit IRC15:49
*** diogogmt has joined #kolla15:49
*** tzn has quit IRC15:50
*** inc0 has joined #kolla15:58
pbourkesdake: just saw ping16:01
sdakecan you backport one of your recent changes for httpd.pid16:01
pbourkeyup should be already done16:01
pbourkehttps://review.openstack.org/#/c/242106/16:01
diogogmti’m starting to get this errors when reloading the vagrant vm: http://pastebin.com/PAxgJAGv16:10
sdakethanks16:11
diogogmti see a comment in the Vagrantfile pointing to this thread: http://askubuntu.com/questions/317338/how-can-i-increase-disk-size-on-a-vagrant-vm16:13
diogogmtwhat if instead we increased the size of the base image being used?16:13
diogogmtit looks like the base image being used is puppetlabs/centos-7.0-64-puppet, however, i don’t see any scripts to provision the vm with docker, ansible, python and all the other dependencies — how is the provision is done on the vagrant vm>16:14
diogogmt?16:14
diogogmtcan i use a different base image instead?16:15
pbourkediogogmt: it should have enough space out of the box16:16
diogogmtit only has a 20GB disk16:16
pbourkediogogmt: what steps have you taken so far16:16
pbourkediogogmt: yes but it maps /var/lib/docker to the host16:16
pbourkediogogmt: the provisioning is done using bootstrap.sh16:16
pbourkeunder kolla/dev/vagrant16:16
diogogmtpbourke: ok, i’ll take a look at the script, thanks16:17
*** kproskurin has joined #kolla16:17
diogogmtregarding the /var/lib/docker, i’m running on a mac, are you saying that all the docker images are being stored under /var/lib/docker ?16:17
diogogmton my mac instead of the vm?16:17
SamYaplesdake: i had a successful deploy16:19
SamYapleno heat or horizon16:19
SamYaple230MB left of ram16:19
sdakeyou can't run kolla on a mac16:20
pbourkesdake: no he's using a vm on his mac16:20
sdakeunless you use vagrant16:20
pbourkeyes16:20
sdakeoh vms will work16:20
sdakepbourke nice summary of the sessions :)16:21
pbourkediogogmt: checking again, only the registry storage goes to the host16:21
sdake+2: )16:21
pbourkesdake: cheers16:21
pbourkediogogmt: so maybe the disk is too small to build everything :/16:21
SamYaplesdake: thought of how we can skirt the docker registry on multinode16:22
SamYapletrue multinode16:22
openstackgerritMerged openstack/kolla: Clean up keystone httpd pid files for RPM distros  https://review.openstack.org/24210616:22
diogogmtpbourke: i had the same issue before with some of my docker hosts running with vagrant, i had to manually increase the size of the box by converting the box back to vdi and then using lvm to increase the disk16:23
pbourkecan anyone recommend dirt cheap public cloud that could work for kolla dev?16:23
diogogmtpbourke: do you see any issues if i use a different base image instead of puppetlabs/centos ?16:23
sdakerax woild work but it is fairly pricy i think16:23
pbourkediogogmt: yeah resizing can be a pain though :( I think another should work16:23
pbourkediogogmt: if you find a more suitable one let us know maybe we can update the vagrantfile16:24
sdakepbourke is a "top lad" a good thing?16:25
pbourkediogogmt: generally I dont build everything... you can use --profile default when building16:26
pbourkesdake: I think so?!16:26
pbourkediogogmt: that way you save resources but get everything you need for a working openstack16:26
diogogmtpbourke: i’ll check if the puppetlabs base image has lvm, if it does i might give it a try to resize it — do you have any concerns if we use ubuntu instead of centos for the vagrant base image?16:27
diogogmtpbourke: what do you mena by building everything? if we don’t build locally does kolla uses public images for some of the services?16:28
*** daneyon has joined #kolla16:28
openstackgerritMerged openstack/kolla: Update kollaglue docker repository URL  https://review.openstack.org/24211816:29
openstackgerritSam Yaple proposed openstack/kolla: Allow disabling of sysctl values  https://review.openstack.org/24185616:30
openstackgerritSam Yaple proposed openstack/kolla: bootstrap takes place on the wrong node  https://review.openstack.org/24198916:30
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Deploy multinode in gate  https://review.openstack.org/24150016:30
SamYaplethats two bugs for my multinode gate16:31
*** daneyon_ has joined #kolla16:32
pbourkediogogmt: oh sorry I thought you were building your own images16:33
pbourkediogogmt: so you're running out of space just from a deploy?16:33
diogogmtpbourke: no, i got the out of space problem while building the images16:34
sdake20gb is not enough storage16:34
sdake40gb16:34
sdakeis what you need16:34
diogogmti’ve been following the instructions from the wiki page16:34
pbourkeyeah I thought the image we were using was 4016:34
sdakewhich wiki page?16:34
*** tzn has joined #kolla16:34
sdakewe dont have wiki docs that I know of16:34
diogogmthttps://github.com/openstack/kolla/blob/master/doc/vagrant-dev-env.rst16:34
sdakedocs.openstack.org/developer/kolla16:34
diogogmtso first i was building the images: kolla-build, and later planning to deploy them16:35
diogogmtbut i hit the out of space error before i got to the deploy part16:35
*** daneyon has quit IRC16:35
pbourkeIf we mounted /var/lib/docker to the host that should fix things I think16:36
diogogmtsdake: pbourke: looks like the puppetlabs/cento7 vm is using lvm to manage the partions, i guess i’ll just increase the size of the base image manually then16:39
diogogmtlater i can push the new resized box to https://atlas.hashicorp.com/scalefactory/boxes/16:39
diogogmtdo you think 40GB is enough?16:40
diogogmtor should i resize to 60GB to avoid future problems?16:40
pbourkego with 60GB sure16:42
pbourkeIm wondering would it good if you supplied your instructions for resizing to the docs rather than pushing the box16:43
pbourke(though you're free to push the box also of course :))16:44
pbourkepeople may be more comfortable using the puppetlabs maintained one and resizing themselves rather than a user submitted one16:44
diogogmtpbourke: makes sense, i’ll do it both ways16:48
diogogmtquick question regarding the branches, is master considered stable or is it better to go with liberty?16:49
*** odyssey4me_ has joined #kolla16:51
pbourkediogogmt: liberty would be more stable16:52
*** Ti-mo- has joined #kolla16:53
pbourkediogogmt: though master should work most of the time16:53
diogogmtpbourke: ok, i’ll start with liberty just to get a hang and move to master eventually16:55
diogogmtthanks for the pointers16:55
pbourkesounds good16:56
*** sdake_ has joined #kolla16:56
*** Ti-mo- has quit IRC16:57
*** egonzalez has quit IRC16:59
*** barra204 has joined #kolla16:59
*** Ti-mo- has joined #kolla17:00
*** sdake has quit IRC17:04
*** v1k0d3n has quit IRC17:04
*** odyssey4me has quit IRC17:04
*** Ti-mo has quit IRC17:04
*** shakamunyi has quit IRC17:04
*** inc0 has quit IRC17:09
*** achanda has joined #kolla17:26
*** unicell1 has quit IRC17:33
*** kproskurin has quit IRC17:38
*** achanda has quit IRC17:43
*** ssurana has joined #kolla17:44
*** ssurana has quit IRC17:45
*** achanda has joined #kolla17:45
harmwguys, are there reasons to _not_ venture into logging using the docker fluentd-driver?17:46
harmwinstead of syslog17:46
pbourkeonly one I can think of is liberty is out in the wild using rsyslog so we'd need a migration path if we decided it was worth switching17:51
*** ssurana has joined #kolla17:53
harmwtrue, forgot that, offering fluentd as 2nd option sounds more friendly in that case :)17:53
pbourkeoptions are always nice :)17:54
harmwbut perhaps a full efk stack could be added17:54
pbourkeim just reading about it now17:54
harmwin essence, it plain simple17:54
pbourkeis it a fancy way of saying we'll store your logs in json?17:54
harmw:)17:54
harmwplus, it buffers17:55
*** kfox1111_ is now known as kfox111117:55
harmwbtw, that Mesos discussion is awesome17:56
harmwI wonder where we'll be heading17:57
sdake_we are leading with ansible17:57
*** sdake_ is now known as sdake17:57
sdakeand a bare metal underlay17:57
harmwofcourse we are17:57
sdakewe are experimenting with mesos as the underlay and openstack as an overlay17:57
sdakeatm, we have bare metal as an overlay17:57
sdakerather underlay17:58
harmwbut using marathon to manage containers is cool :)17:58
harmwwe're using it at $work17:58
*** mbound has quit IRC17:58
*** unicell has joined #kolla18:09
*** ryansb has joined #kolla18:13
sdakebmace you around18:13
sdakebmace__ around?18:13
bmace__yup18:14
*** achanda has quit IRC18:19
*** tzn has quit IRC18:20
*** tzn has joined #kolla18:24
*** tzn has quit IRC18:28
*** vilobhmm11 has joined #kolla18:34
*** jasonsb has quit IRC18:38
*** achanda has joined #kolla18:41
*** cemason has quit IRC18:45
*** cemason has joined #kolla18:46
*** gfidente has quit IRC18:48
openstackgerritDavid Moreau Simard proposed openstack/kolla: Fix documentation links in the README  https://review.openstack.org/24218818:50
*** cemason has quit IRC18:51
dmsimardbtw <3 kolla, have high hopes for you guys. Good talks in Tokyo too :)18:52
rhalliseyI <3 kolla18:53
rhalliseyt-shirts sdake18:53
rhalliseydmsimard, thanks :)18:53
sdakedmsimard welcome to the party :)18:54
sdakeglad you liked the talk and demo :)18:54
sdakerhallisey still working on stickers ;-)18:54
rhalliseyselinux bug in docker 1.918:56
dmsimardI'm at Red Hat doing upstream RDO stuff and been toying with the idea of using Kolla for some stuff, please let me know if I can help in some way :)18:56
rhallisey:/18:56
*** mfalatic_ has joined #kolla18:56
sdakedmsimard fyi for the future, docs changes do not require bug ids - just to speed thigns up18:56
rhalliseydmsimard, what stuff were you planning on using kolla for.  Curious18:58
*** mfalatic has quit IRC19:00
dmsimardrhallisey: Nothing worth mentioning yet :)19:00
sdakedmsimard rhallisey is one of your coworkers :)19:00
* dmsimard nods19:00
sdakenothign worth mentioning - sounds dangerous :)19:00
sdakedmsimard there is all kinds of areas to help out in19:01
dmsimardBah, I see potential in kolla for doing things that triple-o, packstack or rdo manager do right now but the hamster in my head is still at the spinning stage19:01
rhalliseydmsimard, do you happend to have +2 powers in rdo community?19:01
rhalliseydmsimard, neat19:01
rhalliseydmsimard, tripleo has come container support atm. Added it to the compute node19:02
dmsimardI assisted in some talks around magnum and kubernetes too - apparently there's some murano kubernetes cluster manager app or something too19:03
dmsimardrhallisey: +2 where? In packaging ?19:03
rhalliseydmsimard, ya19:03
dmsimardrhallisey: no but I work closely with people who do :)19:04
rhalliseydmsimard, can you ask someone to +2 https://review.gerrithub.io/#/c/251378/19:04
rhallisey:)19:05
rhalliseyno one responded when I asked earlier19:05
*** tzn has joined #kolla19:05
*** JoseMello has joined #kolla19:05
sdakedmsimard the cool part about kolla vs those other tools yoou mentioned is that it fits right in with rhel atomic19:08
dmsimardrhallisey: no added requirements to the package ?19:08
sdakethose other tools require writeable /usr, where atomic has read only /usr19:08
sdakekolla designed to work with read/only /usr19:09
sdakedmsimard rhallisey can fill you in in more detail, I have limited exposure to this tech19:09
dmsimardsdake: my interest in kolla has a lot to do with the quick and easily reproducible deployments (think containers vs rdo manager that takes >1hr), just for the sake of CI this is *huge*19:10
sdakewell with koll yuo still have to build the images19:10
sdakewhich on my high end gaer takes about 30 minutes19:10
dmsimardOnce the images are built, though19:10
sdakeroger that19:10
dmsimardI also like that you have the possibility to install either from source or from binaries19:11
dmsimardLots of good design decisions :)19:11
ryansbhas anyone gotten kolla going on a remote atomic host?19:11
sdakeyo ryansb19:11
ryansbI'm trying to set the DOCKER_HOST env variable and it's not taking the address19:11
sdakei htink rhallisey had it running at one point but he could add mor edetail19:11
rhalliseyryansb, tripleo deploys onto atomic19:12
ryansbsdake: \o19:12
ryansbrhallisey: so if I want to build images against a remote docker host, how do I specify that?19:12
sdakeryansb what are you working on these days - heat still?19:12
ryansbyup, heat & some director19:12
dmsimardrhallisey: +2'd19:13
rhalliseydmsimard, thanks!19:13
rhalliseyryansb, I don't quite understand your setup. You want a remote host build images?19:14
ryansbrhallisey: so I've got kolla cloned locally, and my docker host is a separate box (baremetal atomic install)19:15
ryansbso what I want to do is have the build script use the equivalent of "docker -H atomic.local:4243 {subcommand}"19:16
rhalliseyOhh interesting I've never done that.  I usually build locally19:16
sdakestackoverflow says19:16
sdakeexport DOCKER_HOST=tcp://192.168.59.103:237519:16
sdakeis that what your doing ryansb ?19:16
sdakenever tried DOCKER_HOST either19:16
sdakebut one thing to keep in mind19:16
sdakeI find it highly unlikekly our build.py processes DOCKER_HOST environment variable19:17
sdakeyo uwill probably need to patch kolla to do such a thing19:17
rhalliseyya that would be a nice addition19:17
ryansbactually you do19:17
ryansbmy problem was that I didn't have the "tcp://" prefix19:18
ryansbdocker -H doesn't require the prefix19:18
sdakegoogle ftw ;-)19:18
rhalliseynice19:18
ryansbyou use kwargs_from_env() https://github.com/openstack/kolla/blob/master/kolla/cmd/build.py#L5719:18
ryansbso you get it for free19:18
ryansbthanks for the tcp nudge, now we'll see if the images build properly19:19
ryansbrhallisey: yeah, my atomic baremetal has nicer hardware than my workstation, figured it might be faster19:20
rhalliseyryansb, cool.  Using atomic for dev work can be a tough. No yum XD.  But I'm sure you know your way around19:21
ryansbI actually rarely use the atomic tools currently, mostly just to remote docker host info19:22
ryansberr, "just to host remote docker containers" is what I meant19:22
*** jasonsb has joined #kolla19:24
*** Hui has joined #kolla19:24
rhalliseyryansb, cool19:24
HuiHi, my latest neutron-server deployment fails from binary19:24
Hui Waiting for bootstrap container to exit > {"changed": true, "cmd": ["docker", "wait", "bootstrap_neutron"], "delta": "0:00:00.043556", "end":19:25
Huidoes anyone have similar problem; it was working last few days19:25
rhalliseyHui, does the container show up in restarting?19:28
Huithis is a neutron bootstrap container19:29
rhalliseyright, but you might still see it around19:29
Huiyes19:29
Huiin one of the compute node19:29
Huibecause it is "runonce" on any node19:30
Huido you know how to check the log?19:30
rhalliseywell you can start the container by hand and shell into it19:30
rhalliseyand run the commands manually19:30
rhalliseyand see what they split back out at you19:31
rhalliseycould be a db issue19:31
Huibut I see other service like glance and nova is booted fine19:31
jasonsbharmw: were you thinking of using fluentd on client side to forward the logs?19:32
rhalliseyHui, I'm not sure. Run the neutron-server container with '-it' flags and execute '/bin/bash'19:37
rhalliseythen run entend-start.sh19:38
*** blahRus1 has joined #kolla19:38
rhalliseyhopefully will reveal something19:38
*** blahRus has quit IRC19:39
*** inc0 has joined #kolla19:51
inc0hey19:51
*** mbound has joined #kolla19:53
sdakeyo inc020:22
sdakeup late20:22
*** ryansb has quit IRC20:26
*** ryansb has joined #kolla20:28
*** ryansb has quit IRC20:28
*** ryansb has joined #kolla20:28
*** achanda has quit IRC20:35
harmwjasonsb: at $work we're using fluentd on each docker host (lets call those the clients) and they send data per interval to the fluentd containers that reside on the ElasticSearch nodes, from where they inject into localhost:920020:49
harmwand all services (containers) log their JSON to the local fluentd container, as is done using docker (stdout/stderr)20:50
dmsimardrhallisey: so, anyway, do you know about delorean ? There's probably a way to hook image builds in there :)21:04
Huirhallisey, thanks. I found that somehow the neutron.conf is not pushed to all nodes21:05
rhalliseydmsimard, I do know about delorean, but not much21:05
rhalliseyHui, oh nice21:05
Huiso if that node is picked as a bootstrap node, the bootstrap of neutron will fail21:05
rhalliseydmsimard, how can we hook in images builds.  That's sounds exactly what I'm looking for21:05
rhalliseyHui, see if you can file a bug around it21:06
rhalliseynot sure how that happened21:06
dmsimardrhallisey: well, I don't have anything to show yet but delorean essentially watches upstream repos and then builds packages whenever there's a commit. It shouldn't be very hard to build an image instead of a package. Just have to figure out how to do it in a smart and efficient way.21:07
rhalliseydmsimard, oh neat.  I was going to have a job that periodically builds the containers21:08
rhalliseybut that sounds much better21:08
Huirhallisey, sure, I will21:09
rhalliseyHui, cool thanks21:09
dmsimardrhallisey: well, this is for the RDO packaging side - I don't know what the other distro have/use21:11
sdakehttps://www.youtube.com/watch?v=BKYJuYsT4z4&feature=youtu.be21:13
rhallisey^ check out 38 min.  I'm on stage! :D21:15
sdakedmsimard you can regex the build21:15
sdakepresentation was pretty solid21:16
sdaketimed it right ;-)21:16
dmsimardYou guys might want to A) pin that to liberty in the liberty branch or B) use the GA repo https://github.com/openstack/kolla/blob/stable/liberty/docker/base/Dockerfile.j2#L52-L5321:19
jasonsbharmw: neat.  any particular reason to use fluentd on source side?21:19
dmsimardThose links are the mitaka repos now (since liberty was cut)21:19
jasonsbharmw: is it to have same tooling to handle streamed syslog style and also picking up logs from files?21:19
harmwjasonsb: buffering :)21:20
sdakei thought we did that, but dms if you have links to the repo or want to wsubmit a patch itwould be appreciated21:20
jasonsbah ok21:20
harmwthe local container receives logs and pushes them per interval (10s) to fluentd container on ES hosts21:20
jasonsbis it qualitatively different than that provided by rsyslog?21:20
dmsimardI can submit a fix but what do you want ? Do you want to use the latest tested packages (closer to trunk liberty) or the stable packages (closer to release/milestone)21:20
harmwplus, we add some keys if the logs come from the crappy docker stdout/stderr21:20
harmwwell, it's k/v JSON21:21
jasonsbrsyslog has queues i was thinking21:21
rhalliseydmsimard, agreed21:21
harmwbut rsyslog logs in lines, not in JSON21:21
harmwright?21:22
harmwcapturing stacktraces sucks bigtime that way21:22
rhalliseydmsimard, stable liberty21:23
jasonsbthis is true.  you have to parse the thing while making sure to keep the log lines in correct order21:23
dmsimardrhallisey: okay I'll submit a fix for that, straight in the liberty branch though since master we'll keep mitaka (master)21:23
jasonsbotoh you don't have to worry about local storage if you pay that piper21:24
rhalliseydmsimard, so into master and add 'backport: liberty'21:24
dmsimardrhallisey: You want to pin liberty in master ?21:24
jasonsbharmw: regardless of the tradeoffs, this is an interesting topic.  have to look at it closer with operational POV21:25
rhalliseydmsimard, no sorry into liberty21:25
rhalliseyI thought the whole time that branch was pinned to liberty21:26
rhalliseydidn't notice21:26
*** achanda has joined #kolla21:28
sdakedmsimard for stable/liberty we want to use liberty release, for masster, we want to use delorean21:31
inc0we have additional problem with rsyslog21:31
inc0I can't get TRACEBACK to log to rsyslog:/21:32
inc0that's strange thing because everything else is logged21:32
*** cemason has joined #kolla21:35
Huirhallisey, filed a bug at https://bugs.launchpad.net/kolla/+bug/151359821:35
openstackLaunchpad bug 1513598 in kolla "neutron bootstrap container fails due to missing neutron.conf" [Undecided,New] - Assigned to Hui Kang (huikang27)21:35
rhalliseycool thanks Hui21:36
harmwjasonsb: we're using it in production, works like a breeze :)21:37
harmwlet me know if you need more info21:37
Huirhallisey  my pleasure.  If you have time, could you validate it?21:37
jasonsbharmw: will be in touch.  i am interested in the cardinality of the source vs sink for fluentd21:37
rhalliseyya I'll try to get around to it21:38
*** cemason has quit IRC21:39
*** _tzn has joined #kolla21:41
*** tzn has quit IRC21:41
*** inc0 has quit IRC21:43
*** absubram has quit IRC21:44
openstackgerritDavid Moreau Simard proposed openstack/kolla: Pin the liberty branch to the liberty RPM repositories  https://review.openstack.org/24222921:46
*** JoseMello has quit IRC21:52
rhalliseyHui, https://review.openstack.org/#/c/241989/321:53
rhalliseyhttps://bugs.launchpad.net/kolla/+bug/151343921:53
openstackLaunchpad bug 1513439 in kolla "bootstrap process can happen on the wrong node" [Critical,In progress] - Assigned to Sam Yaple (s8m)21:53
rhalliseyHui, ^^ that patch will fix it21:53
openstackgerritMerged openstack/kolla: Fix documentation links in the README  https://review.openstack.org/24218821:54
sdakedmsimard if you plan to continue contributing to kolla, would you like to join the kolla-drivers team?21:57
sdakethis team is responsible for bug triage of kolla (basically setting bug states on new bugs in kolla)21:57
*** shardy has quit IRC21:58
dmsimardI don't have any "official" time to spend on kolla right now :(21:59
sdakeroger, open offer, keep in mind for future21:59
* dmsimard nods21:59
*** rhallisey has quit IRC22:00
*** unicell1 has joined #kolla22:01
*** unicell has quit IRC22:02
*** unicell1 has quit IRC22:10
*** unicell has joined #kolla22:10
diogogmtcan somebody please clarify if there is a way on kolla to pull the images from a public repo instead of building them locally? i just want o build the image for a couple of services we want to add patches for, not all the openstack services images22:14
*** achanda has quit IRC22:22
Huirhallisey, thanks22:25
*** cloudnautique has quit IRC22:29
*** alisonh has quit IRC22:31
sdakediogogmt atm the only tagged release of kolla is DOA becaue of a bug that was added after we tagged the repo to an upstream repository22:36
sdakediogogmt we fixed the problem in stable/liberty, however, we have not retagged a new 1.0.1 release22:36
sdakesince we don't have a tag, we have not pushed a liberty releae to docker hub22:36
sdakeso the short term answer is no22:37
sdakeI guess we need to tag a 1.0.1 release but I've been super jet-lagged this week and my sleep is not good22:37
sdakebasically I haven't had eneough energy to test and releae a 1.0.1 tag since summit22:37
sdakediogogmt if you already have imagess you can build specific imaes - build.py takes a regex as an argument that identifies the images to build22:40
sdakeSamYaple are yo uabout22:42
*** diogogmt has quit IRC22:44
*** daneyon_ has quit IRC22:46
*** alisonh has joined #kolla22:49
*** achanda has joined #kolla22:59
*** _tzn has quit IRC23:02
*** tzn has joined #kolla23:07
asalkeldanyone running ubuntu 15.10 and kolla works? ubuntu wants to upgrade  it's self23:08
* asalkeld giving it a go23:11
*** sdake has quit IRC23:12
*** sdake has joined #kolla23:14
*** asalkeld has quit IRC23:15
*** tzn has quit IRC23:16
SamYaplesdake: yea23:16
*** tzn has joined #kolla23:16
sdakeSamYaple say can you test that build.py change on ubuntu wfm on centos23:18
sdakeHui around?23:19
openstackgerritMerged openstack/kolla: Use kolla_internal_address in the example rc file  https://review.openstack.org/23999223:19
*** Hui has quit IRC23:25
openstackgerritSteven Dake proposed openstack/kolla: Make two types of openrc example efiles  https://review.openstack.org/24225623:25
sdakeSamYaple I inadvertnatly set w+1 to a patch and I think it caused the child to fail to merge, and recheck may not fix this type of oops23:27
*** achanda has quit IRC23:31
*** achanda has joined #kolla23:32
*** Ti-mo- has quit IRC23:34
*** Ti-mo has joined #kolla23:36
*** asalkeld has joined #kolla23:40
sdakesamyaple do you tihnk you will be able to tackle the cap codebase, or willing to pass it to another cr?23:41
sdakefor m1 (dec 4th)23:41
asalkelddam, upgrading a desktop takes for ever ...23:43
sdakemoring asalkeld23:44
asalkeldhowdie, sorting a new spec rev. now23:44
openstackgerritAngus Salkeld proposed openstack/kolla: Spec: Deploy Kolla images using Mesos  https://review.openstack.org/24108623:48
openstackgerritMerged openstack/kolla: Remove vip for rabbitmq  https://review.openstack.org/23577723:49
*** blahRus1 has quit IRC23:49
SamYaplesdake: capping all of the versions?23:51
SamYaplei can do that23:51
sdakenah cap permissions23:51
SamYapleno23:51
SamYaplewe need to drop user perms first23:51
sdakeagree23:51
SamYaplecap permissions are just icing on the cake anyway23:52
SamYaplemost people dont know what that is23:52
sdakei gueess i se it as one job but its probably two blueprints23:52
SamYapleobsolete/superceed23:52
SamYaplesetup up the one blueprint with work items23:52
sdakeif we are going to distribute the work i prefer etherpad for that23:54
sdakeyou ahd expressed interest in doing this work23:54
sdakewondering if you going to do it alone or want to distribute23:54
SamYapleid like to start using the work items rather than an etherpad23:56
SamYapleits why they are there23:56
SamYapleany cores around that can give me an ack for this so I can backport the patch afterward? https://review.openstack.org/#/c/241856/23:57
*** jasonsb_ has joined #kolla23:57
sdakeSamYaple back to hte original q do you want to do the job alone or distribute23:57
SamYaplei want to see the work items involved first :)23:58
sdakewe can try it23:58
sdakeopen to giving it a go23:59
sdakeif it doesn't work as well as etherpad we can stick with etherpad23:59
sdakeif it is as effective we can switch permanetly23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!