Tuesday, 2016-02-23

SamYapleyo yo00:00
SamYaplemaybe there is some confusion here?00:01
asalkeldhey SamYaple00:01
*** ayoung has joined #kolla00:02
SamYapleasalkeld: https://github.com/openstack/kolla-mesos/blob/master/config/mariadb/templates/galera.cnf.j2#L300:02
SamYaplethat is the problem00:02
SamYaplethats certainly ansible variables, no?00:02
asalkeldSamYaple: i have a patch series that will soon kill that00:02
SamYaplei thought so00:02
SamYaplethats what i said in the midcycle00:02
SamYaplei brought it up in a few reviews back00:02
SamYapleim just looking for a clean break or more full integration00:03
sdakewe won't be able to split the repos until this problem is solved00:03
asalkeldthat was largely me being lazy, and thinking "hey look i can just copy the configs"00:03
SamYaplethis halfway non-sense is just going to be a problme00:03
SamYaplesdake: this has nothing to do with spliting out kolla-ansible...00:03
asalkeldi'll clean up soon00:03
SamYapleasalkeld: i think it was a great first step00:03
sdakeya prototype then tidy00:03
sdakestandard engineering practice00:04
sdakenow may not be right time to tidy00:04
sdakebut its coming00:04
asalkeldi'll post what i have so you can see00:04
SamYapleanyway asalkeld i think were good00:04
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Make the memcached dependency optional  https://review.openstack.org/28331600:04
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Test the rendering of nova.conf.j2  https://review.openstack.org/28331700:04
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Derive the role/group from the APP_ID  https://review.openstack.org/28331800:04
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Move service zookeeper nodes  https://review.openstack.org/28331900:04
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: remove unused get_new_node_id() in start.py  https://review.openstack.org/28332000:04
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Add Template functions for common uses  https://review.openstack.org/28332100:04
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Use new template functions  https://review.openstack.org/28332200:04
sdakekzaitsev_mb ok wound me plz00:05
asalkeldSamYaple: https://review.openstack.org/28332200:05
openstackgerritLingfeng Xiong proposed openstack/kolla: Glance image path customization  https://review.openstack.org/28284000:05
SamYapleyea asalkeld that looks alot better00:06
*** kzaitsev_mb has quit IRC00:06
SamYaplethe one other thing taht was brought up was duplicating the extend_start.sh script for (i think) keystone00:06
SamYapleI dont see a need to duplicate that unless a _change_ is needed00:07
SamYaplenihilifer saw and agreed to that as well00:07
SamYapledo you disagree?00:07
SamYapleand if a change is truly needed, shouldn't it be in the kolla repo anyway?00:07
asalkeldSamYaple: yip, agree - we thought at the time it had to do with something we were doing "special", but turned to be a common bug00:07
SamYaplecool then i think we are on the same page00:08
SamYaplejust bear in mind that config changes to kolla-ansible wont get propogated over to kolla-mesos00:08
*** thumpba has quit IRC00:08
SamYaplewe really need a policy on how to handle that, even if its as simple as requiring a bug to change a config template00:08
SamYaplethat bug could be labeled as kolla and kolla-mesos for tracking00:09
openstackgerritMerged openstack/kolla: Multiple hypervisor driver support for nova  https://review.openstack.org/28294300:09
*** kzaitsev_mb has joined #kolla00:09
asalkeldyes, that is an issue00:10
kzaitsev_mbsdake: ok, so in short: murano has a plugin for glance(glare), that lives inside murano code-base and has to be installed in glance's env for murano to operate correctly with it.00:11
kzaitsev_mbsdake: this kind of setup is currently marked as experimental00:11
sdakei see00:11
SamYaplekzaitsev_mb: does it affect glance if it is installed? or does it require config option to enable?00:12
sdakeplugins are an unsolved problem for us00:12
kzaitsev_mbsdake: so the question is — would there be any problems with kolla way of setting up glance/murano00:12
sdakethis is first i've heard of a plugin in a different repository00:12
SamYaplesdake: horizon.00:12
SamYaplethats all horizon is00:12
sdakeform a source perspective, you could just install murano in the glance container and pull the plugin from there00:12
SamYapleplugins in different repo00:12
sdakebut, my obvious question, w,hy not export the plugin to a third repo?00:12
sdakethat would make things alot easier on us00:13
sdakeif it was in a third repo then we could import the plugin instead of all of murano00:13
kzaitsev_mbSamYaple: it affects glare — which is an API and Storage for Artifacts (former glance-v3). It defines a set of properties for murano's artifact00:14
asalkeldSamYaple: sdake i'll make an etherpad and try explain how our stuff is working and some ideas for better commonality - tho' SamYaple you suggesting a "clean break"?00:14
sdakewhich repo is glare in?00:14
sdakedefine clean break asalkeld00:15
asalkeldin templates/*, SamYaple's terminology00:15
SamYaplekzaitsev_mb: i dont think we can include that then00:15
kzaitsev_mbsdake: that's actually the reason I'm asking in the first place, we've been thinking about pros/cons of storing it in the separate repo for some time. Plugin itself is actually quite small (a handfull of python lines)00:15
SamYaplekzaitsev_mb: if it affects glance by the very fact it is installed, i dont think we can include it in the repo as it stands00:16
asalkeldi'll start here: https://etherpad.openstack.org/p/kolla-set_configs00:16
SamYapleif it is only turned on via config options, thats a different story00:16
sdakeif a plugin is going to be in a larger scope repo, it should relate to that larger scoped repo00:16
kzaitsev_mbso storing it in a separate repo seemd like an overkill, when we first implemented the thing, as experimental feature =)00:17
sdakefor example, if yoru plugin is for glare, why nto put it in the glare repo rather then the murano repo?00:17
kzaitsev_mbsdake: glare lives in glance's repo btw. it was just moved to a separate process, but still the code lives there00:17
sdakeok why not store in glance repo?  they dont want in their repo?00:17
* sdake wtb session at summit on plugins00:18
kzaitsev_mbsdake: yep, that's another option we have. Would it solve packageing issues for you to any extent?00:18
kzaitsev_mbsdake: afaik that's how searchlight works with plugins (stores all of them within it's own repos)00:19
sdakekzaitsev_mb hmm well I understand the rationale, its harder making new repos for each plugin00:20
sdakebut this is a common practice in other projects (making new repos)00:20
sdakerather hten overloading one repo with a bunch of plugins00:20
kzaitsev_mbTBH — I'm wheighting the options here, since it basicaly affects all the packaging folks00:21
kzaitsev_mbSamYaple: well, glare is designed that way, i.e. just installing the plugin enables it and adds all the artifact-endpoints, to API it(glare) provides00:22
sdakekzaitsev_mb its too late to do anything for mitaka on this point anyway00:22
sdakekzaitsev_mb I'd suggest we have a session at summit with the projects that use plugins00:22
sdakewhat I need is one way for plugisn to happen across openstack00:22
sdakewhatever that way is, we will adapt to it00:23
sdakeI think what we have now is two ways to do plugins00:23
sdakeone in repo and one out of repo00:23
kzaitsev_mbsdake: you're probably right about mitaka =)00:23
sdakeI am not certain on this point00:23
kzaitsev_mbyep, we've seen multiple ways, when we investigated how other projects store their plugins00:23
SamYaplekzaitsev_mb: I am unfamiliar with glare so youll have to forgive the ignorance on the subject. This seems like something that wont happen until newton00:24
SamYaplecan you touch base then?00:24
sdakeya mitaka is locked up - no new features - the minerals and vespene gas are all used up ;)00:24
SamYaplesdake: "1 way for plugins" doesnt make sense when most plugins are installed different ways00:24
kzaitsev_mbsdake: maybe one last question — which way would you prefer most: murano-glare-plugin in murano-repo, murano-glare-plugin in glare repo or murano-glare-plugin in a standalone repo. To crown it all?00:25
kzaitsev_mbSamYaple: touch base? =)00:26
sdakeoff the cuff, a separate repo would be best00:26
sdakebut i'd really have to think about it and analyze how other projects do plugins00:26
sdakei do recognize its overkill to put a plugin in a separate repositroy00:26
sdakebut it makes packegers lives alot easier00:26
sdake(of which we are:)00:27
sdakethe second choice would be for all plugins to live in the repo to which the plugin plugs into00:27
sdakefor example a glare plugin would be in a glare repo00:28
sdakebut this is not how horizon behaves00:28
sdakeI htink they expect people to put plugins in their own repository00:28
sdakekzaitsev_mb the bottom line is we can't dictate to other projects how to package their plugins00:28
kzaitsev_mbsdake: scheduling a session at summit actually sounds like a really nice idea to me, not sure if I'm coming, but I'll definitelly ping folks to try and reach out00:28
sdakeso we should follow best practices already sorted out by the community00:28
sdakehopefully there aren't separate best practices ;)00:29
kzaitsev_mbsdake: sounds fair00:29
sdakehow to package/how to distribute above00:29
sdakekolla packages, murano distributes00:29
sdakethis will likely be a fishbowl session since i'd like a wide attendance from the other project coress if possible00:30
kzaitsev_mbthanks for the insights, now I've got something to think about and to discuss with murano/glance communities.00:31
sdakeif you get any data, please hit us up onthe mailing list with a [kolla] tag :)00:32
*** kproskurin has quit IRC00:33
kzaitsev_mbsure I'll do =)00:36
sdake1 USD who can tell me what repo creates http://releases.openstack.org/mitaka/index.html00:40
sdakeUPDATED: MON FEB 22 17:05:11 2016, COMMIT CA484E400:40
openstackgerritSam Yaple proposed openstack/kolla: cloud-archive contains newer version of haproxy  https://review.openstack.org/28333100:44
*** salv-orlando has quit IRC00:45
*** sdake_ has joined #kolla00:49
*** sdake_ has quit IRC00:49
*** sdake_ has joined #kolla00:49
*** sdake has quit IRC00:51
sdake_ok folks time to play with the kiddos before they are all grown up and don't want to talk to me from 20-30 :)00:51
sdake_SamYaple I keep getting ads in google chrome for the westin pointsette00:53
sdake_(the one where you live)00:53
* sdake_ thinks google knows too much aboutme ;)00:53
*** sdake_ has quit IRC00:56
*** thumpba has joined #kolla00:59
*** kzaitsev_mb has quit IRC01:00
*** cui has joined #kolla01:03
openstackgerritAngus Salkeld proposed openstack/kolla: Skip extend_start.sh if KOLLA_SKIP_EXTEND_START is defined  https://review.openstack.org/28333901:09
openstackgerritJeffrey Zhang proposed openstack/kolla: [WIP] Reconfigure keystone service  https://review.openstack.org/28138001:09
*** tfukushima has joined #kolla01:11
*** iceyao has joined #kolla01:16
*** iceyao_ has joined #kolla01:21
openstackgerritMerged openstack/kolla: Add elasticsearch repo to gate  https://review.openstack.org/28327501:22
*** iceyao has quit IRC01:23
*** tzn has joined #kolla01:25
*** alisonh has quit IRC01:27
*** tzn has quit IRC01:30
*** cui has quit IRC01:42
*** ByPasS has quit IRC01:43
*** alisonh has joined #kolla01:45
*** iceyao_ has quit IRC01:47
*** iceyao has joined #kolla01:47
*** ByPasS has joined #kolla01:49
*** hparekh has quit IRC01:53
*** kzaitsev_mb has joined #kolla01:56
*** hparekh has joined #kolla01:57
*** sdake has joined #kolla01:57
*** iceyao has quit IRC01:58
*** iceyao has joined #kolla01:58
*** iceyao has quit IRC02:03
*** iceyao_ has joined #kolla02:03
*** sdake has quit IRC02:04
*** tfukushima has quit IRC02:04
*** sdake has joined #kolla02:07
*** tfukushima has joined #kolla02:11
*** dims has quit IRC02:13
*** jasonsb has joined #kolla02:15
*** rhallisey has quit IRC02:18
*** kzaitsev_mb has quit IRC02:19
*** salv-orlando has joined #kolla02:22
*** salv-orlando has quit IRC02:30
*** neophy has joined #kolla02:33
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Skip the default sourcing of extend_start in kolla start.sh  https://review.openstack.org/28335602:33
*** sdake has quit IRC02:38
*** sdake has joined #kolla02:39
*** iceyao_ has quit IRC02:41
*** achanda has joined #kolla02:41
*** iceyao has joined #kolla02:41
*** sdake has quit IRC02:41
*** tzn has joined #kolla02:44
openstackgerritJeffrey Zhang proposed openstack/kolla: [WIP] Reconfigure keystone service  https://review.openstack.org/28138002:44
*** tzn has quit IRC02:48
openstackgerritMartin André proposed openstack/kolla: Fix building centos kolla-toolbox  https://review.openstack.org/28290602:54
Jeffrey4l_why we need mariadb-libs in the openstack-base image? and why the `Percona-XtraDB-Cluster-shared-55` is installed when the percona-release repo is enabled.03:02
Jeffrey4l_mandre, do u have any idea? ^^03:02
mandreJeffrey4l_, not sure why mariadb-libs was there in the first place03:11
mandremaybe it was a requirement of MySQL-python at the time03:12
mandreto answer your second question, percona repo sets different priorities and obsoletes a lot of packages that's why you often get different packages than you expect03:14
mandreJeffrey4l_, where is Percona-XtraDB-Cluster-shared-55 installed?03:14
*** unicell has quit IRC03:14
Jeffrey4l_mandre, show the details in the launchpad later.03:15
openstackgerritMartin André proposed openstack/kolla: Remove unneeded deps  https://review.openstack.org/28336303:15
*** kzaitsev_mb has joined #kolla03:16
*** yuanying_ has joined #kolla03:16
*** yuanying has quit IRC03:19
*** yuanying_ has quit IRC03:20
Jeffrey4l_mandre, see the bug https://bugs.launchpad.net/kolla/+bug/154858903:21
openstackLaunchpad bug 1548589 in kolla "kolla-toolbox fails to build" [Critical,New] - Assigned to Jeffrey Zhang (jeffrey4l)03:21
*** kzaitsev_mb has quit IRC03:21
mandreJeffrey4l_, oh I see03:22
*** Jeffrey4l_ is now known as Jeffrey4l03:22
mandrelike I said, percona sets a different priority for their packages, and they obsolete mariadb packages03:23
mandreso when you have the percona repo enabled (we do) and you want to install mariadb-libs it really installs Percona-XtraDB-Cluster-shared-5503:23
mandreJeffrey4l, http://logs.openstack.org/31/283331/1/check/gate-kolla-dsvm-build-centos-binary/e18aebe/console.html#_2016-02-23_00_56_52_84603:25
Jeffrey4lmandre, yea, saw that.03:25
Jeffrey4lbrb03:25
*** chandankumar has joined #kolla03:28
*** tpot has left #kolla03:30
*** salv-orlando has joined #kolla03:34
Jeffrey4lback03:38
Jeffrey4lmandre, should we remove the MariaDB-shared? is it the TODO you mentioned in the code? https://github.com/openstack/kolla/blob/master/docker/kolla-toolbox/Dockerfile.j2#L603:40
Jeffrey4lremove the MariaDB-shared instead.03:40
*** klint has joined #kolla03:40
*** dims has joined #kolla03:49
Jeffrey4lmandre, ^03:50
*** salv-orlando has quit IRC03:50
*** unicell has joined #kolla03:55
*** sdake has joined #kolla04:00
*** yuanying has joined #kolla04:08
*** Marga_ has quit IRC04:12
*** kzaitsev_mb has joined #kolla04:18
*** chandankumar has quit IRC04:19
*** kzaitsev_mb has quit IRC04:23
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Use new template functions  https://review.openstack.org/28332204:24
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Add Template functions for common uses  https://review.openstack.org/28332104:24
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: remove unused get_new_node_id() in start.py  https://review.openstack.org/28332004:24
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Move service zookeeper nodes  https://review.openstack.org/28331904:24
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Derive the role/group from the APP_ID  https://review.openstack.org/28331804:24
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Test the rendering of nova.conf.j2  https://review.openstack.org/28331704:24
*** salv-orlando has joined #kolla04:26
*** sdake has quit IRC04:28
*** Marga__ has joined #kolla04:28
*** Marga__ has quit IRC04:32
*** salv-orlando has quit IRC04:32
mandreJeffrey4l, I submitted https://review.openstack.org/283363 to remove MariaDB-shared04:32
openstackgerritVikram Hosakote proposed openstack/kolla: Updated doc/quickstart.rst to help new contributors  https://review.openstack.org/28329804:38
*** alisonh has quit IRC04:55
openstackgerritDave McCowan proposed openstack/kolla: Implement Option for Two VIPs to separate API traffic  https://review.openstack.org/28251304:55
openstackgerritLingfeng Xiong proposed openstack/kolla: Allow cinder-volume to be configured to use nfs  https://review.openstack.org/27991304:56
openstackgerritDave McCowan proposed openstack/kolla: Implement Option for Two VIPs to separate API traffic  https://review.openstack.org/28251304:56
openstackgerritLingfeng Xiong proposed openstack/kolla: Allow cinder-volume to be configured to use nfs  https://review.openstack.org/27991304:56
*** alisonh has joined #kolla04:59
*** coolsvap|away has quit IRC05:06
*** Marga_ has joined #kolla05:12
*** chandankumar has joined #kolla05:15
*** Marga_ has quit IRC05:16
*** kzaitsev_mb has joined #kolla05:19
*** Marga_ has joined #kolla05:23
*** kzaitsev_mb has quit IRC05:24
*** thumpba has quit IRC05:31
*** Marga_ has quit IRC05:33
*** Marga_ has joined #kolla05:33
openstackgerritMerged openstack/kolla: Fix building centos kolla-toolbox  https://review.openstack.org/28290605:37
*** sdake has joined #kolla05:41
nihilifersdake: ok, will rebase my kolla-ansible upgrade patches. i keep them as wip, because well, i didn't test them properly on multinode yet :(05:51
*** dims has quit IRC05:54
*** sdake has quit IRC05:57
*** tfukushima has quit IRC06:04
*** tfukushima has joined #kolla06:06
openstackgerritMichal Rostecki proposed openstack/kolla: Pin Mesos to 0.27.1 version  https://review.openstack.org/28309106:19
openstackgerritJeffrey Zhang proposed openstack/kolla: [WIP] Reconfigure keystone service  https://review.openstack.org/28138006:26
*** kzaitsev_mb has joined #kolla06:31
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Use new template functions  https://review.openstack.org/28332206:38
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Add Template functions for common uses  https://review.openstack.org/28332106:38
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: remove unused get_new_node_id() in start.py  https://review.openstack.org/28332006:38
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Move service zookeeper nodes  https://review.openstack.org/28331906:38
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Derive the role/group from the APP_ID  https://review.openstack.org/28331806:38
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Test the rendering of nova.conf.j2  https://review.openstack.org/28331706:38
*** unicell has quit IRC06:40
*** unicell has joined #kolla06:40
openstackgerritMichal Rostecki proposed openstack/kolla: Pin Mesos to 0.27.1 version  https://review.openstack.org/28309106:47
*** kzaitsev_mb has quit IRC06:50
*** v1k0d3n has quit IRC06:55
*** v1k0d3n has joined #kolla06:56
*** macsz has joined #kolla07:01
*** akwasnie has joined #kolla07:01
*** neophy has quit IRC07:07
*** Jeffrey4l has quit IRC07:10
openstackgerritDaniel Gonzalez Nothnagel proposed openstack/kolla: Fix vagrant bootstrapping for ubuntu  https://review.openstack.org/28306907:11
openstackgerritMichal Rostecki proposed openstack/kolla: Pin Mesos to 0.27.1 version  https://review.openstack.org/28309107:15
*** tzn has joined #kolla07:22
*** Jeffrey4l has joined #kolla07:24
*** akwasnie has quit IRC07:25
*** tzn has quit IRC07:27
*** Jeffrey4l has quit IRC07:35
*** akwasnie has joined #kolla07:40
*** salv-orl_ has joined #kolla07:41
*** salv-orl_ has quit IRC07:41
*** mikelk has joined #kolla07:43
*** kzaitsev_mb has joined #kolla07:46
*** achanda has quit IRC07:49
*** Jeffrey4l has joined #kolla07:52
openstackgerritEric Lemoine proposed openstack/kolla: Remove duplicate code in nova extend_start.sh  https://review.openstack.org/28308207:58
*** kzaitsev_mb has quit IRC07:59
openstackgerritMichal Rostecki proposed openstack/kolla: Pin Mesos to 0.27.1 version  https://review.openstack.org/28309107:59
akwasniehi08:11
openstackgerritMichal Rostecki proposed openstack/kolla: Pin Mesos to 0.27.1 version  https://review.openstack.org/28309108:13
elemoine_hi akwasnie08:21
*** subscope has joined #kolla08:32
*** jmccarthy has quit IRC08:33
*** subscope has quit IRC08:36
*** macsz has quit IRC08:37
openstackgerritEric Lemoine proposed openstack/kolla: Change Jinja2 expressions in heka.json.j2  https://review.openstack.org/28344708:44
*** macsz has joined #kolla08:44
*** jmccarthy has joined #kolla08:45
*** akwasnie has quit IRC08:46
*** subscope has joined #kolla08:47
*** akwasnie has joined #kolla08:47
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Tag Mesos slaves with their OpenStack roles  https://review.openstack.org/28345208:51
*** kzaitsev_mb has joined #kolla08:56
openstackgerritYongfeng Du proposed openstack/kolla: Clean the KOLLA_CONFIG_STRATEGY setting  https://review.openstack.org/28345709:02
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: [WIP] Define roles of OpenStack services  https://review.openstack.org/28345809:03
openstackgerritYongfeng Du proposed openstack/kolla: Clean the KOLLA_CONFIG_STRATEGY setting  https://review.openstack.org/28345709:16
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: [WIP] Define roles of OpenStack services  https://review.openstack.org/28345809:17
*** subscope has quit IRC09:20
*** achanda has joined #kolla09:22
*** achanda has quit IRC09:27
*** akwasnie has quit IRC09:29
*** pbourke has quit IRC09:31
*** pbourke has joined #kolla09:31
*** akwasnie has joined #kolla09:32
*** kzaitsev_mb has quit IRC09:33
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Mistral logs  https://review.openstack.org/28317709:33
openstackgerritMD NADEEM proposed openstack/kolla: Tempest ubuntu binary container  https://review.openstack.org/27729009:38
*** subscope has joined #kolla09:43
*** salv-orlando has joined #kolla09:43
*** mikelk has quit IRC09:45
*** mikelk has joined #kolla09:45
openstackgerritEric Lemoine proposed openstack/kolla: Remove Rsyslog entirely  https://review.openstack.org/28315009:53
*** subscope has quit IRC09:57
*** tzn has joined #kolla10:01
*** tzn has quit IRC10:01
*** tzn has joined #kolla10:01
*** subscope has joined #kolla10:02
*** tzn has quit IRC10:03
*** liyi has left #kolla10:06
*** liyi has joined #kolla10:06
*** gfidente has joined #kolla10:07
*** gfidente has joined #kolla10:07
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: [WIP] Define roles of OpenStack services  https://review.openstack.org/28345810:21
*** kzaitsev_mb has joined #kolla10:26
*** bmace has quit IRC10:39
*** bmace has joined #kolla10:40
*** cui has joined #kolla10:47
*** tfukushima has quit IRC10:53
*** iceyao has quit IRC10:55
openstackgerritMerged openstack/kolla: cloud-archive contains newer version of haproxy  https://review.openstack.org/28333111:03
ajafohttps://tech.vijayp.ca/linux-kernel-bug-delivers-corrupt-tcp-ip-data-to-mesos-kubernetes-docker-containers-4986f88f7a19#.2ervo0iy711:15
*** cui has quit IRC11:17
*** subscope has quit IRC11:21
*** achanda has joined #kolla11:25
*** achanda has quit IRC11:31
openstackgerritMerged openstack/kolla: Remove unneeded deps  https://review.openstack.org/28336311:31
*** salv-orlando has quit IRC11:41
*** JoseMello has joined #kolla11:47
*** kzaitsev_mb has quit IRC11:51
*** subscope has joined #kolla11:58
*** salv-orlando has joined #kolla12:15
*** rhallisey has joined #kolla12:26
*** stvnoyes has quit IRC12:36
*** stvnoyes has joined #kolla12:38
*** mbound has joined #kolla12:41
*** mbound_ has joined #kolla12:46
*** masterbound has joined #kolla12:48
openstackgerritJeffrey Zhang proposed openstack/kolla: Reconfigure keystone service  https://review.openstack.org/28138012:48
openstackgerritJeffrey Zhang proposed openstack/kolla: Add the default empty reconfig.yml file to all roles  https://review.openstack.org/28355112:48
*** mbound has quit IRC12:49
*** mbound has joined #kolla12:49
openstackgerritMerged openstack/kolla: Tempest ubuntu binary container  https://review.openstack.org/27729012:50
openstackgerritJeffrey Zhang proposed openstack/kolla: Add the default empty reconfig.yml file to all roles  https://review.openstack.org/28355112:50
openstackgerritJeffrey Zhang proposed openstack/kolla: Reconfigure keystone service  https://review.openstack.org/28138012:50
*** mbound_ has quit IRC12:51
*** masterbound has quit IRC12:53
*** iceyao has joined #kolla12:58
*** subscope has quit IRC13:05
*** mandre has quit IRC13:16
*** mandre has joined #kolla13:17
*** subscope has joined #kolla13:22
*** achanda has joined #kolla13:28
*** secate has joined #kolla13:28
*** subscope has quit IRC13:32
*** athomas has quit IRC13:32
*** achanda has quit IRC13:33
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: [WIP] Define roles of OpenStack services  https://review.openstack.org/28345813:36
*** dwalsh has joined #kolla13:39
*** salv-orlando has quit IRC13:41
*** mbound has quit IRC13:42
*** athomas has joined #kolla13:43
*** subscope has joined #kolla13:45
*** subscope has quit IRC13:55
*** mbound has joined #kolla13:57
*** klint has quit IRC13:57
*** salv-orlando has joined #kolla14:01
*** subscope has joined #kolla14:04
*** jasonsb has quit IRC14:06
*** inc0 has joined #kolla14:07
inc0good morning14:08
inc0Jeffrey4l, around?14:08
akwasniehi inc014:08
inc0(or rather I'll try later;))14:08
Jeffrey4linc0, yea14:08
inc0oh, great:)14:08
Jeffrey4lsup14:08
inc0https://review.openstack.org/#/c/283278/3 what do you mean by password auth?14:09
Jeffrey4linc0, check this. https://review.openstack.org/#/c/277085/14:10
Jeffrey4linc0, https://review.openstack.org/#/c/277085/1/ansible/roles/haproxy/templates/keepalived.conf.j214:10
Jeffrey4lthe keepalived has a authentication function which is disable now. But I think it is need to enable for security reason.14:11
Jeffrey4lAnd the best time to enable this, is the upgrade stage.14:12
SamYaplemorning14:13
Jeffrey4lbecause it will may cause some downtime14:13
Jeffrey4lmorning SamYaple14:13
inc0when does it ask for password tho?14:13
inc0I'm not convinced about that...14:13
SamYapleJeffrey4l: yea auth is a good idea for keepalived14:14
SamYapleneutron does it as well14:14
SamYapleinc0: it never _asks_ for a password14:14
SamYapleits in hte ocnfgi14:14
inc0it's for someone tap to the cluster by just running keepalived in same subnet?14:14
Jeffrey4lsecurity. other service may use the same vrrp_instance and virtual_router_id.14:15
SamYapleyea inc0 its really best practice at this point14:15
SamYaplei think theres a patch in the queue for it already too14:15
inc0I'd say let's not do it in upgrade play14:15
inc0there is no reason to14:16
inc0if we merge this patch, upgrade will take it into account automatically14:16
SamYapleinc0: its not part of the upgrade persay, it would be part of the config14:16
SamYapleso you would push configs14:16
SamYaplestop non-active14:16
SamYaplerestart active, start non-actives14:16
SamYaplethat would pull in password14:16
SamYapleits just a config change14:16
inc0keepalived doesn't have signup tho?14:16
SamYaple?14:17
inc0service keepalived reload14:17
inc0without restart14:17
SamYapleim confused. during upgrade container is recreated14:17
*** macsz has quit IRC14:17
inc0in any case, you can propagate this config change with upgrade play, easily14:17
inc0I mean if you want to enable this config14:17
SamYaplei dont think this requires any change to keepalied upgrade14:17
inc0wouldn't it be simpler to push configs and then service keepalived reload14:18
inc0I get that, I'm asking about reconfigure keepalived14:18
Jeffrey4lchange the auth will case the keepalive do not know each other. and in some time period, there may be multi actives.14:18
inc0btw our gates got pretty bad, have you noticed?14:18
SamYapleso im confused, you wouldnt run reconfigure without having first run upgrade14:19
SamYapleso upgrade handles the new config like it should, why does this matter for reconfigure14:19
inc0ok, whatever, in any case14:19
*** akwasnie has quit IRC14:19
inc0if you merge your review Jeff, kolla-ansible upgrade -t haproxy will do well for propagation of it14:19
inc0SamYaple, "One or more undefined variables: 'host' is undefined" what's that?14:22
inc0have you seen this one before?14:23
Jeffrey4lSamYaple, you means add the auth in the reconfigure stage for customer?14:24
SamYapleinc0: yea host isn't defined!14:24
SamYaplewhere are you seeing that?14:24
SamYaplelikely you copied a string out of a loop14:24
SamYapleJeffrey4l: i mean add it in before mitaka-3 and it will automatically get included when someone upgrades14:25
inc0yeah, that will be it14:25
inc0carry on14:25
inc0:P14:25
SamYaple:)14:25
*** jtriley has joined #kolla14:26
Jeffrey4lyea14:26
*** mbound_ has joined #kolla14:29
dave-mccowaninc0 along those lines, should router_id also be configurable?  if other services are using keepalive on the same subnet, there can be a collision with our hard coded 51.14:30
*** mbound has quit IRC14:30
*** inc0 has quit IRC14:31
*** achanda has joined #kolla14:31
*** inc0 has joined #kolla14:32
*** dwalsh has quit IRC14:33
*** achanda has quit IRC14:35
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: [WIP] Define roles of OpenStack services  https://review.openstack.org/28345814:39
*** mbound_ has quit IRC14:43
*** ayoung has quit IRC14:46
*** dmsimard is now known as dmsimard|afk14:48
SamYapledave-mccowan: not if its using auth14:53
dave-mccowanSamYaple  so if two services (or two kolla devs) use 51 on the same subnet, it's OK, as long as they have different passwords?14:54
*** spisarski has joined #kolla14:54
*** dims has joined #kolla14:54
*** sdake has joined #kolla14:55
sdakemorning14:57
sdakenihilifer don't worry about testing multinode14:57
sdakenihilifer that is not a requirement for merging14:57
sdakenihilifer testing single node is good enough14:57
sdakenihilifer i don't ant to hear the "I couldn't test on multinode so hte test was blocked for weeks on end" thing anymore :)14:58
*** rmart04 has joined #kolla14:59
sdakelets just get em merged, if folks can't or don't want to test multinode, we either a) need a gate to handle multinode HA or b) need to all test that model periodically and fix what problems we find14:59
*** inc0 has quit IRC14:59
*** inc0 has joined #kolla15:01
openstackgerritMerged openstack/kolla: Remove duplicate code in nova extend_start.sh  https://review.openstack.org/28308215:01
*** inc0_ has joined #kolla15:03
*** dwalsh has joined #kolla15:03
*** salv-orl_ has joined #kolla15:05
*** dave-mcc_ has joined #kolla15:05
*** inc0 has quit IRC15:06
*** salv-orlando has quit IRC15:07
*** stvnoyes has quit IRC15:07
*** dave-mccowan has quit IRC15:08
*** stvnoyes has joined #kolla15:08
*** subscope has quit IRC15:11
*** mbound has joined #kolla15:14
*** salv-orl_ has quit IRC15:17
*** haplo37 has joined #kolla15:26
*** liyi has quit IRC15:27
*** macsz has joined #kolla15:33
*** mbound has quit IRC15:34
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: [WIP] Define roles of OpenStack services  https://review.openstack.org/28345815:34
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Remove meaningless try-except block from ZooKeeper write  https://review.openstack.org/28364015:34
*** fthiagogv has joined #kolla15:36
*** inc0_ has quit IRC15:36
*** inc0 has joined #kolla15:36
openstackgerritMerged openstack/kolla: Skip extend_start.sh if KOLLA_SKIP_EXTEND_START is defined  https://review.openstack.org/28333915:38
openstackgerritMerged openstack/kolla: Change Jinja2 expressions in heka.json.j2  https://review.openstack.org/28344715:38
*** iceyao has quit IRC15:39
*** chandankumar has quit IRC15:40
SamYapledave-mcc_: I beilieve so, but we can check15:40
*** limamauricio has joined #kolla15:45
limamauriciosomeone already deploy ubuntu + source with ceph?15:46
*** inc0_ has joined #kolla15:49
*** jmccarthy has quit IRC15:49
*** jmccarthy has joined #kolla15:50
*** blahRus has joined #kolla15:50
*** inc0 has quit IRC15:53
*** chandankumar has joined #kolla15:55
SamYaplelimamauricio: i do15:59
limamauriciodid you have any problem with ssh?16:05
limamauricioi'm rebuilding now, but yesterday i had some problems with ssh16:06
SamYaplewhat do you mean?16:07
limamauricioi'll try again, if the same error happen I'll give you more details16:09
limamauricio:)16:09
*** v1k0d3n has quit IRC16:11
*** dwalsh has quit IRC16:13
*** subscope has joined #kolla16:13
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: [WIP] Define roles of OpenStack services  https://review.openstack.org/28345816:15
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add volume for /var/lib/kolla  https://review.openstack.org/28366616:15
*** v1k0d3n has joined #kolla16:15
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: [WIP] Define roles of OpenStack services  https://review.openstack.org/28345816:16
*** dwalsh has joined #kolla16:22
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: [WIP] Define roles of OpenStack services  https://review.openstack.org/28345816:23
limamauricioSamYaple, no errors in deploy :)16:23
*** v1k0d3n has quit IRC16:23
*** v1k0d3n has joined #kolla16:24
*** subscope has quit IRC16:27
*** achanda has joined #kolla16:33
*** dwalsh has quit IRC16:34
*** dims has quit IRC16:37
*** salv-orlando has joined #kolla16:37
*** salv-orlando has quit IRC16:37
openstackgerritMerged openstack/kolla: Make Heka collect Mistral logs  https://review.openstack.org/28317716:37
openstackgerritMerged openstack/kolla: Remove Rsyslog entirely  https://review.openstack.org/28315016:37
*** salv-orlando has joined #kolla16:37
inc0_how I hate ansible...16:38
inc0_some clever guy decided that if there is no secondary ip on interface, why bother adding this dict key at all16:38
*** achanda has quit IRC16:38
SamYapleinc0_: |default({})16:39
*** unicell1 has joined #kolla16:40
*** unicell has quit IRC16:41
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: HAProxy and Keepalived upgrade  https://review.openstack.org/28327816:41
*** Jeffrey4l has quit IRC16:42
inc0_SamYaple, I did some testing on almost-multinode, would you be kind enough to test it out on actual multimode16:42
inc0_?16:42
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: HAProxy and Keepalived upgrade  https://review.openstack.org/28327816:43
*** subscope has joined #kolla16:45
*** inc0___ has joined #kolla16:45
*** fgimenez has joined #kolla16:46
*** inc0___ has quit IRC16:46
*** inc0 has joined #kolla16:46
*** inc0___ has joined #kolla16:47
*** inc0___ has left #kolla16:48
*** inc0_ has quit IRC16:48
inc0lots of reconnects today..16:48
limamauricioSamYaple, i have this error now -> http://pastebin.com/qNwWcTtp16:49
*** kzaitsev_mb has joined #kolla16:54
*** v1k0d3n has quit IRC16:57
*** v1k0d3n has joined #kolla17:04
*** mikelk has quit IRC17:05
limamauriciosomeone have this error  Traceback (most recent call last):File "/usr/local/bin/nova", line 7, in <module>     from novaclient.shell import main   File "/usr/local/lib/python2.7/dist-packages/novaclient/shell.py", line 44, in <module>17:06
limamauricioi'm using ubuntu/source17:06
*** dims has joined #kolla17:09
*** fgimenez has quit IRC17:09
pbourkelimamauricio: how did you install python-openstackclient17:13
limamauricioyes :(17:13
limamauricioi did pip install -U python-openstackclient17:14
sdakelima can you paste the full traceback in a paste service17:15
limamauricioi'll try using apt-get now17:15
pbourkemaybe the packaging is broken on python-openstackclient17:15
sdakeI need to see what caused an exception17:15
limamauriciohttp://pastebin.com/qNwWcTtp17:15
*** inc0 has quit IRC17:16
*** inc0 has joined #kolla17:16
*** unicell1 has quit IRC17:20
*** jtriley has quit IRC17:26
*** sdake_ has joined #kolla17:28
*** ayoung has joined #kolla17:29
*** GheRivero has joined #kolla17:29
*** sdake has quit IRC17:30
sdake_yo ayoung17:32
sdake_thanks for the blog post mentioining kolla17:32
ayoungsdake_, so I watched a video17:32
sdake_helps to get the word out ;)17:32
*** sdake_ is now known as sdake17:33
sdakewhich one17:33
ayoungand I found out that Kolla does not rhyme with Holla17:33
sdakehaha17:33
sdakekoll-uh17:33
*** subscope has quit IRC17:33
ayoungCola17:33
sdakeright17:33
sdakepeople go both ways17:33
sdakeI actually don't know which is correct17:33
ayoungsdake, I so want Kolla to be the way we deploy the overclouds with Tripleo17:33
limamauriciosdake, did you see the traceback?17:34
sdakelimamauricio no i did not did you paste the full one?17:34
sdakeayoung rhallisey and jpeeler are working on that atm17:34
openstackgerritMerged openstack/kolla: Pin Mesos to 0.27.1 version  https://review.openstack.org/28309117:34
sdakebut possibly in a different way then you have in mind17:34
ayoungsdake, I know.  It was a rhallisey video that taught me the correct pronunciation.  I need to finish watching it17:34
rhalliseydaneyon_, convinced me it was cola :D17:35
ayoungI need to learn if I canreplace Puppet calls with Ansible for setting up Federation.17:35
sdakelimamauricio what distro are you in17:35
*** achanda has joined #kolla17:35
sdakeyou could probably fix it by pip install netifaces17:35
limamauricioubuntu17:36
limamauriciosource17:36
limamauriciook17:36
sdakeor apt-get python-netifaces17:36
sdakeor whatever that package is called in ubuntu17:36
sdakei'd go with a distro packaged version rathre then pip personally17:36
sdakeImportError: No module named netifaces17:36
sdakethis is why everything is bust17:36
sdakealso 404's on neutron here: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}17:37
sdakethat is a problem17:37
sdakelimamauricio did yo usource your openrc?17:37
limamauricioyes17:38
limamauricio:/17:38
limamauricioi'll try again17:38
*** ayoung has quit IRC17:39
*** achanda has quit IRC17:40
*** ayoung has joined #kolla17:40
*** vhosakot has joined #kolla17:40
*** SiRiuS__ has joined #kolla17:40
vhosakotthanks for the great comments team for the quickstart updates :)17:41
sdakehav a link?17:42
vhosakotsdake: https://review.openstack.org/28329817:42
vhosakotsdake: I got more comments than the changes :)17:42
vhosakotsdake: I assumed a brand new VM and a _very_ new user... like no git, no docker, no ansible, etc etc..17:43
sdakeya with quickstart changes, I recommend surgical improvements rather then complete rewrites ;)17:43
vhosakotsdake: cool, agreed :)17:44
*** kzaitsev_mb has quit IRC17:45
*** inc0_ has joined #kolla17:46
inc0_SamYaple, do you have moment?17:46
limamauriciosdake, apt-get python-netifaces worked for me17:48
limamauricio:)17:48
sdakelimamauricio happy to help17:48
limamauricioThanks17:48
*** inc0 has quit IRC17:48
SamYapleinc0_: shoot17:52
inc0_SamYaple, wanna take upgrade or rabbit or galleria?17:53
inc0_or galleria even17:54
inc0_galera fas17:54
inc0_AUTOCORRECT YOU DUMB M*****F****17:54
sdakeinc0_ if your on a mac you can turn autocorrect off17:54
sdakei do - it makes my life much easier17:55
sdakeyou can set it per app or systemwide17:55
inc0_problem is, I did17:55
sdakei just set it system wide17:55
sdakebuilds take f0rever!!17:56
inc0_ok, azure xchat has it's own setting for that17:56
inc0_SamYaple, for galera upgrades...is it any harder than just rolling upgrade of containers?17:57
SamYaplei have galera upgrades17:58
SamYaplei can also do rabbit.... or walk someone else through the tricky part17:59
sdakei can do rabbit if you explain the tricky parts18:00
sdakefolks should be working on essential, then high, then med, then low blueprints in priority order18:00
sdakeall the infrastructure upgrades are high18:01
sdakeSamYaple if you could do neutron instead of rabbit that would be fantastimical :)18:01
sdaketag is 2nd-4th whenever we think its most stable18:01
SamYapleill have mariadb and neutron done by this weekend18:03
SamYaplei might do rabbitmq too rather than explain it18:03
SamYapleclusterer makes it really easy18:04
*** spisarski1 has joined #kolla18:04
*** spisarski has quit IRC18:04
sdakeok whichever works - I guess I'll finish the job on drop-root since my hopes of people finishing up the last few things were dashed ;)18:04
sdakedocker engine 1.10.2 is out btw18:05
*** rmart04 has quit IRC18:07
*** achanda has joined #kolla18:07
*** achanda has quit IRC18:10
*** unicell has joined #kolla18:12
*** unicell has quit IRC18:12
*** unicell has joined #kolla18:12
SamYaplehey guys I dont want to see this half merged18:16
SamYaplehttps://review.openstack.org/#/c/282840/18:16
SamYapleeither we need to hammer out this work (which should have happened in liberty) in its entirety, or we shouldnt merge this18:17
sdakewhy are people filing bugs for blueprint work18:17
sdakeif anything semlls like a feature at this point and its nto in the tracker, please -2 it18:17
sdakeand explain it will be un -2'ed at start of newton18:18
*** inc0_ has quit IRC18:19
*** inc0 has joined #kolla18:23
sdakevhosakot re your docs change, can you submit a patch stream for each individual change, rather then a big change to all the docs18:24
sdakethat way changes can be selectively dropped18:24
sdakeby you via a git rebase18:24
sdakean alternative to this is to submit patches one at a time and deal with the merge conflicts as they occur18:25
sdakethere are about 10 different changes in that doc18:25
vhosakotsdake: yup, I was thinking the same too.. so, I will submit a separate patch set for each section and mention the section name in the commit mesage.. how does this sound ?18:26
sdakewfm18:26
openstackgerritMerged openstack/kolla-mesos: Make the memcached dependency optional  https://review.openstack.org/28331618:27
vhosakotsdake: cool... ther should not be any more emails "Kolla deploy errors", "kolla not coming up", blah blah :)18:27
openstackgerritMerged openstack/kolla: Fix vagrant bootstrapping for ubuntu  https://review.openstack.org/28306918:27
sdakeI can tell your highly optimistic :)18:28
*** gfidente has quit IRC18:28
sdake113 containers18:31
vhosakotsdake: I'm serious.. should be like DevStack... works the very first time right out of the box... neutron is also like that (atleast Liberty).. very stable... I think operators/users will be turned off if something does not work out of the box the very first time18:31
sdakei finally have a bare metal machine operational18:31
sdakevhosakot your correct about the part people don't give you alot of room in eval18:31
sdakefortunately eval of openstack gets a little more slack because its such a big investment for a company to make18:32
vhosakotsdake: yep... stability stability stability is important... kolla's end-result is great... I've playing with it the the past few days after it is up... we should address the newest/dumbest user...18:33
sdakethe proper term is naive18:34
sdake;)18:34
vhosakot:)18:34
sdake(to kolla)18:34
vhosakotsdake: cool :)18:35
*** dwalsh has joined #kolla18:36
*** achanda has joined #kolla18:37
*** achanda has quit IRC18:45
inc0SamYaple, since I'm done with haproxy, walk me through rabbitmq plz18:47
inc0I want to understand it anyway18:47
dave-mcc_ayoung ping i have a kolla/keystone question18:48
ayoungdave-mcc_, 1 dec, keystone meeting18:48
sdakequick q, this has always bothered me18:49
SamYapleinc0: one of the nodes is the "gospel node" it must be stopped last.18:49
sdakeand i'm curious if someone fixed in december when i was out18:49
SamYapleinc0: they can all be started in any order18:49
sdakedocker rebuilds all containers when you do a build and then a build --push (this rebuilds)18:50
sdakeis there any way to avoid the second rebuild18:50
SamYapleinc0: `rabbitmqctl eval 'rabbit_clusterer:status().'`18:50
sdakei know in the old days of ocker setting the mtime to 0 or consistently fixed this problem18:50
SamYaplethat will tell you the info needed to find gospel node18:50
SamYaplesdake: the mtime to 0 only fixes some issues18:50
SamYapleand it still doen18:50
SamYapledoes*18:51
sdakeagreed18:51
sdakebut not all18:51
*** inc0 has quit IRC18:51
SamYaplefor source builds it rebuilds all containers. period18:51
*** achanda has joined #kolla18:51
SamYaplenew tarballs get pushed up minuntely18:51
SamYapleor so ive found18:51
*** inc0 has joined #kolla18:51
*** dave-mcc_ is now known as dave-mccowan18:52
inc0SamYaple, network went to sleep -.- so rabbitmqctl eval gets me gospel node, this node dies last18:52
inc0anything else?18:52
SamYaplethats it!18:52
sdakeso the answer then is no, there isn't a way to push indepndent of build in build.py18:52
sdake?18:52
inc0simple enough then18:52
SamYaplehowever this only needs to happen when the rabbitmq cluster is different breaking version18:52
SamYapleerlang can often exist in the same cluster with different versions18:53
inc0for now let's assume worst case scenerio18:53
inc0so it is breaking18:53
SamYapleinitial upgrade script should just assume (like nova conductor) that if versions are different just hard stop18:53
SamYaplewe can get smarter latter18:53
SamYaplesdake: docker push <image>18:53
inc0ok will do18:53
inc0nihilifer, I'll assign rabbitmq upgrade to myself ok?18:54
*** athomas has quit IRC18:56
*** subscope has joined #kolla18:56
*** subscope has quit IRC19:08
sdakedocker push takes 6 minutes on my SSD gear19:11
sdake1.5 GB for ubuntu source19:11
sdakeconsumed by a docker bindmount19:11
*** jtriley has joined #kolla19:15
*** athomas has joined #kolla19:19
openstackgerritVikram Hosakote proposed openstack/kolla: Updated doc/quickstart.rst - Part 1  https://review.openstack.org/28329819:21
sdakeSamYaple what pops out of your mind when I say "MariaDB recovery playbook"19:23
vhosakotwill submit smaller updates in smaller patch sets so that reviewing is easier...19:25
vhosakotCould folks review part 1  -  https://review.openstack.org/283298 ?19:26
*** chandankumar has quit IRC19:26
vhosakotI will rebase and resubmit more patches19:27
*** diogogmt has joined #kolla19:30
*** jtriley has quit IRC19:35
sdakeubuntu binary has really come a long way19:37
sdakei dont know if it deploys but it mostly builds ;-)19:37
*** jtriley has joined #kolla19:38
*** Marga_ has quit IRC19:38
ayoungdave-mccowan, OK, what you got?19:38
dave-mccowanayoung kolla deploys keystone behind apache with wsgi.  https://github.com/openstack/kolla/blob/master/ansible/roles/keystone/templates/wsgi-keystone.conf.j219:40
ayoungdave-mccowan, yep19:40
dave-mccowanthe apache is eating the X-Forwarded-For header, so the remote ip address is not showing up the keystone logs.19:40
ayoungdave-mccowan, Om nom nom19:41
dave-mccowanwhat's the best practice line-of-config to add to that line to make it right?19:41
ayoungdave-mccowan, I'm not aware of it...let's see...19:42
SamYaplesdake: same work that pops out when i think "upgrade mariadb"19:42
dave-mccowansomething like "WSGITrustedHeaders X-Forwarded-For", but that's not right. or maybe i need to import a module19:42
SamYapleayoung: o/19:42
ayoungdave-mccowan, so...what does this break?19:44
*** salv-orlando has quit IRC19:44
ayoungdave-mccowan, is it this: http://blog.dscpl.com.au/2015/07/redirection-problems-when-proxying-to.html19:45
dave-mccowanThe keystone access log says that every request originats from the Kolla VIP address, instead of an accurate record of who made the requests.19:45
*** jtriley has quit IRC19:46
ayoungdave-mccowan, is the Kolla VIP also acting as an Apache proxy in this case, or is there a separate apache instance per service?19:46
ayoungdave-mccowan, does WSGITrustedProxyHeaders X-Forwarded-Server X-Forwarded-Port X-Forwarded-For  make a difference?19:47
dave-mccowanayoung haproxy catches all the requests at the VIP, the apache proxy is inside the container with keystone alone.19:48
dave-mccowanayoung that's what i expected to work, but apache didn't recognize that command.  is there an import module that goes with that?19:48
sdakeSamYaple thanks19:49
ayoungdave-mccowan, ah, that might be a more recent version of mod_wsgi?19:49
ayoung"I'm trying to use it in normal mod_wsgi which I compiled from source to version 4.4.13 (I'm not familar with mod_wsgi-express )"19:49
*** inc0 has quit IRC19:50
dave-mccowanyour first link made me think that Docker might also be doing some header eating... i'll have to check that next.19:50
SamYapledave-mccowan: i dont think so19:50
ayoungdave-mccowan, http://blog.dscpl.com.au/2015/06/proxying-to-python-web-application.html  and the follow up http://blog.dscpl.com.au/2015/07/redirection-problems-when-proxying-to.html  look informative, but I really don;t know.  Let me know what you find19:50
SamYapledave-mccowan: we use --net=host, so i wouldnt waste time on the docker front19:50
SamYaplei think you need to properly set it in hapoxy conf reading the issue so far19:51
SamYaple'option forward' is set in haproxy dave-mccowan?19:51
dave-mccowanSamYaple i have set haproxy conf correctly in my env.  Nova, Glance, etc.  all report the correct address now in the api logs.  but keystone does not.  i blame httpd.conf.19:51
SamYaplefair enough19:52
SamYapledave-mccowan: what patchset is this based around?19:52
dave-mccowanjust off master.  i opened a bug today that the VIP is the only reported address in kolla logs.19:53
dave-mccowani expect the fix to be two lines of template.  one to haproxy.cfg.j2 and one to keystone-wsgi.conf.j2.19:53
SamYapleah i just checked19:54
SamYapleoption forwardfor is the default19:54
dave-mccowaninteresting.  i don't think that's the behavior i saw, but maybe only looked at keystone before adding "option forwardfor".19:55
SamYapledave-mccowan: i see the issue19:56
SamYaplewsgi is not logging forwardfor :)19:56
SamYapleneed somethign like...19:56
SamYapleLogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy19:56
SamYaplei just pulled that example out of some old script i had19:56
SamYapleprobably got that on the interwebs somewhere19:57
SamYapleare you saying keystone.log should contain this info?19:57
dave-mccowanSamYaple ayoung thanks!  i'll test that out.19:57
dave-mccowanyes, i think that would be the correct behavior for an audit log.19:57
SamYaplewell in this case I think it will only show up in keystone-apache-public-access.log19:58
*** jtriley has joined #kolla20:00
*** inc0 has joined #kolla20:01
vhosakotSamYaple: After running "mount --make-shared /run" on Ubuntu, the neutorn_dhcp_agent container did not throw the APIError/HTTPError.  So should I mention the steps to delete the neutron_dhcp_agent container and rerun "kolla-ansible deploy"  ?20:04
vhosakotSamYaple: "mount --make-shared /run" is already in the quickstart20:04
SamYaplevhosakot: i had to do that, so at the very least its in consistent and i think it should be mentioned20:04
vhosakotSamYaple: along with "mount --make-shared /run"  ?20:05
vhosakotSamYaple: or set MountFlags to shared if you're on CentOS20:05
*** inc0 has quit IRC20:06
openstackgerritVikram Hosakote proposed openstack/kolla: Updated doc/quickstart.rst - Part 1  https://review.openstack.org/28329820:12
*** subscope has joined #kolla20:16
sdakegetting some of this when building from source INFO:__main__:nova-base:Ignoring python-ironic-inspector-client: markers "python_version=='3.4'" don't match your environment20:31
SamYapleyea thats normal20:35
SamYapleits just a warning20:35
sdakeSamYaple when I say "lights out mariadb recovery" what pops out?20:37
SamYaplethe same thing that popped out when you asked me an hour ago and i responded20:38
sdakecool20:38
sdakethey are not quite the same sentence which is why i asked twice ;-)20:38
sdakewe have a lights out mariadb recoveyr blueprint20:39
sdakedo we need a special playbook just for that, or are we expecting operators to run the upgrade playbook?20:39
sdakecurious if a recovery action makes sense20:40
*** jtriley has quit IRC20:43
*** jtriley has joined #kolla20:43
SamYaplerecovery playbooks makes sense for mariadb, not so much for rabbitmq or the rest20:46
SamYaplethat said upgrade playbook will probably work20:46
*** dwalsh has quit IRC20:48
sdakerabbitmq needs some other type of recovery mechanism, or none at all?20:49
SamYaplenone at all20:49
SamYapleclusterer is amazing20:49
*** jtriley has quit IRC20:52
*** dmsimard|afk is now known as dmsimard20:57
*** dims has quit IRC21:01
*** dims has joined #kolla21:03
*** unicell has quit IRC21:04
*** unicell has joined #kolla21:05
*** salv-orlando has joined #kolla21:05
openstackgerritVikram Hosakote proposed openstack/kolla: Updated doc/quickstart.rst - Part 1  https://review.openstack.org/28329821:08
*** unicell has quit IRC21:12
*** unicell has joined #kolla21:15
*** unicell has quit IRC21:25
*** salv-orl_ has joined #kolla21:27
*** salv-orlando has quit IRC21:30
*** rhallisey has quit IRC21:32
*** jtriley has joined #kolla21:35
*** sdake has quit IRC21:37
*** jtriley has quit IRC21:44
*** unicell has joined #kolla21:54
*** jtriley has joined #kolla22:00
*** dims has quit IRC22:02
vhosakotSamYaple: I added the timeout for 120 seconds in requests.get() in kolla/cmd/build.py and ran tox... when I run tox, I see py27 and pep8 test passed, but py34 and pypy tests failed... is it expected ?22:11
vhosakotSamYaple: I dont have python 3 and pypyp installed on my mchine22:12
vhosakotSamYaple: doesn't tox install py34 and pypy in venv ?22:12
*** kzaitsev_mb has joined #kolla22:12
vhosakotCan anyone confirm if tox passes in kolla ? py34 and pypy fail for me22:18
vhosakotpy27 and pep8 pass fine22:18
*** sdake has joined #kolla22:30
sdakeSamYaple good news about clusterer22:30
sdakejust brainstorming here, what else would this recovery action do?22:30
sdakebesides mariadb "upgrade" model22:31
sdakereorder the start of processes that we know cause problems when they are started out of order?22:31
sdakereorder/restart22:31
sdakecommonly getting this error  when building docker containers centos-source22:35
sdakeINFO:__main__:nova-network:[Errno 12] Cannot allocate memory: '/var/cache/yum/x86_64/7/base/mirrorlist.txt'22:35
*** unicell has quit IRC22:36
*** unicell has joined #kolla22:36
sdakethis may be the cause of the threaded build failing22:37
*** dims has joined #kolla22:40
sdakehttps://bugs.launchpad.net/kolla/+bug/154901922:43
openstackLaunchpad bug 1549019 in kolla "memory allocation errors on centos+source build" [Critical,Confirmed]22:43
SamYapleare your running out of memory sdake....22:46
sdake32gb available 4gb in use22:47
SamYaplewhen the error occurs?22:47
sdakedocker is taking 0.5% of memory22:47
SamYapleor after building stops22:47
sdakebut the systemd file has memory limits22:47
openstackgerritVikram Hosakote proposed openstack/kolla: Add timeout to requests.get() in kolla/cmd/build.py  https://review.openstack.org/28384422:48
sdakewhen the error occurs i have 4gb in use22:48
*** subscope has quit IRC22:48
sdakei am on docker 1.10.2 however22:48
sdakebut the other builds worked fine22:48
sdakeit seems to be source builds triggering the issue22:48
sdakei built ubuntu source22:48
sdakeubuntu binary22:48
sdakenow centos source is failing22:49
sdakeI ran with -T 3 this time22:49
sdakesame results22:49
SamYapleno idea22:49
sdakethis may be the same problem others were running into22:50
SamYapleim 1.10.2 ubuntu-source22:50
sdakeya seems wierd22:50
sdakelet me rekick the docker daemon22:50
sdakeits definatel not a memory leak22:50
sdakeor the docker process would show the memory above 0.5%22:51
sdakesystemd can limit memory usage22:51
sdakevia cgroups22:51
sdakeand docker has that nonsense in there22:51
*** rhallisey has joined #kolla22:51
sdakeits possible 1.10.2 has a memory leak that bumps upa gainst that limit22:51
sdakeya now docker is at 0.2% memory usage22:52
sdakeafter a restart22:52
sdakeand a build in progress22:52
sdake1.5gb virt usage22:52
sdakerather it looks like 1.5mb22:53
sdakedocker restart does not resolve the problem22:53
sdakenothing in dmesg to indicate prolbems22:54
sdakesystemd limits threads to 1 million threads and files to 1 million files22:56
sdakein the docker case22:56
sdakerunning on bare metal22:58
*** mkoderer has quit IRC22:58
*** diogogmt has quit IRC22:58
*** vhosakot has quit IRC22:58
*** mandre has quit IRC22:58
*** yuanying has quit IRC22:58
*** akscram has quit IRC22:58
*** gtt116_ has quit IRC22:58
*** macsz has quit IRC22:58
*** Allen_Gao has quit IRC22:58
*** asalkeld has quit IRC22:58
*** mdnadeem has quit IRC22:58
*** intr1nsic has quit IRC22:58
*** ajafo has quit IRC22:58
*** limamauricio has quit IRC22:58
*** dtturner has quit IRC22:58
*** rhallisey has quit IRC22:58
*** dims has quit IRC22:58
*** athomas has quit IRC22:58
*** ayoung has quit IRC22:58
*** blahRus has quit IRC22:58
*** pbourke has quit IRC22:58
*** elemoine_ has quit IRC22:58
*** logan- has quit IRC22:58
*** bradjones has quit IRC22:58
*** clayton has quit IRC22:58
*** SiRiuS__ has quit IRC22:58
*** JoseMello has quit IRC22:58
*** bmace has quit IRC22:58
*** daneyon_ has quit IRC22:58
*** zigo has quit IRC22:58
*** GheRivero has quit IRC22:58
*** nihilifer has quit IRC22:58
*** vincent_vdk has quit IRC22:58
*** zaneb has quit IRC22:58
*** salv-orl_ has quit IRC22:58
*** stvnoyes has quit IRC22:58
*** alisonh has quit IRC22:58
*** dmsimard has quit IRC22:58
*** tpeoples has quit IRC22:58
*** cloudnull has quit IRC22:58
*** Slower has quit IRC22:58
*** bigjools has quit IRC22:58
*** Kennan has quit IRC22:58
*** kbyrne has quit IRC22:58
*** kzaitsev_mb has quit IRC22:58
*** zeus has quit IRC22:58
*** slagle has quit IRC22:58
*** sean-k-mooney has quit IRC22:58
*** jmccarthy has quit IRC22:58
*** secate has quit IRC22:58
*** mkoderer__ has quit IRC22:58
*** elemoine has quit IRC22:58
*** kfox1111 has quit IRC22:58
*** dolpher has quit IRC22:58
*** jpeeler has quit IRC22:58
*** thomasem has quit IRC22:58
*** harmw has quit IRC22:58
*** serverascode has quit IRC22:58
*** ravla has quit IRC22:58
sdakeirc just broke22:58
*** sdake is now known as jpeeler22:58
*** jpeeler is now known as 64MAAECDI23:00
*** ravla has joined #kolla23:01
*** serverascode has joined #kolla23:01
*** harmw has joined #kolla23:01
*** thomasem has joined #kolla23:01
*** jpeeler has joined #kolla23:01
*** dolpher has joined #kolla23:01
*** ajafo has joined #kolla23:01
*** intr1nsic has joined #kolla23:01
*** mdnadeem has joined #kolla23:01
*** asalkeld has joined #kolla23:01
*** Allen_Gao has joined #kolla23:01
*** macsz has joined #kolla23:01
*** bradjones has joined #kolla23:01
*** clayton has joined #kolla23:01
*** logan- has joined #kolla23:01
*** elemoine_ has joined #kolla23:01
*** pbourke has joined #kolla23:01
*** blahRus has joined #kolla23:01
*** ayoung has joined #kolla23:01
*** athomas has joined #kolla23:01
*** dims has joined #kolla23:01
*** rhallisey has joined #kolla23:01
*** kbyrne has joined #kolla23:01
*** Kennan has joined #kolla23:01
*** bigjools has joined #kolla23:01
*** Slower has joined #kolla23:01
*** cloudnull has joined #kolla23:01
*** tpeoples has joined #kolla23:01
*** 64MAAECDI is now known as sdake23:01
*** SiRiuS__ has joined #kolla23:01
*** GheRivero has joined #kolla23:01
*** JoseMello has joined #kolla23:01
*** bmace has joined #kolla23:01
*** daneyon_ has joined #kolla23:01
*** zigo has joined #kolla23:01
*** nihilifer has joined #kolla23:01
*** vincent_vdk has joined #kolla23:01
*** zaneb has joined #kolla23:01
*** mkoderer has joined #kolla23:01
*** diogogmt has joined #kolla23:02
*** vhosakot has joined #kolla23:02
*** mandre has joined #kolla23:02
*** yuanying has joined #kolla23:02
*** akscram has joined #kolla23:02
*** gtt116_ has joined #kolla23:02
*** vhosakot has quit IRC23:02
*** jmccarthy has joined #kolla23:03
*** secate has joined #kolla23:03
*** mkoderer__ has joined #kolla23:03
*** elemoine has joined #kolla23:03
*** kfox1111 has joined #kolla23:03
*** zeus has joined #kolla23:03
*** slagle has joined #kolla23:03
*** sean-k-mooney has joined #kolla23:03
*** limamauricio has joined #kolla23:03
*** dtturner has joined #kolla23:03
*** GheRivero has quit IRC23:04
*** jtriley has quit IRC23:06
*** salv-orl_ has joined #kolla23:06
*** stvnoyes has joined #kolla23:06
*** alisonh has joined #kolla23:06
*** dmsimard has joined #kolla23:06
*** dmsimard1 has joined #kolla23:06
*** dmsimard has quit IRC23:07
*** dmsimard1 is now known as dmsimard23:07
*** exploreshaifali has joined #kolla23:08
*** exploreshaifali has quit IRC23:08
*** Marga_ has joined #kolla23:09
*** Marga_ has quit IRC23:15
*** Marga_ has joined #kolla23:16
*** SiRiuS__ has quit IRC23:18
sdakeSamYaple well if I run build.py --base centos --type source things go balistic23:24
sdakeif I run build.py --type source things work fine23:24
sdakeit looks like the image prefix is not being set properly23:24
sdakebut looking at the code, it looks correct23:24
sdakeanyway I will have to dig into debugging that at a later time today as dinner is incoming23:25
*** GheRivero has joined #kolla23:31
*** JoseMello has quit IRC23:32
*** diogogmt has quit IRC23:36
*** haplo37 has quit IRC23:49
*** kzaitsev_mb has joined #kolla23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!