Thursday, 2016-02-04

*** alisonh has joined #kolla00:02
*** JoseMello has quit IRC00:08
*** yuanying has quit IRC00:09
*** yuanying has joined #kolla00:16
*** jasonsb has joined #kolla00:16
*** tzn has joined #kolla00:19
*** yuanying has quit IRC00:19
*** yuanying has joined #kolla00:22
*** achanda has joined #kolla00:27
*** tzn has quit IRC00:35
*** sdake has joined #kolla00:35
*** ssurana has quit IRC00:37
*** farid has quit IRC00:37
*** sdake_ has joined #kolla00:50
*** sdake has quit IRC00:54
*** Allen_Gao has joined #kolla00:54
SamYapleinc0: here00:55
SamYapleinc0: you still around?00:55
inc0yeah00:55
inc0what was that about?00:55
SamYapleok so you want bad news first?00:56
inc0bring it on00:56
SamYapleasalkeld: yes things changed, yes ive seen that before, ill tell you in a second00:56
SamYapleok inc000:56
SamYaplesuccessfully removed and used a _new_ libvirt image, no loss of qemu process00:56
SamYapleissue is the new libvirt contaienr did _not_ have a record of the running qemu instance00:57
SamYaplethe reason for this is we are missing a file mount (i beielve) with needed info00:57
SamYaplethats the good news I suppose00:57
SamYaplebad news is it _still_ seems well have downtime with this issue00:58
SamYaplebut we won't have downtime moving foward00:58
inc0so, we might be able to reconnect to qemu00:58
inc0if we find out exactly what libvirt does to reconnect00:58
SamYaplethere is a reconnect command00:58
SamYaplevirsh qemu-reconnect <pid>00:59
SamYaplebut thats not the whole picture00:59
*** Kennan is now known as Kennan_Vacation00:59
SamYapleim working on identifying the issue for loss of info in libvirt00:59
SamYaplebut i think it will still require a hard downtime to fix00:59
SamYaplenot positive though00:59
inc0I'd assume libvirt has some qemu-pid file somewhere00:59
SamYaplevirsh qemu-attach <pid> ***00:59
SamYaplei know the pid01:00
SamYapleit didnt reattach01:00
inc0and it disapears with container01:00
SamYaplelike I said, its not wanting to reattach01:00
inc0you know, but new libvirt doesn't01:00
SamYapleyes it does01:00
SamYaplei ran that command01:00
SamYapleit errors01:00
inc0ah, I see01:00
inc0any helpful error?01:00
*** jtriley has joined #kolla01:00
inc0I'll ask around if anyone in my glorious company had anything to do with libvirt, maybe they could help01:01
SamYapleasalkeld: that issue is a keystone v2/v3 issue. i remember it from when i rwote the module01:01
SamYapleinc0: kinda. im still working through it01:01
SamYapleits possible, thats all ill say01:02
inc0well, it's hopeful01:02
SamYapleits definetly happening long term01:02
inc0does it attach if we move libvirt outside?01:02
SamYaplebut since we missed needed info we may need a hard downtime to fix01:02
SamYaplenah but thats another issue01:02
SamYaplesince it doenst have access to nova mountpoint01:02
inc0a yeah01:02
SamYapleit would otherwise01:03
inc0well it kinda does if we navigate through /var/lib/docker01:03
SamYaplebut then again we wouldnt need to restart libvirt ;)01:03
inc0apart from possible fs problems01:03
SamYapleit would work apart from that yes01:03
inc0but we're not doing it, too ugly even for me01:03
inc0and can very well fail horribly01:04
SamYaplethats a preq for libvirt on the host just so you know01:04
SamYaplebind mounting host /var/lib/nova and /var/lib/libvirt01:04
inc0another issue01:04
inc0to discuss over a beer01:04
SamYapleeh you say ugly I say useful01:05
SamYaplethats how yaodu did it01:05
SamYapleim not saying its better than what we have, but its not _bad_01:05
inc0I'm saying navigating through /var/lib/docker is ugly, bindmountig is ok01:05
SamYapleOHHH01:05
SamYapleyea i agree01:05
SamYapleno /var/lib/docker01:05
inc0so we need to figure out how exactly libvirt attach works01:06
inc0maybe there is indeed missing file, but one we can scoop out before tearing down old container01:06
SamYaplethats my thinking01:06
SamYapleim searching for said files01:06
SamYapleno luck01:06
inc0technically containers are immutable, and since it's reattaching it doesn't keep anything in memory01:06
SamYapleall i know about are files in /var/lib/libvirt and /var/lib/nova01:06
SamYapleits likely a db somewhere i dont know about01:07
inc0I'll look for someone who knows libvirt01:07
SamYaplei was at dinner right after discovery above01:08
SamYapleso im checking again01:08
asalkeldthanks SamYaple01:09
*** neilus has joined #kolla01:10
SamYapleasalkeld: my goal is to switch those modules over to the shade modules before mitaka01:10
SamYaplebut thats just playbook changes01:10
SamYapleyou can do that in mesos right now01:10
asalkeldok01:10
asalkeldi think longer term we want to just not use ansible for this stuff01:11
*** yuanying has quit IRC01:11
*** yuanying has joined #kolla01:12
SamYapleasalkeld: yea honestly mesos is way more kolla-ansibly than i thought it would be01:12
*** iNeilus has quit IRC01:12
SamYaplefrankly you guys could just call the openstackclient inside the kolla-ansible container and be done with it01:13
asalkeldyip01:15
*** CheKoLyN has quit IRC01:19
*** yuanying has quit IRC01:21
*** Jeffrey4l has joined #kolla01:22
*** yuanying has joined #kolla01:23
openstackgerritSam Yaple proposed openstack/kolla: Convert Neutron to thin containers  https://review.openstack.org/27271501:28
openstackgerritSam Yaple proposed openstack/kolla: Correct issue with virtualenv rootwrap  https://review.openstack.org/27550401:28
*** sdake_ has quit IRC01:33
openstackgerritSam Yaple proposed openstack/kolla: Convert Neutron to thin containers  https://review.openstack.org/27271501:37
*** Jeffrey4l has quit IRC01:39
*** tfukushima has joined #kolla01:40
*** alisonh has quit IRC01:47
*** alisonh has joined #kolla01:51
*** farid has joined #kolla02:03
Allen_GaoSamYaple: when attaching qemu process to libvirt,  *   - The '-name' and '-uuid' arguments should have been set (not mandatory, but strongly recommended)02:10
Allen_GaoSamYaple: And FYI, It must have been started with a monitor socket using the UNIX   domain socket protocol.02:10
Allen_GaoSamYaple: And ensure that No device hotplug/unplug, or other configuration changes can   have been made via the monitor since it started.02:12
SamYapleAllen_Gao: yup02:15
SamYapletwas the case here02:15
SamYaplebut now rabbitmq is breaking for me so i cant retest02:15
*** inc0 has quit IRC02:41
*** tfukushima has quit IRC02:49
*** tfukushima has joined #kolla03:01
*** sdake has joined #kolla03:01
*** unicell has quit IRC03:06
*** yuanying has quit IRC03:15
*** yuanying has joined #kolla03:19
*** yuanying has quit IRC03:24
*** jtriley has quit IRC03:28
*** thumpba has joined #kolla03:34
*** yuanying has joined #kolla03:35
*** dims has joined #kolla03:37
*** yuanying_ has joined #kolla03:41
*** Jeffrey4l has joined #kolla03:42
*** farid has quit IRC03:43
*** yuanying has quit IRC03:44
*** sdake has quit IRC03:45
*** sdake has joined #kolla03:52
*** dims has quit IRC03:53
*** thumpba has quit IRC03:53
*** thumpba has joined #kolla03:54
*** salv-orl_ has joined #kolla03:54
*** salv-orlando has quit IRC03:58
*** yuanying_ has quit IRC04:06
*** yuanying has joined #kolla04:07
*** yuanying has quit IRC04:08
*** yuanying has joined #kolla04:08
*** jasonsb has quit IRC04:09
*** jasonsb has joined #kolla04:10
*** daneyon_ has joined #kolla04:11
*** macsz has quit IRC04:12
*** thumpba_ has joined #kolla04:12
*** daneyon has quit IRC04:12
*** tpeoples has quit IRC04:12
*** macsz has joined #kolla04:13
*** thumpba has quit IRC04:13
*** sdake has quit IRC04:13
*** Allen_Gao has quit IRC04:13
*** jpeeler has quit IRC04:13
*** emperor has quit IRC04:14
*** jasonsb has quit IRC04:15
*** jasonsb has joined #kolla04:16
*** tpeoples has joined #kolla04:16
*** bradjones has quit IRC04:19
*** elemoine has quit IRC04:19
*** Allen_Gao has joined #kolla04:19
*** Jeffrey4l_ has joined #kolla04:20
*** elemoine has joined #kolla04:20
*** jpeeler has joined #kolla04:21
*** bradjones has joined #kolla04:21
*** bradjones has quit IRC04:21
*** bradjones has joined #kolla04:21
*** thumpba_ has quit IRC04:25
*** tfukushima has quit IRC04:34
*** Allen_Gao has left #kolla05:04
*** Allen_Gao has joined #kolla05:04
*** dratushnyy has joined #kolla05:14
*** chandankumar has joined #kolla05:34
*** tfukushima has joined #kolla05:34
*** tfukushima has quit IRC05:40
*** sdake has joined #kolla05:40
dratushnyyHi05:43
*** tfukushima has joined #kolla05:44
*** sdake_ has joined #kolla05:45
*** sdake has quit IRC05:45
kjellyHi dratushnyy05:49
*** sdake_ has quit IRC06:13
*** sdake has joined #kolla06:13
*** dratushnyy has quit IRC06:17
*** dratushnyy has joined #kolla06:18
*** dratushnyy has quit IRC06:18
*** dave-mccowan has quit IRC06:30
*** dwalsh has joined #kolla06:33
*** Marga_ has joined #kolla06:56
*** dwalsh has quit IRC07:03
*** dratushnyy has joined #kolla07:12
*** salv-orl_ has quit IRC07:13
*** Jeffrey4l has quit IRC07:19
*** Jeffrey4l has joined #kolla07:32
*** unicell has joined #kolla07:37
*** sdake has quit IRC07:56
openstackgerritMerged openstack/kolla: Add double quotes in ceph_cache_mode  https://review.openstack.org/27585807:57
*** fgimenez has joined #kolla08:06
*** Marga_ has quit IRC08:10
*** Marga_ has joined #kolla08:10
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510908:19
*** dwalsh has joined #kolla08:21
*** akwasnie has joined #kolla08:23
*** gfidente has joined #kolla08:24
openstackgerritMerged openstack/kolla-mesos: Add requirements.txt to venv  https://review.openstack.org/27591608:25
*** sdake has joined #kolla08:26
*** jmccarthy has joined #kolla08:27
*** neilus has quit IRC08:41
*** akwasnie has quit IRC08:44
*** neilus has joined #kolla08:44
*** ppetit has joined #kolla08:44
*** achanda has quit IRC08:45
*** akwasnie has joined #kolla08:46
openstackgerritJeffrey Zhang proposed openstack/kolla: Move the install base type check to kolla-build  https://review.openstack.org/27462908:48
*** achanda has joined #kolla08:48
*** neilus has quit IRC08:49
*** Allen_Gao has quit IRC08:56
*** chandankumar has quit IRC08:56
*** rhallisey has joined #kolla09:04
*** dratushnyy has quit IRC09:09
*** Allen_Gao has joined #kolla09:12
*** macsz has quit IRC09:14
*** macsz has joined #kolla09:15
*** esikachev has joined #kolla09:15
ajafoo/09:15
openstackgerritAlicja Kwasniewska proposed openstack/kolla: Added Elasticearch and its deployment.  https://review.openstack.org/26771409:17
*** openstackgerrit has quit IRC09:17
*** openstackgerrit has joined #kolla09:17
akwasnieajafo: hi:)09:21
*** mbound has joined #kolla09:24
elemoinehi09:24
akwasniehi elemoine09:26
elemoinehi akwasnie09:26
elemoineakwasnie, looking forwarding to seeing your ES code merged, to start Heka/ES/Kibana integration09:30
*** dwalsh has quit IRC09:30
akwasnieelemoine: me too and looking forward to see how heka works with ES and kibana!:) you've done a lot good work with heka already09:33
elemoinethanks09:33
*** achanda has quit IRC09:33
elemoinein the future we may need more specific configurations for ES and Kibana09:33
elemoinefor example for ES we may want to define what fields are analyzed and not analyzed09:34
akwasnieyes, we should improve it, but I think that it can be done in next steps09:36
elemoineright, that's also the way I see it09:37
akwasnieI will be glad to help you with Heka/ES/Kibana integration and improvement, if you need any help09:38
elemoineexcellent09:40
elemoinewe can start the integration when ES is merged09:40
akwasnieyes09:41
*** kjelly is now known as kejlly_09:42
*** Jeffrey4l has quit IRC09:43
*** tzn has joined #kolla09:45
*** Jeffrey4l has joined #kolla10:01
*** gfidente has quit IRC10:10
*** gfidente has joined #kolla10:15
*** gfidente has quit IRC10:15
*** gfidente has joined #kolla10:15
*** dwalsh has joined #kolla10:27
*** Jeffrey4l has quit IRC10:29
*** neilus has joined #kolla10:29
*** tfukushima has quit IRC10:30
*** tfukushima has joined #kolla10:32
*** neilus has quit IRC10:35
*** dwalsh has quit IRC10:40
*** dwalsh has joined #kolla10:40
*** fgimenez has quit IRC10:41
*** fgimenez has joined #kolla10:42
*** sdake has quit IRC10:44
*** tzn has quit IRC10:48
*** Allen_Gao has quit IRC10:48
*** esikachev has quit IRC10:52
*** esikachev has joined #kolla10:53
*** akwasnie has quit IRC10:53
*** salv-orlando has joined #kolla10:54
*** dwalsh has quit IRC11:01
*** tfukushima has quit IRC11:04
*** Allen_Gao has joined #kolla11:06
*** dims has joined #kolla11:12
*** egonzalez90 has joined #kolla11:23
*** egonzalez90 is now known as egonzalez11:24
openstackgerritMerged openstack/kolla: gnocchi is busted missing tooz for source  https://review.openstack.org/27589811:24
*** ppetit has quit IRC11:25
*** kproskurin has joined #kolla11:25
openstackgerritMerged openstack/kolla: Move the install base type check to kolla-build  https://review.openstack.org/27462911:31
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510911:36
pbourkeSamYaple: can we sway you on https://review.openstack.org/#/c/26821111:37
openstackgerritMD NADEEM proposed openstack/kolla: Update kernel version in ubuntu-bootstrap.sh  https://review.openstack.org/27617811:42
openstackgerritSam Yaple proposed openstack/kolla: Revert "fix miss rabbitmq config when hostname include special character"  https://review.openstack.org/27618011:47
SamYapleuughhhhhh11:47
SamYaplepbourke: Jeffrey4l_ you both approved the initial patch, can you approve the revert? https://review.openstack.org/#/c/276180/11:48
SamYaplei never even saw the patch in the queue or I would hae blocked :/11:48
SamYaplethis is why my rabbitmq has been busted for two days11:48
pbourke:(11:48
pbourkewho cares about ubuntu11:48
SamYapleohly everyone11:48
pbourkewhy did all the gates go green11:49
SamYapleit breaks on certain ubuntu kernels11:50
SamYaplebecause erlang has direct kernel talkings11:50
*** fgimenez has quit IRC11:51
SamYaplepbourke: https://github.com/openstack/kolla/blob/master/ansible/roles/rabbitmq/templates/rabbitmq.config.j2#L211:52
*** fgimenez has joined #kolla11:53
*** fgimenez has quit IRC11:53
*** fgimenez has joined #kolla11:53
SamYaplepbourke: official guide has no quotes, no quotes works for me, quotes breaks for me. need the revert11:54
pbourkeyup that's fine11:55
pbourkeunfortunate for the guy who has quotes in his hostname though11:55
SamYaplei have dashes in my hostname...11:56
SamYaplehttps://bugs.launchpad.net/kolla/+bug/154023411:56
openstackLaunchpad bug 1540234 in kolla "rabbitmq cluster is not work" [Undecided,Confirmed] - Assigned to weiyu (weiyu)11:56
SamYaplesee last comment11:56
SamYapleits literally the inverse of the above comments11:56
pbourkeok that's good to know11:56
SamYaplehe is using centos though11:57
limamauricioCan I deploy ceph all-in-one?11:58
limamauricioAll running only in one vm11:59
SamYaplelimamauricio: yes, but it requires a bit more knowleged about ceph11:59
limamauricio:/11:59
SamYaplespecfically you need to set the replicas to 1 or update teh failure domain of the object to osd instead of host11:59
limamauricioSam, can you see this error http://pastebin.com/zgvCppsy ?12:02
limamauricioI'm following this guide http://docs.openstack.org/developer/kolla/ceph-guide.html12:02
limamauriciobut I have it now :/12:02
SamYaplelimamauricio: are you on a completely clean environment? no leftover contaienrs, configs, or volumes from a previous deploy?12:02
SamYaple`docker volume ls`12:03
SamYaple`ls /etc/kolla`12:03
SamYaple`docker ps -a`12:03
*** neilus has joined #kolla12:03
SamYapleonce you get the evironment clean the above commands shouldn't show _any_ reference to ceph12:03
limamauricioI' have to clean all?12:03
SamYapleat least teh ceph stuff12:03
SamYaplewhat you see if from a failed parital deploy12:04
limamauricioOk12:04
*** athomas has joined #kolla12:07
*** neilus has quit IRC12:08
*** Allen_Gao has quit IRC12:17
*** dwalsh has joined #kolla12:19
openstackgerritSam Yaple proposed openstack/kolla: Make libvirt reestablish connection with qemu  https://review.openstack.org/27619312:22
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510912:27
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Switch from mysql-python to pymysql everywhere  https://review.openstack.org/27619412:27
*** Allen_Gao has joined #kolla12:30
limamauricioSam, monitor data directory at '/var/lib/ceph/mon/ceph-localhost' does not exist: have you run 'mkfs'?12:32
*** salv-orlando has quit IRC12:32
limamauricioi ran mkfs.ext412:32
limamauriciomkfs.ext4 /dev/sdb :/12:32
*** iceyao has joined #kolla12:34
SamYaplelimamauricio: no thats for the osd12:34
SamYaplethe mon doesnt need a disk like that12:35
limamauricioI need set enable_ceph_rgw: "yes"?12:36
limamauricioand create /etc/kolla/config/ceph.conf file?12:37
SamYapleif you want to use the radosgw you do12:37
SamYaplebut its not required12:37
limamauriciook12:38
*** iceyao has quit IRC12:38
openstackgerritMD NADEEM proposed openstack/kolla: Update kernel version in ubuntu-bootstrap.sh  https://review.openstack.org/27617812:41
*** salv-orlando has joined #kolla12:46
*** mkoderer has quit IRC12:49
*** vincent_1dk is now known as vincent_vdk12:49
*** vincent_vdk has quit IRC12:49
*** vincent_vdk has joined #kolla12:49
*** elemoine- has joined #kolla12:52
*** dave-mccowan has joined #kolla12:53
*** mdnadeem has quit IRC12:53
*** elemoine has quit IRC12:53
*** mdnadeem has joined #kolla12:54
*** egonzalez has quit IRC12:58
*** mkoderer has joined #kolla13:03
*** iceyao has joined #kolla13:04
*** dwalsh has quit IRC13:08
*** iceyao has quit IRC13:14
*** inc0 has joined #kolla13:21
inc0hey good morning13:21
inc0SamYaple, up and about already?13:22
*** dwalsh has joined #kolla13:25
limamauricioSomeone did kolla-ansible deploy with ceph works?13:28
limamauricioall-in-one13:29
inc0aio ceph needs additional config13:29
limamauricioI'm following this guide http://docs.openstack.org/developer/kolla/ceph-guide.html13:29
limamauricioI never worked with ceph13:30
inc0at the bottom there is paragraph about aio13:30
inc0in Managing ceph13:31
limamauricioWhen I ran kolla-ansible deploy I always get the same error13:31
limamauricioTASK: [glance | Pulling cephx keyring]13:31
inc0because ceph doesn't work in aio without additional config, I just said that:P13:31
inc0that's normal and expected behaviour13:31
inc0ceph normally isn't prepared to be aio13:31
inc0you can do this by following ceph guide, there is paragraph in "Managing ceph" about that13:32
limamauriciook13:32
*** gfidente has quit IRC13:39
openstackgerritvenkatamahesh proposed openstack/kolla: Use uppercase 'S' in word "OpenStack"  https://review.openstack.org/27623013:43
*** gfidente has joined #kolla13:47
*** gfidente has quit IRC13:47
*** gfidente has joined #kolla13:47
*** Jeffrey4l has joined #kolla13:48
*** sdake has joined #kolla13:50
*** neilus has joined #kolla13:51
*** neilus has quit IRC13:56
*** Allen_Gao has quit IRC13:57
mdnadeemajafo, ping13:58
mdnadeemajafo, I am new to kolla and looking for to start contribution in kolla, would you mind if i can help you to complete this bp https://blueprints.launchpad.net/kolla/+spec/binary-ubuntu ?14:01
limamauricioDid someone here deploy ceph aio successfully? If yes, please tell me how.14:03
*** dwalsh has quit IRC14:03
limamauricioI did the additional config, but not works.14:04
*** barra204 has quit IRC14:04
*** dwalsh has joined #kolla14:06
*** shakamunyi has joined #kolla14:06
*** fgimenez has quit IRC14:06
*** fgimenez has joined #kolla14:10
*** fgimenez has quit IRC14:10
*** fgimenez has joined #kolla14:10
*** dims_ has joined #kolla14:10
ajafomdnadeem: I had to leave this bp, at the moment but sure if you can it would be nice14:11
ajafomdnadeem: this can be little helpful https://review.openstack.org/#/c/260069/14:11
*** dims has quit IRC14:12
inc0SamYaple, so quick question, I did spawn vm, killed docker and started docker, vm dies as any other process14:17
inc0if it is not the case for you, maybe they 'broke' something in 1.10 which helps14:18
*** tzn has joined #kolla14:20
*** salv-orlando has quit IRC14:20
rhalliseyinc0, does this only happen in 1.10?14:21
inc0not sure14:22
inc0I'm on 1.9 and vm dies14:22
rhalliseyinc0, reason I'm asking is I probably going to help the RHT docker people debug this and I'd like to pin the version14:22
inc0Sam told me yesteday that it does not14:22
inc0there might be other causes tho, not sure yet14:22
rhalliseyinc0, kk14:23
inc0we need to pinpoint it14:23
inc0about version pin, you will need at least 1.1014:23
inc0docker14:23
inc0is it available in centos?14:23
inc0I mean will it be by kolla release?14:23
*** salv-orlando has joined #kolla14:24
rhalliseysorry didn't mean pin the version like that14:24
*** tzn has quit IRC14:24
inc0thin neutron container forces at least 1.1014:24
rhalliseypinpoint would be more accurate14:24
inc0ok14:24
rhalliseybut pin agree if there is an issue14:24
inc0yeah, we need to check that one out14:24
openstackgerritvenkatamahesh proposed openstack/kolla: Use uppercase 'S' in word "OpenStack"  https://review.openstack.org/27623014:24
*** tzn has joined #kolla14:26
sdakeSamYaple around?14:27
sdakeinc0 SamYaple can both of you look at https://bugs.launchpad.net/kolla/+bug/154186214:27
openstackLaunchpad bug 1541862 in kolla "stable/liberty vagrant aio fails at "Creating haproxy mysql user"" [Undecided,New]14:27
inc0Chris already answered and it seems to be good, let me double-check tho14:28
inc0ah14:29
inc0wrong, it seems he pinpointed problem but vagrant need fixing14:29
*** tzn has quit IRC14:35
sdakei see its a dupe14:37
sdakei dont think vagrant has anything to do with it14:37
sdakehere is the master dupe https://bugs.launchpad.net/kolla/+bug/154186214:37
openstackLaunchpad bug 1541862 in kolla "stable/liberty fails at "Creating haproxy mysql user" because of module error" [Critical,Triaged]14:37
sdakewe need a stable target in launchpad i guess14:38
*** ftcpops has joined #kolla14:49
inc0I should start using vagrant15:00
inc0I keep hearing lots of problems with it and can't help with them15:00
sdakehalf our dev team uses vagrant15:01
sdakebut inc0 i feel the same way15:01
*** daneyon_ has quit IRC15:01
inc0I never did and old habits die hard I guess15:01
sdakeinc0 https://bugs.launchpad.net/kolla/liberty/+bug/153096215:04
openstackLaunchpad bug 1530962 in kolla liberty "mariadb "Creating haproxy mysql user" task failing in stable/liberty" [Critical,Confirmed]15:04
pbourkeinc0: there's a reason you keep hearing lots of problems with it ;)15:05
sdakehttps://bugs.launchpad.net/kolla/liberty/+bug/153096215:05
dmsimardnihilifer: that's your commit, right ? https://review.openstack.org/#/c/233939/ (sorry if it's another Michal)15:06
inc0yeah its nihilifer15:06
dmsimardthere are still build failures in gate jobs that this commit was supposed to prevent15:08
dmsimardand funny thing is that jenkins is still returning +115:08
dmsimardhere's an example: http://logs.openstack.org/63/275463/3/check/gate-kolla-dsvm-build-centos-binary/1a707b4/console.html#_2016-02-03_16_44_39_096 (scroll to the end)15:08
inc0sdake, https://github.com/openstack/kolla/blob/stable/liberty/requirements.txt we don't even have six there15:08
dmsimardam I missing something here ?15:08
dmsimardI just looked at the centos binary job from https://review.openstack.org/#/c/275463/15:09
dmsimardhttp://logs.openstack.org/63/275463/3/check/gate-kolla-dsvm-build-centos-binary/1a707b4/console.html#_2016-02-03_16_27_45_997 is returning failure because mistral is not available as centos binary so I would've expected https://github.com/openstack/kolla/blob/75343870e395e257c8d8267ccd9c74e70c7fceb5/tests/test_build.py#L74 to prevent that from building in the first place15:11
dmsimardor wait, it doesn't prevent it from building, it ignores if there is an error building it15:12
sdakeinc0 someone backported something requiring six15:13
sdakethat is option a15:13
sdakeor option b is15:13
sdakethe kolla-ansible dockerfile six deepndency changed in some way15:14
sdakeSamYaple used a specific tag for ansible15:14
sdakeso it should never break15:14
sdakebut it is15:14
inc0one of deps in requirements might have dependency on six and version changed15:15
inc0bottom line, we need to find version of six that works and pin to it15:15
sdakeinc0 its happening in the kolla_ansible module15:18
sdakenot in ansible15:18
*** asalkeld has quit IRC15:20
inc0I'd say shade is problematic15:20
inc0we dont explicitly install six in kolla-ansible, but we use shade and we don't pin version on it15:20
*** mimizone has joined #kolla15:21
sdakethere is a worka round in the bug15:21
openstackgerritEvgeny Sikachev proposed openstack/kolla-mesos: Add unit-tests for chronos.py  https://review.openstack.org/27629815:24
*** blahRus has joined #kolla15:26
*** fgimenez has quit IRC15:28
*** akwasnie has joined #kolla15:28
*** fgimenez has joined #kolla15:28
*** achanda has joined #kolla15:30
*** jtriley has joined #kolla15:37
inc0sdake, I see it now15:38
limamauricioSamYaple, now i have an error in this task: [ceph | Bootstrapping Ceph OSDs]15:38
inc0we install git ansible instead of pip install ansible15:38
limamauricio:/15:38
inc0and it was alpha15:38
limamauricioi'm using centos15:39
inc0ansible 2.0 is released so now we should just install it with pip15:39
*** neilus has joined #kolla15:40
*** achanda has quit IRC15:42
limamauriciofailed: [localhost] => (item=(0, {u'device': u'/dev/sdb', u'fs_uuid': u''})) => {"changed": true, "failed": true, "item": [0, {"device": "/dev/sdb", "fs_uuid": ""}]} msg: Container exited with non-zero return code15:42
*** jtriley has quit IRC15:43
*** dratushnyy has joined #kolla15:44
*** neilus has quit IRC15:44
*** tzn has joined #kolla15:46
*** alyson_ has joined #kolla15:48
*** alyson_ has quit IRC15:48
*** JoseMello has joined #kolla15:48
*** alyson_ has joined #kolla15:49
*** tzn has quit IRC15:57
SamYapleinc0: ping15:58
inc0SamYaple, here, qemu gets restarted for me :/15:58
*** jtriley has joined #kolla15:58
SamYapleinc0: I have 100% confirmed on latest head it does _not_ get restarted at all for me15:59
SamYapleqemu processes stick around15:59
SamYapleworld is good15:59
inc0I believe you, I wonder what's wrong with my case15:59
inc0maybe they changes something in 1.1015:59
inc0I'm running 1.915:59
SamYapleno i checked 1.9 as well15:59
SamYapleand 1.8 for that matter16:00
inc0so here is what I do, I run vm, ps aux | grep qemu16:00
inc0then I watch vm pid16:00
inc0if I do docker rm -f nova_libvirt pid dies16:00
inc0if I kill -9 docker, pid stays, but then if I start docker, pid dies16:00
SamYapledoes pid actually die, or does it get a new pid?16:00
SamYapledoes qemu process survive is my question16:00
inc0I'm checking pid16:01
*** esikachev has quit IRC16:01
inc0also libvirt shows qemu instance as down16:01
inc0after restart of container16:01
SamYaplei have a patch for that16:01
inc0I've seen, +2 it16:01
inc0I was sure we do share /run16:01
inc0well we used to;)16:02
inc0SamYaple, you use qemu+kvm or just qemu?16:02
SamYapleinc0: +kvm16:03
SamYapleand as i thought ll along _that_ might be the key16:03
SamYaplethe kernel tie in16:04
SamYaplebut if that is the case maybe we can replicate it16:04
inc0I run just qemu, so yeah16:04
SamYaplei can test just kvm16:04
SamYaplejust qemu*16:04
inc0yeah, do that16:04
inc0trying just kvm would be hard;)16:04
SamYaplehey can you give me a +2 on this revert inc0 ? Its killing me and my testing right now https://review.openstack.org/#/c/276180/16:05
sdake2142 commits in liberty, 2896 commits in master16:05
sdakeour velocity is off the hook16:05
inc0we have merge fest tho:P16:05
sdakeinc016:06
sdakeshall I take careof that ansible 2.0.0 thing then?16:06
sdakeI need it done asap16:06
inc0I'm rebuilding containers now16:06
sdakeor are you working on it already16:06
inc0for tests16:06
SamYaplesdake: what ansible thing?16:06
inc0SamYaple, https://bugs.launchpad.net/kolla/liberty/+bug/153096216:06
openstackLaunchpad bug 1530962 in kolla liberty "six version dependency in kolla_ansible appears to be incorrect." [Critical,Confirmed]16:06
sdakeSamYaple https://bugs.launchpad.net/kolla/liberty/+bug/153096216:06
sdakeyup16:07
SamYapleoh yea ive seen that16:07
sdakestable is broken16:07
inc0I'm pinning ansible to 2.0.0.2 and will test master16:07
sdakehave demo of stable in 3 days16:07
SamYaplethats from an ordering issue16:07
SamYapleon installs16:07
SamYaplesee https://review.openstack.org/#/c/268863/16:07
inc0anyway, do we still need to keep ansible from source?16:08
SamYapleyea16:08
SamYapleansible 2.0 is broken with mysql16:09
SamYaplemore specifically16:09
SamYapleits broken with the way we do the grants16:09
sdakethanks SamYaple16:09
SamYaplethey broke it in rc8 (again)16:09
*** kproskurin has quit IRC16:09
SamYapleive already fixed the issue once16:09
SamYaplehttps://github.com/ansible/ansible-modules-core/pull/193416:09
inc0mandre, already in France?:)16:10
sdakeinc0 here is the problem https://review.openstack.org/#/c/268863/216:13
sdakeit would be interesting to understand what the real issue is - if only there were a way to record information in the bug tracker about what was happening with a bug ;)16:14
SamYaplesdake: mandre describes it16:14
sdakechanging orders of install isn't going to do a damn thing16:14
inc0sdake, mandre said that it does not fix issue16:14
inc0in review16:14
SamYaplesix has always been a dep16:14
*** farid_ has joined #kolla16:14
SamYapleits just conflicting16:14
SamYapleinc0: but its an ordering thing still16:15
SamYapleit fixes it when installed manually16:15
sdakeit does but not automatically?16:15
SamYaplesdake: it _will_ fix it chaning order of installs16:15
sdakeoh you mean not with pip -r requirements.txt16:15
SamYaplethats like 90% of teh pip issue16:15
SamYaplenot just that16:15
SamYaplebut with host conflicts as well16:16
sdakemy understanding this is not on the host but this is on the container16:16
sdakeis that correct?16:16
SamYaplethis is a dep problem in the container, yes16:17
sdakereading pypi bug report16:17
SamYaplehonestly i think we can just bump ansible to master version.... let me check16:18
sdakewill that fix this problem?16:18
sdakeyou mean 2.0.0 version16:18
SamYapleno16:19
inc02.0.0.2 is current pypi latest16:19
SamYaplewe cannot bump16:19
SamYapleplease read my comments above16:19
inc0I know16:19
SamYaplei meant what we have in master16:19
inc0ah so backport ansible version from our current master?16:20
openstackgerritSam Yaple proposed openstack/kolla: Bump liberty kolla-ansible continer ansible  https://review.openstack.org/27632816:20
*** dwalsh has quit IRC16:21
SamYaplesdake: inc0 ^16:21
SamYaplethats a safe operation for liberty since no changes occured16:21
SamYapleanything rc8 and up will break our playbooks16:21
inc0and you think it will fix issue at hand?16:21
SamYaplesince we don't see it in master, i would say yes16:23
*** dwalsh has joined #kolla16:23
sdakeSamYaple let me test it16:23
SamYaplei have a patch for bumping kolla-ansible's ansible to non-rc version, but like i said it breaks our playbooks16:23
sdakeif it tests out, i'll approve16:23
SamYaplei would have to adjust it all16:23
sdakelet me test the review in queue not the ansible 2.0.0 thing16:24
SamYapleinc0: was that a no on the revert merge? It kills me since it breaks my deploys and i have to manually fix when testing this qemu stuff https://review.openstack.org/#/c/276180/16:24
inc0SamYaple, I just want to test it16:24
inc0I try to figure out what happend16:25
SamYaple\' is a special char in rabbitmq config16:25
inc0did someone break our configs and we did merge it?16:25
SamYapleyup16:25
inc0dug16:25
SamYaplelook at the commit im reverting16:25
inc0duh16:25
limamauricioSamYaple, now i have an error in this task: [ceph | Bootstrapping Ceph OSDs]16:25
limamauriciofailed: [localhost] => (item=(0, {u'device': u'/dev/sdb', u'fs_uuid': u''})) => {"changed": true, "failed": true, "item": [0, {"device": "/dev/sdb", "fs_uuid": ""}]} msg: Container exited with non-zero return code16:25
SamYaplelimamauricio: you should have a boostrap_osd_* container16:26
SamYaplelook at the logs from that container16:26
SamYaple`docker logs bootstrap_osd_*`16:26
inc0SamYaple, what happend to our CI?16:27
inc0no functional tests after deploy?16:27
sdakeinc0 we are working on it16:27
inc0also I did deploy succesfully on ubuntu today16:27
inc0with vm booting16:28
inc0and all16:28
sdakeinc0 the problem  is ram, I've got a couple cats sorting out the ram issue as best they can16:28
SamYapleinc0: this only affects hostnames with certain chars16:28
SamYaplesdake: thats not related16:28
inc0ok..16:28
SamYapleinc0: look at bug, its all there16:28
inc0what chars are these?:P16:28
sdakeif i seem incoherent, 3 hrs of sleep for 2 weeks + 12 hours of manual labor every day will do that to you :)16:29
inc0nah that was me being unclear16:29
sdakei am so exhausted16:30
sdakeand in so much pain i just want to cry16:30
*** shardy has joined #kolla16:30
SamYaplesdake: you dont have to be here16:30
sdakei have definately put my body throug hthe ringer16:30
SamYapleyou can sleep16:30
SamYapleyoull get more done if you sleep first16:30
sdakei did get a night of sleep last night full 10 hours16:30
sdakeit was glorious16:30
sdakei honestly can't sleep anymore today16:30
sdakether eason iwas getting 3 hrs of sleep is the contractor dudes show up at 4am-5am16:31
inc0sdake, man you are breaking apart this release;)16:31
sdakeand I go  to bed at midnight gettting prepped for the next day16:31
sdakeinc0 how do you mean16:31
inc0first tooth, now sleep depravation16:31
sdakethe good  thing is, once this remodel is done (2 weeks) I'm going to be back at full 18 hr days of work I actually like doing - yay :)16:32
*** esikachev has joined #kolla16:32
inc0get some sleep first;)16:32
sdakenot tired16:32
sdakebody tired16:32
sdakei have full r&r before midcycle16:32
sdakeso should be at 100% for that :)16:32
sdakeI'll need it to keep up with you youngins :)16:33
inc0yeah I'm going to SC Saturday and will stay over a week there16:33
inc0for r&r16:33
sdakei am in monday-wed16:33
sdakei leave wed at 3:30 pm16:33
openstackgerritMerged openstack/kolla: Revert "fix miss rabbitmq config when hostname include special character"  https://review.openstack.org/27618016:33
inc0SamYaple, try out qemu plz16:34
sdakethats so the folks on the west coast can get back home without getting stuck for another hotel night16:34
SamYapleinc0: wow 14 seconds after merge you ask16:34
SamYaplecant even git pull htat fast16:34
inc0give some amphetamine to your packet-carrying-pidgeons16:35
sdakei had a friend that used crystal meth when he started working16:35
sdakehe blew up a lawyers car and went to jail16:35
sdakehe himself was an attorney16:35
sdakegot disbarred16:35
sdakehis life was a disaster for about 15 years16:36
sdakeI stil ldont think hes clean16:36
sdakeamphetemine is not to be touched :)16:36
inc0yeah, for pidgeons tho...16:36
pbourkesdake: "friend"16:36
inc0lsd is better for programming;)16:36
pbourke;)16:37
sdakepbourke best friend in fact16:37
*** salv-orlando has quit IRC16:37
sdakei really tried to support him for a long time16:37
sdakebut eventually gave up on him16:37
sdakei dont think lsd and writing code woudl work out16:37
sdakebut whatever your poison :)16:38
openstackgerritMerged openstack/kolla: Make libvirt reestablish connection with qemu  https://review.openstack.org/27619316:38
sdakeMine is Nuvigil16:38
inc0lsd and architectural problem solving does16:38
sdakehttps://en.wikipedia.org/wiki/Armodafinil16:38
sdakei tried mushrooms on college16:38
sdakenever lsd16:39
sdakei think they are similiar16:39
inc0they are not16:39
sdakei definately coudln't do anything on mushrooms16:39
inc0mushrooms are halucinogen, lsd is psychodelic16:39
sdakemushrooms rocked but i wouldn't do that again16:39
sdakedude they are both hallucingens :)16:39
*** esikachev has quit IRC16:40
inc0not really, lsd changes perception but don't add new stuff to it...at least for me16:40
limamauricioSamYaple, log from logs bootstrap_osd_0 http://pastebin.com/n8XKvd7q16:40
inc0but I did hear Mozart in 3d ;)16:40
*** ppetit has joined #kolla16:40
inc0and I mean spatial 3d not just stereo16:41
SamYaplelimamauricio: looking16:41
SamYaplelimamauricio: it cannot connect to your ceph-mon16:41
SamYaplelimamauricio: is this an AIO?16:41
limamauricioyes16:41
limamauricioaio16:41
SamYapleis the ceph-mon running?16:42
SamYaplethis is master branch too correct?)16:42
limamauriciowait16:42
limamauricio82b41e32a35f        localhost:4000/kollaglue/centos-source-ceph-mon:2.0.0        "kolla_start"            8 minutes ago       Restarting (1) About a minute ago                            ceph_mon16:42
limamauriciono :/16:42
SamYapleyoull want to check that log16:42
SamYaplesee whats up16:42
limamauriciook16:42
limamauricioSam http://pastebin.com/JyNfJW5B16:43
inc0what version of docker you're running?16:44
inc0limamauricio,16:44
limamauricioDocker version 1.9.1, build a34a1d516:44
limamauricioI'm trying multinode too16:45
*** fgimenez has quit IRC16:45
SamYaplelimamauricio: you need to pick one patch and stick with it for now16:45
SamYaplethis is all caused by having a dirty environtment16:45
SamYaplei would recommend a full clean of containers, configs, and docker volumes16:45
inc0and remember to re-partition your osd disk16:45
*** fgimenez has joined #kolla16:47
limamauriciookay :(16:47
limamauricioI will do everything from scratch16:47
SamYaplelimamauricio: you dont need to rebuild your container images, but you can16:47
pbourkeSamYaple: did we not backport kolla_ansible to liberty16:48
inc0do it just in case16:48
SamYapleyou do need to clean everythign else, all ceph configs in /etc/kolla, all containers for ceph deploy, all ceph volumes in `docker volume ls`16:48
pbourkeI mean kolla_docker16:48
SamYaplepbourke: nooo16:48
limamauricioI cleaned everything and i have the same error16:48
inc0and we shouldn't pbourke16:48
SamYaplethats a very large change in playbooks across the board16:48
inc0too big of a change16:48
SamYapleits like a 30% change16:49
pbourkeinc0: upgrade testing is a bi**h16:49
inc0that's what people will experience16:49
inc0so yeah I do know that16:49
pbourkeinc0: as you have to swap versions of docker/docker-py16:49
inc0yup16:49
pbourkehmm16:49
inc0and try to do this without tearing down your vms16:49
pbourke>_<16:49
inc0that's why I'm freaking out whole week16:50
SamYaplei can tear it down with out hitting vms :P16:50
inc0SamYaple, we're getting there16:50
inc0if it works for qemu+kvm I'm ok with that16:50
*** rhallisey has quit IRC16:50
inc0if someone is running qemu he shouldn't care about vm downtime anyway16:50
*** sdake has quit IRC16:51
*** sdake has joined #kolla16:52
SamYaplewow youre just generalizing!16:52
SamYaplenice16:52
*** shardy has quit IRC16:52
inc0well, have you ever seen qemu without kvm prod?16:52
inc0and performance of these vms would be so low that I'm not even afraid about LM-ing them, even over pidgeons on crystal meth16:53
*** dwalsh has quit IRC16:56
sdakeSamYaple did you confirm that restart of containers keeps qemu-kvm active?16:57
sdakebecause a couple days ago you said that was not the case in your testing16:57
SamYapleinc0: please run `ps -p 2666 -o f` where 2666 is a qemu without kvm process running from container16:57
SamYaplesdake: i said processes i did not say qemu specifically (didnt test qemu)16:58
sdakeok noted16:58
sdakeso very likely qemu is still operational16:58
sdakeoh16:58
sdakeduh16:58
sdakegot it you tried different processes16:58
sdakeSamYaple what processes did you see were retained on a libvirt restart?16:59
inc0sdake, this is required for libvirt to reconnect to instances: https://review.openstack.org/#/c/276193/17:02
SamYapleinc0: that merged, but was a master specfic patch17:02
SamYapleinc0: what was the output of the command?17:02
sdakei knew that change was wrong inc017:02
sdakei even left a comment about it :)17:03
inc0SamYaple, redeploying now,17:03
inc0got some strange errors in the process17:03
SamYaplesdake: you got lucky is all17:04
SamYaplethe only reason /run was there in the first place was to share socket with nova-compute17:05
*** ftcpops has quit IRC17:05
SamYapleit just _so_ _happened_ to contain other libvirt stuff too17:05
sdakeno it was to share a socket with systemd!17:05
sdakeatleast on centos ;)17:05
SamYaplebut systemd doesnt use it17:05
SamYapleyou are thinking of teh cgroup mount17:05
SamYaplenova-compute used the socket17:06
SamYaplesystemd never touched it17:06
sdakeok well whtaever wfm17:06
*** daneyon has joined #kolla17:06
sdakeinc0 that patch was already merged17:06
*** _ftcpops has joined #kolla17:06
sdakeinc0 so nothign to do there17:06
inc0yeah, I noticed17:06
inc0this was today17:06
*** daneyon_ has joined #kolla17:06
sdakeinc0 so with that qemu works correctly ?17:07
sdakeon container upgrade17:07
inc0still testing but we have hope17:07
sdakecorrectly being defined as no downtime or lost state of vm17:07
*** gfidente has quit IRC17:08
SamYaplejeeeeeezzz ugghhh17:09
sdakeSamYaple you may be right about systemd + cgroups17:10
SamYapleinc0: finally figured this out17:10
inc0go ahead17:10
SamYaplesdake: i know i am specifically about cgroups because ubuntu 14.04 doesnt need them17:10
SamYaple16.04 will though17:10
SamYapleinc0: ok17:10
SamYapleits a PITA17:10
SamYapleand you will confirm it when you show me the output of my requested command17:10
SamYapleinc0:17:10
SamYaple     P_INMEM    0x00004   Loaded into memory17:10
SamYaple     P_NOCLDSTOP    0x00008   No SIGCHLD when childr17:10
*** daneyon has quit IRC17:11
SamYapleoop wrong two17:11
SamYaple     P_CONTROLT     0x00002   Has a controlling terminal17:11
SamYaple     P_INMEM    0x00004   Loaded into memory17:11
SamYaplethe kvm+qemu process has those two flags set on the process17:11
SamYaplei want to know what _just_ qemu has17:11
*** unicell has quit IRC17:12
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Switch from mysql-python to pymysql everywhere  https://review.openstack.org/27619417:13
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Cleanup named volumes  https://review.openstack.org/27523717:13
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Use docker_remove_delay for auto-removing containers  https://review.openstack.org/27021317:13
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Remove rsyslog from all profiles  https://review.openstack.org/27015117:13
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Use private network in default settings  https://review.openstack.org/26954517:13
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Use volume for MariaDB  https://review.openstack.org/27025217:13
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml  https://review.openstack.org/27456017:13
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510917:13
SamYaplehmmmmm inc017:15
SamYaplemy stuff sticks around whether it is KVM or TCG accel17:16
sdakewhat is tcg17:16
SamYaplenon-kvm17:16
sdakeinc0 are you on centos or ubuntu?17:16
sdakeSamYaple  thanks17:16
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510917:17
inc0sdake, ubuntu17:17
inc0nested vm17:17
sdakeok well maybe nested vms just dont work17:17
sdakeSamYaple  did you try neseted vms?17:17
*** fgimenez has quit IRC17:17
SamYapleinc0: im nested vm17:17
SamYapleim nested kvm and qemu17:17
inc0but not qemu over xen;)17:17
SamYapleno17:17
sdakekerenels he same?17:18
SamYaplebut thats because only RAX uses xen :P17:18
inc0rax public cloud \o/17:18
inc0and amazon17:18
SamYapletrue17:18
sdakeinc0 your o nrax and amazon?17:18
SamYapleinc0: your kernel is probably 3.13 no?17:18
sdakewell christ that is a huge delta17:18
inc0SamYaple, it's 15.10 so its 4+17:18
SamYaple4.217:18
SamYaplehmmm yea it should be working17:19
inc04.2.0 yeah17:19
sdakeinc0 did you sa you running on rackspaces's cloud?17:19
SamYaplecan you walk through your exact proceedure for me? because im all sunshine and daisys over her17:19
*** ssurana has joined #kolla17:19
inc0yeah17:19
sdakeSamYaple are you runnign on your bare metal or rax cloud?17:19
inc0SamYaple, as soon as I figure out why the hell my kolla is broken17:19
SamYaplei just did a full nova_libvirt NEW IMAGE containerrecrate with upgrade from qemu 2.3 --> 2.5 and it all works17:20
inc0no downtime? oO17:20
SamYaplei have a 2.3 qemu vm and a 2.5 qemu vm running side by side under nova control17:20
SamYapleno downtime17:20
SamYaplei was ssh'd in the whole time17:20
inc0O.O17:20
sdakeyes this worked in liberty17:20
sdakei spent weeks on it17:20
sdakeand it sounds like i t still works17:20
sdakesomething in inc0's environment is making it not work17:20
sdakeif you guys could s ort out what that is that would rock17:21
inc0n exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 0 anyone seen anything like that? oO17:21
sdakebecause this problem will be repeated in the field im sure17:21
*** mbound has quit IRC17:21
inc0http://paste.openstack.org/show/485995/17:22
limamauricioSamYaple, inc0 I have the same error. ¬¬'17:22
limamauricioceph aio does not work17:23
inc0hold on17:23
limamauriciookay17:23
limamauricio:)17:23
sdakelimamauricio did yo ufollow directions on setting u p1 osd only?17:23
sdakeyou only have 1 osd right?17:23
limamauricioI think yes, how i chack it?17:24
sdakemoment17:24
limamauriciocheck*17:24
sdakehttp://docs.openstack.org/developer/kolla/ceph-guide.html17:24
inc0limamauricio, ansible --version17:25
sdake[global]17:25
sdakeosd pool default size = 117:25
sdakeosd pool default min size = 117:25
inc0you should have 1.9.117:25
limamauricio1.9.1?17:25
limamauricioI have 1.9.417:25
inc0well less than 217:25
sdakethats the version yo uwant to be on17:25
sdake1.9.417:26
limamauricioi'm trying to deploy master17:26
inc0my mistake17:26
sdakedo you have hat global section limamauricio17:26
limamauricioyes17:26
limamauricio:(17:26
limamauriciohttp://pastebin.com/fFepGZKf17:26
limamauricioceph_mon logs ^17:26
sdakelimamauricio i have tested ceph all in one in liberty and that worked for me, but I have not tested master ceph aio lately17:27
sdakeceph multinodeworked forme as of two weeks ago17:27
SamYaplelimamauricio: you have a _very_ old version of the ceph container17:27
inc0I did test ceph multinode from master like week ago17:27
sdakemy lab is torn apart in a remodel so i can't test it17:27
SamYaplewhen was the last time you rebuild?17:27
SamYapleplease post `docker images` limamauricio17:27
limamauriciook17:27
sdakelimamauricio i am also working on another critical issue here and can't help you live debug atm17:27
inc0yeah, you should do rebuilt every time you checko out new code17:27
inc0sdake, did you test this six problem>?17:28
sdakeinc0 i had to build a vm17:28
sdakei am getting there17:28
inc0kk17:28
*** neilus has joined #kolla17:28
sdakegetting ready to build containers now17:28
limamauriciohttp://pastebin.com/2DsqA3B217:29
limamauriciohttp://pastebin.com/Vv64xVku grep ceph17:29
limamauriciook sdake, thank for your support17:30
limamauricioI'll to deploy multinode17:31
SamYaplelimamauricio: you havent rebuild in 2 weeks!17:31
SamYapleyou are way outdated17:31
limamauricio:O17:31
limamauricioI have to build the whole week?17:31
limamauriciosorry, but i don't know :/17:32
SamYapleyou have to rebuild everytime somethign changes anythign in docker/*17:32
SamYaplethis is master branch, its constantly changing17:32
limamauriciook17:32
limamauriciook17:32
limamauriciosorry17:32
SamYapleno problem, but thats the cause of all your problmes17:32
limamauricioi'll run build again :)17:32
inc0SamYaple, output of ps -p 2666 -o f => F 617:33
limamauricioThank you so much17:33
*** neilus has quit IRC17:33
SamYapleinc0: ok same with me17:33
*** akwasnie has quit IRC17:33
SamYapleim trying to figure out a way to extact KTHREAD info17:33
SamYaplei need to get PF_KTHREAD of the process17:34
SamYapleits flag 0x0020000017:34
inc0SamYaple, http://paste.openstack.org/show/486000/17:34
inc0vm dies17:34
inc0Docker version 1.9.1, build a34a1d517:35
*** athomas has quit IRC17:35
*** salv-orlando has joined #kolla17:39
*** salv-orlando has quit IRC17:44
SamYapleinc0: I cant figure out this thread thing17:45
SamYapleif my process is a KTHREAD process then we have a thing to test17:45
SamYaplebut I can't even figure that our17:46
SamYapleout*17:46
SamYaple(also im not entirely sure on the exact underlying difference)17:46
dmsimardnihilifer: would like to chat with you when you're around - regarding the split of jobs per distro and install type17:48
*** unicell has joined #kolla17:48
*** esikachev has joined #kolla17:51
sdakebe back in 1 hour - lunch17:56
openstackgerritEvgeny Sikachev proposed openstack/kolla-mesos: Add unit-tests for chronos.py  https://review.openstack.org/27629817:57
*** daneyon_ has quit IRC17:59
*** Marga_ has quit IRC18:05
*** Marga_ has joined #kolla18:05
pbourkeSamYaple: how do you feel about moving variables from globals.yml to all.yml and leaving globals as all commented out18:11
pbourkeSamYaple: for the vars set in globals that aren't in all.yml its impossible to use group_vars/host_vars18:11
SamYaplepbourke: some vars are required in globals.yml18:11
SamYaplemost are common options18:11
pbourketrue18:12
*** sdake has quit IRC18:12
pbourkenetwork_interface is one that's not though18:12
SamYapleit is a requirement though18:12
pbourkewhat do you mean its not a requirement18:13
SamYapleit is a requirement18:13
pbourketo have it the same on every node/18:13
pbourke?18:13
SamYaplewell no but that it needs to be in globals.yml for most installs18:15
pbourkeyeah18:16
*** salv-orlando has joined #kolla18:24
*** salv-orlando has quit IRC18:28
*** asalkeld has joined #kolla18:28
*** ppetit has quit IRC18:29
openstackgerritJeffrey Zhang proposed openstack/kolla: Ensuring the openvswitch_db is ready before creating bridges  https://review.openstack.org/27557318:30
*** Jeffrey4l has quit IRC18:35
*** Marga_ has quit IRC18:38
*** unicell has quit IRC18:46
*** unicell has joined #kolla18:46
*** neilus has joined #kolla18:52
*** neilus has quit IRC18:57
*** jasonsb has quit IRC19:02
*** esikachev has quit IRC19:03
claytonSamYaple: do me a favor and run docker run -v /dev:/dev ubuntu and check what /dev/ptmx looks like before and after?19:03
claytonI'd love to just be going crazy.  Upgraded a dev environment to docker 1.9.1 and still seeing crazy behavior with ptys19:04
*** Marga_ has joined #kolla19:04
*** Marga_ has joined #kolla19:04
*** sdake has joined #kolla19:05
sdakeyo19:05
sdakeinc0 did you get that patch tested?19:05
sdakewow house dusty19:07
sdakeSamYaple did you get that wiki page setup with the midcycle details?19:15
sdakenm found it19:16
*** neilus has joined #kolla19:16
inc0sdake, was on lunch19:18
sdakeya me too19:18
inc0we were debugging libvirt19:18
sdakejust got back19:18
inc0seems like 1.10 doesn't restart vms19:18
inc01.9 does19:18
inc0seems like a bug tho...fortunate for us but bug nonetheless19:18
*** neilus has quit IRC19:28
sdakesamyaple if you ahve the discount link please put in wiki when you have an opportunity19:48
sdakeinc0 can you do me a solid and double check this page for errors19:48
sdakehttps://wiki.openstack.org/wiki/Sprints/KollaMitakaSprint19:48
sdakelike in the next 10 minutes19:48
sdakeinc0 good news about 1.10 - that is which version we are rolling with19:48
inc0still confirming but it does seems like it19:50
inc0as for midcycle site, can't see anything wrong19:50
sdakeinc0 can you check that sprint page i want to get it on the ml19:50
sdakethere is still a todo for the hotel code19:51
sdakebut other then that looks pretty solid19:51
inc0and an etherpad19:51
*** salv-orlando has joined #kolla19:54
inc0sdake, I found another bug in stable19:56
inc0docker-py 1.7 doesn't seem to be working, we need to pin requirements to ==1.1.019:56
sdakethis is during deploy or buidl?19:58
sdakeI am building now with 1.4.019:58
sdakeso we don't have to pin to 1.1.019:59
inc0deploy19:59
inc0of stable19:59
*** sdake_ has joined #kolla20:02
sdake_inc0 i have meeting in 1 hour20:03
sdake_lets rock20:03
*** dratushnyy has quit IRC20:03
sdake_let me get my network back20:03
inc0kk20:03
*** sdake has quit IRC20:03
*** sdake has joined #kolla20:04
sdakeinc0 all my containers but gnocchi are built20:05
sdakelet me try a deploy20:05
inc0go for it20:05
*** sdake_ has quit IRC20:08
inc0so rolling back to docker-py 1.1 and rebuilding container seems to help20:10
inc0docker-py 1.4 works for you?20:10
*** neilus has joined #kolla20:17
*** severion has joined #kolla20:18
SamYapleclayton: ok give me a minute20:26
*** severion has quit IRC20:30
claytonI suspect your /dev/ptmx is already a symlink20:31
SamYapleclayton: it matches my host perms20:31
SamYaple /dev/ptmx on my host (and in the container) is a dev node20:31
SamYaplenot a symlink20:31
claytonyou're using docker 1.9.1?20:32
SamYapleyes20:32
SamYapleand docker 1.1020:32
claytonok, I can't explain that at all.  as near as I can tell libcontainer is hard coded to make that a symlink in the container20:33
SamYaplewhere do you see that?20:33
claytonhttps://github.com/docker/docker/blob/master/vendor/src/github.com/opencontainers/runc/libcontainer/rootfs_linux.go#L49-L5820:34
claytonI compiled a debug version of the docker engine, and it always passes entries in with config.Devices20:34
*** fthiagogv_ has joined #kolla20:37
SamYapleroot@y-work-laptop01:~# ls -l /dev/ptmx; docker run --rm -it -v /dev/:/dev/ ubuntu ls -l /dev/ptmx20:37
SamYaplecrw-rw-rw- 1 root tty 5, 2 Feb  4 20:36 /dev/ptmx20:37
SamYapleclayton: ^^20:37
SamYaplecrw-rw-rw- 1 root tty 5, 2 Feb  4 20:36 /dev/ptmx20:37
claytonweird, clearly something is different.  I'm running of out things it could be20:38
SamYaplewhats your host os and kernel?20:38
sdakeinc0 i'm getting unrecognized status from pull20:38
sdakeI have pull policy set to missing20:38
sdakeand am using the registry20:38
sdakeI htink the module is not working as expected20:38
*** neilus has quit IRC20:38
claytonTrusty, 3.13.0-67-generic20:38
SamYaplesdake: for liberty?20:38
sdakeSamYaple yes20:39
sdakestable/liberty20:39
SamYaplesdake: are you running docker registry v2?20:39
sdakei am not using a registry20:39
sdakei want to just use the cache20:39
SamYaplethen you shouldn't be pull an image20:39
SamYapleyou likely have your naming wrong20:39
SamYapleyou are attempting to pull an image for sure20:39
sdakeI see that20:39
SamYapleclayton: just fyi, it is recommened/required to run 3.19 or 4.2 kernel on Trusty20:39
sdakeit says "Pullling image docker.io ..."20:40
claytonyeah, I'd like to upgrade, but we're waiting for 16.04.20:40
SamYapleso you likely dont have that image build20:40
SamYapleclayton: you can pull in the latest lts kernel20:40
SamYapleits an official lts kernel in the official repos20:40
claytonsadly there is no newer LTS kernel.  There are newer kernels you can run on LTS and they're supported, but only for 6 months20:40
SamYaplethere is...20:40
SamYapleno no20:41
SamYaplelook at the deprecation map20:41
SamYaplehttps://wiki.ubuntu.com/Kernel/LTSEnablementStack20:41
sdakeopenstack-release set improperly samyaple20:41
sdakei'm good20:41
SamYapleif you are saying you must keep kernel for 2 years clayton, then youre right20:41
SamYapleotherwise you ahve many options20:41
claytonSamYaple: yeah, support for all o ftghem but 3.13 expires the day xenial releases20:42
sdakeContainer command not found or does not exist20:42
sdakenever seen that error before20:42
sdakedocker 1.1020:42
sdakedocker-py 1.420:42
SamYapleclayton: despite what that chart shows, thats not true20:42
SamYapleclayton: you have a 3 month grace period20:42
SamYapleif you look at the super deatailed one toward the bottom it shows that clayton20:43
claytonI can try a newer kernel anyway, it'll be good to rule that out20:43
*** macsz1 has joined #kolla20:43
SamYaplesdake: liberty you need docker 1.8.220:43
sdakeyou said it would work with 1.1020:43
SamYapleno i said containers will run20:43
sdakeoh20:44
SamYapleI said playbooks would not20:44
sdakeyou didn't say the second part, but I see :) thanks20:44
SamYaplei did repeatedly20:44
sdakegood info to have20:44
SamYapleill get the logs if youd like20:44
sdakewell i must have mised it because of these annoyign security rules at cisco20:44
sdakei'll roll back to docker 1.8220:44
*** inc0_ has joined #kolla20:44
SamYapleits all good20:44
SamYapleyou should run your IRC in a vm somewhere20:44
SamYapleto not have that issue20:44
claytonSamYaple: at this point I think it's unlikely we'll find a problem that we'll upgrade for before xenial releases anyway.  it's just too close not to wait unless there is a real show stopper20:45
SamYapleI understand clayton, and im not sure that is the issue20:46
claytonSamYaple: reproduced the same issue with 3.1920:46
SamYaplebut just thorwing that out there20:46
claytonyeah, it's worth ruling out20:46
claytonwe have some weird aufs stuff we run into that I suspect upgrading would fix also20:46
SamYapleclayton: https://github.com/docker/docker/issues/1946420:46
SamYapleapparently it is fixed for you in docker 1.10?20:47
*** inc0 has quit IRC20:47
*** macsz has quit IRC20:47
*** fthiagogv has quit IRC20:47
claytonwell, I can try 1.10 in a bit20:47
SamYapleclayton: https://github.com/docker/docker/issues/1946420:48
SamYapleoops20:48
claytonhah, that's a bit of a hack of a fix20:48
SamYapleclayton: https://master.dockerproject.org/ *20:48
claytonI already have the code building locally, it'll be easy to just switch to master and build that20:48
claytonoh, didn't realize those were the binaries, that's definitely easier.  thanks20:50
SamYapleyup20:50
SamYaplebuilt per commit too20:50
claytonwelll, and I get to inadverently check how long it takes to build the hashes for the local images20:51
claytonSamYaple: yeap, that fixes it20:52
SamYapleclayton: well good news is that will be out in like 3 days20:52
SamYapleit was slated fr a release today20:52
SamYapleand kolla will require it as soon as it lands20:53
claytonSamYaple: I saw something about that, we can wait20:53
claytonso why does this work for you on 1.9.1?20:53
SamYapleim on 15.1020:54
claytonSamYaple: I think part of this is misconfiguration on ubuntu also.  it's not setting ptmxmode when it mounts /dev/pts.  if it did then this wouldn't matter20:54
claytonit would change /dev/ptmx to be a symlink to /dev/pts/ptmx and it would still just keep working, since they would have the same permissions20:55
claytonSamYaple: v1.10.0 was tagged 30 minutes ago20:56
SamYaplew00t20:56
SamYapleawesome clayton20:56
inc0_SamYaple, http://paste.openstack.org/show/486014/ => have you ever seen sth like this?20:58
inc0_trying to deploy liberty -.-20:58
*** neilus has joined #kolla21:05
SamYapleinc0_: looking21:05
sdakelooks like an api change to me21:06
sdakethere was something like that in 1.5 of docker-py too21:06
sdakeannoying21:06
sdakehave to cap docker-py would be my guess21:06
inc0_I did21:06
inc0_1.1.021:06
sdakepip show docker-py21:07
inc0_1.1.021:07
SamYapleinc0_: yea21:07
inc0_told you, and in requirements we have =>1.1.0 so by default installs latest greatest21:07
SamYaplethats ansible docker module21:08
SamYapleand docker 1.8.2<21:08
inc0_ahh...I have docker 1.8.221:08
inc0_.3**21:08
SamYapleyup21:08
SamYaplethat was the breaking change21:08
SamYaplemid version!21:08
sdakeright21:08
* sdake wacks docker21:08
* sdake thinks there was an ulterior motive for that nonsense21:09
SamYapleit was an honest mistake21:09
SamYaplethey were solving a different issue21:09
SamYapleit just should have required a bump in x.y.z Y version21:09
SamYaplebut they missed it21:09
sdakeshould hae released 1.8.4 with revert, bumped versions21:09
*** absubram has quit IRC21:09
sdakeanyway21:10
sdakecan't fix the past :)21:10
SamYapleeh damage was done21:10
claytonSamYaple: just verified, the 1.10 packages are in the apt repos21:18
SamYapleclayton: yup already installed21:19
claytonI'm going to put that into dev and see how it works out21:19
SamYapletesting my needs-docker1.10-patches now21:19
SamYapleabout to repush them21:19
*** neilus has quit IRC21:25
*** farid_ has quit IRC21:27
*** ssurana has quit IRC21:34
sdakeinc0 when i downgraded to docker 1.8.2 my docker images got removed, so i have to rebuild21:35
* sdake groans21:35
sdake30-45 mins inc21:35
inc0_same here21:36
inc0_just finished;)21:36
sdakei'm just using docker from epel21:36
sdakerather hten docker-engine21:36
inc0_well, we need to make that crystal clear, that with upgrade you need new docker21:37
inc0_probably won't be in rh distros yet21:37
sdakeinc0_  was that a downgrade from 1.8.3 that caused that?21:37
sdakeinc0_ yes i've got an action to write the docs on upgrade21:37
sdakei'll take that on21:37
inc0_well I had to remove /var/lib/docker to make it work21:37
inc0_soo...I did succesfully deploy aio stable with docker 1.8.221:40
SamYapleinc0_: with my patch in queue?21:42
SamYapleor ithout?21:42
inc0_vanilla21:42
inc0_granted I can't run haproxy because it's aio21:43
SamYaplebut you an't run deploy with vanilla21:43
SamYapleright?21:43
inc0_and I think haproxy task was one that bug reported21:44
SamYaplethen why did I do this ? https://review.openstack.org/#/c/276328/21:44
inc0_I did deploy vanilla21:44
inc0_just did21:44
inc0_not with you patch21:44
inc0_once I solved docker version problem, it deployed21:44
inc0_people did report bug for that21:44
inc0_just didnt happen to me21:45
*** inc0__ has joined #kolla21:46
inc0__irc is fleaky;/21:46
*** inc0__ is now known as inc021:46
*** inc0_ has quit IRC21:49
*** sdake has quit IRC21:49
inc0sooo...good news is that I kill -9ed docker 1.8.2 and vm keeps runing22:02
SamYapleahhhhahaharhharharharhhaghaghaghrgahgahg22:03
inc0bad news is after docker upgrade to 1.10 libvirt containers fails to start22:03
SamYapledocker 1.10 isn't letting me do shared mountpoint22:03
inc0wait..waht? isn't that exactly why we want docker 1.10?22:03
*** neilus has joined #kolla22:04
SamYapleyea22:04
SamYapleits breaking22:04
SamYapleError response from daemon: Cannot start container 1642edde5add723fa9253b638e407a679e5e129304cfe17e78f0124e02e04a1f: Path /run/netns is mounted on /run/netns but it is not a shared mount.22:04
SamYaplethis is official binaries22:04
SamYapleofficial 1.10 tagged released bienaries22:05
*** absubram has joined #kolla22:06
inc0soo upgrading docker leaves you in broken openstack limbo22:08
inc0that's bad.22:08
*** absubram_ has joined #kolla22:08
inc0some of containers failed to start22:08
SamYapleinc0: you are stopping the containres first right?22:08
*** neilus has quit IRC22:08
SamYaplethen starting them in correct order?22:08
inc0no22:08
inc0but that's a good point22:09
SamYaplethey have deps on eachother and docker provides no start assurance22:09
SamYaplei think docker starts in order of container id22:09
inc0however I don't think that's the case22:09
inc0because one of failing containers is nova_libvirt and it doesn't really have any deps22:09
inc0and other than that restarting containers that failed should work22:10
*** absubram has quit IRC22:11
*** absubram_ is now known as absubram22:11
*** absubram has quit IRC22:14
SamYapleahhhhhh22:16
SamYaplegot it22:16
SamYaplenew default option22:16
SamYaplewe need to set system config paramaters in the docker daemon to make this mount sharing work22:16
inc0make sure to write it down for upgrade-docker playbook22:16
SamYapleim still not sure about that inc022:17
SamYaplewe dont have a docker-install playbook for a reason22:17
SamYaplewhy would we have an upgrade one?22:18
inc0well upgrade will be pretty non-trivial22:18
inc0even if not as playbook, we need very detailed guide at very least22:18
*** neilus has joined #kolla22:18
*** bmace has quit IRC22:18
SamYaplematters not. the reason we dont have an install playbook is because that is host level22:18
SamYapledetailed guide 100%22:18
SamYaplethe playbook is what im on the fence about22:18
*** bmace has joined #kolla22:19
SamYaplei would say we would need an install playbook before upgrade, and we had prevoiusly vetoed the install playbook22:19
inc0well I think if we keep it completely separated, it might not be too bad22:19
SamYaplesee above22:19
inc0I know22:19
SamYapleok well thats just a discussion we need to have again22:19
inc0just my opinion, detailed guide is ok22:19
SamYaplebecase it was vetoed since "kolla doesnt tough host"22:19
SamYapletouch*22:19
inc0yeah I know, but at the same time we want seamless upgrades22:20
inc0and to peform it as seamless as possible...we need procedure that will be non-trivial at best22:21
inc0so SamYaple I've been thinking about neutron-agents container rebuild22:22
inc0we use vxlans as default right?22:22
inc0then we should be able to migrate between routers pretty easily22:22
SamYapleinc0: we don't migrate anything, the operator does22:23
*** neilus has quit IRC22:23
inc0again, procedure22:23
inc0guide, howto22:23
SamYaplevxlan or not, it doesnt matter22:23
SamYaplemoving router to new host is same procedure22:23
inc0if someone uses HA routers, should be pretty easy right?22:24
SamYapleyea that dosent actualy require any action on thier part22:24
SamYaple.... when it works22:24
SamYaplewhen it doesnt work it still doesnt require action from them22:24
SamYaplebut networking drops22:24
inc0but neutron's problems are neutron's problems22:25
SamYapleyup22:25
inc0ok22:25
SamYapleto migrate legacy router is pretty simple22:25
inc0so best case scenerio, we actually can do upgrade with no vm downtime22:25
SamYapleyea22:26
SamYaplenetwork is going down period22:26
SamYapleno getting aroudn that ever22:26
SamYapleuntil thin containers22:26
inc0yeah, I get that22:26
inc0but can we make it a blip?22:26
inc0not full-fledget, over-10s long downtime?22:26
SamYaplethat assumes no issues22:26
SamYaplethere are alwasy issues moving routers22:26
inc0so here's deal, we want to demo upgrades22:27
inc0so we have benefit of creating configs way we want22:27
SamYaplei would not do live demo if I were you22:30
SamYaplethats just me though22:30
SamYapleit was.... not prepped last time22:30
SamYaplei view that talk as a failure tbh22:30
SamYapleit could have been much better22:31
inc0I'll run it 100 times before22:31
SamYapleyea22:31
SamYaplestill22:31
inc0live demo adds to "wow, cool" factor tremendously22:31
inc0but I'll have recording just in case22:31
SamYaplesure22:31
inc0we'll see, right now I want upgrades at all ;(22:32
SamYaplewe should have hard downtime upgrade now and incremental improvements inc022:33
SamYapleif those improvments come before mitaka... great!22:33
SamYaplebut we need to have _something_22:33
inc0so thing is you can do no-downtime-for-vm now22:33
inc0I did that22:33
inc0assuming no issues ofc, but yeah, it can be done22:34
SamYaplethats with kill -922:34
*** tzn has joined #kolla22:34
inc0I just want at least that22:34
SamYapleim -2 on automating that22:34
inc0yes, hacky22:34
SamYapleim ok with documentation22:34
inc0I'm ok with no automation of that22:34
inc0just a howto22:34
SamYaplecool22:34
inc0very detialed22:34
SamYapleyea22:34
SamYaplehow and why22:34
SamYapleand why its fixed in 1.1122:34
inc0automation will go for openstack services22:35
SamYaplei like how pip now says "still running stup.py for...."22:35
SamYapleits really useful22:35
inc0that being said, we need to upgrade docker to 1.10 / 1.11 and run liberty on it22:35
inc0decouple docker upgrade from service upgrade22:35
inc0yeah, I noticed that too22:36
inc0so it will probably look like "upgrade docker, then manually start containers in correct order -> here is start_containers_in_correct_order.sh what will do it"22:37
inc0and you're running liberty on docker 1.1122:37
inc0then you run our plays22:37
*** tzn has quit IRC22:38
SamYaplewe can backport the kolla_docker module to liberty and _not_ change the plays22:38
SamYaplebut add a new play for "start in correct order"22:38
inc0thing is, it would super cool to run neutron upgrade just once22:38
inc0we can have it in kolla-mitaka22:39
*** salv-orl_ has joined #kolla22:39
inc0anyway, that's on me, I'll prep it tomorrow22:39
inc0as a sh for now,22:40
inc0I'm going off for now, cyas22:40
SamYaplebye inc022:40
*** inc0 has quit IRC22:40
*** sdake has joined #kolla22:41
*** salv-orlando has quit IRC22:42
SamYaplew00000t successful docker 1.10 deploy22:42
openstackgerritSam Yaple proposed openstack/kolla: Convert Neutron to thin containers  https://review.openstack.org/27271522:46
*** _ftcpops has quit IRC22:46
openstackgerritSam Yaple proposed openstack/kolla: Correct issue with virtualenv rootwrap  https://review.openstack.org/27550422:46
*** jtriley has quit IRC22:50
sdakeSamYaple nice22:56
sdakenow if we could just get stable, the thing we released, to work, we woudl be golden ;)22:56
*** neilus has joined #kolla22:57
sdakeSamYaple i'm getting an undefined variable attribute ipv422:58
sdakeseen that one?22:59
sdakeansible 1.9.4-1.el722:59
*** neilus has quit IRC23:01
sdaketerrible error for a configuration problem23:02
sdakeyup well i can duplicate that kolla_ansible problem23:05
*** dave-mccowan has quit IRC23:05
sdakestable/liberty is busted23:05
sdakei'll try your patch now23:05
sdakeall day to get a working test environment23:05
* sdake groans23:05
*** blahRus has quit IRC23:13
*** pbourke has quit IRC23:15
*** pbourke has joined #kolla23:15
*** JoseMello has quit IRC23:18
*** neilus has joined #kolla23:22
*** neilus has quit IRC23:26
*** aginwala has joined #kolla23:27
*** dims_ has quit IRC23:31
*** dims has joined #kolla23:32
sdakeSamYaple can you ahve a look at that patch you submitted this morning - it is spitting out an oserror 13 related to semaphores23:34
sdake(perm denied)23:34
*** neilus has joined #kolla23:37
*** stvnoyes has quit IRC23:42
*** stvnoyes has joined #kolla23:43
*** neilus has quit IRC23:48
*** aginwala has quit IRC23:51
*** aginwala has joined #kolla23:56
*** daneyon has joined #kolla23:58
*** daneyon_ has joined #kolla23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!