Thursday, 2017-04-06

sbezverksdake: nah, the nosiest part is colling vents00:03
sbezverkfor cpus00:03
sbezverkthey scream like crazy initially00:03
sbezverkbut then quiet down a bit00:04
eanylinsdake: Just saw your message00:08
eanylinsdake> eanylin can you confirm you did get 4.0.0 images running and booting a vm with source00:08
eanylin<sdake> with centos-source imgaes?00:08
eanylinyes I did00:08
*** shashank_t_ has joined #openstack-kolla00:08
eanylinbut I still havent figure out how to ssh/ping the cirros vm...00:08
*** rwsu has quit IRC00:09
eanylinseems like rwellum made huge progress today00:09
*** chas has joined #openstack-kolla00:09
sdakeeanylin ya he seemed to do so00:10
sdakeI guess that EXPORT is one solution and ~/.kube/config is another solution00:10
eanylinsdake: Yeah, I think he sure did00:10
sdakeeanylin i wasn't clear if he got a vm ssh-able into00:10
eanylinsdake: I gave him the info this morning00:10
eanylinsdake: Before I start work00:10
eanylinsdake: I thought his VM wento error state?00:11
eanylinwent into*00:11
sdakedont know00:11
sdakemine does00:11
eanylinsdake: Was reading the messages from above, seems like thats where he left off00:12
eanylinsdake: Yeah, I am not sure why too, you mentioned you had problems with rabbit00:12
sdakeeanylin i didn't debug it00:12
sdakejust got back from jerking around at the dentist00:12
eanylinsdake: Ok00:12
eanylinsdake: Lol00:12
openstackgerritAbel Lopez proposed openstack/kolla-kubernetes master: Make Ironic honor dependencies  https://review.openstack.org/45393100:13
eanylinsdake: Placement and Cell0 came up without issue for you right?00:14
sdakeyup00:14
eanylinI think those are the critical ones00:14
*** chas has quit IRC00:14
eanylinI had problems with those00:14
eanylinand also with the bug that was reported in RDO00:14
eanylinmy VM came up after fixing those00:14
eanylinsdake: I am leaving office, will come back online later..00:17
*** Pavo has quit IRC00:19
*** eanylin has quit IRC00:23
*** lucasxu has joined #openstack-kolla00:24
*** Pavo has joined #openstack-kolla00:25
spsuryamorning guys00:31
*** zhurong has joined #openstack-kolla00:33
*** lrensing has quit IRC00:33
*** MarginHu has joined #openstack-kolla00:34
*** lrensing has joined #openstack-kolla00:37
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Recovering ironic after gate job name change  https://review.openstack.org/45393400:38
*** harlowja has quit IRC00:39
*** guoguo has joined #openstack-kolla00:45
*** chas has joined #openstack-kolla00:45
*** chas has quit IRC00:50
*** ediardo has quit IRC00:52
alopthanks sbezverk00:54
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Recovering ironic after gate job name change  https://review.openstack.org/45393400:56
*** alop has quit IRC00:57
*** eanylin has joined #openstack-kolla00:58
sbezverkalop: thank you for catching it.. really surprised how it was working before..00:59
*** lucasxu has quit IRC01:01
*** jrobinson has quit IRC01:02
*** lrensing has quit IRC01:11
*** goldyfruit has joined #openstack-kolla01:13
*** oanson has quit IRC01:17
*** oanson has joined #openstack-kolla01:18
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Recovering ironic after gate job name change  https://review.openstack.org/45393401:20
*** chas has joined #openstack-kolla01:21
*** shashank_t_ has quit IRC01:24
*** shashank_t_ has joined #openstack-kolla01:24
*** caoyuan has joined #openstack-kolla01:26
*** harlowja has joined #openstack-kolla01:26
*** chas has quit IRC01:26
*** harlowja has quit IRC01:26
*** shashank_t_ has quit IRC01:28
*** hrw has quit IRC01:29
*** hrw has joined #openstack-kolla01:31
spsuryasdake: around ?01:31
openstackgerritMerged openstack/kolla-kubernetes master: Updated from global requirements  https://review.openstack.org/45101801:31
*** iceyao has joined #openstack-kolla01:34
*** ediardo has joined #openstack-kolla01:34
openstackgerritchenyingnan proposed openstack/kolla-ansible master: test  https://review.openstack.org/45394301:37
*** jrobinson has joined #openstack-kolla01:41
sdakenew server is out of the box01:43
sdakesbezverk its quiet compared to minime01 :)01:43
sdakenot sure if the rack i bought wil work for it01:43
sdakehave no idea what to do with these rack track things01:43
sdakesup spsurya what an i do for ya01:43
spsuryasdake: next PTG again location :(01:44
sdakehas that been pbuisehd?01:44
sdakeI dont knwo where next ptg is01:45
spsuryasdake: yes. on dev ML01:46
sdakewehre is it located01:46
spsuryain USA01:47
*** Pavo has quit IRC01:47
spsuryaDenver01:48
sdakewell blame the foundation - I dont have anything to do with that decision :(01:51
sdakei hear ya01:51
*** fooliouno has joined #openstack-kolla01:52
spsurya:(01:54
*** shashank_t_ has joined #openstack-kolla01:57
*** chas has joined #openstack-kolla01:57
*** oanson has quit IRC01:58
*** goldyfruit has quit IRC01:59
*** oanson has joined #openstack-kolla02:00
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Fix ceilometer meters for swift service  https://review.openstack.org/45237302:01
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Update the references link for some project  https://review.openstack.org/45055202:01
*** chas has quit IRC02:03
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Fix ceilometer meters for swift service  https://review.openstack.org/45237302:03
openstackgerritwangwei proposed openstack/kolla-ansible master: Add sudo command for non-root account to mount osd  https://review.openstack.org/43929702:07
*** unicell has quit IRC02:09
openstackgerritwangwei proposed openstack/kolla-ansible master: Add sudo command for non-root account to mount osd  https://review.openstack.org/43929702:11
openstackgerritwangwei proposed openstack/kolla-ansible master: Add sudo command for non-root account to mount osd  https://review.openstack.org/43929702:12
openstackgerritwangwei proposed openstack/kolla-ansible master: Add remove-garbage function in kolla-ansible  https://review.openstack.org/40236202:33
*** chas has joined #openstack-kolla02:33
*** MarginHu has quit IRC02:38
*** chas has quit IRC02:38
openstackgerritjimmygc proposed openstack/kolla-ansible master: Make glance filesystem_store_datadir configurable.  https://review.openstack.org/45209402:45
*** shashank_t_ has quit IRC02:49
*** shashank_t_ has joined #openstack-kolla02:50
*** shashank_t_ has quit IRC02:54
*** vhosakot has joined #openstack-kolla03:09
*** brad[] has quit IRC03:09
*** chas has joined #openstack-kolla03:09
*** lamt has joined #openstack-kolla03:13
*** chas has quit IRC03:14
*** caoyuan_ has joined #openstack-kolla03:20
*** lrensing has joined #openstack-kolla03:20
*** caoyuan has quit IRC03:21
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Fix ceilometer meters for swift service  https://review.openstack.org/45237303:22
*** brad[] has joined #openstack-kolla03:24
*** zhubingbing_ has joined #openstack-kolla03:24
*** jascott1_ has joined #openstack-kolla03:31
*** chas has joined #openstack-kolla03:36
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move nova scheduler related parameters to [scheduler] group  https://review.openstack.org/45396603:37
*** gkadam has joined #openstack-kolla03:40
*** chas has quit IRC03:40
*** gkadam has quit IRC03:43
*** vhosakot has quit IRC03:44
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Fix ceilometer meters for swift service  https://review.openstack.org/45237303:45
*** dave-mccowan has quit IRC03:46
*** gkadam has joined #openstack-kolla03:49
*** jtriley_ has joined #openstack-kolla04:02
*** erhudy_ has joined #openstack-kolla04:03
*** hrww has joined #openstack-kolla04:06
openstackgerritMerged openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939504:09
*** hrw has quit IRC04:10
*** jtriley has quit IRC04:10
*** erhudy has quit IRC04:10
*** hrww is now known as hrw04:10
*** erhudy_ is now known as erhudy04:10
*** markmcclain has quit IRC04:12
*** markmcclain has joined #openstack-kolla04:13
*** zhubingbing_ has quit IRC04:29
*** sayanta__ has quit IRC04:34
*** unicell has joined #openstack-kolla04:36
*** ediardo has quit IRC04:42
*** lamt has quit IRC04:45
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move nova scheduler related parameters to [scheduler] group  https://review.openstack.org/45396604:50
*** skramaja has joined #openstack-kolla04:53
*** zhubingbing has joined #openstack-kolla04:53
*** lrensing has quit IRC04:59
*** jtriley_ has quit IRC05:04
*** caoyuan_ has quit IRC05:05
*** caoyuan has joined #openstack-kolla05:06
*** sayantan_ has joined #openstack-kolla05:11
*** satyar has joined #openstack-kolla05:11
*** guoguo has quit IRC05:15
*** rwsu has joined #openstack-kolla05:30
*** ksumit has quit IRC05:43
*** jaosorior_away is now known as jaosorior06:16
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Remove all Ubuntu 14.04 related documentation  https://review.openstack.org/45366806:26
*** Serlex has joined #openstack-kolla06:43
*** skramaja_ has joined #openstack-kolla06:44
*** skramaja has quit IRC06:45
*** targon has joined #openstack-kolla06:47
*** skramaja_ has quit IRC06:49
*** caowei has joined #openstack-kolla06:49
*** skramaja_ has joined #openstack-kolla06:51
*** skramaja_ is now known as skramaja06:53
*** matrohon has joined #openstack-kolla06:56
*** targon has quit IRC06:58
*** targon has joined #openstack-kolla07:01
*** jrobinson has quit IRC07:02
*** pcaruana has joined #openstack-kolla07:02
*** jascott1_ has quit IRC07:05
*** jascott1_ has joined #openstack-kolla07:05
*** jascott1_ has quit IRC07:10
*** athomas has joined #openstack-kolla07:12
*** eanylin has quit IRC07:15
*** bjolo_ has joined #openstack-kolla07:27
*** jmccarthy has quit IRC07:28
*** cuongnv has joined #openstack-kolla07:29
*** magicboiz has quit IRC07:34
*** manheim has joined #openstack-kolla07:42
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Support non-x86 architectures (aarch64, ppc64le)  https://review.openstack.org/43094007:42
*** manheim has quit IRC07:42
*** sayantan_ has quit IRC07:42
*** manheim has joined #openstack-kolla07:42
*** sayantan_ has joined #openstack-kolla07:43
*** rmart04 has joined #openstack-kolla07:43
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: gnocchi-base: add some devel packages for non-x86  https://review.openstack.org/45370007:44
*** chas has joined #openstack-kolla07:44
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ironic-pxe: handle non-x86 architectures  https://review.openstack.org/43481707:45
*** magicboiz has joined #openstack-kolla07:46
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: debian: move to stretch  https://review.openstack.org/43445307:47
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: handle rtslib(-fb) package names and dependencies  https://review.openstack.org/45080507:51
*** huzhengchuan has joined #openstack-kolla07:51
*** skramaja has quit IRC07:53
*** skramaja has joined #openstack-kolla07:56
openstackgerritweichuancheng proposed openstack/kolla-ansible master:  add clean osd ceph osd  disk info task  https://review.openstack.org/45402507:57
*** nathharp has joined #openstack-kolla08:02
openstackgerritweichuancheng proposed openstack/kolla-ansible master:  add clean osd ceph osd  disk info task  https://review.openstack.org/45402508:03
*** sayantan_ has quit IRC08:18
*** sayantan_ has joined #openstack-kolla08:18
*** shardy has joined #openstack-kolla08:19
*** magicboiz has quit IRC08:23
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: handle rtslib(-fb) package names and dependencies  https://review.openstack.org/45080508:26
*** magicboiz has joined #openstack-kolla08:26
*** magicboiz has quit IRC08:26
*** blallau has joined #openstack-kolla08:26
*** magicboiz has joined #openstack-kolla08:27
*** mgoddard has joined #openstack-kolla08:29
*** magicboiz has quit IRC08:32
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Support non-x86 architectures (aarch64, ppc64le)  https://review.openstack.org/43094008:35
*** egonzalez has joined #openstack-kolla08:37
openstackgerritweichuancheng proposed openstack/kolla-ansible master:  add clean osd ceph osd  disk info task  https://review.openstack.org/45402508:43
*** magicboiz has joined #openstack-kolla08:43
*** sayantan_ has quit IRC08:59
zhubingbinghi gusys09:06
openstackgerritzhubingbing proposed openstack/kolla-kubernetes master: Add elasticsearch support helm  https://review.openstack.org/45405009:07
*** zhubingbing has quit IRC09:10
*** zhubingbing has joined #openstack-kolla09:13
*** Deys1 has left #openstack-kolla09:21
*** cuongnv has quit IRC09:23
openstackgerritMerged openstack/kolla-ansible master: Modify the hosts of the post-deploy.yml playbook  https://review.openstack.org/43682209:25
*** caowei has quit IRC09:25
*** caowei has joined #openstack-kolla09:31
*** yingjun has quit IRC09:35
*** caowei has quit IRC09:37
*** caowei has joined #openstack-kolla09:38
*** satyar has quit IRC09:47
*** iceyao has quit IRC09:50
*** iceyao has joined #openstack-kolla09:51
*** shashank_t_ has joined #openstack-kolla09:51
*** shashank_t_ has quit IRC09:56
*** gfidente has joined #openstack-kolla10:00
*** gfidente has quit IRC10:00
*** gfidente has joined #openstack-kolla10:00
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Support non-x86 architectures (aarch64, ppc64le)  https://review.openstack.org/43094010:02
*** caoyuan has quit IRC10:13
openstackgerritMerged openstack/kolla master: debian: enable all images enabled for Ubuntu  https://review.openstack.org/43278710:14
*** caoyuan has joined #openstack-kolla10:15
*** caoyuan has quit IRC10:17
hrwyay!10:23
*** manheim has quit IRC10:25
openstackgerritBertrand Lallau proposed openstack/kolla master: Remove Ubuntu Trove postinst script hack  https://review.openstack.org/45137810:28
*** manheim has joined #openstack-kolla10:28
*** gkadam is now known as gkadam-mtg10:33
*** gfidente has quit IRC10:36
*** pbourke has quit IRC10:41
*** pbourke has joined #openstack-kolla10:43
*** magicboiz has quit IRC10:56
*** magicboiz has joined #openstack-kolla10:56
*** magicboiz has quit IRC10:58
*** dave-mccowan has joined #openstack-kolla11:06
*** magicboiz has joined #openstack-kolla11:07
*** jtriley has joined #openstack-kolla11:08
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: openstack-base: Percona-Server is x86-64 only  https://review.openstack.org/44996511:13
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: mariadb: handle lack of external repos on non x86  https://review.openstack.org/44483211:15
*** rwallner has joined #openstack-kolla11:16
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: handle ppc64le  https://review.openstack.org/43481011:17
*** rwallner has quit IRC11:20
*** rwallner has joined #openstack-kolla11:21
*** magicboiz has quit IRC11:21
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Support non-x86 architectures (aarch64, ppc64le)  https://review.openstack.org/43094011:23
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance  https://review.openstack.org/45373811:34
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance  https://review.openstack.org/45373811:34
*** magicboiz has joined #openstack-kolla11:35
*** caoyuan has joined #openstack-kolla11:38
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance  https://review.openstack.org/45373811:41
*** athomas has quit IRC11:44
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla master: Reparent kolla-toolbox from openstack-base  https://review.openstack.org/43502311:50
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Update keystone sanity check to use built in shade  https://review.openstack.org/43803011:50
*** athomas has joined #openstack-kolla11:54
*** gfidente has joined #openstack-kolla11:56
openstackgerritzhubingbing proposed openstack/kolla-ansible master: Test  https://review.openstack.org/45413511:58
*** jtriley has quit IRC12:00
openstackgerritweichuancheng proposed openstack/kolla-ansible master:  add clean osd ceph osd  disk info task  https://review.openstack.org/45413612:00
*** srwilkers has joined #openstack-kolla12:08
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Add tacker binary for rpm based distros  https://review.openstack.org/45414412:09
*** bjolo has joined #openstack-kolla12:09
*** goldyfruit has joined #openstack-kolla12:10
*** bjolo_ has quit IRC12:10
srwilkersmorning12:15
sbezverksrwilkers: good morning12:16
*** goldyfruit has quit IRC12:16
sbezverksrwilkers: please review this one if you have time: https://review.openstack.org/#/c/453934/12:16
srwilkerssbezverk, sure man. looking now12:21
*** jtriley has joined #openstack-kolla12:25
openstackgerritweichuancheng proposed openstack/kolla-ansible master:  add clean osd ceph osd  disk info task  https://review.openstack.org/45415312:26
*** rwallner has quit IRC12:31
*** rwallner has joined #openstack-kolla12:32
*** chas has quit IRC12:33
*** chas has joined #openstack-kolla12:34
*** chas has quit IRC12:39
*** rwellum has joined #openstack-kolla12:39
*** lamt has joined #openstack-kolla12:41
*** zhurong has quit IRC12:42
*** jtriley has quit IRC12:43
*** athomas has quit IRC12:45
*** chas has joined #openstack-kolla12:45
*** huzhengchuan has quit IRC12:47
*** chas has quit IRC12:49
*** mbruzek has joined #openstack-kolla12:50
*** mbruzek has joined #openstack-kolla12:50
*** eaguilar has joined #openstack-kolla12:51
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Remove neutron_metadata_socket volume from some agents  https://review.openstack.org/45416712:51
*** jaosorior has quit IRC12:52
*** jaosorior has joined #openstack-kolla12:53
*** eanylin has joined #openstack-kolla12:54
*** manheim has quit IRC12:58
*** goldyfruit has joined #openstack-kolla13:04
*** bjolo_ has joined #openstack-kolla13:05
*** chas has joined #openstack-kolla13:05
*** bjolo has quit IRC13:05
*** chas has quit IRC13:09
*** chas has joined #openstack-kolla13:11
*** Pavo has joined #openstack-kolla13:12
*** cristicalin has joined #openstack-kolla13:12
*** eaguilar_ has joined #openstack-kolla13:13
*** chas has quit IRC13:15
*** eaguilar has quit IRC13:16
*** manheim has joined #openstack-kolla13:16
*** eanylin has quit IRC13:19
*** gkadam-mtg has quit IRC13:21
*** eaguilar has joined #openstack-kolla13:24
*** jtriley has joined #openstack-kolla13:25
pbourkemnaser: egonzalez: got a minute? Any idea what im doing wrong here? https://review.openstack.org/#/c/453738/4/ansible/roles/rabbitmq/tasks/reconfigure.yml13:26
*** eaguilar_ has quit IRC13:26
* mnaser looks13:26
pbourkefailing with 'item' is undefined13:26
*** esharao has joined #openstack-kolla13:27
mnaseri wonder if the when doesn't get parsed as part of the with_items pbourke13:28
egonzalezpbourke, guess missing project_name in rabbitmq site https://review.openstack.org/#/c/453738/4/ansible/site.yml13:28
egonzalezit is in murano13:28
pbourkeyeah rabbitmq defaults has it though13:28
pbourkei was wondering that though I'll try it13:28
pbourkemnaser: also wondering that :)13:28
*** lrensing has joined #openstack-kolla13:29
mnaseraccording to bcoca "with_ clauses ALWAYS evaluate before the when clause, otherwise we cannot use item and condition loop iterations. "13:29
pbourkemnaser: it works in the original code though13:29
pbourkeah I think i see the problem13:31
sdakemorning13:31
mnaserwhat was it? im curious13:31
pbourkemnaser: role_rabbitmq_group should be role_rabbitmq_groups13:32
mnaserso i guess the undefined error propagated up13:32
pbourkewas just a really hard to parse error13:32
mnaser#justansiblethings13:32
pbourkeha13:32
pbourkeeverything is working now13:32
pbourkeexcept a vhost per service13:32
pbourkeneed to figure out how I'll template that13:32
mnaseri spent half a day once13:32
mnaserbecause someone created an empty folder with the role name13:32
mnaserin another folder which had a higher priority in roles_path13:33
pbourkewe get a lot of complaints about how hard ansible errors are to understand13:33
egonzalezhttps://s-media-cache-ak0.pinimg.com/736x/a2/9a/bc/a29abc6432badfba5106344c11c88029.jpg13:33
mnaser:P13:33
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance  https://review.openstack.org/45373813:34
openstackgerritnarasimha18sv proposed openstack/kolla-ansible master: enable CORS for Gnocchi and Keystone for grafana datasource  https://review.openstack.org/45418613:35
*** eanylin has joined #openstack-kolla13:35
*** skramaja has quit IRC13:36
*** iniazi_ has quit IRC13:37
*** shardy has quit IRC13:37
*** jmontleon has quit IRC13:38
*** Pavo has quit IRC13:38
*** iniazi_ has joined #openstack-kolla13:38
*** jmontleon has joined #openstack-kolla13:41
sdakesbezverk are you running cimc3.0 or cimc2.0?13:42
*** jmontleon has quit IRC13:43
*** mbruzek has quit IRC13:44
*** Pavo has joined #openstack-kolla13:45
sbezverksdake: 3.013:46
sdake3.0 is superior13:46
sdakethis machine is so quiet13:46
*** iceyao has quit IRC13:46
sbezverksdake: https://review.openstack.org/45393413:47
sbezverksdake: after kfox1111 renamed experimental gates and added new ones, it is completely broken13:47
sdakegot it13:47
*** jistr is now known as jistr|afk13:47
sbezverksdake: this PS at least restores ironic13:48
sbezverksdake: thanks. I left a comment last night in the deployment guild, I think it will be important to address it13:50
sdakesbezverk cool i will address as soon as I've got my box operational13:50
sdakesbezverk assuming I can do so :)13:50
*** jaosorior is now known as jaosorior_away13:55
openstackgerritMerged openstack/kolla-kubernetes master: Recovering ironic after gate job name change  https://review.openstack.org/45393413:55
sdakesbezverk do you ahe secure boot enabled on your system?13:55
sdakei took my pcie 400gb out of one of my minidelsl and stuck it in my c220m413:56
sdakeand it doesn't seem to recognize it in the uefi menu13:56
sdakehowever the uefi shell recognizes it13:56
sdakethe boot menu that is13:56
sdakei guess i'm going to try a reinstall13:56
sbezverksdake: sorry for next two hours I am hosting debug session, cannot chat..13:57
sdakeenjoy13:57
*** japestinho has quit IRC14:00
*** lamt has quit IRC14:00
*** manheim has quit IRC14:01
*** wagiel has joined #openstack-kolla14:01
*** manheim has joined #openstack-kolla14:02
*** haplo37 has quit IRC14:02
openstackgerritsean mooney proposed openstack/kolla-ansible master: introduce contrib playbook for ovs-dpdk  https://review.openstack.org/40887214:03
*** mbruzek has joined #openstack-kolla14:04
*** mbruzek has quit IRC14:04
*** mbruzek has joined #openstack-kolla14:04
openstackgerritsean mooney proposed openstack/kolla master: introduces support for the OVS DPDK dataplane  https://review.openstack.org/34235414:06
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla master: Make rabbitmq log dir configurable  https://review.openstack.org/45373514:06
*** manheim has quit IRC14:06
*** pomac_ has joined #openstack-kolla14:08
*** sean-k-mooney has joined #openstack-kolla14:08
*** manheim has joined #openstack-kolla14:10
sean-k-mooneyo/14:10
*** akwasnie_ has joined #openstack-kolla14:11
*** pomac has quit IRC14:11
*** akwasnie has quit IRC14:11
*** akwasnie_ is now known as akwasnie14:11
*** sindhu has quit IRC14:11
*** haplo37 has joined #openstack-kolla14:11
*** portdirect has quit IRC14:12
*** common has quit IRC14:12
*** common has joined #openstack-kolla14:12
*** portdirect has joined #openstack-kolla14:12
*** sindhu has joined #openstack-kolla14:13
*** berendt has quit IRC14:14
*** classicsnail has quit IRC14:14
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add vhost to external rabbitmq for Murano  https://review.openstack.org/45420614:15
*** berendt has joined #openstack-kolla14:15
pbourkemnaser: nice idea wrt vhosts in the external rabbitmq14:16
pbourkemnaser: all good to go14:16
*** classicsnail has joined #openstack-kolla14:17
mnaserpbourke looks good but i take it this isn't really tested in gate as part of the gate, or is it?14:18
pbourkemnaser: hmm I guess not14:18
* mnaser really wants to take sometime to implement scenario based testing like puppet-integration does14:18
mnaserhttps://github.com/openstack/puppet-openstack-integration - this is really good way of maximizing our coverage14:19
mnaserpbourke dont blame you though.. most new roles have zero testing in gate14:19
pbourkethe fact rabbit is green in the gate with all the tweaks gives a little confidence14:20
mnaserpbourke yeah that it doesn't affect the existing infra14:25
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Neutron: cleanup /lib/modules volume mount  https://review.openstack.org/45421414:26
*** targon has quit IRC14:27
*** shashank_t_ has joined #openstack-kolla14:27
manheimis there a known issue with the horizon 4.0.0 image from kolla dockerhub?14:27
manheimelements just floating around14:28
manheimubuntu/binary14:29
*** bjolo_ has quit IRC14:36
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Revert "Remove useless variable glance_backend_file"  https://review.openstack.org/45421814:36
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Revert "Remove useless variable glance_backend_file"  https://review.openstack.org/45421814:36
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Revert "Remove useless variable glance_backend_file"  https://review.openstack.org/45421814:37
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Only start one glance_api when using file backend  https://review.openstack.org/44865414:37
*** esharao_ has joined #openstack-kolla14:38
*** cristicalin has quit IRC14:38
*** magicboiz has quit IRC14:41
*** magicboiz has joined #openstack-kolla14:42
*** magicboiz has quit IRC14:46
*** magicboiz has joined #openstack-kolla14:47
*** yingjun has joined #openstack-kolla14:49
*** jmontleon has joined #openstack-kolla14:50
kfox1111morning.14:51
*** magicboiz has quit IRC14:51
*** manheim has quit IRC14:52
*** manheim has joined #openstack-kolla14:52
*** jistr|afk is now known as jistr14:53
*** manheim has quit IRC14:56
*** manheim has joined #openstack-kolla14:57
*** jmccarthy has joined #openstack-kolla14:57
*** matrohon has quit IRC15:03
sdakesup kfox111115:03
sdakekfox1111 my server is deplooyed ;)15:03
*** magicboiz has joined #openstack-kolla15:04
kfox1111sdake: cool.15:06
*** rmart04 has quit IRC15:08
*** sayantan_ has joined #openstack-kolla15:16
*** jmontleon has quit IRC15:18
*** chas has joined #openstack-kolla15:18
*** caoyuan has quit IRC15:20
mnaseruhm15:23
mnaserhas there been any thought into why we run vms inside the libvirt container15:23
mnaserthis kinda means that if our libvirt container ever gets goofy, we lose all the VMs inside of it if we need to restart15:23
*** jmontleon has joined #openstack-kolla15:24
*** jmontleon has quit IRC15:25
*** caoyuan has joined #openstack-kolla15:26
*** jmontleon has joined #openstack-kolla15:27
*** jmontleon has quit IRC15:27
*** pcaruana has quit IRC15:28
*** yangyapeng has joined #openstack-kolla15:28
*** jmontleon has joined #openstack-kolla15:29
*** jmontleon has quit IRC15:31
sbezverkkfox1111: can you look into 2 experimental jobs you added. everytime when check experimental is run, they got into queued state and never let other jobs to complete the run..15:31
kfox1111queued state?15:32
kfox1111thats very strange. that means the code isn't even running.15:32
*** jmontleon has joined #openstack-kolla15:33
*** manheim has quit IRC15:33
kfox1111got a ps to inspect?15:33
kfox1111is it upgrade jobs or 4.x jobs?15:33
*** jmontleon has quit IRC15:34
sdakemnaser this is not accurate - we use pid=host15:35
sbezverkkfox1111: just do check experimental on any ps and you will see15:36
sdakethe vms are ran in the host's PID namespace - if the libvirt container goes byebye, systemd inherits the vm processes15:36
sdakemnaser they are still completely controllable by a new libvirt bringup15:36
kfox1111sbezverk: just hoping you have one arleady going.15:36
mnasersdake interesting, never heard of that option15:36
mnaserthats great15:36
*** jmontleon has joined #openstack-kolla15:36
sdakemnaser yup - sorted that out about 2.5 years ago :)15:36
mnaserreally neat15:36
sdakemnaser its the standard in kolla - not an optional feature15:36
sdakeonly way to make libvirt work properly in a container15:36
* mnaser wonders how cgroups deal with that15:37
sbezverkkfox1111: unfortunately the ironic job I pushed to fix gate rename got already merged15:37
*** jmontleon has quit IRC15:37
sbezverkso nothing handy15:37
kfox1111sbezverk: ok. I'll try something then.15:37
zhubingbinghi guys15:38
*** blallau has quit IRC15:38
sbezverkkfox1111: thanks, cause it might cause infra the issue as experimental job sits there for hours because of two upgrade jobs in queued state15:38
egonzalezsup zhubingbing15:38
*** jmontleon has joined #openstack-kolla15:39
zhubingbinghi egonzalez15:39
kfox1111sbezverk: ultimately, we probably should talk to them.15:40
*** caoyuan has quit IRC15:42
kfox1111the config looks ok to me... lets ask infra15:44
*** chas has quit IRC15:46
sdakesbezverk is the ironic experimental job functional?15:46
*** chas has joined #openstack-kolla15:46
sbezverksdake: centos one yes15:46
sdakecool15:46
sdakeso I can make the centos one non-experimental?15:47
sbezverkkfox1111: but do you see the issue I am talking about?15:47
kfox1111sbezverk: yeah.15:47
*** zhurong has joined #openstack-kolla15:47
sbezverksdake: running ironic on main gate sounds a bit overkill, I mean it is not something like every other person would do15:48
sdakesbezverk the jobs run in parallel15:48
sdakesbezverk there is no such thing s "overkill" in a check job :)15:48
kfox1111sbezverk: but lots of things we change could kill that gate.15:48
kfox1111if its ready, lets move it to non experimental.15:48
sdakethe moer gates we have the more sasurance we have the sofftware is functional on master15:48
sbezverksdake: right but it will run every time, would you want to test ironic dpeloyment on every run?15:48
*** zhurong has quit IRC15:49
sdakesbezverk don't see why not15:49
kfox1111sdake: +1.15:49
sbezverkkfox1111: sure, I think it is stable enough15:49
*** fooliouno has quit IRC15:49
kfox1111k. lets promote it then. :)15:49
sdakein a perfect world we would have like 50 check jobs ;)15:49
sdakesbezverk its alsso the only thing that gates 4.0.0 iiuc15:50
*** chas has quit IRC15:50
*** vhosakot has joined #openstack-kolla15:53
kfox1111sbezverk: infra found the issue. ps up for review.15:54
*** chas has joined #openstack-kolla15:57
sbezverkkfox1111: great :)15:59
SamYaplemnaser: the cgroups are the same as the host. so the VM actually runs in the same cgroups as it would on the host. thats why if you arent using KVM then you _do_ lose all your VMs when you do something to the libvirt container16:03
mnaserSamYaple i see, cool, that clears out a lot of things16:04
berendthave you guys already seen https://github.com/openstack/loci?16:05
berendtLightweight OCI compatible images for OpenStack Projects16:05
berendtlooks a little bit redundant to our docker images16:06
SamYapleberendt: it was originally a PoC for a new structure here in Kolla. i moved it to its own project because it has different goals than kolla16:07
berendtSamYaple: is there a mission statement or something like this?16:07
SamYaplewould be nice to merge it back one day, but its unlikely to take those technics and apply them to kolla due to the differences16:07
SamYapleberendt: not as yet. but it doesnt include _deployment_ in any form. its generic images with CICD in mind16:09
SamYaplemnaser: technically, we could drop pid=host from the libvirt container, but it requires a bit of tweaking and it isn't worth it in the end16:09
SamYaplemnaser: but its doable!16:10
openstackgerritMerged openstack/kolla-ansible master: Update keystone sanity check to use built in shade  https://review.openstack.org/43803016:10
*** caoyuan has joined #openstack-kolla16:10
*** chas has quit IRC16:13
*** chas has joined #openstack-kolla16:13
*** jaosorior_away has quit IRC16:14
*** jaosorior has joined #openstack-kolla16:15
*** chas has quit IRC16:17
*** yingjun has quit IRC16:20
*** nathharp has quit IRC16:20
mnaserSamYaple if you dont mind because i see a comment of yours over here - https://github.com/openstack/kolla-ansible/blob/master/ansible/library/kolla_docker.py#L471-L48716:23
mnaserim trying to run against centos 7 distro shipped docker and it seems the pull returns this status instead:  Digest: sha256:c0929c9637f0332d973454a3675a56a76f281bbe71f5e1b57bb5f7bb1032542e16:23
mnaserresulting in the predictable failure of - {"changed": false, "failed": true, "msg": "Unknown status message: Digest: sha256:1e19adc5532ddfd741440d3c07ee0a0de0dc77b4699ba67464069059be9d0552"}16:24
mnaserany ideas how to best work around this or clear it up?  we'd kinda like to stick to RH shipped docker rpms16:24
SamYaplemnaser: what version is the RH docker?16:24
mnaserSamYaple docker-1.12.6-11.el7.centos.x86_6416:25
mnaserand as far as i know, that should be supported16:25
SamYaplehmmm. mnaser that is far newer than when i wrote that block16:25
SamYapleyou should dump the entire status object16:25
*** yangyapeng has quit IRC16:25
mnasersomeone ran to the same issue here - https://bugs.launchpad.net/kolla/+bug/166805916:25
openstackLaunchpad bug 1668059 in kolla "Failed to Starting kolla-toolbox container" [Critical,Triaged]16:25
mnaseralright, ill try to see if there is something that can help identify if something changed or didnt16:26
SamYaplemnaser: youll need to dump the status object before L47116:26
*** egonzalez has quit IRC16:26
SamYapleonce you dump the object the answer should become clear, but ping me if it isnt16:26
*** Mech422 has quit IRC16:26
SamYaplethis will probably require a patch16:26
SamYaplethe docker hub just went through a change and it broke things16:27
mnaserSamYaple no worries, i can write it up (this is pulling images from registry:2 tho)16:27
SamYaplemnaser: hmmmm16:28
SamYaplewell anyway, dumping that object will help16:29
*** caoyuan has quit IRC16:35
*** chas has joined #openstack-kolla16:35
*** ksumit has joined #openstack-kolla16:36
mnaserSamYaple http://paste.openstack.org/show/605638/16:38
*** scsnow has quit IRC16:38
mnaserso looks like the status is omitted for some reaosn16:39
SamYaplemnaser: looks like the response has changed16:43
mnaserSamYaple tho i do have a machine running docker-engine-1.11.0-1.el7.centos.x86_64 with no issues16:43
SamYapleso what you need to find out is if that message means it pulled a _new_ image, or its still the old image16:43
mnaserand i was testing with 17.03 and it didnt have that issue16:43
mnaserso i wonder if its some weird redhat thing.16:43
SamYapleyou may need to record the sha of the current image and then compare after pull16:43
*** zhubingbing has quit IRC16:44
SamYaplemnaser: id be willing to be its some edge result you are triggering in docker itself16:44
mnaser#justdockerthings16:44
SamYaplemnaser: that whole section is just for idempotency results16:44
SamYaple:)16:44
mnaseryeah16:44
*** eragavi has joined #openstack-kolla16:44
mnaseri notice that even when i do docker pull16:45
mnaserit doesnt report the status at the end16:45
mnaser(using the cli)16:45
SamYapleyea mnaser, just right in a test case for it. i think the best may be to do an image compare if its missing status16:45
SamYaplerecord image sha first, then if you cant parse status fall back to comparing sha16:46
SamYapleor hell, just replace the whole section with a compare sha section16:46
SamYapleend result will be the same16:46
mnaseryeah16:46
mnaserif image doesnt exist, old_sha1 will just be none and changed=True16:46
mnaseras in old_sha1 != new_sha116:47
mnaserbut seriously, is this how the docker api works?  just plain text?16:47
mnaserno structure of the sha116:47
*** jaosorior is now known as jaosorior_away16:48
SamYaplemnaser: that is correct16:48
SamYapledocker api is awful16:48
SamYapleits always changing to16:48
SamYaplenot compatible16:48
mnasermight as well as just send plain text16:48
mnaserlol16:48
SamYaplethey do16:49
SamYaplethey just wrap it in json to be ReSTful16:49
mnaser[r for r in respose.split("\n")] <= we have a json api now16:49
mnaserok it looks like docker-py has an api for managing images16:50
mnaserreturns an image object16:50
mnaser(or we could just use docker_image the ansible built in but /shrug(16:51
*** fooliouno has joined #openstack-kolla16:53
*** rmart04 has joined #openstack-kolla16:54
mnaseroo this is gonna be a nice refactor.  if it works.16:56
*** rmart04 has quit IRC17:01
kfox1111sbezverk: the jobs should be fixed.17:05
*** unicell has quit IRC17:05
jmccarthyHmm ok kolla_docker.py / ansible question - is there a way to grab the snippet ansible wants to run on a failed task ? I tried setting ANSIBLE_KEEP_REMOTE_FILES=1 but it didn't work. I have an odd (unsupported?) situation where one of my target nodes is python 2.6 and want to try and run it manually like on the node17:07
*** mgoddard has quit IRC17:07
kfox1111sbezverk: that ps wasn't rechecked after the version 3 fix landed.17:08
kfox1111should be fine. do you want to do another recheck?17:08
sbezverkkfox1111: I would prefer to be sure17:08
kfox1111k.17:09
sbezverkthere is no urgency to delete poor glance ;)17:09
kfox1111true.17:10
*** chas has quit IRC17:15
*** gfidente has quit IRC17:15
*** chas has joined #openstack-kolla17:15
*** chas has quit IRC17:20
*** harlowja has joined #openstack-kolla17:23
*** shashank_t_ has quit IRC17:30
*** shashank_t_ has joined #openstack-kolla17:31
*** vhosakot_ has joined #openstack-kolla17:32
*** vhosakot has quit IRC17:32
*** ksumit has quit IRC17:33
openstackgerritMerged openstack/kolla-kubernetes master: Make Ironic honor dependencies  https://review.openstack.org/45393117:37
*** mbruzek has quit IRC17:38
*** unicell has joined #openstack-kolla17:39
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Switch kolla_docker to rely on SHA256 for image changes  https://review.openstack.org/45426517:39
sdakeboy having an HTMLv5 console is so epic17:40
sdakeas opposed to a pile of hacks to connect my server to my display17:40
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Fix documentation for kolla_docker image pulling  https://review.openstack.org/45426617:40
*** jmontleon has quit IRC17:41
mnaserSamYaple https://review.openstack.org/#/c/454265/17:46
mnaserappreciate any input17:46
esharao_Hi folks..17:46
esharao_having some issues with canal using 1.6.1 of k8s17:46
esharao_ cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d17:47
esharao_anyone else faced similar issue?17:47
SamYaplemnaser: already reviewed17:47
mnaserinteresting issue17:47
*** jmontleon has joined #openstack-kolla17:48
*** jmontleon has quit IRC17:48
*** mgoddard has joined #openstack-kolla17:49
*** jmontleon has joined #openstack-kolla17:50
*** ksumit has joined #openstack-kolla17:50
*** ksumit has quit IRC17:51
*** manheim has joined #openstack-kolla17:52
kfox1111esharao_: haven't seen that.17:54
esharao_hmm tried it on 2 diff Vms running centos and i see the same issue :(17:55
kfox1111was it working on 1.6.0 or did you come from 1.5?17:56
*** manheim has quit IRC17:57
esharao_hmm it was working on 1.6.018:03
*** mbruzek has joined #openstack-kolla18:03
*** mgoddard has quit IRC18:04
sdakeesharao_ note I am running through the instructions right now on a freshly deployed peice of bare metal18:07
esharao_kool sdake..18:07
*** jascott1_ has joined #openstack-kolla18:10
*** jascott1_ has quit IRC18:10
*** vhosakot has joined #openstack-kolla18:11
*** vhosakot_ has quit IRC18:14
openstackgerritQin Wang (qwang) proposed openstack/kolla-kubernetes master: [WIP] Condition and tags in requirements.yaml for keystone service  https://review.openstack.org/44446118:18
openstackgerritPradeep Kilambi proposed openstack/kolla master: Add ceilometer agent ipmi docker file  https://review.openstack.org/45428118:18
*** lucasxu has joined #openstack-kolla18:22
inc0hello18:23
sdakegah step 1 for me enters a crashlopobackoff18:24
esharao_sdake .. same here18:25
sdakehttps://da.gd/wwXmT -> https://paste.fedoraproject.org/paste/vGcUqAPSlO90-3jnpD9My15M1UNdIGYhyRLivL9gydE=/18:25
sdakeesharao_ you have the same probelm as I do?18:25
sdakeesharao_ this *was* working yesterday18:25
sdakewhich implies something broke upstream18:25
srwilkershey inc018:26
esharao_my canal-etcd is going into crashlookBackoff (this is happening since yesterday afternoon)18:26
inc0hey Steve18:26
srwilkershows it going dude?18:26
inc0all good18:27
esharao_sdake i see this in the logs - http://paste.openstack.org/show/605651/18:27
SamYapleinc0: im here too!18:27
srwilkersSamYaple, go away18:28
inc0lots of meetings:)18:28
sdakeesharao_ which command do you run to get that output?18:28
inc0hey Sam, hows LOCI doing? You still like us?18:28
inc0:P18:28
sdakeesharao_ i'll verify if i have same result in my env18:28
esharao_sdake - that was in the /var/log/messages18:28
sdakeif so - should be easy enough to sort out18:28
inc0j/k18:29
SamYaplesrwilkers: :(18:29
*** mbruzek has quit IRC18:29
esharao_i did change log level to debug in canal.yaml18:29
inc0I know you do18:29
SamYapleinc0: pfff you think you could get rid of me?18:29
SamYapleim always here. lurking18:29
srwilkersSamYaple, just kidding. i <3 yoj18:29
inc0I need to drink you to uncounciousness and steal your ssh keys for gerrit18:30
SamYapleyea yoj is pretty cool18:30
SamYapleinc0: i use passwords for gerrit18:30
inc0good, even easier18:30
SamYaplei pasted in my private ssh key as the password18:30
inc0does your password invole horses and staples?18:30
*** jascott1_ has joined #openstack-kolla18:30
jascott1_o/18:31
SamYaplejascott1_: o/18:31
jascott1_what up SamYapl?18:31
jascott1_+e18:31
srwilkersyo jascott1_18:31
jascott1_hey srwilkers18:31
jascott1_I guess inc0 will have to represent at summit since I cannot go :/18:32
*** mgoddard has joined #openstack-kolla18:32
jascott1_he will have to have all my drinks18:33
inc0I'll drink your share18:33
jascott1_:D18:33
inc0see? great minds think alike18:33
srwilkersjascott1_, this saddens me18:33
inc0so are minds of pathological alcoholics18:33
jascott1_lol18:33
jascott1_+cold_medicine18:33
SamYaplejascott1_: :( who will we all bum cigarettes from?18:34
jascott1_haha18:34
*** rwallner has quit IRC18:34
SamYapleyoure the only one that remembers to buy them18:34
inc0in Poland we had cold medicine that in huge doses gave you trip like LSD18:34
jascott1_who will the entire city of Boston bum cigarettes from?18:34
jascott1_inc0 wow was that over the counter?18:34
jascott1_OTC!18:34
inc0indeed it was18:34
inc0probably it also made your liver cry bloody tears18:35
inc0but who cares18:35
sdakeesharao_ how did you enable debugging in calico?18:35
jascott1_shutup whinny liver!18:35
sdakeinc0 we have cold medication like that in the US too18:35
sdakejascott1_ probably rbergeron18:35
sdakerather in canal esharao_18:36
*** mkoderer has quit IRC18:37
jascott1_sdake rbergeron?18:38
*** mbruzek has joined #openstack-kolla18:38
*** mbruzek has quit IRC18:38
*** mbruzek has joined #openstack-kolla18:38
sdakejascott1_ indeed - for bumming cigs18:38
esharao_sdake http://paste.openstack.org/show/605653/18:38
sdakeshe brings atleast a carton for other peole18:38
jascott1_ah haha18:38
jascott1_smart18:38
jascott1_srwilkers have you gotten to do anything with openstack-golang?18:40
srwilkersjascott1_, not yet sadly18:40
jascott1_i havent either but hope to very soon18:40
*** Manheim has joined #openstack-kolla18:42
*** mbruzek has quit IRC18:45
*** MasterOfBugs has joined #openstack-kolla18:46
sdakeesharao_ here is the error I get: https://da.gd/vEVh -> https://paste.fedoraproject.org/paste/sJVVhMpUPJ3YtqP2KVWpTV5M1UNdIGYhyRLivL9gydE=/18:46
sdakeesharao_ which is diferent from yours18:46
sdakekfox1111 sbezverk any ideas on what that crap means in that log? ^^18:47
*** mbruzek has joined #openstack-kolla18:47
*** mbruzek has quit IRC18:47
*** mbruzek has joined #openstack-kolla18:47
sdakeApr  6 04:45:38 beast-01 kubelet: E0406 04:45:38.506399   15897 pod_workers.go:182] Error syncing pod 8e6ff0fb-1aba-11e7-b580-2cd02dbe3274 ("kube-dns-3913472980-j1n6h_kube-system(8e6ff0fb-1aba-11e7-b580-2cd02dbe3274)"), skipping: failed to "SetupNetwork" for "kube-dns-3913472980-j1n6h_kube-system" with SetupNetworkError: "NetworkPlugin cni failed to set up pod \"kube-dns-3913472980-j1n6h_kube-system\" network: open /run/flannel/subnet.env: no such18:48
sdake file or directory"18:48
esharao_sdake.. i get similar error as well18:48
esharao_i did not paste all that.. in the original error..18:48
sdakeesharao_ we debug from the top of the error not ht ebottom :)18:49
kfox1111maybe fluentd is not started when it tried.18:49
kfox1111may not be a real issue.18:49
esharao_http://paste.openstack.org/show/605655/18:49
sdakekfox1111 something is rela aobut it - canal enters a crashloopbackoff18:49
kfox1111yeah. kube-dns relies on canal. so that make sense. just a symptom.18:50
esharao_sdake i agree.. thats why i looked at it and then the original paste was what i thougth is the root cause for me18:50
sdakeesharao_ maybe i didn't go far enough back18:50
sdakeesharao_ let me look furthe rback18:50
esharao_sdake.. i found that error message after kubectl delete -f canal.yaml and then a re-apply18:51
kfox1111not sure how safe it is to delete canal.18:51
*** manheim_ has joined #openstack-kolla18:52
*** manheim_ has quit IRC18:52
*** manheim_ has joined #openstack-kolla18:53
sdakeesharao_ ls -l /opt/cni/bin -> paste plz18:54
sdakeesharao_ follow that up with getenforce18:54
esharao_one sec18:54
esharao_sdake.. you got it. getenforce was enforcing changed and trying again18:57
sdakeesharao_ you will probably ahve to reboot18:58
sdakeesharao you can permeantly trun off selinux by modifying /etc/selinux/config18:59
sdakeesharao yo uwill probably ahe to tear down your env18:59
sdakenote I dont have selinux enabled18:59
sdakeso that isn't my problem :(18:59
esharao_ahh ok.. i did reboot afrter i ran clean.sh thats where i ran into issue18:59
sbezverksdake: I would guess, etcd for canal is not confiugred properly19:09
kfox1111sbezverk: https://review.openstack.org/#/c/446274/ seems happy with ironic now.19:09
esharao_sdake it now works.. thanks19:15
sdake[preflight] Some fatal errors occurred:19:17
sdake/proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 119:17
sdake[preflight] If you know what you are doing, you can skip pre-flight checks with `--skip-preflight-checks`19:17
sdakeany ideas?19:17
rwellumsdake:  https://www.irccloud.com/pastebin/88wMFKWq/19:23
rwellumThat's what I did yesterday19:23
rwellumIt's the 'why' that eludes me.19:23
sdakerwellum i'm right there with ya :)19:25
*** eaguilar has quit IRC19:25
sdakerwellum i dont see that in default centos19:25
sdakerwellum did you add it in?19:25
rwellumYes added it in, ran 'sudo sysctl -p'19:26
*** shashank_t_ has quit IRC19:26
rwellumRemember I'm using a VM so I am a little less concerned about borking things up19:26
sdakeApr  6 04:21:11 beast-01 dockerd-current: time="2017-04-06T04:21:11.038059994-07:00" level=error msg="Handler for GET /v1.24/images/quay.io/calico/cni:v1.6.2/json returned error: No such image: quay.io/calico/cni:v1.6.2"19:27
*** shashank_t_ has joined #openstack-kolla19:27
sdake[root@beast-01 ~]# docker pull quay.io/calico/cni/v1.6.219:28
sdakeUsing default tag: latest19:28
sdakeTrying to pull repository quay.io/calico/cni/v1.6.2 ...19:28
sdakeGet https://quay.io/v2/calico/cni/v1.6.2/manifests/latest: name invalid: Nested repositories are not supported. Found: calico/cni/v1.6.219:28
*** bmace has quit IRC19:28
*** bmace has joined #openstack-kolla19:28
sdakeoops had a typo19:28
sdakehmm - time to reload my os I guess19:28
* sdake growls19:29
openstackgerritnarasimha18sv proposed openstack/kolla-ansible master: enable CORS for Gnocchi and Keystone for grafana datasource  https://review.openstack.org/45418619:29
*** rwallner has joined #openstack-kolla19:29
*** rwallner has quit IRC19:34
*** openstack has joined #openstack-kolla19:48
rwellumsdake: sure19:50
jmccarthyHmm is there much call for support of python 2.6 ? Dictionary comprehension in kolla_docker.py is giving me grief at the moment :/19:50
kfox1111I think 2.6 is officially unsupported now?19:55
kfox11112.6 was kept around for el6 support which is not supported anymore?19:55
kfox1111the goal is for 3.x support which is so much easier when 2.6 isn't supported.19:55
kfox1111since 3.x and 2.7 are mostly compatable but 2.6 isn't.19:55
*** lrensing has quit IRC20:02
*** harlowja has quit IRC20:03
*** leseb has quit IRC20:12
sbezverkkfox1111: has fernet got merged?20:12
kfox1111sbezverk: needs a second still.20:13
*** imcsk8 has quit IRC20:13
sbezverkkfox1111: got it, I wanted to get it running on my cluster20:13
kfox1111go for it. :) let me kow if you run into any issues.20:14
*** Pavo has quit IRC20:14
kfox1111if you get it running, that would be great. there's one missing feature to fully complete it.20:14
kfox1111to tweak the job to be optionally a cronjob.20:15
kfox1111might be easier to test that one on a real system.20:15
*** imcsk8 has joined #openstack-kolla20:21
*** jmccarthy has quit IRC20:21
sbezverkkfox1111: will do as soon as it gets merged..20:24
kfox1111k.20:24
kfox1111sdake: inc0: https://review.openstack.org/#/c/446274/ please20:24
*** Serlex has quit IRC20:25
*** markmcclain has quit IRC20:26
*** markmcclain has joined #openstack-kolla20:27
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735620:30
sdakeok20:30
sdakerwellum eanylin esharao want to give that a spin?20:31
rwellumsdake: sure but it'll be a bit later for me.20:31
sdakerwellum ack20:32
sdakekfox1111 can you ahe a review of that deploy doc20:33
sdakei really want ot get it merged so people can actually see it rendered20:33
sdakeeven if it isn't quite right20:33
sdakethat will help me figure out how to make it right faster20:33
*** ksumit has joined #openstack-kolla20:34
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Switch kolla_docker to rely on SHA256 for image changes  https://review.openstack.org/45426520:34
*** jtriley has quit IRC20:36
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Remove un-used Keystone test path  https://review.openstack.org/45433320:39
*** leseb has joined #openstack-kolla20:39
rwellumsdake: did you successfully deploy a VM?20:40
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/iscsi  https://review.openstack.org/45285420:40
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/mariadb  https://review.openstack.org/45285520:40
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/horizon  https://review.openstack.org/45285220:40
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/keystone  https://review.openstack.org/45285320:40
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/glance  https://review.openstack.org/45285120:40
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Modify WARNING into RST format  https://review.openstack.org/45286020:40
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/rabbitmq  https://review.openstack.org/45285820:40
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/skydns  https://review.openstack.org/45285920:40
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/memcached  https://review.openstack.org/45285620:40
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove services/neutron  https://review.openstack.org/45285720:40
sdakerwellum i can't get kubernetes runnig nnow...20:40
sdakerwellum i have a new machine20:40
openstackgerritMerged openstack/kolla-kubernetes master: Fernet Token Support  https://review.openstack.org/44627420:40
sdakerwellum although the guide was GTG 2 days ago from my pov :)20:40
sdakerwellum it needs a bit of work still - however- without it rendered evyerone is looking in gerrit  tryign to figure out what it means20:40
sdakerwellum whereas a rendering - everyoen works on same copy20:41
jemcevoyinc0:  Did you want to park at my place and take the streetcar to the streetcar to the PDX CloudNative Meetup or will you be taking the MAX?20:41
rwellumsdake: I'm fine with that.20:42
kfox1111sdake: reviewed.20:43
*** david-lyle has quit IRC20:44
*** jrobinson has joined #openstack-kolla20:45
kfox1111sbezverk: merged.20:48
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Remove python-ceilometer from test-requirements.txt  https://review.openstack.org/45433720:49
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Remove python-barbicanclient from test-requirements.txt  https://review.openstack.org/45433820:49
sbezverkkfox1111: nice..20:50
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Remove python-heatclient from test-requirements.txt  https://review.openstack.org/45433920:52
*** haplo37 has quit IRC20:53
*** g3ek has quit IRC20:53
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Remove python-swiftclient from test-requirements.txt  https://review.openstack.org/45434020:53
mnaserok ive made enough noise20:53
*** wagiel has quit IRC20:55
openstackgerritMohammed Naser proposed openstack/kolla-ansible master: Remove graphiz from test-requirements.txt  https://review.openstack.org/45434220:55
sdakekfox1111 thanks20:56
*** mgoddard has quit IRC20:56
sdakekfox1111 good catch20:59
*** openstackstatus has quit IRC20:59
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735621:00
*** openstackstatus has joined #openstack-kolla21:01
*** ChanServ sets mode: +v openstackstatus21:01
*** g3ek has joined #openstack-kolla21:02
*** haplo37 has joined #openstack-kolla21:03
*** harlowja has joined #openstack-kolla21:03
kfox1111sdake: missed one: https://review.openstack.org/#/c/447356/46..47/doc/source/deployment-guide.rst21:03
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735621:03
sdakeindeed21:04
* sdake fires himself21:04
sdakeok - gotta go get kid from school21:04
kfox1111+221:04
sdaketx21:04
sdakeinc0 sbezverk ^^21:04
kfox1111interesting.... the newer flannel daemonset doesn't need an etcd anymore.21:06
kfox1111I may need to start adopting it on some of my clouds now.21:06
*** openstackgerrit has quit IRC21:07
sbezverksdake: looks almost ok, but you did not address my comment, hence -121:07
*** chatter29 has joined #openstack-kolla21:07
chatter29hey guys21:07
chatter29allah is doing21:07
chatter29sun is not doing allah is doing21:07
chatter29to accept Islam say that i bear witness that there is no deity worthy of worship except Allah and Muhammad peace be upon him is his slave and messenger21:07
*** chatter29 has quit IRC21:10
*** lamt has joined #openstack-kolla21:11
mnasersigh..21:15
*** ksumit has quit IRC21:25
*** fooliouno has quit IRC21:26
sdakesbezverk do you mean the stuff about ubuntu?21:31
sdakesbezverk ubuntu will be handled in a followup21:31
sdakesbezverk if you mean something else I am happy to address it now21:31
sbezverksdake: regardless, the right approach is to check what docker uses as a driver and use the same21:32
sbezverkthen you becomes version independent21:32
sbezverkdocker info | grep cgroup21:32
sdakeworking on it21:33
sbezverkthen take that driver name and use it in kubelet command line21:33
sbezverkvery very easy21:33
*** lucasxu has quit IRC21:36
*** ssurana has joined #openstack-kolla21:36
*** openstackgerrit has joined #openstack-kolla21:42
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735621:42
sdakesbezverk ^21:42
kfox1111sbezverk: its extremely sad kubelet doesn't just do that itself. :/21:42
sdakeindeed21:42
sdakeespecially since kubelet can run with other CRIs21:42
sbezverkkfox1111: agree21:43
rwellumsdake -there are some points in the guide currently that really only apply to bare-metal. I haven't commented on these because it's a deployment guide, not a development guide. Is that ok?21:44
sdakerwellum i htink its ok21:44
sdakerwellum if people use it as a dev guide, wfm :)21:44
sbezverksdake: not that it matter, but normally you try to keep ip addresses groupped21:44
sdakeour dev env is busted atm21:44
sbezverkexample instead of using .100 for canal etcd, I would suggest to use .11 which is next available after kube-dns21:45
rwellumSo that include the $(hostname) issue I have? I don't want to comment on it again.21:45
sdakesbezverk i just want to get what has worked for others on docs.oo :)21:46
sdakerwellum not sure how to solve that one21:46
sdakerwellum i think we were ogin gto webex earlier this week and didn't to sort out a command for that21:46
sbezverksdake: in the networking world these things tare taken seriously21:46
sdakei doubt anyone that is serious will read our guide and think "thats how to deploy kubernetes, done and out" :)21:47
sdakeatleast it explains the steps :)21:47
sbezverksdake: that is why I did not commented in the doc, it is just FYI21:47
sdakethanks - ithis is a living doc21:48
sdakei am hopeful kubeadm does a better job of documenting their own project in the future21:48
sdakeatm they dont21:48
kfox1111sdake: its beta. only so much can be done.21:49
sdakeagreed21:49
sdakeparts of it are alpha21:50
kfox1111but, yeah. :)21:50
sdakeso if you ack it i can go PTFO :)21:55
sdakeTIA21:55
rwellumReview sent sdake - sorry a few more issues21:55
sdakeyou guys are killing me :)21:55
rwellum'50' is the magic number21:56
sbezverkrwellum: why stop on 50, let's make it 100 ;)21:58
rwellumThere were much more pedantic reviewers than me at Cisco - I was one of the better one fwiw... :)21:59
*** openstack has joined #openstack-kolla22:04
sbezverksdake ubuntu 16.0422:04
*** dasTor_ has joined #openstack-kolla22:04
sdakewerid22:04
sdakedoes it not use systemd?22:04
sbezverkit defaults to cgroupfs22:04
sdakei see22:04
sbezverkfor docker22:04
sdakethanks22:04
rwellumsdake: +1 for me and thanks!22:05
sdakerwellum thank you :)22:05
*** dasTor has quit IRC22:05
kfox1111helm v2.3.0 released.22:05
sbezverkkfox1111: oh man, a bit of stability at least for a few days ;)22:06
kfox1111hehe.22:09
sdakefinally22:10
sdakedid someone merge the cinder patch btw?22:10
sdakethe cinder removal?22:10
kfox1111one of the removal patches merged.22:10
sdakecool22:10
kfox1111dont remember which.22:10
sdakei resubmitted them so i didn't have to type recheck 10 times22:10
sdakebekuz i'm alzy :)22:10
sdakecinder was the first one22:10
sdakekfox1111 can you ack: https://review.openstack.org/#/c/447356/22:11
kfox1111done.22:13
*** mbruzek has quit IRC22:14
*** srwilkers has quit IRC22:14
*** chas has joined #openstack-kolla22:16
*** chas has quit IRC22:20
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Change maxdepth to 1 so docs are more readable  https://review.openstack.org/45437322:29
*** Manheim has quit IRC22:29
esharao_sdake have you seen 403 error for placement api?22:30
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: The dnsguide is outdated now that kubernetes includes dns  https://review.openstack.org/45437522:30
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove labels.rst  https://review.openstack.org/45437622:31
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove host-setup  https://review.openstack.org/45437722:32
*** rhallisey has quit IRC22:33
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove kubernetes-setup.rst  https://review.openstack.org/45437822:33
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove installation.rst  https://review.openstack.org/45437922:34
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Update to Helm v2.3.0  https://review.openstack.org/45438022:35
*** esharao has quit IRC22:35
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove quickstart.rst  https://review.openstack.org/45438122:36
sdakeesharao_ no22:36
*** esharao has joined #openstack-kolla22:36
sdakeesharao_ however i haven't looked22:36
sdakeesharao ^^22:36
*** manheim has joined #openstack-kolla22:37
*** esharao_ has quit IRC22:40
sbezverksdake: you need to change your nick for something like "eraser" or "destroyer" ;) why not to delete these files in one shot ??22:40
sdakesbezverk i want them to be individually judged22:41
*** esharao has quit IRC22:41
sdakeI'm not sure if there is somethign someone wants to keep22:41
sdakeif i delete everything its very difficult to add it back in - versus a rebase - easy to drop a patch22:41
sdakesdake DESTROYER OF WORLD22:41
*** eragavi has quit IRC22:41
sdakes22:41
sdakesweet Jeffrey4l added our docs-only gate :)22:44
sdakeJeffrey4l thanks dude :)22:45
openstackgerritMerged openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735622:46
mnaseranyone ever heard this? https://rook.io/22:48
kfox1111mnaser: no. weird....22:54
kfox1111is it a fork of ceph?22:54
mnaserkfox1111 still trying to get it.. it seems confused at what it is22:54
mnaseris it a fork is it a deployment tool is it a bunch of kuberentes scripts on top22:55
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove installation.rst  https://review.openstack.org/45437922:56
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove kubernetes-setup.rst  https://review.openstack.org/45437822:56
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove host-setup  https://review.openstack.org/45437722:56
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove labels.rst  https://review.openstack.org/45437622:56
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove quickstart.rst  https://review.openstack.org/45438122:56
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove outdated DNS guide  https://review.openstack.org/45437522:56
kfox1111mnaser: weird.22:56
mnasersdake while you're at it with docs is this necessary to live in kolla-ansible? https://github.com/openstack/kolla-ansible/blob/master/specs/kubernetes-deployment.rst22:57
mnaserits duplicated here - https://github.com/openstack/kolla/blob/master/specs/kubernetes-deployment.rst22:57
mnaserand i think its more related to kolla rather than kolla-ansible22:57
sdakemnaser we dont touch the specs dir22:57
mnasereven if its duplicated?22:57
sdakesomeone might be willing to approve it but i'm not submitting it :)22:58
sdakesepcs are immutable22:58
sdakeonce they go in they dont come out22:58
mnaseryeah i agree with that 100%22:58
sdakethey are also advisory22:58
sdakesomeone else might make a different call as a reviewer22:59
sdakethat has always been my policy and the project's policy in general22:59
sdake(to avoid specs that is :)22:59
*** sayantan_ has quit IRC22:59
mnasersdake i agree with you, just having them duplicated in two places seems confusing.  i feel like that one would belong in kolla and not kolla-ansible23:00
mnaseranyawys, far more important things to do than that, just a small personal nit :D23:00
*** jascott1_ has quit IRC23:00
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove keystone guide  https://review.openstack.org/45438823:05
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove mariadb guide  https://review.openstack.org/45438923:06
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove memcached guide  https://review.openstack.org/45439023:06
*** jemcevoy has quit IRC23:07
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove multi-node guide.  https://review.openstack.org/45439223:07
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Add warning to development environment documentation  https://review.openstack.org/45439323:09
*** adil452100 has joined #openstack-kolla23:09
*** adil452100 has quit IRC23:10
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: py34 is no longer supported in our tox.ini  https://review.openstack.org/45439423:12
*** vhosakot has quit IRC23:14
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Remove usage.rst  https://review.openstack.org/45439823:14
sdakerwellum eanylin https://docs.openstack.org/developer/kolla-kubernetes/deployment-guide.html23:17
*** Pavo has joined #openstack-kolla23:17
sdakeour docs wont look like such a disaster as soon s the aboe patches are merged ;)23:17
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment guide had a formatting error  https://review.openstack.org/45439923:19
sdakesbezverk kfox1111 can you merge https://review.openstack.org/454399 plz23:19
*** manheim has quit IRC23:20
sdakeso - one of many documentation purges we will probably undergo as a deliverable23:21
sdakekolla itself went through 4 or 5 during its first 2 years :)23:21
*** jascott1_ has joined #openstack-kolla23:25
*** shashank_t_ has quit IRC23:26
*** shashank_t_ has joined #openstack-kolla23:27
*** sayantan_ has joined #openstack-kolla23:30
*** shashank_t_ has quit IRC23:31
*** lrensing has joined #openstack-kolla23:37
*** imcsk8 has quit IRC23:40
*** lrensing has quit IRC23:43
*** jascott1_ has quit IRC23:45
*** jascott1_ has joined #openstack-kolla23:45
*** jtriley has joined #openstack-kolla23:47
*** imcsk8 has joined #openstack-kolla23:47
*** lrensing has joined #openstack-kolla23:48
*** david-lyle has joined #openstack-kolla23:48
kfox1111sbezverk: still seem to have a race maybe on multinode bringup. http://logs.openstack.org/80/454380/1/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/fb9b79423:49
*** eaguilar has joined #openstack-kolla23:51
*** jascott1_ has quit IRC23:51
*** goldyfruit has quit IRC23:52
spsuryamorning folks23:53
sdakecheck out this review: https://review.openstack.org/#/c/439740/23:55
sdakedell ironic CI voting on it23:55
sdakewho woulda guessed23:55
*** lrensing has quit IRC23:56
spsuryasdake: Deployment guide rocking23:57
sdakespsurya its merged atleast :)23:57
sdakewhether it works or not, thats another matter :)23:57
spsuryasdake: yesteryday I left some comment23:57
spsuryasdake: I saw23:57
sdakespsurya now you can submit reviews for changes23:58
spsuryavery nice23:58
mnasersdake that was a bug way back (the ci)23:58
sdakemnaser ok23:58
mnaserdell ironic ci was throwing that warning for every single review coming into openstack23:58
mnaser:p23:58
spsuryasdake: sure I will23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!