Thursday, 2018-10-04

*** jhebden has joined #tripleo00:02
*** jcoufal has quit IRC00:03
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179256000:10
openstackLaunchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr)00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287200:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571800:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)00:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]00:10
*** agopi__ is now known as agopi00:12
*** jcoufal has joined #tripleo00:15
*** spartakos has quit IRC00:19
*** dtrainor has joined #tripleo00:22
*** jcoufal has quit IRC00:24
*** sai_p has quit IRC00:28
*** tristanC has quit IRC00:33
*** tristanC has joined #tripleo00:33
*** rlandy|afk has quit IRC00:35
*** codrcodz has joined #tripleo00:42
*** codrcodz has quit IRC00:50
*** jtomasek has joined #tripleo01:03
*** jtomasek has quit IRC01:07
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179256001:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287201:10
openstackLaunchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr)01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571801:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)01:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]01:10
*** huynq has joined #tripleo01:14
*** dtrainor has quit IRC01:18
*** mrsoul has quit IRC01:20
*** codrcodz has joined #tripleo01:22
*** thrash is now known as thrash|g0ne01:30
*** huynq has quit IRC01:50
*** huynq has joined #tripleo01:56
*** codrcodz has quit IRC02:01
*** jtomasek has joined #tripleo02:02
*** jtomasek has quit IRC02:07
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179256002:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287202:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571802:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr)02:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)02:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]02:10
*** lblanchard has quit IRC02:40
*** ramishra has joined #tripleo02:45
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179256003:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287203:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571803:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr)03:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)03:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]03:10
*** udesale has joined #tripleo03:14
*** shyamb has joined #tripleo03:45
shyambHi04:01
*** ykarel|away has joined #tripleo04:02
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179256004:10
openstackLaunchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr)04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287204:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571804:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)04:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]04:10
*** shyamb has quit IRC04:22
*** shyamb has joined #tripleo04:22
*** saneax has joined #tripleo04:35
*** Petersingh has joined #tripleo04:42
Tenguhello there05:01
*** Petersingh is now known as Petersingh|afk05:05
*** udesale has quit IRC05:09
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179256005:10
openstackLaunchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr)05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287205:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571805:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)05:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]05:10
*** shyamb has quit IRC05:11
*** shyamb has joined #tripleo05:11
*** udesale has joined #tripleo05:15
*** iranzo has joined #tripleo05:23
*** ykarel|away is now known as ykarel05:23
*** shyamb has quit IRC05:26
*** sai_p has joined #tripleo05:35
*** Petersingh|afk is now known as Petersingh05:43
*** jfrancoa has joined #tripleo05:48
*** quiquell|off is now known as quiquell05:49
*** iranzo has quit IRC05:49
*** udesale has quit IRC05:53
*** shyamb has joined #tripleo05:53
*** mschuppert has joined #tripleo05:55
shyambHi05:56
shyambTengu: Can I use two kvm servers to deploy overcloud on vms?05:57
shyambI meant some overcloud nodes will be on kvm0 and some nodes will be on kvm1, is it possible?05:57
Tengushyamb: as long as you have a working network between the two nodes and a working virtualbmc for the "fake IPMI", I don't see any cons.05:58
*** udesale has joined #tripleo05:58
shyambhmm, ok05:58
shyambTengu: Thanks05:59
shyambI need try it05:59
*** ksambor has joined #tripleo05:59
shyambI doubt about PXE boot only05:59
*** agurenko has joined #tripleo06:04
*** jtomasek has joined #tripleo06:05
*** ratailor has joined #tripleo06:06
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179256006:10
openstackLaunchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr)06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287206:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571806:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)06:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]06:10
*** iurygregory has joined #tripleo06:15
hjensasshyamb: either ensure both kvm hosts are on the same L2 network and DHCP should be fine.06:20
*** dciabrin has joined #tripleo06:21
shyambhjensas: ok, thanks06:21
hjensasshyamb: If you cannot have them on the same L2 network you can deploy a dhcp-relay server on the kvm host not running the undercloud and deploy a routed-spine-and-leaf architecture.06:21
*** holser_ has joined #tripleo06:21
shyambok06:22
shyambwill try first approach06:22
*** florianf|afk is now known as florianf06:26
*** moshele has joined #tripleo06:30
*** yprokule has joined #tripleo06:33
*** cylopez has joined #tripleo06:34
*** cylopez has left #tripleo06:35
mschuppertjaosorior: morning! could you +w https://review.openstack.org/594145 ?06:38
*** rcernin has quit IRC06:38
*** moshele has quit IRC06:38
*** rcernin has joined #tripleo06:38
*** shyamb has quit IRC06:38
*** pcaruana has joined #tripleo06:40
*** shyamb has joined #tripleo06:44
*** Petersingh is now known as Petersingh|afk06:56
*** slaweq has joined #tripleo06:57
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179256007:10
openstackLaunchpad bug 1792560 in tripleo "Upgrades in CI still using Q->master instead of R->master release" [Critical,In progress] - Assigned to Jiří Stránský (jistr)07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287207:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571807:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)07:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]07:10
*** rcernin has quit IRC07:10
*** bogdando has joined #tripleo07:11
*** Petersingh|afk is now known as Petersingh07:20
*** aufi has joined #tripleo07:23
*** skramaja has joined #tripleo07:23
*** ykarel_ has joined #tripleo07:24
*** ykarel has quit IRC07:26
*** shardy has joined #tripleo07:26
*** ykarel__ has joined #tripleo07:28
*** ykarel_ has quit IRC07:30
*** udesale has quit IRC07:30
*** shyamb has quit IRC07:38
*** Petersingh is now known as Petersingh|lunch07:38
*** shyamb has joined #tripleo07:38
*** ykarel_ has joined #tripleo07:41
*** ykarel__ has quit IRC07:44
*** tosky has joined #tripleo07:45
*** ykarel__ has joined #tripleo07:46
*** ykarel__ is now known as ykarel07:46
*** ykarel has quit IRC07:47
*** ykarel_ has quit IRC07:48
*** yprokule has quit IRC07:51
*** yprokule has joined #tripleo07:51
*** udesale has joined #tripleo07:53
*** jpena|off is now known as jpena07:53
*** ykarel has joined #tripleo07:54
*** pcaruana has quit IRC07:55
*** amoralej|off is now known as amoralej07:55
*** pcaruana has joined #tripleo07:57
*** shyamb has quit IRC07:59
*** rdopiera has joined #tripleo08:00
*** chem has joined #tripleo08:02
Tengujaosorior: standalone with podman+selinux: done.08:04
Tengujaosorior: I'm surprised, it was unexpectedly easy to do, compared to the undercloud :D08:04
Tengujaosorior: https://review.openstack.org/#/c/607557/ #enjoy :)08:10
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287208:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571808:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)08:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]08:10
*** akrivoka has joined #tripleo08:10
*** ykarel_ has joined #tripleo08:12
jaosoriorTengu: you're on fire man!08:13
*** derekh has joined #tripleo08:13
Tengujaosorior: well, I'm still surprised. but apparently, it actualy DOES work as it should. I'll do the same for a standalone with ceph.08:14
Tenguguess that one will be nastier.08:14
*** ykarel has quit IRC08:15
Tengujaosorior: that said, some stuff for the nova containers aren't really nice I think - disabled labels + privileged is kind of... unsecure.08:16
Tengujaosorior: not sure if we can do better on that.08:16
Tengumaybe using the "capabilities" instead of privileged?08:16
Tenguenabling labels isn't an option due to the affected host directories (/etc and the like).08:16
jaosoriorTengu: yeah, that would be the next step, replace privileged containers with capabilities profiles08:17
Tengujaosorior: I think docker has the same flags available right?08:18
*** ykarel__ has joined #tripleo08:20
*** paramite|clone has joined #tripleo08:21
*** ykarel_ has quit IRC08:23
*** ykarel_ has joined #tripleo08:23
*** sai_p has quit IRC08:24
d0ugalweshay, sshnaidm, marios|rover: Sorry, I wasn't well yesterday. Let me know if I can help with Mistral issues today!08:24
*** gfidente has joined #tripleo08:25
*** ykarel__ has quit IRC08:26
d0ugalReading the bug now anyway...08:26
gfidentemarios I am not sure how about yojfulness today but this is just the backport to rocky https://review.openstack.org/#/c/607652/08:27
mariosd0ugal: ack thanks seems like its more on the ironic side so far thrash also checked it out08:27
gfidentejistr go go go https://review.openstack.org/#/c/607588/08:28
jistrgfidente: yea thanks :)08:28
*** gkadam has joined #tripleo08:30
*** gkadam has quit IRC08:31
d0ugalmarios: okay, cool. I'll take a quick look anyway and comment if I have anything to add.08:37
*** Petersingh|lunch is now known as Petersingh08:40
mariosthanks d0ugal08:42
sshnaidmd0ugal, thanks, I think toure started the investigation, let's talk to him later when he's online08:43
*** panda|off is now known as panda08:44
bogdandoowalsh: hi, around? PTAL https://review.openstack.org/#/c/60755708:45
*** ykarel__ has joined #tripleo08:45
*** udesale has quit IRC08:46
*** shyamb has joined #tripleo08:46
*** ykarel_ has quit IRC08:48
*** ykarel_ has joined #tripleo08:49
*** udesale has joined #tripleo08:49
jaosoriorTengu: AFAIK, for capabilities it's a bit different. We need to provide a separate capabilities list.08:49
jaosoriorwhich would be a json file08:50
*** ykarel__ has quit IRC08:51
owalshbogdando: hey, /me looks09:01
*** ykarel_ has quit IRC09:07
*** ykarel_ has joined #tripleo09:07
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287209:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571809:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)09:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]09:10
*** ykarel__ has joined #tripleo09:10
*** ykarel_ has quit IRC09:13
Tenguowalsh bogdando jaosorior thanks for your feedback on that change :).09:17
Tenguit's not yet finished, have a couple of issues with the "ceph" in standalone (for now I did a simple, without ceph deploy)09:18
*** ykarel has joined #tripleo09:18
*** ykarel__ has quit IRC09:19
*** salmankhan has joined #tripleo09:24
akrivokahow do I access overcloud nodes after a oooq run? stuff from https://docs.openstack.org/tripleo-quickstart/latest/accessing-overcloud.html doesn't work - I have no overcloud hosts listed in the ssh.config.ansible file and nova list returns empty09:25
*** shyamb has quit IRC09:29
*** shyamb has joined #tripleo09:30
*** ykarel has quit IRC09:32
*** ykarel has joined #tripleo09:32
*** paramite|clone has quit IRC09:35
*** ykarel_ has joined #tripleo09:35
*** ykarel has quit IRC09:38
*** shyamb has quit IRC09:40
*** ykarel_ is now known as ykarel09:43
*** paramite has joined #tripleo09:54
*** dtantsur|afk is now known as dtantsur09:55
jistrchem, matbu, jfrancoa: hmm we're not running all deploy steps in upgrade it seems?09:58
jistrhttps://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps.j2#L63609:58
jistrintention or bug?09:58
jistrlooks like we have two uses of what '_max' in the variable names means09:59
*** moshele has joined #tripleo09:59
jistrdeploy tasks use the _max variables as "last step", while update/upgrade tasks use the _max variables as "number of steps", and given that we start from 0 for the update tasks, this creates some confusion10:00
jistrchem, matbu, jfrancoa: the deploy_steps_max usage in the link is likely incorrect but i'm not sure... will investigate it further and probably redo the update/upgrade steps conventions to match the deploy steps conventions10:01
jistrso that we always iterate to _max in all tasks, and not somewhere to _max and in other places to _max-110:01
chemjistr: I think it's the correct idiom10:01
chemjistr: we have 5 steps max I believe10:01
chemjistr: in puppet tripleo nothing comes at step >= 610:02
jistrchem: ah i thought the range function would generate inclusive range on the top side https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps.j2#L10410:02
jistrwhich confused me10:02
* jistr double checks jinja docs10:02
chemjistr: I remember having the same doubt a while ago but it was cleared somehow10:03
*** paramite has quit IRC10:03
jistrchem, matbu, jfrancoa: indeed range(1,6) generates numbers 1..510:03
*** paramite has joined #tripleo10:03
jistrso all's ok, sorry for the noise :)10:04
jistrthanks chem10:04
*** udesale has quit IRC10:04
chemjistr: pretty sure I have an old abandonned patch where I had excatly the same doubt :)10:05
jistrhaha :)10:05
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287210:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179366510:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571810:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1793665 in tripleo "Jobs fails on validate-tempest: Unexpected response code received" [Critical,Triaged]10:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]10:10
*** udesale has joined #tripleo10:13
ykarelakrivoka, is overcloud installed? i guess u have not passed --tags all during installation from oooq10:15
akrivokaykarel: how can I tell if the overcloud was installed? the oooq run finished successfully10:16
ykarelakrivoka, check logs10:17
akrivokaykarel: this is the command I ran: bash quickstart.sh --clean --teardown all --no-clone --release master $VIRTHOST10:17
ykarelakrivoka, ^^ no --tags all in this10:17
ykarelso it will just install undercloud and will create all required scripts to install overcloud10:17
ykarelakrivoka, so if u login to undercloud, u can see that /home/stack/overcloud-deploy.sh exists there10:18
ykarelu can simply run that to install overcloud10:18
Tengu:) I have the same kind of trick for my tripleo-lab10:18
akrivokaykarel: oh I see10:18
akrivokaykarel: thanks10:18
akrivokawould be nice to have that info in docs, https://docs.openstack.org/tripleo-quickstart/latest/devmode.html currently points to a broken link...10:19
ykarelakrivoka, u can propose a patch to fix that here: https://raw.githubusercontent.com/openstack/tripleo-quickstart/master/doc/source/devmode.rst10:21
ykareljust need to adjust the text there10:22
jaosoriorbandini: so, this works then https://review.openstack.org/#/c/607343/3 ?10:25
bandinijaosorior: so yes mschuppert and I tested that. I'll dig into the CI failures after lunch10:26
*** shyamb has joined #tripleo10:27
*** skramaja_ has joined #tripleo10:35
*** skramaja has quit IRC10:36
*** kopecmartin|ruck is now known as kopecmartin|scho10:38
*** slaweq_ has joined #tripleo10:38
*** shyamb has quit IRC10:38
*** shyamb has joined #tripleo10:38
*** udesale has quit IRC10:39
*** udesale has joined #tripleo10:40
*** slaweq has quit IRC10:40
*** paramite has quit IRC10:55
*** slaweq__ has joined #tripleo10:56
*** slaweq_ has quit IRC10:57
*** pcaruana has quit IRC11:02
*** pcaruana has joined #tripleo11:02
*** jpena is now known as jpena|lunch11:03
*** amoralej is now known as amoralej|lunch11:03
*** janki has joined #tripleo11:06
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287211:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179366511:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571811:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell)11:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,Triaged]11:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]11:10
jankiEmilienM, hey11:20
mandretrown|outtypewww: good morning, when you have time can you uploadthe etcd image from the fedora registry to the tripleorocky namespace on the dockerhub? the same one we uploaded to tripleomaster should do11:21
*** paramite has joined #tripleo11:21
mandretrown|outtypewww: the missing image is the current issue for why our backports fail scenario009 -> http://logs.openstack.org/95/607195/1/check/tripleo-ci-centos-7-scenario009-multinode-oooq/3b9034f/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz11:23
*** aufi has quit IRC11:23
*** Petersingh is now known as Petersingh|afk11:24
*** pvc has joined #tripleo11:25
pvchi guys11:25
pvcanyone help me with overcloud. im using baremetal servers with vlan networking11:25
*** raildo has joined #tripleo11:41
*** iranzo has joined #tripleo11:43
*** shyamb has quit IRC11:44
*** asbishop has joined #tripleo11:45
*** shyam89 has joined #tripleo11:45
*** dprince has joined #tripleo11:45
*** slagle has quit IRC11:51
pvchi mandre can you help me11:53
gfidentelove for a backport https://review.openstack.org/#/c/607840/ ? :D11:59
*** morazi has joined #tripleo11:59
mandrepvc: did you solve the problem with your node not getting an address?12:00
*** lblanchard has joined #tripleo12:02
pvcfor now im using a bond-with-vlans12:04
*** rh-jelabarre has joined #tripleo12:04
pvci will introspect my overcloud wait12:04
*** jpena|lunch is now known as jpena12:04
pvcit can get an ip address mandre but it cannot up the interface. But when i do ifup <interface name> it have an ip address12:04
*** jcoufal has joined #tripleo12:04
mandrepvc: I do not know what your problem is :(12:05
*** med_ has joined #tripleo12:08
weshaymarios, hey.. re: https://bugs.launchpad.net/tripleo/+bug/179366512:10
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,Triaged]12:10
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287212:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179366512:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571812:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]12:10
weshaymarios, the tempest tests in those jobs are generally understood to be the bare min for coverage12:10
*** huynq has quit IRC12:10
weshayunlike something like fs2012:10
weshaymarios, so let's look which tests are failing and run it by ptl's12:11
*** Petersingh|afk is now known as Petersingh12:11
*** thrash|g0ne is now known as thrash12:11
weshaymarios, fyi.. this snip does not offer any insight http://logs.openstack.org/81/602681/3/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/16ad793/job-output.txt.gz#_2018-10-04_06_46_01_15665012:12
weshaybetter to link to the tempest html12:12
honzajtomasek: so many tiny patches!  i love it12:12
*** trown|outtypewww is now known as trown12:13
weshayarxcruz, kopecmartin|scho I'm looking at tempest failures that do not have tempest.html or stackwiz12:13
weshayhttp://logs.openstack.org/81/602681/3/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/16ad793/logs/undercloud/home/zuul/tempest.log.txt.gz12:13
marios|roverweshay: ack will dig some more after this call12:13
arxcruzweshay: https://tree.taiga.io/project/tripleo-ci-board/issue/77?kanban-status=202773412:13
weshayarxcruz++12:14
*** rfolco has joined #tripleo12:14
trownmandre: sure thing I will do that this morning... did you see my patch for the atomic install issue? We have passing CI on the OSA container patch with that!12:15
kopecmartin|schoweshay, great, we should make sure, that it's expected to fail there ..  maybe the failure is ignored because it may fail , i don't know12:15
trownmandre: do you have an etherpad with all the patches on it? I would like to bug shardy to try to get everything merged up to the containerized patch... Then maybe we should try 3.11?12:16
mandretrown: yeah that's awesome! time to merge all-the-things12:17
mandretrown: https://etherpad.openstack.org/p/tripleo-openshift-patches for the list of patches12:17
pvchi mandre how can i successfully clean the overcloud failed  deploy ?12:18
pvcit seems it have caching on their side mandre12:18
*** rlandy has joined #tripleo12:18
honzaarxcruz: what timezone are you in?12:19
arxcruzhonza: berlin12:19
mandrepvc:  "openstack stack delete overcloud --wait" should do it12:19
mandrepvc: this will tear down the provisioned nodes and remove the stack12:20
pvcstack already deleted12:20
pvcbut it seems its already have the prev config12:20
mandrethen you're good to go with a new deployment12:20
pvcbut ipxe not working and one node booting on tftp one booting for httpboot12:21
weshaymarios|rover, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/2ffd88c/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-10-04_02_15_4612:21
honzaarxcruz: i was wondering if you could help me figure out how to iterate on tempest tests for tripleo-ui --- my understanding is that all the tempest plugins are packaged in rdo, then packaged together into openstack-tempest-all, and then installed in the tempest docker image12:22
Tenguo_O apparently gate kills jobs or something? a change is being re-queued for the 3rd time now :(.12:22
marios|roverweshay: ack thanks. so other than that, also some infra issues on ovb jobs filed https://bugs.launchpad.net/tripleo/+bug/1796069 and me kopecmartin|scho  have updated https://review.rdoproject.org/etherpad/p/ruckrover-sprint20 with those fyi12:22
openstackLaunchpad bug 1796069 in tripleo "OVB jobs are failing for "No valid host was found. , Code: 500" during the overcloud deployment" [High,Triaged] - Assigned to Marios Andreou (marios-b)12:22
arxcruzhonza: yes...12:22
honzaarxcruz: this image is then pulled on the undercloud in certain ci jobs, and tests are executed inside12:23
trownshardy: if you have some free time today everything in https://etherpad.openstack.org/p/tripleo-openshift-patches with "READY" just needs 1x +2/+W ... If we get everything "READY" there merged we will have working OSA running from container12:23
weshaymarios|rover, ah k.. wanted to show you one thing12:23
arxcruzhonza: yup12:23
honzaarxcruz: it seems like a really long chain to get a test into ci12:23
*** ssbarnea_ has joined #tripleo12:23
honzaarxcruz: i would like to submit the test and the code that is tested by that test at the same time, and have ci test my code for me12:23
honzaarxcruz: is such a thing possible?12:24
marios|roverweshay: ?12:24
shardytrown: ack will go through the list shortly12:24
jtomasekhonza: glad to hear that:)12:24
weshaymarios|rover, failed gate job http://logs.openstack.org/41/601241/16/gate/tripleo-ci-centos-7-containers-multinode/5a05846/job-output.txt.gz#_2018-10-03_21_37_01_70964612:24
arxcruzhonza: so, you already have the package for this test ?12:24
weshayhttp://logs.openstack.org/90/605090/10/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/6edb958/job-output.txt.gz#_2018-10-03_18_20_58_50087212:24
marios|roverweshay: ah then we have yet another thing12:25
weshaymarios|rover, kopecmartin|scho /me wondering if jobs running on qemu have a greater chance of failing12:25
jaosoriorweshay: isn't that warning unrelated?12:25
*** ssbarnea|bkp2 has quit IRC12:25
weshayjaosorior, it's not failing on that.. however the job times out12:25
weshay :)12:25
honzaarxcruz: we have a project set up for tempest-tripleo-ui but no packaging at the moment (working on it already)12:25
jaosorioralthough it is quite fishy that we can't modprobe the kvm_intel module12:25
*** amoralej|lunch is now known as amoralej12:26
*** aufi has joined #tripleo12:26
arxcruzhonza: and you want to have it running in our ci in some featureset right ?12:26
arxcruzor you have your own job already set ?12:26
weshayjaosorior, rfolco found that not all the hardware is capable12:27
weshayjaosorior, it does inject the module when possible12:27
jaosoriorok12:27
honzaarxcruz: there are some open patches to that effect; one in tripleo-ci to create a new job, one in oooq to create a new featureset12:28
*** ade_lee has quit IRC12:28
rfolcoweshay, I am working on that role (set-libvirt-type) again to set a default (kvm) and then use default for ovb (multinode-baremetal) coz we cannot run that role on ovb (inventory is unknown)12:28
honzaarxcruz: here is the topic https://review.openstack.org/#/q/topic:selenium-ci+(status:open+OR+status:merged)12:29
arxcruzhonza: okay, so i think you have everything in place already :D12:29
rfolcoweshay, context: https://review.openstack.org/#/c/601584/12:29
arxcruzhonza: chandankumar is the one who create the docker image, i'm not familiar to how to insert a new rpm into that image12:29
*** lblanchard has quit IRC12:30
weshayarxcruz, we have this failing the gate atm12:31
weshayessentially upstream is down12:31
weshayarxcruz, can you give me 10-15min on your best guess http://logs.openstack.org/81/602681/3/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/16ad793/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-10-04_06_45_0412:31
weshayjaosorior, marios|rover ^12:31
marios|roverweshay: thanks12:32
weshayarxcruz, I'm looking as well12:32
jaosoriorqX_X12:32
honzaarxcruz: i see how it's done in the docker file in kolla; i'm just wondering if there is a way to inject the tests outside of the whole rpm toolchain, e.g. if i wanted to test a zuul change, like we do in the reproducer12:32
honzaarxcruz: thanks12:32
*** quiquell is now known as quiquell|lunch12:33
weshayImportError: Class ceph cannot be found (['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 32, in import_class\n    return getattr(sys.modules[mod_str], class_str)\n', "AttributeError: 'module' object has12:33
jtomasekhonza: maybe the tests could live in separate repository from the tempest ui package and the package would just be used to run them?12:35
marios|roverweshay: isee what you mean one of them is failing for 2018-10-04 04:32:53 | {0} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops [31.405731s] ... FAILED http://logs.openstack.org/98/604298/20/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/9aad81f//logs/undercloud/home/zuul/tempest.log.txt.gz12:35
weshayhrm.. it's not the ceph error12:36
marios|roverweshay: cos you mentioned networking at some point on the call earlier12:36
*** social has joined #tripleo12:36
marios|roverweshay: my survey of four is 50/50 that network test and then the other 2018-10-04 04:41:18 | {0} tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern [29.796705s] ... FAILED12:37
*** tosky__ has joined #tripleo12:37
arxcruzweshay: seeing a lot of this in nova compute: http://logs.openstack.org/81/602681/3/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/16ad793/logs/subnode-2/var/log/containers/nova/nova-compute.log.txt.gz#_2018-10-04_06_46_11_19712:37
weshaymarios|rover, arxcruz http://logs.openstack.org/81/602681/3/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/16ad793/logs/subnode-2/var/log/extra/errors.txt.gz#_2018-10-04_06_36_10_50412:37
*** tosky is now known as Guest839912:37
*** tosky__ is now known as tosky12:37
weshayya.. +1 arxcruz12:38
marios|roverweshay: interesting. both of the tempest traces have that 2018-10-04 04:50:16 |     Details: {u'message': u'No valid host was found. ', u'code': 500, u'created': u'2018-10-04T04:50:05Z'}12:38
marios|roverso could be very relevant12:38
arxcruzweshay: plus, is trying to create instance on RegionOne, instead of regionOne12:38
arxcruzagain, this come back to haunt us12:38
arxcruzweshay: http://logs.openstack.org/81/602681/3/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/16ad793/logs/subnode-2/var/log/containers/nova/nova-compute.log.txt.gz#_2018-10-04_06_45_08_35412:39
weshayarxcruz, again?12:39
arxcruzweshay: well, for me at least hehehe12:39
arxcruzwhen i was working on rhos... we had problems12:39
*** Guest8399 has quit IRC12:39
arxcruzi don't know why regionOne instead of default RegionOne12:39
weshay2018-10-04 06:47:12.298 8 WARNING nova.scheduler.client.report [req-acf88fb6-a57f-42e6-a5fe-4ed401bf778e - - - - -] The placement API endpoint was not found.: EndpointNotFound: ['internal'] endpoint for placement service in RegionOne region not found12:40
honzajtomasek: i was thinking of adding a mechanism into the tempest image whereby you could request that plugins are refreshed from git/gerrit before running tests12:40
weshayarxcruz, hrm.. why is that just a warning.. if that ends up being fatal?12:40
weshay0-04 06:47:12.291 8 ERROR nova.compute.resource_tracker [req-acf88fb6-a57f-42e6-a5fe-4ed401bf778e - - - - -] Skipping removal of allocations for deleted instances: ['internal'] endpoint for placement service in RegionOne region not found: EndpointNotFound: ['internal'] endpoint for placement service in RegionOne region not found12:40
weshayah. .it's an error12:40
*** agopi has quit IRC12:41
arxcruzweshay: is not a warning12:41
arxcruzyeah, it's an error12:41
weshayarxcruz, is that the root cause?12:41
arxcruzweshay: probably, there's no compute node on RegionOne12:41
*** agopi has joined #tripleo12:41
honzajtomasek: that way the job running for a patch in tripleo-ui could check the commit message for any depends-ons, and optionally check out a patch from tempest-tripleo-ui inside the container12:41
arxcruzweshay: but checking tempest, is looking for regionOne12:41
arxcruzso it might be something in nova side12:41
weshaymarios|rover, ok.. we should update the bug title12:41
weshayarxcruz, is that fixed in tht or nova?12:41
jtomasekhonza: nice!12:41
weshaywe should be experts in this by now12:41
arxcruzweshay: tht12:42
weshayarxcruz, http://codesearch.openstack.org/?q=RegionOne&i=nope&files=&repos=tripleo-heat-templates12:42
weshayhttp://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/docker/services/novajoin.yaml#n5212:43
arxcruzweshay: http://logs.openstack.org/81/602681/3/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/16ad793/logs/subnode-2/var/log/config-data/nova/etc/nova/nova.conf.txt.gz12:43
arxcruzweshay: there are two regions set12:43
honzajtomasek: we'll probably have to go through the whole spec/blueprint dance before starting though12:43
weshayAHHHH12:44
weshayarxcruz, one is RegionOne and the other is regionOne12:44
arxcruzweshay: well, it's in different sections12:44
weshayjaosorior, http://logs.openstack.org/81/602681/3/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/16ad793/logs/subnode-2/var/log/config-data/nova/etc/nova/nova.conf.txt.gz12:44
arxcruzbut it should be regionOne in both12:44
jaosoriorwtf12:45
*** agopi has quit IRC12:45
arxcruzsince we don't have RegionOne12:45
marios|roverweshay: arxcruz added comment #812:45
*** agopi has joined #tripleo12:45
arxcruzperhaps check other jobs that haven't failed if nova.conf has both [r|R]egionOne12:46
jaosoriorweshay: where is RegionOne set?12:46
*** kopecmartin|scho is now known as kopecmartin|ruck12:46
*** agopi has quit IRC12:47
jaosoriorwe do have a parameter in t-h-t called KeystoneRegion, which should be setting that12:47
jaosoriorwe default to 'regionOne'12:47
*** agopi has joined #tripleo12:47
jaosorioractually, zzzeek_ was working on multi-region support for TripleO and did find out we weren't setting the region for some services. zzzeek_ do you have a list handy of what we were missing?12:47
*** slagle has joined #tripleo12:47
arxcruzmarios|rover: on #8 where? O.o12:48
*** agopi has quit IRC12:48
weshayjaosorior, arxcruz afaict.. we're consistent in tht http://codesearch.openstack.org/?q=regionone&i=fosho&files=&repos=tripleo-common,tripleo-heat-templates12:48
*** agopi has joined #tripleo12:48
jaosoriorweshay: it's probably that we just missed setting it in some configuration... so we need to find those.12:49
weshayjaosorior, what other repos?12:49
weshaynova?12:49
jaosoriornot another repo, just missing from t-h-t12:49
marios|roverarxcruz: https://bugs.launchpad.net/tripleo/+bug/1793665 comment 812:50
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,Triaged]12:50
jaosoriorcause OpenStack's global default is Region one AFAIK12:50
marios|roverarxcruz: i started looking initially at what tempest test failing/if it was the same but in tmeantime the nova error became the star added at the first paragraph12:50
arxcruzk12:50
arxcruzjaosorior: and WHY we set to regionOne?12:51
jaosoriorI don't have that answer12:51
* arxcruz put the meme y u set to regionOne12:51
jaosoriorweshay, arxcruz: Do we have a bug for the region?12:52
weshayjaosorior, arxcruz nova uses RegionOne, tht uses regionOne12:52
weshaywe have old ones for sure12:52
weshayjaosorior, arxcruz I never followed this as closely as I should have, but why would we not just match what Nova defaults to?12:52
weshaymwhahaha, ^12:52
* weshay thinks he's going to yell at me for asking12:53
jaosoriorweshay: this setting predates OSP7, who knows :D12:53
arxcruzI had a patch before to change it to RegionOne, but it would break the upgrades job, at least that was the response in that time to not change it to RegionOne12:54
arxcruzbut it's been 2 years or more12:54
jaosoriorweshay, arxcruz: we were using the wrong hiera keys the value changes in puppet-novpa12:54
jaosoriorweshay, arxcruz: https://review.openstack.org/60795212:54
jaosoriorarxcruz: changing it to RegionOne is a no-go.12:54
arxcruzjaosorior: i'm already in peace with that decision :P12:54
jaosoriorthat would surely break people12:54
*** agurenko has quit IRC12:55
arxcruzand that's why it's important to have tempest running in all jobs :)12:55
arxcruzjaosorior: weshay ^12:55
jaosoriorarxcruz: indeed12:55
jaosorioranyway12:55
jaosoriorgot a fix12:55
weshayarxcruz, totally12:55
jaosoriorweshay, arxcruz: https://review.openstack.org/607952 do we have a bug that I can link it to?12:55
*** moshele has quit IRC12:56
jaosoriorcause we'll need it in rocky too12:56
arxcruzjaosorior: i guess https://bugs.launchpad.net/tripleo/+bug/179366512:56
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,Triaged]12:56
weshayjaosorior, Closes-Bug: ?12:56
jaosoriorarxcruz: thanks. Submitted.12:56
marios|roverarxcruz: nice12:57
jaosoriorweshay: that's what I was asking for.12:57
weshay:)12:57
jaosoriordidn't know what bug number to put12:57
marios|roverarxcruz: weshay but, why are we only seeing it now?12:57
jaosoriornow I did.12:57
marios|roverarxcruz: new tempest test?12:57
jaosoriormarios|rover: cause we were setting that value before.12:57
*** agurenko has joined #tripleo12:57
jaosoriorbut it changed in puppet-nova12:57
jaosoriorwe were using os_region_name12:57
jaosoriorand it changed to be region_name12:57
marios|roverjaosorior: ack thanks12:57
*** med_ has quit IRC12:57
jaosoriorthe deprecation was set for a release already (or not sure how many)12:57
jaosoriorand this time they finally deleted os_region_name12:57
jaosoriorthe same will happen for the interface12:57
jaosoriorso I added it to the review12:58
jaosoriorah, fuck, we don't have os_interface in rocky12:58
jaosoriorthat'll have to be a separate patch12:58
*** ansmith has joined #tripleo13:00
ykareljaosorior, looks like nova::placement::interface is not valid attribute, i can't see at13:01
ykarelhttps://github.com/openstack/puppet-nova/blob/master/manifests/placement.pp13:01
jaosoriorykarel: just fixed it.13:02
ykareljaosorior, ack13:02
jaosoriorykarel, weshay, marios|rover, arxcruz: https://review.openstack.org/#/c/607952/ this only changes the region13:02
jaosoriorykarel: this changes the interface https://review.openstack.org/#/c/607953/13:02
weshayarxcruz, any changes required to tempest to follow that?13:02
arxcruzweshay: nope13:02
weshayarxcruz++13:02
jaosoriormarios|rover: can you revisit the patch? I had to update it after you scored it.13:03
*** quiquell|lunch is now known as quiquell13:05
marios|roverjaosorior: ack in call will check bit13:06
marios|roverthanks jaosorior13:07
weshayjaosorior, the next thing to understand is the WHYYY13:08
weshaywhy did that error just raise itself13:08
weshaywhat checks / gates did we miss there13:08
marios|roverweshay: comment on that review from yatin13:08
* weshay looks13:08
marios|roverweshay: https://review.openstack.org/#/c/60506913:08
marios|rovermerged 15 hours ago13:08
weshayah..13:09
weshaymwhahaha, we removed tripleo jobs there intentionally.. so that is ours I guess13:09
weshaytripleo-puppet-ci-centos-7-undercloud-containers13:10
jaosoriorweshay: no worries, we'll test that shit out with standalone next time :D13:10
weshaydid not catch that error13:10
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287213:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179366513:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571813:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)13:10
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles)13:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]13:10
jaosoriorwhat does undercloud-containers do?13:10
jaosoriordoes it run tempest?13:10
weshayjaosorior, http://logs.openstack.org/69/605069/1/check/tripleo-puppet-ci-centos-7-undercloud-containers/2b6b18e/logs/tempest.html.gz13:11
jaosoriordafuuuq13:11
jaosoriorand it didn't crash13:11
jaosoriorweeel I dunno13:11
ykarelseems it's not running server create test13:11
ykareljust api13:11
jaosoriornone of you are based in Brussels, right?13:12
weshaylet's update that.. has plenty of time to run13:12
weshayarxcruz, ^13:12
*** lblanchard has joined #tripleo13:14
arxcruzokay, which test you want to run?13:14
arxcruzdefault scenario?13:14
arxcruzjaosorior: i think, i'm the closest one to Brussels :P13:15
jaosoriorarxcruz: I'll head there today and will be there for the weekend. So if you're around, let me know :D13:15
arxcruzjaosorior: haha, I wish! All those beers...13:16
jaosoriorhaha indeed :D13:17
*** vinaykns has joined #tripleo13:17
jaosoriorso yeah13:20
jaosoriortomorrow I won't be around folks13:20
*** ratailor has quit IRC13:21
*** slagle has quit IRC13:22
*** shyam89 has quit IRC13:23
*** ade_lee has joined #tripleo13:25
*** holser_ has quit IRC13:34
mwhahahaso regionOne vs RegionOne predates me. i blame someone else13:34
*** ykarel_ has joined #tripleo13:36
*** hjensas has quit IRC13:36
*** slaweq__ is now known as slaweq13:36
*** holser_ has joined #tripleo13:36
rnoriegamwhahaha, ping13:37
mwhahaharnoriega: sup?13:37
rnoriegamwhahaha, hey buddy! quick question, have we deprecate the --use-heat when installing the undercloud?13:37
*** mjturek has joined #tripleo13:37
*** ykarel has quit IRC13:38
*** fultonj has quit IRC13:39
*** slagle has joined #tripleo13:41
shardyrnoriega: it's deprecated because the heat/containers install is default now13:41
rnoriegashardy, thanks! so it will fail if we use that argument13:42
mwhahaharnoriega: yea in master that is no longer a thing13:42
rnoriegacool, good to know13:42
rnoriegashardy, mwhahaha, is this up-to-date? https://docs.openstack.org/tripleo-docs/latest/install/installation/installing.html13:43
mwhahaharnoriega: yea except the deprecated note needs to go away13:43
rnoriegamwhahaha, ack! thanks!13:43
*** agopi has quit IRC13:44
redrobotHello #tripleo friends!  I'm trying to set up a dev environment using tripleo quickstart, but I keep getting image errors about docker images not being available.13:45
redrobotSo I'm trying to understand the process of how those images get onto the overcloud nodes.13:45
redrobotCan someone point me in the right direction?13:45
*** pcaruana has quit IRC13:50
*** ykarel_ is now known as ykarel13:50
*** fultonj has joined #tripleo13:51
*** bfournie has joined #tripleo13:56
jaosoriormandre: do you know by any chance? ^^13:56
rookanyone ever ran into this ?13:57
rook(undercloud) [stack@microbrow-19 heat]$ openstack tripleo container image prepare -e /home/stack/containers-prepare-parameter.yaml13:57
rookError while fetching server API version: ('Connection aborted.', error(13, 'Permission denied'))13:57
*** gkadam has joined #tripleo14:00
slagleEdge squad meeting time14:00
slagle#startmeeting TripleO Edge Squad Meeting14:00
openstackMeeting started Thu Oct  4 14:00:32 2018 UTC and is due to finish in 60 minutes.  The chair is slagle. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: TripleO Edge Squad Meeting)"14:00
openstackThe meeting name has been set to 'tripleo_edge_squad_meeting'14:00
slagleping slagle, emilien, csatari, jaosorior, owalsh, fultonj, gfidente, hjensas, jtomasek,thrash, bogdando, dtantsur, rbrady14:00
slagle#info remove or update your nick from the Meeting Template on the etherpad if you want (or don't want) to be ping'd for the start of the meeting14:00
dtantsuro/14:00
slagle#link https://etherpad.openstack.org/p/tripleo-edge-squad-status14:00
gfidenteo/14:00
jaosorioro/14:00
slagleAnyone can use the #link, #action, #help, #idea, #agreed, and #info commands, not just the moderatorǃ14:00
fultonjo/14:01
slagle#topic Agenda14:01
slagle* Review past action items14:01
slagle* One off items14:01
slagle* Goals for the week14:01
*** openstack changes topic to "Agenda (Meeting topic: TripleO Edge Squad Meeting)"14:01
slagle#topic Review past action items14:02
*** openstack changes topic to "Review past action items (Meeting topic: TripleO Edge Squad Meeting)"14:02
*** agopi has joined #tripleo14:02
slagleACTION: all: review https://etherpad.openstack.org/p/tripleo-edge-mvp and discuss (slagle, 14:10:04)14:02
slaglewe said we'd review that etherpad from bogdando14:02
slaglei see comments from a few folks14:03
slaglei reviewed it, especially around Goal 4 since that was the ask14:03
fultonjme too14:03
slaglei found it a little difficult to understnad, but I left some comments there14:03
fultonjbogdando: anything we should talk about now?14:04
slaglebogdando: perhaps you could take a look and we can continue the discussion on the etherpad for now?14:04
cgoncalvesmwhahaha, EmilienM: hi. could you please approve https://review.openstack.org/#/c/587015/ ? thanks!14:04
zzzeek_jaosorior: i know nova is one, i'd have to run a test to find all the rest14:04
fultonj#agreed continue disucssion in etherpad14:05
slaglehehe :)14:05
slagleok, next up14:05
slagleACTION: fultonj to    get feedback from tripleo CI group on    https://blueprints.launchpad.net/tripleo/+spec/split-controlplane-ci    (fultonj,    14:29:39)14:05
bogdandoslagle, fultonj: yes, pleae let's keep it for offline14:06
fultonjthat AI will be ongoing14:07
fultonjplanning to...14:07
fultonj1. prototype14:07
fultonj2. then get feedback on submission which uses standlone like https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/standalone-jobs.yaml#L1414:07
fultonjstill workkig on prototype14:07
slagleok sounds good14:08
slaglehow is that going?14:08
slagleor is that the next action item we were going to discuss...14:08
fultonj#info - following https://etherpad.openstack.org/p/tripleo-compute-standalone14:08
fultonj#info - few days ago ran into http://paste.openstack.org/show/73111614:08
fultonj#info - Why is RoleNetIpMap missing new edge node 192.168.24.3? http://paste.openstack.org/show/73112014:08
fultonj#info - Stack knows about new node http://paste.openstack.org/show/73112114:08
fultonj#info - Reproducing in my env today and might ping for help if still stuck14:08
fultonj#info - https://gitlab.com/fultonj/tripleo-standalone-edge14:08
fultonjslagle: that's what i'm currently stuck on perhaps after meeting and after i reproduce i could get help dubbing?14:09
fultonjdebugging*14:09
* dtantsur thinks fultonj just invented twitter over irc14:09
slaglefultonj: sure, it probably didn't find an IP for the compute node14:10
*** hamzy has quit IRC14:10
slaglemaybe a problem with DeployedServerPortMap or HostnameMap14:10
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287214:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179366514:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571814:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)14:10
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles)14:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]14:10
slagle#action fultonj and slagle to continue debugging standalone edge deployment14:10
fultonjthanks slagle14:11
owalsho/14:11
*** yprokule has quit IRC14:11
slagleACTION: fultonj try    standalone for    https://blueprints.launchpad.net/tripleo/+spec/split-controlplane-ci    based on    https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/standalone-jobs.yaml#L14    (fultonj,    14:38:07)14:11
slaglealright, we just discussed that14:12
slagleso moving on14:12
slagle#topic One Off14:12
*** openstack changes topic to "One Off (Meeting topic: TripleO Edge Squad Meeting)"14:12
fultonj#info - abishop not here but made progress on active/active cinder in standalone14:12
fultonjand cinder AZs14:12
bogdando#action https://etherpad.openstack.org/p/tripleo-edge-mvp is needs clarification for the special snowflakes, what do we understand as that when comparing it to the federated openstack services (perhaps)14:13
bogdandodtantsur: ^^14:13
bogdandojust captured the irc logs, not really invented those snowflakes :)14:13
dtantsurnot sure I get it, what's the question for me?14:13
slaglebogdando: the only context i was able to capture from the etherpad was that it was implied that "using ansible" = "special snowflake"14:14
bogdandodtantsur: yes please as I think when we were talking about architecture differences for undercloud vs overcloud, we used that term14:14
slagleso i don't understand what is meant by special snowflake14:14
dtantsurlet's probably try use less metaphors and more specific wording?14:15
fultonjdtantsur: +114:15
bogdandoagreeing on terms is already a goal 1 :D14:15
fultonjw.r.t terms i think it helps to be consistent with edge working group14:16
*** jaosorior has quit IRC14:16
slaglefultonj: yes, so one of the things I was going to do try and do over the next week is gather all the links/references that we've been compiling into a single spot14:16
*** Petersingh has quit IRC14:16
slagleperhaps the sqaud etherpad for now14:16
*** Petersingh has joined #tripleo14:17
*** Petersingh is now known as Petersingh|gone14:17
dtantsurI think what is meant under "special snowflake" here is actually a concern about growing the amount of logic specific to tripleo14:17
*** Petersingh|gone has quit IRC14:17
slagleb/c there was also some confusion around "Far Edge" in the etherpad. it's a thing that's been defined outside of TripleO, so we need to understand what is meant by it14:17
dtantsuras opposed to tripleo relying more on other upstream openstack projects14:17
* dtantsur is the primary hater of the term "Far Edge"14:17
fultonjhttps://www.dropbox.com/s/255x1cao14taer3/MVP-Architecture_edge-computing_PTG.pptx?dl=0#14:17
dtantsurhow do I block IRC messages containing ^^^14:18
dtantsur?14:18
slagledtantsur: understood :) but you'd have to convince the Edge WG to not use it14:18
fultonj^ i don't love the term but i'll use it because others do outside of tripleo in the dge working group14:18
bogdandoslagle, dtantsur: updated special snowflakes14:18
bogdandohttps://etherpad.openstack.org/p/tripleo-edge-mvp14:18
dtantsurslagle: maybe I'm just the grumpy old man here :)14:18
slagledtantsur: nah :). i hear where you're coming from14:19
dtantsurI suspect the "Edge" terminology makes good marketing, so we may not be able to change it globally14:19
slagledtantsur: honestly it was driven by groups who are already actually deploying this stuff. and they found it useful to have these classifications14:19
*** iranzo has quit IRC14:19
bogdandoslagle: did I wrote that right?14:20
dtantsurthen I guess we should just start our documentation with a glossary :)14:20
dtantsurwhich is a good idea anyway14:20
bogdandodtantsur: ^^ please take a look that etherpad for 11th time :)14:20
bogdandoline 24-2714:20
dtantsurbogdando: can we s/special snowflake/something non-metaphoric/g please?14:20
bogdandogo for it!14:20
dtantsurI have a gut feeling it's not used exactly in this sense, but as a non-native speaking I cannot grasp it in full14:21
slagledtantsur: as for tripleo growing more logic that is not openstack, not sure i totally agree there. tripleo is part of openstack, as are all the other deployment projects14:21
bogdandojust please make sure it not gets too much of tripleo specific! :)14:21
dtantsurslagle: right, it's not necessary a bad thing14:21
dtantsurbut it's maintenance cost, and if we can avoid it - let's avoid it14:21
bogdandoas every vendor these days seems like has a bunch of snowflakes for Edge thene14:21
bogdandotheme*14:22
slagledtantsur: openstack doesn't have a tenant facing API service for software config/deployment that is sufficient enough for openstack or edge deployment14:22
slagleHeat is about it14:22
dtantsure.g. in metalsmith case, the metalsmith library is under ironic governance, so we share its maintenance14:22
dtantsurwe could write this code in tripleo-common, then we would have to maintain it14:22
slagleand i don't plan on using Heat to deploy thousands of edge nodes :)14:22
dtantsurlol, yeah14:22
*** iranzo has joined #tripleo14:23
* dtantsur ponders an openstack software config service14:23
slaglei do think it's important that we drive things with existing or new consistent interfaces. for now, that means t-h-t14:23
slagleso i look at ansible/config-download as a potential way to scale that to the edge14:24
dtantsuranyway, for me personally the most interesting things is what (if anything) happens with bm provisioning and management14:25
*** agurenko_ has joined #tripleo14:26
*** agurenko has quit IRC14:26
slagle#action slagle to work on documenting some terms and gathering links/references14:26
owalshbogdando: for distributed undercloud, what exactly is the goal? is it federation to make is possible (fast) to deploy to edge from centre, a local undercloud at each edge to tolerate network partitions, both, neither?14:26
dtantsurmy concerns were around replacing ironic as an operator facing API (or completely) with something specific to tripleo14:27
dtantsurthis is *probably* where the snowflakes speech came from14:27
dtantsurowalsh: that's the question, and I don't think we have solved it yet14:27
*** tosky__ has joined #tripleo14:27
slagledtantsur: for bm? i don't think there's any reason to do that. i'm not aware of anything going in that direction14:27
*** tosky has quit IRC14:28
*** tosky__ is now known as tosky14:28
owalshdtantsur: ack14:28
*** hjensas has joined #tripleo14:28
owalshdtantsur: so both :-)14:28
*** aufi_ has joined #tripleo14:28
dtantsurslagle: the question is how deeply we change the bm management workflow on the central undercloud (the Edge layer if I'm not mixing things up again)14:29
*** aufi has quit IRC14:29
*** bogdando has quit IRC14:29
dtantsur(it may not feel the right question for right now, but the resulting architecture of the undercloud may depend on it)14:29
*** saneax has quit IRC14:30
slaglewell maybe the new workflow will be used for both14:32
slagleor should be14:32
*** ykarel is now known as ykarel|afk14:32
owalsh+114:32
*** bogdando has joined #tripleo14:32
slagleanything else before we wrap up?14:33
slagle#endmeeting14:33
*** openstack changes topic to "Welcome to Rocky | CI Status: YELLOWish? (seems to be stable, please be careful) - ovb is sad still | https://docs.openstack.org/tripleo-docs/latest/"14:34
slaglethanks!14:34
openstackMeeting ended Thu Oct  4 14:33:59 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:34
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tripleo_edge_squad_meeting/2018/tripleo_edge_squad_meeting.2018-10-04-14.00.html14:34
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tripleo_edge_squad_meeting/2018/tripleo_edge_squad_meeting.2018-10-04-14.00.txt14:34
openstackLog:            http://eavesdrop.openstack.org/meetings/tripleo_edge_squad_meeting/2018/tripleo_edge_squad_meeting.2018-10-04-14.00.log.html14:34
dtantsurslagle: you mean, hiding all current ironic actions inside a mistral workflow?14:34
*** Vorrtex has joined #tripleo14:34
dtantsurthis is probably what bogdando and I are concerned about14:34
*** mwhahaha changes topic to "Welcome to Rocky | CI Status: RED - master blocked DO NOT APPROVE/RECHECK - ovb is sad still | https://docs.openstack.org/tripleo-docs/latest/"14:35
slagledtantsur: not at all14:36
*** skramaja_ has quit IRC14:36
*** mwhahaha changes topic to "Welcome to Stein | CI Status: RED - master blocked DO NOT APPROVE/RECHECK - ovb is sad still | https://docs.openstack.org/tripleo-docs/latest/"14:36
slagledtantsur: the interface can stand on its own, but we may need automation around that14:37
slaglesimilar to what we've done for config-download imo14:37
slaglewe have commands to generate inventory, export config, apply config14:37
dtantsurI'm quite looking forward to some whiteboarding in Berlin, if you'll be around14:37
slaglebut it's also all automated via a mistral workflow14:38
owalshare we considering workflows that orchestrate remote underclouds?14:38
slaglebecause of the need for a HTTP based api14:38
dtantsurright14:38
dtantsurowalsh: this is how I understand it14:38
toureslagle so you are looking for a workflow which would act as a helper function in a sense to provide all configurable inputs, if I understand you correctly ?14:39
owalshdtantsur: ack, so I was thinking that the lesson from cells in nova is to have one code paths to ensure it's maintained/tested14:39
dtantsurowalsh: this is a part of my concern here. maybe if we do it, we need to use it even for non-edge case..14:39
dtantsurbut this may mean more work to replicate certain ironic features14:39
dtantsurdunno, I still have hard time fitting the whole picture in my head14:40
owalshdtantsur: yea, but it's definitely a better approach long term I think14:40
dtantsurowalsh: another option is to have independent ironic's at each Far Edge, and just tell operators to log in there14:41
dtantsurwithout any central management from the Edge (?) location14:41
owalshdtantsur: but maybe we need both, so that an edge can be managed in isolation... why I was wondering if the requirements are clear for distributed undercloud14:42
bogdandodtantsur: I will highly likely be there14:42
*** shyamb has joined #tripleo14:43
dtantsurowalsh: sure, the ironic API at the Far Edge won't go anywhere. the question is about whether and how we expose bm management facilities from the central location14:43
dtantsuractually, we have 3 options: 1. make ironic do aggregation for us (I'm looking into it, but may take time / be rejected), 2. provide tripleo workflows to do it, 3. do nothing.14:43
slagledtantsur: what about the distributed conductor model?14:44
bogdando> why I was wondering if the requirements are clear for distributed undercloud14:44
bogdandothis ^^14:44
dtantsurslagle: you mean, my initial idea? it required stretching rabbit and db between Edge and Far Edge. otherwise it's pretty neat.14:44
owalshdtantsur: so really you need the equivalent of nova cell_v2 for ironic....14:45
dtantsurideally, we need cells for the whole openstack :)14:45
*** hamzy has joined #tripleo14:45
dtantsurironic may be easier, it does not need any centrally stored state (IMO)14:47
owalshdtantsur: TBH though, I think Far Edge == a cell tower, and Edge == a (relatively) local hub, so latency shouldn't be very high14:47
*** kopecmartin|ruck is now known as kopecmartin|scho14:47
owalshit that specific use case at least14:47
slagledtantsur: rabbitmq to the far edge perhaps for the deployed compute nodes. i don't think the db would need to be stretched to the far edge14:47
slagleand the uc's rabbitmq only to the edge14:47
dtantsurslagle: if we use cells, we don't need rabbitmq to be stretched, do we?14:48
*** med_ has joined #tripleo14:48
*** itlinux has quit IRC14:48
slagleperhaps not. i'm mainly thinking uc here.14:48
bogdandoowalsh: I think the latency is not an only thing to worry about, but isolation14:48
bogdandoand how long it takes being allowed operating isolated14:49
bogdandoand how do we want to sync states later14:49
dtantsurslagle: okay, so we have an option to place the undercloud on Edge, while only having ironic-conductor (and whatever is needed for sw deployment) on Far Edge14:50
owalshbogdando: do we need to replicate state?14:50
dtantsurslagle: it's line 112 in https://etherpad.openstack.org/p/tripleo-edge-squad-status14:50
*** jchhatbar has joined #tripleo14:50
bogdandoowalsh: that's also the question14:51
bogdandoI'd say at least for monitoring status14:51
dtantsurbogdando, owalsh, the problem is database.. I'm not sure how it will behave in case of split-brain14:51
bogdandoand there is also images and identity things14:51
bogdandoglance/ironic images I mean14:51
owalshdtantsur: but isn't Far Edge just the computes?14:51
slagledtantsur: yea i still don't see why conductor would be needed at the far edge14:51
bogdandodtantsur: poorly14:51
dtantsurowalsh: I'm talking about UC right now14:51
slaglecan't everything be done over http?14:52
bogdandobut may be ok if read only mode for db14:52
owalshdtantsur: so we would have a conductor at each FarEdge site?14:52
dtantsurslagle: good that you ask, let's open a can of worms :)14:52
dtantsurowalsh: it's just one of the options14:52
owalshdtantsur: API nodes are at the Edge site, and I'd expect it to be close enough to the Far Edge to not worry about it14:52
dtantsurslagle: we need something to bootstrap iPXE OR switch to virtual media14:53
dtantsuralso solve DHCP, but it's also possible (we're working on DHCP via virtual media, for example)14:53
*** janki has quit IRC14:53
dtantsurcurrently DHCP, TFTP and iSCSI are problems from ironic perspective14:54
nhichersshnaidm: any news regarding mistral timeout ?14:54
*** med_ has quit IRC14:54
hjensaswhy does it have to be over http? As long as it is over IP it's fine?14:54
slagleseems like that could be done with something smaller than conductor, that wouldn't require stretching uc rabbitmq14:54
*** thrash is now known as thrash|biab14:54
slagleso new device boots up at far edge, gets dhcp/ipxe bootstrap from "something", phones home to conductor. conductor does remote deployment14:55
owalshdtantsur: are we assuming the Far Edge nodes are not on the same network segment for DHCP etc...?14:55
slaglethe direct deployment or something? i thought ironic had a thing14:55
slaglewhere it would tell IPA how to do the deployment, etc. entirely via http14:56
dtantsurslagle, owalsh, line 97 https://etherpad.openstack.org/p/tripleo-edge-squad-status14:56
hjensasI think "something" there is either virtual-media or dhcp-relay? TFTP servers can be at Edge as long as the provisioning network at far-edge knows how to route?14:56
dtantsurhjensas: the same link above14:58
owalshdtantsur: ack, but is "Edge" really a central location?14:58
bogdandoowalsh, dtantsur: I thought a tipical for Far Edge is connections to Edge via WAN (no L2)14:59
*** toure is now known as toure|biab14:59
dtantsurowalsh: from our perspective - yes14:59
bogdandotypical*14:59
slagledtantsur: somethhing like that, yea. with many conductors14:59
owalshbogdando: that's what I'm asking, do we have L2 or not?14:59
slagleand distributed glance storage14:59
dtantsurowalsh, bogdando, my notes on the etherpad assume no L215:00
bogdandogood question, would be nice to assume we  do not15:00
dtantsurwith L2 things are essentially the same as now15:00
* owalsh gotta run to another meeting, dtantsur thanks for the brain dump, I'll take a look later15:01
slagletoure|biab: hey, sorry, i didn't understand your earlier question15:07
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287215:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179366515:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571815:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)15:10
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles)15:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]15:10
*** holser_ has quit IRC15:11
*** holser_ has joined #tripleo15:11
*** quiquell is now known as quiquell|off15:18
sshnaidmnhicher, hi, I think toure|biab was on that yesterday, maybe he has updates today, let's ask him when he's onlin15:19
*** toure|biab is now known as toure15:19
sshnaidmnhicher, the problem as I understood was that heat engine started later than needed15:20
sshnaidmlet toure fix me if I'm wrong ^15:20
sshnaidmnhicher, and this is caused by slow storage...15:20
*** sshnaidm is now known as sshnaidm|afk15:20
touresshnaidm nhicher that was the mistral engine logs15:20
touresshnaidm nhicher so the issue is stemming from slow storage, when heat looks to retrieve templates from swift there are timeouts hit15:21
toureit looks as though heat retries and fails15:22
*** shyamb has quit IRC15:23
*** iranzo has quit IRC15:25
*** slagle has quit IRC15:25
*** moguimar has quit IRC15:26
nhichertoure: I can reproduce for each run, but got +1 2 or 3 times, check https://review.rdoproject.org/r/#/c/1656615:26
nhichertoure: do you know how I can fix that issue ?15:27
tourethis job looks as though it completed15:28
nhichertoure: check 'Toggle CI', a lot of failing jobs15:30
tourenhicher am I missing something15:30
toureah15:30
rookmandre: hey - got a second?15:30
rookmandre: I tried restarting the mistral executor contianer15:31
rookbut still get the Permission Deined15:31
*** agopi is now known as agopi|long_lunch15:31
*** dprince has quit IRC15:33
*** hjensas has quit IRC15:35
sshnaidm|afktoure, maybe it's possible to increase the timeout anyhow? At least just for test15:36
*** sshnaidm|afk is now known as sshnaidm15:36
*** udesale has quit IRC15:36
mandrerook: hmm that sucks... are you trying to pull images from a public registry?15:39
*** thrash|biab is now known as thrash15:39
mandrerook: if not, maybe you need to 'docker login' prior to running the prepare step?15:39
toureso far I /win 2115:40
toureoops15:41
rookmandre: local docker15:42
rookhm15:42
rookdocker log?15:42
rooklogin*15:42
*** hamzy_ has joined #tripleo15:42
*** bogdando has quit IRC15:43
mandrerook: the image prepare step will pull container images from a remote registry and push them locally on the undercloud -- does the remote registry require auth?15:43
*** hamzy has quit IRC15:44
mandrerook: you can try to docker pull one of the image from the remote registry, see if it complains15:44
*** agurenko_ has quit IRC15:46
*** hjensas has joined #tripleo15:46
*** jchhatbar has quit IRC15:46
*** dprince has joined #tripleo15:49
*** ykarel|afk is now known as ykarel15:52
*** jfrancoa has quit IRC15:55
*** itlinux has joined #tripleo15:57
*** jaganathan has joined #tripleo15:58
weshaymarios|rover, kopecmartin|scho packstack also broke https://review.openstack.org/#/c/607983/16:00
marios|roverweshay: ack /me about to call it for today will follow up in the morning have a good one16:04
marios|roverthanks weshay16:04
*** trown is now known as trown|lunch16:04
*** jaganathan has quit IRC16:05
bnemecrook: https://review.openstack.org/#/c/531913/2/doc/source/install/containers_deployment/overcloud.rst16:07
bnemecI don't know what happened to that note. I don't see it on master anymore. :-/16:07
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287216:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179366516:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571816:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles)16:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]16:10
*** hjensas has quit IRC16:10
*** dtantsur is now known as dtantsur|afk16:10
*** ykarel_ has joined #tripleo16:18
itlinuxhi to all.. it's almost friday :)16:18
*** sai_p has joined #tripleo16:19
*** ykarel_ is now known as ykarel|away16:20
*** ykarel has quit IRC16:20
*** slagle has joined #tripleo16:20
sai_pHi, in tripleo overcloud deployment, I am trying to make sure puppet configuration for service A is done before it starts the puppet configuration for service B. Can anyone let me know how it can be achieved.16:22
*** hjensas has joined #tripleo16:27
mwhahahasai_p: i guess it depends, but we do have steps in the deployment process and the containers process as well16:27
mwhahahasai_p: if you really want to ensure the configs are done at different times then they should probably be in different steps (like 4/5)16:27
*** aufi_ has quit IRC16:40
Tenguhmm, in fact the bot listing incoming changes is dead right?16:41
mwhahahait's in other channels16:42
mwhahahawill have to ask16:42
Tenguok - not that urgent compared to the other things ;)16:42
*** holser_ has quit IRC16:45
*** raterch has joined #tripleo16:49
Tenguok folks, see you tomorrow. Good luck with that nasty boy of a CI16:49
raterchI'm having an issue with deploy writing a containers-default-parameters.yaml to swift that doesn't match the one in my plan-environment.yaml. Any ideas?16:50
*** moshele has joined #tripleo16:53
*** ykarel|away has quit IRC16:58
*** salmankhan has quit IRC17:04
*** shardy has quit IRC17:04
*** derekh has quit IRC17:04
*** trown|lunch is now known as trown17:04
*** paramite has quit IRC17:04
*** ksambor has quit IRC17:05
*** morazi has quit IRC17:06
*** morazi has joined #tripleo17:07
*** gkadam has quit IRC17:08
*** ramishra has quit IRC17:09
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287217:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179366517:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571817:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles)17:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]17:10
EmilienMweshay, mwhahaha : I hope we can land this one this week: https://review.openstack.org/#/c/600517/17:12
EmilienMit would be a nice milestone17:12
weshayEmilienM, https://review.openstack.org/#/c/607952/ and https://review.openstack.org/#/c/607904/17:12
weshayjust need to land for sanity17:13
EmilienMweshay: ok? nothing to do with my podman work, right?17:13
weshaybfournie, ping re: https://review.openstack.org/#/c/584452/17:13
weshayEmilienM, just clearing up results17:13
EmilienMweshay: click on my link please17:14
EmilienMweshay: it's containerized undercloud job only17:14
EmilienMso nothing to do with other jobs17:14
weshayit's master17:14
weshayso it does17:14
bfournieweshay: hey17:14
weshaybfournie, hey man.. so wondering if it's realistic at all to have ironic setup 3rd party jobs w/ tripleo ovb17:14
weshayI see fujitsu, ibm, cisco17:15
weshayit would be nice to have ovb in there as third party right?17:15
weshayfujitsu's tests are taking 2+ hours17:16
bfournieweshay: I'm deferring to dtantsur|afk re: update to trello card17:17
*** pcaruana has joined #tripleo17:17
weshayah.. /me reads17:18
weshaybtw https://logs.rdoproject.org/52/584452/7/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/7bf44d0/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-10-04_16_24_5017:18
bfournieweshay: hmm, different error17:19
bfournieweshay: no valid host is usually infrastructure17:19
weshayya.. but related to ironic I think17:19
weshayor that17:19
bfournieweshay: will look at some more logs17:19
*** jpena is now known as jpena|off17:21
weshaybfournie, who does CI in your DFG?17:23
trownmandre: I pushed the etcd container to tripleorocky17:23
bfournieweshay: mainly derek17:24
hjensasweshay: So placement return no candidates - https://logs.rdoproject.org/52/584452/7/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/7bf44d0/logs/undercloud/var/log/containers/nova/nova-scheduler.log.txt.gz#_2018-10-04_16_23_01_61917:24
weshaynice choice17:24
bfournieyeah17:24
bfourniehjensas: ah17:25
weshayhjensas, thanks17:25
hjensasweshay: earlier on there is some Warnings about "endpoint for placement service in RegionOne region not found" as well.17:26
weshayhjensas, ya.. we have a fix for that.. puppet broke us last night17:26
*** gfidente is now known as gfidente|afk17:28
weshayhjensas, bfournie Ib7717911aba3267f855ac6682b0144bfe92034fb17:28
weshaysorry https://review.openstack.org/#/c/607952/17:29
hjensasweshay: ok, let's hope for a short re-check party on that one then.17:30
weshayadded as a dep17:30
*** akrivoka has quit IRC17:31
bfourniecool17:31
bfournierelocating, bbiab17:32
*** bfournie has quit IRC17:32
*** amoralej is now known as amoralej|off17:37
*** vinaykns has quit IRC17:38
*** morazi_ has joined #tripleo17:48
*** morazi has quit IRC17:48
fultonjslagle: i've reproduced http://paste.openstack.org/show/731116 do you have any time today to help me debug?17:49
slaglefultonj: sure17:50
slaglefultonj: care to tmate?17:51
fultonjyep17:51
weshayhjensas, can you please vote on https://review.openstack.org/#/c/607904/ ?17:56
hjensasweshay: done.17:57
weshaythanks17:58
*** ykarel|away has joined #tripleo17:59
rookbnemec: interesting, that was missing18:00
rookbnemec: thanks18:00
*** pcaruana has quit IRC18:01
*** ykarel_ has joined #tripleo18:04
*** holser_ has joined #tripleo18:05
*** holser_ has quit IRC18:05
*** dciabrin has quit IRC18:05
weshaymwhahaha, gate18:05
weshaymwhahaha, need to remove any ci changes18:05
weshayhttps://review.openstack.org/60692518:05
*** holser_ has joined #tripleo18:06
*** ykarel|away has quit IRC18:07
*** rbrady is now known as rbrady-afk18:09
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287218:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179366518:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571818:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles)18:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]18:10
mwhahahaweshay: yea i missed that on i guess18:22
weshaymwhahaha, trying https://review.openstack.org/#/c/608051/18:27
*** lblanchard has quit IRC18:30
*** moshele has quit IRC18:36
*** raterch has quit IRC18:39
*** lblanchard has joined #tripleo18:40
*** bfournie has joined #tripleo18:50
rookbnemec: yo - ovb question - if pxe is failing, what is your protocol to look into. mtu looks right. I applied the patches. any thing else?18:51
*** agopi|long_lunch is now known as agopi18:54
*** jcoufal has quit IRC18:57
mwhahaharook: check the console of the nodes to see what its doing?18:59
rookmwhahaha going to switch to the noop driver.19:00
rookgive that a shot19:00
touresshnaidm nhicher so I poured through all the logs on that gate, just following the trail and it looks like my initial suspicion is correct, storage is killing this job, the proof was confirmed with errors from nova, it is not able to create any resources on disk19:00
weshaytoure++19:01
weshaythank you19:01
touresure thing19:01
nhichertoure: thanks, so no way to fix, we only need better storage ?19:02
toureyeah19:02
toureunavailable resources is a tough one19:03
touresorry for the bad news, wish there was a work around19:03
weshaynhicher, having fun yet?19:04
nhicherweshay: =), thanks for your help toure19:06
touresure thing19:07
toureany time19:07
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287219:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179366519:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571819:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles)19:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]19:10
*** diw has joined #tripleo19:10
*** ykarel__ has joined #tripleo19:13
diwHi, I was wondering if someone could help me get staging-ovirt support in my undercloud.  I'm on CentOS7 and I've copied /usr/share/python-tripleoclient/undercloud.conf.sample to ~/underclloud.conf with the only change adding "enabled_hardware_types = ipmi,redfish,ilo,idrac,staging-ovirt" but when my undercloud finishes installing running openstack baremetal driver list returns an empty list.19:14
*** ykarel__ has quit IRC19:14
*** ykarel__ has joined #tripleo19:14
*** ykarel_ has quit IRC19:15
*** ykarel_ has joined #tripleo19:17
diwAs far as I can see ironic-api is running correctly.19:17
*** mjturek has quit IRC19:17
*** ykarel__ has quit IRC19:20
*** ykarel__ has joined #tripleo19:21
*** ykarel_ has quit IRC19:23
*** ykarel__ has quit IRC19:26
diwI've just found an error in the conductor log - DriverNotFoundInEntrypoint: Could not find the following items in the 'ironic.hardware.types' entrypoint: staging-ovirt.  Did I miss a step to ensure the staging drivers get added to the container?19:30
*** gfidente|afk has quit IRC19:43
*** slagle has quit IRC19:43
*** jcoufal has joined #tripleo19:48
*** mjturek has joined #tripleo19:49
*** mjturek has quit IRC20:00
*** jcoufal_ has joined #tripleo20:03
*** raterch has joined #tripleo20:04
raterchis there anyone in here who works on the tripleo mistral plans?20:04
*** jcoufal has quit IRC20:05
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287220:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179366520:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571820:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)20:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1793665 in tripleo "multinode-oooq-container scenarios and Fs016/17/20 periodic jobs fails on validate-tempest: Unexpected response code received" [Critical,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles)20:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]20:10
*** raterch has quit IRC20:10
*** mjturek has joined #tripleo20:11
*** mjturek has quit IRC20:17
*** mjturek has joined #tripleo20:19
*** slagle has joined #tripleo20:19
*** mjturek has quit IRC20:23
*** med_ has joined #tripleo20:26
EmilienMfultonj: https://review.openstack.org/#/c/604357/3/environments/undercloud.yaml20:34
EmilienMit broke podman integration FYI20:34
EmilienMbecause when ceph-ansible isn't installed, /usr/share/ceph-ansible doesn't exist20:34
EmilienMand in contrario to Docker, podman doesn't create missing dirs (by design/choice)20:35
fultonjEmilienM: i see, what do you think we should do?20:35
EmilienMfultonj: 3 options20:35
EmilienM1) add a dep to tripleoclient to deploy ceph-ansible20:36
EmilienM2) add a host prep task in mistral-executor serviec to install ceph-ansible20:36
EmilienM3) add a host prep task to create the directory in mistral-executor service20:36
EmilienMbut this :20:36
EmilienMhttps://review.openstack.org/#/c/604357/3/environments/undercloud.yaml20:36
EmilienMthis is ugly20:36
EmilienMyou could have created enable_ceph in undercloud.conf20:37
*** morazi_ has quit IRC20:37
EmilienMso what option should we take?20:37
EmilienMI thought we could have podman in gate this week20:37
EmilienMbut you ruined my dream :(20:38
weshaywooot.. EmilienM mwhahaha https://review.openstack.org/#/c/607952/ merged20:38
EmilienMweshay: good20:38
mwhahahak let me restore all my stuff20:38
EmilienMfultonj: so what's the plan?20:38
weshayEmilienM, mwhahaha should we +2 EmilienM's patch first20:38
EmilienMweshay: no it'll fail in gate20:38
EmilienMweshay: ceph guys broke us20:38
mwhahahawhich patch20:39
EmilienMhttps://review.openstack.org/#/c/60435720:39
EmilienMmwhahaha: my podman stuff20:39
*** holser_ has quit IRC20:39
mwhahahaoh20:39
*** slagle has quit IRC20:39
mwhahahayea i don't want that stuff :D20:39
fultonji like #1 or #220:39
weshayhttps://review.openstack.org/#/c/600517/20:39
EmilienMfultonj: it makes deps bigger...20:39
*** jtomasek has quit IRC20:39
weshaypodman-0.9.3.1-3.gita3c4ce6.el7.x86_6420:40
fultonj#3 seems like a quicker fix20:40
fultonj#3 to stop bleeding, consider #120:40
fultonjEmilienM: couldn't we do #3 to keep the dream alive :)20:41
*** ansmith has quit IRC20:41
EmilienMfultonj: sure and then we have customers calling because they have an empty ceph-ansible directory :D20:42
EmilienMand missed to install that rpm20:42
EmilienMnah i'll add it as a dependency20:42
EmilienMit's all good20:42
*** tosky has quit IRC20:42
EmilienMone more :D20:42
mwhahahahow much MB is added to the install because of ceph-ansible?20:42
fultonjEmilienM: thanks20:42
mwhahahais it pretty small or does it pull in the world20:43
EmilienMmwhahaha: let me see20:43
*** tosky has joined #tripleo20:43
fultonj2M20:43
EmilienMone sec20:43
*** med_ has quit IRC20:43
*** rdopiera has quit IRC20:43
*** hamzy_ has quit IRC20:44
EmilienMNo package ceph-ansible available20:45
EmilienMby default20:45
EmilienMI bet we need a special repo?20:45
fultonjcentos storage sig20:45
fultonje.g. http://mirror.centos.org/centos/7/storage/x86_64/ceph-luminous/20:45
EmilienMso we need to fix RDO repo first20:46
EmilienMto include that repo by default20:46
EmilienMright now I'm a dumb user, I deploy repos from doc and I can't install ceph-ansible20:46
EmilienMif I add ceph-ansible as a dep of tripleoclient20:46
EmilienMwe break the world20:47
EmilienMbecause http://mirror.centos.org/centos/7/storage/x86_64/ceph-luminous/ repo isn't installed by dfault20:47
mwhahahalet's just revert it and add it correctly20:47
EmilienMyes20:47
fultonjsudo -E tripleo-repos current-tripleo-dev ceph20:47
bandiniany takers for an easy pike backport ? https://review.openstack.org/#/c/601077/20:47
EmilienMfultonj: i'll let you guys figure20:47
EmilienMbut this chance isn't good20:47
*** mjturek has joined #tripleo20:48
* EmilienM files a bug and revert20:48
fultonjhttps://docs.openstack.org/tripleo-docs/latest/install/installation/installation.html#installing-the-undercloud20:48
EmilienM"If you intend to deploy Ceph in the overcloud, or configure the overcloud to use an external Ceph cluster, and are running Pike or newer, then install ceph-ansible on the undercloud:"20:48
EmilienMwhat if I don't intend?20:49
EmilienMI still need it20:49
fultonjwith that change, yes20:49
EmilienMso if you want me to add it as a dependency20:49
EmilienMyou'll need to change the doc, the default RDO repos and quickstart20:49
mwhahahawhich do we ship ceph-ansible by default elsewhere?20:50
fultonjright, so it takes a few steps to add it as a dep20:50
fultonjceph tools20:50
fultonjthough it's still cross shipped20:50
fultonjas a dep downstream20:50
fultonji don't think we can add it as a dep quickly without some coordination and agreement20:51
fultonjso i thought #3 for bleeding20:51
EmilienM#3 is hiding the issue20:51
fultonjtemporarily20:51
EmilienMbut fine20:51
fultonjbut yes20:51
mwhahahait would be better to only include that mount if ceph is enabled or do some host prep step to install ceph-ansible that throws a warning if it fails20:51
fultonjvalid bug to be assigned to me or gfidente20:52
EmilienMfultonj: i'm filling one20:53
fultonjEmilienM: thanks20:53
EmilienMfultonj: so you're proposing a patch?20:53
EmilienMor me20:54
fultonjfor #3?20:54
fultonji can if you can show me where prep script is20:54
*** mjturek has quit IRC20:54
*** lblanchard has quit IRC20:54
*** trown is now known as trown|outtypewww20:54
fultonjunless you want to do it (you're probably faster but i can once i know where to put the check)20:54
mwhahahahttps://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/mistral-engine.yaml#L12420:55
EmilienMfultonj: https://bugs.launchpad.net/tripleo/+bug/179618820:55
EmilienMthe bug20:55
openstackLaunchpad bug 1796188 in tripleo "Undercloud fails to deploy (with Podman) if ceph-ansible isn't installed" [High,Triaged]20:55
fultonjEmilienM: i can file patch20:56
EmilienMmwhahaha: thats not mistral executor20:57
fultonji'll add you to review +alex20:57
EmilienMfultonj: thx for your help20:57
mwhahahaoh thought it was engine20:57
fultonjengine20:57
mwhahahaeitherway, something like that20:57
fultonjvs exec20:57
fultonjit's exec20:57
EmilienMwe need exec20:57
mwhahahahttps://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/mistral-executor.yaml#L14020:57
fultonji rememver these files20:57
mwhahahafine since you can't s/enginer/executor20:57
EmilienMfultonj: if you want to test it in your box, change undercloud.conf to have container_cli = podman20:58
EmilienMand you'll see how that works20:58
fultonjEmilienM: wow20:58
fultonjno other patches needed ?20:58
EmilienMfultonj: oh and disable selinux please, we have an ongoing bug that is being fixed20:59
EmilienMfultonj: nope20:59
EmilienMfultonj: I told you, we were ready to go in gate20:59
EmilienMand my dreams are gone !20:59
mwhahahaweshay: can you pike the appropriate people to review https://review.openstack.org/#/c/604487/20:59
weshaymwhahaha, murderer21:00
*** mwhahaha changes topic to "Welcome to Stein | CI Status: YELLOWish? - ovb is sad still | https://docs.openstack.org/tripleo-docs/latest/"21:00
*** dprince has quit IRC21:04
*** thrash is now known as thrash|g0ne21:04
diwI've been doing some digging it looks like openstack-ironic-staging-drivers isn't included in the container, I'm guessing I need to rebuild the ironic-conductor container and use it locally? Or do you think the kolla folks would add it to the ironic-conductor spec?21:07
mwhahahathat's a question for the kolla folks :D i couldn't see why not21:08
mwhahahaif you need it in tripleo, you could always propose it to our overrides but it would likely be accepted in kolla21:08
diwI was working off https://redhatstackblog.redhat.com/2018/07/10/virtualize-your-openstack-control-plane-with-red-hat-virtualization-and-red-hat-openstack-platform-13/21:09
diwI'm guessing the RHOSP official images must include it in the image?21:09
mwhahahano idea21:09
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287221:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571821:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)21:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]21:10
diwI'll see about proposing it to the kolla guys.21:10
mwhahahadiw: oh no so i think that's for the undercloud which in Queens was not containerized21:11
mwhahahaso the driver could just be installed on the undercloud21:11
mwhahahawith Rocky+ it's containerized and we would need that driver installed in the ironic container21:11
mwhahahai wouldn't see why you couldn't land that in kolla21:11
*** agopi has quit IRC21:12
*** itlinux has quit IRC21:14
*** ade_lee has quit IRC21:14
diwI'll raise it with them in the morning.  Thanks21:15
fultonjEmilienM: https://review.openstack.org/#/c/608086/21:16
* fultonj has to test21:17
*** itlinux has joined #tripleo21:17
EmilienMfultonj: looking21:17
EmilienMfultonj: looks good to me21:18
fultonjok, cool21:18
fultonjEmilienM: is there an experimental job?21:18
EmilienMno21:18
EmilienMfultonj: I8942811c210f442d3cb630721c5105b725293d2921:18
EmilienMI'm about to run depends-on21:18
EmilienMfultonj: https://review.openstack.org/#/c/600517/21:19
fultonjok, cool21:19
EmilienMmwhahaha: where is the gerrit bot?21:19
EmilienMon pto or what21:19
mwhahahaEmilienM: so i asked21:19
mwhahahait thinks it is here21:19
mwhahahabut isn't21:19
mwhahahathey need to patch up the bot21:19
EmilienMlol21:19
mwhahahaso it's off telling us all the things are merging, but isn't actually doing it21:20
mwhahahahttp://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-10-04.log.html#t2018-10-04T16:43:4921:20
*** toure is now known as toure|gone21:22
mwhahahaweshay: https://review.openstack.org/#/c/606046/ is the patch to enable the extra tests in the standalone21:23
*** diw has quit IRC21:25
fultonjEmilienM: in other news https://gitlab.com/fultonj/tripleo-standalone-edge21:29
fultonji was able to get it working for thanks for getting it started ^21:30
fultonji have a script to export the info from heat to generate the environment file inputs based on slagle's work so no manual edits were required21:31
EmilienMfultonj: wow nice21:35
EmilienMhttps://gitlab.com/fultonj/tripleo-standalone-edge/blob/master/create-compute-stack-env.sh21:36
EmilienMi see21:36
*** panda is now known as panda|off21:36
*** vinaykns has joined #tripleo21:36
fultonja little hacky but a start21:36
* fultonj has a hard stop now21:37
fultonjttyl21:37
*** Vorrtex has quit IRC21:49
*** weshay is now known as weshay_pto21:51
*** itlinux has quit IRC22:05
*** jcoufal_ has quit IRC22:07
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287222:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571822:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)22:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]22:10
*** social has quit IRC22:10
*** med_ has joined #tripleo22:11
mwhahahalooks like rdo cloud is no bueno again22:12
mwhahahahttps://logs.rdoproject.org/27/607727/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/d2f2064/job-output.txt.gz#_2018-10-04_21_47_06_53960822:12
*** social has joined #tripleo22:22
*** itlinux has joined #tripleo22:23
*** bdodd has quit IRC22:25
*** itlinux has quit IRC22:25
*** rcernin has joined #tripleo22:28
*** vinaykns has quit IRC22:37
*** social has quit IRC22:42
*** bdodd has joined #tripleo22:42
*** social has joined #tripleo22:44
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179287223:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/179571823:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)23:10
openstackLaunchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged]23:10
*** tosky has quit IRC23:17
*** slagle has joined #tripleo23:35

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!