| *** hamzy_ has joined #tripleo | 00:01 | |
| *** hamzy_ is now known as hamzy | 00:02 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 00:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 00:10 |
| *** slagle has quit IRC | 00:15 | |
| *** slagle has joined #tripleo | 00:15 | |
| *** rcernin has quit IRC | 00:27 | |
| *** rcernin has joined #tripleo | 00:29 | |
| *** rbrady-afk has quit IRC | 00:52 | |
| *** med_ has quit IRC | 01:01 | |
| *** mjturek has joined #tripleo | 01:01 | |
| *** mjturek has quit IRC | 01:02 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 01:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 01:10 |
| *** mrsoul has joined #tripleo | 01:17 | |
| *** med_ has joined #tripleo | 01:18 | |
| *** mschuppert has quit IRC | 01:20 | |
| *** moshele has joined #tripleo | 01:39 | |
| *** moshele has quit IRC | 01:48 | |
| *** agopi has joined #tripleo | 01:57 | |
| *** itlinux has joined #tripleo | 02:05 | |
| *** ramishra has joined #tripleo | 02:09 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 02:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 02:10 |
| *** rh-jelabarre has quit IRC | 02:57 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 03:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 03:10 |
| *** agopi has quit IRC | 03:21 | |
| *** agopi has joined #tripleo | 03:21 | |
| *** rlandy has quit IRC | 03:22 | |
| *** jaganathan has joined #tripleo | 03:23 | |
| *** udesale has joined #tripleo | 03:49 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 04:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 04:10 |
| *** sri_ has quit IRC | 04:13 | |
| *** ykarel__ has joined #tripleo | 04:15 | |
| *** med_ has quit IRC | 04:16 | |
| *** ykarel__ is now known as ykarel | 04:34 | |
| *** diw has joined #tripleo | 04:39 | |
| *** Petersingh has joined #tripleo | 04:41 | |
| *** rcernin has quit IRC | 04:42 | |
| *** Petersingh is now known as Petersingh|afk | 04:43 | |
| *** rcernin has joined #tripleo | 04:46 | |
| *** holser_ has joined #tripleo | 04:48 | |
| *** holser_ has quit IRC | 04:51 | |
| *** flaper87 has quit IRC | 04:52 | |
| *** jbadiapa has quit IRC | 04:52 | |
| *** d0ugal has quit IRC | 04:52 | |
| *** dhill_ has quit IRC | 04:52 | |
| *** shyamb has joined #tripleo | 04:54 | |
| *** flaper87 has joined #tripleo | 04:57 | |
| *** jbadiapa has joined #tripleo | 04:57 | |
| *** d0ugal has joined #tripleo | 04:57 | |
| *** dhill_ has joined #tripleo | 04:57 | |
| *** shyamb has quit IRC | 05:01 | |
| *** agopi has quit IRC | 05:03 | |
| *** shyamb has joined #tripleo | 05:07 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 05:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 05:10 |
| *** Petersingh|afk is now known as Petersingh | 05:16 | |
| *** ratailor has joined #tripleo | 05:18 | |
| *** shyamb has quit IRC | 05:22 | |
| *** agopi has joined #tripleo | 05:24 | |
| *** shyamb has joined #tripleo | 05:35 | |
| *** jtomasek has joined #tripleo | 05:45 | |
| *** shyamb has quit IRC | 05:47 | |
| *** iranzo has joined #tripleo | 05:55 | |
| Tengu | hello there | 06:00 |
| *** shyamb has joined #tripleo | 06:01 | |
| *** jfrancoa has joined #tripleo | 06:03 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 06:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 06:10 |
| *** pvc has quit IRC | 06:10 | |
| *** dciabrin has joined #tripleo | 06:14 | |
| *** shyamb has quit IRC | 06:15 | |
| *** shyamb has joined #tripleo | 06:15 | |
| *** ksambor has joined #tripleo | 06:22 | |
| *** Petersingh is now known as Petersingh|afk | 06:25 | |
| *** mschuppert has joined #tripleo | 06:26 | |
| *** udesale has quit IRC | 06:39 | |
| *** kopecmartin|scho is now known as kopecmartin|ruck | 06:41 | |
| *** cylopez has joined #tripleo | 06:45 | |
| *** quiquell|off is now known as quiquell|brb | 06:50 | |
| *** udesale has joined #tripleo | 06:53 | |
| Tengu | quiquell|brb: heya! apparently there are some other needs introduced by the removal of xinetd: http://logs.openstack.org/30/607130/2/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/41bf62c/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-10-05_06_04_00 | 06:56 |
| Tengu | quiquell|brb: I'll check that shortly. | 06:56 |
| *** pcaruana has joined #tripleo | 06:57 | |
| Tengu | o_O the code in t-h-t shows a test to ensure xinetd is still active. weird. | 06:58 |
| *** Petersingh|afk is now known as Petersingh | 07:04 | |
| *** rcernin has quit IRC | 07:04 | |
| *** quiquell|brb is now known as quiquell | 07:08 | |
| quiquell | Tengu: hello there | 07:08 |
| quiquell | Tengu: Maybe it's related to promotions and is using and old THT version | 07:09 |
| *** pvc has joined #tripleo | 07:09 | |
| *** jpena|off is now known as jpena | 07:10 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 07:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 07:10 |
| pvc | hi guys how can i configure overcloud baremetal with ipv6 networkgni | 07:10 |
| quiquell | Tengu: Ahh I see can we check the flag there ? | 07:10 |
| pvc | quiquell what should i use for my baremetal node with vlan for overcloud | 07:12 |
| pvc | single-nic-vlands? | 07:12 |
| Tengu | quiquell: ah, if it's using an old package... what can I do for you on that topic? I'm no guru | 07:13 |
| *** shyamb has quit IRC | 07:16 | |
| *** amoralej|off is now known as amoralej | 07:17 | |
| Tengu | erf.. apparently this is the only failure for one of my patch in the gate :'( | 07:18 |
| *** holser_ has joined #tripleo | 07:18 | |
| Tengu | and not due to my code change :'((( | 07:18 |
| quiquell | Tengu: first we have to know where this check is introduced and if we can use the step 0 flag for xinetd to check that it's there | 07:19 |
| quiquell | Tengu: btw, it's very natural to not restart something that is not present so add a guard there make sense | 07:19 |
| Tengu | quiquell: hmm, there is a guard for that already. | 07:20 |
| Tengu | at least in master code. | 07:20 |
| Tengu | quiquell: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/ironic-pxe.yaml#L98-L106 | 07:20 |
| *** huynq_ has joined #tripleo | 07:23 | |
| Tengu | quiquell: the test: tripleo-ci-centos-7-containerized-undercloud-upgrades | 07:23 |
| *** shardy has joined #tripleo | 07:28 | |
| *** shyamb has joined #tripleo | 07:28 | |
| *** skramaja has joined #tripleo | 07:29 | |
| *** skramaja_ has joined #tripleo | 07:38 | |
| *** skramaja has quit IRC | 07:38 | |
| *** Petersingh is now known as Petersingh|afk | 07:38 | |
| *** bogdando has joined #tripleo | 07:39 | |
| quiquell | Tengu: It's just a matter of backporting to rocky and others https://github.com/openstack/tripleo-heat-templates/blob/stable/rocky/docker/services/ironic-pxe.yaml#L98 | 07:45 |
| *** tosky has joined #tripleo | 07:45 | |
| quiquell | Tengu: Going to backport this https://github.com/openstack/tripleo-heat-templates/commit/8e127bce07ce0579f7f4e6728410de38de5ecaf1 | 07:45 |
| quiquell | jistr: Are you backporting this already ^ ¿ | 07:46 |
| *** ykarel has quit IRC | 07:46 | |
| Tengu | quiquell: ah, ok. thanks :) | 07:46 |
| *** ykarel has joined #tripleo | 07:46 | |
| Tengu | hm. something seems to have changed at some point for my podman+selinux work X( | 07:47 |
| Tengu | all generated files have the wrong type ;_; | 07:47 |
| quiquell | Tengu: I suspect we have go beyond something and discovered new issues | 07:47 |
| quiquell | Tengu: what generated files ? | 07:47 |
| Tengu | quiquell: the ones in kolla/config_files - nothing to do with the current issue in gate. | 07:48 |
| quiquell | Tengu: The backporing is pending review https://review.openstack.org/#/c/607966/ | 07:48 |
| Tengu | quiquell: can't do more than +1 | 07:48 |
| quiquell | Tengu: me too | 07:49 |
| quiquell | bogdando, marios|rover: Can you hel workflow this https://review.openstack.org/#/c/607966/ ? | 07:49 |
| Tengu | thank you bogdando :) | 07:54 |
| *** shyamb has quit IRC | 08:01 | |
| marios | quiquell: sorry folks didn't see in time | 08:04 |
| *** salmankhan has joined #tripleo | 08:08 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 08:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 08:10 |
| *** rdopiera has joined #tripleo | 08:11 | |
| quiquell | marios: How are the gates ? we need this to move move RDO jobs to new workflow https://review.openstack.org/#/c/608080/ | 08:31 |
| marios | quiquell: check jobs are green this morning the failing scenarios are working now looks like | 08:33 |
| marios | quiquell: as you can see grafana zuul queue is at 7 hours | 08:33 |
| marios | quiquell: i think slowly getting back to 'working' till it breaks again :) | 08:33 |
| quiquell | marios: It's ok if I ask at #opensack-infra about prioritize it ? we cannot go back to normal with promotions at new workflow without it | 08:34 |
| quiquell | marios: Not purging it just put it in the top | 08:34 |
| marios | quiquell: ack | 08:34 |
| marios | kopecmartin|ruck: ^ | 08:34 |
| *** derekh has joined #tripleo | 08:35 | |
| mandre | marios: seems like the upgrade job is failing | 08:35 |
| quiquell | mandre, kopecmartin|ruck: thanks | 08:35 |
| mandre | marios: see https://review.openstack.org/#/c/601241/ | 08:35 |
| mandre | http://logs.openstack.org/41/601241/16/check/tripleo-ci-centos-7-undercloud-upgrades/701e728/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-10-05_07_01_57 | 08:35 |
| marios|rover | mandre: thanks checking (I saw top two at http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 failing check jobs undercloud-upgrades was going to check in a bit) | 08:38 |
| marios|rover | mandre: so your review will fix the Could not find the requested service xinetd: host ? | 08:39 |
| marios|rover | or whatever the root was | 08:39 |
| marios|rover | mandre: oh no sorry | 08:39 |
| mandre | marios|rover: nah, my patch is blocked because of this new CI issue | 08:39 |
| marios|rover | i just read it | 08:40 |
| *** moshele has joined #tripleo | 08:40 | |
| mandre | I just wanted to let you know of this new failure | 08:40 |
| marios|rover | yeah ok thanks mandre we'll look into it (cc kopecmartin|ruck ) | 08:40 |
| *** panda|off is now known as panda | 08:40 | |
| mandre | also in case you have an idea what that is or how to fix it... | 08:41 |
| marios|rover | mandre: well its not a known one or i missed it ;) | 08:41 |
| *** ykarel_ has joined #tripleo | 08:44 | |
| *** moshele has quit IRC | 08:45 | |
| kopecmartin|ruck | marios|rover, it seems it's not only that job, tripleo-ci-centos-7-containerized-upgrades failed because of that too http://logs.openstack.org/00/606100/5/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/6e31d3e/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-10-05_04_30_22 | 08:45 |
| kopecmartin|ruck | marios|rover, I'll check bugs and report it if it's not already reported | 08:46 |
| mandre | marios|rover: this is our bug https://bugs.launchpad.net/tripleo/+bug/1792527 | 08:46 |
| openstack | Launchpad bug 1792527 in tripleo "Undercloud upgrade rocky -> master broken on ironic" [Critical,Fix released] - Assigned to Jiří Stránský (jistr) | 08:46 |
| mandre | jistr has a patch | 08:46 |
| *** ykarel has quit IRC | 08:47 | |
| marios|rover | ack mandre kopecmartin|ruck the error is coming from this task? https://github.com/openstack/tripleo-heat-templates/blob/051f4e0059c4b314754154b320270f59349d686d/docker/services/ironic-pxe.yaml#L102 | 08:48 |
| marios|rover | jistr: patch? ^ | 08:48 |
| marios|rover | kopecmartin|ruck: did you file a bug or should i go ahead? | 08:49 |
| kopecmartin|ruck | marios|rover, they already have a bug https://bugs.launchpad.net/tripleo/+bug/1792527, we can comment there | 08:49 |
| openstack | Launchpad bug 1792527 in tripleo "Undercloud upgrade rocky -> master broken on ironic" [Critical,Fix released] - Assigned to Jiří Stránský (jistr) | 08:49 |
| marios|rover | wow thats an old bug | 08:50 |
| mandre | marios|rover: I'm not sure jistr's patch causes the bug of fixes it :) | 08:50 |
| marios|rover | oh but its master upgrade so ok i mean rocky->master | 08:50 |
| marios|rover | but still | 08:50 |
| kopecmartin|ruck | marios|rover, upgrades jobs work there https://review.openstack.org/#/c/590774/ | 08:50 |
| mandre | the patch in master merged yesterday, and there it the rocky backport that is about to merge | 08:50 |
| mandre | so with the rocky backport we should be good, correct? | 08:51 |
| *** ykarel_ is now known as ykarel|lunch | 08:51 | |
| marios|rover | mandre: yeah job looks green on both patches | 08:52 |
| marios|rover | thanks jistr :) | 08:53 |
| mandre | marios|rover: yeah upgrade job in master is upgrading rocky -> master afaik | 08:55 |
| mandre | so jistr patch will fix the gate | 08:55 |
| *** aufi_ has joined #tripleo | 08:55 | |
| marios|rover | mandre: ack we have to wait for next runs then https://review.openstack.org/#/c/560445 currently running and the previous one was fail on that undercloud-upgrades | 08:57 |
| *** shyamb has joined #tripleo | 08:58 | |
| *** gfidente has joined #tripleo | 09:01 | |
| *** janki has joined #tripleo | 09:05 | |
| *** chandankumar has joined #tripleo | 09:06 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 09:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 09:10 |
| *** agopi has quit IRC | 09:14 | |
| jistr | marios|rover, mandre: which job is broken? the backport should only affect non-master jobs i think... | 09:16 |
| mandre | jistr: I've seen the upgrade jobs fail in https://review.openstack.org/#/c/601241/ | 09:17 |
| jistr | mandre: that should have been fixed by the patch which went to master though | 09:18 |
| marios|rover | jistr: this http://logs.openstack.org/24/607624/1/check/tripleo-ci-centos-7-undercloud-upgrades/89b0b58/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz | 09:19 |
| *** sai_p has quit IRC | 09:19 | |
| mandre | jistr: isn't it failing still because we're upgrading *from rocky | 09:19 |
| jistr | mandre: we had the same issue on this patch, but the master merge fixed it https://review.openstack.org/#/c/590774/ | 09:19 |
| marios|rover | jistr: its hitting the bug that your patch fixes | 09:19 |
| mandre | and rocky doesn't have the patch yet? | 09:19 |
| jistr | mandre: on upgrading R->m we're using master templates. Or at least we should. If not, it's a bug which that patch can't help with. | 09:20 |
| marios|rover | yeah but should be using the master upgrade tasks | 09:20 |
| marios|rover | so why is master gate still hitting it then? | 09:20 |
| mandre | hmm | 09:20 |
| jistr | mandre: the UC job had stale t-h-t which was fixed by this one https://review.openstack.org/#/c/605745/ | 09:20 |
| * jistr opens the log from marios|rover | 09:20 | |
| *** pdeore has joined #tripleo | 09:21 | |
| jistr | mandre, marios|rover: my guess is that we somehow regressed to using Rocky templates for the upgrade to master, instead of master templates. Previously we successfully fixed it with this https://review.openstack.org/#/c/605745/ but i'm not sure what's wrong now. | 09:23 |
| jistr | mandre, marios|rover: if we fix the backport, that might fix the immediate problem, because Rocky templates will contain the fixed tasks. But it will only hide the true root cause, that we're using Rocky templates where we should be using master. | 09:25 |
| *** salmankhan has quit IRC | 09:25 | |
| jistr | s/fix the backport/merge the backport/ | 09:25 |
| jistr | marios|rover, mandre: i'm even tempted to block the backport from landing to force correct solution to be investigated :D | 09:26 |
| mandre | jistr: the only issue if you do that is that you'll prevent anything from merging in master :) | 09:27 |
| *** akrivoka has joined #tripleo | 09:27 | |
| jistr | mandre: or we should switch the UC upgrade to non-voting, as it's not CIing what it should be | 09:27 |
| marios|rover | jistr: yeah sounds plausible otherwise why doesn't that job have the 'right' tasks (i mean that we are using old templates) | 09:27 |
| jistr | marios|rover: yea | 09:28 |
| mandre | jistr: yeah I like it better | 09:28 |
| marios|rover | jistr: so really we don't want your backport to 'fix' it | 09:28 |
| marios|rover | jistr: but it will :) until we hit some other issue from the rocky templates :) | 09:28 |
| jistr | marios|rover: yea exactly | 09:29 |
| jistr | marios|rover: the other problem is that we need the patch in Rocky legitimately, it fixes undercloud minor upgrade in Rocky :D | 09:31 |
| jistr | so the backport was proposed for that reason | 09:31 |
| marios|rover | jistr: ack lets not block merging that (or the gate in general) but i'll continue to dig for the repo issue | 09:32 |
| marios|rover | kopecmartin|ruck: fyi ^ | 09:32 |
| jistr | marios|rover++ ack thanks a lot | 09:32 |
| *** salmankhan has joined #tripleo | 09:32 | |
| *** Petersingh|afk is now known as Petersingh | 09:33 | |
| Tengu | gfidente: hello! I think you pushed a patch mounting ceph-ansible into mistral container right? even if we don't have ceph-ansible installed apparently. | 09:36 |
| Tengu | gfidente: and, if docker creates host missing directories, this is not the case for podman, so you patch is crashing when deploying the undercloud with podman: ceph-ansible isn't installed, this no /usr/share/ceph-ansible directory on the host, thus podman crashes | 09:36 |
| Tengu | gfidente: so what's the right thing to do? install ceph-ansible on the nodes running mistral thingy no matter if we actually use ceph, or just ensure the directory exists? | 09:37 |
| jistr | Tengu, gfidente: ceph-ansible should be a dependency of t-h-t | 09:40 |
| Tengu | hmmm. | 09:40 |
| jistr | i thought it already is | 09:40 |
| Tengu | jistr: I have puppet-ceph installed right now. | 09:41 |
| Tengu | but no ceph-ansible. | 09:41 |
| Tengu | jistr: do you know the command to check deps for an RPM? | 09:41 |
| Tengu | it doesn't show up with "yum info <package>" :( | 09:41 |
| jistr | Tengu: not off the top of my head but i recall it was easy to find | 09:41 |
| *** paramite has joined #tripleo | 09:42 | |
| Tengu | ah, apparently rpm -qpR | 09:43 |
| jistr | Tengu: https://serverfault.com/questions/199743/how-to-list-rpm-dependencies | 09:43 |
| jistr | yea | 09:43 |
| *** gkadam has joined #tripleo | 09:43 | |
| Tengu | yeah, so nope. | 09:43 |
| jistr | Tengu: ah ha so it didn't merge yet, just found this: https://review.rdoproject.org/r/#/c/16362/ | 09:43 |
| Tengu | at least for openstack-tripleo-heat-templates-9.0.0-0.20181005085601.11ab791.el7.noarch it's not a dep. | 09:44 |
| *** shyamb has quit IRC | 09:44 | |
| Tengu | damn. | 09:44 |
| Tengu | so yeah, I really need that nasty boy in here. | 09:44 |
| Tengu | I can force the package install easily in my lab, but for CI and other things wanting to use podman, we have to get that new package :). | 09:45 |
| Tengu | thanks for the info/link, jistr ! | 09:45 |
| jistr | np :) | 09:45 |
| gfidente | jistr Tengu yeah I had a patch for that | 09:46 |
| gfidente | jistr Tengu meantime I think I have to make sure we install ceph-ansible in all jobs | 09:46 |
| Tengu | gfidente: yep | 09:47 |
| Tengu | so if ceph-ansible is a dep, it's already nice, but that means we will probably need to add new sources. | 09:47 |
| *** shyamb has joined #tripleo | 09:47 | |
| Tengu | as the default tripleo-repos -b <version> current doesn't add ceph, of course. | 09:47 |
| *** chandankumar has quit IRC | 09:48 | |
| gfidente | Tengu nice thanks for pointing it out, do you want to add a patch for that? | 09:48 |
| Tengu | gfidente: I'm a bit busy with my tests for podman :/ | 09:49 |
| Tengu | gfidente: guess you'll be faster than me, since I don't know where I have to push a patch :/ | 09:49 |
| gfidente | ok I will | 09:49 |
| Tengu | gfidente: note, tripleo-repos -b master current ceph does the trick for example. | 09:49 |
| marios|rover | jistr: mandre maybe this is why | 09:50 |
| marios|rover | jistr: using fs 47 in previous releases script https://github.com/openstack-infra/tripleo-ci/commit/19172abff6e8ec0aa6494c4858e6ffa55900786b | 09:50 |
| marios|rover | s/in/with | 09:50 |
| gfidente | Tengu jistr this seems to be blocked https://review.rdoproject.org/r/#/c/16362/ | 09:51 |
| jistr | marios|rover: the job was green on that patch though... cc quiquell ^ | 09:51 |
| Tengu | gfidente: yeah, as you need to add the repositories first. | 09:52 |
| marios|rover | jistr: ack ok might not be it but looking for recent things that might affect | 09:52 |
| Tengu | gfidente: would be great to get in touch with anyone from the #openstack-infra I think | 09:52 |
| marios|rover | jistr: either way it means there is a bug in the emit release if it was that (but job was green so..) | 09:53 |
| jistr | marios|rover: yea (FWIW re CI results on that patch: https://review.openstack.org/#/c/590774/) | 09:53 |
| jistr | marios|rover: it's likely going to be something that merged later than ^^. We had the job green for about a day i think. | 09:54 |
| marios|rover | jistr: ack thanks jistr ! i believe you | 09:54 |
| * marios|rover clicks intensely | 09:54 | |
| *** ykarel|lunch is now known as ykarel | 09:54 | |
| jistr | marios|rover: yea i'm linking mainly in case you find it helpful :D maybe the logs there or something, for comparison with current state. Though i'm not sure what to look at. | 09:55 |
| *** shyamb has quit IRC | 09:55 | |
| jistr | marios|rover: i already checked that the undercloud-upgrade.sh has correct yum update still, so maybe the /etc/yum.repos.d/ content is wrong now | 09:55 |
| jistr | (?) | 09:55 |
| marios|rover | jistr: thanks, yeah i am also just poking at the moment not sure | 09:56 |
| marios|rover | jistr: if you come up with aything add it to the bug | 09:56 |
| quiquell | marios|rover, jistr: Let me know if you need any help, I was involve with the r->m sh... stuff | 09:57 |
| marios|rover | quiquell: ack, basically the problem is we are hitting https://bugs.launchpad.net/tripleo/+bug/1792527 (see comment 6 for examples from today) but we should not be, if that job is using master templates for the upgrade. THe only way to hit it is if the job is using rocky templates | 09:58 |
| openstack | Launchpad bug 1792527 in tripleo "Undercloud upgrade rocky -> master broken on ironic" [Critical,Fix released] - Assigned to Jiří Stránský (jistr) | 09:58 |
| marios|rover | quiquell: so question is why are we using rocky templates for the tripleo-ci-centos-7-undercloud-upgrades | 09:58 |
| gfidente | Tengu so I am looking at the tripleo-repos code | 09:58 |
| gfidente | it doesn't add any repo by default | 09:58 |
| gfidente | it's expecting you to provide a list of repos to enable | 09:58 |
| Tengu | gfidente: yep | 09:59 |
| Tengu | so we'd probably need to update the way tripleo-repos is called. | 09:59 |
| *** chandankumar has joined #tripleo | 09:59 | |
| gfidente | Tengu I don't think so | 09:59 |
| gfidente | it looks fine as it is | 09:59 |
| Tengu | else it won't enable the ceph-luminous (or any other ceph release wanted by the branch we're activate) | 09:59 |
| Tengu | gfidente: well, if you don't get the ceph repo, you can't install ceph-ansible. | 10:00 |
| Tengu | so it will basically fail installing openstack-tripleo-heat-templates | 10:00 |
| Tengu | no good ;) | 10:00 |
| quiquell | marios|rover: We just point it to rocky, maybe before it was using queens templates | 10:02 |
| quiquell | marios|rover: We are not suppose to use rocky templates for undercloud install and master templates for undercloud-upgrade ? | 10:02 |
| marios|rover | quiquell: jistr or, it is using master tht but latest delorean build doesn't include the fix since master only merged 20 hours ago anyway? | 10:02 |
| marios|rover | quiquell: ack maybe we *are* pointing at master? | 10:03 |
| quiquell | marios|rover: btw we have stop promotions | 10:03 |
| *** shyamb has joined #tripleo | 10:03 | |
| quiquell | marios|rover: We are also testing estuff with the new workflow, maybe we have break something | 10:03 |
| quiquell | marios|rover: We are just playing with tripleo-ci-testing but not with tripleo-ci | 10:03 |
| quiquell | marios|rover: I think upstream uses tripleo-ci | 10:03 |
| jistr | marios|rover: hm latest delorean-current should have that, and we should be using -current for the whitelisted tripleo packages | 10:03 |
| marios|rover | jistr: for example via /yum.repos.d on that job as you say, http://logs.openstack.org/45/560445/155/check/tripleo-ci-centos-7-undercloud-upgrades/06186ad/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz i see openstack-tripleo-heat-templates-9.0.0-0.20181002082153.5ed260c.el7.src.rpm2018-10-02 08:23 | 10:03 |
| marios|rover | jistr: ah right delorean-current has newer | 10:04 |
| marios|rover | jistr: openstack-tripleo-heat-templates-9.0.0-0.20181005014339.051f4e0.el7.noarch.rpm2018-10-05 01:45 | 10:04 |
| jistr | yea | 10:05 |
| jistr | http://logs.openstack.org/45/560445/155/check/tripleo-ci-centos-7-undercloud-upgrades/06186ad/logs/undercloud/etc/yum.repos.d/delorean-current.repo.txt.gz | 10:05 |
| quiquell | marios|rover: so we are just missing promotions | 10:05 |
| marios|rover | quiquell: ack | 10:05 |
| quiquell | marios|rover: I think we can not use Depends-On merged stuff to test it | 10:06 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 10:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 10:10 |
| marios|rover | quiquell: whats the story on promotions? | 10:10 |
| marios|rover | 13:03 < quiquell> marios|rover: btw we have stop promotions | 10:11 |
| *** phuongnh has joined #tripleo | 10:11 | |
| marios|rover | quiquell: we have that ironic thing but is there something else? | 10:12 |
| quiquell | marios|rover: https://tree.taiga.io/project/tripleo-ci-board/task/104?kanban-status=1447277 | 10:13 |
| marios|rover | quiquell: migration? me checks | 10:13 |
| marios|rover | quiquell: ah | 10:13 |
| marios|rover | yeah | 10:13 |
| *** phuongnh has quit IRC | 10:16 | |
| *** shyamb has quit IRC | 10:18 | |
| *** udesale has quit IRC | 10:18 | |
| *** itlinux has quit IRC | 10:24 | |
| *** chandankumar has quit IRC | 10:26 | |
| *** sshnaidm is now known as sshnaidm|off | 10:35 | |
| *** jtomasek has quit IRC | 10:43 | |
| Tengu | gfidente: so yeah, adding the required source + ceph-ansible package allows to deploy the undercloud with podman again. | 10:50 |
| *** shyamb has joined #tripleo | 10:51 | |
| fultonj | Tengu gfidente did you see https://review.openstack.org/#/c/608086/ ? | 10:52 |
| gfidente | fultonj yeah shall we merge? | 10:53 |
| Tengu | well, YES we should merge asap | 10:53 |
| Tengu | wasn't sure this was the right thing to do | 10:53 |
| fultonj | Tengu: EmilienM and i talked about it yesterday | 10:54 |
| Tengu | but now that you point an already existing patch, fultonj, go on. I think we talked about it | 10:54 |
| Tengu | ok :) | 10:54 |
| Tengu | so yeah. obviously go for it. | 10:54 |
| Tengu | fultonj: the setype is probably NOT required though | 10:54 |
| fultonj | Tengu: i think we want to make ceph-ansible a dep but it won't be done easily so we wanted to at least ensure the directory is present | 10:54 |
| Tengu | fultonj: you won't write in it right? | 10:54 |
| fultonj | Tengu: you want line 160 deleted? | 10:55 |
| fultonj | https://review.openstack.org/#/c/608086/1/docker/services/mistral-executor.yaml | 10:55 |
| *** holser_ has quit IRC | 10:55 | |
| Tengu | fultonj: "want" is a big statement - I can check how it works with selinux if you want | 10:56 |
| fultonj | https://review.openstack.org/#/c/600517 | 10:56 |
| fultonj | ^ depends on that patch | 10:56 |
| Tengu | fultonj: lemme kick in my lab once more and deploy an undercloud with that change. | 10:56 |
| fultonj | Tengu: ok, cool | 10:57 |
| Tengu | fultonj: we don't have selinux enforcing on CI ;) | 10:57 |
| fultonj | Tengu: yeah, i understand there's a bug with that ATM | 10:57 |
| Tengu | ^^ | 10:57 |
| * fultonj back in a few minutes | 10:57 | |
| Tengu | so. hmm. branch, rebase, blah-blah, run the lab. | 10:57 |
| *** paramite has quit IRC | 10:58 | |
| *** paramite has joined #tripleo | 10:58 | |
| *** jchhatbar has joined #tripleo | 10:59 | |
| *** janki has quit IRC | 11:00 | |
| Tengu | fultonj: hm, why didn't you use the loop right above your addition in fact? | 11:00 |
| Tengu | its purpose is to create the directories anyway :) | 11:01 |
| *** jchhatbar has quit IRC | 11:01 | |
| *** jchhatbar has joined #tripleo | 11:02 | |
| Tengu | fultonj: lab's being built, feedback in about an hour or so. | 11:02 |
| *** Petersingh is now known as Petersingh|afk | 11:02 | |
| *** moshele has joined #tripleo | 11:02 | |
| *** jpena is now known as jpena|lunch | 11:04 | |
| Tengu | fultonj, gfidente : added a comment to https://review.openstack.org/608086 for the setype, will come back once I know :). | 11:04 |
| *** jchhatba_ has joined #tripleo | 11:06 | |
| *** jtomasek has joined #tripleo | 11:07 | |
| *** jchhatbar has quit IRC | 11:09 | |
| *** dtantsur|afk is now known as dtantsur | 11:09 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 11:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 11:10 |
| *** moguimar has joined #tripleo | 11:12 | |
| *** moshele has quit IRC | 11:20 | |
| *** quiquell is now known as quiquell|lunch | 11:20 | |
| *** paramite has quit IRC | 11:21 | |
| *** moshele has joined #tripleo | 11:21 | |
| *** paramite has joined #tripleo | 11:21 | |
| fultonj | Tengu: it seemed distinct from "create persistent data directory", but I could have renamed the task | 11:22 |
| gfidente | fultonj so I am not sure this is what we want https://review.openstack.org/#/c/608086/1 | 11:22 |
| gfidente | fultonj we just want to make sure we install ceph-ansible in the undercloud | 11:22 |
| fultonj | gfidente: yes, but then we'd have to patch RDO and the doc coordinate w/ downstream | 11:22 |
| gfidente | fultonj yeah we need to add ceph-ansible as dep to tht | 11:23 |
| fultonj | gfidente: so ultimately, we DO want to patch RDO and the doc coordinate w/ downstream | 11:23 |
| fultonj | but in order to quickly get podman undercloud working we did that patch | 11:23 |
| gfidente | fultonj yeah this is what we discussed with mburned | 11:23 |
| Tengu | fultonj: good point indeed. and if we don't need the setype as I suspect, we can keep it in a distinct task. | 11:23 |
| gfidente | fultonj right the submission to mark it as dep in tht is a bit slow https://review.rdoproject.org/r/#/c/16362/ | 11:27 |
| gfidente | fultonj I'll comment pointing to it | 11:27 |
| gfidente | fultonj when it's ready we can remove the workaround | 11:27 |
| Tengu | +1 | 11:27 |
| gfidente | fultonj++ | 11:28 |
| fultonj | :) | 11:28 |
| *** moguimar has quit IRC | 11:28 | |
| *** salmankhan has quit IRC | 11:29 | |
| *** moguimar has joined #tripleo | 11:34 | |
| *** huynq_ has quit IRC | 11:38 | |
| *** moguimar has quit IRC | 11:39 | |
| Tengu | fultonj: so without setype set, it's a "user_t" - I think container_t has access to it without any issue, at least read-only. | 11:40 |
| Tengu | still running. | 11:40 |
| Tengu | step3, that's when mistral_executor is started I think. | 11:41 |
| fultonj | ack | 11:41 |
| Tengu | so I should get a result pretty soon now. | 11:41 |
| fultonj | thanks Tengu when i have your results i can tweak the patch accordingly | 11:42 |
| *** quiquell|lunch is now known as quiquell | 11:42 | |
| *** dciabrin has quit IRC | 11:42 | |
| *** moguimar has joined #tripleo | 11:43 | |
| *** jtomasek has quit IRC | 11:48 | |
| Tengu | so step 5, mistral_executor is running for about a minute now. | 11:52 |
| jistr | shardy: here's the cleanup of old pcmk stuff https://review.openstack.org/#/c/608242 | 11:52 |
| jistr | cc bandini ^ :) | 11:53 |
| bandini | jistr: NICE! ;) | 11:55 |
| bandini | will look at it today | 11:55 |
| rascasoft | mwhahaha, hey, so I get my first RDOPhase2 Rocky success in ages, because of https://review.openstack.org/#/c/607950/, can you expand more your concern? Because to me we are fine | 11:56 |
| *** rh-jelabarre has joined #tripleo | 11:56 | |
| *** holser_ has joined #tripleo | 11:58 | |
| *** agopi has joined #tripleo | 11:58 | |
| bandini | jistr: another pcmk cleanup https://review.openstack.org/#/c/580305/ | 11:59 |
| *** agopi has quit IRC | 12:00 | |
| Tengu | fultonj: can confirm: no need to add the setype: The Undercloud has been successfully installed. without it :) | 12:00 |
| jistr | bandini: can we remove the 'WIP' from commit message or are you testing? | 12:00 |
| *** jtomasek has joined #tripleo | 12:00 | |
| fultonj | Tengu: is setype harmful? | 12:00 |
| fultonj | or just unecessary | 12:01 |
| *** jtomasek has quit IRC | 12:01 | |
| Tengu | fultonj: unecessary at most. | 12:01 |
| Tengu | fultonj: it will most than probably be reset to usr_t upon selinux related update. | 12:01 |
| *** jtomasek has joined #tripleo | 12:01 | |
| bandini | jistr: it was an old patch, I will remove it | 12:02 |
| Tengu | so, CIX mtg right now :) | 12:02 |
| fultonj | Tengu: i think we should merge asis then | 12:02 |
| Tengu | fultonj: fine for me | 12:02 |
| fultonj | Tengu: thanks | 12:02 |
| Tengu | just added a note about it, but not harmful. | 12:02 |
| Tengu | at least I see no issue. | 12:02 |
| fultonj | i think we're not yet supposed to +w | 12:02 |
| Tengu | well, it shouldn't be needed, and in fact, it MUST not be done anyway | 12:03 |
| Tengu | it's a content managed by an rpm, pushing stuff in there is bad ;) | 12:03 |
| *** med_ has joined #tripleo | 12:03 | |
| EmilienM | fultonj, gfidente : please take a look at my comment on https://review.rdoproject.org/r/#/c/16362/ | 12:06 |
| *** jpena|lunch is now known as jpena | 12:06 | |
| *** moguimar has quit IRC | 12:08 | |
| gfidente | EmilienM hey thanks for jumping in | 12:08 |
| *** ratailor has quit IRC | 12:08 | |
| *** moguimar has joined #tripleo | 12:09 | |
| EmilienM | gfidente: stop breaking us ! | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 12:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 12:10 |
| gfidente | EmilienM oh god, why did you install ceph-ansiblein the container in the first place? | 12:10 |
| gfidente | :D | 12:10 |
| EmilienM | gfidente: I didn't :P | 12:11 |
| gfidente | well I didn't either | 12:11 |
| gfidente | it shouldn't have been | 12:11 |
| gfidente | but now to deal with the repos | 12:11 |
| gfidente | you mean we should add it to delorean? | 12:11 |
| slagle | jistr: i've got a failed undercloud upgrade job on https://review.openstack.org/#/c/607701/ that appears to be because it's missing this patch https://review.openstack.org/#/c/602550/ | 12:11 |
| slagle | jistr: any idea how that could be? | 12:11 |
| *** udesale has joined #tripleo | 12:12 | |
| jistr | slagle: we discussed it earlier today, probably we somehow regressed to using Rocky templates for upgrading to master. The job was green for at least a day before this started happening again. | 12:12 |
| jistr | AFAIK | 12:12 |
| *** Petersingh|afk is now known as Petersingh | 12:13 | |
| jistr | slagle: even more unfortunately, there's a backport proposed to Rocky because it fixes undercloud minor update on Rocky: https://review.openstack.org/#/c/607966/ | 12:13 |
| jistr | slagle: once that merges the gate will get "fixed", but not really fixed, the problem will just be hidden/lurking | 12:13 |
| slagle | heh | 12:13 |
| slagle | ok :) | 12:14 |
| slagle | is there a bug open? i'll file one if not | 12:14 |
| jistr | marios|rover: probably there's not a bug open right? ^ | 12:14 |
| EmilienM | gfidente: read my comment I explained everything | 12:15 |
| jistr | marios|rover: i mean separately from the one which was already closed 2 days ago | 12:15 |
| EmilienM | gfidente: you need to patch puppet-dlrn | 12:15 |
| marios | jistr: no i didn't file one yet | 12:15 |
| marios | jistr: wanted to see if it was an actual issue...problem is the job shows lots of success today too sec; | 12:15 |
| marios | jistr: (just came off a call gimme a min) | 12:15 |
| jistr | sure | 12:16 |
| *** lblanchard has joined #tripleo | 12:16 | |
| *** asbishop has quit IRC | 12:16 | |
| marios | tripleo-ci-centos-7-undercloud-upgrades | 12:17 |
| marios | jistr: check ^ in http://cistatus.tripleo.org/ | 12:17 |
| marios | jistr: and show successes | 12:17 |
| jistr | marios: we should be looking at undercloud upgrades *containers* | 12:17 |
| marios | same | 12:17 |
| *** Petersingh is now known as Petersingh|gone | 12:17 | |
| marios | jistr: i mean both have similar pattern | 12:18 |
| marios | jistr: e.g.g. here http://logs.openstack.org/45/560445/156/check/tripleo-ci-centos-7-undercloud-upgrades/f1f2f6b/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-10-05_09_47_34 | 12:18 |
| jistr | marios: ah really, that's interesting | 12:18 |
| jistr | slagle: ^ | 12:18 |
| marios | jistr: the examples in comment #6 are all for 7-undercloud-upgrades | 12:21 |
| honza | jtomasek: florianf: tosky: i'm currently blocked by this, could you give it a quick once over? https://review.openstack.org/#/c/608251/ | 12:23 |
| *** morazi has joined #tripleo | 12:23 | |
| jtomasek | honza: on in | 12:24 |
| jtomasek | *it | 12:24 |
| florianf | honza: ack | 12:24 |
| *** boazel has joined #tripleo | 12:24 | |
| *** rlandy has joined #tripleo | 12:25 | |
| *** dprince has joined #tripleo | 12:25 | |
| tosky | honza: I can +1, but that's it, no +2 :) | 12:27 |
| *** amoralej is now known as amoralej|lunch | 12:27 | |
| honza | tosky: np, just wanted yours on it | 12:27 |
| *** agopi has joined #tripleo | 12:27 | |
| honza | tosky: your eyes* | 12:27 |
| *** dtantsur is now known as dtantsur|brb | 12:27 | |
| shardy | jistr: nice, thanks for picking that up! :) | 12:33 |
| *** salmankhan has joined #tripleo | 12:33 | |
| *** holser_ has quit IRC | 12:37 | |
| *** lblanchard has quit IRC | 12:39 | |
| *** trown|outtypewww is now known as trown | 12:41 | |
| chem | marios: quiquell so what would be the best way to include a pip package in there https://review.openstack.org/#/c/604756/ as ansible-lint is not yet installed at that time. | 12:44 |
| chem | marios: quiquell (hi) | 12:44 |
| chem | marios: quiquell is there some standart ? | 12:44 |
| marios | chem: don't know about standard but quickstart has some bootstrap dependencyies but where you need it on the undercloud/virt host? i think those are for virt host | 12:45 |
| chem | marios: it's for the undercloud | 12:45 |
| quiquell | chem: maybe requirements.txt at tq or tqe | 12:47 |
| chem | quiquell: marios oki, fine by me, it has the disadvantage to impact all deployments though | 12:48 |
| marios | chem: if its for the job, you might consider something in the repo_cmd_after | 12:49 |
| quiquell | chem: humm that's right | 12:49 |
| marios | chem: for example https://github.com/openstack/tripleo-quickstart/blob/38c42aab9abe37ab531e5aae7fc8721f5b0109e5/config/release/tripleo-ci/master.yml#L122 | 12:50 |
| *** ansmith has joined #tripleo | 12:50 | |
| chem | marios: hum ... it doesn't feel right in the release file | 12:51 |
| chem | marios: quiquell what about adding a simple pip resource in the standalone.yml playbook, would that be ok ? | 12:52 |
| quiquell | chem: Yep, totally if it's ok with standalone standards | 12:52 |
| chem | quiquell: marios oki, I'll try that and see how it goes. | 12:54 |
| chem | quiquell: marios thanks for the input. | 12:54 |
| marios | chem: ack exactly propose it and worst case someoen will tell you what the right way is | 12:54 |
| *** shyamb has quit IRC | 12:55 | |
| *** holser_ has joined #tripleo | 12:56 | |
| *** jcoufal has joined #tripleo | 13:01 | |
| pdeore | Hi everyone, can someone review this please, https://review.openstack.org/#/c/598560/ ? | 13:04 |
| *** toure|gone is now known as toure | 13:05 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 13:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 13:10 |
| *** ykarel has quit IRC | 13:12 | |
| *** moshele has quit IRC | 13:13 | |
| *** moshele has joined #tripleo | 13:14 | |
| *** vinaykns has joined #tripleo | 13:19 | |
| *** moshele has quit IRC | 13:22 | |
| quiquell | marios: gates broken at upgrades | 13:27 |
| *** ramishra has quit IRC | 13:28 | |
| *** panda is now known as panda|off | 13:31 | |
| *** med_ has quit IRC | 13:31 | |
| *** sai_p has joined #tripleo | 13:32 | |
| *** skramaja_ has quit IRC | 13:34 | |
| *** quiquell is now known as quiquell|off | 13:34 | |
| marios | quiquell|off: o/ | 13:34 |
| marios | sorry i was trying to get back on myirc bouncer connection/vpn issue | 13:35 |
| marios | quiquell|off: do you mean the undercloud upgrade job we were talking about earlier? | 13:35 |
| EmilienM | bandini: one million dollar question | 13:37 |
| EmilienM | bandini: can we get CI using pcmk-podman-rhel7 so we can make progress on upstream testing? | 13:37 |
| *** ykarel has joined #tripleo | 13:38 | |
| *** pdeore has quit IRC | 13:38 | |
| EmilienM | bandini: or move these packages in delorean-master-deps | 13:38 |
| *** amoralej|lunch is now known as amoralej | 13:39 | |
| bandini | EmilienM: we'd have to use a separate repo somehow, because if we add those to a generic delorean-master repo we'd get it everywhere which would break a bunch of other things | 13:39 |
| *** ykarel has quit IRC | 13:39 | |
| bandini | because we'd pull in pcmk 2 on existing ci jobs and that would be bad | 13:39 |
| *** ykarel_ has joined #tripleo | 13:39 | |
| *** boazel_ has joined #tripleo | 13:44 | |
| EmilienM | jpena, amoralej ^ what can we do here in your opinion? | 13:45 |
| *** boazel has quit IRC | 13:47 | |
| *** dtantsur|brb is now known as dtantsur | 13:48 | |
| amoralej | EmilienM, bandini, sorry i don't follow | 13:49 |
| amoralej | there is a differnt pacmaker build somewhere? | 13:49 |
| EmilienM | amoralej: we are working on replacing docker by podman | 13:49 |
| EmilienM | amoralej: on the overcloud, pacemaker version needs to change for that matter | 13:50 |
| EmilienM | amoralej: yes, bandini built a new version of pcmk for centos7 | 13:50 |
| EmilienM | amoralej: I have a copy of this repo here: https://files.macchi.pro:8443/pcmk-podman-rhel7/ | 13:50 |
| amoralej | but iiuc it will break other things | 13:51 |
| EmilienM | amoralej: yes, I'm not sure what exactly, but pcmk2 will not work everywhere as of today iiuc | 13:51 |
| bandini | yeah we'd also need specially built {galera,redis,rabbitmq} containers with that repo, (only for this podman CI job) | 13:52 |
| EmilienM | amoralej: I'm looking for a repo we can use and rely on, and gradually switch over in TripleO CI | 13:52 |
| amoralej | EmilienM, then what you need is to use it for a specific job? | 13:52 |
| EmilienM | for example, we would switch container-multinode first | 13:52 |
| EmilienM | then maybe one of the OVB jobs | 13:52 |
| bandini | shall we discuss this a bit more in detail in our upcoming call? | 13:52 |
| EmilienM | bandini: fine | 13:52 |
| amoralej | ok | 13:52 |
| bandini | amoralej: I have some package requests for the fedora-stable repo btw., will send you a shopping list ;) | 13:53 |
| amoralej | ok, no problem :) | 13:53 |
| *** asbishop has joined #tripleo | 13:53 | |
| bandini | shouldn't be a crazy amount of them | 13:53 |
| amoralej | bandini, we've had issues with rabbitmq and ssl testing it with packstack | 13:55 |
| bandini | amoralej: oh do share any details you might have on that | 13:56 |
| *** EmilienM is now known as EvilienM | 13:57 | |
| amoralej | bandini, logs in http://logs.openstack.org/59/607659/13/check/packstack-fedora-integration-scenario-py3-tempest/d2861d3/logs/ | 13:58 |
| bandini | amoralej: how hard would it be to either reproduce on my end or get and env showing the issue? (note: I have not used packstack in my whole life) | 14:02 |
| mwhahaha | EvilienM: undercloud upgrades broken? http://logs.openstack.org/80/608080/2/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/52824fe/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-10-05_12_49_02 | 14:05 |
| EvilienM | mwhahaha: https://review.openstack.org/#/c/607966/ | 14:06 |
| EvilienM | apparently we need that | 14:06 |
| mwhahaha | ah ok | 14:06 |
| mwhahaha | EvilienM: but shouldn't it have the latest THT? | 14:07 |
| mwhahaha | EvilienM: that code is in master and this was the upgrade to master, so is this case where we're running the upgrade but we're still using THT from rocky? | 14:07 |
| EvilienM | mwhahaha: not sure why Rocky templates are used during the upgrade | 14:07 |
| mwhahaha | ie we need to ensure tht is correctly pulled in when we update python-tripleoclient | 14:07 |
| mwhahaha | EvilienM: because we don't have a >= dependency for THT in python-tripleoclient | 14:07 |
| mwhahaha | EvilienM: we need to start using that to ensure when we upgrade python-tripleoclient that the correct version of THT is pulled in | 14:08 |
| EvilienM | https://github.com/rdo-packages/tripleoclient-distgit/blob/rpm-master/python-tripleoclient.spec#L136 | 14:08 |
| EvilienM | Requires: openstack-tripleo-heat-templates >= 9.0.0 | 14:08 |
| EvilienM | we probably need to bump that | 14:09 |
| *** ykarel_ is now known as ykarel | 14:09 | |
| mwhahaha | or quickstart is doing something silly | 14:09 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 14:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 14:10 |
| *** mcornea has joined #tripleo | 14:10 | |
| *** ade_lee has joined #tripleo | 14:11 | |
| EvilienM | mwhahaha: why does it fail *now* is also a good question | 14:11 |
| EvilienM | but I guess it's because we released 9.0.0 | 14:11 |
| mwhahaha | EvilienM: good question indeed | 14:12 |
| *** mjturek has joined #tripleo | 14:12 | |
| *** lblanchard has joined #tripleo | 14:12 | |
| EvilienM | oh | 14:12 |
| EvilienM | I know | 14:12 |
| EvilienM | so we released 9.0.0 and 9.0.0 is the rocky-rc2 tag | 14:13 |
| EvilienM | mwhahaha: now that we released rocky we should tag stein | 14:13 |
| mwhahaha | EvilienM: oh don't we need to bump all the version numbers now? | 14:13 |
| EvilienM | 10.0.0.x | 14:13 |
| mwhahaha | yea | 14:13 |
| EvilienM | ok i'm doing it | 14:13 |
| EvilienM | mwhahaha: for the xinetd thing I'm not sure if whether or not we need a new tag though | 14:14 |
| EvilienM | (in rocky) | 14:14 |
| mwhahaha | no it's find for rocky | 14:14 |
| mwhahaha | because xinetd will exist | 14:15 |
| EvilienM | k | 14:15 |
| gfidente | EvilienM so something like https://review.rdoproject.org/r/#/c/16459 ? | 14:15 |
| mwhahaha | it's the containerized -> containerized upgrade that we're doing now | 14:15 |
| gfidente | EvilienM what else? | 14:15 |
| EvilienM | gfidente: yes that's all | 14:15 |
| EvilienM | gfidente: see other patches I had to adjust | 14:15 |
| gfidente | EvilienM which other patches? | 14:15 |
| EvilienM | gfidente: in puppet-dlrn | 14:16 |
| EvilienM | mwhahaha: https://review.openstack.org/#/c/608276/ | 14:18 |
| mwhahaha | EvilienM: https://review.openstack.org/#/c/607727/ trade you | 14:18 |
| EvilienM | so demanding | 14:19 |
| mwhahaha | DO SOMETHING OF VALUE | 14:19 |
| *** dansmith is now known as SteelyDan | 14:20 | |
| EvilienM | mwhahaha, weshay_pto : also https://review.openstack.org/#/c/600517/ | 14:22 |
| *** cylopez has quit IRC | 14:22 | |
| gfidente | Tengu what email address do you use in gerrit? | 14:22 |
| EvilienM | gfidente: cjeanner@r...com | 14:23 |
| *** ykarel is now known as ykarel|afk | 14:23 | |
| *** iurygregory has quit IRC | 14:23 | |
| gfidente | EvilienM you know I am not sure what I am doing here, I'll just add the repo and pray oooq won't complain | 14:24 |
| gfidente | EvilienM oooq is an obfuscation layer | 14:24 |
| *** paramite has quit IRC | 14:25 | |
| EvilienM | gfidente: https://review.openstack.org/#/c/602973/ | 14:25 |
| EvilienM | do something like https://review.openstack.org/#/c/602973/5/config/release/tripleo-ci/promotion-testing-hash-master.yml | 14:25 |
| EvilienM | gfidente: btw the repo HAS to be mirrored by infra | 14:25 |
| *** paramite has joined #tripleo | 14:25 | |
| EvilienM | otherwise it's no go | 14:25 |
| EvilienM | I told you, making this package requires isn't the easiest option here | 14:25 |
| gfidente | EvilienM ok so there is the change to create /usr/share/ceph-ansible as a workaround | 14:26 |
| gfidente | which other option? | 14:26 |
| gfidente | EvilienM we could keep it optional to install it on the undercloud, as it used to be? | 14:29 |
| EvilienM | an option would have been to introduce enable_ceph in undercloud.conf | 14:29 |
| gfidente | and the option would have/ | 14:32 |
| gfidente | ? | 14:32 |
| EvilienM | in a mtg | 14:33 |
| *** iurygregory has joined #tripleo | 14:34 | |
| *** aufi_ has quit IRC | 14:34 | |
| *** panda|off has quit IRC | 14:36 | |
| *** panda has joined #tripleo | 14:37 | |
| *** udesale has quit IRC | 14:38 | |
| *** diw has quit IRC | 14:39 | |
| fultonj | gfidente EvilienM i see https://review.openstack.org/#/c/607982 was rechecked. i assume when it's green that we'll merge it | 14:45 |
| *** numans has quit IRC | 14:47 | |
| EvilienM | fultonj: I don't get it | 14:50 |
| EvilienM | fultonj: it's a DNM patch | 14:50 |
| fultonj | EvilienM: ok, i wasn't sure if we were going to merge and then unmerge but keeping it DNM is fine too | 14:51 |
| fultonj | gfidente: ^ fyi | 14:51 |
| fultonj | gfidente: i don't think we need to put it in Rocky as docker won't complain if the dir isn't there | 14:52 |
| fultonj | s/put it in/backport it to/ | 14:52 |
| *** moguimar has quit IRC | 14:55 | |
| bogdando | PTAL https://review.openstack.org/#/c/607514/ | 14:56 |
| bogdando | mwhahaha, EvilienM: | 14:56 |
| *** paramite has quit IRC | 14:56 | |
| EvilienM | bogdando: no -x option when cherry-picking? | 14:57 |
| bogdando | I normally do that | 14:57 |
| bogdando | missed somewhere? | 14:57 |
| EvilienM | you probably used the UI to backport before the patch was mergedi n master | 14:57 |
| mwhahaha | cause it was a revert not a cherry pick of a revert? | 14:57 |
| bogdando | oh, that's not a cherry pick | 14:57 |
| EvilienM | but too late mwhahaha approved | 14:57 |
| EvilienM | ah ok | 14:57 |
| bogdando | just a revert | 14:57 |
| EvilienM | anyway | 14:57 |
| EvilienM | it's fine | 14:57 |
| *** rbrady has joined #tripleo | 14:59 | |
| *** ksambor has quit IRC | 15:02 | |
| *** jaganathan has quit IRC | 15:06 | |
| *** jfrancoa has quit IRC | 15:07 | |
| *** mmethot_ has quit IRC | 15:08 | |
| *** med_ has joined #tripleo | 15:09 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 15:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 15:10 |
| *** mmethot has joined #tripleo | 15:10 | |
| *** bnemec is now known as beekneemech | 15:27 | |
| marios | EvilienM: can you check here when you get a chance please undercloud upgrade https://bugs.launchpad.net/tripleo/+bug/1796341 cc mwhahaha | 15:27 |
| openstack | Launchpad bug 1796341 in tripleo "The tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task" [Critical,In progress] | 15:27 |
| EvilienM | marios: it's already in gate | 15:28 |
| marios | orly? | 15:28 |
| EvilienM | marios: https://review.openstack.org/#/c/607966/ | 15:28 |
| EvilienM | since your morning | 15:28 |
| marios | EvilienM: yeah thats not the issue | 15:28 |
| EvilienM | uh | 15:29 |
| marios | EvilienM: the question is why is it still happening in master undercloud upgrade | 15:29 |
| EvilienM | ah | 15:29 |
| marios | EvilienM: since that fix merged in master like september | 15:29 |
| EvilienM | marios: we need https://review.openstack.org/#/c/608282/ | 15:29 |
| EvilienM | that's why | 15:29 |
| mwhahaha | marios: we tagged final version so package version issues | 15:29 |
| marios | EvilienM: so one theory is we are somehow reverting to using rocky templates still there | 15:29 |
| EvilienM | we pull tags from rocky | 15:29 |
| mwhahaha | let me clear the gate of stuff that's going to fail | 15:29 |
| EvilienM | yeah | 15:29 |
| marios | rlandy: fyi discussion here about that job ^^ | 15:29 |
| marios | kopecmartin|ruck: cc ^ | 15:29 |
| marios | EvilienM: looking | 15:30 |
| rlandy | following thanks | 15:30 |
| EvilienM | jtomasek, honza, jrist : we need https://review.openstack.org/#/c/608282/ | 15:30 |
| EvilienM | mwhahaha: please don't purge https://review.openstack.org/#/c/608276/ and https://review.openstack.org/#/c/608282/ | 15:30 |
| EvilienM | I'm asking if we can release stein today but I suspect they'll say no | 15:31 |
| mwhahaha | EvilienM: i'm just purging the stuff in front of https://review.openstack.org/#/c/607966/ | 15:31 |
| EvilienM | mwhahaha: good | 15:31 |
| EvilienM | if https://review.openstack.org/#/c/608282/ can't land soon, we'll need to make the containerized undercloud non voting on master maybe | 15:32 |
| EvilienM | if we have another problem than xinetd | 15:32 |
| *** ykarel|afk is now known as ykarel|ptotillwe | 15:33 | |
| *** ykarel|ptotillwe is now known as ykarel|ptotwedne | 15:34 | |
| jrist | lol I tried to type EmilienM | 15:34 |
| jrist | but it didn't autocomplete | 15:34 |
| mwhahaha | EvilienM: plz propose the non-voting | 15:34 |
| mwhahaha | we should just land that | 15:34 |
| jrist | EvilienM: I'm pretty sure that's fine | 15:34 |
| jrist | who are you asking about releasing today? | 15:34 |
| jrist | we have a ton of things that need to go into Stein | 15:34 |
| jrist | but this should be fine to start | 15:35 |
| EvilienM | mwhahaha: on master only right now right? | 15:35 |
| mwhahaha | EvilienM: yes | 15:35 |
| EvilienM | jrist: EmilienM isn't here today, his evil is replacing in the meantime | 15:36 |
| jrist | clearly | 15:36 |
| EvilienM | jrist: you can ask me for any -1 or -2, it's with pleasure I'll do it | 15:36 |
| jrist | bahahaha | 15:36 |
| *** mjturek has quit IRC | 15:37 | |
| EvilienM | nothing funny | 15:37 |
| EvilienM | but ok | 15:37 |
| *** aufi_ has joined #tripleo | 15:38 | |
| *** pcaruana has quit IRC | 15:39 | |
| EvilienM | mwhahaha, marios, rlandy : https://review.openstack.org/608291 | 15:41 |
| EvilienM | damn wrong job | 15:41 |
| chem | jistr: maybe I've missed something, do we have something tracking http://logs.openstack.org/69/605369/7/check/tripleo-ci-centos-7-undercloud-upgrades/20db73f/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-10-05_12_53_38 | 15:41 |
| EvilienM | mwhahaha: try #2 | 15:41 |
| *** mjturek has joined #tripleo | 15:41 | |
| chem | that's Evil EvilienM | 15:42 |
| mwhahaha | oh silly | 15:42 |
| jistr | chem: not yet maybe, we discussed it earlier with slagle and marios but ended up talking away from it | 15:43 |
| jistr | chem: i mean reporting a LP for it | 15:43 |
| jistr | chem: here's a fake fix https://review.openstack.org/#/c/607966/ | 15:44 |
| chem | jistr: so we still need a lp, right ? | 15:44 |
| jistr | chem: probably yes :) | 15:45 |
| marios | jistr: chem yeah https://bugs.launchpad.net/tripleo/+bug/1796341 | 15:45 |
| openstack | Launchpad bug 1796341 in tripleo "The tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task" [Critical,In progress] | 15:45 |
| jistr | chem: sorry haven't been following the most recent discussion | 15:45 |
| jistr | marios: ack thanks | 15:45 |
| chem | jistr: marios oki, so this is merged, just need recheck, thanks for the chat :) | 15:45 |
| marios | and EvilienM reckons we need https://review.openstack.org/#/c/608282/ | 15:45 |
| marios | jistr: chem ^ | 15:45 |
| marios | chem: no it isn't about the xinetd fix or recheck | 15:46 |
| marios | chem: it merged to master ages ago | 15:46 |
| jistr | chem: it's not a real fix though :) | 15:46 |
| marios | chem: question was why is master still hitting it | 15:46 |
| jistr | chem: just hiding symptoms. We chatted about this earlier in the channel http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2018-10-05.log.html | 15:46 |
| marios | 18:29 <@EvilienM> marios: we need https://review.openstack.org/#/c/608282/ | 15:47 |
| marios | 18:29 <@EvilienM> that's why | 15:47 |
| marios | 18:29 <@mwhahaha> marios: we tagged final version so package version issues | 15:47 |
| jistr | chem: http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2018-10-05.log.html#t2018-10-05T12:11:46 | 15:47 |
| jistr | marios: hm but why Stein tag matters if we should be upgrading to master, not Stein? | 15:48 |
| jistr | EvilienM: ^ | 15:48 |
| EvilienM | stein is current master no? | 15:49 |
| chem | jistr: marios CI is black magic again ... man I though I was past that :) | 15:49 |
| EvilienM | there is no black magic here : | 15:49 |
| EvilienM | the latest tag in THT is a tag for Rocky | 15:49 |
| EvilienM | so when you deploy python-tripleoclient /rocky on your system | 15:50 |
| EvilienM | try "yum update python-tripleoclient" | 15:50 |
| EvilienM | it'll do nothing | 15:50 |
| EvilienM | because there is new nvr | 15:50 |
| EvilienM | so I'm fixing this today, with a new release (first in stein), with higher tags | 15:50 |
| EvilienM | so when you'll run yum update you'll get updates for tripleo components | 15:50 |
| EvilienM | is it black magic? :) | 15:51 |
| chem | EvilienM: make senses, thanks for the clear explanation | 15:51 |
| EvilienM | cool | 15:51 |
| EvilienM | EvilienM is nice today, feeling weird | 15:51 |
| chem | EvilienM: you shouldn't talk to you like that, it's definitively weird :) | 15:51 |
| EvilienM | :-o | 15:52 |
| jistr | EvilienM: ack ok. i guess i don't understand how it works because i thought 1) we should be upgrading to dlrn-current build (always latest) rather than a specific tag and 2) the dlrn-current.repo file should provide that master content, which would be (hopefully?) newer in RPM version than the one in Rocky. But as long as someone understands how it works, we're fine :D | 15:53 |
| chem | jistr: or maybe something is wrong we repo-setup | 15:53 |
| chem | s/we/with/ | 15:53 |
| EvilienM | jistr: no you need a new NVR to upgrade | 15:54 |
| jistr | chem: yea that's what i thought (kinda still think...) | 15:54 |
| EvilienM | no matter what is in delorean | 15:54 |
| EvilienM | you need a new tag | 15:54 |
| EvilienM | that's how yum/dnf works | 15:54 |
| mwhahaha | we hit this at the start of every cycle | 15:54 |
| mwhahaha | like clockwork | 15:54 |
| EvilienM | yes :D | 15:54 |
| mwhahaha | so let's have this discussion again in 6 months | 15:54 |
| EvilienM | jistr: only one thing to remember: you can't upgrade a package if no new NVR is provided (e.g. 10.0.0 here) | 15:54 |
| EvilienM | on a fresh deployment though, it'll take master | 15:55 |
| EvilienM | because it's what is built by delorean | 15:55 |
| EvilienM | but when you upgrade an existing package, yum/dnf needs to see a new tag | 15:55 |
| EvilienM | does it makes sense? | 15:55 |
| jistr | EvilienM: yea makes sense now, thanks :) Somehow i thought Master RPMs were always +1 in version than the latest stable, i didn't realize we need to tag Stein to make sure that Master is +1 from Rocky too. | 15:56 |
| EvilienM | let's take an example : | 15:57 |
| EvilienM | https://trunk.rdoproject.org/centos7-master/current/python-tripleoclient-10.5.1-0.20181003201555.01eca26.el7.src.rpm | 15:57 |
| EvilienM | https://trunk.rdoproject.org/centos7-rocky/current/python-tripleoclient-10.6.1-0.20181004041331.5e03a6c.el7.noarch.rpm | 15:57 |
| jistr | EvilienM: yea that's bad :) | 15:57 |
| EvilienM | it's a weird example, tripleoclient in rocky gets a newer tag | 15:57 |
| EvilienM | wtf | 15:57 |
| mwhahaha | HEH | 15:58 |
| chem | that's make no sense :) | 15:58 |
| EvilienM | ok now I'm confused too | 15:58 |
| jistr | LOL | 15:58 |
| EvilienM | we can poke amoralej maybe | 15:58 |
| jistr | 17:50 <@EvilienM> is it black magic? :) | 15:59 |
| amoralej | what's the question? | 15:59 |
| EvilienM | jistr: meh | 15:59 |
| amoralej | rocky being higher that master? | 15:59 |
| EvilienM | amoralej: yeah | 15:59 |
| amoralej | like every cycle :) | 15:59 |
| amoralej | until first ta | 15:59 |
| amoralej | tag | 15:59 |
| marios | jistr: should be the chan topic? | 15:59 |
| EvilienM | oh ok | 15:59 |
| chem | we've been the full loop here :) | 15:59 |
| jistr | marios: :D | 15:59 |
| EvilienM | amoralej: https://review.openstack.org/#/c/608282/ | 16:00 |
| amoralej | yeah, i saw it coming | 16:00 |
| *** jchhatba_ has quit IRC | 16:00 | |
| *** akrivoka has quit IRC | 16:01 | |
| *** iurygregory has quit IRC | 16:02 | |
| *** rdopiera has quit IRC | 16:04 | |
| *** salmankhan has quit IRC | 16:04 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 16:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796341 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1796341 in tripleo "The tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task" [Critical,In progress] | 16:10 |
| *** cylopez has joined #tripleo | 16:11 | |
| *** jpena is now known as jpena|off | 16:12 | |
| *** itlinux has joined #tripleo | 16:12 | |
| *** derekh has quit IRC | 16:19 | |
| *** gkadam has quit IRC | 16:21 | |
| mwhahaha | bandini: https://review.openstack.org/#/c/607634/ is ok for queens right? | 16:21 |
| *** jtomasek has quit IRC | 16:21 | |
| EvilienM | bandini: in all your testing, have you already hit https://github.com/containers/buildah/issues/933 ? | 16:24 |
| *** bogdando has quit IRC | 16:26 | |
| *** social has quit IRC | 16:27 | |
| *** social has joined #tripleo | 16:27 | |
| *** boazel_ has quit IRC | 16:29 | |
| *** trown is now known as trown|lunch | 16:30 | |
| *** boazel has joined #tripleo | 16:30 | |
| *** gfidente has quit IRC | 16:37 | |
| *** aedc has joined #tripleo | 16:41 | |
| *** ansmith has quit IRC | 16:42 | |
| *** pcaruana has joined #tripleo | 16:42 | |
| *** ansmith has joined #tripleo | 16:45 | |
| *** salmankhan has joined #tripleo | 16:49 | |
| *** pcaruana has quit IRC | 16:50 | |
| *** jtomasek has joined #tripleo | 16:52 | |
| *** amoralej is now known as amoralej|off | 17:05 | |
| *** salmankhan has quit IRC | 17:09 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 17:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796341 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1796341 in tripleo "The tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task" [Critical,In progress] | 17:10 |
| *** cylopez has quit IRC | 17:12 | |
| *** dtantsur is now known as dtantsur|afk | 17:18 | |
| *** ansmith has quit IRC | 17:25 | |
| *** vinaykns has quit IRC | 17:27 | |
| *** florianf has quit IRC | 17:29 | |
| *** ansmith has joined #tripleo | 17:30 | |
| *** lblanchard has quit IRC | 17:31 | |
| *** shardy has quit IRC | 17:31 | |
| *** trown|lunch is now known as trown | 17:32 | |
| *** ykarel|ptotwedne has quit IRC | 17:33 | |
| EvilienM | dprince, mwhahaha : for early feedback https://review.openstack.org/#/c/608319/ - feel free to propose another idea | 17:35 |
| *** aedc has quit IRC | 17:36 | |
| EvilienM | it's really a poc now, so open for feedback | 17:36 |
| dprince | EvilienM: commented | 17:38 |
| dprince | EvilienM: I think you are going to need to effectively do the rm /var/lib/docker/* even if you use your patch anyway | 17:40 |
| *** holser_ has quit IRC | 17:42 | |
| *** moshele has joined #tripleo | 17:44 | |
| *** med_ has quit IRC | 17:46 | |
| *** salmankhan has joined #tripleo | 17:53 | |
| *** salmankhan has quit IRC | 17:55 | |
| *** salmankhan has joined #tripleo | 17:55 | |
| *** dsneddon has quit IRC | 17:59 | |
| bandini | mwhahaha: yes that is fine for queens | 17:59 |
| bandini | EvilienM: lemme look | 17:59 |
| bandini | EvilienM: nope I don't think we have seen that one | 18:00 |
| *** moshele has quit IRC | 18:02 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796341 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 18:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1796341 in tripleo "The tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task" [Critical,In progress] | 18:10 |
| *** vinaykns has joined #tripleo | 18:13 | |
| *** jcoufal has quit IRC | 18:18 | |
| *** jcoufal has joined #tripleo | 18:18 | |
| *** zaneb has quit IRC | 18:20 | |
| *** dprince has quit IRC | 18:20 | |
| *** zaneb has joined #tripleo | 18:20 | |
| *** slaweq has quit IRC | 18:37 | |
| *** slaweq has joined #tripleo | 18:38 | |
| *** iranzo has quit IRC | 18:56 | |
| *** ansmith has quit IRC | 18:58 | |
| *** raildo has quit IRC | 18:59 | |
| *** raildo has joined #tripleo | 18:59 | |
| *** zul has quit IRC | 19:02 | |
| *** zul has joined #tripleo | 19:03 | |
| *** aufi_ has quit IRC | 19:09 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 19:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796341 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1796341 in tripleo "The tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task" [Critical,In progress] | 19:10 |
| *** ansmith has joined #tripleo | 19:14 | |
| *** moshele has joined #tripleo | 19:20 | |
| *** khyr0n has joined #tripleo | 19:27 | |
| *** slagle has quit IRC | 19:35 | |
| *** jcoufal has quit IRC | 19:38 | |
| *** ansmith has quit IRC | 19:39 | |
| *** ansmith has joined #tripleo | 19:40 | |
| Tengu | EvilienM: hm, log collection still not merged, and won't be until we get a correction for the upgrade with xinetd merged -.-' | 19:42 |
| EvilienM | Tengu: I'll take care of that | 19:42 |
| EvilienM | Tengu: it merged | 19:42 |
| EvilienM | now we wait for dlrn build | 19:42 |
| Tengu | EvilienM: ah, good to hear then :). | 19:42 |
| Tengu | anyway, it wouldn't merge even if I didn't rebase it. sorry for that additional delay anyway :/ | 19:42 |
| Tengu | hey that means Monday should a good day for podman :D | 19:44 |
| Tengu | hopefully I'll be able to merge some of my changes for it as well. | 19:45 |
| *** morazi has quit IRC | 19:50 | |
| *** boazel has quit IRC | 19:50 | |
| *** boazel has joined #tripleo | 19:56 | |
| *** dsneddon has joined #tripleo | 19:57 | |
| *** agopi is now known as agopi|away | 19:59 | |
| *** ansmith has quit IRC | 20:03 | |
| dpeacock | I'm outta here for Thanksgiving weekend. See you Tuesday folks. :-) | 20:09 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 20:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796341 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1796341 in tripleo "The tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task" [Critical,In progress] | 20:10 |
| *** raildo has quit IRC | 20:22 | |
| *** boazel has quit IRC | 20:31 | |
| rlandy | EvilienM: https://review.openstack.org/#/c/608291/ - worth rechecking now? should tripleo-ci-centos-7-undercloud-upgrades also go to non-voting? | 20:31 |
| mwhahaha | bandini: wasn't there a bug about the mysql_init_bundle stuff not being rerun on subsequent updates? | 20:34 |
| *** boazel has joined #tripleo | 20:42 | |
| *** salmankhan has quit IRC | 20:49 | |
| *** vinaykns has quit IRC | 20:54 | |
| *** kopecmartin|ruck is now known as kopecmartin|off | 21:00 | |
| *** mjturek has quit IRC | 21:10 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 21:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796341 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 21:10 |
| openstack | Launchpad bug 1796341 in tripleo "The tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task" [Critical,In progress] | 21:10 |
| *** med_ has joined #tripleo | 21:10 | |
| *** toure is now known as toure|gone | 21:15 | |
| *** mcornea has quit IRC | 21:15 | |
| *** holser_ has joined #tripleo | 21:17 | |
| *** holser_ has quit IRC | 21:19 | |
| *** med_ has quit IRC | 21:20 | |
| *** trown is now known as trown|outtypewww | 21:21 | |
| *** itlinux has quit IRC | 21:50 | |
| *** moshele has quit IRC | 21:55 | |
| *** slagle has joined #tripleo | 22:00 | |
| *** EvilienM is now known as EmilienM | 22:06 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 22:10 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796341 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1796341 in tripleo "The tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task" [Critical,In progress] | 22:10 |
| *** itlinux has joined #tripleo | 22:19 | |
| *** itlinux has quit IRC | 22:22 | |
| *** rlandy has quit IRC | 22:24 | |
| *** artom has quit IRC | 22:54 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1792872 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1795718 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1796341 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Marios Andreou (marios-b) | 23:10 |
| openstack | Launchpad bug 1795718 in tripleo "Exception: Error inspecting image: docker://docker.io/ server misbehaving" [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1796341 in tripleo "The tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task" [Critical,In progress] | 23:10 |
| hjensas | The action raised an exception [action_ex_id=a801ad7f-1403-454e-a1f0-241e9f71e28f, action_cls='<class 'mistral.actions.action_factory.GetOvercloudConfig'>', attributes='{}', params='{u'container_config': u'overcloud-config', u'container': u'overcloud'}'] | 23:14 |
| hjensas | string longer than 2147483647 bytes | 23:14 |
| *** boazel has quit IRC | 23:17 | |
| *** rfolco has quit IRC | 23:23 | |
| *** bdodd_ has joined #tripleo | 23:34 | |
| *** bdodd has quit IRC | 23:35 | |
| *** zaneb has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!