| *** furlongm has quit IRC | 00:05 | |
| *** furlongm has joined #rdo | 00:05 | |
| *** fragatina has quit IRC | 00:26 | |
| rdogerrit | Alan Pevec created rdoinfo: Initial pin for Oslo and OpenStack clients http://review.rdoproject.org/r/1730 | 00:29 |
|---|---|---|
| *** dyasny has quit IRC | 00:30 | |
| *** limao has joined #rdo | 00:33 | |
| *** sdake has joined #rdo | 00:39 | |
| *** ayoung has joined #rdo | 00:45 | |
| *** sdake_ has joined #rdo | 00:50 | |
| *** sdake has quit IRC | 00:52 | |
| *** sdake has joined #rdo | 00:55 | |
| *** sdake_ has quit IRC | 00:55 | |
| *** dmacpher has joined #rdo | 00:57 | |
| *** ayoung has quit IRC | 01:00 | |
| *** jcoufal has joined #rdo | 01:23 | |
| *** jcoufal_ has joined #rdo | 01:28 | |
| *** jcoufal has quit IRC | 01:31 | |
| *** crazyraven has joined #rdo | 01:34 | |
| *** iranzo has quit IRC | 01:43 | |
| *** rlandy has quit IRC | 01:47 | |
| *** sdake has quit IRC | 01:59 | |
| *** jcoufal_ has quit IRC | 01:59 | |
| *** gildub has quit IRC | 01:59 | |
| *** jcoufal has joined #rdo | 02:00 | |
| *** julim has joined #rdo | 02:04 | |
| *** rhallisey has quit IRC | 02:11 | |
| *** pilasguru has quit IRC | 02:13 | |
| *** danielbruno has joined #rdo | 02:13 | |
| *** jcoufal_ has joined #rdo | 02:32 | |
| *** jistr has quit IRC | 02:32 | |
| *** jcoufal has quit IRC | 02:33 | |
| *** Liuqing has joined #rdo | 02:33 | |
| *** jistr has joined #rdo | 02:33 | |
| *** crossbuilder_ has joined #rdo | 02:35 | |
| *** crossbuilder has quit IRC | 02:35 | |
| *** jcoufal_ has quit IRC | 02:38 | |
| *** jhershbe has quit IRC | 02:44 | |
| *** Liuqing_ has joined #rdo | 02:52 | |
| *** paragan has joined #rdo | 02:55 | |
| *** Liuqing has quit IRC | 02:55 | |
| *** richm has quit IRC | 02:56 | |
| *** coolsvap has joined #rdo | 03:00 | |
| *** pilasguru has joined #rdo | 03:09 | |
| *** limao has quit IRC | 03:21 | |
| *** limao has joined #rdo | 03:25 | |
| *** pilasguru has quit IRC | 03:35 | |
| *** anilvenkata has joined #rdo | 03:39 | |
| *** vimal has joined #rdo | 03:42 | |
| *** imcleod has quit IRC | 03:45 | |
| *** Amita has joined #rdo | 03:51 | |
| *** links has joined #rdo | 04:03 | |
| *** vimal has quit IRC | 04:12 | |
| *** dtrainor has quit IRC | 04:20 | |
| *** dtrainor has joined #rdo | 04:27 | |
| *** vimal has joined #rdo | 04:33 | |
| *** pcaruana has quit IRC | 04:34 | |
| *** jtomasek has quit IRC | 04:39 | |
| *** Alex_Stef has joined #rdo | 04:40 | |
| *** gildub has joined #rdo | 04:44 | |
| *** ohochman has left #rdo | 04:49 | |
| *** abregman has joined #rdo | 04:54 | |
| *** jtomasek has joined #rdo | 04:58 | |
| *** chandankumar has joined #rdo | 04:58 | |
| *** jhershbe has joined #rdo | 05:00 | |
| *** ekuris has joined #rdo | 05:00 | |
| *** nehar has joined #rdo | 05:00 | |
| *** oshvartz has quit IRC | 05:02 | |
| *** danielbruno has quit IRC | 05:03 | |
| *** anilvenkata has quit IRC | 05:05 | |
| *** linuxaddicts has joined #rdo | 05:07 | |
| *** jubapa has quit IRC | 05:08 | |
| *** eliska has joined #rdo | 05:09 | |
| *** gildub has quit IRC | 05:14 | |
| *** rdas has joined #rdo | 05:20 | |
| *** satya4ever has joined #rdo | 05:21 | |
| *** nyechiel has joined #rdo | 05:25 | |
| *** limao has quit IRC | 05:29 | |
| *** limao has joined #rdo | 05:31 | |
| *** jtomasek has quit IRC | 05:38 | |
| *** pilasguru has joined #rdo | 05:41 | |
| *** pilasguru has quit IRC | 05:46 | |
| *** yfried has joined #rdo | 05:50 | |
| *** Liuqing_ has quit IRC | 05:57 | |
| *** Liuqing has joined #rdo | 05:59 | |
| *** Liuqing has quit IRC | 06:02 | |
| *** pgadiya has joined #rdo | 06:02 | |
| *** Liuqing has joined #rdo | 06:04 | |
| *** tshefi has joined #rdo | 06:11 | |
| *** anilvenkata has joined #rdo | 06:15 | |
| *** milan has joined #rdo | 06:15 | |
| *** jprovazn has joined #rdo | 06:16 | |
| *** leanderthal|afk is now known as leanderthal | 06:18 | |
| *** oshvartz has joined #rdo | 06:20 | |
| *** aludwar has quit IRC | 06:23 | |
| *** gildub has joined #rdo | 06:25 | |
| *** aludwar has joined #rdo | 06:26 | |
| rdogerrit | rdo-trunk created openstack/openstackclient-distgit: python-openstackclient: failed to build 0bc2d83 http://review.rdoproject.org/r/1731 | 06:28 |
| *** smeyer has joined #rdo | 06:29 | |
| *** jeckersb is now known as jeckersb_gone | 06:31 | |
| *** edannon has joined #rdo | 06:31 | |
| *** mosulica has joined #rdo | 06:31 | |
| *** Amita has quit IRC | 06:35 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-newton-current @ http://tinyurl.com/gud2vup |#| Build failure on centos7-master/current: python-openstackclient: http://trunk.rdoproject.org/centos7-master/report.html | 06:37 |
| *** anshul has joined #rdo | 06:37 | |
| *** tesseract- has joined #rdo | 06:39 | |
| *** choirboy has quit IRC | 06:42 | |
| *** jubapa has joined #rdo | 06:43 | |
| *** garrett has joined #rdo | 06:49 | |
| *** Son_Goku has quit IRC | 06:49 | |
| *** Son_Goku has joined #rdo | 06:50 | |
| *** shardy has joined #rdo | 06:52 | |
| *** pcaruana has joined #rdo | 06:55 | |
| *** florianf has joined #rdo | 07:00 | |
| *** florianf has quit IRC | 07:00 | |
| *** dmacpher has quit IRC | 07:01 | |
| *** jubapa has quit IRC | 07:04 | |
| *** nmagnezi has joined #rdo | 07:04 | |
| *** nmagnezi has quit IRC | 07:04 | |
| *** nmagnezi has joined #rdo | 07:05 | |
| *** dgurtner has joined #rdo | 07:06 | |
| *** dgurtner has joined #rdo | 07:06 | |
| *** rcernin has joined #rdo | 07:06 | |
| *** satya4ever has quit IRC | 07:06 | |
| *** jubapa has joined #rdo | 07:06 | |
| *** fzdarsky has joined #rdo | 07:06 | |
| *** gildub has quit IRC | 07:09 | |
| *** florianf has joined #rdo | 07:09 | |
| *** Amita has joined #rdo | 07:12 | |
| *** KarlchenK has joined #rdo | 07:14 | |
| *** ade_b has joined #rdo | 07:14 | |
| *** iranzo has joined #rdo | 07:20 | |
| *** iranzo has quit IRC | 07:20 | |
| *** iranzo has joined #rdo | 07:20 | |
| *** milan has quit IRC | 07:21 | |
| *** gildub has joined #rdo | 07:27 | |
| *** ifarkas_afk is now known as ifarkas | 07:28 | |
| *** jlibosva has joined #rdo | 07:34 | |
| *** satya4ever has joined #rdo | 07:34 | |
| *** mcornea has joined #rdo | 07:34 | |
| *** pnavarro has joined #rdo | 07:35 | |
| *** milan has joined #rdo | 07:35 | |
| *** jpich has joined #rdo | 07:40 | |
| *** vaneldik has quit IRC | 07:41 | |
| *** vaneldik has joined #rdo | 07:41 | |
| *** vaneldik has quit IRC | 07:46 | |
| rdogerrit | hguemar proposed openstack/openstackclient-distgit: python-openstackclient: drop ClientManager patch http://review.rdoproject.org/r/1731 | 07:51 |
| *** kaminohana has quit IRC | 07:55 | |
| *** rdas has quit IRC | 07:55 | |
| *** Son_Goku has quit IRC | 07:59 | |
| *** Son_Goku has joined #rdo | 08:00 | |
| *** ushkalim has joined #rdo | 08:01 | |
| *** vaneldik has joined #rdo | 08:05 | |
| *** milan has quit IRC | 08:05 | |
| *** jpich has quit IRC | 08:07 | |
| *** tumble has joined #rdo | 08:08 | |
| *** fragatina has joined #rdo | 08:11 | |
| *** fragatina has quit IRC | 08:11 | |
| *** fragatina has joined #rdo | 08:12 | |
| *** rdas has joined #rdo | 08:19 | |
| *** hewbrocca-afk is now known as hewbrocca | 08:21 | |
| *** abregman has quit IRC | 08:23 | |
| *** abregman has joined #rdo | 08:23 | |
| *** mvk has quit IRC | 08:24 | |
| *** egallen has joined #rdo | 08:26 | |
| *** jpich has joined #rdo | 08:27 | |
| *** derekh has joined #rdo | 08:29 | |
| *** rdas has quit IRC | 08:30 | |
| *** paramite has joined #rdo | 08:32 | |
| *** pblaho has quit IRC | 08:32 | |
| *** zoli_gone-proxy is now known as zoliXXL | 08:33 | |
| *** jtomasek has joined #rdo | 08:33 | |
| *** rdas has joined #rdo | 08:33 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-newton-current @ http://tinyurl.com/gud2vup |#| Build failure on centos7-master/current: python-openstackclient: http://trunk.rdoproject.org/centos7-master/report.html | 08:34 |
| *** pblaho has joined #rdo | 08:35 | |
| *** flepied2 is now known as flepied | 08:38 | |
| *** indy21 has joined #rdo | 08:38 | |
| *** pnavarro has quit IRC | 08:43 | |
| *** limao has quit IRC | 08:46 | |
| *** mvk has joined #rdo | 08:51 | |
| *** Goneri has joined #rdo | 08:52 | |
| *** milan has joined #rdo | 08:58 | |
| *** gildub has quit IRC | 08:59 | |
| *** hynekm has joined #rdo | 09:02 | |
| *** pnavarro has joined #rdo | 09:02 | |
| *** saneax_AFK is now known as saneax | 09:05 | |
| *** edannon_ has joined #rdo | 09:08 | |
| *** edannon has quit IRC | 09:08 | |
| *** akrivoka has joined #rdo | 09:09 | |
| *** gfidente has joined #rdo | 09:12 | |
| *** gfidente has quit IRC | 09:12 | |
| *** gfidente has joined #rdo | 09:12 | |
| *** limao has joined #rdo | 09:12 | |
| *** ihrachys has joined #rdo | 09:12 | |
| *** ihrachys has quit IRC | 09:13 | |
| *** gszasz has joined #rdo | 09:22 | |
| *** paragan has quit IRC | 09:24 | |
| *** mbound has joined #rdo | 09:25 | |
| *** nyechiel has quit IRC | 09:25 | |
| *** ihrachys has joined #rdo | 09:29 | |
| hewbrocca | dmsimard: I guess we're still stalled on the promote job? | 09:31 |
| *** DaveJ__ has joined #rdo | 09:33 | |
| *** edannon_ has quit IRC | 09:38 | |
| *** egallen_ has joined #rdo | 09:38 | |
| *** egallen has quit IRC | 09:38 | |
| *** egallen_ is now known as egallen | 09:38 | |
| number80 | hewbrocca: still too early for him :) | 09:40 |
| hewbrocca | oh, that's right -- montreal | 09:40 |
| number80 | one job is running right now | 09:40 |
| number80 | https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo-delorean-promote-master/ | 09:40 |
| hewbrocca | hoping to hit the lucky combination, I guess | 09:41 |
| hewbrocca | meh | 09:41 |
| number80 | yup | 09:41 |
| *** shaunm has quit IRC | 09:42 | |
| *** fbo has joined #rdo | 09:49 | |
| *** shaunm has joined #rdo | 09:50 | |
| *** edannon_ has joined #rdo | 09:51 | |
| *** apevec has joined #rdo | 09:52 | |
| *** mvk has quit IRC | 09:56 | |
| *** mvk has joined #rdo | 09:57 | |
| *** zoliXXL is now known as zoli|lunch | 10:05 | |
| *** Liuqing has quit IRC | 10:07 | |
| *** DV has quit IRC | 10:11 | |
| *** Goneri has quit IRC | 10:13 | |
| *** nehar has quit IRC | 10:15 | |
| *** limao has quit IRC | 10:15 | |
| *** _degorenko|afk is now known as degorenko | 10:18 | |
| *** kaminohana has joined #rdo | 10:21 | |
| *** Goneri has joined #rdo | 10:23 | |
| *** Amita has quit IRC | 10:23 | |
| *** jubapa_ has joined #rdo | 10:23 | |
| *** Amita has joined #rdo | 10:23 | |
| *** jubapa_ has quit IRC | 10:28 | |
| *** ushkalim_ has joined #rdo | 10:28 | |
| *** ushkalim__ has joined #rdo | 10:29 | |
| *** nehar has joined #rdo | 10:29 | |
| *** trozet has quit IRC | 10:29 | |
| *** tosky has joined #rdo | 10:30 | |
| *** ushkalim has quit IRC | 10:32 | |
| *** pblaho has quit IRC | 10:32 | |
| *** nyechiel has joined #rdo | 10:33 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-newton-current @ http://tinyurl.com/gud2vup |#| Build failure on centos7-master/current: python-openstackclient: http://trunk.rdoproject.org/centos7-master/report.html | 10:34 |
| *** gkadam has joined #rdo | 10:39 | |
| *** DV has joined #rdo | 10:40 | |
| *** gildub has joined #rdo | 10:44 | |
| *** DV has quit IRC | 10:45 | |
| *** pblaho has joined #rdo | 10:54 | |
| *** crazyraven has quit IRC | 10:58 | |
| apevec | weshay, trown, slagle - we'll be soon blocked on fix in tripleoclient https://bugs.launchpad.net/tripleo/+bug/1605876 | 11:02 |
| openstack | Launchpad bug 1605876 in tripleo "openstack baremetal import --json instackenv.json fails with "PluginAttributeError: _auth_params"" [Critical,Triaged] - Assigned to Brad P. Crochet (brad-9) | 11:02 |
| apevec | OSC has reverted private attributes | 11:02 |
| *** trozet has joined #rdo | 11:04 | |
| apevec | https://review.rdoproject.org/r/#/c/1731/2 | 11:04 |
| slagle | apevec: thrash|g0ne is working on it | 11:06 |
| slagle | we got a promote in tripleo-ci this morning | 11:07 |
| slagle | maybe rdo will promote before this breaks yet again | 11:07 |
| slagle | i think the problem is actually in mistralclient fwiw | 11:08 |
| apevec | ah then move LP to mistralclient | 11:09 |
| slagle | i just did | 11:09 |
| apevec | thanks | 11:09 |
| *** paragan has joined #rdo | 11:09 | |
| *** rdo has quit IRC | 11:10 | |
| rdogerrit | Merged openstack/openstackclient-distgit: python-openstackclient: drop ClientManager patch http://review.rdoproject.org/r/1731 | 11:10 |
| apevec | RDO promote is unlikely, HA job has only chance to pass on 1/4 of ci.centos machines | 11:10 |
| slagle | oh | 11:10 |
| apevec | but weshay said he'll re-run on internal HW then we could promote manually | 11:11 |
| apevec | based on his result | 11:11 |
| apevec | weshay, ^ let me know how is it going | 11:11 |
| apevec | take lastest hash where oooq minimal passed and ha failed | 11:11 |
| *** trown|outtypewww is now known as trown | 11:17 | |
| *** gildub_ has joined #rdo | 11:17 | |
| apevec | social, is there LP now for InstanceDeployFailure: Failed to provision instance f6d9453a-d9f1-4206-ac2d-97c6b495a171: Timeout reached while waiting for callback for node 82080397-4963-4aa2-ac35-5f79dabd262e ? | 11:18 |
| trown | actually looks like we had everything except minimal pass, (and that failed on the NodeNotLocked thing), I just tried a "retry" there | 11:18 |
| apevec | trown, crap :) | 11:18 |
| trown | I dont think the multijob will actually continue if it succeeds, but if it passes I am going to promote manually the next phases | 11:18 |
| *** egallen has quit IRC | 11:18 | |
| apevec | trown, see above, we have now OSC/mistralclient in consistent | 11:18 |
| apevec | so if you promote, we need one older hash | 11:19 |
| *** weshay has quit IRC | 11:19 | |
| trown | apevec: right I will promote what ran in last job, not consistent | 11:19 |
| apevec | ack | 11:19 |
| apevec | trown, what about those i-p-a timeouts? Was it a neutron bug filed? | 11:19 |
| apevec | I see that in logs | 11:20 |
| apevec | https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-minimal-411/undercloud/var/log/nova/nova-compute.log.gz | 11:20 |
| apevec | that's 25. TBD ironic/nova timeout from yesterday | 11:20 |
| trown | oh you are right it didnt fail with NodeNotLocked but the other one social was talking about | 11:22 |
| apevec | yeah, is ther LP# ? | 11:23 |
| *** nyechiel_ has joined #rdo | 11:23 | |
| trown | I dont understand what the bug or workaroud is for that | 11:23 |
| *** nyechiel has quit IRC | 11:23 | |
| apevec | social mentiond worked, /me looks back | 11:23 |
| apevec | *workaround | 11:23 |
| *** DV has joined #rdo | 11:24 | |
| *** Goneri has quit IRC | 11:24 | |
| trown | found it: [11:35:01] <social> trown: /etc/neutron/plugins/ml2/openvswitch_agent.ini ovsdb_interface = vsctl | 11:24 |
| apevec | yes, http://eavesdrop.openstack.org/irclogs/%23rdo/%23rdo.2016-07-25.log.html#t2016-07-25T15:35:01 | 11:25 |
| trown | but I dont see any explanation of what is going on and why that works around it... or if that is just a better default | 11:25 |
| apevec | but we need neutron bug# so that jlibosva ihrachys can validate it | 11:25 |
| apevec | jlibosva, ihrachys - any more on ^ last I saw from social was " issue with neutron using ovs native instead vsctl, it causes ovs to timeout and reconnect which should not disrupt anything but it drops all the connections eg dd fails for ironic" | 11:26 |
| trown | nevermind found enough to file a bug | 11:26 |
| ihrachys | wat where who why? | 11:26 |
| trown | [10:25:00] <social> trown: this is issue with neutron using ovs native instead vsctl | 11:26 |
| trown | [10:25:28] <social> trown: it causes ovs to timeout and reconnect which should not disrupt anything but it drops all the connections eg dd fails for ironic | 11:26 |
| apevec | trown, irc logs ++ :) | 11:27 |
| apevec | ihrachys, same here :) | 11:27 |
| ihrachys | apevec: please elaborate, I still to drink my coffee | 11:27 |
| apevec | we're just copy/pasting what social said, garb him :) | 11:27 |
| ihrachys | what is the workaround for? | 11:28 |
| ihrachys | what's the issue? | 11:28 |
| ihrachys | link to a bug or what? | 11:28 |
| dmsimard | apevec, number80, pabelanger: we're still having that meeting in 30 right ? I'm gonna need extra caffeine. | 11:28 |
| *** fultonj has joined #rdo | 11:29 | |
| apevec | dmsimard, I'll be late up to 30min | 11:29 |
| apevec | number80, ^ feel free to start or resched | 11:29 |
| *** gildub_ has quit IRC | 11:29 | |
| *** gildub has quit IRC | 11:29 | |
| *** gildub has joined #rdo | 11:30 | |
| ihrachys | apevec: we need a bug reported if you want anyone to look at it | 11:30 |
| dmsimard | I don't have a huge amount of time, have a pediatrician appointment 2h past the meeting beginning | 11:30 |
| *** nehar has quit IRC | 11:30 | |
| dmsimard | Figured 1h would've been enough | 11:30 |
| *** Goneri has joined #rdo | 11:31 | |
| dmsimard | apevec: think it's important that you're there | 11:31 |
| trown | apevec: ihrachys, https://bugs.launchpad.net/neutron/+bug/1534110 seems maybe related? | 11:34 |
| openstack | Launchpad bug 1534110 in neutron "OF native connection sometimes goes away and agent exits" [Medium,New] - Assigned to YAMAMOTO Takashi (yamamoto) | 11:34 |
| apevec | trown, hmm, January | 11:35 |
| *** ushkalim__ has quit IRC | 11:35 | |
| apevec | why it started hitting us only now? | 11:35 |
| *** ushkalim_ has quit IRC | 11:36 | |
| trown | oh right... k, I will just file a new bug with logs and what information we have | 11:36 |
| apevec | number80, dmsimard - prev. meeting got moved to 2pm, so I've now full overlap, please reschedule | 11:36 |
| trown | well it is racy, and the timeout in the ovs-agent logs is just an RPC timeout probably because the undercloud is very overtaxed | 11:36 |
| apevec | trown, so under-powerd CI is actually good :) | 11:37 |
| apevec | exposing all the races | 11:37 |
| *** aortega has joined #rdo | 11:37 | |
| trown | so maybe the behavior was there, but we didnt have the issue becuase the undercloud was less overwhelmed | 11:37 |
| * trown blames mistral | 11:37 | |
| *** egallen has joined #rdo | 11:38 | |
| *** lucasagomes is now known as lucas-hungry | 11:38 | |
| ihrachys | trown: could be. | 11:40 |
| *** egafford has joined #rdo | 11:40 | |
| ihrachys | apevec: because we switched to native lately | 11:40 |
| *** fzdarsky is now known as fzdarsky|lunch | 11:41 | |
| ihrachys | trown: please reuse the bug unless you clearly see it's not the same thing | 11:41 |
| ihrachys | trown: once you are done, I think we can bump the bug to High and target to N because it's now the default. | 11:42 |
| trown | ihrachys: ok, I will add to current bug... I definitely dont understand enough to know it is not the same thing | 11:42 |
| ihrachys | ajo: ^ any news on the bug ? https://bugs.launchpad.net/neutron/+bug/1534110 | 11:42 |
| openstack | Launchpad bug 1534110 in neutron "OF native connection sometimes goes away and agent exits" [Medium,New] - Assigned to YAMAMOTO Takashi (yamamoto) | 11:42 |
| ihrachys | trown: does the agent exit for you though? | 11:43 |
| ihrachys | because the bug describes the exit of the agent. | 11:43 |
| *** ade_b has quit IRC | 11:43 | |
| ihrachys | we may need to pull in jlibosva and ajo and otherwiseguy to look at the results you have. so post them first, then we'll try to get folks on board. | 11:43 |
| trown | ihrachys: ya I dont see the agent actually exit, just RPC timeout, and at the same time connection Ironic was using to dd image drops | 11:45 |
| *** jlibosva has quit IRC | 11:45 | |
| apevec | ihrachys, trown - what do you think about using social's workaround? /etc/neutron/plugins/ml2/openvswitch_agent.ini ovsdb_interface = vsctl | 11:47 |
| apevec | and where would that fit in tripleo? | 11:47 |
| trown | we would have to wire it into puppet on the undercloud | 11:47 |
| *** ushkalim_ has joined #rdo | 11:48 | |
| ihrachys | apevec: that's fine for short term but we should not ship N with it | 11:48 |
| *** ushkalim__ has joined #rdo | 11:48 | |
| ihrachys | apevec: because then we diverge from u/s | 11:48 |
| *** rdo has joined #rdo | 11:49 | |
| apevec | ack | 11:49 |
| trown | it is not a simple fix to do correctly in instack-undercloud either | 11:49 |
| apevec | yes, only as a workaround to unblock RDO promotion, we're stuck > 2 week due to various issues :( | 11:49 |
| trown | putting a hack in to quickstart would work, but I get a lot of shit for that | 11:50 |
| apevec | heh | 11:50 |
| apevec | I was about to suggest sed or crudini 1-liner :) | 11:50 |
| apevec | ok, I've updated issue 25. | 11:51 |
| *** zoli|lunch is now known as zoli | 11:51 | |
| trown | I put my understanding in https://bugs.launchpad.net/neutron/+bug/1534110 | 11:51 |
| openstack | Launchpad bug 1534110 in neutron "OF native connection sometimes goes away and agent exits" [Medium,New] - Assigned to YAMAMOTO Takashi (yamamoto) | 11:51 |
| *** zoli is now known as zoliXXL | 11:51 | |
| trown | would help to have social confirm or deny as I am pretty much just regurgitating from IRC logs | 11:51 |
| *** rhallisey has joined #rdo | 11:53 | |
| *** weshay has joined #rdo | 11:53 | |
| ajo | ihrachys, was it here where you asked me about https://bugs.launchpad.net/neutron/+bug/1534110 ? , my IRC client crashed | 11:54 |
| openstack | Launchpad bug 1534110 in neutron "OF native connection sometimes goes away and agent exits" [Medium,New] - Assigned to YAMAMOTO Takashi (yamamoto) | 11:54 |
| ajo | ihrachys, no news | 11:54 |
| *** richm has joined #rdo | 11:54 | |
| *** iberezovskiy|off is now known as iberezovskiy | 11:55 | |
| *** paramite is now known as paramite|afk | 11:55 | |
| ihrachys | ajo: seems like rdo ci hits it | 11:56 |
| *** pkovar has joined #rdo | 11:56 | |
| ajo | ihrachys, I guess it's more likely to happen now since we switched to native :/ | 11:57 |
| ihrachys | ajo: that's why it hits them | 11:57 |
| ajo | ihrachys, any idea of what's the rate? I guess we need to bump priority on the bug if we're all native now | 11:57 |
| ihrachys | ajo: which sucks because if we don't guard against timeouts, then it's not on par with prev interface | 11:57 |
| *** DV has quit IRC | 11:57 | |
| ihrachys | ajo: I said it before, yes; High and target to N | 11:58 |
| ajo | ihrachys, what's the difference betweem critical an high? | 11:58 |
| ihrachys | ajo: though trown is to check it's the same thing and report his logs | 11:58 |
| ajo | from my point of view, neutron ref impl is broken with this out in the wild | 11:58 |
| *** ade_b has joined #rdo | 11:59 | |
| ihrachys | ajo: critical blocks all development like gate breakage, high is everything else of high importance | 11:59 |
| ajo | ack | 11:59 |
| ajo | so High | 11:59 |
| ajo | I'm bumping it then | 11:59 |
| trown | ihrachys: I posted logs... they are not that interesting from ovs-agent side... just RPC timeout at the same time as Ironic connection drops | 12:00 |
| *** shardy is now known as shardy_lunch | 12:01 | |
| number80 | lol, I still managed to be the first in the meeting | 12:01 |
| ajo | trown, but RPC is different to openflow | 12:01 |
| *** imcleod has joined #rdo | 12:02 | |
| * number80 had to run under that hot weather | 12:02 | |
| ajo | trown, what you reported is something else :) | 12:02 |
| ajo | trown, doesn't it recover from that? | 12:02 |
| weshay | trown, where did the min job fail.. no tempest errors afaict | 12:02 |
| trown | ajo: right, but ironic connection is via iscsi for dd'ing image... and that drops at the same time | 12:02 |
| ihrachys | ajo: probably a timeout triggers some reconf in ryu or smth | 12:03 |
| ihrachys | so flows are broken. | 12:03 |
| ihrachys | that would be my guess for the connection flip | 12:03 |
| weshay | dam.. inventory | 12:03 |
| trown | weshay: it failed on this issue we are discussing ^ | 12:03 |
| *** nyechiel_ has quit IRC | 12:04 | |
| trown | weshay: I did retry and that is failing the same way, so I am not returning the node so I can hack on it... want to test the a proposed workaround there | 12:04 |
| social | trown: from the comment they aren't using native | 12:05 |
| *** rdas has quit IRC | 12:05 | |
| social | ajo: are you sure you don't use native in ovs agent? I could only reproduce it with native | 12:05 |
| *** rdas has joined #rdo | 12:06 | |
| trown | social: default is definitely native in my conf | 12:06 |
| social | trown: we could for now deliver patched config in neutron or make puppet to flip it | 12:07 |
| *** thrash|g0ne is now known as thrash | 12:07 | |
| number80 | apevec, dmsimard, pabelanger: meeting rescheduled 3 hours later than original schedule | 12:08 |
| social | ajo: how is the memory on the machines you hit this? do you have some link on failed jobs I could skim through? | 12:08 |
| number80 | fbo: ping stable builds | 12:09 |
| trown | social: probably meant for me? https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-minimal-411 is most recent example | 12:09 |
| trown | social: machines are definitely under heavy load | 12:09 |
| fbo | number80, pong | 12:10 |
| *** DV has joined #rdo | 12:11 | |
| dmsimard | number80: not going to be there :( | 12:11 |
| dmsimard | number80: I should be back from an appointment like an hour and a half after that | 12:12 |
| *** dpeacock has joined #rdo | 12:12 | |
| number80 | dmsimard: then, try to provide a schedule that fits Alan too | 12:14 |
| number80 | I guess today won't work | 12:16 |
| dmsimard | yeah ... | 12:16 |
| *** gema has quit IRC | 12:16 | |
| *** rdas has quit IRC | 12:16 | |
| *** toanju has joined #rdo | 12:16 | |
| *** gema has joined #rdo | 12:17 | |
| number80 | I can make an exception to my no-meeting fridays | 12:17 |
| *** Liuqing has joined #rdo | 12:18 | |
| social | trown: it's heartbeats lost in amqp? | 12:18 |
| *** danielbruno has joined #rdo | 12:19 | |
| number80 | looks like there's a nice bug in webob | 12:20 |
| *** kgiusti has joined #rdo | 12:21 | |
| social | trown: Jul 26 12:49:39 undercloud ovs-vswitchd[10666]: ovs|00043|rconn|ERR|br-ctlplane<->tcp:127.0.0.1:6633: no response to inactivity probe after 5 seconds, disconnecting | 12:21 |
| *** linuxaddicts has quit IRC | 12:22 | |
| social | trown: but the ovs exception about status report was at 10:49:32.787 in openvswitch-agent.log and in server.log I see neutron complaining about 2 dead agents at 10:49:32.719 | 12:23 |
| *** nyechiel_ has joined #rdo | 12:24 | |
| *** jlibosva has joined #rdo | 12:24 | |
| *** gildub has quit IRC | 12:26 | |
| trown | social: looking at a live CI environment now, bottleneck is much more CPU than memory... not even sure we would expect an actual user to have such resource constraints | 12:27 |
| social | jlibosva: starting 2016-07-26 10:49:28.788 in https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-minimal-411/undercloud/var/log/neutron/openvswitch-agent.log.gz | 12:27 |
| trown | might try just giving VMs more vcpus | 12:27 |
| *** ohochman has joined #rdo | 12:28 | |
| jlibosva | social: interesting I can see the native idl is used but there is no ovsdb_interface option seen in the logs | 12:29 |
| jlibosva | social: what's wrong with rpc there? | 12:30 |
| social | jlibosva: Jul 26 12:49:39 it looses connection | 12:30 |
| *** Goneri has quit IRC | 12:30 | |
| jlibosva | social: because of the same reason on flat network? | 12:31 |
| social | jlibosva: yep the ovs thing | 12:31 |
| jlibosva | social: have you tried to run it several time with vsctl interface to be sure it's caused by native interface? | 12:31 |
| social | jlibosva: in my case vsctl didn't fail not sure if that helps | 12:33 |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-newton-current @ http://tinyurl.com/gud2vup |#| Build failure on centos7-master/current: python-openstackclient: http://trunk.rdoproject.org/centos7-master/report.html | 12:34 |
| jlibosva | social: it would be good to be sure it wasn't a coincidence, you tried it just once and native fails sporadically, right? | 12:34 |
| social | jlibosva: I tried vsctl 4 times and it didn't fail, native fails obviously only under heavy load | 12:34 |
| jlibosva | social: ok, I thought it was only once | 12:34 |
| *** julim has quit IRC | 12:35 | |
| *** rodrigods has quit IRC | 12:35 | |
| *** rodrigods has joined #rdo | 12:35 | |
| *** ohochman has left #rdo | 12:35 | |
| *** lucas-hungry is now known as lucasagomes | 12:39 | |
| *** jcoufal has joined #rdo | 12:40 | |
| *** rlandy has joined #rdo | 12:42 | |
| rdobot | [sensu] RESOLVED: master.monitoring.rdoproject.org - check-delorean-newton-current @ http://tinyurl.com/gud2vup |#| No build failures detected: http://trunk.rdoproject.org/centos7-master/report.html | 12:45 |
| *** jeckersb_gone is now known as jeckersb | 12:46 | |
| *** Goneri has joined #rdo | 12:48 | |
| *** akshai has joined #rdo | 12:50 | |
| *** yfried has quit IRC | 12:50 | |
| social | jlibosva: just reproduced it locally again with native | 12:50 |
| jlibosva | social: coming | 12:50 |
| * social hides under table | 12:51 | |
| *** paramite|afk is now known as paramite | 12:51 | |
| *** morazi has joined #rdo | 12:51 | |
| *** vimal has quit IRC | 12:51 | |
| weshay | https://review.rdoproject.org/etherpad/p/rdo-infra-scrum | 12:56 |
| weshay | in 4 min | 12:56 |
| chandankumar | apevec, https://bugzilla.redhat.com/show_bug.cgi?id=1318765 and https://bugzilla.redhat.com/show_bug.cgi?id=1355886 any thing more to add on this | 12:57 |
| openstack | bugzilla.redhat.com bug 1318765 in Package Review "Review Request: openstack-sahara-tests - Sahara Scenario Test Framework" [Unspecified,Assigned] - Assigned to apevec | 12:57 |
| openstack | bugzilla.redhat.com bug 1355886 in Package Review "python-horizon-tests-tempest : Tempest Integration of Horizon" [Unspecified,Assigned] - Assigned to apevec | 12:57 |
| *** dmsimard is now known as dmsimard|afk | 12:58 | |
| *** eliska has quit IRC | 12:58 | |
| *** linuxaddicts has joined #rdo | 12:59 | |
| *** severion has joined #rdo | 13:00 | |
| *** choirboy has joined #rdo | 13:00 | |
| *** _elmiko is now known as elmiko | 13:01 | |
| weshay | trown, you avail? | 13:01 |
| weshay | panda, you are welcome as well | 13:02 |
| trown | coming | 13:02 |
| *** amuller has joined #rdo | 13:02 | |
| *** linuxgeek_ has joined #rdo | 13:03 | |
| rdogerrit | Fabien Boucher proposed config: Initial commit to activate the stable branch build on CBS http://review.rdoproject.org/r/1727 | 13:03 |
| *** unclemarc has joined #rdo | 13:04 | |
| *** Liuqing has quit IRC | 13:06 | |
| *** linuxaddicts has quit IRC | 13:07 | |
| weshay | bkero, mtg | 13:07 |
| weshay | https://redhat.bluejeans.com/u/whayutin/ | 13:07 |
| *** Liuqing has joined #rdo | 13:08 | |
| rdogerrit | Merged config: Initial commit to activate the stable branch build on CBS http://review.rdoproject.org/r/1727 | 13:09 |
| *** chandankumar has quit IRC | 13:10 | |
| social | jlibosva: I think you want to stop by | 13:10 |
| *** alexismonville has quit IRC | 13:10 | |
| social | jlibosva: it failed and recovered ,) | 13:10 |
| *** alexismonville has joined #rdo | 13:10 | |
| social | we have "and now for something completly different" moment | 13:10 |
| jlibosva | social: coming | 13:11 |
| *** mlammon has joined #rdo | 13:12 | |
| *** Son_Goku has quit IRC | 13:12 | |
| *** Son_Goku has joined #rdo | 13:13 | |
| *** dustins has joined #rdo | 13:14 | |
| *** Liuqing has quit IRC | 13:14 | |
| *** eliska has joined #rdo | 13:14 | |
| *** shardy_lunch is now known as shardy | 13:15 | |
| *** Liuqing has joined #rdo | 13:15 | |
| *** nyechiel_ has quit IRC | 13:15 | |
| social | jlibosva: I'll ping you in ~20 min with env ready again | 13:15 |
| *** hynekm has quit IRC | 13:16 | |
| *** rdo has quit IRC | 13:18 | |
| *** Liuqing has quit IRC | 13:19 | |
| *** jeckersb is now known as jeckersb_gone | 13:20 | |
| jlibosva | social: acknowledged | 13:20 |
| *** jeckersb_gone is now known as jeckersb | 13:24 | |
| *** julim has joined #rdo | 13:24 | |
| *** nmagnezi has quit IRC | 13:26 | |
| *** rook has quit IRC | 13:27 | |
| rdogerrit | Fabien Boucher created config: Add newton-distgit-cbs-validate job and activate it for newtow-rdo branch http://review.rdoproject.org/r/1732 | 13:29 |
| *** pgadiya has quit IRC | 13:29 | |
| social | win 12 | 13:32 |
| *** Son_Goku has quit IRC | 13:32 | |
| pabelanger | apevec: number80: dmsimard|afk: apologies for not making the meeting. Looks like it's in 90mins now? | 13:34 |
| larsks | dmsimard|afk: did you get sorted out with your docker exec question yesterday? I didn't see it until late last night... | 13:34 |
| *** egonzalez has joined #rdo | 13:35 | |
| number80 | pabelanger: well, dmsimard|afk can't | 13:36 |
| number80 | so we have to figure out a better timeslot this week | 13:36 |
| *** fzdarsky|lunch is now known as fzdarsky | 13:36 | |
| * number80 is going to Poland next week and apevec will be on PTO after that | 13:36 | |
| number80 | as both of you have kids, I suppose that you'll take PTO in august too | 13:37 |
| *** hynekm has joined #rdo | 13:39 | |
| *** eliska has quit IRC | 13:40 | |
| rdogerrit | hguemar proposed openstack/novaclient-distgit: Added py2 and py3 subpackage http://review.rdoproject.org/r/1618 | 13:44 |
| *** links has quit IRC | 13:44 | |
| *** joelinux73 has joined #rdo | 13:44 | |
| rdogerrit | Fabien Boucher proposed config: Add newton-distgit-cbs-validate job and activate it for newton-rdo branch http://review.rdoproject.org/r/1732 | 13:47 |
| *** ayoung has joined #rdo | 13:48 | |
| *** dyasny has joined #rdo | 13:50 | |
| hewbrocca | kbsingh: hey, did you have any more inspiration about what might be screwing us up on the non-Intel chassis? | 13:51 |
| *** dyasny has quit IRC | 13:51 | |
| apevec | pabelanger, what time would work for you later today? | 13:51 |
| hewbrocca | I don't know where we left that yesterday | 13:51 |
| *** dyasny has joined #rdo | 13:51 | |
| apevec | number80, pabelanger - dmsimard|afk is back after 16:30 UTC | 13:51 |
| number80 | UTC ? | 13:52 |
| number80 | I thought it was EDT | 13:52 |
| apevec | number80, that's what he said, lemme check | 13:52 |
| apevec | I should be back around 4:30PM UTC (12:30PM EST). | 13:52 |
| apevec | number80, reschedule to that slot | 13:53 |
| *** eliska has joined #rdo | 13:53 | |
| apevec | then we'll see if that works out | 13:53 |
| number80 | wfm | 13:53 |
| number80 | it should be quick | 13:53 |
| kbsingh | hewbrocca: on the machine side, the only key metric that might be relevant here is the per-code compute capacity on the amd chassis is 60% of what it is on the intel machines | 13:53 |
| apevec | hewbrocca, trown might have a solution: https://review.openstack.org/347371 | 13:53 |
| kbsingh | hewbrocca: but since the core count on the AMD machines is twice that of the intel ones - overall machine capacity is about the same | 13:53 |
| kbsingh | if we can get to using more cores | 13:54 |
| apevec | kbsingh, yep, that's what trown's patch is doing | 13:54 |
| hewbrocca | kbsingh: I see, so each blade is less beefy, but there are more blades | 13:54 |
| hewbrocca | OK, more cores | 13:54 |
| *** jmelvin has joined #rdo | 13:54 | |
| *** jcoufal has quit IRC | 13:54 | |
| kbsingh | apevec: that looks good! | 13:54 |
| hewbrocca | oh, wow | 13:54 |
| hewbrocca | Are we trying it? | 13:55 |
| *** gszasz has quit IRC | 13:55 | |
| trown | hewbrocca: ya I confirmed it worked on a CI node that failed without it | 13:55 |
| hewbrocca | ROCK | 13:55 |
| kbsingh | \o/ | 13:55 |
| hewbrocca | trown: good man | 13:55 |
| trown | hewbrocca: running CI on it to make sure we dont break liberty/mitaka | 13:55 |
| trown | because more CPU = more workers = more memory... and we had memory issues in the past | 13:55 |
| hewbrocca | right | 13:55 |
| *** gszasz has joined #rdo | 13:55 | |
| trown | thanks kbsingh for the analysis on the AMD CPUs that helped me know how to fix it | 13:56 |
| kbsingh | there are 2 more things that i am going to work on that should help - (1) bstinson is going to work with facilities to see if we can move ram around and bring up another 64 intel machines with 32gb ram ( we have 64 of these, sitting with 16gb, which is no use ) | 13:56 |
| hewbrocca | kbsingh: that would be magnificent | 13:56 |
| kbsingh | and (2) i am going to work on changing the fair-spread logic of machine allocation towards a more 'best-available' | 13:56 |
| *** dneary has joined #rdo | 13:57 | |
| kbsingh | that way, we cant be sure ( atleast the user cant specify a pool per se, but we get to always hand out intel > amd ; and for jobs that need multiple machines we always handout nodes that have 10gb fabric locally | 13:57 |
| trown | neat | 13:57 |
| *** ekuris has quit IRC | 13:58 | |
| *** hynekm has quit IRC | 14:03 | |
| *** rdo has joined #rdo | 14:03 | |
| *** nyechiel_ has joined #rdo | 14:04 | |
| *** jcoufal has joined #rdo | 14:06 | |
| *** kaminohana has quit IRC | 14:11 | |
| *** Amita has quit IRC | 14:12 | |
| *** limao has joined #rdo | 14:12 | |
| *** limao_ has joined #rdo | 14:13 | |
| *** sdake has joined #rdo | 14:14 | |
| *** limao has quit IRC | 14:16 | |
| *** dustins has quit IRC | 14:17 | |
| *** sdake_ has joined #rdo | 14:18 | |
| *** mengxd has joined #rdo | 14:20 | |
| *** sdake has quit IRC | 14:20 | |
| number80 | didn't saw that we merged https://review.rdoproject.org/r/#/c/1685/ \o/ | 14:23 |
| *** dustins has joined #rdo | 14:23 | |
| *** laron has joined #rdo | 14:23 | |
| * number80 now can build dependencies slightly faster | 14:23 | |
| *** jdob has quit IRC | 14:25 | |
| *** limao_ has quit IRC | 14:26 | |
| *** anshul has quit IRC | 14:26 | |
| *** anshul has joined #rdo | 14:27 | |
| *** jdob has joined #rdo | 14:27 | |
| jruzicka | nice hack :) | 14:28 |
| jruzicka | err, solution | 14:28 |
| *** rdo has quit IRC | 14:28 | |
| *** tosky has quit IRC | 14:30 | |
| *** eliska has quit IRC | 14:30 | |
| number80 | jruzicka: it's a hack, that's why I never dared to merge it myself :) | 14:34 |
| *** Liuqing has joined #rdo | 14:34 | |
| *** egallen has quit IRC | 14:34 | |
| *** KarlchenK has quit IRC | 14:38 | |
| *** pabelanger has quit IRC | 14:39 | |
| *** pabelanger has joined #rdo | 14:39 | |
| ihrachys | jlibosva: you are on timeout thing right? | 14:39 |
| *** pnavarro has quit IRC | 14:40 | |
| jlibosva | ihrachys: partially, I'm working with social, he can reproduce it locally | 14:40 |
| apevec | trown, I'll merge 23. under 25. ack? https://etherpad.openstack.org/p/delorean_master_current_issues | 14:41 |
| trown | ya, they are similar... my patch for command_timeout did actually help HA job :) | 14:41 |
| *** zhenguo has quit IRC | 14:41 | |
| trown | it just failed on deploy timeout of 90 minutes though after that | 14:42 |
| jlibosva | ihrachys: if you want an update, it's because of ovs interfaces switch. since we switched both in the short period, we don't know which cause it or whether it's the combination | 14:42 |
| social | jlibosva: I forgot to ping you | 14:42 |
| ihrachys | jlibosva: I just wanted to make sure it has not slipped thru the cracks | 14:42 |
| jlibosva | social: I came after 20 minutes but your seat was empty | 14:42 |
| social | jlibosva: yeah I'm here and env is ready | 14:43 |
| social | I should have gave you shell | 14:43 |
| jlibosva | ihrachys: we're on it, people already spotted they can't install openstack | 14:43 |
| jlibosva | s/openstack/rdo/ | 14:43 |
| ihrachys | ok cool. visibility :P | 14:43 |
| jlibosva | social: I'll give you my brand new old public key | 14:44 |
| *** zoliXXL is now known as zoli|brb | 14:50 | |
| *** aderyugin has quit IRC | 14:52 | |
| *** aderyugin has joined #rdo | 14:53 | |
| *** jokke_ has quit IRC | 14:54 | |
| *** aderyugin has quit IRC | 14:54 | |
| *** zoli|brb is now known as zoli | 14:54 | |
| *** zoli is now known as zoliXXL | 14:54 | |
| *** aderyugin has joined #rdo | 14:54 | |
| *** edannon_ has quit IRC | 14:54 | |
| *** number80 has quit IRC | 14:55 | |
| *** aderyugin has quit IRC | 14:55 | |
| *** rdo has joined #rdo | 14:57 | |
| *** toanju has quit IRC | 14:59 | |
| *** aderyugin has joined #rdo | 15:00 | |
| *** jhershbe has quit IRC | 15:00 | |
| *** coolsvap is now known as coolsvap_ | 15:01 | |
| *** rdo has quit IRC | 15:03 | |
| *** laron has quit IRC | 15:03 | |
| *** laron has joined #rdo | 15:04 | |
| *** rcernin has quit IRC | 15:05 | |
| *** mbound has quit IRC | 15:07 | |
| *** anshul has quit IRC | 15:12 | |
| *** satya4ever has quit IRC | 15:14 | |
| *** nyechiel has joined #rdo | 15:18 | |
| *** laron has quit IRC | 15:19 | |
| *** metabsd has quit IRC | 15:19 | |
| *** nyechiel_ has quit IRC | 15:19 | |
| *** READ10 has joined #rdo | 15:21 | |
| *** gkadam has quit IRC | 15:21 | |
| *** aderyugin has quit IRC | 15:24 | |
| *** aderyugin has joined #rdo | 15:24 | |
| *** tosky has joined #rdo | 15:25 | |
| *** nmagnezi has joined #rdo | 15:25 | |
| *** saneax is now known as saneax_AFK | 15:29 | |
| *** ade_b has quit IRC | 15:29 | |
| *** Liuqing has quit IRC | 15:29 | |
| *** leanderthal is now known as leanderthal|afk | 15:30 | |
| *** rdo has joined #rdo | 15:30 | |
| *** nmagnezi has quit IRC | 15:31 | |
| *** linuxgeek_ has quit IRC | 15:32 | |
| *** tshefi has quit IRC | 15:33 | |
| *** smeyer has quit IRC | 15:37 | |
| *** linuxaddicts has joined #rdo | 15:39 | |
| *** ifarkas is now known as ifarkas_afk | 15:40 | |
| apevec | weshay, myoung, trown - just to double-check, you're re-testing internally this hash? https://ci.centos.org/job/rdo-promote-get-hash-master/536/console | 15:40 |
| weshay | apevec, aye | 15:41 |
| apevec | that's the one where, of all jobs!, only minimal failed | 15:41 |
| *** garrett has quit IRC | 15:41 | |
| apevec | cool | 15:41 |
| apevec | yeah, ha job was lucky and hit dusty | 15:42 |
| trown | apevec: ya, I actually disable promote, and put that hash manually in get hash job | 15:42 |
| trown | apevec: waiting on CPU patch to pass CI, then will rerun promote with hash from 536 | 15:42 |
| *** akrivoka has quit IRC | 15:43 | |
| rdogerrit | Frederic Lepied proposed rdoinfo: Initial pin for Oslo and OpenStack clients http://review.rdoproject.org/r/1730 | 15:43 |
| trown | apevec: we can also skip image building in that run, since we already have an image for that hash | 15:43 |
| trown | so maybe after lunch we have a promote | 15:43 |
| *** tesseract- has quit IRC | 15:43 | |
| apevec | indeed | 15:43 |
| apevec | right, your patch doesn't affect image building | 15:43 |
| trown | yep, just undercloud vm | 15:44 |
| *** number80 has joined #rdo | 15:45 | |
| *** weshay is now known as weshay_brb | 15:45 | |
| *** abregman has quit IRC | 15:49 | |
| number80 | seems that CBS is stalled | 15:49 |
| number80 | oh no, just SCL jamming the builders | 15:50 |
| number80 | https://cbs.centos.org/koji/tasks | 15:50 |
| number80 | One good reason to not like koschei | 15:50 |
| number80 | (koschei is generic DLRN-like continuous delivery that uses koji) | 15:51 |
| apevec | number80, oh, SCL SIG uses that? | 15:54 |
| *** panda is now known as panda|bbl | 15:55 | |
| apevec | number80, flepied - I think we can now merge both https://review.rdoproject.org/r/#/q/topic:newton-uc | 15:55 |
| apevec | as a starting point, so we can setup new dlrn instance | 15:55 |
| apevec | I'll send update-uc script in followup | 15:56 |
| flepied | apevec: yes but we need to fix the script to insert 'source-branch' | 15:56 |
| apevec | with improved projects for filter | 15:56 |
| apevec | flepied, yep, that too | 15:56 |
| *** weshay_brb is now known as weshay | 15:56 | |
| apevec | I'll re-run it and it should not make changes to your patchset | 15:57 |
| flepied | apevec: yep | 15:57 |
| *** indy21 has quit IRC | 15:58 | |
| *** mosulica has quit IRC | 15:58 | |
| trown | ok promote kicked now with twice the cpus... https://ci.centos.org/view/rdo/view/promotion-pipeline/ getting lunch | 15:59 |
| *** laron has joined #rdo | 15:59 | |
| *** trown is now known as trown|lunch | 15:59 | |
| hewbrocca | woop woop | 15:59 |
| hewbrocca | nice work trown|lunch | 15:59 |
| *** mengxd has quit IRC | 16:01 | |
| weshay | apevec, did I loose rights to add repos to redhat-openstack? | 16:04 |
| *** pgadiya has joined #rdo | 16:04 | |
| *** pgadiya has quit IRC | 16:05 | |
| apevec | you shouldn't have, lemme check | 16:05 |
| apevec | weshay, this guy https://github.com/weshayutin is Owner | 16:06 |
| apevec | not sure who is under helment... | 16:06 |
| *** tumble has quit IRC | 16:06 | |
| *** Amita has joined #rdo | 16:06 | |
| *** oshvartz has quit IRC | 16:07 | |
| weshay | apevec, :) | 16:07 |
| weshay | lolz | 16:07 |
| weshay | weird.. github changed something when forking.. | 16:07 |
| *** nmagnezi has joined #rdo | 16:08 | |
| *** mbound has joined #rdo | 16:08 | |
| *** anilvenkata has quit IRC | 16:08 | |
| *** nyechiel has quit IRC | 16:09 | |
| apevec | speaking of forks, f25 was branched today | 16:09 |
| apevec | number80, jruzicka ^ when is py3 deadline for Fedora? | 16:10 |
| apevec | number80, oh, Flock is next week? | 16:11 |
| number80 | apevec: I haven't been nagged again but I'm nearly finished | 16:12 |
| jruzicka | number80++ | 16:13 |
| number80 | currently, I'm waiting CBS resources to be freed so that I can close it all | 16:13 |
| *** mbound has quit IRC | 16:13 | |
| number80 | tripleoclient/tripleocommon are the only exception but that needs more work and they're not in fedora | 16:14 |
| jruzicka | ooo always is something extra | 16:14 |
| number80 | I'm not commenting that :) | 16:14 |
| jlibosva | ihrachys: then again, which timeout did you mean couple of hrs back? Just making sure it's not a different issue | 16:14 |
| number80 | I need more coffee | 16:15 |
| ihrachys | jlibosva: my understanding was that a rpc timeout in agent triggered something in ryu that made connectivity blink | 16:15 |
| jlibosva | ihrachys: ack, that sounds like it | 16:16 |
| *** richm has quit IRC | 16:16 | |
| jlibosva | ihrachys: do you know if it happens on jenkins jobs? | 16:16 |
| *** richm has joined #rdo | 16:16 | |
| ihrachys | jlibosva: that's all I knew | 16:16 |
| kbsingh | number80: i highly encourage you to report on this with a 'can we increase capacity' for CBS | 16:17 |
| apevec | kbsingh, is there bug# for that? | 16:18 |
| kbsingh | number80: we have the ability to increase build capacity in CBS almost 300% if needed | 16:18 |
| kbsingh | apevec: you should be good to go ahead and file one | 16:18 |
| kbsingh | cc: bstinson :) | 16:18 |
| apevec | I'll let number80 file one for CBS | 16:19 |
| number80 | kbsingh: well, that's one of the rare occasion I'm waiting, during the w-e, I have as many builders as I want :) | 16:19 |
| kbsingh | ( just going by the fact that number80 said things were hectic almost an hour back, and from the looks of it, thre are still 70+ jobs in the queue, we should be ok to add more capacity there ) | 16:19 |
| number80 | 300% good | 16:19 |
| number80 | I won't complain to have more builders available :) | 16:19 |
| kbsingh | we have 2 machines there now ( bstinson might correct me ) - and we can add another 6 ( 2 more or less now, 4 more would mean moving things around ) | 16:20 |
| number80 | well, let's be future-proof and expect more usage of CBS :) | 16:21 |
| *** hrw has quit IRC | 16:21 | |
| *** vaneldik has quit IRC | 16:22 | |
| *** hrw has joined #rdo | 16:22 | |
| *** zoliXXL is now known as zoli|gone | 16:23 | |
| kbsingh | number80: go ahead and file for more resources, we might not treat it as urgent for now since you think its not urgent, but we can get some more builders added in the next few weeks for sure | 16:24 |
| kbsingh | did trown|lunch's work to spread load a bit, mean we got a promote ? | 16:24 |
| kbsingh | ( or is that still running ) | 16:25 |
| hewbrocca | kbsingh: it's running now | 16:25 |
| hewbrocca | https://ci.centos.org/view/rdo/view/promotion-pipeline/ | 16:25 |
| kbsingh | woo | 16:25 |
| apevec | damn, one packstack job failed | 16:25 |
| apevec | we'll need to re-run it | 16:26 |
| *** ushkalim__ has quit IRC | 16:26 | |
| *** ushkalim_ has quit IRC | 16:26 | |
| apevec | dmsimard|afk, when you're back https://ci.centos.org/job/weirdo-master-promote-packstack-scenario001/409/ | 16:26 |
| *** pilasguru has joined #rdo | 16:27 | |
| *** zoli|gone is now known as zoli_gone-proxy | 16:28 | |
| apevec | hmm, cirros image | 16:28 |
| apevec | but not during create i.e. d/l error | 16:28 |
| *** jlibosva has quit IRC | 16:29 | |
| imcsk8 | apevec: i'll take a look also | 16:30 |
| *** derekh has quit IRC | 16:30 | |
| apevec | imcsk8, looks like a race in Packstack::Provision ? | 16:31 |
| *** pilasguru has quit IRC | 16:32 | |
| apevec | image create followed by image set | 16:32 |
| apevec | it kept retrying but it wasn't enough | 16:33 |
| apevec | https://ci.centos.org/artifacts/rdo/weirdo-master-promote-packstack-scenario001/409/packstack/logs/latest/manifests/172.19.2.94_controller.pp.log.txt.gz | 16:33 |
| *** spr1 has joined #rdo | 16:34 | |
| *** jpich has quit IRC | 16:35 | |
| *** mcornea has quit IRC | 16:35 | |
| *** rwsu has joined #rdo | 16:37 | |
| *** chem has quit IRC | 16:37 | |
| apevec | imcsk8, also what dmsimard|afk and I discussed yesterday, we could use cached cirros in packstack jobs, like puppet jobs: https://github.com/openstack/puppet-openstack-integration/blob/6c7cb5a39d2b54a222605d9caf9973e8059e34e5/run_tests.sh#L153 | 16:39 |
| number80 | kbsingh: done https://bugs.centos.org/view.php?id=11223 I can't change assignee of add bstinson as CC though | 16:40 |
| * number80 only has ticket creation + commenting perms in mantis | 16:40 | |
| *** jhershbe has joined #rdo | 16:40 | |
| *** florianf has quit IRC | 16:42 | |
| *** lucasagomes is now known as lucas-dinner | 16:44 | |
| apevec | imcsk8, hmm, this is all within glance_image ? | 16:45 |
| imcsk8 | apevec: that sounds cool | 16:45 |
| apevec | that's what packstack is using right? | 16:45 |
| *** chandankumar has joined #rdo | 16:45 | |
| apevec | number80, getting f25 spam from f25 too? | 16:46 |
| apevec | err from pkgdb | 16:46 |
| *** danielbruno has quit IRC | 16:46 | |
| imcsk8 | apevec: packstack uses this one: https://github.com/openstack/packstack/blob/master/packstack/plugins/provision_700.py#L33 | 16:46 |
| * apevec needs to tune his fedora-notifs | 16:46 | |
| *** dmsimard|afk is now known as dmsimard | 16:47 | |
| apevec | imcsk8, yep, we'd need to check if that works with file:// | 16:47 |
| apevec | but also, this race should be solved in puppet-glance | 16:48 |
| number80 | apevec: not yet but I suspect that fedmsg is lagging | 16:48 |
| EmilienM | what race? | 16:48 |
| EmilienM | is there a launchpad somewhere? | 16:48 |
| imcsk8 | apevec: i'll check, it does not seem very problematic | 16:48 |
| apevec | EmilienM, https://ci.centos.org/artifacts/rdo/weirdo-master-promote-packstack-scenario001/409/packstack/logs/latest/manifests/172.19.2.94_controller.pp.log.txt.gz | 16:48 |
| apevec | EmilienM, not yet | 16:48 |
| apevec | image set after image create | 16:49 |
| apevec | relies on retries | 16:49 |
| apevec | and we could mask it by using locally cached cirros image I guess | 16:49 |
| *** ccamacho is now known as ccamacho|awawawa | 16:50 | |
| *** ccamacho|awawawa is now known as ccamacho|away | 16:50 | |
| dmsimard | apevec, pabelanger, number80: I'm back | 16:50 |
| apevec | but right now it just failed in rdo promotion https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo-delorean-promote-master/539/ | 16:50 |
| dmsimard | apevec: yeah the cirros image is starting to be a bit troublesome | 16:50 |
| apevec | dmsimard, welcome back! | 16:50 |
| dmsimard | apevec: I saw it as well | 16:51 |
| apevec | dmsimard, it will be fun it oooq pass in that run :) | 16:51 |
| apevec | it -> if | 16:51 |
| dmsimard | apevec: we can definitely cache it in review.rdo but I don't see how we could cache it in ci.centos since they're "virgin installs" | 16:51 |
| dmsimard | apevec: I wonder if the cloud SIG could host it | 16:51 |
| number80 | dmsimard, apevec, pabelanger: then ready for a quick call? | 16:52 |
| apevec | hmm, we could package it as RPM ? | 16:52 |
| apevec | number80, let's | 16:52 |
| dmsimard | apevec: not a crazy idea .. it's not that big | 16:52 |
| kbsingh | if you need to pre-seed content, you can host it on the artifacts machine, then rsync it down before kicking stuff off | 16:52 |
| kbsingh | humm the entire image as an rpm ? | 16:52 |
| dmsimard | kbsingh: cirros is pretty small | 16:52 |
| dmsimard | smaller than the kernel-dev packages :p | 16:53 |
| kbsingh | have you guys seen centos-min ? i believe its down to 70MB | 16:53 |
| apevec | kbsingh, url? | 16:53 |
| pabelanger | number80: sure | 16:53 |
| apevec | sounds interesting | 16:53 |
| number80 | kbsingh: that's awesome | 16:53 |
| number80 | (I'm saying that as a former fedora cloud guy) | 16:53 |
| *** jhershbe has quit IRC | 16:53 | |
| dmsimard | number80, pabelanger, apevec: ok let's do the meeting then, meet you in the BJ in 1 min | 16:54 |
| *** fragatina has quit IRC | 16:54 | |
| *** pilasguru has joined #rdo | 16:54 | |
| kbsingh | apevec: https://github.com/cgwalters/centos-dockerbase-minimal is a good place to start from | 16:54 |
| kbsingh | apevec: we can trick it down even further, just need to be careful what we call it | 16:54 |
| number80 | ah , it's using cgwalters libhif yum-min clone | 16:54 |
| number80 | if you remove yum and cloud-init, you can win a lot of space | 16:55 |
| kbsingh | clone as in, it does 'yum install' | 16:55 |
| *** rdas has joined #rdo | 16:55 | |
| number80 | yep | 16:55 |
| number80 | well, for CI usage, you don't need much more | 16:55 |
| pabelanger | dmsimard: URL? | 16:56 |
| dmsimard | pabelanger: PM | 16:56 |
| *** mvk has quit IRC | 16:57 | |
| *** fragatina has joined #rdo | 17:00 | |
| *** nmagnezi has quit IRC | 17:01 | |
| *** hewbrocca is now known as hewbrocca-afk | 17:04 | |
| *** fzdarsky is now known as fzdarsky|afk | 17:06 | |
| *** paramite has quit IRC | 17:06 | |
| *** fragatina has quit IRC | 17:07 | |
| *** laron has quit IRC | 17:09 | |
| *** jtomasek has quit IRC | 17:10 | |
| *** apevec_ has joined #rdo | 17:12 | |
| *** shardy has quit IRC | 17:13 | |
| *** dsneddon has quit IRC | 17:13 | |
| *** dsneddon has joined #rdo | 17:13 | |
| *** trown|lunch is now known as trown | 17:14 | |
| *** apevec has quit IRC | 17:16 | |
| *** metabsd has joined #rdo | 17:17 | |
| metabsd | Hi :) I just install a RDO packstack --allinone. Now I want to add more storage space to deploy a VM ou Image. Help! | 17:18 |
| *** nehar has joined #rdo | 17:19 | |
| *** fzdarsky|afk has quit IRC | 17:26 | |
| *** milan has quit IRC | 17:28 | |
| larsks | metabsd: generally, that just means increasing the space available to whatever contains /var/lib/nova/instances (or mounting a new filesystem at that location). | 17:29 |
| *** READ10 has quit IRC | 17:30 | |
| *** rdas has quit IRC | 17:30 | |
| *** jhershbe has joined #rdo | 17:34 | |
| *** vaneldik has joined #rdo | 17:35 | |
| *** vaneldik has left #rdo | 17:36 | |
| *** pkovar has quit IRC | 17:37 | |
| *** sdake_ is now known as sdake | 17:37 | |
| EmilienM | apevec_: new error on trunk http://paste.openstack.org/show/ilg5IMSvyUycj3JwbdFb/ | 17:38 |
| *** spr1 has quit IRC | 17:38 | |
| *** spr1 has joined #rdo | 17:38 | |
| *** nehar has quit IRC | 17:40 | |
| apevec_ | nodejs ? | 17:41 |
| apevec_ | @epel ? | 17:41 |
| *** livelace has joined #rdo | 17:41 | |
| EmilienM | yeah | 17:42 |
| *** hynekm has joined #rdo | 17:42 | |
| number80 | wait | 17:43 |
| number80 | I'm building nodejs 4.4.7 | 17:43 |
| number80 | in RDO -testing | 17:43 |
| *** spr1 has quit IRC | 17:44 | |
| *** jhershbe has quit IRC | 17:44 | |
| *** rpioso has joined #rdo | 17:44 | |
| dmsimard | tristanC: are you there or I guess moving ? | 17:44 |
| *** gfidente is now known as gfidente|afk | 17:44 | |
| *** READ10 has joined #rdo | 17:47 | |
| *** ihrachys has quit IRC | 17:47 | |
| *** spr1 has joined #rdo | 17:47 | |
| *** Tenhi has joined #rdo | 17:48 | |
| *** jschlueter is now known as jschlueter|food | 17:48 | |
| *** jschlueter|food is now known as jschlueter | 17:48 | |
| *** mlammon is now known as mlammon|afk | 17:49 | |
| *** toanju has joined #rdo | 17:50 | |
| EmilienM | number80: that's why it broke or? | 17:50 |
| trown | apevec_: upstream still uses epel | 17:50 |
| trown | we wont hit that in RDO | 17:50 |
| *** imcleod has quit IRC | 17:50 | |
| *** DaveJ__ has quit IRC | 17:50 | |
| dmsimard | epel ? why ? | 17:51 |
| *** spr1 has quit IRC | 17:51 | |
| *** spr1 has joined #rdo | 17:51 | |
| apevec_ | https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo-delorean-promote-master/539/ - only packstack failed! | 17:52 |
| *** apevec_ is now known as apevec | 17:52 | |
| trown | apevec: ya rerun is queued up waiting on executor | 17:52 |
| *** spr1 has quit IRC | 17:52 | |
| dmsimard | cirros image failure again .. | 17:52 |
| *** mvk has joined #rdo | 17:52 | |
| trown | I am just going to rerun that hash until it passes | 17:52 |
| *** spr1 has joined #rdo | 17:52 | |
| apevec | trown, promote or die! | 17:53 |
| trown | without building images it is only about 90 minutes a run | 17:53 |
| trown | we will get it | 17:53 |
| dmsimard | at that point we should probably promote manually.. lol | 17:53 |
| *** tosky has quit IRC | 17:53 | |
| trown | I guess | 17:53 |
| apevec | yeah, let's promote that hash | 17:53 |
| trown | k | 17:53 |
| dmsimard | what hash is it | 17:53 |
| apevec | but keep the current run | 17:53 |
| dmsimard | I can do | 17:53 |
| apevec | maybe we get lucky in packstack :) | 17:53 |
| apevec | dmsimard, ah found that old dlrn HA proposal: https://github.com/javierpena/delorean-instance/blob/master/docs/delorean-instance.md it's different story | 17:54 |
| *** mburned is now known as mburned_out | 17:54 | |
| pabelanger | dmsimard: if you want to create an account on launchpad.net and enroll it into openstack.org, that will be the user zuul.rdoproject.org can use | 17:54 |
| apevec | dmsimard, but let's keep that for the next week when jpena is back | 17:54 |
| trown | dmsimard: 96/af/96af93d36151b04fc82c5c1db3615311bfda544e_d13b9e38 | 17:54 |
| EmilienM | number80: can you let us know if the nodejs failure is on your side? | 17:55 |
| dmsimard | apevec: yeah that's no good | 17:55 |
| trown | I will promote the image | 17:55 |
| EmilienM | all tripleo CI jobs are failing atm | 17:55 |
| *** iberezovskiy is now known as iberezovskiy|off | 17:55 | |
| *** Tenhi has quit IRC | 17:55 | |
| dmsimard | apevec: I think we can compromise and keep the current structure relatively intact if we switch from sql to a central mysql database ? | 17:55 |
| trown | EmilienM: node is coming from EPEL... rdo doesnt manage EPEL | 17:55 |
| dmsimard | it's sqlalchemy | 17:55 |
| *** degorenko is now known as _degorenko|afk | 17:55 | |
| trown | EmilienM: this comes up once every couple months... but upstream wants EPEL so this is what happens | 17:55 |
| EmilienM | trown: ok so something broke in EPEL | 17:55 |
| dmsimard | so we could have multiple "workers" for a release | 17:55 |
| dmsimard | pabelanger: ack | 17:56 |
| EmilienM | trown: have we tried to deploy tripleo CI without epel? | 17:56 |
| EmilienM | trown: Puppet CI doesn't deploy EPEL at all and it works fine | 17:56 |
| *** priteau has joined #rdo | 17:57 | |
| apevec | dmsimard, we also need central "repo" mashing place, but yeah central DB too | 17:57 |
| trown | EmilienM: RDO quickstart image does not have EPEL either | 17:57 |
| *** jhershbe has joined #rdo | 17:57 | |
| *** spr1 has quit IRC | 17:57 | |
| trown | it is totally possible to not use EPEL, but there are hardcoded dib things that set it up | 17:57 |
| apevec | EmilienM, number80 - Updated By: 1:libuv-1.8.0-1.el7.x86_64 (delorean-newton-testing) | 17:58 |
| apevec | that's new dep for nodejs update that number80 is doing | 17:58 |
| number80 | yeah, it's slowly coming | 17:59 |
| apevec | so it's a mix which is breaking | 17:59 |
| apevec | epel on its own should work | 17:59 |
| *** hynekm has quit IRC | 17:59 | |
| apevec | EmilienM, but what requires nodejs? which job is that? | 17:59 |
| number80 | I need this one before having 4.4.1 | 17:59 |
| number80 | https://cbs.centos.org/koji/taskinfo?taskID=102899 | 17:59 |
| *** dtrainor has quit IRC | 17:59 | |
| number80 | jobs are launched sequentially, but CBS queue is full | 17:59 |
| number80 | well, was full | 18:00 |
| EmilienM | apevec: tripleo | 18:00 |
| trown | EmilienM: this is all that is required to not use EPEL https://github.com/redhat-openstack/ansible-role-tripleo-image-build/blob/master/templates/dib-prepare-centos7-default.sh.j2#L29-L36 | 18:00 |
| EmilienM | trown: https://review.openstack.org/347499 | 18:00 |
| trown | I have brought it up before though and folks dont seem interested | 18:00 |
| EmilienM | trown: i AM interested. | 18:01 |
| trown | EmilienM: ya you will have to remove it from the DIB elements though | 18:01 |
| EmilienM | trown: can you look my patch and tells if I missed something? | 18:01 |
| number80 | EPEL must die # do not take this seriously! | 18:01 |
| *** panda|bbl is now known as panda | 18:01 | |
| EmilienM | trown: can you help me with that? | 18:01 |
| trown | sure | 18:02 |
| social | trown: I was on training any update on the neutron issue? | 18:02 |
| trown | EmilienM: /usr/share/diskimage-builder/elements/base/install.d/99-dkms is the tricky one as it is in the base element | 18:02 |
| trown | so my easy hack of just removing it probably will not fly upstream | 18:02 |
| *** ppowell has joined #rdo | 18:03 | |
| number80 | trown: worst-case would be adding it in -common | 18:03 |
| number80 | or have a separate repo for upstream purposes | 18:03 |
| number80 | mini-EPEL | 18:03 |
| trown | social: I increased the number of vCPUs of the undercloud VM to not create the issue | 18:03 |
| trown | social: since it only happens when things are bogged down | 18:04 |
| * number80 running low on energy | 18:04 | |
| trown | social: not a solution, but fixes CI | 18:04 |
| EmilienM | trown: where in ooo-ci could I apply your patch? | 18:04 |
| trown | EmilienM: not sure we can apply that in ooo-ci, as we get the "undercloud image" from infra (it is just infra centos image) | 18:05 |
| *** anilvenkata has joined #rdo | 18:06 | |
| trown | EmilienM: which is built by DIB | 18:06 |
| social | trown: and it worked? | 18:06 |
| trown | social: ya, it seems to have improved the pass rate pretty dramatically | 18:06 |
| trown | from 0 to non-zero for sure | 18:06 |
| trown | so infinite percent increase | 18:06 |
| social | ^_^ ok, I'll try to keep looking into this | 18:06 |
| *** leanderthal|afk has quit IRC | 18:07 | |
| *** jlibosva has joined #rdo | 18:07 | |
| trown | EmilienM: checking your patch though, I will help get it removed from tripleo-ci if we dont get blocked | 18:07 |
| EmilienM | k | 18:08 |
| EmilienM | lunch now | 18:08 |
| *** laron has joined #rdo | 18:08 | |
| *** dtrainor has joined #rdo | 18:12 | |
| *** toanju has quit IRC | 18:13 | |
| rdogerrit | Honza Pokorny created rdoinfo: Add openstack-tripleo-ui http://review.rdoproject.org/r/1733 | 18:17 |
| *** gszasz has quit IRC | 18:23 | |
| *** jprovazn has quit IRC | 18:27 | |
| *** chandankumar has quit IRC | 18:28 | |
| *** spr1 has joined #rdo | 18:30 | |
| *** jubapa_ has joined #rdo | 18:31 | |
| *** visibilityspots has quit IRC | 18:34 | |
| *** sdake has quit IRC | 18:35 | |
| *** laron has quit IRC | 18:35 | |
| *** visibilityspots has joined #rdo | 18:35 | |
| *** sdake has joined #rdo | 18:35 | |
| *** jhershbe has quit IRC | 18:37 | |
| *** anilvenkata has quit IRC | 18:41 | |
| dmsimard | apevec: we've reached disk space treshold again on internal dlrn | 18:42 |
| *** mlammon|afk is now known as mlammon | 18:43 | |
| *** aortega has quit IRC | 18:43 | |
| *** nyechiel has joined #rdo | 18:48 | |
| *** weshay has quit IRC | 18:49 | |
| *** jubapa_ has quit IRC | 18:54 | |
| *** dpeacock has quit IRC | 18:56 | |
| *** itlinux has joined #rdo | 18:56 | |
| *** dgurtner has quit IRC | 19:05 | |
| *** abregman has joined #rdo | 19:12 | |
| *** rain has joined #rdo | 19:12 | |
| *** rain is now known as Guest90133 | 19:13 | |
| *** sdake has quit IRC | 19:15 | |
| rdogerrit | Merged openstack/novaclient-distgit: Added py2 and py3 subpackage http://review.rdoproject.org/r/1618 | 19:17 |
| *** Guest90133 has quit IRC | 19:18 | |
| *** aortega has joined #rdo | 19:18 | |
| *** imcleod has joined #rdo | 19:22 | |
| *** jcoufal has quit IRC | 19:23 | |
| *** spr1 has quit IRC | 19:25 | |
| *** spr1 has joined #rdo | 19:25 | |
| *** nyechiel has quit IRC | 19:26 | |
| *** dneary has quit IRC | 19:28 | |
| *** spr1 has quit IRC | 19:29 | |
| *** Guest90133 has joined #rdo | 19:30 | |
| *** livelace has quit IRC | 19:30 | |
| *** milan has joined #rdo | 19:32 | |
| *** mburned_out is now known as mburned | 19:35 | |
| *** sdake has joined #rdo | 19:36 | |
| *** dgurtner has joined #rdo | 19:36 | |
| *** dgurtner has joined #rdo | 19:36 | |
| metabsd | How can assign a physical network card to my virtual switch ? | 19:36 |
| *** READ10 has quit IRC | 19:37 | |
| *** imcleod has quit IRC | 19:39 | |
| *** spr1 has joined #rdo | 19:41 | |
| *** weshay has joined #rdo | 19:42 | |
| *** dgurtner has quit IRC | 19:42 | |
| *** zeroshft has joined #rdo | 19:43 | |
| EmilienM | number80: any update on node? | 19:43 |
| EmilienM | tripleo ci is broken now | 19:43 |
| number80 | EmilienM: currently on it | 19:45 |
| number80 | I'll build 4.4.1 as a temp workaround | 19:46 |
| number80 | (it's good enough but I prefer building latest 4.4.7 which requires refreshing patches) | 19:46 |
| EmilienM | number80: please ping me when it's done so I can test it | 19:47 |
| *** milan is now known as _milan_ | 19:48 | |
| slagle | trown: if we can remove epel, let's do it | 19:48 |
| slagle | i dont know that anyone is necessary opposed to it | 19:49 |
| trown | slagle: I think between my patch and EmilienM's it will be removed | 19:49 |
| slagle | maybe we can just enable it on demand, such as for this bmc image | 19:49 |
| trown | https://review.openstack.org/347499 | 19:49 |
| *** arif-ali has quit IRC | 19:49 | |
| trown | slagle: I am not even sure it is needed there | 19:49 |
| trown | python-crypto is packaged | 19:49 |
| slagle | ok | 19:49 |
| apevec | trown, bnemec "python2-crypto doesn't exist without EPEL" ?? | 19:51 |
| trown | apevec: maybe it didnt at some point..l. it is definitely in delorean-deps now | 19:52 |
| bnemec | apevec: http://paste.openstack.org/show/542169/ | 19:52 |
| apevec | bnemec, yeah, replace it with python-crypto :) | 19:53 |
| apevec | python2-* is provided by python-* | 19:53 |
| apevec | err, other way around :) | 19:53 |
| apevec | so yum install python-crypto should always work | 19:53 |
| apevec | epel or without | 19:54 |
| *** _milan_ has quit IRC | 19:54 | |
| bnemec | Yeah, I'll look into changing it. | 19:57 |
| bnemec | For the moment I think we should just leave it alone though. There's no runtime dependency there, it only gets used if we rebuild the bmc image. | 19:57 |
| bnemec | Which needs to happen pretty much never. :-) | 19:57 |
| apevec | just curious, what is that image? | 19:59 |
| bnemec | It's the fake BMC that we use in CI for providing IPMI control of OpenStack instances. | 19:59 |
| bnemec | Part of OVB. | 19:59 |
| trown | apevec: how long does it typically take from promotion to showing up on http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tested// | 20:00 |
| apevec | trown, depens on CDN edge node you're hitting | 20:01 |
| apevec | trown, dmsimard - btw what failed in https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo-delorean-promote-master/540/ run? | 20:01 |
| *** coolsvap_ has quit IRC | 20:01 | |
| apevec | ha, puppet, packstack :( | 20:01 |
| dmsimard | looking | 20:01 |
| apevec | trown, that was w/ increased vcpu# ? | 20:02 |
| imcsk8 | apevec: checking | 20:02 |
| dmsimard | puppet-openstack-scenario002 TestNetworkBasicOps.test_network_basic_ops | 20:02 |
| trown | apevec: ya | 20:02 |
| apevec | so still not enough for HA job? :( | 20:02 |
| dmsimard | apevec: looks like a oslo.privsep stacktrace | 20:03 |
| dmsimard | first time I see one of those: https://ci.centos.org/artifacts/rdo/weirdo-master-promote-puppet-openstack-scenario002/408/puppet-openstack/logs/nova/nova-compute.txt.gz | 20:03 |
| trown | maybe not 100% of the time, but I bet it will pass sometimes on the slower chassis now | 20:03 |
| number80 | https://cbs.centos.org/koji/taskinfo?taskID=103010 | 20:03 |
| number80 | building | 20:03 |
| *** ppowell has quit IRC | 20:03 | |
| trown | which is an improvement :P | 20:03 |
| apevec | it is, but we have other odds to chase, so better make this 1.0 factor :) | 20:04 |
| dmsimard | apevec: looks like we could be hitting https://bugs.launchpad.net/oslo.privsep/+bug/1593743 | 20:07 |
| openstack | Launchpad bug 1593743 in oslo.privsep "(privsep) u'systool -c fc_host -v' failed. Not Retrying." [Undecided,New] - Assigned to Walt Boring (walter-boring) | 20:07 |
| trown | sounds boring | 20:08 |
| dmsimard | sounds like a trunk lib issue | 20:08 |
| dmsimard | EmilienM: fyi ^ | 20:08 |
| EmilienM | sounds super boring | 20:09 |
| apevec | reported month ago... | 20:09 |
| *** akshai has quit IRC | 20:09 | |
| dmsimard | added a comment, I'll try and hunt down that mister boring | 20:11 |
| dmsimard | I'll add that to both etherpads .. | 20:13 |
| *** fragatina has joined #rdo | 20:15 | |
| dmsimard | hm, turns out it might just be a logging issue | 20:15 |
| apevec | "That dropped off after we blacklisted the os-brick 1.4.0 version from global-requirements which contains the changes for using oslo.privsep." | 20:15 |
| apevec | oh, red herring? | 20:15 |
| dmsimard | dunno, I'll look some more | 20:16 |
| apevec | yeah, real ERROR at the end of compute log is ConnectTimeout: Request to https://[::1]:9696/v2.0/ports.json timed out | 20:17 |
| dmsimard | apevec: well, the actual tempest traces are http://paste.openstack.org/show/542172/ | 20:18 |
| *** jeckersb is now known as jeckersb_gone | 20:18 | |
| dmsimard | digging in the neutron logs is like searching for a needle in a haystack though | 20:19 |
| weshay | myoung, fyi.. https://review.openstack.org/#/c/346733/ | 20:19 |
| * myoung looks | 20:19 | |
| metabsd | neutron use openvswitch ? | 20:20 |
| *** zeroshft has quit IRC | 20:22 | |
| dmsimard | apevec: right, I see the ports.json timeout now | 20:22 |
| dmsimard | metabsd: neutron is a network abstraction, it has several backends for networking. openvswitch is one of those, linuxbridge is another one. | 20:23 |
| metabsd | dmsimard: I try to acces my instance by the network. I setup packstack --allinone. I try to find how I can attach physical network to my ovs. I think --allinone use OVS. I'm right ? | 20:24 |
| imcsk8 | metabsd: yes | 20:25 |
| *** kgiusti has left #rdo | 20:25 | |
| number80 | EmilienM: ok, untagging libuv, I have to patch nodejs to use openssl 1.0.1 | 20:26 |
| number80 | this is no trivial change | 20:27 |
| number80 | doable but not trivial | 20:27 |
| EmilienM | number80: k | 20:27 |
| apevec | number80, untagging won't remove it from buildlogs | 20:27 |
| number80 | the harder being that openssl has shitty documentation | 20:27 |
| number80 | apevec: haven't we changed that? | 20:28 |
| apevec | it's still accumulating afaict | 20:28 |
| *** abregman has quit IRC | 20:29 | |
| apevec | but, how does nodejs get into tripleo-ci ? | 20:29 |
| apevec | that should be only for tripleo-ui jobs? | 20:30 |
| number80 | I thought that metadata would be regenerated | 20:30 |
| number80 | well, I may fix it tonight, but not 100% sure | 20:30 |
| apevec | untag it for now, and we'll see | 20:31 |
| dmsimard | apevec: can't find a legit issue with that puppet job, it looks like it may just be a flap - not seeing it anywhere else. I'll wait and see if it reproduces. | 20:31 |
| dmsimard | apevec: I noted the privsep stacktrace in the etherpad nonetheless | 20:31 |
| apevec | yeah, must be regular flapping, it worked w/ same hash run before | 20:32 |
| *** imcleod has joined #rdo | 20:33 | |
| *** spr1 has quit IRC | 20:39 | |
| *** jtomasek has joined #rdo | 20:40 | |
| *** jtomasek has quit IRC | 20:41 | |
| *** jtomasek has joined #rdo | 20:41 | |
| imcsk8 | dmsimard: i found this on the console log, i'm not sure if it's relevant: https://paste.fedoraproject.org/396004/46956581/ | 20:44 |
| *** dneary has joined #rdo | 20:48 | |
| *** imcleod has quit IRC | 20:50 | |
| *** fultonj has quit IRC | 20:52 | |
| *** KarlchenK has joined #rdo | 20:54 | |
| *** arif-ali has joined #rdo | 20:57 | |
| rdogerrit | Merged rdoinfo: Normalize rdo.yml http://review.rdoproject.org/r/1729 | 20:58 |
| *** KarlchenK has quit IRC | 21:00 | |
| *** julim has quit IRC | 21:02 | |
| *** KarlchenK has joined #rdo | 21:02 | |
| *** amuller has quit IRC | 21:02 | |
| *** trown is now known as trown|outtypewww | 21:02 | |
| trown|outtypewww | weshay: so green https://dashboards.rdoproject.org/rdo-dev | 21:03 |
| *** unclemarc has quit IRC | 21:03 | |
| weshay | wha hoooo | 21:03 |
| weshay | jschlueter, ^ | 21:03 |
| *** rlandy has quit IRC | 21:06 | |
| *** rlandy has joined #rdo | 21:07 | |
| *** KarlchenK has quit IRC | 21:08 | |
| *** mburned is now known as mburned_out | 21:08 | |
| *** iranzo has quit IRC | 21:13 | |
| apevec | trown|outtypewww, it hurts my eyes | 21:14 |
| *** KarlchenK has joined #rdo | 21:14 | |
| apevec | also green with "1 issue" :) | 21:14 |
| *** iranzo has joined #rdo | 21:14 | |
| apevec | ah systool thing | 21:14 |
| apevec | changed to TBD | 21:15 |
| myoung | woot! | 21:22 |
| *** dustins has quit IRC | 21:24 | |
| *** jlibosva has quit IRC | 21:28 | |
| *** dgurtner has joined #rdo | 21:32 | |
| *** dgurtner has joined #rdo | 21:32 | |
| *** linuxaddicts has quit IRC | 21:33 | |
| *** linuxaddicts has joined #rdo | 21:36 | |
| *** dgurtner has quit IRC | 21:37 | |
| *** rhallisey has quit IRC | 21:41 | |
| *** jubapa_ has joined #rdo | 21:44 | |
| number80 | good progress with node => https://cbs.centos.org/koji/watchlogs?taskID=103018 | 21:44 |
| number80 | I have patches that builds | 21:44 |
| *** ayoung has quit IRC | 21:45 | |
| *** ccamacho|away has quit IRC | 21:46 | |
| number80 | that | 21:46 |
| *** jmelvin has quit IRC | 21:51 | |
| *** mlammon has quit IRC | 21:56 | |
| *** apevec has quit IRC | 21:56 | |
| *** gildub has joined #rdo | 21:57 | |
| *** openstackgerrit has quit IRC | 22:03 | |
| *** openstackgerrit has joined #rdo | 22:04 | |
| *** ChanServ sets mode: +v openstackgerrit | 22:04 | |
| *** sdake has quit IRC | 22:09 | |
| *** KarlchenK has quit IRC | 22:13 | |
| *** Alex_Stef has quit IRC | 22:15 | |
| number80 | EmilienM: is your CI still broken? | 22:17 |
| *** priteau has quit IRC | 22:23 | |
| number80 | I have working build of node.js 4.4.7 \o/ | 22:24 |
| *** egallen has joined #rdo | 22:24 | |
| *** erlon has quit IRC | 22:26 | |
| number80 | found a job failing due to libuv/nodejs | 22:30 |
| *** iranzo has quit IRC | 22:33 | |
| *** gildub has quit IRC | 22:35 | |
| *** limao has joined #rdo | 22:38 | |
| *** aortega has quit IRC | 22:40 | |
| *** rpioso has quit IRC | 22:44 | |
| *** sdake has joined #rdo | 22:47 | |
| *** openstackgerrit has quit IRC | 22:48 | |
| *** sdake_ has joined #rdo | 22:49 | |
| *** openstackgerrit has joined #rdo | 22:49 | |
| *** ChanServ sets mode: +v openstackgerrit | 22:49 | |
| *** Son_Goku has joined #rdo | 22:49 | |
| *** sdake has quit IRC | 22:52 | |
| *** morazi has quit IRC | 22:55 | |
| *** Son_Goku has quit IRC | 23:12 | |
| *** Son_Goku has joined #rdo | 23:13 | |
| *** elmiko is now known as _elmiko | 23:15 | |
| *** jubapa_ has quit IRC | 23:16 | |
| *** egafford has quit IRC | 23:18 | |
| *** limao has quit IRC | 23:21 | |
| itlinux | hello all, has anyone done the undercloud in an HA mode? | 23:22 |
| *** thrash is now known as thrash|g0ne | 23:38 | |
| *** fragatina has quit IRC | 23:52 | |
| *** kaminohana has joined #rdo | 23:54 | |
| *** egallen has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!