Tuesday, 2016-02-09

*** dmacpher has quit IRC00:13
*** eil397 has quit IRC00:27
jidar{get_attr: [Compute, hosts_entry]} doesn't work either00:33
* jidar groans00:33
*** rpothier has quit IRC00:58
*** dmacpher has joined #tripleo01:05
*** rbrady has quit IRC01:09
*** rbrady has joined #tripleo01:10
*** weshay_xchat has joined #tripleo01:14
*** rbrady has quit IRC01:18
*** rbrady has joined #tripleo01:20
*** tiswanso has quit IRC01:38
*** weshay_xchat has quit IRC02:02
openstackgerritColleen Murphy proposed openstack/diskimage-builder: Add --version option to disk-image-create  https://review.openstack.org/27765802:07
*** shivrao has quit IRC02:11
*** tiswanso has joined #tripleo02:14
jidar>Note The OS::TripleO::NodeExtraConfigPost applies configuration to all nodes, there is currently no per-role NodeExtraConfigPost interface.02:14
jidar:( :( :( :(02:14
*** thrash is now known as thrash|g0ne02:22
*** sthillma has quit IRC02:30
*** Marga_ has quit IRC02:30
*** Marga_ has joined #tripleo02:34
*** Marga_ has quit IRC02:40
*** rhallisey has quit IRC02:52
*** sbalukoff has quit IRC03:00
*** lblanchard has quit IRC03:26
*** panda has quit IRC03:37
*** panda has joined #tripleo03:37
*** links has joined #tripleo03:41
*** cwolferh has quit IRC03:54
*** Marga_ has joined #tripleo03:54
*** cwolferh has joined #tripleo04:08
*** rlandy has quit IRC04:09
*** cwolferh has quit IRC04:11
*** cwolferh has joined #tripleo04:12
*** ayoung has quit IRC04:15
*** dshulyak has joined #tripleo04:24
*** dshulyak has quit IRC04:27
*** sbalukoff has joined #tripleo04:30
*** tiswanso has quit IRC04:30
*** tzumainn has quit IRC05:08
*** masco has joined #tripleo05:09
*** rbrady has quit IRC05:13
*** rbrady has joined #tripleo05:13
*** rbrady has quit IRC05:18
openstackgerritGraeme Gillies proposed openstack/instack-undercloud: Added a new element to configure ansible inventory from nova  https://review.openstack.org/27768805:21
*** Marga_ has quit IRC05:24
*** cwolferh has quit IRC05:27
*** jtomasek has quit IRC05:32
*** ukalifon has joined #tripleo05:34
*** ukalifon has quit IRC05:40
*** ukalifon1 has joined #tripleo05:40
*** ukalifon2 has joined #tripleo05:41
*** ukalifon2 has quit IRC05:41
*** ukalifon1 has quit IRC05:41
*** ukalifon has joined #tripleo05:41
*** anande has joined #tripleo05:45
*** anande has quit IRC05:46
*** NobodyCam has quit IRC05:48
*** dmacpher has quit IRC05:50
*** NobodyCam has joined #tripleo05:50
*** cwolferh has joined #tripleo05:58
*** Marga_ has joined #tripleo06:06
*** dcain has quit IRC06:07
*** ukalifon has quit IRC06:21
*** cwolferh has quit IRC06:33
*** ukalifon has joined #tripleo06:48
*** jprovazn has joined #tripleo06:51
*** jprovazn has quit IRC06:52
*** jprovazn has joined #tripleo06:52
*** cwolferh has joined #tripleo06:59
*** ukalifon has quit IRC07:13
*** shivrao has joined #tripleo07:37
*** panda has quit IRC07:37
*** panda has joined #tripleo07:37
*** jtomasek__ has quit IRC07:38
*** athomas has joined #tripleo07:40
*** shivrao has quit IRC07:46
*** hjensas has joined #tripleo07:55
*** hjensas has quit IRC07:55
*** hjensas has joined #tripleo07:55
*** jcoufal has joined #tripleo07:59
*** liverpooler has joined #tripleo08:03
*** aufi has joined #tripleo08:06
*** fgimenez has joined #tripleo08:11
*** fgimenez has quit IRC08:11
*** fgimenez has joined #tripleo08:11
*** oshvartz has joined #tripleo08:11
*** pblaho has joined #tripleo08:11
*** ukalifon1 has joined #tripleo08:17
*** mkovacik has joined #tripleo08:24
*** shardy has joined #tripleo08:30
*** devvesa has joined #tripleo08:35
*** jaosorior has joined #tripleo08:37
openstackgerritMerged openstack/tripleo-heat-templates: Create linux bridge vlans environments  https://review.openstack.org/27020508:50
jaosorioryay, I see the CI is working again! :D though there are so many jobs pending that it'll take a while to get back to business08:52
openstackgerritMerged openstack/tripleo-heat-templates: Fix MidoNet errors  https://review.openstack.org/27020708:52
shardyjaosorior: actually those two had passes from before the most recent breakage08:53
jaosoriorshardy: Yeah, I actually went to check08:53
shardycistatus looks a bit more healthy08:53
jaosoriorshardy: Though it seems to me that CI is broken in another way now, since a lot of the new ones have failures08:53
shardyjaosorior: ah, cool, yeah it looks like things are at least running again now :)08:53
openstackgerritJaume Devesa proposed openstack/tripleo-heat-templates: Create linux bridge vlans environments  https://review.openstack.org/27773808:53
jaosoriorso there is the mongodb issue which I was trying to fix, and now another failure... :(08:53
* shardy sighs08:54
jaosoriorshardy, anyway, still in Brno?08:54
shardyjaosorior: No, I'm back home now, and you?08:54
jaosoriorstill here08:54
jaosoriorI leave tomorrow08:54
*** morazi has joined #tripleo08:58
*** mbound has joined #tripleo08:59
openstackgerritJaume Devesa proposed openstack/tripleo-heat-templates: Fix MidoNet errors  https://review.openstack.org/27774209:02
*** jaosorior has quit IRC09:03
*** jaosorior has joined #tripleo09:08
*** jaosorior has quit IRC09:08
*** jaosorior has joined #tripleo09:09
*** lucas-dinner is now known as lucasagomes09:10
jaosoriorshardy: So the tests are failing with something like this http://logs.openstack.org/88/268288/3/check-tripleo/gate-tripleo-ci-f22-ha/2d11ca3/console.html#_2016-02-09_01_52_13_11509:16
jaosoriorshardy: Are we using v3 yet? It seems that the neutronclient tries to access keystone through the v3 API and gets a NotFound error there09:18
*** ifarkas has joined #tripleo09:19
shardyjaosorior: yes there have been various patches moving things over to v3, but I assume we won't have landed any without passing CI09:21
jaosoriorshardy: Well, the CI has been out since before Friday, so it might have been something that merged from friday until today :/09:22
shardyI just meant, if we've broken things I'd have never expected CI to pass09:25
jaosoriorshardy: the puppet modules are built from git, right?09:26
shardyyeah, something changing there is a likely candidate09:27
jaosoriorshardy: Can I rebuild the overcloud images with the latest puppte manifests using tripleo.sh?09:28
shardyjaosorior: yes, just re-run tripleo.sh --overcloud-images09:28
jaosoriorshardy: Thanks dude09:28
shardythat looks likely to be related09:29
shardyI'll propose a pin to test09:29
shardyEmilienM: ^^ FYI when you wake up09:29
jaosoriorshardy: that looks like it could be it09:30
jaosoriorshardy: Well, i rant he tripleo.sh with --overcloud-images and all I got is that the images are up to date09:30
shardyjaosorior: you can remove the existing image first09:31
jaosoriorshardy: overcloud-full.qcow2?09:33
openstackgerritSteven Hardy proposed openstack/tripleo-common: Pin puppet-nova  https://review.openstack.org/27775609:39
*** derekh has joined #tripleo09:39
*** dtantsur|afk is now known as dtantsur09:42
jaosoriorshardy: Nice, hopefully that does the trick for now. Though I suspect we need to finally ditch the setting up of keystone endpoints through os-cloud-config in order to take that patch into use09:43
dtantsurmorning folks! is "ha" job still down or should I recheck things?09:44
shardydtantsur: we're waiting to see if https://review.openstack.org/277756 fixes the latest breakage from puppet-nova09:45
jaosoriordtantsur: The now all jobs are broken09:45
jaosoriordtantsur: And I think HA will be still broken09:45
shardyYeah, I'd delay rechecking until we see the results of that09:45
dtantsurgot it, thanks09:45
jaosoriorI have a patch trying to fix the HA job, but the CI had an outage in the weekend. But aparently dprince and derekh fixed it :D09:46
derekhjaosorior: the cloud is back running as of last night09:47
jaosoriorderekh: I noticed, thanks dude!09:47
*** mbound has quit IRC09:47
derekhlooks like both ha and nonha are failing now though09:47
jaosoriorderekh: and ceph too. That is what shardy is trying to address in this CR https://review.openstack.org/27775609:48
derekhjaosorior: ^^ so we probably now have 2 problems09:48
derekhjaosorior: ack09:48
derekhjaosorior: will check back when the job is finished09:48
*** fgimenez has left #tripleo09:49
jaosoriorderekh: would it be possible to get an overview at some point on how the CI nodes are working and such? If an outage happens like the one from this weekend, I would also like to be able to help09:49
derekhjaosorior: sure, I put some info in an email a while back I'll dig it up later and point you at it09:50
jaosoriorderekh: Thanks09:50
*** NobodyCam has quit IRC09:52
*** mbound has joined #tripleo09:55
*** tosky has joined #tripleo09:59
*** NobodyCam has joined #tripleo09:59
*** jtomasek__ has joined #tripleo10:10
*** jtomasek__ has quit IRC10:10
*** jtomasek has joined #tripleo10:11
*** Marga_ has quit IRC10:12
*** mcornea has joined #tripleo10:13
jaosoriorshardy: So I think I did it wrong when rebuilding the images... since the controller creation actually passed in my deployment :/10:15
*** gfidente has joined #tripleo10:15
jaosoriordamn, I think I needed to rebuild the undercloud in order to reproduce the error10:17
*** electrofelix has joined #tripleo10:18
*** r-mibu has quit IRC10:20
*** r-mibu has joined #tripleo10:21
*** jaosorior has quit IRC10:29
*** jaosorior has joined #tripleo10:29
*** deva_ has joined #tripleo10:30
*** NobodyCa1 has joined #tripleo10:33
jaosorioralright, the same error occurs for the VMs created by the overcloud10:36
*** NobodyCa1 has quit IRC10:38
*** deva_ has quit IRC10:38
*** mgould has joined #tripleo10:49
*** dprince has joined #tripleo10:52
*** weshay_xchat has joined #tripleo10:55
*** ukalifon1 has quit IRC10:57
*** mbound has quit IRC11:02
*** mbound has joined #tripleo11:15
*** ukalifon1 has joined #tripleo11:22
*** shardy has quit IRC11:34
*** shardy has joined #tripleo11:36
*** panda has quit IRC11:37
*** panda has joined #tripleo11:38
*** jcoufal has quit IRC11:38
EmilienMshardy: yeah I'm sorry if this patch annoys you11:53
EmilienMshardy: dmsimard is working on a patch in THT and we also need to patch undercloud11:53
EmilienMthe problem is this patch is required for the recent nova in mitaka11:53
EmilienMit broke our CI (when running trunk)11:53
EmilienMshardy: we need https://review.openstack.org/#/c/276790/11:54
EmilienMbut CI is failing11:55
shardyEmilienM: Ok, np, I've proposed a temporary pin via https://review.openstack.org/#/c/277756/11:57
shardywaiting on CI to prove that works around the issue11:57
EmilienMshardy: awesome, it will let us the time to patch tripleo11:57
EmilienMshardy: sorry for the breakage... the patch was supposed to be backward compatible at some point11:58
*** trown has joined #tripleo11:58
EmilienMshardy: I'll take care of this today and make sure we go back on puppet-nova master11:59
shardyEmilienM: great, thanks! :)11:59
*** trown has quit IRC11:59
*** ukalifon1 has quit IRC12:00
*** trown has joined #tripleo12:00
*** trown is now known as trown|outtypewww12:02
*** lucasagomes is now known as lucas-hungry12:13
openstackgerritMerged openstack/puppet-tripleo: Remove all 'validate_array' statements  https://review.openstack.org/26747412:14
dprincederekh, shardy do we know the cause of the java.io.IOException: Unexpected termination of the channel?12:25
dprincein CI12:25
shardydprince: Not seen that one yet - I was looking at the nova/neutron NotFound error ref https://review.openstack.org/#/c/277756/12:26
derekhdprince: havn't been looking, trying to reproduce something else here, normally  that is an interuption in communication between jenkins and the nodes12:26
shardydprince: CI jobs were broken over the weekend tho, I'm not yet sure what the solution was12:26
dprinceshardy: there was an overcloud controller issue derekh fixed yesterday12:26
dprinceshardy: but this java.io exception is new from the hudson/jenkins slave it appears12:27
*** Goneri has joined #tripleo12:27
derekhdprince: historically I don't think that particular problem has been a problem with the cloud, jenkins could have been rebooted or something, is it still happening?12:27
dprincederekh: I've seen a couple12:27
*** dmacpher has joined #tripleo12:29
derekhdprince: They mentioned yesterday something about switching links in the data center, I wonder if its still ongoing ... just a theory12:33
dprincederekh: not sure, we'll just have to ask12:35
jaosoriorshardy: From what I see it seems that the nova/neutron NotFound error is due to a wrong keystone configuration12:36
openstackgerritMiles Gould proposed openstack/python-tripleoclient: Use Ironic API v1.11 to support ENROLL state  https://review.openstack.org/27220612:36
openstackgerritMiles Gould proposed openstack/python-tripleoclient: Remove tripleoclient.baremetal wrapper  https://review.openstack.org/26533612:36
jaosoriorwhere the auth_url is set to keystone_endpoint/v2.0 while the plugin to be used is already set as v3password, which will fail12:37
jaosoriorso I've been manually changing the configuration to see what works12:37
EmilienMshardy: http://logs.openstack.org/90/276790/4/check-tripleo/gate-tripleo-ci-f22-nonha/dc068b3/console.html#_2016-02-08_23_04_33_47512:38
EmilienMit means your version of nova does not contain the change we had to make our CI passing12:38
EmilienMdo you pin nova? if yes, to which commit?12:38
*** openstackgerrit_ has joined #tripleo12:38
jaosoriorEmilienM: Which commit did you use to make CI pass?12:43
*** jcoufal has joined #tripleo12:44
*** marcusvrn_ has joined #tripleo12:44
EmilienMjaosorior: I don't know exactly but a very recent, maybe 4 or 5 days12:44
jaosoriorEmilienM: Did you mean this one? https://github.com/openstack/puppet-nova/commit/d09868a59c451932d67c66101b725182d7066a1412:45
EmilienMjaosorior: no... In Nova itself ;-)12:45
jaosoriorEmilienM: I see12:46
*** rlandy has joined #tripleo12:54
*** Marga_ has joined #tripleo12:55
*** thrash|g0ne is now known as thrash12:57
*** trown|outtypewww is now known as trown12:58
*** masco has quit IRC13:01
*** dshulyak has joined #tripleo13:02
*** lucas-hungry is now known as lucasagomes13:07
*** jayg|g0n3 is now known as jayg13:12
EmilienMshardy: I need to know about nova, do you pin it?13:14
*** dshulyak has quit IRC13:15
*** links has quit IRC13:17
EmilienMderekh, dprince: ^ in case you know too13:23
*** aufi has quit IRC13:24
*** akuznetsov has joined #tripleo13:24
slagleEmilienM: we get it from http://trunk.rdoproject.org/centos7/current-tripleo/13:26
slaglelooks like it's from january 25th13:26
EmilienMok I think we're missing a commit13:27
EmilienMthat will break v2 auth for neutron notifs13:27
EmilienMwe had to patch puppet-nova https://github.com/openstack/puppet-nova/commit/d09868a59c451932d67c66101b725182d7066a1413:27
derekhEmilienM: is that the reason for the puppet-nova pin shardy proposed ?13:32
EmilienMyeah I think13:32
derekhEmilienM: ok13:32
EmilienMdmsimard has a patch in THT to fix it, but we need a recent nova to test it13:33
derekhthe nonha job for that pin looks like its nearly finished, if it passed I'll merge it, then we can figure out the problem with HA deployments13:33
derekhonce that is fixed we can move to a more recent current-tripleo13:34
dprincein order to bump Delorean we'll also need this: https://review.openstack.org/#/c/275607/13:34
trownoh right we need that on the undercloud too13:35
*** akuznetsov has quit IRC13:36
*** dcain has joined #tripleo13:39
*** bvandenh has joined #tripleo13:39
openstackgerritDerek Higgins proposed openstack/tripleo-common: Pin puppet-nova  https://review.openstack.org/27775613:44
derekhshardy: your patch made us successfully deploy the overcloud13:45
derekhshardy: but then the heat stack for the ping test failed, see update ^^13:45
derekhshardy: The fact that it passed the overcloud deploy, I think is enough to prove we need the pin, I'm +1 for merging it now13:46
derekhdprince: slagle ^13:46
*** akuznetsov has joined #tripleo13:46
*** ukalifon has joined #tripleo13:52
*** openstackgerrit_ has quit IRC13:56
derekhjaosorior: Now that ^^ has merged, didn't you say you had a patch to fix HA deployments ? wanna recheck it to see how it goes.13:57
derekhjaosorior: hold that idea, it hasn't merged yet13:58
shardyderekh: +1, thanks13:58
shardyderekh: I was wondering, should we really be setting global (overcloud/undercloud) pins at the top, e.g under "Temporary workarounds"?13:59
shardythat might avoid other folks making the same mistake I did13:59
jaosoriorderekh: Will do14:00
*** akuznetsov has quit IRC14:00
jaosoriorderekh: Ah, so it hasn't merged?14:01
dprincemeeting time14:01
jaosoriordprince: What was the channel/14:01
openstackgerritLucas Alvares Gomes proposed openstack/python-tripleoclient: Use the grub2 element instead of grub2-deprecated  https://review.openstack.org/27784214:01
dprincejaosorior: openstack-meeting214:03
dprincejaosorior: openstack-meeting-alt14:03
*** rpothier has joined #tripleo14:08
*** akuznetsov has joined #tripleo14:08
*** jpeeler has joined #tripleo14:11
*** jpeeler has quit IRC14:11
*** jpeeler has joined #tripleo14:11
openstackgerritMerged openstack/tripleo-common: Pin puppet-nova  https://review.openstack.org/27775614:13
openstackgerritRyan Brady proposed openstack/tripleo-common: WIP Mistral Experiment  https://review.openstack.org/27528514:15
*** links has joined #tripleo14:16
*** lblanchard has joined #tripleo14:17
openstackgerritGeorge Shuklin proposed openstack/diskimage-builder: Add DIB_APT_SOURCES_INLINE to apt-sources element.  https://review.openstack.org/27785014:20
*** eggmaster has joined #tripleo14:20
*** yamahata has joined #tripleo14:21
*** NobodyCa1 has joined #tripleo14:22
*** tiswanso has joined #tripleo14:24
*** tosky_ has joined #tripleo14:25
*** rhallisey has joined #tripleo14:26
*** tosky has quit IRC14:26
*** tzumainn has joined #tripleo14:26
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Forcibly clear parameters (now passed as parameter_defaults)  https://review.openstack.org/25667014:26
*** stendulker has joined #tripleo14:29
*** NobodyCa1 has quit IRC14:29
*** tiswanso has quit IRC14:30
dmsimarddprince: made a comment on https://review.openstack.org/#/c/275607/14:31
*** tiswanso has joined #tripleo14:31
*** dshulyak has joined #tripleo14:32
*** mgould has quit IRC14:34
*** stendulker has quit IRC14:46
*** mgould has joined #tripleo14:49
*** julim_ has quit IRC14:49
jpeelerdprince: could you do a quick review of https://review.openstack.org/#/c/265366/ ?14:50
*** egafford has joined #tripleo14:51
*** pradk has joined #tripleo14:52
openstackgerritDavid Moreau Simard proposed openstack/tripleo-heat-templates: Nova now requires an api database to be created  https://review.openstack.org/27679014:52
dmsimarddprince: ^ new patchset with nova_api user14:53
*** links has quit IRC14:54
dprincejpeeler: +2, but the HA ci should patch I think14:55
dprincedmsimard: thanks14:55
jaosoriorderekh: Sorry, I'm still in a meeting. But I'm not sure if there's a bug for the HA problem14:55
jpeelerdprince: you need me to recheck to get it approved?14:55
dprincejpeeler: CI outage right now, so I'd wait a bit14:55
jpeelerit all passed at one point...14:56
jpeelerwithout changes14:56
derekhjaosorior: I rechecked it anyways, lets wee what happens14:56
dmsimardEmilienM: did you figure out if tripleo was good on the nova-neutron config ?14:56
dmsimardnova api database patches are in review for tripleo and nova api database patch is merged in packstack, looking at nova-neutron now14:57
*** rbrady has joined #tripleo14:57
EmilienMdmsimard: no14:57
*** hjensas has quit IRC14:58
trowndmsimard: EmilienM: dprince: derekh, maybe we should revive the the etherpad to start listing all issues in tripleo preventing us from moving the tripleo-current pin to current14:59
dmsimardtrown: ok I'll add the current bugs/reviews that I am aware of there15:00
trowndmsimard: I just archived what was there, since that was for a previous promote15:01
derekhtrown: ack, good idea, btw, our periodic job is now actually testing current so we should see it pass when things are fixed15:01
derekhtrown: finding the results of the periodic job is anothor thing15:02
derekhtrown: once we merge this, I'll set up the report page, https://review.openstack.org/#/c/271370/215:02
trownshardy: can we merge that without CI ^, since it is just reporting15:03
*** dtantsur is now known as dtantsur|brb15:03
shardytrown: if derekh is happy with that then yes, I think so15:05
dmsimardEmilienM: does tripleo need anything for the gnocchi pipeline issue ?15:06
EmilienMdmsimard: no15:06
*** tiswanso has quit IRC15:06
*** tiswanso has joined #tripleo15:09
*** shardy has quit IRC15:28
*** dshulyak has quit IRC15:30
openstackgerritRajini Ram proposed openstack/tripleo-heat-templates: Update Dell Storage Center api port setting  https://review.openstack.org/27788815:31
*** dcain has left #tripleo15:32
*** aufi has joined #tripleo15:33
devvesashardy: Sorry. I couldnt attend to today's meeting cause I had another one in the same time. Does the success of this patch https://review.openstack.org/#/c/277756/15:33
devvesamean that it can fix the stable/liberty CI errors?15:33
*** panda has quit IRC15:38
*** panda has joined #tripleo15:38
slaglederekh: is there a way to clear the zuul queue, and would it make sense to do so?15:41
slaglewe know it's all going to fail, with the proposed fix being https://review.openstack.org/#/c/276701/15:41
slaglewhich is not first in the queue :)15:41
*** tserong has quit IRC15:42
*** julim has joined #tripleo15:45
derekhslagle: no way to do it as far as I know, unless some people in #infra have a magic button15:46
*** marcusvrn_ has quit IRC15:47
*** tserong has joined #tripleo15:49
*** shardy has joined #tripleo15:51
derekhslagle: we could sabotage the running jobs to fail quicker ;-)15:51
*** dmacpher is now known as dmacpher-afk15:52
*** liverpooler has quit IRC15:53
*** mcornea has quit IRC16:02
*** stendulker has joined #tripleo16:05
*** aufi has quit IRC16:07
*** rpothier has left #tripleo16:07
*** oshvartz has quit IRC16:09
*** mbound has quit IRC16:14
ryansbthere needs to be some kind of a "this patch fixes the gate, kill other jobs and requeue them after this one"16:17
ryansb*some kind of a switch for16:17
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Don't regenerate the overcloud passwords if the Heat stack exists  https://review.openstack.org/27566116:23
derekhdprince: slagle so the overcloud looks like it still has problems, only 9 ACTIVE vms and new ones going to error state16:23
*** dtantsur|brb is now known as dtantsur16:24
dprincederekh: okay, I just asked clark in infra about the timeouts. He seemed to think they were on our side16:24
derekhdprince: http://paste.openstack.org/show/486456/16:25
*** ukalifon has quit IRC16:26
derekhdprince: looks like its lost cound of the user resources, we're not running 308 VM's16:26
dprincederekh: I wonder if post-reboot of the controller if we might should just spin up new overcloud compute nodes again?16:26
d0ugaltzumainn: ^ updated the review.16:27
jaosoriorderekh, dprince, shardy: Has the fix in nova that EmilienM was talking about landed yet?16:27
derekhdprince: maybe...no sure I thought we did this reboot in the past and didn#t have to reboot the controllers16:29
dprincederekh: i think it has always be a bit tricky, the reboots16:30
dprincein theory it should be fine though16:30
tzumainnd0ugal, thanks, I'll take a look in a bit!16:30
*** akuznetsov has quit IRC16:31
derekhdprince: yup its always been tricky alright ;-)16:31
derekhdprince: now rebooting the computes, I think thats how we ended up with the arp flood the last time, I ended up having to rebuild them16:32
derekhdprince: how would that fix the DB of used quota16:32
dprincederekh: It makes me thinks the computs are communicating16:32
dprincederekh: should automatically refresh16:33
*** mcornea has joined #tripleo16:35
derekhdprince: sec, just checking stasts for one of the ndoes to see if they match up16:36
*** mbound has joined #tripleo16:40
*** yamahata has quit IRC16:47
derekhdprince: so, all of the instances currently running jenkins nodes are on the same compute node, I buch of them probably got overwhelmed yesterday when the controller got fixed and we opened the floodgate16:48
derekhdprince: going to bounce a few of them now16:50
*** stendulker has quit IRC16:54
*** d0ugal has quit IRC17:02
*** dsneddon_brno has joined #tripleo17:03
*** jaosorior has quit IRC17:04
*** bvandenh has quit IRC17:06
*** yamahata has joined #tripleo17:13
openstackgerritDmitry Tantsur proposed openstack/instack-undercloud: Enable extra hardware data collection and processing for ironic-inspector  https://review.openstack.org/27797417:15
jidarhey guys, is there any way at all to limit "OS::TripleO::NodeExtraConfigPost" to just controllers or compute hosts, and if not can I do something only on UPDATE that would achieve the same solution?17:16
jidarright now I'm looking at having the puppet component run only on if $::hostname =~ /.*controller.*/ { ... } or similar17:17
jidarbut that's sort of pokey17:17
openstackgerritDmitry Tantsur proposed openstack/instack-undercloud: Enable extra hardware data collection and processing for ironic-inspector  https://review.openstack.org/27797417:18
*** morazi has quit IRC17:19
*** jcoufal has quit IRC17:19
*** jprovazn has quit IRC17:21
*** sthillma has joined #tripleo17:22
*** athomas has quit IRC17:23
*** tiswanso has quit IRC17:24
-openstackstatus- NOTICE: Gerrit is restarting now, to alleviate current performance impact and WebUI errors.17:25
*** sthillma_ has joined #tripleo17:25
*** tiswanso has joined #tripleo17:26
*** sthillma has quit IRC17:27
*** sthillma_ is now known as sthillma17:27
*** mbound has quit IRC17:30
*** devvesa has quit IRC17:32
*** masco has joined #tripleo17:34
*** jaosorior has joined #tripleo17:39
gfidentejidar, I think you'd have to add a role specific extraconfigpost17:44
gfidentein the templates17:44
*** mkovacik has quit IRC17:44
openstackgerritDmitry Tantsur proposed openstack/instack-undercloud: Enable extra hardware data collection and processing for ironic-inspector  https://review.openstack.org/27797417:48
*** jtomasek has quit IRC17:48
*** eil397 has joined #tripleo17:50
*** jaosorior has quit IRC17:53
openstackgerritxin wu proposed openstack/tripleo-heat-templates: Include big switch puppet modules for deploying overcloud  https://review.openstack.org/27194017:55
derekhdprince: gotta run, all the failing compute nodes have been rebuilt, zull is now running about 30 jobs (and increasing), we were running 8 or 917:56
dprincederekh: I will watch it, thanks17:57
*** derekh has quit IRC17:58
openstackgerritxin wu proposed openstack/tripleo-heat-templates: Include big switch puppet modules for deploying overcloud  https://review.openstack.org/27195317:59
*** shardy has quit IRC18:00
*** penick has joined #tripleo18:00
*** gfidente has quit IRC18:02
*** bnemec has joined #tripleo18:02
*** penick has quit IRC18:05
*** eil397 has quit IRC18:07
*** electrofelix has quit IRC18:07
*** mgould has quit IRC18:08
*** eil397 has joined #tripleo18:08
*** tzumainn has quit IRC18:10
*** lucasagomes is now known as lucas-dinner18:11
*** tzumainn_ has joined #tripleo18:13
*** tzumainn_ is now known as tzumainn18:14
*** tosky_ has quit IRC18:17
*** xarses has joined #tripleo18:17
*** rhallisey has quit IRC18:18
*** rhallisey has joined #tripleo18:19
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Increase size of connection tracking table  https://review.openstack.org/27576918:20
*** alop has joined #tripleo18:20
*** akuznetsov has joined #tripleo18:23
*** akuznetsov has quit IRC18:24
*** akuznetsov has joined #tripleo18:24
*** akuznetsov has quit IRC18:25
*** shivrao has joined #tripleo18:28
*** akuznetsov has joined #tripleo18:28
*** akuznetsov has quit IRC18:28
*** bvandenh has joined #tripleo18:30
*** bvandenh_ has joined #tripleo18:31
*** sthillma has quit IRC18:34
*** bvandenh has quit IRC18:35
openstackgerritgreghaynes proposed openstack/diskimage-builder: Install docker for tests  https://review.openstack.org/27802118:35
*** bnemec changes topic to "TripleO | CI status: http://tripleo.org/cistatus.html | Docs: http://tripleo.org/"18:37
*** bvandenh_ has quit IRC18:37
openstackgerritgreghaynes proposed openstack/diskimage-builder: Fail functests if refusing to run tests  https://review.openstack.org/17700218:38
*** akuznetsov has joined #tripleo18:39
*** masco has quit IRC18:41
*** eil397 has quit IRC18:41
*** mcornea has quit IRC18:42
openstackgerritgreghaynes proposed openstack/diskimage-builder: Dont pass xattrs to tar if its unsupported  https://review.openstack.org/17755918:42
*** trown is now known as trown|lunch18:42
*** akuznetsov has quit IRC18:44
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add tests for building *-minimal images  https://review.openstack.org/18116218:44
SlickNikWe have now moved on to: https://etherpad.openstack.org/p/trove-mitaka-sprint-future18:45
*** rhallisey has quit IRC18:45
*** rhallisey has joined #tripleo18:48
openstackgerritgreghaynes proposed openstack/diskimage-builder: Dont pass xattrs to tar if its unsupported  https://review.openstack.org/17755918:49
openstackgerritgreghaynes proposed openstack/diskimage-builder: Properly fail/trap in eval_run_d  https://review.openstack.org/25973418:55
*** tiswanso has quit IRC19:01
*** tiswanso has joined #tripleo19:02
*** bvandenh_ has joined #tripleo19:04
*** eil397 has joined #tripleo19:06
*** akuznetsov has joined #tripleo19:07
*** akuznetsov has quit IRC19:07
*** akuznetsov has joined #tripleo19:08
*** dtantsur is now known as dtantsur|afk19:09
*** akuznetsov has quit IRC19:09
*** akuznetsov has joined #tripleo19:13
*** akuznetsov has quit IRC19:14
xarsesdmsimard:  should we get the outstanding puppet-ceph tripleo tests killed? 4h,20m and 3h,23 min are never going to finish https://jenkins07.openstack.org/job/gate-tripleo-ci-f22-nonha/639/ https://jenkins07.openstack.org/job/gate-tripleo-ci-f22-nonha/642/19:16
*** penick has joined #tripleo19:17
dmsimardxarses: I'm not very knowledgeable about those jobs, EmilienM or trown|lunch would know19:17
dmsimardxarses: Can you even kill jobs that are in progress ? I've never had to do that before.19:18
xarsesIIRC infra can19:18
dmsimardSurely there's an enforced timeout on the job19:19
* dmsimard shrugs19:19
xarsesI would have figured 2h - 2h, 30m based on the average 1h 35m runtime19:19
*** akuznetsov has joined #tripleo19:20
*** akuznetsov has quit IRC19:25
xarsesdmsimard: hmm, nvm they have not been running that long, they have been waiting that long19:29
EmilienMdmsimard, xarses: they don't slow you down19:29
EmilienMFYI they are not voting and you don't need them to merge any patch19:29
*** eil397 has quit IRC19:30
*** eil397 has joined #tripleo19:31
*** jtomasek_ has joined #tripleo19:31
*** eil397 has quit IRC19:33
*** eil397 has joined #tripleo19:36
*** ayoung has joined #tripleo19:37
*** mkovacik has joined #tripleo19:37
*** panda has quit IRC19:37
*** panda has joined #tripleo19:38
*** akuznetsov has joined #tripleo19:41
*** akuznetsov has quit IRC19:41
*** akuznetsov has joined #tripleo19:42
*** akuznetsov has quit IRC19:43
*** trown|lunch is now known as trown19:44
*** akuznetsov has joined #tripleo19:45
*** akuznetsov has quit IRC19:50
*** akuznetsov has joined #tripleo19:54
*** akuznetsov has quit IRC19:55
*** akuznets_ has joined #tripleo19:55
*** akuznetsov has joined #tripleo19:57
*** dprince has quit IRC19:59
*** akuznets_ has quit IRC20:00
*** dprince has joined #tripleo20:00
*** akuznetsov has quit IRC20:01
*** ifarkas has quit IRC20:02
*** eil397 has quit IRC20:06
*** rcernin has joined #tripleo20:08
*** tiswanso has quit IRC20:14
*** tiswanso has joined #tripleo20:15
*** Marga_ has quit IRC20:23
*** NobodyCa1 has joined #tripleo20:34
jidargah, gfidente left.. anybody know what he/she meant when they suggested role specific extraconfigpost?20:37
*** hjensas has joined #tripleo20:37
*** hjensas has quit IRC20:37
*** hjensas has joined #tripleo20:37
*** NobodyCa1 has quit IRC20:41
trowndmsimard: does the error posted here fit with the nova-neutron keystonev3 issue, or do you think it is something else? https://etherpad.openstack.org/p/delorean_master_current_issues20:44
dmsimardtrown: that's exactly it yeah20:44
*** weshay_xchat has quit IRC20:45
dmsimardtrown: see https://review.openstack.org/#/c/276932/5/manifests/network/neutron.pp20:45
trowndmsimard: cool, so looks like tripleo will need both an instack-undercloud and tripleo-heat-templates patch for that too20:46
trownI will start hacking on the undercloud part20:46
dmsimardtrown: the packstack patch looks fairly simple https://review.openstack.org/#/c/277867/20:46
dmsimardtrown: OPM was just updated explicitely to carry the puppet-nova patch but it's not built by delorean yet (last I check)20:46
dmsimardso keep that in mind20:47
trownah, so I need that puppet-nova too?20:47
trownmakes sense...20:48
trownpinche OPM20:48
*** NobodyCa1 has joined #tripleo20:48
*** deva_ has joined #tripleo20:50
jidarI'm looking @ https://gist.github.com/b1322dd307acd7eb7e82, and I don't see how it's possible to filter on any compute host controller host when deploying the post-config stuff?20:50
jidarall I see available is NodeExtraConfigPost, so how do I hook that in as a controller based deploy or a compute based deploy20:51
jidarI mean, I guess I could just write out a huge bash script that parses the hostname but that's piss poor design :(20:53
*** NobodyCa1 has quit IRC20:53
*** dsneddon_brno has quit IRC20:54
*** Goneri has quit IRC20:54
*** deva_ has quit IRC20:54
dmsimardtrown: I'll let you know when opm has built20:58
*** akuznetsov has joined #tripleo20:58
*** rlandy has quit IRC20:58
*** Marga_ has joined #tripleo20:59
trowndmsimard: cool, for hacking purposes I will just use the puppet-nova module from source20:59
trowndmsimard: looks like the instack-undercloud patch will be fairly simple, although I should probably take the opportunity to move that config to hiera20:59
dmsimardtrown: just don't make it too complicated so it takes a long time to merge :)21:00
dmsimarddo a follow up patch if necessary21:00
*** rlandy has joined #tripleo21:00
trowndmsimard: lol, ya I would not even do it if I didnt think it would get -1 if I did not :)21:00
*** eil397 has joined #tripleo21:01
*** akuznetsov has quit IRC21:03
*** eil397 has quit IRC21:08
*** Goneri has joined #tripleo21:09
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Increase size of connection tracking table  https://review.openstack.org/27576921:21
*** weshay_xchat has joined #tripleo21:23
trownis there a way to use a DIB_INSTALLTYPE variable to install a single puppet module from master? or is it all source or all OPM?21:23
*** akuznetsov has joined #tripleo21:24
*** dprince has quit IRC21:31
*** lblanchard has quit IRC21:34
*** tiswanso has quit IRC21:37
bnemectrown: I believe it should be possible to install single module from source.  See the second Source admonition in step 5 on http://docs.openstack.org/developer/tripleo-docs/installation/installation.html#installing-the-undercloud21:41
trownbnemec: ah cool, I think that may not work, but good to know it is supposed to :p21:42
*** tiswanso has joined #tripleo21:43
openstackgerritEthan Gafford proposed openstack/tripleo-heat-templates: Trove Integration  https://review.openstack.org/23324021:49
*** Marga_ has quit IRC21:49
openstackgerritSteve Baker proposed openstack/tripleo-common: Convenience methods in base model class  https://review.openstack.org/26882121:50
openstackgerritSteve Baker proposed openstack/tripleo-common: Implement ViewManager list_software_deployments  https://review.openstack.org/26882221:50
openstackgerritSteve Baker proposed openstack/tripleo-common: Implement ViewManager list_stack_failures  https://review.openstack.org/27054621:50
openstackgerritSteve Baker proposed openstack/tripleo-common: Implement ViewManager show_software_deployment  https://review.openstack.org/27054521:50
*** Marga_ has joined #tripleo21:50
openstackgerritSteve Baker proposed openstack/python-tripleoclient: openstack overcloud software deployment show  https://review.openstack.org/27111321:51
openstackgerritSteve Baker proposed openstack/python-tripleoclient: openstack overcloud failures  https://review.openstack.org/27111421:51
openstackgerritSteve Baker proposed openstack/python-tripleoclient: openstack overcloud software deployment list  https://review.openstack.org/25267021:51
stevebakerhey, there has been a complete lack of reviews on these changes ^. I think they provide a huge UX improvement, but only if they, you know, land21:52
trownbnemec: yep... export DIB_INSTALLTYPE_puppet_nova=source did not get me puppet_nova from source :(21:53
trownstevebaker: ya I think we have unfortunately had some pretty major CI outages that are still be unraveled21:53
*** akuznetsov has quit IRC21:54
stevebakertrown: ok, as long as people know those changes are there. fwiw they are low risk of regression changes - they just make GET calls and format the results21:55
trownstevebaker: ya it has been on my list to give them a go for a while now21:56
*** Marga_ has quit IRC21:57
*** Marga_ has joined #tripleo21:57
dmsimardtrown: is keystone v3 even setup in tripleo ?21:58
*** jpeeler has quit IRC21:58
trowndmsimard: ya21:58
trownbnemec: slagle, I think I found an ill effect of https://github.com/openstack/tripleo-puppet-elements/commit/4b20834b702700612c473b70940d3b69656aa63b21:59
dmsimardtrown: okay... tried to find things v3 or domain related and didn't find them21:59
openstackgerritJames Slagle proposed openstack/os-cloud-config: Add []'s around ipv6 address when creating keystone clients  https://review.openstack.org/27811121:59
trownhmm... although without that, it is not possible to rerun undercloud install... that -f does overwrite the link from doing a single source puppet install though22:03
trowndeleting the module you want to get from source from /usr/share/openstack-puppet/modules works though :)22:05
*** egafford has quit IRC22:05
dmsimardlol.. yeah, been there done that22:05
dmsimardrm -rf <module>; git clone <module>22:05
* trown cant wait until that split is complete22:06
*** ooolpbot has joined #tripleo22:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged]22:10
dmsimardtrown: all those get_input, where are they fetched from ? https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/compute.yaml#L50322:13
trowndmsimard: hehe, welcome to THT :) https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/compute.yaml#L61522:15
dmsimardthat doesn't exactly help me, how do I know what's the value "hidden" in get_param: [EndpointMap, KeystoneAdmin, uri] ?22:16
trownwhich comes from https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/compute.yaml#L270-L274, which points out that the EndpointMap is set via the resource registry22:16
trownhaving fun yet? :)22:17
dmsimardyeah okay I just found the endpoint map thing22:17
trownhttps://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.yaml is where the top level resource registry is22:18
dmsimardwhat did I get myself into22:19
trownturtles all the way down22:19
dmsimardlooks like there's no mention of keystone v322:21
dmsimardI probably need to create v3 UriSuffix'd endpoints ?22:22
trowndmsimard: what I was thinking was we could just do all of the nova-neutron stuff via hieradata like https://review.openstack.org/#/c/239109/22:23
dmsimardyeah, see the comment that confirms what I was saying https://review.openstack.org/#/c/239109/4/puppet/hieradata/controller.yaml22:25
trownI tested that patch and it worked though...22:26
dmsimardokay, I'll hack something together then22:26
trownwoot Overcloud Deployed22:27
*** rhallisey has quit IRC22:27
trownnot that it will function without fixing THT, but at least that confirms the undercloud patch I made works22:27
trowndmsimard: thanks for delving into tripleo :)22:28
*** julim has quit IRC22:32
*** trown is now known as trown|outtypewww22:32
*** dsneddon_brno has joined #tripleo22:34
openstackgerritDavid Moreau Simard proposed openstack/tripleo-heat-templates: Update nova::network::neutron variables to drop deprecated parameters  https://review.openstack.org/27813422:39
dmsimardEmilienM: There doesn't seem to be any keystone v3 endpoint "mapped" in tht (context: nova-neutron v3 changes). Do you think I should add keystone v3 entries in the map with an urisuffix v3 to be able to use it ? Here: https://github.com/openstack/tripleo-heat-templates/blob/master/network/endpoints/endpoint_map.yaml#L238-L26922:41
dmsimardbnemec: would you have an opinion on ^ ? nova-neutron needs to be configured to use keystone v3 in mitaka22:46
dmsimardi.e https://github.com/openstack/puppet-nova/commit/d09868a59c451932d67c66101b725182d7066a1422:46
bnemecdmsimard: I actually thought we were already talking keystone v3 for notifications: https://github.com/openstack/tripleo-heat-templates/commit/56a141cacbbc76bdd909c02e0e670a97676cb21722:49
*** pradk has quit IRC22:49
dmsimardbnemec: Hence why I'm poking you :P trown reported the issue in point 2 https://etherpad.openstack.org/p/delorean_master_current_issues22:50
dmsimard(We had to fix this explicitely to point to /v3 in puppet-openstack-integration tests as well as packstack)22:50
dmsimardIs it expected that the default is v3 when there is no suffix to the endpoint or something ?22:51
bnemecdmsimard: I thought shardy had said before that in theory everyone should be using unversioned keystone endpoints and letting the client handle version selection.22:51
bnemecUnfortunately he's not around. :-/22:52
dmsimardDo we have an unversioned endpoint in undercloud as well ? That's where trown got the trace in the etherpad22:52
openstackgerritJohn Trowbridge proposed openstack/instack-undercloud: Update nova-neutron configuration  https://review.openstack.org/27813722:52
dmsimard(I'm really not familiar with tripleo, just trying to help getting CI to finally pass)22:53
dmsimardwell, there's trown's undercloud review22:53
*** tiswanso has quit IRC22:54
*** trown|outtypewww is now known as trown22:55
dmsimardbnemec: this is the original bug where I reported the issue https://bugs.launchpad.net/puppet-nova/+bug/154248622:56
openstackLaunchpad bug 1542486 in puppet-nova "nova-compute stack traces with BadRequest: Specifying 'tenant_id' other than authenticated tenant in request requires admin privileges" [Critical,New] - Assigned to Emilien Macchi (emilienm)22:56
dmsimardnotice the unversioned endpoint in the config (provided by puppet-openstack-integration) that yields the error while devstack's "hardcoded" /v3 works22:57
trownI am actually still here for a bit :)... dmsimard I did not actually test the overcloud after I got it working, I am testing the patch above now, and will actually pingtest the overcloud this time to see if we even have anything broken22:57
bnemecWell, there's this: https://review.openstack.org/#/c/248500/12/elements/puppet-stack-config/puppet-stack-config.yaml.template22:57
*** rcernin has quit IRC22:58
bnemecWhich is when we switched the undercloud to v3.  It looks like notifications weren't previously using those hiera values though. :-/22:58
dmsimardbnemec: the problem with THT is all those get_params, is there a way to feed them a /v3 without adding v3 endpoints in the map ?22:59
bnemecOh hey, that's where shardy made the comment about unversioned endpoints too.22:59
bnemecGood to know I didn't imagine that.23:00
*** dsneddon_brno has quit IRC23:00
bnemecdmsimard: I think you'd have to add explicit KeystoneV3 endpoints.  If you do, I'd prefer to base it on https://review.openstack.org/#/c/275437/ so we don't have a giant merge conflict waiting to happen.23:02
dmsimardoh man that review is nuts23:02
dmsimardok :p23:02
bnemecdmsimard: Most of it is generated though. :-)23:04
openstackgerritDavid Moreau Simard proposed openstack/tripleo-heat-templates: Update nova::network::neutron variables to drop deprecated parameters  https://review.openstack.org/27813423:05
trownwhoa, that is an awesome patch23:05
dmsimardbnemec: how close are we to merging that realistically23:06
dmsimardbnemec: only concern I have with basing my patch on top of that is takes a long time to merge and I depend on it23:06
bnemecdmsimard: I don't know.  CI appears to be completely hosed again.23:07
dmsimardtrown: I'll try to send a patch in tomorrow morning, taking off ttyl23:10
*** ooolpbot has joined #tripleo23:10
openstackLaunchpad bug 1543810 in tripleo "Jenkins errors killing many CI jobs" [Critical,Triaged]23:10
*** ooolpbot has quit IRC23:10
trowndmsimard: thanks, I will also try to dig into it tomorrow. have a good night23:11
*** thrash is now known as thrash|g0ne23:12
*** jtomasek_ has quit IRC23:12
*** mkovacik has quit IRC23:13
*** penick has quit IRC23:18
*** mkovacik has joined #tripleo23:19
openstackgerritJohn Trowbridge proposed openstack/instack-undercloud: Update nova-neutron configuration  https://review.openstack.org/27813723:21
*** hjensas has quit IRC23:23
*** alop has quit IRC23:28
*** bvandenh_ has quit IRC23:29
*** panda has quit IRC23:37
*** panda has joined #tripleo23:38
*** eil397 has joined #tripleo23:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!