Friday, 2016-09-16

*** Sarath has joined #tripleo00:05
*** mlupton has quit IRC00:11
*** mlupton has joined #tripleo00:14
*** tosky has quit IRC00:16
*** mlupton has quit IRC00:18
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: [mitaka-only] mysql: never add brackets to mysql_bind_host
*** mlupton has joined #tripleo00:21
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Convert deploy steps to jinja2 loop
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Convert UpdateWorkflow to support composable roles
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Convert AllNodesExtraConfig to support composable roles
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Add fluentd client service
EmilienMlarsks: ok, let's try again :)00:24
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: Migrate to using osc-lib
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: mysql: never add brackets to mysql_bind_host
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: get_host_info: get repos list
*** mlupton has quit IRC00:39
*** Sarath has quit IRC00:44
*** skramaja_ has joined #tripleo00:47
*** skramaja has quit IRC00:47
*** mlupton has joined #tripleo00:50
*** mlupton has quit IRC00:52
*** dmacpher has quit IRC00:54
*** jcoufal has joined #tripleo00:55
*** bana_k has quit IRC01:14
*** mlupton has joined #tripleo01:32
*** mlupton has quit IRC01:36
*** kbyrne has quit IRC01:40
*** dmacpher has joined #tripleo01:43
*** kbyrne has joined #tripleo01:47
*** fultonj has quit IRC01:48
*** limao has joined #tripleo01:58
*** thrash is now known as thrash|g0ne01:59
*** egafford has quit IRC02:08
*** trozet has quit IRC02:11
*** jlinkes has quit IRC02:15
*** kbyrne has quit IRC02:15
*** kbyrne has joined #tripleo02:16
*** mlupton has joined #tripleo02:17
*** limao has quit IRC02:18
*** tzumainn has quit IRC02:18
*** limao has joined #tripleo02:19
*** jlinkes has joined #tripleo02:23
ayoungEmilienM, credentials looks good.  Think we can risk un-pegging Keystone02:24
*** trozet has joined #tripleo02:25
*** mlupton has quit IRC02:27
*** absubram has quit IRC02:28
*** mlupton has joined #tripleo02:31
*** absubram has joined #tripleo02:40
*** mlupton has quit IRC02:41
*** absubram has quit IRC02:51
*** jlinkes has quit IRC02:52
*** jlinkes has joined #tripleo02:52
*** pmannidi has quit IRC02:59
*** pmannidi has joined #tripleo02:59
dtrainorI'm trying to introspect some nodes.  I see the ipxe dialogue, they start to come up, and then they infinitely fail trying to download agent.kernel03:02
*** mlupton has joined #tripleo03:09
dtrainor bug 1364079 in ipxe "iPXE hangs with an infinite stream of different errors" [Unspecified,Closed: worksforme] - Assigned to rhos-maint03:11
*** mlupton has quit IRC03:15
*** TicToc has quit IRC03:20
*** TicToc has joined #tripleo03:21
*** limao has quit IRC03:25
*** jlinkes has quit IRC03:48
*** bana_k has joined #tripleo04:00
*** jlinkes has joined #tripleo04:04
*** TicToc has quit IRC04:21
*** TicToc has joined #tripleo04:25
*** rcernin has quit IRC04:37
*** jlinkes has quit IRC04:42
*** absubram has joined #tripleo04:43
*** absubram_ has joined #tripleo04:44
*** absubram has quit IRC04:48
*** absubram_ is now known as absubram04:48
*** absubram has quit IRC04:50
*** jaosorior has joined #tripleo04:57
*** jlinkes has joined #tripleo04:58
*** radeks has joined #tripleo05:04
*** radeks has quit IRC05:07
*** mlupton has joined #tripleo05:12
*** bvandenh has quit IRC05:12
*** absubram has joined #tripleo05:13
*** absubram_ has joined #tripleo05:14
*** fragatina has quit IRC05:15
*** mlupton has quit IRC05:16
*** absubram has quit IRC05:18
*** absubram_ is now known as absubram05:18
*** pmannidi has quit IRC05:21
*** mpsairam has joined #tripleo05:24
*** TicToc has quit IRC05:25
*** TicToc has joined #tripleo05:26
*** mlupton has joined #tripleo05:29
*** pmannidi has joined #tripleo05:30
*** skramaja_ is now known as skramaja05:34
*** bvandenh_ has joined #tripleo05:38
*** dsariel has joined #tripleo05:38
*** rlandy|bbl is now known as rlandy05:39
*** benoit has quit IRC05:40
*** pgadiya has joined #tripleo05:40
*** rcernin has joined #tripleo05:43
*** bana_k has quit IRC05:43
*** benoit has joined #tripleo05:44
*** benoit has quit IRC05:49
*** benoit has joined #tripleo05:50
*** bana_k has joined #tripleo05:55
*** benoit has quit IRC05:56
*** jlinkes has quit IRC05:57
*** jlinkes has joined #tripleo05:57
*** benoit has joined #tripleo05:58
*** benoit has quit IRC06:03
*** benoit has joined #tripleo06:05
*** rajinir has quit IRC06:05
*** benoit has quit IRC06:10
*** benoit has joined #tripleo06:11
*** rcernin has quit IRC06:14
*** benoit has quit IRC06:17
*** benoit has joined #tripleo06:18
*** rcernin has joined #tripleo06:19
*** flepied has quit IRC06:19
*** jprovazn has joined #tripleo06:20
*** rasca has joined #tripleo06:20
*** pcaruana has joined #tripleo06:23
*** benoit has quit IRC06:24
*** benoit has joined #tripleo06:25
*** TicToc has quit IRC06:26
*** absubram has quit IRC06:29
*** fragatina has joined #tripleo06:29
*** aufi has joined #tripleo06:29
*** TicToc has joined #tripleo06:30
*** jlinkes has quit IRC06:30
*** pmannidi has quit IRC06:32
*** fragatina has quit IRC06:33
*** benoit has quit IRC06:38
*** jbadiapa has joined #tripleo06:38
*** benoit has joined #tripleo06:38
*** pmannidi has joined #tripleo06:42
*** jlinkes has joined #tripleo06:42
*** matbu|brb is now known as matbu06:43
*** benoit has quit IRC06:44
*** benoit has joined #tripleo06:45
*** bana_k has quit IRC06:45
*** benoit has quit IRC06:51
*** jtomasek_ has joined #tripleo06:52
*** benoit has joined #tripleo06:52
*** dciabrin|away has quit IRC06:55
*** florianf has joined #tripleo06:57
dtrainorI have some nodes in ironic that I can't delete.  When I try, I'm told:  Failed to delete node 77b3a651-b395-4d70-85c4-a7053d725899: Node 77b3a651-b395-4d70-85c4-a7053d725899 is associated with instance bdaf8737-14b3-4ee5-94bb-724407db9df3. (HTTP 409)06:57
dtrainorI don't know which Instance UUID displayed in 'ironic node-list' this is referring to06:58
*** benoit has quit IRC06:58
dtrainorThe only thing I could think of which it would be referring to is a deployment or stack in heat - I have neither of those.06:59
*** david-lyle_ has joined #tripleo07:00
*** david-lyle has quit IRC07:00
*** benoit has joined #tripleo07:00
*** flepied has joined #tripleo07:03
*** flepied has quit IRC07:04
*** dtantsur|afk is now known as dtantsur07:05
*** flepied has joined #tripleo07:06
*** dciabrin|away has joined #tripleo07:09
*** benoit has quit IRC07:12
*** mcornea has joined #tripleo07:12
*** Jokke_ has quit IRC07:13
*** benoit has joined #tripleo07:13
*** bandini has joined #tripleo07:16
*** dciabrin|away is now known as dciabrin07:17
*** dmacpher has quit IRC07:17
*** rhefner has quit IRC07:18
*** adarazs has quit IRC07:19
*** zoli_gone-proxy has quit IRC07:19
*** cschwede has quit IRC07:19
*** psanchez has quit IRC07:20
*** jmiu has quit IRC07:20
*** marios has quit IRC07:20
*** stevebaker has quit IRC07:20
*** gchamoul has quit IRC07:20
*** tobias_fiberdata has joined #tripleo07:20
*** myoung|bbl has quit IRC07:20
*** jschlueter has quit IRC07:21
*** slagle has quit IRC07:21
*** psanchez has joined #tripleo07:21
*** slagle has joined #tripleo07:22
d0ugalHow is CI lookin'?07:22
*** marios has joined #tripleo07:22
*** myoung has joined #tripleo07:22
*** zoli_gone-proxy has joined #tripleo07:22
*** stevebaker has joined #tripleo07:23
*** tobias-fiberdata has quit IRC07:23
*** jmiu has joined #tripleo07:23
shadowerd0ugal: paints a lovely red picture07:23
*** adarazs has joined #tripleo07:23
*** jlinkes has quit IRC07:23
*** gchamoul has joined #tripleo07:23
*** jlinkes has joined #tripleo07:23
d0ugalshadower: yay07:24
*** jschlueter has joined #tripleo07:24
*** cschwede has joined #tripleo07:25
*** absubram has joined #tripleo07:25
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Verify that the Deployment Plan creation was successful
*** jpena|off is now known as jpena07:26
openstackgerritDougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names.
*** david-lyle has joined #tripleo07:28
*** david-lyle_ has quit IRC07:29
*** akuznetsov has joined #tripleo07:29
*** TicToc has quit IRC07:30
*** abehl has joined #tripleo07:32
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Pass the timeout to the deploy workflow
*** TicToc has joined #tripleo07:34
*** tobias_fiberdata has quit IRC07:35
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove the environments from Mistral when removing from Swift
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Add an optional timeout when waiting for websocket messages
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove the get_hiera_key function
*** pmannidi_ has joined #tripleo07:36
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Update the Mistral action names
*** pmannidi has quit IRC07:37
*** jtomasek_ has quit IRC07:39
*** florianf_ has joined #tripleo07:39
d0ugalWhen patches are proposed to master, do people propose them to newton at the same time?07:40
*** panda|Zz is now known as panda07:40
d0ugalI ask, because I have been waiting until they are about to merge - but that often means I do it late if they merge when I am not around07:40
d0ugaland then EmilienM beats me to it and I bet he is mad that I'm not doing it :)07:40
*** florianf has quit IRC07:44
*** florianf_ is now known as florianf07:44
*** florianf_ has joined #tripleo07:44
mariosd0ugal: yeah depends on the case, but i have in the passed simultanously proposed to stable, but then -1 it so is clear we are waiting for master first07:45
mariospast even wow07:45
d0ugalmarios: k, thanks - I guess I should start doing that07:46
*** athomas has joined #tripleo07:48
*** tobias_fiberdata has joined #tripleo07:49
shadowerwait, so any unmerged newton patches must be submitted against the newton branch, too?07:50
jaosoriorshadower: for python-tripleo and tripleo-common07:50
jaosoriort-h-t and puppet-tripleo are not branched yet07:50
d0ugalshadower: EmilienM has probably been doing all of yours too lol07:51
shadowerah okay07:51
* shadower will have a look07:51
mariosshadower: d0ugal you've been emilien'd07:51
d0ugalmarios: actually, has tripleo-common been branched?07:51
shadowerwhat about instack-undercloud?07:51
d0ugalnow I am confused.07:52
d0ugalMaybe it is just tripleoclient07:52
mariosd0ugal: i am not sure.. i thought as jaosorior said, common and pyuthon-tripleo but github can quickly answer your question...07:52
shadowerdoesn't appear so:
d0ugalSeems -common doesn't have a newton branch yet.07:52
shadowerall I see is liberty and mitaka07:52
marios no newton here yet afaics07:52
shadowerphew :-)07:53
d0ugalI don't understand why we branched the client only? Isn't that normally the last to be done :/07:53
jaosoriord0ugal: supposedly libraries come first07:53
*** tobias-fiberdata has joined #tripleo07:53
*** florianf_ has quit IRC07:53
d0ugaljaosorior: oh, I guess that sorta makes sense. I am not aware of anyone using it as a library tho' - so maybe we can remove that status?07:54
d0ugaltripleo-common is more of a library07:54
d0ugaltripleoclient isn't even in global-requirements :)07:54
*** tobias_fiberdata has quit IRC07:56
*** yamahata has joined #tripleo07:57
jaosoriord0ugal: gotta talk to shardy about it I guess07:58
d0ugalYeah, probably a bit late :)07:58
*** hjensas has joined #tripleo07:58
*** shardy has joined #tripleo07:58
*** akuznetsov has quit IRC08:00
*** jpich has joined #tripleo08:01
*** jkraj has joined #tripleo08:01
*** gfidente has joined #tripleo08:02
*** ohamada has joined #tripleo08:03
*** openstackgerrit has quit IRC08:03
*** openstackgerrit has joined #tripleo08:04
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Update the Mistral action names
*** ccamacho has joined #tripleo08:09
*** fragatina has joined #tripleo08:15
*** dtantsur is now known as dtantsur|bbl08:16
openstackgerritDougal Matthews proposed openstack/tripleo-common: Add template processing to the update plan workflow.
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fix the default plan creation
openstackgerritDougal Matthews proposed openstack/tripleo-common: Return the result of create_plan in create_deployment_plan workflow
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Verify that the Deployment Plan creation was successful
*** fragatina has quit IRC08:19
*** lucas-dinner is now known as lucasagomes08:21
*** yamahata has quit IRC08:21
*** dbecker has quit IRC08:24
*** derekh has joined #tripleo08:26
*** dbecker has joined #tripleo08:28
*** _milan_ has joined #tripleo08:30
*** TicToc has quit IRC08:35
*** shardy has quit IRC08:40
*** TicToc has joined #tripleo08:42
*** saneax-_-|AFK is now known as saneax08:42
*** electrofelix has joined #tripleo08:43
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fix the default plan creation
openstackgerritDougal Matthews proposed openstack/tripleo-common: Return the result of create_plan in create_deployment_plan workflow
openstackgerritDougal Matthews proposed openstack/tripleo-common: Add template processing to the update plan workflow.
*** ohamada has quit IRC08:57
*** ohamada_ has joined #tripleo08:57
*** ohamada_ has quit IRC08:57
*** ohamada_ has joined #tripleo08:58
openstackgerritMerged openstack/tripleo-heat-templates: Populate vnc_api_lib.ini on compute nodes with OpenContrail
openstackgerritMerged openstack/python-tripleoclient: Updated from global requirements
openstackgerritSaravanan KR proposed openstack/os-net-config: Add mac address to the DPDK mapping file
*** tobias-fiberdata has quit IRC09:09
*** ooolpbot has joined #tripleo09:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1624274 in tripleo "CI: OVB jobs consistently fail to get environments" [Critical,Triaged]09:10
*** pmannidi_ has quit IRC09:12
derekhjistr: fyi, I restarted rabbit on the rh1 controller, I think that has worted the problem, see the comment I added to your bug09:13
jistrderekh: awesome, thanks!09:14
derekhjistr: that was over an hour ago, so we should see some passes soon09:15
*** abregman has joined #tripleo09:17
openstackgerritMerged openstack/tripleo-ui: Update version to match current release
shadowerderekh: should we start trying rechecks or wait a bit longer?09:21
shadowermine failed but that was about 2 hrs ago09:21
derekhshadower: a couple wouldn't do any harm but I wouldn't go crazy, the jobs are now getting testenvs, to that problem is solved09:22
derekhshadower: but now I'm looking at errors creating the overcloud09:22
derekh2016-09-16 09:02:45.558651 | 2016-09-16 09:02:42Z [CephStorage]: CREATE_FAILED  ResourceInError: resources.CephStorage.resources[0].resources.CephStorage: Went to status ERROR due to "Message: Unknown, Code: Unknown"09:22
shadoweroh that's a fantastic error explanation09:23
* shadower has two +Ad patches that are waiting for a gate pass for days now. I'll try reverifying one09:23
derekh2016-09-16 09:03:15.000 | | eac4cf4b-23fa-4333-bd3c-813c30832378 | overcloud-cephstorage-0 | ERROR  | -          | NOSTATE     |                     |09:24
derekhwell that would do it09:24
*** sshnaidm has quit IRC09:25
*** shardy has joined #tripleo09:26
tbarronmarios: when you get a chance, needs a little syntax fix09:27
*** sshnaidm has joined #tripleo09:27
mariostbarron: thanks ack gimme 2 mins will sort it out09:27
tbarronmarios: np, i know you are doing many things :)09:28
*** dtantsur|bbl is now known as dtantsur09:28
openstackgerritFlorian Fuchs proposed openstack/tripleo-ui: Migrate Deploy action to Mistral
openstackgerritSaravanan KR proposed openstack/os-net-config: Add mac address to the DPDK mapping file
jaosorioraww... still seeing failures getting environment in ovb :(09:48
openstackgerritMerged openstack/tripleo-heat-templates: Unset Keystone public_endpoint
openstackgerritMerged openstack/python-tripleoclient: Use the hexdigest of the path to make the filename unique in swift
openstackgerritMerged openstack/tripleo-quickstart: Remove external requirements
*** tobias-fiberdata has joined #tripleo09:50
derekhjaosorior: can you point me at one09:53
openstackgerritMerged openstack/python-tripleoclient: Add `openstack overcloud plan deploy`
jaosoriorderekh: and
*** tosky has joined #tripleo09:56
derekhjaosorior: thanks09:56
*** akrivoka has joined #tripleo09:58
derekhjaosorior: 2 different errors   looking09:59
openstackgerritMarios Andreou proposed openstack/puppet-tripleo: Add manila-netapp backend to manila class and tidy up generic
openstackgerritMarios Andreou proposed openstack/puppet-tripleo: Fixup manila-cephfs native backend defaults
*** zoli_gone-proxy is now known as zoliXXL10:03
openstackgerritMerged openstack/python-tripleoclient: Tripleoclient leaks temporary files
*** tobias_fiberdata has joined #tripleo10:08
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Update the Mistral action names
*** bvandenh_ has quit IRC10:08
d0ugalderekh: The second is the error we had before in CI10:08
d0ugalderekh: we now know that to be /cc therve10:09
openstackLaunchpad bug 1624284 in Mistral "MessagingTimeout when executing mistral actions" [Undecided,Confirmed]10:09
*** kbyrne has quit IRC10:09
*** ooolpbot has joined #tripleo10:10
*** tobias-fiberdata has quit IRC10:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1624274 in tripleo "CI: OVB jobs consistently fail to get environments" [Critical,Triaged]10:10
derekhd0ugal: the second error I pasted ? that is an error on the RH1 cloud controller, not in a CI job10:10
d0ugalderekh: This one:
d0ugalderekh: oh :)10:10
therveYeah it doesn't seem to come from mistral10:10
d0ugalsorry, I read that too quickly10:11
derekhd0ugal: no prob10:11
*** kbyrne has joined #tripleo10:11
*** TicToc has quit IRC10:11
mariosgfidente: revisit please when yuo get a chance
openstackgerritMerged openstack/instack-undercloud: Introduce 'enable_validations' option
*** zoliXXL is now known as zoli|lunch10:18
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Pass the timeout to the deploy workflow
derekhok, I think all of the env stacks that failed overnight are causing extra load on things as heat had been trying to delete them (and their many resources) but failing,10:19
derekhcleaning things up now10:19
*** abregman has quit IRC10:21
*** skramaja has quit IRC10:22
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Add an optional timeout when waiting for websocket messages
openstackgerritFlorian Fuchs proposed openstack/tripleo-ui: Migrate Deploy action to Mistral
d0ugaltherve: So, you know how we create the default plan at install time?10:24
therved0ugal, Somewhat10:24
d0ugaltherve: The only reason that CI has been working, is because that fails10:24
d0ugalAs soon as I fix it, the problem comes back10:24
therveSo we really need a quick solution in mistral10:25
d0ugaltherve: Any ideas? :)10:25
d0ugaltherve: and I assume since we are seeing this in CI consistently - that there is a good chance users will hit it10:26
therveYeah it's a fundamental issue. Mistral doesn't hit it in its gate because there is not enough testing10:27
tbarronmarios: i deleted my overcloud stack, virt-customized the overcloud-full image again with this tiime, and attempted to redeploy but hit immediately - and 'heat stack-list' shows empty.10:27
tbarronmarios: i guess there is an issue with re-deploys?  I nuked all my vms and started with freshly updated packages and git clones this morning and10:28
therved0ugal, Using anything but the threading executor would make the CI happy, I believe10:28
therveWhether or not it's correct in the general is another issue10:28
tbarronmarios: can do that again, but just want to check real quick if there's a less drastic approach to picking up the latest update10:29
mariostbarron: not sure what happened with that paste... i was going to say perhaps the heat stac wasn't gone yet by te time you started a redeploy, so it thought it was updating?10:29
d0ugaltherve: I guess we could also do something like this: Very pseudocode-y, but hopefully makes sense.10:31
tbarronmarios: well, i did 'heat stack-delete'; 'watch heat stack-list' till i saw it deleted, then picked up the latest manila.pp, virt-customized overcloud, uploaded to glance, and re-deployed.  so the heat stack should have been gone.10:31
d0ugalbut that would be a big change for us at this point.10:31
therveYeah. And mistral would still be broken10:32
d0ugaltherve: :)10:32
tbarronmarios: and there are of course no overcloud.yaml and overcloud-without-mergepy.yaml in my THT, as i menationed everything was built  fresh this morning10:33
tbarronmarios: i dunno OOO well enough to tell who expects them to be there10:33
mariostbarron: well looks like it may be a client issue d0ugal do you have any idea what the issue is? tbarron deplyoed, deleted the stack, updated images and tries to deploy again but gets
*** thrash|g0ne is now known as thrash10:35
mariosd0ugal: is it stale plan data or something? looks like is getting /trying and failing to get something from swift?10:35
* marios caffeinne brb10:36
tbarronmarios: dougal exactly, and note in the paste messages about removing current and uploading new plan files.  I didn't see those on the first deploy atempt.10:36
tbarrondobson: marios the first deploy *did* have a 404 for overcloud-without-mergepy.yaml though10:37
tbarrons/dobson/dougal/ - sorry dobson10:37
tbarronah, d0ugal ^^^^  didn't see the '0', sorry10:38
tbarrond0ugal: marios so on the first deploy attempt there wasn't the stuff about removiing current plan and updating new plan, which makes sense10:39
tbarrond0ugal: marios but on the first deploy there *was* a 404 for overcloud-without-mergepy.yaml but no 404 for overcloud.yaml10:40
tbarrond0ugal: marios it just got that one 404 and kept on chugging10:40
d0ugaltbarron: I think that error can actually  be ignored.10:41
tbarrond0ugal: on the second deploy, as you see in there is a second series of 404s, for both of the old overcloud*.yaml files10:41
d0ugaltbarron: It's related to a change I think shardy made - it it looking for both the files in swift - we should change it to be more clearly debug info because everyone is asking about it :)10:41
tbarrond0ugal: well i did ignore it on the first deploy, on the second I paid attention because the deploy stopped instead of continuing on10:41
d0ugaltherve: did it stop with an error?10:42
d0ugaltbarron: oh, I see10:42
tbarrond0ugal: line 32 in that paste was the last i saw10:42
d0ugaltbarron: That is odd.10:42
tbarrond0ugal: that's why i was inclined to blame the 404s10:42
d0ugalYeah, I am more inclined to take them seriously now :)10:43
* d0ugal looks at the code10:43
tbarrond0ugal: and the fact that i didn't see the second set of 404s, for both the old overcloud*yaml fiiles, the firs time, only the single 404 for overcloud-without-mergepy.yaml10:43
mariosd0ugal: thanks ... tbarron i wouldn't nuke my env yet.. i mean yes you should be able to do this (redeploy) fine ! hanve't come across the issue you're seeing here before though.10:45
d0ugaltbarron: yeah, so this is where it is happening:
mariosd0ugal: could/would manually deleting the plan help? would a new one just be created on next attempt?10:45
d0ugalmarios: It might help and yes it would10:46
d0ugalmarios, tbarron: openstack overcloud plan delete overcloud10:46
*** adarazs is now known as adarazs_lunch10:47
tbarrondobson: marios done - and 'openstack overcloud plan list' shows empty - now just re-deploy?10:49
mariostbarron: yeah see what happens.. also sanity check all the env files etc you are passing10:49
mariostbarron: ironic nodes all available and no heat stack right?10:50
tbarrond0ugal: ^^ (i'm a slow learner, did your nick wrong again)10:50
d0ugaltbarron: lol, sorry for being awkward :)10:50
d0ugaltbarron: Yeah, just redeploy10:50
d0ugalI think we need to change this plan management stuff, causing too many problems10:51
d0ugalI really need some help with it from somebody that understands Heat better10:51
d0ugalWe have this bug which is related to it:
openstackLaunchpad bug 1622683 in tripleo "Updating plans breaks deployment" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal)10:51
tbarron'heat stack-list' is empty, 'ironic node-list' shows all 'baremetals' available10:52
tbarronk, here goes10:52
tbarronit's creatiing a new swift container for the plan, that looks better :)10:53
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fix the default plan creation
tbarron404 on overclous-without-mergepy.yaml but it is contiinuting on, like my first deploy10:54
tbarrond0ugal: marios thanks, over that hump and I didn't nuke everything!10:54
d0ugaltbarron: np, sorry for the plan related issues :(10:55
mariostbarron: AND its friday \o/10:55
mariosd0ugal: thanks :)10:55
* tbarron observes that in OOO it feels so good to quit hitting ones head on wall10:55
tbarron^^^ couldn't resist, that's true everywhere of course10:56
d0ugaltbarron: we require extra head hitting so that when you get beyong that it feels even better!10:56
tbarrond0ugal: rofl10:56
mariostbarron: sorry for the pain and it really is as bad as it gets, i mean with the puppet-tripleo so you need to inject the images too... i.e. not just grab templates and go10:57
mariostbarron: please keep fighting :) we are waiting for you to tell us it works10:58
mariostbarron: once we hear that i think we could land the series today or whenever we start landing things again10:58
jaosoriortbarron: or you could use swift to update the puppet manifest on the next overcloud deploy10:58
mariostbarron: we have at least one +2 everywhere i think now10:59
tbarronmarios: i will; i know that there are a lot of changes in flight and big infra stuff in OOO has happened this cycle, so i understand10:59
*** tobias_fiberdata has quit IRC11:00
tbarronmarios: i will declare victory just as soon as I can, minimum viable product is great with me.  I know rc1 is imminent :)11:00
tbarronjaosorior: thanks,, something to read during the deploy attempt :)11:00
mariostbarron: ack... yeah as longs as it works, we can tidy up or add stuff easily once base is in. plus we are dealing with a general backend tidy up AND adding netapp and cephfs backends so is  a lot there already.11:01
tbarronjaosorior: that looks cool, the next optimizations after virt-customize for faster workflow11:04
mariostbarron: regarding rc1 yes... i think we got a reprieve with the ci issues yesterday so would be awesome to get it in, otherwise we start to risk not landing (still couple more weeks but .... would be nice not to have to start backporting everything)11:05
*** ccamacho is now known as ccamacho|lunch11:05
tbarronmarios: ack11:05
gfidentemarios, here you mean basically replace true with an echo?11:09
*** lucasagomes is now known as lucas-hungry11:09
*** ooolpbot has joined #tripleo11:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1624274 in tripleo "CI: OVB jobs consistently fail to get environments" [Critical,Triaged]11:10
mariosgfidente: well only if we care... that was the question11:10
mariosgfidente: if we don't care if it pass/fail then it is fine like that11:10
gfidentemarios, thing is the directory might not exist at all11:12
*** tobias_fiberdata has joined #tripleo11:12
gfidentemarios, let's use an echo, I'd have to recheck it anywa11:12
gfidentethanks :)11:12
mariosjistr: can you see comment at - ill revote on the others too if need be (I was +2 on all 3 of those reverts)11:14
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Fixes the Ceph upgrade scripts
jaosoriorderekh: hey dude, could you check this commit out?
mariosgfidente: nice thanks voted11:16
mariostbarron: btw sounds like rc1 is bumped so you can go back to one spoon of coffee for now ;) thought you'd be please to hear
* tbarron pulls the syringe back out of his vein11:23
marioslol intravenous caffeinne is serious dedication11:24
jaosoriormarios: can you check this out?
*** paramite has joined #tripleo11:36
*** pgadiya has quit IRC11:38
*** fultonj has joined #tripleo11:38
jaosoriorshadower, jistr hey guys, if you have time can you check this commit out?
mariosjaosorior: ack11:39
derekhjolooking at it now, so is the ssl job broken?11:41
d0ugaltbarron: You'll be happy to hear I am hitting the swift 404 errors in CI :)11:42
jistrjaosorior: lgtm11:43
jaosoriorderekh: So, the current stuff works. But when trying to test zaqar websocket's behind HAProxy, which actually takes proxies into account, it keeps failing. Hoping that does the trick, cause it works locally (on different deployments)11:43
derekhjaosorior: ok11:43
tbarrond0ugal: well, i guess it's good not to feel alone, but sorry about CI11:44
derekhAs if CI wasn't bad enough /me has just delete 19 random ports from neutron on the RH1 overcloud by accident11:44
d0ugalEmilienM: Morning11:44
tbarronmarios: so now we hit a weird mongodb error, certainly unrelated to your changes:
*** tobias_fiberdata has quit IRC11:45
derekhEmilienM: howdy, testenvs were failing to get created overnight, I've been cleaning thing up so I think they are in better shape now11:46
tbarronmarios: Error: /Stage[main]/Tripleo::Profile::Base::Database::Mongodb/Mongodb_replset[tripleo]: Could not evaluate: rs.add() failed to add host to replicaset tripleo: replSetReconfig command must be sent to the current replica set primary.\u001b[0m\n"11:46
EmilienMderekh: thanks a lot11:46
*** abregman has joined #tripleo11:46
tbarronmarios: unfort it's early enough that manila.conf isn't getting updated and no attempt to start manila services: so i can't confirm that your patches are working yet11:47
mariostbarron: wow that entire paste has only one instance of 'error'... not seen that before... yeah it would be before the cluster services are brought up11:47
mariostbarron: i mean i am not sure why you are seeing that... may be worth browsing the recent bugs for tripleo?11:48
tbarronmarios: i wonder if anyone else is seeing that mongodb deploy error11:48
derekhEmilienM: at least I'm starting to seem some ovb jobs passing now, although I may have just cause some to fail by deleting some neutron ports I shouldn't have deleted11:48
tbarronmarios: yeah, i'll browse11:48
*** masco has quit IRC11:49
*** jpena is now known as jpena|lunch11:49
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Cleanup the previous plan when deploying
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fix the default plan creation
*** egafford has joined #tripleo11:51
derekhI'm off for the weekend before things start to go downhill
d0ugalderekh: nice!11:52
*** trown|outtypewww is now known as trown11:53
*** adarazs_lunch is now known as adarazs11:53
EmilienMderekh: can we close ?11:56
openstackLaunchpad bug 1624274 in tripleo "CI: OVB jobs consistently fail to get environments" [Critical,Triaged]11:56
derekhEmilienM: we can downgrade it I think, will come up with a more permanent solution later11:57
*** derekh changes topic to "TripleO : | | Meetings On Tuesdays at 14:00 UTC in #openstack-meeting-alt"11:57
derekhEmilienM: ack11:58
d0ugalgah, ipv6 is super annoying :)11:59
derekhbnemec: slagle bug 1624274  , I think we need something to consume the messages from ceilometer's queue, otherwise the whole thing just grows11:59
openstackbug 1624274 in tripleo "CI: OVB jobs consistently fail to get environments" [High,In progress]
derekhbnemec: slagle and I think causes use problems, as it must be slowing down the other queues11:59
*** dprince has joined #tripleo12:00
slaglederekh: i thought we'd shut down ceilometer?12:00
slaglewhy do we need it12:00
derekhbnemec: dprince slagle: yes thats the problem, things are writing to the queues that ceilometer normally consums, but nothing is consuming the messages12:00
derekhdprince: RE.
openstackLaunchpad bug 1624274 in tripleo "CI: OVB jobs consistently fail to get environments" [High,In progress]12:01
EmilienMderekh, slagle, dprince : fyi
EmilienMderekh: ah, we should maybe stop to send events on this queue12:01
slaglederekh: k, i understand now12:02
slaglewe had stopped it due to the high load on the controller12:02
EmilienMor set a low ttl to the messages12:02
derekhslagle: yup12:02
dprincederekh: so we need ceilometer then?12:02
*** jayg|g0n3 is now known as jayg12:03
derekhdprince: I think either turn back on ceilo or cron job to pruge the notify queues or the ttl EmilienM suggested12:04
*** ccamacho|lunch is now known as ccamacho12:04
EmilienMshardy: have you seen the failures on ?12:04
dprincederekh: we can leave it on then for now12:04
EmilienMshardy: sounds like transient12:04
derekhEmilienM: that  tripleo-cd-admin file is now deprecated, I'm not sure if its used anywhere any longer12:05
derekhEmilienM: you should add it here too
dprincederekh: we disabled it as part of the services we thought we didn't use12:05
EmilienMderekh: ok I will12:05
EmilienMshardy: what is worries me is - failures on scenario003 look valid12:05
shardyEmilienM: looking now - is confusing as it actually looks like the job worked fine12:06
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add Emilien Macchi ssh key to TripleO admins
shardyEmilienM: Yes, the fluentd patch failures do look real12:08
shardyI'll see if I can see where the problem is12:08
slaglederekh: which queues in rabbit?
shardythere's a syntax error somewhere in the service template outputs12:08
jistrEmilienM, shardy: also the HA job failure might be valid -- 31mError: /Stage[main]/Tripleo::Profile::Base::Database::Mongodb/Mongodb_replset[tripleo]: Could not evaluate: Can't find master host for replicaset tripleo12:08
EmilienMshardy: right12:08
jistri just got an env up, i'll try to deploy with that patch locally12:09
openstackgerritMartin André proposed openstack/puppet-tripleo: Manage tripleo-ui configuration files with puppet
*** lucas-hungry is now known as lucasagomes12:11
slagleEmilienM: did anything get added in our jobs to attempt to collect logs in post_test_hook?12:12
slagleEmilienM: b/c we have some jobs failing at that step, even though the pingtest succeeded:12:12
d0ugaljtomasek: If I have a plan in tripleo-ui and I want to update it, how do I do that?12:13
d0ugaljtomasek: Delete and re-create?12:13
d0ugaljtomasek: (update the templates etc.)12:13
EmilienMslagle: no we haven't anything added12:13
*** abehl has quit IRC12:14
*** anshul_ has joined #tripleo12:14
EmilienMi think we are unlucky, log collections was close to the timeout limit12:14
EmilienMhmm no, 1h13 is not too bad12:15
*** mbound has quit IRC12:15
slagleactually, i see the FAILURE earlier12:15
slaglemaybe coming from postci?12:15
jtomasekd0ugal: we currently add/overwrite files in swift12:15
*** mbound has joined #tripleo12:15
EmilienM2016-09-16 11:00:50.811542 | Job timeout set to: 95 minutes12:16
tbarronjistr: EmilienM shardy that mongodb replicaset error you cite looks like what I hit with deploy attempt with fresh packages and git clones this morning:
d0ugaljtomasek: oh, fun :)12:16
jistrtbarron: hmm yea... the error message isn't exactly the same, but it's very similar12:17
EmilienMlooking at
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add CephRgw to roles_data.yaml
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Use osd_pool_default_* puppet parameters when creating the pools
jistrok so i'll try to first deploy HA without any modifications12:18
EmilienMslagle: it could be a multinode/infra issue with networking12:18
*** zoli|lunch is now known as zoli12:18
shardylarsks: Hey, looks like an error crept into somewhere12:19
slagleEmilienM: yea i see an error in
jtomasekd0ugal: when I try to deploy with latest tripleo-heat-templates I am getting Failed to validate nested template: Property error: resources[9].properties: Property KeystoneCredential0 not assigned12:19
shardylarsks: I also added a question re the snmp credentials12:19
slagle2016-09-16 11:00:17,775 p=14535 u=zuul |  fatal: [node]: FAILED! => {"failed": true, "msg": "Failed to connect to the host via ssh."}12:19
EmilienMprobably a network issue with bluebox cloud12:20
openstackgerritDmitry Tantsur proposed openstack/instack-undercloud: Fix nova-related deprecation warnings
d0ugaljtomasek: I suspect you will hit similar plan updating problems if you have enough testing :(12:20
slagleEmilienM: let's go with that :)12:20
EmilienMjtomasek: I wrote that code12:20
d0ugaljtomasek: I've not seen that before, sounds like a parameter isn't set?12:20
EmilienMjtomasek: the property is generated in tripleoclient12:20
d0ugaljtomasek: it is a new one:
EmilienMit's even backported !12:21
d0ugalEmilienM: :)12:21
d0ugaljtomasek: I'd check with rbrady to see how he is getting on with the password generation stuff12:22
d0ugalI am not sure that is going to make Newton at this point :/12:22
jtomasekEmilienM, d0ugal : is it going to get into a Mistral action? is Ryan working on that?12:22
d0ugaljtomasek: Yeah, it should be part of his general password generation stuff12:22
d0ugaljtomasek: because that didn't exist yet we had to let people add more to tripleoclient.12:23
jtomasekd0ugal: I'd say it is a blocker for GUI then12:23
*** pradk has joined #tripleo12:23
d0ugaljtomasek: Sure, but a blocker doesn't make it any easier to do :)12:23
d0ugaljtomasek: That is the start of it, but it doesn't update the CLI12:23
openstackgerritDougal Matthews proposed openstack/tripleo-common: Port password generation from tripleoclient to tripleo-common
dprincepabelanger: could we try this late this afternoon?
dprincepabelanger: if it breaks then revert over the weekend?12:25
*** rhallisey has joined #tripleo12:25
EmilienMdprince: I'm doing a recheck on it12:25
EmilienMto see if it pass now ;-)12:25
d0ugalEmilienM: chances are we are going to hit that mistral timeout error against soon12:25
therved0ugal, The mistral patch seems to have worked, no?12:26
EmilienMAug 25 is last year for me in CI world12:26
EmilienMd0ugal: why?12:26
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Add mongo config settings in collector service templates
therveAt least we don't see that error anymore12:26
d0ugaltherve: oh, I hadn't seen result yet12:26
d0ugaltherve: right, yeah, it did12:26
d0ugaltherve: but should they accept it as is?12:26
d0ugaltherve: I don't really know.12:26
therved0ugal, No idea :)12:27
slagleEmilienM: isnt tested by CI12:27
openstackLaunchpad bug 1624284 in Mistral "MessagingTimeout when executing mistral actions" [Undecided,Confirmed]12:27
dprinceEmilienM: I'm not 100 sure a recheck would test that actually12:27
slagleEmilienM: we'd have to deploy the script onto the te-broker12:27
d0ugalEmilienM: therve found how to reproduce it, we have not fixed anything. So I guess any user with the same setup could well hit it too12:27
d0ugalEmilienM: and the only reason we don't hit it is because something else is broken12:27
dprinceexactly, we have to install it manually12:27
d0ugalEmilienM: CI was never really fixed, just broken quietly :)12:28
* d0ugal -> lunch and dog walk12:29
EmilienMdprince: ok, so we can maybe approve it. /me just making sure we don'tr break CI again12:29
EmilienMd0ugal: wait12:31
EmilienMd0ugal: where do you test in tripleO?12:31
EmilienMtherve: thx12:32
dprinceEmilienM: sure, we might even test it in place first12:32
EmilienMsighs at mistral :( please don't break us during a release12:32
slagledprince: EmilienM : which ceileometer service should we start up on rh1 to clear the queue? openstack-ceilometer-collector?12:34
dprinceslagle: maybe just restart all the ones that were running before12:34
dprinceslagle: we stopped them to save some CPU power12:34
slagledprince: i'm not sure how i would tell which ones were running before12:35
tbarronjistr: if you decide that the mongodb replset 'can't find master' and 'replSetReconfig command must be sent to the current replica set primary' are likely at root the same problem and want to look at my deployment, ping me as I'll likely leave my beaker machine in that state until i find a way to unblock12:36
osphi, can anyone provide details on how i can get an ldap backend configured in heat during a tripleo deployment? In my parameters i can supply parameters to keystone.conf but can't seem to get a file create within /etc/keystone/domains12:36
slaglepradk: which ceilometer service consumes events from the rabbit queue?12:36
gfidentetherve, can you give a look at and see if you can spot anything wrong with my comments there?12:37
gfidentelooks like we should be able to use batch_create and rolling_update updating the template12:37
pradkslagle, collector12:38
slaglepradk: can i start just collector? or does it need other ceilometer services?12:39
slaglepradk: backstory is that we disabled ceilometer services in our cloud due to cpu usage, but now we are seeing the queue fill up12:39
slagleand we think that's causing a bottleneck12:39
thervegfidente, So 1) Resources aren't tied to template versions12:39
pradkslagle, yea if the services are continue to publish it will fill up quickly12:39
slaglewe don't actually care about the messages, just want to clear the queue12:40
thervegfidente, 2) batch_create is not a property, it's an update policy key12:40
gfidentetherve, dah, that's why then12:40
gfidenteso it goes12:40
gfidente  batch_create:12:40
pradkslagle, mongo and gnocchi still running ?12:40
gfidente    max_batch_size: 112:40
dprinceslagle: I just ran this:
*** noslzzp has joined #tripleo12:40
shadowerjaosorior: could you have a look at ? +2 and the gate passes O_o12:40
gfidentetherve, maybe you can comment there how to use it?12:41
pradkslagle, so collector will clear up that queue but that has to go somewhere.. which is by default to mongo and gnocchi for events and metrics respectively12:41
thervegfidente, Yep looks about right.12:41
dprinceslagle: anything else you think we should enable?12:41
gfidentetherve, thanks!12:41
slaglepradk: k, mongod is up12:41
pradkslagle, so long as they are up, it should do it12:41
slagledprince: don't think so. i see the queue going down12:41
*** noslzzp has quit IRC12:42
slaglecpu load was high, but it was rabbitmq that was consuming a lot, so maybe once the queue is empty, it will settle some12:42
*** flepied has quit IRC12:42
*** noslzzp has joined #tripleo12:42
slagleactually, the load is high b/c of "mprime"12:43
slagledid we leave a benchmark running? :)12:43
shadowerjaosorior: thanks!12:43
pradkslagle, can you check if the notification agent is running.. you probably will need that up too12:43
slaglepradk: openstack-ceilometer-notification? it'sup12:44
slagleit's up12:44
*** rlandy has joined #tripleo12:44
pradkslagle, cool12:44
slaglebnemec: do you have mprime running on the rh1 overcloud controller?12:45
*** masco has joined #tripleo12:45
*** fultonj_ has joined #tripleo12:45
jaosoriorgfidente: hey dude, could you check this commit out?
EmilienMmatbu: do you have any progress on upgrade testing?12:48
EmilienMpanda: do you have progress on ipv6 testing?12:48
*** pkovar has joined #tripleo12:49
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add ipv6 nic-configs
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Fix use of batch_create in CephMon major upgrade template
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types
*** Goneri has joined #tripleo12:50
jistrtbarron: hmm btw i didn't seem to hit any problem on stack-create...12:50
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Fix use of batch_create in CephMon major upgrade template
tbarronjistr: well, maybe i'll retry then.  the only changes I have are for manila, in puppet-tripleo and THT, but naively I don't see how they would have anything to do with mongodb replset issues12:52
gfidenteshadower, can you vote on if we got it right ?12:52
gfidentetherve, ^^ added you there as well12:53
shadowergfidente: did you really mean me? That's the first time I'm seeing that change12:53
gfidenteshadower, yes I did :)12:53
tbarronjistr: and fresh packages and complete rebuild from vms on up this morniing ...12:54
matbuEmilienM: i'm working on upgrade bugs, but still blocking on underclod install hanging, any help would be welcome .. i think it's infra related12:54
thervegfidente, You can only have one of the 2 keys12:54
matbuEmilienM: but i have probably set something wrong that cause thishanging12:54
gfidentetherve, ack -1 please12:54
EmilienMmatbu: do you have logs? have you reported a bug?12:54
jistrtbarron: hmm yea i've also rebuilt from scratch today (both undercloud and overcloud)12:54
gfidentetherve, though seems problematic12:54
gfidenteon the first upgrade we don't have that resource so we'll want batch_create12:55
jistrtbarron: could be that we have something intermittent perhaps, we'll see now that we have CI running if there are some jobs that will hit this problem still12:55
gfidenteon a further update the rolling_update was meant to have same effect12:55
matbuEmilienM: yep, but no very useful (log i mean, i asked sshnaidm and slagle for help)12:55
gfidentetherve, can we get it to do both create/update always on one at a time?12:55
*** tzumainn has joined #tripleo12:56
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service
*** hjensas has quit IRC12:57
thervegfidente, I don't know, I don't think so12:57
gfidentetherve, auch!12:58
gfidentesrsly? :)12:58
thervezaneb, Do you know?12:58
EmilienMlarsks: it's weird scenario003 failed, scenario003 deploys sahara (scenario002 was working with cinder)12:58
larsksshardy, that was a rebase mistake; thanks for catching it.12:58
gfidenteit's kind of a show stopper :)12:58
*** jpena|lunch is now known as jpena12:58
*** jcoufal_ has joined #tripleo12:58
larsksEmilienM, if only I could get an overcloud deploy not to fall over early on locally... :/12:58
larsksd0ugal, do you know off the top of your head if those bugs I was hitting yesterday have a resolution yet?12:59
thervegfidente, Hum no I'm wrong, sorry12:59
zanebrolling update only creates/updates up to batch_size resources at a time12:59
zanebwas that the question?12:59
therveYou'd better test that, though. More than making sure that the template validates :)12:59
thervezaneb, So you can specify both batch_create and rolling_update in the policy?13:00
zanebI believe so. batch_create affects only the original creation of the group13:00
therve(side note, that's an horrible interface, but don't mind me :))13:01
zanebrolling_updates affects only subsequent changes to the group13:01
zanebtherve: blah blah historical reasons...13:01
thervegfidente, So forget me, that fix looks good :)13:01
gfidentetherve, zaneb yeah the expected behaviour we want is what zaneb described13:02
gfidentecause on the initial upgrade of tirpleo the resource doesn't exist, so it goes into create mode13:02
gfidenteon further attempts it goes into update mode13:02
gfidentebut we always want 1 by 113:02
*** lblanchard has joined #tripleo13:03
*** jaosorior has quit IRC13:03
*** jaosorior has joined #tripleo13:04
zanebdoes SoftwareDeploymentGroup have both of those policies?13:04
therveHopefully by inheritance13:05
zanebapparently it does13:05
* zaneb checks code13:05
zanebtherve: it didn't used to have either, so it's not picking it up just from inheritance13:06
thervezaneb, Not sure what you mean13:06
zanebin fact it can't, because you can't specify min_in_service on a SoftwareDeploymentGroup13:06
gfidenteit inherits ResourceGroup13:08
therveOh, it overrides the schema13:08
zanebit didn't support it at all before that, despite inheriting from ResourceGroup13:09
zanebinterestingly, there is a bug there13:09
thervezaneb, Because update_policy_schema was overriden?13:09
zanebbecause it actually *doesn't* override the schema when it should13:10
zanebtherve: yes13:10
zanebso min_in_service is included and a lot of the code in that patch is dead
therveWhy does it need to override it?13:11
*** tobias_fiberdata has joined #tripleo13:11
*** ayoung_ has joined #tripleo13:11
therveOh of course13:11
zanebtherve: because it defines (but does not use) a different schema for rolling_update13:11
therveBecause there is no way to use the proper copy of rolling_update_schema13:11
* zaneb raises bug13:11
zanebat least it landed in Newton!13:12
zanebwe can fix in rc2 ;)13:12
gfidentezaneb, therve thanks guys :)13:12
gfidentewe can probably vote on anyway13:12
gfidenteas it fixes the syntax error anyway13:13
zanebgfidente: yeah, just reviewed it13:13
zanebindentation is messed up, but otherwise its fine13:13
thervegfidente, So talking about template,
thervegfidente, Where does ExternalPort come from?13:13
gfidentezaneb, ack updating there13:13
gfidentetherve, sec13:14
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Fix use of batch_create in CephMon major upgrade template
tobias_fiberdataovercloud plan list, what plan is this exactly? and what does it do?13:14
derekhslagle: notifications.error   , I guess there is an error somewhere populating that too13:16
gfidentetherve, yeah seems wrong13:16
tobias_fiberdatadpeloying and getting stuck on this: Uploading new plan files13:17
tobias_fiberdata, could someone guide me?13:17
thervegfidente, There is a bunch of those in that patch13:17
gfidentetherve, well external_v6 does create the ExternalPort resource13:18
gfidentetherve, it's the _from_pool which need fixing :(13:18
gfidentetherve, thanks13:18
gfidentetherve, curious how you spotted it?13:19
thervegfidente, I validated all templates in the repo13:19
*** Jokke_ has joined #tripleo13:19
jristrandom question13:22
jristI filed a bug13:22
jristit says13:22
jristPlease if you are a developer, self-triage the bug. We do not need to wait for another developer to confirm that this is a bug.13:22
openstackgerritMerged openstack/tripleo-heat-templates: Set client protocol for glance registry client
jristdoes that just mean setting the importance?13:23
jristor more?13:23
openstackgerritMerged openstack/tripleo-incubator: Allow Emilien Macchi to be root on TripleO Cloud
thervezaneb, Should we try to test this...13:24
*** akshai has joined #tripleo13:25
trownjrist: importance, and also the "triaged" flag13:25
jristtrown: see that's what I needed to know :) thanks13:25
trownjrist: targeting to a milestone is also good13:25
jristdid that13:25
zanebtherve: there were unit tests already13:25
thervezaneb, Not the useful kind apparently13:26
jristtrown: confirmed or triaged?13:26
zanebtherve: well, it supported too *much*. it's hard to test for that13:26
trownjrist: I set triaged for bugs I file, or confirmed on bugs I am looking at that someone else filed13:26
trownnot sure it matters though13:26
thervezaneb, I guess :)13:26
zanebtherve: because for any given feature there are an infinite number of things it isn't intended to do ;)13:27
thervezaneb, I'd be ok if it'd just tested min_in_services though :p13:27
*** akshai_ has joined #tripleo13:27
*** mbound has quit IRC13:28
jpichjrist: Seems the preference in TripleO is Triaged, according to
jristjpich: thanks for the clarification13:29
jristmight need a launchpad patch to point to that13:30
*** akshai has quit IRC13:30
trownoh TIL13:32
trownwill stop using confirmed all together13:32
*** flepied has joined #tripleo13:35
openstackgerritMarios Andreou proposed openstack/puppet-tripleo: Add manila-netapp backend to manila class and tidy up generic
*** mgarciam has joined #tripleo13:37
shadowerFolks, can we merge this?  It will generate better validations docs and exercise the publish-docs job :-)13:39
shadowerjaosorior ^ ?13:39
openstackgerritMarios Andreou proposed openstack/puppet-tripleo: Fixup manila-cephfs native backend defaults
*** limao has joined #tripleo13:42
*** rodrigods has quit IRC13:45
*** rodrigods has joined #tripleo13:45
jristjtomasek: how do I test in the UI?13:45
jristjtomasek: I have it pulled down13:46
jristbut I'm not sure what changed13:46
jristperhaps that's the point? :)13:46
jristthe little pencil?13:46
jtomasekjrist: yep13:47
jristI just get infinite spinner13:47
gfidented0ugal, can you point to a commit in tripleoclient older than the one calling mistral action to do the jinja compiling?13:50
gfidentea version which can be used without overcloud.j2.yaml13:50
jtomasekjrist: hmpf, no idea, I'll look into it when I get back13:51
jristjtomasek: yeah sounds good. thanks13:51
jristjtomasek: ping me when you're back13:51
openstackgerritMerged openstack/tripleo-validations: Generate documentation for validations
*** zephcom has left #tripleo13:52
gfidented0ugal, looks like up to 8th of sept?13:53
tobias_fiberdataslagle, shardy, could anyone of you help out? We are running newton undercloud and deploying our overcloud, and it says "Uploading new plan files13:54
tobias_fiberdata", what exactly does that mean?13:54
pandaEmilienM: I didn't go past the swift error, I was testing primarily with experimental jobs ...13:55
*** saneax is now known as saneax-_-|AFK13:55
shardytobias_fiberdata: we copy the template files from the local filesystem into a swift container (same name as the overcloud you're deploying), and a mistral environmeent (again named e.g "overcloud")13:56
*** anshul_ has quit IRC13:56
shardytobias_fiberdata: combined those two things are referred to as a "plan"13:56
shardybecause it contains all the stuff needed to deploy an overcloud13:56
tobias_fiberdatashardy, ah okey, so basically it does that on the undercloud node?13:56
tobias_fiberdatathat explains why it takes ages13:56
shardytobias_fiberdata: Yeah13:56
shardyshouldn't take that long, few seconds perhaps13:57
tobias_fiberdataour undercloud aint the fastest thing in the world :P13:57
tobias_fiberdatawell then something is wrong13:57
tobias_fiberdatacause i've started this 10mins ago13:57
shardyYeah, that's defintely wrong, it's just copying a few files via some API calls13:57
tobias_fiberdatado you have any clue if there's any logs about this?13:58
jaosoriorthrash: I think this is needed for the zaqar websocket stuff to work in CI
shardytobias_fiberdata: I'd check the mistral logs /var/log/mistral/mistral-server.log13:59
shardysounds like something went wrong but the error wasn't reported to the client13:59
shardythat happens via zaqar, so ensure the zaqar services are running OK14:00
gfidentetobias_fiberdata, I have seen sometimes errors in mistral trying to reach zaqar on the wrong endpoint14:00
gfidentedefaulting to localhost:someport instead of the websockets endpoint14:00
tobias_fiberdatalooks like ZaqarAction.queue_post failed: <class 'requests.exceptions.ConnectionError'>: HTTPConnectionPool(host='',14:00
tobias_fiberdataokey gfidente14:00
gfidentetobias_fiberdata, though in your case looks like it's going to the right socket?14:01
*** cdearborn has joined #tripleo14:01
tobias_fiberdata[Errno 111] ECONNREFUSED',))14:01
tobias_fiberdatait says this aswell14:01
tobias_fiberdataperhaps it's blocking something?14:01
*** coolsvap has quit IRC14:02
gfidentewhich port is it going to?14:02
gfidentecan you compare that with endpoint list of undercloud?14:02
*** jcoufal__ has joined #tripleo14:03
tobias_fiberdatado you want the websocket one?14:04
derekhbnemec: I went into sysctl.conf for persist the conntract settiong we set last friday and found this
gfidentetobias_fiberdata, well you just want to make sure mistral is trying to reach zaqar on the right ip:port so compare those14:05
derekhbnemec: so I've bumped up the timouts again (not quite as high as they were), will keep and eye on it and persist what ever we finish up on14:05
*** jlinkes has quit IRC14:05
*** jcoufal_ has quit IRC14:06
*** jlinkes has joined #tripleo14:06
jistrEmilienM, shardy, tbarron: FYI i was able to reproduce the MongoDB replicaset problem (same message as in CI, different than what tbarron posted). It's most probably some kind of intermittent issue / race condition, as running the same puppet replset resource for the 2nd time worked just fine.14:07
gfidentetobias_fiberdata, and to which one of the two is mistral going?14:07
openstackgerritMerged openstack/instack-undercloud: Deploy validations SSH key in post config
tobias_fiberdatagonna check the logs14:07
openstackgerritMerged openstack/tripleo-heat-templates: Fixes the Ceph upgrade scripts
jistri'll at least report it now, so far i don't see what would be the cause14:07
gfidentetobias_fiberdata, I think we only launch the websocket one14:07
gfidenteso you probably don't have anything on 888814:07
tobias_fiberdataZaqarAction.queue_post failed: <class 'requests.exceptions.ConnectionError'>: HTTPConnectionPool(host='', port=888814:07
d0ugallarsks: I don't think they do yet - not sure.14:07
*** [1]cdearborn has joined #tripleo14:07
d0ugalgfidente: Yeah, that sounds about right.14:08
d0ugalgfidente: The 8th, it was recent[14:08
gfidented0ugal, ack, thanks!14:08
gfidentetobias_fiberdata, so that's not good, I think mistral is supposed to look for the zaqar-websocket endpoint14:08
jaosoriord0ugal: is that so? Thought we used the HTTP API from zaqar. At least mistral should be using it to create queues from there. And subsequently use the websocket endpoint to actually use that queue14:09
tobias_fiberdatagfidente, we installed this undercloud machine yesterday14:09
gfidentefrom master?14:09
tobias_fiberdatagfidente, not sure14:10
gfidenteusing ?14:10
*** jaosorior has quit IRC14:10
tobias_fiberdatawe used the latest repos because mitaka was not working very well for us14:10
tobias_fiberdatanot no14:10
tobias_fiberdatasorry this14:11
*** ramishra has quit IRC14:11
*** hjensas has joined #tripleo14:11
*** hjensas has joined #tripleo14:11
openstackgerritMiles Gould proposed openstack/instack-undercloud: Update undercloud.conf.sample
openstackgerritMiles Gould proposed openstack/instack-undercloud: Enable introspection of UEFI nodes by default
gfidentetobias_fiberdata, so I think this could be an issue in tripleo-common14:12
tobias_fiberdatagfidente, can we do an upgrade on the undercloud machine and hope that is correcting it to those ports it's supposed to be?14:13
*** ramishra has joined #tripleo14:13
tobias_fiberdatai mean if there's any change from yesterday14:13
gfidenteright I would remove the three delorean .repo files from /etc/yum.repos.d14:13
gfidentere-curl those (as per docs)14:13
gfidenteand try a tripleo-common update14:13
tobias_fiberdatawe'll give it a shot14:13
tobias_fiberdatathanks alot gfidente14:14
gfidentegood luck :)14:14
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Cleanup the previous plan when deploying
slaglebnemec: do we need the mprime process running on the rh1 overcloud?14:17
*** tobias-fiberdata has joined #tripleo14:17
slagleit's eating up cpu14:17
jristflorianf: did you pull down 367993 too or just visual review?14:17
jristflorianf: curious if you're having the issue I'm having
*** [4]cdearborn has joined #tripleo14:18
bnemecslagle: I shut it off.  It was niced, so it shouldn't have been taking priority over anything else, but it sounds like we found the issue.14:18
slaglebnemec: ok14:19
slaglewasn't sure :). i just saw it using a lot of cpu14:19
slaglewhen i checked this morning14:19
bnemecThat was my attempt to get the cpu to stop scaling down without being able to actually get into the bios and change the setting.14:19
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Fix _from_pool_v6.yaml str_split
*** tobias_fiberdata has quit IRC14:20
EmilienMjistr: ok, any solution?14:21
*** cdearborn has quit IRC14:21
florianfjrist: I pulled it down14:21
jistrEmilienM: no, i don't know yet why the puppet module fails on that. When re-run, it succeeds, it must be something temporary.14:21
jristflorianf: did it load ok for you?14:22
jristlike the panel had content?14:22
florianfjrist: yes, both panels have content14:23
*** fultonj has quit IRC14:23
florianfjrist: I don't see that error14:23
tobias-fiberdatagfidente, there was 2 newer packages14:23
tobias-fiberdatafor tripleo-common14:23
*** fultonj_ is now known as fultonj14:23
tobias-fiberdataso we'll try that14:23
jristflorianf: I wonder what I'm missing14:23
*** hjensas has quit IRC14:23
florianfjrist: updating changes (to the services for a role) works for me too14:24
jristwell I can't do that14:24
jristbecause the panel fails14:24
jristspinner then error in console14:24
jristhonza: merge conflict
jristflorianf: can I get a +2 on
jristand a +1 workflow plzzzzzz <314:25
honzajrist: yes14:25
shardytobias-fiberdata: note that if tripleo-common changed, you may need to either re-run openstack undercloud install, or manually refresh the mistral actions/workflows (depending on what changed in the update)14:25
florianfjrist: oh, I thought I already did that...14:26
shardythat's the manual approach, which copies what the undercloud install does internally14:26
florianfjrist: +2'ed14:26
jristthx flaper8714:26
jristthanks florianf14:26
jristflorianf: it's bugging me because I test on two machines14:26
bnemecslagle: derekh: Did you re-enable ceilometer too?  I had disabled all the stuff that we shut off.14:26
jristflorianf: thank youuuu14:26
florianfjrist: yeah, good idea to change that14:27
derekhbnemec: I havn't touched it at all, we could just call the rabbit command to clear the queue every hour14:27
jristit's just for dev but14:27
*** mbound has joined #tripleo14:27
jristit's useful14:27
slaglebnemec: probably not. dprince started them14:29
*** mbound has quit IRC14:31
*** mbound has joined #tripleo14:31
*** [1]cdearborn has quit IRC14:32
openstackgerritMerged openstack/tripleo-ui: Have dev server listen everywhere instead of just local
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton
*** bnemec has quit IRC14:37
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: POC DO NOT MERGE: Add virt-setup option to
trownpanda: ^14:39
pandatrown: finally! :) looking14:40
EmilienMpanda: my question was about experimental job, did you make progress on making it pass? Does it deploy ipv6 correctly?14:42
openstackgerritMerged openstack/puppet-tripleo: Fix wrong flag name for VNC Proxy in HAProxy
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Use osd_pool_default_* puppet parameters when creating the pools
pandaEmilienM: I released a patch for the swift problem, made it dependent on the ha-ipv6 patch for tripleo-ci, then was waiting for CI to stabilize again. I launched another check now on that patch.14:47
*** jlinkes has quit IRC14:48
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo: Add swift proxy for ceilometer middleware
*** jcoufal__ has quit IRC14:48
*** bnemec has joined #tripleo14:49
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove openstackclient imports in the new parameters command
openstackgerritJiri Stransky proposed openstack/puppet-tripleo: Wait for MongoDB connections before creating replset
EmilienMpanda: here ?
EmilienMpanda: the depends-on is
jistrEmilienM: ^^ here's an attempt for the MongoDB fix, but given that the issue is intermittent, only time can tell if it works or not14:51
EmilienMand it's not swift it's nova14:51
EmilienMjistr: nice!!14:51
EmilienMjistr: looking14:51
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: elements: opensuse: Add support for openSUSE Leap
pandaEmilienM: and I have to remove that Depends-On on my first patch, since it is on the same project, it doesn't do anything14:54
*** athomas has quit IRC14:54
EmilienMpanda: you should rather do the other way around imho14:55
EmilienMso you can have multiple depends-on in tripleo-ci patch14:55
pandaEmilienM: ok, I would probably have arrived at this solution at the third patch on the experimental ipv6 job14:56
pandaEmilienM: but I'll invert after these results14:56
EmilienMpanda: we really need to make progress on this thing14:57
EmilienMpanda: we can't releae newton without ipv6 support14:57
bnemecpanda: EmilienM: Swift is broken in ipv6 though:
openstackLaunchpad bug 1623672 in tripleo "Swift failing to deploy in ipv6" [Critical,In progress] - Assigned to Ben Nemec (bnemec)14:57
bnemecMy first patch resulted in broken hieradata, and I haven't been able to figure out the right puppet to fix it.14:58
bnemecAlthough I haven't had a lot of time to look into it the past couple of days either.14:58
openstackgerritMiles Gould proposed openstack/instack-undercloud: Enable introspection of UEFI nodes by default
pandabnemec: that is the same error I'm trying to fix here
bnemecpanda: Ah, cool.15:00
*** jcoufal_ has joined #tripleo15:03
*** athomas has joined #tripleo15:05
pandaEmilienM: any suggestion on how to speed up this process even when CI is down ?15:06
shardyalembic.script.revision.RevisionError: Requested revision 4b47ea298795 overlaps with other requested revisions d6a12e637e2815:06
shardyhrm, anyone seen that trying to upgrade neutron on the undercloud?15:07
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: elements: opensuse: Add support for openSUSE Leap
*** jkraj has quit IRC15:13
*** dtantsur is now known as dtantsur|pto15:13
*** rcernin has quit IRC15:15
therved0ugal, Is there a list of tripleoclient command that happens during a ci run?15:16
*** jkraj has joined #tripleo15:19
shardytherve: CI uses, which calls tripleoclient:15:19
shardyso you can look at that and see how its called, does that help?15:19
shardyyou can also run that script locally with same/similar inputs to CI15:19
therveshardy, And things like plan create?15:20
shardytherve: openstack overcloud deploy does a plan create internally, we don't yet explicitly test the seperated steps of create plan, deploy plan15:21
shardywe'll need to do that soon tho15:21
therveHum okay15:21
EmilienMpanda: which process?15:21
*** bnemec has quit IRC15:22
jpichtherve: "undercloud install" should be creating a plan with the default templates as well15:22
pandaEmilienM: moving forward with the experimental ipv6 job15:23
thervejpich, Ahah, that's the one I was missing, thanks15:23
thervejpich, It doesn't wait for the plan to be created though :/15:24
*** aufi has quit IRC15:24
therveThat sounds like it may be an issue15:24
EmilienMpanda: moving forward like how?15:24
*** bnemec has joined #tripleo15:25
pandaEmilienM: make progress15:25
*** ramishra has quit IRC15:25
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Update README about running a full end to end deployment
EmilienMpanda: when CI is down, we can't make any progress15:25
jpichtherve: It sounds like it could be yeah... and there's also an issue with creating the default plan atm that means it's not being created at all15:25
jpich(I think d0ugal has a patch up for that, just hit this on a brand new undercloud locally and about to test it)15:26
therveCool, I'm making progress understanding this though15:27
* therve bbl15:27
*** ramishra has joined #tripleo15:27
EmilienMpanda, bnemec also
openstackLaunchpad bug 1605363 in tripleo "[Newton] ipv6 HA deployments are currently broken" [Critical,Triaged]15:28
EmilienMbandini: have you an update about ^ ?15:28
EmilienMis it fixed ?15:28
bandiniEmilienM: I put it on my todo to test it with master. won't get to it today though15:29
EmilienMbandini: ok, please let me know, it sounds quite critical15:30
*** jkraj has quit IRC15:30
bandiniEmilienM: will do, thanks for checking15:30
*** matbu is now known as matbu|brb15:38
jpichtherve: And thank you for that! ( does resolve the particular issue I hit fwiw)15:39
openstackgerritMerged openstack/tripleo-heat-templates: Convert deploy steps to jinja2 loop
*** [4]cdearborn has quit IRC15:42
*** ebalduf has joined #tripleo15:43
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Add mongo config settings in collector service templates
EmilienMheh, mitaka jobs are green again with
EmilienMnow liberty15:49
*** jcoufal__ has joined #tripleo15:49
mariosEmilienM: gfidente revote please when you get a chance thanks15:49
*** electrofelix has quit IRC15:50
*** jcoufal_ has quit IRC15:50
EmilienMmarios: -215:50
gfidentemarios, I told you indetation would kill you15:50
gfidentethat's why they invented python15:51
gfidenteI like enforced indentation15:51
gfidentenot the -lint jobs15:51
mariosyou know i had to fight to even get rake lint to pass locally15:51
mariosi mean had to install stuff, not done it before15:52
gfidenteyeah for me it wasn't installing a couple of gems15:52
gfidentethe other day15:52
gfidentebundle install15:52
marios(or lately, gems in general haven't touched for loong time)15:52
EmilienMmarios: wow, you installed lint locally?15:52
EmilienMyou ok? :)15:52
mariosEmilienM: no, i mean it wasn't passing 'rake lint'15:53
mariosEmilienM: had to install the right dependencies (gems) to get it to work, bundler helped in the end15:53
EmilienMbundle install?15:53
EmilienMthat all you need ;)15:53
gfidenteyeah when it passes15:54
mariosEmilienM: well i also had to manually install puppet for some reason15:54
ayoungdprince, BTW, EmilienM 's changes to get Credentials initialized have all landed.  I confirmed they worked last night.  We should be able to un-peg Keystone now15:54
mariosEmilienM: there was a dependency issue so bundle install didn't pass15:54
mariosEmilienM: perhaps these things are just easier for you ;)15:54
EmilienMmarios: nothing is easy man15:54
*** benoit has quit IRC15:54
mariosEmilienM: gfidente thanks guys15:55
ayoungalso, the keystone upstream changed such that there is a null key used during the migration process to deal with keys, so the breakage was removed.  I understand why they did what they did, but was able to convince them it was not something we could accept.15:55
ayoungeither way, we should be able to unpeg Keystone from N315:55
EmilienMmarios: anytime15:55
mariosgfidente: please readd here too
mariosEmilienM: if you have time ^^^ related one15:56
EmilienMayoung: it's already unpin ...15:56
ayoungEmilienM, excellent15:56
EmilienMayoung: you're late15:56
EmilienMwe unpinned like last week15:56
EmilienM2016-09-07 14:22 Emilien Macchi      o Revert "Pin Keystone to Newton milestone 3"15:57
EmilienM9 days ago15:57
ayoungEmilienM, I can't track it all.  Just wanted to make sure I was backing you guys up.15:57
EmilienMayoung: well, hopefully we can track them all15:57
EmilienMotherwise our CI would be broken every day.15:58
EmilienMmarios: sure, looking15:58
EmilienMmarios: i'll trust you for testing it, as we don't any testing for manila15:58
EmilienMmarios: puppet code looks ok15:58
mariosEmilienM: right, yeah tbarron is testing that stuff he is commenting about things passing failing.. we are waiting to hear final ok but he was blocked today on unrelated issue 14:45 < tbarron> marios: so now we hit a weird mongodb error, certainly unrelated to your changes:
mariosEmilienM: thanks16:00
EmilienMcool yw16:00
openstackgerritMerged openstack/tripleo-heat-templates: [mitaka-only] mysql: never add brackets to mysql_bind_host
*** zoli is now known as zoli|gone16:02
*** derekh has quit IRC16:02
beaglesdprince: I've been looking at  - I'd like to try to find a reasonable way to fix this somehow in the templates, but I'm coming up with nada16:03
openstackLaunchpad bug 1623155 in tripleo "Neutron L3 HA isn't apparently being enabled anywhere" [High,Confirmed]16:03
zoli|gonehave a nice weekend16:03
beaglesdprince: the alternative is to revert the change to the tripleoclient that removed enabling neutron L3 HA when the controller count > 116:04
beaglesdprince: could use your insight here16:04
*** panda is now known as panda|bbl16:04
*** zoli|gone is now known as zoli_gone-proxy16:04
beaglesshardy too if he's around ^^^16:04
shardybeagles: can we use the equals function to convert e.g ControllerCount == 1 to a boolean?16:07
shardyit's going to mean the service is still tied to the Controller Role, but at least it won't be hard-coded in tripleoclient16:07
beaglesshardy: yup... is the ControllerCount available to where NeutronL3HA is set?16:07
beaglesshardy: if so, that'd be awesomely easy16:08
shardybeagles: it should be, it's passed in via parameter_defaults like everything else16:08
* shardy thinks for a role agnostic way to do it16:08
beaglesshardy: ooo... actually I have to be a bit more slick than that. I have to make a conditional I think, because we don't want to enable it if controllercount > 1 and dvr is enabled16:09
shardybeagles: perhaps nest two if functions?16:10
beaglesshardy: ah of course, yeah16:10
shardyThese are bleeding edge functions that just landed in Heat16:10
shardywhat could possibly go wrong ;)16:10
*** cdearborn has joined #tripleo16:10
beaglesshardy: I briefly considered suggesting a conditional resource for roles that resourcegroups with counts >1, but it got complicated pretty fast.16:11
EmilienMshardy: our CI looks pretty good now, if I propose rc1, stable/newton will be created and we'll have to backport all the things we want from master to stable/newton. Do we want that?16:11
shardyActually I think the or example which combines or, equals and not gets you pretty close?16:11
beaglesshardy: yeah. I'll give it as hot16:12
beaglesI mean shot .16:12
shardyEmilienM: Lets look at the FFE blueprint status - It'd be nice to cut an RC1 so we can really focus on bugfixes for an RC216:13
shardybut ideally I'd prefer we didn't carry lots of FFEs into the RC2 (ideally none)16:13
*** colonwq has quit IRC16:14
*** absubram has quit IRC16:14
*** masco has quit IRC16:15
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: Swift add_devices.pp IPv6 handling
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types
EmilienMpanda|bbl: done ^16:16
*** fragatina has joined #tripleo16:16
EmilienMshardy: right16:16
EmilienMshardy: maybe could we create a gerrit topic with all patches we want in rc116:16
gfidentebeagles, add me on review pls16:17
beaglesgfidente: ack16:17
shardyEmilienM: It looks like there's going to be a few FFEs still to land, including the last custom-roles patches and the large fluentd client one16:17
gfidentethe idea was that neutron/l3 would be disabled on dev environments16:17
*** mcornea has quit IRC16:17
gfidenteand that the default would work fine for 3 controllers16:17
EmilienMshardy: ok for gerrit topic? it will help reviewers16:18
gfidentebut having the logic in the template would be nicer16:18
shardyEmilienM: I'm fine with branching now tho, I guess it'd be good to align with all the other projects, and it will help us focus on what we really need to land for the final release16:18
shardyEmilienM: Yes, that's a good idea, thanks16:18
EmilienMshardy: i'm starting something but I'm afraid to miss patches, I'll ask you to review16:18
*** colonwq has joined #tripleo16:19
shardyEmilienM: did you check with dhellmann that it's OK for us to miss the RC1 deadline given that most projects are cycle-trailing?16:19
EmilienMmarios, shardy: is in FFE?16:20
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: Move rabbit's clustering port away from the ephemeral port range
EmilienMshardy: yes I did16:20
EmilienMhe said that's not too bad, as long as we release our final release on time16:20
*** fragatina has quit IRC16:20
shardyEmilienM: Ok, thats good16:20
EmilienMshardy: please add patches in
EmilienMerr sorry wrong ling16:21
shardyEmilienM: yeah that's an FFE
EmilienMshardy: well the patch is in bad shape16:21
shardymarios, Jokke_: what's the status of ?16:21
shardyshould we defer it to Ocata-1?16:21
shardyEmilienM: agreed16:22
EmilienMmerge conflict, -1 from marios, CI not passing16:22
EmilienMI don't want to be pessimist but...16:22
shardyYeah, we're going to have to start deferring things pretty soon as we've pushed things pretty far with FFEs already16:22
EmilienMshardy: I'm not putting bugs in rc1 topic16:23
gfidenteshardy, EmilienM is the gerrit branch for bug fixes too?16:23
EmilienMjust features16:23
EmilienMideally, just features16:24
gfidenteah that just answers it :)16:24
shardyEmilienM: Yep, lets focus on the features, then we can groom the bugs for RC216:24
EmilienMbug fixes can still be backported16:24
*** dprince has quit IRC16:24
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient: Do not use selinux-permissive for the CentOS image
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Revert "Create overcloud images for liberty using EXT4"
mariosshardy: i was waiting for Jokke_ to fix that up ... i think it is a really easy fix...16:26
EmilienMparamite, larsks: about ops tools, is last blocker?16:26
shardymarios: yeah I nearly did it myself but wasn't sure if you were already on it16:26
shardywe're running out of time for Newton, can you push an update?16:27
EmilienMshardy: do we have all you need for custom roles in ?16:27
*** osp has quit IRC16:27
mariosshardy: EmilienM that tht depends on the puppet-tripleo side which I asked EmilienM  to vote on
EmilienMakrivoka, jrist: hey, can you add patches related to "node tagging workflow" in please ?16:27
EmilienMmarios: please add tripleo/rc1 gerrit topic to all manila cephfs patches16:28
shardyEmilienM: Yes, just those two remaining patches16:28
mariosshardy: yeah will try update before i finish up today16:28
jristI thought I did16:28
shardywe then need CI coverage and docs, but they can be done independent of the release16:28
mariosJokke_: you around?16:28
mariostbarron: any idea? ^16:28
EmilienMshardy: ack16:29
jristEmilienM: how do I add it?16:29
jtomasekjrist: your error happens probably because with latest tripleo-heat-templates the heat validate fails again16:29
jristjtomasek: ah that is possible. this is latest master16:29
jristjtomasek: what templates should I use? got a hash?16:29
jtomasekjrist: use my capabilities patch, that works.16:30
jristfigure. ha16:30
EmilienMjrist: there is a button in Gerrit16:30
jristpatch the patches16:30
EmilienM"Edit topic"16:30
jtomasekjrist: git review -d <mypatchid>16:30
*** absubram has joined #tripleo16:30
jristI know how to do that :)16:30
jristthanks jtomasek16:30
* EmilienM afk lunch16:30
jristEmilienM: change to tripleo/rc1 ?16:30
jristha you did it16:31
* EmilienM afk_real16:31
jtomasekjrist: this work is supposed to fix the heat validate hopefully
jristwoot nice thanks16:31
jristso I could pull that too :)16:31
jristoh yheah16:31
jristI saw this16:31
gfidenteis it okay for me to tag a couple of patches tripleo-rc1 ?16:31
*** rasca has quit IRC16:32
jristd0ugal: on above patch 368150 should we recheck again?16:32
EmilienMif they are related to the blueprints in yes16:32
shardygfidente: Yes, but please only tag FFE feature patches, or bugfixes that should block the release and can't wait for RC216:32
EmilienMotherwise no.16:32
jristor does it happen after a rebase automatically16:32
*** tbonds has joined #tripleo16:32
gfidenteshardy, wait we said no bugs16:32
shardygfidente: critical release blockers only16:32
tbarronmarios: i haven't talked to Jokke_ today16:33
EmilienMmarios, gfidente: please do the same for manila cephfs :-)16:33
shardylike, we land that list, then we tag the release16:33
*** ebalduf has quit IRC16:33
*** jpich has quit IRC16:33
openstackgerritAdriano Petrich proposed openstack/tripleo-heat-templates: GATE TEST, please ignore
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add integration with Manila CephFS Native driver
mariostbarron: ack thanks16:35
mariosshardy: done EmilienM will do but maybe monday are you guys cuttin rc1 today? I thought 26th?16:36
gfidenteshardy, honestly I am not sure how come not everybody hits
openstackLaunchpad bug 1623552 in tripleo "heatclient resolves paths for types and get_file calls that then don't make sense in swift" [Critical,Confirmed]16:36
EmilienMmarios: no we don't cut today16:36
EmilienMwe cut as soon as we have critical features merged16:37
shardymarios: that's the deadline for final RC's, we're trying to cut an RC1 before then (same as other projects), but we're going to miss the RC1 window (which closes today) by a few days due to all the FFEs we had16:38
shardy(combined with CI issues, as normal)16:38
mariosshardy: ack thanks16:38
shardymarios: we'll do an RC2 with a bunch of bugfixes, but we're trying to get the FFEs cleared away before RC116:38
*** masco has joined #tripleo16:39
shardygfidente: ack, we definitely need to fix that, I've not yet reproduced locally but will try, and see if I can help with the fix16:41
gfidenteshardy, I am sorry I wanted to look into it today16:42
gfidentebut had more ceph stuff :(16:42
gfidenteshardy, though if after an upload you try16:43
gfidenteswift download container16:43
gfidentein the downloaded files you get the file:// references16:43
*** jaosorior has joined #tripleo16:44
shardygfidente: Yeah, that should actually be OK provided they match the keys in the files map we pass to heat16:44
gfidentethe paths are okay16:45
shardybut it sounds like we're failing to get those files because we add the file:// prefix before resolving the file contents and adding it to the map16:45
gfidenteyes exactly16:45
jaosoriorEmilienM: hey dude, read your comment on the vnc proxy CR. if you have time, could you make the puppet-nova change? Else I'll check that out on monday.16:45
*** bfournie has quit IRC16:45
gfidenterequests/sessions fails with
openstackLaunchpad bug 1623552 in tripleo "heatclient resolves paths for types and get_file calls that then don't make sense in swift" [Critical,Confirmed]16:45
gfidentesorry "No connection adapters were found"16:45
gfidenteso either we strip it out and make those relatives, or we point to swift16:46
gfidenteit seems16:46
EmilienMjaosorior: puppet-tripleo you mean?16:46
*** bfournie has joined #tripleo16:46
shardygfidente: yeah, I'm not sure which of those will work, but in theory heatclient should do all this for us16:46
*** fragatina has joined #tripleo16:46
shardyI had to fight with heatclient a bit previously to get the template_object stuff to work, so will take a look16:47
*** lucasagomes is now known as lucas-dinner16:47
*** thrash is now known as thrash|f00dz16:47
jaosoriorEmilienM: aaah, i thought you meant to do that change directly in puppet-nova16:47
*** abregman is now known as abregman|afk16:48
EmilienMjaosorior: no16:49
*** fragatina has quit IRC16:49
*** fragatina has joined #tripleo16:49
jaosoriorEmilienM: alright. That could work.16:49
gfidentethanks EmilienM for checking those as well16:50
gfidenteme going afk16:50
*** jprovazn has quit IRC16:54
*** gfidente has quit IRC16:55
*** ohamada_ has quit IRC16:55
Jokke_marios, shardy, tbarron: I will have the fix up still today17:00
*** bana_k has joined #tripleo17:00
Jokke_just haven't pushed it out yet17:00
*** rajinir has joined #tripleo17:04
*** fragatina has quit IRC17:05
*** fragatina has joined #tripleo17:06
*** florianf is now known as florianf|afk17:08
*** jcoufal_ has joined #tripleo17:08
*** jcoufal__ has quit IRC17:11
*** tosky has quit IRC17:13
EmilienMlarsks, shardy: so like I said, something is wrong with scenario003, and scenario003 only adds sahara iirc17:13
EmilienMhonza: i'm not going to -1 because we don't have time but please write commit messages, eg
EmilienMakrivoka: you're also commiter on this patch ^17:17
honzaEmilienM: thanks, you're totally right, i need to be pay more attention to those17:18
EmilienMcool np17:18
marioshonza: i updated it17:18
mariossorry Jokke_17:18
marioshonza: apologies, it is long day for me, was a mistake was meant for Jokke_17:19
honzamarios: :)17:19
mariosJokke_: i updated
*** jpena is now known as jpena|away17:19
mariosJokke_: was a quick fix17:19
mariosJokke_: puppet-tripleo side was also updated today so should be good. testing is what we need righ tnow, for this and manila (both the THT change and the puppet-tripleo change have the netapp as parent since it does the tidy up for the backends etc)17:20
*** rhallisey has quit IRC17:21
*** rhallisey has joined #tripleo17:23
tbarronJokke_: notes on my testing are at
tbarronJokke_: obviously your patches will be slightly difft than mine, but you can see how we're doing it17:30
tbarronJokke_: when a deploy fails i run 'heat resource-list --nested-depth 5 overcloud | grep FAILED' and 'heat deployment show <uuid>' to see why and update the review.17:31
bnemectbarron: Jokke_: If you're on master, it's way easier to use "openstack stack failures list overcloud" to find out what failed.17:35
tbarronalso 'ssh heat-admin@<controller-ip> 'sudo grep enabled_ /etc/manila/manila.conf; sudo ls -a /var/log/manila' to see if we got lucky and the deploy is far enough along to update manila.conf and attempt to start up services17:40
tbarronbnemec: thanks for the tip!  i'm sure i'll get opportunity to try that soon :)17:40
*** kjw3 has quit IRC17:44
*** trown is now known as trown|lunch17:48
tbarronbnemec: that is not only more convenient, but it's showing me an error (parameter w/o value from Hiera data file and no default supplied in puppet-tripleo module) that I didn't see before.  Thanks!17:48
*** thrash|f00dz is now known as thrash17:51
*** _milan_ has quit IRC17:51
openstackgerritAdriano Petrich proposed openstack/tripleo-heat-templates: GATE TEST, please ignore
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: WIP Add a function to upgrade from full HA to NG HA
*** akshai_ has quit IRC18:01
*** jcoufal_ has quit IRC18:05
*** kjw3 has joined #tripleo18:05
*** jcoufal_ has joined #tripleo18:06
*** paramite has quit IRC18:13
*** akrivoka has quit IRC18:18
*** jcoufal_ has quit IRC18:20
*** jcoufal_ has joined #tripleo18:20
*** chlong_ has quit IRC18:27
openstackgerritMerged openstack/tripleo-heat-templates: Fix _from_pool_v6.yaml str_split
*** tbonds has quit IRC18:31
EmilienMslagle: this one can be merged also
slaglefor that one, i wasn't actually sure if we needed a bug for that18:32
slagleit's not a feature, i guess it's fine18:32
EmilienMslagle: yeah, it's just helping to remove warnings in puppet catalog18:34
EmilienM(we already have a ton because of deprecations things)18:35
EmilienMbut they should disappear in ocata18:35
*** yamahata has joined #tripleo18:37
*** cwolferh has quit IRC18:37
*** [1]cdearborn has joined #tripleo18:38
*** lhinds has joined #tripleo18:41
*** cdearborn has quit IRC18:42
beagleswoohoo undercloud upgrade no issues18:42
* beagles does a little dance18:43
beaglesit's the little things18:43
*** pkovar has quit IRC18:44
*** bnemec is now known as beekneemech18:45
*** saneax-_-|AFK is now known as saneax18:46
*** athomas has quit IRC18:46
*** abregman|afk is now known as abregman18:48
openstackgerritHonza Pokorny proposed openstack/tripleo-ui: When deploy finishes, show overcloud info
openstackgerritHonza Pokorny proposed openstack/tripleo-ui: Update tripleo-ui-deps RPM
honzajrist: merge conflict resolved
*** trown|lunch is now known as trown19:02
*** jaosorior has quit IRC19:05
*** cwolferh has joined #tripleo19:07
beaglesEmilienM: re: - I added a comment proposing we bump to ocata. We have a partial fix in that will probably do for now.19:07
openstackLaunchpad bug 1612786 in tripleo "Add validation to disallow OVS round-robin bonding" [Medium,In progress] - Assigned to Brent Eagles (beagles)19:07
EmilienMbeagles: rc2 or ocata, up to you19:07
openstackgerritMerged openstack/tripleo-heat-templates: Add hyperconverged-ceph environment to include CephOSD on computes
openstackgerritMerged openstack/tripleo-heat-templates: Fix use of batch_create in CephMon major upgrade template
*** mburned is now known as mburned_out19:11
beaglesEmilienM: mmm... re-reading bnemec's comment there are some examples and doc changes that should be done, so RC-2 probably is appropriate. I'll update19:11
*** absubram has quit IRC19:13
EmilienMbandini: do you have a patch for ?19:15
openstackLaunchpad bug 1623818 in tripleo "RabbitMQ should use predefined ports below ephemeral ports range " [High,In progress] - Assigned to Michele Baldessari (michele)19:15
*** r-mibu has quit IRC19:16
*** r-mibu has joined #tripleo19:17
openstackgerritMerged openstack/instack-undercloud: Update undercloud.conf.sample
*** mgarciam has quit IRC19:22
EmilienMshardy: I updated
EmilienMall patches that landed or are going to land today are rc119:23
EmilienMall patches that do not pass CI or negative review are rc219:23
EmilienMall bugs without patches, and not high or critical are ocata-119:23
EmilienMall bugs without patches, critical or high are rc219:23
EmilienMso in rc1, we still have 3 bugs in progress and these patches
EmilienMin RC2, we have 4 Confirmed, 13 Triaged, 26 In Progress19:24
slagleEmilienM: are you +2 on ?19:26
slagleagree the commit message is bad19:26
EmilienMslagle: yeah I told to honza that commit messages are important.19:27
EmilienMslagle: and no, I won't +2 until ovb jobs are green19:27
slagleyea, i just meant with the commit message as-is19:27
slagleif it passes CI, i'm ok to merge it19:28
slaglei'll +219:28
EmilienMwell, usually I would -1 but since we are close to the release, I'm not blocking it19:28
EmilienMslagle: ok19:28
EmilienMslagle: if CI pass i'll review it and maybe approve it19:28
EmilienMslagle: worries me19:29
EmilienMslagle: gate-tripleo-ci-centos-7-scenario003-multinode fails and that's not normal19:29
slagleyea some response or investigation about that failure on the patch is needed19:30
*** david-lyle has quit IRC19:30
*** david-lyle has joined #tripleo19:30
openstackgerritMerged openstack/tripleo-heat-templates: Convert UpdateWorkflow to support composable roles
*** sarathk has joined #tripleo19:31
slagleEmilienM: scenario001 is also failing19:32
slagleis that expected?19:32
pradkcan we merge this ?19:33
EmilienMslagle: yes scenario001 is failing I think because of ceph19:35
EmilienMslagle: I'll look in a few min19:35
*** dprince has joined #tripleo19:37
EmilienMlarsks: are you around?19:37
*** larsks has left #tripleo19:38
*** larsks has joined #tripleo19:38
larsksEmilienM, sort of :).19:38
*** dprince has quit IRC19:39
*** akshai has joined #tripleo19:39
*** akshai_ has joined #tripleo19:40
EmilienMlarsks: you'll have to help us a bit if you want merged19:42
EmilienMwe're debugging why it doesn't pass functional tests19:42
*** david-lyle has quit IRC19:43
larsksEmilienM, I have been trying to look at that but I've bene stymied by the fact that I can't get an overcloud deploy to start, at all, due to all the bugs in tripleo master.19:43
larsksI'm going to spend some more time with it this evening.19:43
larsksI could use some help from someone!19:43
EmilienMwhat bugs?19:43
EmilienMslagle: for the record, scenario001 is failing because of a gnocchi bug:
*** akshai has quit IRC19:44
EmilienMslagle: really low prio imho, checked with pradk and gnocchi works for him so I'll continue to debug after the release19:44
*** jpena|away is now known as jpena|off19:44
EmilienMslagle: scenario003 error is important though, it sounds like a bad format of heat template in larsks's patch.19:45
bandiniEmilienM: yes I do, not sure why LP did not pick it up19:45
slagleEmilienM: sounds good, as long as we know19:45
EmilienMbandini: please put the patch in the lp manually, so at least we know you work on it.19:46
larsksEmilienM, last I checked there still bugs that mean (a) deploying with custom envrionments doens't work and (b) error reporting is problematic. Do you know if these have been fixed?19:47
larsksEmilienM, I have to perform some kid transport, but will check in a bit.19:47
slaglewhat the launchpad bugs?19:48
openstackgerritMerged openstack/tripleo-heat-templates: Add CephRgw to roles_data.yaml
openstackgerritMerged openstack/instack-undercloud: Fix nova-related deprecation warnings
EmilienMI'm trying the patch locally, there is something wrong in the template i think19:48
slagletripleo-ci has a lot of green and it uses custom environments all over the place19:48
beekneemechslagle: larsks may be referring to the broken --templates parameter19:49
beekneemechI don't actually know if that's fixed because I just haven't tried lately. :-/19:50
slaglemaybe. but that doesnt block this patch19:51
beekneemechYeah.  It does make it a bit of a pita to work on template changes though.19:52
EmilienMpradk: +A lgtm19:53
*** absubram has joined #tripleo19:57
*** openstackstatus has quit IRC19:58
*** openstackstatus has joined #tripleo20:01
*** ChanServ sets mode: +v openstackstatus20:01
*** kjw3 has quit IRC20:08
*** rcarrillocruz has joined #tripleo20:08
openstackgerritMerged openstack/tripleo-heat-templates: Refactor upgrade checks.
*** kjw3 has joined #tripleo20:11
*** jayg is now known as jayg|g0n320:18
*** panda|bbl is now known as panda20:18
*** jcoufal has quit IRC20:18
*** jcoufal_ has quit IRC20:19
larsksEmilienM, in the logs for scenario0003, what don't I see "Uploading new plan files" in the console output?20:19
larsksOr slagle or really anybody who is around....20:20
EmilienMlarsks: any url handy?20:22
larsksMostly I am concerned that I am using locally a different version of things than CI is using...20:23
EmilienMplan created20:24
larsksYes, I see that.20:24
EmilienMthere is no problem in mistral20:24
EmilienMthe prob is 2016-09-16 13:29:10.762448 | 2016-09-16 13:29:05Z [overcloud]: CREATE_FAILED  Resource CREATE failed: The Referenced Attribute (ControllerServiceChain role_data) is incorrect.20:24
larsksBut when I run a deploy, I first see "uploading new plan files".20:24
larsksYes, I see that error, too :)20:24
larsksBut that is not my question right now.20:25
EmilienMlet's look at
EmilienMbecause that error is the reason of overcloud failure20:25
larsksThat error from heat (re; controllerservicechain) often indicates a typo or bad parameter reference in a deeply nested stack.  Unfortunately, heat doens't actually log the source of the error anywhere.20:26
larsksMy first question, if we could just pause, is why the output I see when starting a deploy is different than what I see in CI.  That would help me make sure I am testing in an appropriate environment.20:26
larsksI understand that is not the cause of the error.20:27
EmilienMsomewhere in
EmilienMI'm digging20:27
EmilienMlet's see in heat engine logs20:27
openstackgerritMerged openstack/tripleo-common: Clearer error when the Mistral env already exists
openstackgerritMerged openstack/puppet-tripleo: Add manila-netapp backend to manila class and tidy up generic
larsksEmilienM, never mind. We appear to be having different conversations right now.  I will see if you have some time later.20:28
EmilienMlarsks: i'm looking for the root cause of why your patch fails20:28
larsksEmilienM, yes, and I was asking a different set of questions to try to get a local testing environment that mataches what ci is using.20:29
EmilienMif you want to reproduce the problem, you can use the same environment as tripleo CI scenario00320:29
EmilienMlet me find you the link20:29
larsksSince I can't even *start* a deploy locally, debugging this has been difficult.20:30
*** mburned_out is now known as mburned20:31
thervelarsks, You don't see "uploading new plan files" in the CI because it's a fresh deployment20:34
larskstherve, thanks.  So, with my local environment, all attemtps to start a deploy currently fail with "Exception updating plan: The environment is not a valid YAML mapping data type."20:35
larsksI think I am going to kill it all and start fresh.20:35
thervelarsks, maybe20:36
therveBut restarting from scratch would fix that particular issue20:36
larsksThis has been frustrating enough that I think starting fresh is probably the best idea.20:37
EmilienMwhat I don't understand is why other scenarios are working20:38
EmilienMthe only diff between scenario003 and other is that we have SaharaApi and SaharaEngine services20:38
therveAh, good idea :)20:41
thervelarsks, EmilienM: typo here:
therveWhy it doesn't give an error here is a good question20:43
larskstherve, there are (were?) open heat bugs about validation problems.20:43
therveI bet there are :/20:44
EmilienMtherve: I've been reading this file 10 times20:44
EmilienMtherve: thank you :=20:44
therveNo pb20:45
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Add fluentd client service
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service
EmilienMlarsks: why did you rebase on master yesterday?20:46
EmilienMI rebased it on shardy's patches to avoid merge conflict..20:46
larsksEmilienM, there was a rebase yesterday because I introduced a typo when rebasing on the overcloud.j2.yaml changes.20:46
EmilienMok, I just hope it will pass this time20:47
EmilienMI'm +2'ing it20:47
EmilienMand will approve it tonight if CI is full green (we already had +2 before)20:47
larsksI will keep my fingers crossed.20:47
*** sarathk has quit IRC20:48
therveEmilienM, Is the change in swift-storage ok?20:49
*** noslzzp_ has joined #tripleo20:49
openstackgerritEmilien Macchi proposed openstack/tripleo-common: Add node tagging workflow
*** noslzzp has quit IRC20:50
*** ccamacho has quit IRC20:50
EmilienMwe're hitting a lot20:50
openstackLaunchpad bug 1624420 in tripleo "MongoDB Can't find master host for replicaset tripleo." [High,In progress] - Assigned to Jiří Stránský (jistr)20:50
EmilienMlarsks: why do you add SwiftRawDisks ?20:51
*** ccamacho has joined #tripleo20:51
larsksEmilienM, sounds like another rebase error from rebasing on shardy's big change.20:51
EmilienMsetting up the alert on the mongodb bug20:51
EmilienMit breaks HA job very often20:51
EmilienMlarsks: ok, please double check the patch and submit it again;20:51
*** trown is now known as trown|outtypewww20:53
openstackgerritGabriele Cerami proposed openstack/puppet-tripleo: Swift add_devices.pp IPv6 handling
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service
openstackgerritAlex Schultz proposed openstack/instack-undercloud: Catch runtime exceptions during validation
*** ebalduf has joined #tripleo20:55
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: Wait for MongoDB connections before creating replset
EmilienMjistr: I fixed your patch, there was a typo ^20:56
EmilienMif anyone can +2 it, I think it will make ha job more stable20:56
pandabut ipv6 job did not even start :(20:58
EmilienMyou need to run "check experimental on tripleo-ci patch"20:59
EmilienMpanda: didn't it work?20:59
EmilienMI saw it in zuul this morning20:59
beekneemechpanda: telnet://
*** rhallisey has quit IRC21:02
pandaEmilienM: that's what I do after every patch, but I started a check experimental almost 6 hours ago, and did not receive any result.21:04
pandaEmilienM: maybe I should have waited longer. Now I pushed another patch, probably any older check was canceled21:04
pandabeekneemech: weeeee21:04
beekneemechpanda: Yeah, new patch sets cancel any previous jobs for that change.21:07
pandabeekneemech: even funnier than telnet towel.blinkenlights.nl21:07
beekneemechpanda: It looks like there were a lot of jobs in the queue earlier, and I believe experimental jobs get lowest priority, so it may just not have started.21:07
*** Goneri has quit IRC21:08
openstackgerritMerged openstack/tripleo-specs: Spec for TripleO validations
pandabeekneemech: I have to invert the order of my daily tasks, checks in the morning (Europe) so the queue is smaller.21:09
openstackgerritMerged openstack/tripleo-heat-templates: Expose parameter to enable combination alarms
beekneemechpanda: That is a good plan. :-)21:09
*** ooolpbot has joined #tripleo21:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1624420 in tripleo "MongoDB Can't find master host for replicaset tripleo." [Critical,In progress] - Assigned to Jiří Stránský (jistr)21:10
beekneemechOkay, wtf?  Now every time I try to open a new terminal it telnets to that review.21:11
*** abregman has quit IRC21:15
*** sshnaidm is now known as sshnaidm|away21:16
*** akshai_ has quit IRC21:18
*** [1]cdearborn has quit IRC21:19
*** limao has quit IRC21:21
*** akshai has joined #tripleo21:21
*** ebalduf has quit IRC21:23
*** rlandy has quit IRC21:23
*** akshai has quit IRC21:29
*** fragatin_ has joined #tripleo21:33
pradkEmilienM, i'm trying to add the ceilomiddleware to swift proxy ..
pradkbut ci seems to fail with .. Error: Could not find dependency Class[Ceilometer] for Concat::Fragment[swift_ceilometer] at /etc/puppet/modules/swift/manifests/proxy/ceilometer.pp:107\u001b[0m\n"21:36
pradkseems like a bug in puppet-swift?21:36
*** fragatina has quit IRC21:36
openstackgerritMerged openstack/python-tripleoclient: Remove excessive output when configuring nodes
*** mburned is now known as mburned_out21:38
*** akshai has joined #tripleo21:38
EmilienMpradk: yes21:39
EmilienMlet me git blame21:39
pradkwow thats quite old21:40
*** saneax is now known as saneax-_-|AFK21:40
EmilienM3 years and 5 months21:41
EmilienMI guess you can submit a patch ;-)21:41
EmilienMpuppet-swift has some interesting things sometimes21:41
pradksure i'll look into it first thing monday21:41
pradksurprised no one ran into it21:42
pradki guess its never been used21:42
EmilienMpradk: or ceilometer was installed on same node as swift proxy21:45
EmilienMpradk: at enovance, we built an installer based on puppet and ansible, that used this class21:45
EmilienMand it worked fine because had ceilometer api running on swift proxy nodes21:45
pradkEmilienM, hmm so this is a side effect of composable roles as they dont run on same node any more by default?21:47
EmilienMpradk: yep21:47
EmilienMyou found a good bug21:47
EmilienMjust submit a patch in puppet-swift, one line and we're good21:47
EmilienMmaybe tests needs to be updated21:47
EmilienMpradk: we're releasing puppet modules next week, better to do it asap but it could be backported worst case21:48
*** fragatin_ has quit IRC21:48
*** fragatina has joined #tripleo21:50
*** kjw3 has quit IRC21:50
*** ccamacho has quit IRC21:52
*** myoung is now known as myoung|gone21:54
*** lblanchard has quit IRC22:03
*** akshai has quit IRC22:04
*** ooolpbot has joined #tripleo22:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1624420 in tripleo "MongoDB Can't find master host for replicaset tripleo." [Critical,In progress] - Assigned to Jiří Stránský (jistr)22:10
*** fragatina has quit IRC22:10
*** fragatina has joined #tripleo22:10
*** ayoung_ has quit IRC22:24
*** yamahata has quit IRC22:25
EmilienMit seems like is passing the ovb jobs22:33
EmilienMpingtest worked on both jobs22:33
EmilienMI'll approve it22:33
*** yamahata has joined #tripleo22:37
*** panda is now known as panda|Zz22:38
EmilienMremoving alert on as it will merge in a few22:39
openstackLaunchpad bug 1624420 in tripleo "MongoDB Can't find master host for replicaset tripleo." [Critical,In progress] - Assigned to Jiří Stránský (jistr)22:39
EmilienMbeekneemech: if you still around please22:42
*** dtrainor has quit IRC22:43
*** david-lyle has joined #tripleo22:48
*** absubram has quit IRC22:52
*** noslzzp_ has quit IRC23:25
openstackgerritMonty Taylor proposed openstack/diskimage-builder: Add libselinux-python to yum-minimal
ayoungdeploying the overcloud twice in quick succession  gives me an error, but no update in state.  Is this expected?23:30
ayoungthe stack state is CREATE_COMPLETE23:31
*** cwolferh has quit IRC23:49
*** mburned_out is now known as mburned23:51

Generated by 2.14.0 by Marius Gedminas - find it at!