Tuesday, 2016-09-06

*** TSCHAK has quit IRC00:13
*** fpan has joined #tripleo00:17
*** fpan has quit IRC00:18
*** fpan has joined #tripleo00:18
*** saneax is now known as saneax-_-|AFK00:22
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci: [DO NOT MERGE] Test a fake periodic job, newton 3  https://review.openstack.org/36580200:36
*** limao has joined #tripleo00:41
*** tremble has quit IRC00:52
*** tremble has joined #tripleo00:53
*** tremble has quit IRC00:53
*** tremble has joined #tripleo00:53
*** yamahata has joined #tripleo01:21
*** akshai has joined #tripleo01:56
*** akshai_ has joined #tripleo02:29
*** akshai has quit IRC02:32
*** rhallisey has quit IRC02:33
stevebakerflaper87: hey, ramishra and I would like to chat about the docker-cmd hook when you have the chance02:36
*** limao has quit IRC02:41
*** limao has joined #tripleo02:43
*** limao has quit IRC02:46
*** limao has joined #tripleo02:46
*** bkopilov has quit IRC02:50
*** akshai_ has quit IRC02:52
*** tzumainn has joined #tripleo02:59
*** tzumainn has quit IRC03:11
*** bana_k has joined #tripleo03:41
*** coolsvap_ has joined #tripleo03:52
*** masco has joined #tripleo04:11
*** pgadiya has joined #tripleo04:17
*** bkopilov has joined #tripleo04:24
*** abregman has joined #tripleo04:37
*** links has joined #tripleo04:39
*** bana_k has quit IRC04:39
*** bana_k has joined #tripleo04:58
*** dsariel has joined #tripleo05:10
*** skramaja has joined #tripleo05:11
*** jaosorior has joined #tripleo05:21
*** cwolferh has joined #tripleo05:22
*** jprovazn has joined #tripleo05:22
*** florianf has joined #tripleo05:30
*** liverpooler has quit IRC05:42
*** oshvartz has joined #tripleo05:46
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485905:48
*** saneax-_-|AFK is now known as saneax05:48
jaosoriorbandini: sup dude05:51
*** atom_ has quit IRC05:53
*** mcornea has joined #tripleo05:54
bandinijaosorior: hola, que tal?05:55
*** abregman has quit IRC05:56
*** abregman has joined #tripleo05:57
jaosoriorbandini: nada mal, aquí tratando empezando la mañana... me hace falta café. Tu que tal?05:57
bandinijaosorior: lol a mi me falta cafe tambien :D05:57
bandinime voy a tomar uno ;)05:58
jaosoriorbandini: que tal vas de trabajo?05:59
bkeroquiero un cafe ahora :(06:00
jaosoriorbkero: jaja no sabía que tu también hablas español06:01
openstackgerritSaravanan KR proposed openstack/python-tripleoclient: Tripleoclient leaks temporary files  https://review.openstack.org/33063806:02
bkerojaosorior: perdon, es un lengua segunda pero es la primera lengua por Barcelona06:06
bkeroleguna seguna para mi :)06:06
jaosoriorbkero: bien ahí!06:11
bkerojaosorior: vives en espana?06:12
bkeroperdon para los accentos, mi teclado no tengo06:14
jaosoriorbkero: no pasa nada06:14
jaosoriorbkero: vivo en Finlandia, pero soy mexicano06:15
bkerooooo cool06:15
jaosoriorbkero: tu donde vives?06:16
bkerojaosorior: estados unidos, y tambien06:17
*** anshul has joined #tripleo06:17
jaosoriory tambien?06:17
bkerosoy uno americano06:18
*** hjensas has quit IRC06:18
jaosoriorbkero: cuanto llevas estudiando español?06:19
bkerojaosorior: si, en un universidad comunitario y Duolingo.com. Mi profesora en universidad era Colombina, no habla Ingles06:22
bkeroexcepto Madonna + Evita :)06:23
*** nyechiel_ has joined #tripleo06:24
bkeroperson, es tarde en la noche aqui. buenos noches!06:26
jaosoriorbkero: va, saludos06:27
*** pcaruana has joined #tripleo06:48
*** tremble has quit IRC06:50
jaosoriorshadower: you around yet?06:53
*** tremble has joined #tripleo06:54
shadowerjaosorior: just logged in :-)06:54
shadowerwhat's up06:54
jaosoriorshadower: haha was gonna poke you about reviews :P06:54
jaosoriorshadower: got time?06:55
shadowerjaosorior: sure06:55
shadowerthe fqdn stuff?06:56
jaosoriorshadower: that indeed06:56
shadowerwill do06:56
jaosoriorshadower: and this one  https://review.openstack.org/#/c/365522/506:56
*** zoli_gone-proxy is now known as zoliXXL06:57
*** dsariel has quit IRC06:57
*** mhenkel has joined #tripleo07:00
*** tesseract- has joined #tripleo07:00
openstackgerritAdriano Petrich proposed openstack/tripleo-quickstart: WIP gate upgrade  https://review.openstack.org/34216107:04
*** jpena|off is now known as jpena07:08
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776507:09
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add VIP names to allNodesConfig  https://review.openstack.org/36589507:09
*** liverpooler has joined #tripleo07:10
*** hjensas has joined #tripleo07:11
*** hjensas has quit IRC07:11
*** hjensas has joined #tripleo07:11
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Check for differences between introspected HW  https://review.openstack.org/35397807:16
*** ifarkas_afk is now known as ifarkas07:16
*** leanderthal|afk is now known as leanderthal07:19
*** ebarrera has joined #tripleo07:20
shadowerjaosorior: oh man that's some heavy yaqs/heat templating07:21
jaosoriorshadower: which commit?07:22
*** fzdarsky has joined #tripleo07:22
shadowerMove VIP-related data to AllNodesConfig https://review.openstack.org/#/c/365522/07:22
jaosoriorshadower: haha well, pretty much got it from shardy's work :P http://hardysteven.blogspot.fi/2016/09/complex-data-transformations-with.html07:23
shadowerjaosorior: I haven't read that but plan to07:23
jaosoriorshadower: aaaah, the VIP stuff is pretty much a copy paste of what was already in vip-config, so actually nothing really changed there07:24
shadowerjaosorior: oh! I didn't see the deleted file yet.07:24
jaosoriorshadower: so actually I didn't change much, except that it's now deployed in all the nodes instead of just the controllers07:25
jaosoriorshadower: the stuff that changes is the last bits of the vip-config07:25
jaosoriorshadower: the stuff that sets the virtual ip related stuff (keystone_public_api_vip, admin_vip and that stuff07:25
jaosoriorshadower: so instead of coming from an input, it's now calculated in-place07:25
shadowerright, I see07:26
*** zoliXXL is now known as zoli|brb07:28
*** jpich has joined #tripleo07:30
openstackgerritMartin André proposed openstack/tripleo-common: Fix error when identity file is missing  https://review.openstack.org/36590607:31
openstackgerritMartin André proposed openstack/tripleo-common: Handle missing mistral environment  https://review.openstack.org/36590707:31
*** jbadiapa has joined #tripleo07:32
jaosoriorjistr: hey dude, so I ended up changing the submission that set the cloudname entries for /etc/hosts. Now I don't even change the endpoint map (which is what people where doubting the most), I just get everything via hiera07:33
mandreshadower: ^ a couple of bug fixes for the run validations workflow when it is called before we create the SSH keys07:33
shadowermandre: I'll have a look, thanks07:34
*** aufi has joined #tripleo07:34
*** abregman has quit IRC07:35
*** zoli|brb is now known as zoliXXL07:37
*** jlinkes has joined #tripleo07:39
openstackgerritMerged openstack/tripleo-ui: Add app.conf to .gitignore  https://review.openstack.org/36561807:42
openstackgerritChristopher Brown proposed openstack/tripleo-docs: Change version to fix wwn extensions support  https://review.openstack.org/36591807:48
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485907:49
*** bana_k has quit IRC07:49
*** dbecker has joined #tripleo07:49
*** abregman has joined #tripleo07:51
jaosoriormarios, shadower, jistr: Do you guys mind if I merge this? https://review.openstack.org/#/c/365522/07:52
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Implement Validation Detail modal  https://review.openstack.org/36592107:56
shadowerjaosorior: fine by me, but I should not be merging non-validation stuff yet (which is why I only +2)07:56
jaosoriorshadower: no biggie07:56
shadowerI'm trying to do more non-validation reviews to get familiar & be able to +A again07:57
jaosoriorshadower: makes sense man07:57
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Customize undercloud and overcloud with virt-customize  https://review.openstack.org/36111207:59
mariosjaosorior: no objection from me08:00
jtomasekshadower: this is how the validation detail looks in GUI now http://imgur.com/a/PGchd08:00
*** bvandenh has joined #tripleo08:02
*** dbecker has quit IRC08:02
shadowerjtomasek: right, so it's just the raw output from Ansible. Can you run the "discovery diff" one? It should show a bunch of warnings08:03
* shadower wants to be sure those are displayed, too08:04
*** abregman has quit IRC08:04
jtomasekshadower: which one is it? is that one in tripleo-validations already?08:05
*** abregman has joined #tripleo08:05
shadowerjtomasek: ah right. It's not :-(08:05
shadowerjtomasek: if I sent you a patch, could you add it in easily?08:06
shadower(don't bother for now if you can't)08:06
jtomasekshadower: where is that patch?08:07
shadowerjtomasek: https://review.openstack.org/#/c/353978/08:07
*** jistr is now known as jistr|dentist08:07
shadowerjtomasek: hm, it's failing for me now08:09
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service  https://review.openstack.org/35643008:09
*** ohamada has joined #tripleo08:11
jtomasekshadower: ok, tested it, and fails too. I can check it again whenever you want08:12
*** skramaja has quit IRC08:13
shadowerjtomasek: yeah, thanks. It's not the validation itself but seems we have a bug there somewhere. Investigating now08:13
*** skramaja has joined #tripleo08:13
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: IGNORE - testing manila-generic deploy after tidyup  https://review.openstack.org/36286608:14
*** derekh has joined #tripleo08:15
*** lucas-dinner is now known as lucasagomes08:16
*** b00tcat has joined #tripleo08:21
b00tcatHi, I'm following the tripleo docs and am now in the "Basic Deployment (CLI)" section - there it says that I should install the trunk delorean repos inside the instack-undercloud machine08:22
b00tcathowever in the previous step I installed the `delorean-mitaka` ones, is this step mandatory?08:22
b00tcatforget that, I read below and this is already covered for stable-mitaka :D08:25
flepiedperiodic job succeeded to promote last night or was it promoted by hand?08:30
*** nyechiel_ has quit IRC08:31
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776508:33
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service  https://review.openstack.org/35643008:33
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add parameters for internal TLS  https://review.openstack.org/36594208:33
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add parameters for internal TLS  https://review.openstack.org/36594208:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service  https://review.openstack.org/35643008:35
*** bvandenh has quit IRC08:37
shadowermandre: just discovered this: https://review.openstack.org/#/c/353978/ (see my last comment)08:42
shadowermandre: would appreciate your feedback08:42
openstackgerritMerged openstack/tripleo-heat-templates: Move VIP-related data to AllNodesConfig  https://review.openstack.org/36552208:44
openstackgerritMerged openstack/tripleo-common: Prevent replacement of Port, Net and Subnet  https://review.openstack.org/35341108:45
*** nyechiel_ has joined #tripleo08:45
*** athomas has joined #tripleo08:46
*** abregman has quit IRC08:46
shadowerjtomasek: so a quick fix is to edit /etc/ansible/ansible.cfg, find "sudo_flags" and set it to "-H -S -n -E"08:48
mandreshadower: have you tried the solution #3 you proposed?08:48
shadowermandre: yeah, and it works fine08:48
shadowermandre: actually, I haven't tried #1 but can't see why it wouldn't work. #2 works fine, too08:49
mandreshadower: rather than edit /etc/ansible/ansible.cfg, I prefer you export a variable in run-validations script08:49
mandreshadower: that's the general pattern I adopted to configure ansible08:49
shadowermandre: ah, good point. Not sure that's possible here (but I think so)08:50
shadowermandre: one problem with that is we can no longer use Ansible directly08:50
shadowerhm but we can just use the run-validation script so w/e08:50
shadowerI'll give it a go08:50
mandreshadower: either you'll use the run-validation wrapper or you call ansible-playbook with the the corresponding CLI option08:51
shadowermandre: yeah I think that's fair (though we should document that)08:51
*** dsariel has joined #tripleo08:54
openstackgerritDerek Higgins proposed openstack/tripleo-common: Add port registration to IPMI power driver  https://review.openstack.org/36595308:57
mandreshadower: supposing we go with #2, can we imagine a small helper module that export the environment variables after becoming root?08:58
mandreshadower: this way the validation authors decide when to become root and it's not too painful to have the proper environment variables08:59
*** abregman has joined #tripleo08:59
shadowermandre: yea I was thinking something similar. Though you need to call it before you become root09:00
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Remove the ha2 JOBTYPE  https://review.openstack.org/36341109:00
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485909:01
openstackgerritLucas Alvares Gomes proposed openstack/tripleo-common: Test baremetal: Do not mock at module level  https://review.openstack.org/36595809:01
openstackgerritLucas Alvares Gomes proposed openstack/tripleo-common: Ironic Client: Bump the max_retries and retry_interval  https://review.openstack.org/36569409:02
*** florianf has quit IRC09:02
mandreshadower: right... or we configure ansible to pass the environment variables with sudo like we discussed, and we stop defaulting to 'become: true' in the inventory script09:02
mandreshadower: the 'become: true' will need to be explicit in the validations tasks09:03
openstackgerritLucas Alvares Gomes proposed openstack/tripleo-common: Test baremetal: Do not mock at module level  https://review.openstack.org/36595809:03
*** florianf has joined #tripleo09:04
openstackgerritLucas Alvares Gomes proposed openstack/tripleo-common: Ironic Client: Bump the max_retries and retry_interval  https://review.openstack.org/36569409:04
shadowermandre: right that's an option, too. My proposals were to keep the same functionality, but make the env vars available09:04
shadowermandre: so, setting sudo flags in run-validation works fine09:05
shadowermandre: I'm happy to create that patch if you're fine with that09:06
mandreshadower: let's do it in 2 steps, first set ansible to maintain environment variables with sudo, then in a separate commit get rid of the 'become:true' in the inventory script and be explicit in the validations files that require it09:06
shadowermandre: sounds got to me. Expect patches soon :-)09:07
mandreshadower: great09:07
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add service networks to hieradata  https://review.openstack.org/36547509:09
*** nyechiel_ has quit IRC09:10
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements  https://review.openstack.org/36150109:13
*** limao has quit IRC09:19
openstackgerritTomas Sedovic proposed openstack/tripleo-common: Preserve the shell environment for run-validation  https://review.openstack.org/36598309:20
shadowermandre: that's the first one ^09:20
*** r-mibu has quit IRC09:23
*** jistr|dentist is now known as jistr09:23
shadowermandre: should we remove become from the overcloud nodes, too?09:26
*** nyechiel_ has joined #tripleo09:28
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Remove `become: true` from the inventory script  https://review.openstack.org/36601009:30
shadowermandre: and that's the other one ^09:30
jtomasekshadower: I am getting this now: "No password entered, or found via --os-password or OS_PASSWORD"09:30
shadowerjtomasek: hm. That's exactly what this was supposed to fix09:31
shadowerjtomasek: can you pastebin /etc/ansible/ansible.cfg?09:31
jtomasekshadower: do I need to restart anything after changing that config?09:31
shadowerjtomasek: nope09:32
shadowerat least I don't think. I didn't need to (but I'm not going through Mistral)09:32
shadowerjtomasek: anyway, this should be fixed soon so don't worry about it now09:33
*** r-mibu has joined #tripleo09:33
*** nyechiel_ has quit IRC09:33
b00tcatif my undercloud is a VM inside a baremetal server (which I installed using instack-undercloud), can I tell the undercloud to provision the overcloud controller as a VM on this baremetal server?09:36
*** mcornea is now known as mcornea|afk09:41
*** shardy_afk has joined #tripleo09:43
*** saneax is now known as saneax-_-|AFK09:43
lucasagomesderekh, something really fishy in the unittests for python3.5 and that patch to bump the timeout09:43
lucasagomesderekh, somewhere _get_baremetal_client() is being mocked and it's leaking on the test I just added09:44
* lucasagomes digs more09:44
*** shardy has joined #tripleo09:44
derekhhmm, weird09:46
*** saneax-_-|AFK is now known as saneax09:52
*** akrivoka has joined #tripleo09:55
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Implement Validation Detail modal  https://review.openstack.org/36592109:55
*** dtantsur|afk is now known as dtantsur09:59
*** pgadiya has quit IRC10:02
openstackgerritLucas Alvares Gomes proposed openstack/tripleo-common: Test baremetal: Correctly stop the mocks  https://review.openstack.org/36595810:02
lucasagomesderekh, ^ a-ha, I think I find it10:02
openstackgerritLucas Alvares Gomes proposed openstack/tripleo-common: Ironic Client: Bump the max_retries and retry_interval  https://review.openstack.org/36569410:03
derekhlucasagomes: nice :-)10:03
jpichlucasagomes: Yeah, something similar solves it for me locally as well -- sorry about this10:04
lucasagomesjpich, o/ I didn't look at the other modules, maybe the patch should fix it all10:04
* lucasagomes looks10:04
jpichlucasagomes: Looks like the other place that uses the patcher stops it correctly :)10:05
lucasagomesjpich, yay, great then! Thanks for verifying10:06
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Make the ini-based validations more robust  https://review.openstack.org/36602510:06
jpichlucasagomes: Thanks for finding and fixing this!10:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move AllNodesValidationDeployments into jinja template loop  https://review.openstack.org/33758710:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating  https://review.openstack.org/31567910:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move RedisVirtualIP hiera into VipMap  https://review.openstack.org/36579110:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Convert allNodesConfig properties to composable jinja2  https://review.openstack.org/36579410:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move AllNodesDeployments into jinja template loop  https://review.openstack.org/33726710:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move role ResourceGroups inside the jinja2 loop  https://review.openstack.org/36579310:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move role deployment steps into overcloud.yaml  https://review.openstack.org/36576310:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Remove *ExtraConfig parameters from overcloud.yaml  https://review.openstack.org/36579210:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move per-role NetIpListMap's into jinja template loop  https://review.openstack.org/36474910:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Convert SwiftDevicesAndProxyConfig to composable format  https://review.openstack.org/36474810:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Convert deploy steps to jinja2 loop  https://review.openstack.org/36579610:07
shadowermandre:  https://review.openstack.org/366025 noticed a couple of (unrelated) failures when testing the removed become.10:07
shardyjistr: Hey, I rebased the custom-roles series as it conflicted with jaosorior's patch that landed10:08
shardyjistr: are you still working on the SwiftDevicesAndProxyConfig patches?10:08
shardythose are the head of the branch now, so it'd be good to get them ready to land asap10:08
*** bvandenh has joined #tripleo10:09
mandreshadower: gotcha... do you mind creating an issue in LP for the bug you found?10:10
*** sshnaidm|afk is now known as sshnaidm10:10
shadowermandre: will do10:10
*** pgadiya has joined #tripleo10:10
mandreshadower: thx, I'd like to start tracking bugs more effectively10:12
shadowermandre: yea agreed10:12
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Make the ini-based validations more robust  https://review.openstack.org/36602510:15
*** pgadiya has quit IRC10:15
shadowermandre: done ^10:15
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Make the ini-based validations more robust  https://review.openstack.org/36602510:16
*** nyechiel_ has joined #tripleo10:16
*** pgadiya has joined #tripleo10:17
*** dmanchad has joined #tripleo10:22
*** nyechiel_ has quit IRC10:22
* jistr back from lunch10:25
jistrshardy: hi, yea i just tested it, overcloud deploys and swift seems to work, so i think they're ready to go10:26
jistri'll un-wip the puppet one10:26
shardyjistr: ack, thanks - I've also been testing it locally and all seemed OK, but I've not checked things closely other than a basic deployment completes10:27
*** mcornea|afk is now known as mcornea10:27
*** zoliXXL is now known as zoli|lunch10:28
*** ramishra has quit IRC10:30
*** paramite has joined #tripleo10:31
jistrshardy: i tried also uploading a swift object and looked at the .builder files briefly, they seem to be sane on both controller and swift node10:31
jistrso i hope we're ok there10:31
shardyjistr: Ok, sounds good - lets see if the rebased t-h-t patch passes CI then hopefully we can land them both10:32
*** ramishra has joined #tripleo10:32
*** pgadiya has quit IRC10:35
*** athomas has quit IRC10:37
*** akrivoka has quit IRC10:38
*** athomas has joined #tripleo10:41
*** abregman is now known as abregman|brb10:41
*** pblaho has joined #tripleo10:43
jtomasekmandre, shadower: found a bug here https://github.com/openstack/tripleo-common/blob/master/workbooks/validations.yaml#L12210:43
jtomasekthe action name is wrong10:43
*** tobias_fiberdata has joined #tripleo10:45
*** HenryG_ is now known as HenryG10:46
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Add bootstrap_node and vip_data to hierarchy for all roles  https://review.openstack.org/36604910:47
*** tobias-fiberdata has quit IRC10:48
shadowerjtomasek: ouch, I'll send a patch10:50
jtomasekshadower: we'll need to track it for rc1 and get it merged asap, it blocks running validation groups automatically from GUI10:51
*** pgadiya has joined #tripleo10:51
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Remove excessive output when configuring nodes  https://review.openstack.org/36605110:51
shadowerjtomasek: tell you what: you submit the bug and I'll do the fix in the meantime?10:52
shardyjaosorior: ^^ FYI mcornea found an issue ref bug 1620485 which I think will mean you also can't use your vip_data from allNodesConfig on non-Controller roles10:52
openstackbug 1620485 in tripleo "Deployment fails when MongoDB runs on a different node than controller with missing bootstrap_nodeid hiera" [High,In progress] https://launchpad.net/bugs/1620485 - Assigned to Steven Hardy (shardy)10:52
shardyjaosorior: that patch should fix it I think10:52
jtomasekshadower: ack:)10:52
*** abregman|brb is now known as abregman10:52
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Customize undercloud and overcloud with virt-customize  https://review.openstack.org/36111210:53
jaosoriorshardy: ok10:55
jtomasekshadower, shardy https://bugs.launchpad.net/tripleo/+bug/162057310:58
openstackLaunchpad bug 1620573 in tripleo "Incorrect action name in run_groups validation workflow" [High,New] - Assigned to Tomas Sedovic (tsedovic)10:58
*** dprince has joined #tripleo11:00
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776511:01
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add VIP names to allNodesConfig  https://review.openstack.org/36589511:01
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add parameters for internal TLS  https://review.openstack.org/36594211:01
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service  https://review.openstack.org/35643011:01
openstackgerritTomas Sedovic proposed openstack/tripleo-common: Fix the validation action name in the workbook  https://review.openstack.org/36605511:03
shadowerjtomasek: ^11:03
shadowerjtomasek: I've checked the other action names and they should be all fine. I *have* checked this before but apparently something slipped through :-(11:04
jaosoriorshardy: the fix makes sense. I messed up cause I was seeing in the nodes that vip_data was indeed a file in /etc/puppet. But had forgotten to add it to the hierarchy list :/11:05
jtomasekshadower: yeah, it got through a review. it is quite hard to catch this11:05
jaosoriorshardy: is there a way to pass inputs to a softwareconfig through a way that doesn't involve doing so through a softwaredeployment? I'm looking to passing parameters (with get_param for instance) to a softwareconfig but defer the softwaredeployment to later in the process11:07
jaosoriorshardy: for instance, the TLS stuff that's there at the moment. Would be cool to be able to move that to the haproxy profile... but currently we can't do that cause the software deployment requires the reference to the server.11:07
jaosoriorand we don't have such a reference in the service profiles.11:08
jpichjtomasek, shadower: Wish we could unit tests the workflows' yaml, would have spared me some confusion too :)11:10
*** shardy_afk has quit IRC11:11
shardyjaosorior: Yes, you can just pass the values directly into the SoftwareConfig resource, e.g look at the old VipConfig resource you recently removed :)11:12
shardythat worked like that, as does allNodesConfig11:12
*** rhallisey has joined #tripleo11:13
jaosoriorshardy: but that's a structuredconfig11:13
shardyjaosorior: StructuredConfig works the same as SoftwareConfig, the only difference is some variable subsitution and the output format11:14
jaosoriorshardy: right, so in this case I need to do the get_param calls inside the script definition?11:14
shardyjaosorior: No, you can just define an input, then set the default of the input to get_param: foo11:15
shardyI suppose that works a little different to how we currently use StructuredConfig, you're right11:16
jaosoriorshardy: will the default values appear when inspecting the stack?11:16
*** tremble has quit IRC11:16
shardyjaosorior: define inspecting - you can retrieve the inputs schema for a SoftwareConfig, but I'm not sure why you'd need to?11:16
jaosoriorthe ovb jobs seem broken11:17
jaosoriorderekh: hey dude, any idea what this is? http://logs.openstack.org/83/365583/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/32dbf65/console.html#_2016-09-06_11_08_49_39896711:17
jaosoriorshardy: not the input schema, but the actual inputs given. Or the defaults in this case.11:18
derekhjaosorior: looks like an error while creating testenvs, is it happening now on all jobs?11:19
jaosoriorderekh: I've seen it in 4 jobs now11:19
*** tremble has joined #tripleo11:19
shardyjaosorior: Not sure, it's possibly accessible via the show attribute: http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::SoftwareConfig-attr-show11:19
derekhjaosorior: taking a look11:19
shardyagain, I don't understand why, as you just need to reference the ID and the config encapsulates the input schema, which includes the defaults?11:20
derekhsshnaidm: ^^ fyi, taking a look on the te-broker to find the error11:20
jaosoriorshardy: and there is no way of making inputs hidden?11:20
openstackgerritKeith Schincke proposed openstack/tripleo-heat-templates: Add support for deploying Ceph RGW role  https://review.openstack.org/28902711:20
jaosoriorshardy: if it's key material, it would be best to not show that when someone tries to inspect it.11:21
shardyjaosorior: the parameters can be marked hidden, but we can't hide the value from all API output, or the SoftwarDeployment can't read it11:21
jaosoriorshardy: in this case parameters are marked as hidden, so that part is covered.11:22
shardyjaosorior: Yeah, so that will hide them from casual inspection e.g via horizon or whatever11:22
*** lucasagomes is now known as lucas-hungry11:22
shardybut if you've got credentials to access the heat API, you'll be able to get the values or the SoftwareDeployment couldn't consume them11:22
jaosoriorshardy: alright, so I guess there's no harm in moving the get_param to the default value of those inputs. As we can already view them when checking the softwaredeployment anyway11:24
shardyjaosorior: example https://paste.fedoraproject.org/422773/73161112/11:25
jaosoriorshardy: alright11:25
shardyyou can see the inputs schema there, and it contains the default11:25
jaosoriorshardy: right now I guess that's also the case for the keys that we inject with the current solution, right?11:26
*** bkopilov has quit IRC11:27
shardyjaosorior: yup11:27
*** zephcom has quit IRC11:28
openstackgerritJulie Pichon proposed openstack/tripleo-common: Clean-up messages in the 'configure' workflow  https://review.openstack.org/36606111:28
*** tremble has quit IRC11:28
*** thrash|g0ne is now known as thrash11:28
*** zephcom has joined #tripleo11:29
openstackgerritJulie Pichon proposed openstack/tripleo-common: Clean up messages in the 'configure' workflow  https://review.openstack.org/36606111:29
derekhjaosorior: hmmm, looks like heat is loosing connection to the db11:30
derekhheat/heat-engine.log:2016-09-06 11:18:01.232 46222 INFO heat.engine.stack [-] Stack DELETE FAILED (baremetal_8175): Resource DELETE failed: DBConnectionError: resources.baremetal_env.resources.openstack_baremetal_servers.resources[1]: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')11:30
skramajashardy: we have added an environment file for SR-IOV, can you please take a look? https://review.openstack.org/#/c/363617/ has a +211:30
derekhjaosorior: lots of them ^11:30
*** tremble has joined #tripleo11:30
*** tremble has quit IRC11:30
*** tremble has joined #tripleo11:30
jaosoriorwhat the hell11:30
jaosoriorsshnaidm: you had mentioned there were some issues you were seeing in the periodic jobs involving the heat database. Is this realted?11:31
jaosoriorderekh: wait... is this related to the te-broker issue?11:32
sshnaidmjaosorior, I don't think so, in this case connection is lost, in mine it was incorrect table value11:32
derekhjaosorior: this is a problem on the rh1 cloud itself,11:32
jaosoriorderekh: oh11:32
jaosoriorI don't have access to that :/11:33
jaosoriorderekh: is there a way I can help figure out what's up there? Like a place to check logs or something of the sort11:33
sshnaidmderekh, which log file is it?11:33
derekhjaosorior: nope, the logs are only available to the cloud admins11:33
derekhsshnaidm: heat/heat-engine.log11:34
sshnaidmderekh, on overcloud controller?11:34
shardyskramaja: lgtm, approved - thanks!11:34
derekhsshnaidm: yup11:35
shardyskramaja: is that the last patch to complete the sr-iov blueprint?11:35
skramajathanks shardy.. one more pending..11:35
skramajaCI is failing.. i will let u know once done..11:35
jaosoriorjistr, shardy: got time for a quick review? https://review.openstack.org/#/c/365583 I'm trying to get those FQDN facts to be more usable through t-h-t. So that gets them to match the network names. It passed CI (nonha and ha) in two different runs, but right now the ovb jobs are broken :/11:35
shardyskramaja: Ok, thanks11:35
shadowerjaosorior: can haz a quick review? 2 line patch https://review.openstack.org/#/c/366010/11:36
jaosoriorshadower: sure11:36
derekhsshnaidm: jaosorior I'm guessing some sql queries have gotten too slow due to table's growing11:36
derekhshardy: FYI ^^ problems creating envs on RH111:37
jaosoriorderekh: Are those leftovers in the database? There is a cron-job available to clean those up, no?11:37
sshnaidmderekh, I see this also: http://paste.openstack.org/show/567126/11:38
openstackgerritMerged openstack/tripleo-validations: Remove `become: true` from the inventory script  https://review.openstack.org/36601011:40
derekhsshnaidm: yup, may also be related11:40
derekhjaosorior: doesn't look like we have any clean up cron jobs11:40
flepiedperiodic job succeeded to promote last night or was it promoted by hand?11:40
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Run Validations automatically  https://review.openstack.org/36606811:41
jaosoriorderekh: bummer, but there is this available: heat-manage purge_deleted -g days 3011:42
jaosoriorderekh: to clean up the deleted stacks11:42
*** fragatina has joined #tripleo11:42
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Check for differences between introspected HW  https://review.openstack.org/35397811:42
*** xuao has joined #tripleo11:42
*** dprince has quit IRC11:43
*** fragatina has quit IRC11:43
derekhjaosorior: thanks, I havn't ever run it, but I think we should give it a whirl11:43
*** dprince has joined #tripleo11:43
derekhsshnaidm: bnemec "heat-manage purge_deleted -g days 30" <-- on the RH1 overcloud controller, what do ye think?11:44
openstackgerritMerged openstack-infra/tripleo-ci: Retry stack-delete in case of failure  https://review.openstack.org/35064011:44
sshnaidmflepied, I think manually by weshay11:44
flepiedok thx sshnaidm11:45
shardyderekh: do we even need 30 days of deleted stuff in the DB?11:45
shardyI'm just wondering if anyone ever looks at it11:45
sshnaidmderekh, isn't it cleaned after heat stack is deleted?11:45
shadowerjtomasek: so if you pull tripleo-validations and checkout this: https://review.openstack.org/#/c/353978/ it should be working11:46
shardysshnaidm: No, heat soft-deletes stack like Nova does11:46
*** pkovar has joined #tripleo11:46
shadowerjtomasek: (that's the patch from the morning I wanted to see the details of)11:46
*** tobias-fiberdata has joined #tripleo11:46
derekhshardy: I've never looked at them, we could probably just keep a day or 2 just incase anybody ever want to debug something11:46
shardysome data is deleted when the stack is deleted, not not everything so there's an audit trail e.g the stack owner and events11:46
derekhsshnaidm: the stacks are deleted but in the DB they still exist and marked as deleted11:47
shardyderekh: Yeah, I was thinking something like 7 days would be ample, but I guess any purge is better than none :)11:47
sshnaidmshardy, derekh  so there will be a problem with any long-running openstack?11:48
derekhshardy: ok, before I run it, I want to quickly see if I can turn on slow query logging to see how bad things have gotten11:48
*** jpena is now known as jpena|lunch11:48
jtomasekshadower: hm, I see still the same error. I did git review -d 353978 and sudo pip install .11:49
derekhsshnaidm: possibly, (if this is the actual problem)11:49
shardysshnaidm: Yes, but normally with TripleO deployed OpenStack we setup cron jobs which periodically purge the Heat (and other) DB's11:49
*** tobias_fiberdata has quit IRC11:49
jtomasekshadower: can you try to run that workflow?11:50
sshnaidmderekh, could it be problem with all these non-deleted stacks (DELETE_FAILED) that take resources and don't allow to create a new ones?11:50
*** bfournie has quit IRC11:50
derekhanybody know if you can turn on log_slow_queries on a running mysql server?11:50
*** masco has quit IRC11:50
derekhsshnaidm: yup, we should clean those up also11:50
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Hook internal TLS flag to apache-based services  https://review.openstack.org/36607511:51
shadowerjtomasek: I didn't have time to set up the mistral env properly yet :-( (so I tested it directly with ansible)11:51
shadowerjtomasek: let me have a look11:51
derekhgot it "SET GLOBAL slow_query_log = 'ON'"11:51
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002  https://review.openstack.org/36502611:52
shadowerjtomasek: can you grep for "become" in tripleo-common/scripts/tripleo-ansible-inventory?11:52
shadowerjtomasek: it should come out empty11:52
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: Composable Zaqar services  https://review.openstack.org/33168211:52
derekhsshnaidm: just gonna leave that for a couple of minutes, then we can purge the stale entries and delete the DELETE_FAILED stacks11:52
dtantsurhmm, is it normal for HA on virt to take 1.5 hours to deploy? I think it used to be faster11:52
*** abregman has quit IRC11:53
sshnaidmderekh, ok11:53
*** abregman has joined #tripleo11:54
thrashEmilienM: would you mind checking that I have the depends-on in a sane state for the zaqar patches? We have scenario002 being triggered correctly via zuul.11:54
thrashEmilienM: https://review.openstack.org/#/c/331681/11:54
EmilienMthrash: you need the tripleo-ci patch in depends-on11:54
jaosoriorthrash: are you looking into setting up zaqar for the overcloud?11:54
EmilienMthrash: otherwise you won't have the right pingtest and right template to deploy.11:55
thrashEmilienM: https://review.openstack.org/#/c/365026/ and https://review.openstack.org/#/c/331682/ I switched up the dependency so I think I have it correct now11:55
thrashjaosorior: yes.11:55
EmilienMthrash: yes it's good now!11:55
jtomasekshadower: hmm, I don't have that script11:55
thrashEmilienM: swwweeeeetttt11:55
jaosoriorthrash: I was looking into getting zaqar's websocket endpoint to be terminated by haproxy11:55
shadowerjtomasek: I'm sorry -- tripleo-validations/scripts/...11:55
jaosoriorthrash: but I never got it to work with TLS11:55
jtomasekah, ok11:55
shadowerjtomasek: which come to think of it needs setup.py as well to get it working properly11:56
jaosoriorthrash: I guess you'll need that to get it working11:56
jaosoriorthrash: else you won't get websockets in the overcloud11:56
jtomasekshadower: 'become' is not in that script11:56
thrashjaosorior: https://review.openstack.org/#/c/331682/ is the zaqar composable.11:56
shardyReviews of https://review.openstack.org/#/c/365783/ would be good11:56
jaosorioror you might but when someone enables TLS, it'll break11:56
thrashjaosorior: ack11:57
shardywithout it upload-puppet-modules breaks now we switched to mistral/swift for plan storage11:57
shadowerjtomasek: right, so try running "sudo python setup.py install" for tripleo-validations and that should work11:57
openstackgerritMartin André proposed openstack/tripleo-common: Handle missing mistral environment  https://review.openstack.org/36590711:57
jaosoriorthrash: so this was my attempt at adding it to haproxy https://review.openstack.org/#/c/360329/11:57
jaosoriorthrash: but yeah, I think that needs to work before getting zaqar into the overcloud11:59
thrashjaosorior: +112:00
jaosoriorthrash: TLS for the public endpoints is a pretty common use-case12:00
thrashjaosorior: it won't be enabled by default to start.12:00
jaosoriorthrash: so, if you have any ideas on how to get the websockets over TLS with HAProxy in the middle working... would be cool. I didn't figure it out :(12:00
thrashjaosorior: I'll take a look12:01
*** jeckersb_gone is now known as jeckersb12:01
jaosoriorEmilienM: could you take a look at this? https://review.openstack.org/#/c/365583/12:01
EmilienMjaosorior: sure12:02
jtomasekshadower: I tried a buch of tricks but still the same result http://paste.openstack.org/show/567131/12:02
*** tobias_fiberdata has joined #tripleo12:02
*** zoli|lunch is now known as zoliXXL12:03
*** jeckersb is now known as jeckersb_gone12:03
EmilienMjaosorior: it looks like ovb jobs aren't green12:04
EmilienMand never be12:04
jaosoriorEmilienM: ovb is broken in general12:04
EmilienMthat is not true12:04
jaosoriorderekh: ^^12:04
shadowerjtomasek: does "grep become /usr/bin/tripleo-ansible-inventory" come out empty?12:04
shardyYeah, we've been suffering from some specific problems12:04
EmilienMI saw successful jobs this week end12:04
shardyuntil recently OVB was working much better than our previous CI setup12:04
jaosoriorEmilienM: broke today12:04
EmilienMit was actually fast12:04
EmilienMok but your patch was not passing OVB jobs before either12:05
jtomasekshadower: yes12:05
jaosoriorEmilienM: wasn't it?12:05
*** tobias-fiberdata has quit IRC12:05
EmilienManyway I don't think we should merge this one without OVB vote12:05
EmilienMjaosorior: the ha job wasn't12:05
shadowerjtomasek: I'm so confused. Any chance I could poke around?12:05
EmilienMshardy: what broke?12:05
jtomasekshadower: sure PM12:06
jaosoriorEmilienM: first run failed HA and passed nonha, second run failed nonha and passed ha12:06
*** trown|outtypewww is now known as trown12:06
jaosoriorEmilienM: it's a problem with rh1 cloud itself.12:06
shardyEmilienM: there is an error for heat connecting to the DB which broke some recent jobs, derekh is investigating12:07
jaosoriorEmilienM: this job depends on the puppet change I passed you and is green on ovb https://review.openstack.org/#/c/365585/12:07
shardyand before that we were experiencing performance problems which have been improved by reconfiguring the hardware, and switching back to cached images12:07
*** jayg|g0n3 is now known as jayg12:08
*** saneax is now known as saneax-_-|AFK12:08
*** pgadiya has quit IRC12:10
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Convert SwiftDevicesAndProxyConfig to composable format  https://review.openstack.org/36474812:10
derekhsshnaidm: no slow queries got logged, I'm just ganna run the purge command, or do you want to do it ?12:11
sshnaidmderekh, I will12:12
*** maeca1 has joined #tripleo12:12
derekhsshnaidm: ok12:12
*** akrivoka has joined #tripleo12:12
EmilienMjaosorior: ok, then good12:13
*** fragatina has joined #tripleo12:13
sshnaidmderekh, done12:14
*** saneax-_-|AFK is now known as saneax12:15
jaosoriorEmilienM: hey dude, do you have a write up (or something) on how the scenario tests work in tripleo?12:16
EmilienMjaosorior: it's on my list12:16
EmilienMjaosorior: for now I have written this https://github.com/openstack-infra/tripleo-ci#service-testing-matrix12:16
EmilienMjaosorior: you can see this example https://review.openstack.org/#/c/365026/12:17
EmilienMjaosorior: and https://review.openstack.org/#/c/365023/12:17
jaosoriorI see12:18
jaosoriorEmilienM: are the scenarios deployed in ovb or is it in the multinode setup?12:19
*** akrivoka has quit IRC12:19
*** akrivoka has joined #tripleo12:20
EmilienMjaosorior: multinode12:21
*** liverpooler has quit IRC12:21
*** tobias-fiberdata has joined #tripleo12:21
jaosoriorEmilienM: and is there a way to run those manually?12:22
*** fragatina has quit IRC12:22
*** xuao has quit IRC12:24
*** tobias_fiberdata has quit IRC12:25
*** tobias_fiberdata has joined #tripleo12:25
*** tobias-fiberdata has quit IRC12:26
*** lucas-hungry is now known as lucasagomes12:27
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Run Validations automatically  https://review.openstack.org/36606812:27
*** pgadiya has joined #tripleo12:28
*** rlandy has joined #tripleo12:30
*** zoliXXL is now known as zoli|brb12:32
*** fultonj has joined #tripleo12:32
derekhsshnaidm: the tables don't seem to have gotten any smaller, did the command take long to run ?12:34
sshnaidmderekh, nope12:35
sshnaidmderekh, it was quick12:35
jaosoriorderekh: how many days did you give?12:36
sshnaidmheat-manage purge_deleted -g days 3012:36
derekhsshnaidm: try 7 days maybe to see what happens12:36
jaosoriorderekh, sshnaidm: Might want to try it with 7 days?12:36
jaosoriorderekh: ??12:37
*** myoung|afk is now known as myoung12:37
*** bfournie has joined #tripleo12:38
jaosoriorderekh, sshnaidm if that doesn't work either, might be worth specifying the path to the config file explicitly12:38
derekhjaosorior: snap : we said the same thing at the same time, maybe its an irish thing12:38
sshnaidmderekh, done, try to check now pls12:38
sshnaidmjaosorior, it tries to use: /usr/share/heat/heat-dist.conf,/etc/heat/heat.conf12:39
jaosoriorshardy: ok, so then lets hope that 7 days does the trick12:40
EmilienMif anyone has off-items for our tripleo meeting this week https://etherpad.openstack.org/p/tripleo-meeting-items12:40
*** dmsimard|pto is now known as dmsimard12:43
EmilienMshardy: I saw your link, isn't it for the release item?12:46
shardyEmilienM: Yeah we can cover it during the release topic, I just wanted to highlight it12:47
shardyso folks can start thinking about the status and check out the RC1 link12:47
*** jpena|lunch is now known as jpena12:48
*** jaosorior has quit IRC12:48
*** noslzzp has joined #tripleo12:49
*** jaosorior has joined #tripleo12:49
jaosorior /exit12:49
*** jaosorior has quit IRC12:49
derekhsshnaidm: nope, those 4 tables are still the same size12:49
derekhsshnaidm: who are you running it as? /me wonders if it needs to be root12:50
*** pcaruana has quit IRC12:50
sshnaidmderekh, yep, sudo12:50
*** jaosorior has joined #tripleo12:51
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Customize undercloud and overcloud with virt-customize  https://review.openstack.org/36111212:51
derekhsshnaidm: how about we leave out the days thing alltogether, we shouldn't need them anyways12:51
* derekh is just thinking out loud here12:51
sshnaidmderekh, sorry, don't follow12:52
sshnaidmderekh, you mean to purge all?12:52
derekhsshnaidm: just run "heat-manage purge_deleted" and see what happens, it should only purge things that have already been deleted anyways12:53
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Save console logs from all vms to files  https://review.openstack.org/36536912:54
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Fix aodh auth url to remove suffix  https://review.openstack.org/36511712:54
sshnaidmderekh, done, but it was fast..12:55
sshnaidmderekh, I'm not sure it does something..12:55
EmilienMpradk: /me still investigating why gnocchi/aodh fails in scenario00112:55
EmilienMshardy: when you have time I would like to revisit https://review.openstack.org/#/c/364355/12:55
EmilienMshardy: put your feedback12:56
jaosoriorEmilienM: I've actually seen that same failure in another commit12:56
jaosoriorfrom scenario00112:56
jaosoriordidn't figure it out. Seems to me like the stack creation is left IN_PROGRESS and it doesn't actually say what failed :/12:56
*** tzumainn has joined #tripleo12:56
derekhsshnaidm: ya, looks like it did nothing,12:56
shardyEmilienM: thanks - that's probably a good topic for the meeting actually, as I'd like wider feedback on it12:57
shardyI'm +1 on it tho, we just had discussed os-*-config not being tightly coupled to the other tripleo deliverables12:57
shardyand those (other than os-cloud-config) are consumed outside of tripleo12:58
shardyso perhaps the intermediary model makes sense for those?12:58
derekhshardy: any ideas? purge_deleted doesn't seem to be deleting anything for the large tables...12:58
derekhalso the table size thing mightn't be the actual problem just a hunch12:58
dtantsurshardy, EmilienM, I kind of thought os-net-config can be consumed outside of tripleo too..12:59
shardydtantsur: Yeah, it can, although I'm not sure if anyone does12:59
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Customize undercloud and overcloud with virt-customize  https://review.openstack.org/36111213:00
jaosoriordtantsur: do you know if ironic is running over apache in the undercloud?13:02
dtantsurjaosorior, no, we had problems with it and had to revert13:02
dtantsurthis is a goal for Ocata, I guess13:02
openstackgerritMartin André proposed openstack/tripleo-heat-templates: Add steps to containerized compute deployment  https://review.openstack.org/34692713:03
openstackgerritMartin André proposed openstack/tripleo-heat-templates: WIP: Containerized Services for Composable Roles  https://review.openstack.org/33065913:03
openstackgerritMartin André proposed openstack/tripleo-heat-templates: Bind mount files to run DiD in latest atomic host  https://review.openstack.org/34721813:03
openstackgerritMartin André proposed openstack/tripleo-heat-templates: Be more inclusive in insecure registry regex  https://review.openstack.org/36611113:03
*** bkopilov has joined #tripleo13:03
*** trozet has quit IRC13:04
*** zoli|brb is now known as zoliXXL13:05
*** cdearborn has joined #tripleo13:05
*** Goneri has joined #tripleo13:07
derekhsshnaidm: or maybe there is a filehandle leak somewhere, this seems a tad high13:07
derekh[root@overcloud-controller-0 ~]# wc lsof13:07
derekh  1825664  19669833 316782766 lsof13:07
shadowermandre: so this doesn't pass OS_PASSWORD in which breaks the discovery diff stuff when running from mistral: https://github.com/openstack/tripleo-common/blob/6c39473ce5f647f717c32662e5dc0edd178295f7/tripleo_common/utils/validations.py#L8213:09
*** lblanchard has joined #tripleo13:09
shadowermandre: any ideas how to fix that? The validations should be able to get the auth information to the undercloud somehow13:09
dtantsurshadower, I suspect you can use OS_AUTH_TOKEN to create a session13:10
mandreshadower: you should be able to pass the token the openstack command you use13:10
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-quickstart: Make undercloud service workers configurable  https://review.openstack.org/36611613:10
shadowerdtantsur, mandre: I tried that but it didn't seem to work. Maybe I just used it wrong13:10
dtantsurshadower, please paste your code13:10
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Customize undercloud and overcloud with virt-customize  https://review.openstack.org/36111213:11
openstackgerritWaldemar Znoinski proposed openstack/diskimage-builder: fix systemd resource deadlock  https://review.openstack.org/36579713:11
dtantsurshadower, I think you need this auth plugin: http://docs.openstack.org/developer/keystoneauth/api/keystoneauth1.identity.html#keystoneauth1.identity.v2.Token13:11
dtantsurshadower, or rather http://docs.openstack.org/developer/keystoneauth/api/keystoneauth1.identity.generic.html#keystoneauth1.identity.generic.Token13:12
dtantsurto support both V2 and V313:12
jaosorioryay for generic plugins :D13:12
*** pgadiya has quit IRC13:13
sshnaidmderekh, I don't something about this in journal..13:13
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-quickstart: DO NOT MERGE: Testing workers  https://review.openstack.org/36611913:13
sshnaidmderekh, but found another issue http://paste.openstack.org/show/567154/13:13
shadowerdtantsur: so this is using the commandline client. And afaics python-openstackclient doesn't understand the OS_AUTH_TOKEN env variable13:14
dtantsurshadower, I'm pretty sure I made it work with a token once, lemme try13:14
shadowerdtantsur: thanks. I'll have a look at the plugin stuff in the meantime13:14
sshnaidmderekh, I'm concerned why we can delete any stack there, it seems like vm is still up when heat tries to delete the subnet (with ports)13:15
mandreshadower: openstack command accepts a --os-token CLI option13:17
derekhsshnaidm: ya, there could be a couple of errors, its as if the delete_ports commands arn't working here http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/te-broker/destroy-env and then the stack can't be deleted13:17
*** akshai has joined #tripleo13:18
derekhsshnaidm: ok, heres what I think we can try, restart haproxy, mysql and heat-engine13:18
derekhsshnaidm: then delete all the old resources that had failed to delete or were left behine13:18
derekhsshnaidm: and see what kind of errors we are left with13:18
sshnaidmderekh, ok13:18
mandreshadower: dunno if that helps13:18
*** trozet has joined #tripleo13:19
derekhsshnaidm: ok, you wanna do it or will I?13:19
*** jpena is now known as jpena|off13:19
openstackgerritMerged openstack/tripleo-heat-templates: Added environment file for SR-IOV deployment  https://review.openstack.org/36361713:19
sshnaidmderekh, I'll handle this13:19
shadowermandre: yeah but it seems to be a bit different. It expects the service URL directly (instead of the generic keystone one) and when I did that, I was getting 404 from it13:20
openstackgerritMerged openstack/puppet-tripleo: Make fqdn fact names match service net map  https://review.openstack.org/36558313:20
mandreshadower: i believe you need to use it together with --os-auth-type13:20
shadowermandre: trying that now but unsure which value to pass in :-)13:21
derekhsshnaidm: ok, there might be a lot of old resources to delete so if you want to split it up give me a shout13:21
bandinimatbu: I am seeing this during M->N upgrades (after the undercloud upgrade, during the makor-upgrade-pacemaker-init step): https://paste.fedoraproject.org/422872/73168050/ . Ever seen the same?13:22
dtantsurshadower, mandre, token auth seems broken in OSC 3.2.0... bothering #openstack-sdks now13:22
matbubandini: /me looks13:23
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Use Redis VIP when deploying with keepalived  https://review.openstack.org/36491713:23
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Set Redis VIP on all nodes  https://review.openstack.org/36612813:23
dtantsurshadower, this is what should work according to their docs: openstack --os-token $token --os-auth-url server list13:23
matbubandini: hm weird, i already seen that last week i think, on a CI job, but i didn't reproduce it13:23
bandinimatbu: It is the second time I hit it. Will try to debug more and open a bug then13:24
matbubandini: which newton delorean hash are you using ?13:24
*** tremble has quit IRC13:24
shadowerdtantsur: thanks. How do you get the token btw? Sorry if that looks like a silly question but it gets passed through mistral for me13:25
shadowerwould love to test it manually though13:25
mandreshadower: 'openstack token issue' should do the trick13:26
dtantsuryep, more specifically $ token=$(openstack token issue -f value -c id)13:26
bandinimatbu: https://paste.fedoraproject.org/422875/47316842/13:27
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move role deployment steps into overcloud.yaml  https://review.openstack.org/36576313:27
shardydprince: Hey, I'd appreciate your feedback on ^^ when you get a moment13:27
dprinceshardy: looking13:28
mandrethanks dtantsur, I really like how osc can easily integrate in shell scripts13:28
shardydprince: note that the steps will be moved inside the jinja2 loop (later in that series, still testing it)13:28
*** tremble has joined #tripleo13:28
dtantsurmandre, ++ very cool13:28
shardydprince: also I'm considering optimizing such that role resources aren't generated at all when $roleCount is zero13:28
matbubandini: weird, the current-passed-ci is what is used for : https://ci.centos.org/view/rdo/view/tripleo-periodic/job/tripleo-quickstart-upgrade-major-mitaka-to-master/13:28
shardydprince: but the first step is moving all the Post stuff into overcloud.yaml13:28
dprinceshardy: rather than moving it all into overcloud.yaml13:30
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485913:30
dprinceshardy: could we create a puppet/post.yaml?13:30
dprinceshardy: that is what we are aiming for w/ docker anyways...13:30
shardydprince: I suppose we could, but then I'll have to j2 template two files13:30
shardyI guess that would be OK13:30
bandinimatbu: very odd13:30
dprinceshardy: I think it is actually good that it would be 213:31
bandinimatbu: is there a way for me to trigger that periodic job on a specific patch?13:31
shardydprince: do you have a patch showing how the post.yaml stuff is expected to work?13:31
jristmornin' all13:32
shardydprince: I'm fine with moving it, but we'll need to modify the mistral action to find all j2.yaml files, not just the overcloud one13:32
shardyshouldn't be too hard13:32
dprinceshardy: sharing across baremetal vs. containers is going difficult if not impossible13:33
dprinceshardy: 2 templates I think is a good design actually13:33
dprinceshardy: trying to share things too much is sometimes bad, as is sharing to little13:33
shardydprince: Ok, so we won't be doing the same step approach, or it's just going to be implemented very differently?13:33
shardyI assumed we'd just plug in different *Config resources, which is why I moved them all into nested stacks13:33
dprinceshardy: we'll have steps. but they will mean slightly different things13:34
shardydprince: Ok, cool, I'll try reworking into puppet/post.yaml and see how it looks, thanks for the feedback!13:34
shardy(this patch works, so hopefully will be cut/paste)13:34
*** hjensas has quit IRC13:35
*** jpena|off is now known as jpena13:36
dprinceshardy: there may be a case for keeping some things at the top level too13:38
dprinceshardy: in that we could keep cluster-wide deployment in lockstep across the roles without breakpoints13:38
dprinceshardy: even still, we would need to consider templating mechanism to have different steps for puppet, vs alternatives like containers13:39
shardydprince: perhaps we should land it all at the top-level, then refactor when we know exactly what e.g container integration will look like?13:39
shardydprince: I've tried to keep puppet-isms out of overcloud.yaml, but if you look at the end of that series, it's getting pretty lightweight when j2 templates it all13:40
pradkEmilienM, ok, well logs indicate you still have redis issue, which perhaps means the redis vip patch with keepalived isnt part of your rebase ?13:41
EmilienMpradk: oh right13:41
EmilienMpradk: I also need it13:41
shardydprince: interesting, but we'd still need to template that for all roles, so the steps between roles can be aligned13:41
openstackgerritMartin André proposed openstack/tripleo-heat-templates: Add steps to containerized compute deployment  https://review.openstack.org/34692713:41
openstackgerritMartin André proposed openstack/tripleo-heat-templates: WIP: Containerized Services for Composable Roles  https://review.openstack.org/33065913:41
openstackgerritMartin André proposed openstack/tripleo-heat-templates: Bind mount files to run DiD in latest atomic host  https://review.openstack.org/34721813:41
openstackgerritMartin André proposed openstack/tripleo-heat-templates: Have docker start script honor configuration  https://review.openstack.org/36613813:41
shardydprince: If you're OK with that, I'll move all the post stuff into a puppet/post.yaml13:42
EmilienMpradk: wait, it has it in the serie13:42
dprinceshardy: I like puppet/post.yaml the best for now I think13:42
*** saneax is now known as saneax-_-|AFK13:42
shardythen j2 template it like I did in https://review.openstack.org/#/c/365796/13:42
openstackgerritMerged openstack/puppet-tripleo: Map ctlplane network to management fqdn  https://review.openstack.org/36558813:42
openstackgerritMerged openstack/tripleo-common: Change default deploy artifacts container name  https://review.openstack.org/36578313:42
shardydprince: ack, Ok I'll go with that then, thanks!13:42
dprinceshardy: it doesn't solve the lockstep/no-breakpoints feature thing I mentioned but that is... a new feature. We can do it later13:43
shardydprince: ack, yeah - pretty pushed for time with this already so I'd rather tackle that in ocata13:43
*** pblaho has quit IRC13:43
*** pblaho has joined #tripleo13:44
*** yamahata has quit IRC13:44
*** pblaho has quit IRC13:44
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Fix aodh auth url to remove suffix  https://review.openstack.org/36511713:44
*** tobias_fiberdata has quit IRC13:44
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Fix aodh auth url to remove suffix  https://review.openstack.org/36511713:45
EmilienMpradk: ok now it is :)13:45
*** pblaho has joined #tripleo13:45
jaosoriorEmilienM: did you change anything to get scenario001 to work?13:46
*** xuao has joined #tripleo13:48
EmilienMjaosorior: where does it work?13:51
*** radeks has joined #tripleo13:54
EmilienMtripleo meeting in 5 minutes13:54
*** tobias_fiberdata has joined #tripleo13:56
ansiwenmwhahaha: hi, thanks for reviewing my provider13:59
dprinced0ugal: nice on finally landing this https://review.openstack.org/#/c/342261/14:00
ansiwenmwhahaha: I have no idea if V2_0 or V3 has to be used, so I will use V3 now.14:00
dprinced0ugal: wasn't the easiest patch to land14:01
mwhahahasure, i think v3 is preferred for new stuff14:01
ansiwenmwhahaha: can you comment on the puppet type? I removed the name parameter. but now I realised I need project and user as parameters, because else everithing is run as administrator. but strictly neither is unique, so do I have to mark both as "namevar" ?14:02
mwhahahaansiwen: refresh my memory, which review is it?14:04
ansiwenmwhahaha: sure sorry14:05
ansiwenmwhahaha: https://review.openstack.org/#/c/36458014:05
mandreEmilienM: regarding your comment in https://review.openstack.org/#/c/363167/, should it be a require or a notify?14:06
mwhahahaansiwen: that's a good question, give me a few minutes and i'll get back to you14:07
EmilienMmandre: tripleo meeting atm14:07
ansiwenmwhahaha: I'm currently reading this: https://docs.puppet.com/puppet/latest/reference/lang_resources.html#namenamevar14:07
*** eggmaster has joined #tripleo14:08
d0ugaldprince: heh, thanks :) Yeah, it took a while!14:09
*** jcoufal has joined #tripleo14:09
mandreEmilienM: zut, I forgot the meeting, thanks for the reminder14:10
mwhahahaansiwen: technically i think they would both be the name var since the unique value would be user + project right?14:12
*** trozet has quit IRC14:12
mwhahahaansiwen: i think you might want to use the name as the user by default and then project as an additional option. Alternatively support a name of user@project and then you could just split them apart in code.14:12
*** trozet has joined #tripleo14:13
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Re-add undercloud.yaml  https://review.openstack.org/35203714:13
mwhahahaansiwen: there's something similar in puppet-keystone around a composite namevar, you could check with chem around it since he wrote it i think14:13
ansiwenmwhahaha: actually I'm  not even sure about that... maybe it's possible to create several ec2 credentials per user/project pair. the uniqe id would be the access id, that is created by the provider itself, I guess14:13
mwhahahaoh is it one of those auto generated ids?14:14
mwhahahadoes it support setting the id?14:14
mwhahahaI think the nova flavor provider or the glance image provider might handle a case similarly. i'll have to double check14:14
chemansiwen: mwhahaha I wouldn't go the composite namevar way, it add a layer of complexity not required for this.14:15
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for aodh  https://review.openstack.org/36545714:16
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702914:16
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add class to write overcloud VIPs into /etc/hosts  https://review.openstack.org/35776214:16
ansiwenI don't think it supports proposing an ID14:16
ansiwenchem: so, what would you do?14:16
ansiwenmwhahaha: I don't think it supports proposing an ID14:17
*** michchap has quit IRC14:17
chemansiwen: how can you check that the resource has been created ?14:17
chemansiwen: on the command line14:17
mwhahahathat's unfortunate.14:17
ansiwenchem: well, the commandline returns the credential values when you create them, and you can do "openstack ec2 credentials list"14:18
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Hook internal TLS flag to apache-based services  https://review.openstack.org/36607514:18
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776514:18
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add parameters for internal TLS  https://review.openstack.org/36594214:18
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service  https://review.openstack.org/35643014:18
chemansiwen: can you past a openstack ec2 credentials list result when your resource has been created  ?14:19
d0ugaljtomasek: Are you still cherry-picking any reviews to make the UI work? or has everything landed?14:19
ansiwenchem, mwhahaha: can't I just make both user and project a normal argument, and the title has been chosen in a unique way by the provider-user?14:19
jaosoriorEmilienM: not sure what you meant by "this is not a profile but a regular class".14:20
chemansiwen: yes, that would be the right way for this14:20
EmilienMjaosorior: can we follow the meeting maybe?14:20
EmilienMand do code review a bit later14:20
mwhahahaYou'd still need to be able to determine if it already exists14:20
jaosoriorEmilienM: I'll try... got a meeting in after the tripleo meeting14:20
mwhahahaand if you can do multiple user/project you don't have a way to determine uniqueness14:20
chemansiwen: composite namevar should be support in old puppet is not that great ...14:20
*** links has quit IRC14:21
ansiwenmwhahaha: exists is implemented already14:21
*** oshvartz has quit IRC14:22
ansiwenmwhahaha: actually, the provider also creates the tempest.conf... so it can check there...14:22
mwhahahak then yea just let the name be the regular name var and add user/project as params14:22
ansiwenmwhahaha: so the name var will be unused within the provider code... just by the compiler to identify the resource14:23
chemansiwen: correct14:23
*** xuao has quit IRC14:24
chemansiwen: mwhahaha again, not the best way, but the pragmatic way.  It's not really a driver to the ec2 command, just a driver to the entry line in tempest conf.  So let's keep this simple.14:25
*** zigo has quit IRC14:25
*** xuao has joined #tripleo14:25
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add ipv6 nic-configs  https://review.openstack.org/36447914:25
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types  https://review.openstack.org/36367414:25
*** michchap has joined #tripleo14:26
*** tobias-fiberdata has joined #tripleo14:27
*** akshai has quit IRC14:27
sshnaidmbnemec, btw, did you track all performance tuning anywhere? I mean what was done and where14:29
bnemecsshnaidm: http://etherpad.corp.redhat.com/rh1-profile-switch14:29
b00tcatI'm getting an error when deploying an overcloud with VMs as stated in the documentation - How can I further debug this? http://paste.fedoraproject.org/422923/7216214714:29
sshnaidmbnemec, cool, thanks14:30
bnemecActually, since CI is hosed right now anyway, this might be a good time to switch the controller profile to performance.14:30
bnemecThat _could_ mitigate the issues we're seeing there.14:30
sshnaidmbnemec, yeah, let's try14:30
*** tobias_fiberdata has quit IRC14:30
bnemecderekh: dprince: ^Thoughts on switching the performance profile of the controller now since we're basically in the middle of an outage anyway?14:31
sshnaidmbnemec, just wait for finishing heat deletes..14:31
dprincebnemec: the risk is something else goes wrong and increases the outage14:32
dprincebnemec: do you have a guess on how much this might improve perf on the controller?14:32
*** zigo has joined #tripleo14:32
*** zigo is now known as Guest3914714:33
bnemecdprince: The performance improvement on the computes was about 20% from my testing.14:33
bnemecAlthough that was in vms, whereas everything's running on baremetal on the controller.14:33
dprincebnemec: it is tempting, but I'd not like to increase the outage I think14:33
dprincebnemec: I'd say we wait14:34
dprincebnemec: my vote anyways14:34
derekhbnemec: how heavely loaded is the controller at peak times14:34
*** Ryjedo has joined #tripleo14:34
bnemecderekh: Very.  At times the CPU will spike to 100% for extended periods of time.14:35
*** xuao has quit IRC14:35
bnemecNot often, but I've seen it happen even after tuning the heat-engine workers.14:35
dprincederekh: yep, it gets fairly loaded and is never idle14:35
bnemecsshnaidm: Have you seen the cleanup-stack script on the controller?14:35
derekhbnemec: dprince ok14:35
sshnaidmbnemec, nope14:35
derekhsshnaidm: sounds like you want that script14:35
bnemecI've been using that to delete DELETE_FAILED stacks.14:35
sshnaidmbnemec, sounds useful, where is it?14:36
bnemecsshnaidm: It's on the controller itself in the heat-admin home directory.14:36
bnemecSince I can't get to the heat api from my local system.14:36
derekhbnemec: so the only thing the reboot would help is the testenv creation time, if that is excessive then we should reboot ASAP if it isn't then waiting wouldn't be too bad14:37
*** akshai has joined #tripleo14:37
dprincederekh: my thought was that it could be somehow related to the missing neutron ports issue14:38
bnemecderekh: Well, we'd switch the performance profile, which may help lower the CPU load too.14:38
dprincederekh: load that is, but I haven't confirmed that14:38
*** Guest39147 has quit IRC14:39
derekhbnemec: yup, agreed it may help reduce the load, but from the perspective of the CI runtimes, it will only change the time it takes to create a testenv14:39
derekhunless it also helps the neutron ports problem dprince mentioned14:40
derekh2016-09-05 16:01:05.470859 | + /opt/stack/new/tripleo-ci/toci_gate_test.sh::L233:   sleep 120014:40
derekh2016-09-05 16:10:08.137657 | 2016-09-05 16:10:08,136 - testenv-client - INFO - Received job : {14:40
bnemecderekh: Right, but it may also help with those node locked errors.  If the controller is overloaded and can't respond to the bmc requests, it could cause those timeouts.14:40
derekh9 minutes, could be a lot better14:40
bnemecIt looks like job runtimes had already dropped a lot before this latest outage.14:41
*** zigo_ has joined #tripleo14:42
derekhbnemec: yup, probably a combination of the image caching and perfomance improvements14:42
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton  https://review.openstack.org/34699514:43
*** kjw3 has joined #tripleo14:45
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton  https://review.openstack.org/34699514:45
pabelangerderekh: bnemec: we should whiteboard how it would look like having a local nodepool server manage OVB nodes for tripleo.  Specifically, now the shared network would look14:45
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Handle image location based on CI_ENV and REL_TYPE  https://review.openstack.org/35979614:46
derekhpabelanger: can nodepool manage heat stacks?14:46
d0ugalshardy: Do you think it is a bit late/disruptive to try and do this? https://review.openstack.org/#/c/357682/14:47
ayoungdprince, I see that apevec pinned master to Keyston N3 to avoid the Credentials encryption test breaking.14:47
*** zigo_ has quit IRC14:47
d0ugalshardy: Decided it probably wasn't worth holding the meeting for :)14:47
ayoungI have a proof of concept on using Fernet.14:47
EmilienMjaosorior: ok so profiles are different from regular manifests14:47
dprinceayoung: okay, good to know14:47
EmilienMjaosorior: profiles deploy services14:47
pabelangerderekh: no, it uses shade to launch the nodes14:48
EmilienMjaosorior: think containers. 1 profile could be 1 container14:48
pabelangerI keep forgetting you are using heat to do it14:48
jaosoriorEmilienM: ah, right14:48
ayoungdprince, I'm using a Heat (swift ) artifact.14:48
jaosoriorEmilienM: So that doesn't have to do with the "services" we deploy in the service chain14:48
*** zigo_ has joined #tripleo14:48
jaosoriorcause we have stuff that is not a "profile" in a sense, in that list14:48
ayoungEmilienM, do we have support for Fernet keys in the Keystone puppet module?14:49
derekhpabelanger: I'm sure it can be done without heat, but it would be nice to keep using the OVB heat templates and take advantage to any improvements14:49
jaosoriorEmilienM: such as the tripleo packages, firewall, CA-related things, and I'm trying to add the /etc/hosts additions https://review.openstack.org/#/c/357765/14:49
shardyd0ugal: I guess now is better than after we branch stable/newton14:49
openstackgerritTomas Sedovic proposed openstack/tripleo-common: Allow the validations to run openstack commands  https://review.openstack.org/36617514:49
shardyd0ugal: one question, do we have folks consuming this yet which requires deprecating the old names?14:50
EmilienMayoung: of course14:50
pabelangerderekh: well, we could start with having nodepool manage the images. Build / upload them into the cloud14:50
EmilienMayoung: we even gate on it14:50
shardytzumainn: ^^ are you going to be impacted by changing the mistral action names?14:50
EmilienMayoung: https://github.com/openstack/puppet-openstack-integration#description14:50
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Check for differences between introspected HW  https://review.openstack.org/35397814:50
d0ugalshardy: The CLI and the GUI will need updated at least - if we just remove them that'll break CI until they update.14:51
derekhpabelanger: but for that part we're only using a vanilla centos cloud image14:51
d0ugalI really wish we didn't encourage direct action calls, but that is another subject.14:51
shadowermandre, jtomasek: so https://review.openstack.org/366175 and https://review.openstack.org/353978 should fix the discovery diff issue14:51
shardyd0ugal: Ok, I was thinking it'd just be the workflows inside tripleo-common14:51
EmilienMayoung: https://github.com/openstack/puppet-keystone/blob/master/manifests/init.pp#L411-L41614:51
shardyso if we can't do it in one patch this seems OK14:51
shardyd0ugal: can we switch everything over before the release?14:51
shadowermandre, jtomasek: it does so by fixing up the mistral environment (which is different than I expected)14:51
d0ugalshardy: Nah, some actions are called directly by the CLI14:51
EmilienMjaosorior: right we have stuffs that shouldn't be profiles14:51
EmilienMjaosorior: we should move it later14:51
d0ugalshadower: I think so, the patches are trivial. I can prepare them.14:52
d0ugalshardy: ^14:52
EmilienMjaosorior: but for this new one, i think we should put it as a regular manifest14:52
shadower jtomasek: but the validation will fail until you run node introspection14:52
d0ugalshadower: (sorry)14:52
jaosoriorEmilienM: it already is14:52
shardyd0ugal: relatedly, did you have a chance to check out wiring in the action to render overcloud.yaml?14:52
jaosoriorah wait14:52
shadowerd0ugal: no worries14:52
shardyd0ugal: I was going to take a look later if not14:52
pabelangerderekh: right, but it looks like you online deploy the image once. Then reused it.  Also means you are not using diskimage-builder to build them too14:52
jaosoriorEmilienM: nevermind; yeah, did the change so now it's just manifests/vip_hosts.pp14:52
ayoungEmilienM, But how about a way to say "here is the Key to use"14:52
EmilienMayoung: how to do it? by generating the file?14:53
pabelangermoving to nodepool, we get new images every day, build with diskimage-builder.14:53
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Check for differences between introspected HW  https://review.openstack.org/35397814:53
tzumainnshardy, thanks for asking!  rwsu is currently adding ironic node registration to manageiq, and he's using the mistral workflows and actions there14:53
ayoungEmilienM, well, while I did a tarball, the right way is, I think, to have the hiera data speficy the keys in a hidden field14:53
ayoungEmilienM, http://adam.younglogic.com/2016/09/fernet-overcloud/14:53
derekhpabelanger: sorry hold on I got mixed up there, which nodes are you talking about?14:53
EmilienMayoung: you can specify which directory to use keys and say puppet to not generate the keys14:54
jaosoriorayoung: thought you didn't like injecting keys14:54
ayoungjaosorior, I don't14:54
d0ugalshardy: I started, but I've really made any progress. I've been a bit distracted today if I'm honest.14:54
EmilienMayoung: so puppet will just take care of writting the path in keystone.conf14:54
ayoungjaosorior, but I also hate passwords an we have them all over the place14:54
shardytzumainn: Ok, so it looks like we're going for a late rename of some of them prior to the newton release, is that manageable for you?14:54
openstackgerritMerged openstack/puppet-tripleo: Convert ringbuilder to build devices array  https://review.openstack.org/36473714:54
shardytzumainn: the alternative will be to leave the old names in place for a cycle I guess14:54
ayoungEmilienM, if you look inside the "repo" it looks like this14:54
pabelangerderekh: oops, I was looking at prepare-ovh-cloud.sh, let me find where OVB nodes are created14:54
shardytzumainn: https://review.openstack.org/#/c/357682/8/setup.cfg14:55
tzumainnshardy, I think the rename is manageable, we'll go along with whatever you guys release for newton!14:55
jaosoriorayoung: what happened to the custodia-based approach?14:55
jaosoriorayoung: or can't we do it with barbican? Heat has support for it14:55
shardytzumainn: Ok, obviously we'll be more cautious about this stuff after it's released :)14:55
shardyd0ugal: Ok, np, I think I know what needs to happen so I'll hack on my patch until it works14:55
ayoungEmilienM, http://paste.openstack.org/show/567189/14:55
shardyd0ugal: can you show me an example where an action is called directly please?14:55
jaosoriorayoung: http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Barbican::Secret14:56
shardycut/paste ftw ;)14:56
derekhpabelanger: for the ovb nodes the image that is used is irrelevant, the the CI undercloud boots them with ironic they get the overcloud-full image built during that CI job14:56
ayoungjaosorior, custodia is not going to be ready for a while14:56
ayoungjaosorior, barbican is not deployed on the undercloud, and I don't want to force it just for this.14:56
*** anshul has quit IRC14:56
ayoungbarbican also implies that it is a "pull" and that the controller nodes have credentials in the undercloud keystone14:56
ayoungall do-able, but more complex14:57
*** ifarkas is now known as ifarkas_afk14:57
openstackgerritMerged openstack/puppet-tripleo: Add Ceph RGW listener to HAProxy  https://review.openstack.org/36281614:57
pabelangerderekh: Right. That's the step I missed, I didn't see ironic in the picture. So, in that case, we can't do anything today14:57
EmilienMayoung: I have no idea how to implement the keystone-manage credential setup thing in tripleo14:58
jaosoriorayoung: alright. Then I guess hieradata is the way to go14:58
EmilienMwe do support it now in puppet-keystone but only config/generation14:58
EmilienMin tripleo we need to export the credential somewhere and collect it on other keystone nodes14:58
d0ugalshardy: Creating a plan requires calling actions. https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/workflows/plan_management.py#L85-L8814:58
derekhpabelanger: yup afraid so, we specifically want to test that ironic is working so I'm not sure much can be done there,14:59
d0ugalshardy: create_container is an action call, so is create_deployment_plan - in the middle we upload to swift directly.14:59
d0ugalshardy: I've been -1 to this approach, but so far I've been the only one :)14:59
derekhpabelanger: but as more jobs may move to the multinode model (away from OVB) we might be relying on it less14:59
pabelangerd0ugal: ya, we need to wait for zuulv3. That should expose baremetal provisioning14:59
pabelangerderekh: Yup, or that14:59
d0ugalshardy: the patch florianf linked will also have the GUI call the deploy action directly. We hadn't ever discussed that being called directly.14:59
ayoungEmilienM, I'll get you some guidance.  I think the right thing will be to generate all keys on the undercloud and copy out to the controllers15:00
*** pradk has quit IRC15:00
florianfd0ugal, shardy: We do call some actions directly if we want immediate output (like listing the plans).15:01
EmilienMayoung: right now I have zero time to work on it. i'm focused on getting tripleo RC1 done15:01
shardyd0ugal: Ok, well perhaps that's something to revisit in ocata15:01
d0ugalflorianf, shardy: Yeah, that is one we had discussed as a direct call. I don't think calling deploy is wise.... and this is why I don't like it, it is very vague :/15:01
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Convert SwiftDevicesAndProxyConfig to composable format  https://review.openstack.org/36474815:02
shardyI agree it's not an ideal pattern, perhaps we can refactor things to make the workflows more easily consumable15:02
ayoungEmilienM, this falls into that category.  I'll work on it, but I am in over my head15:02
tzumainnshardy, haha, I trust you guys ;)15:02
d0ugaltzumainn: wow15:02
*** rodrigods has quit IRC15:02
*** rodrigods has joined #tripleo15:02
EmilienMayoung: where does it fall in RC1?15:02
tzumainnd0ugal, I know, right?15:02
jpichflorianf: I still think it's fine for the simple actions, but 'deploy' seems a bit more prone to changes/issues if I understand the UI patch correctly15:02
EmilienMayoung: is there a launchpad bug somewhere?15:02
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Move role deployment steps into overcloud.yaml  https://review.openstack.org/36576315:03
*** jpena is now known as jpena|away15:03
openstackgerritMerged openstack/puppet-tripleo: Include heat base profile outside of step  https://review.openstack.org/35271315:04
ayoungEmilienM, this is the breakage from Friday15:05
florianfjpich, d0ugal: I think in this case it doesn't make much difference from the UI side. So switching to call the workflow sounds fine to me. jtomasek's the owner of the patch, let's see what he says.15:05
*** zoliXXL is now known as zoli|mtg15:05
pabelangerEmilienM: derekh: shardy: dprince: How does this week look to enable shared networks for OVB? https://review.openstack.org/#/c/357308/ This should be a noop change but will allow different projects to interact with the networks created for OVB15:05
d0ugalflorianf: k, thanks :)15:05
jpichflorianf: Cool!15:06
dprincepabelanger: we are in the middle of an outage. Until that is over I would like to wait15:07
dprincepabelanger: otherwise we can't tell if something breaks...15:07
sshnaidmbnemec, your script could be run in parallel, right?15:08
bnemecsshnaidm: Should be.  Even with the script I'm getting weird errors deleting stacks though. :-(15:08
sshnaidmbnemec, do you delete stack now too?15:09
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart: Add config parameter to collect-logs ci-script  https://review.openstack.org/36439415:09
bnemecsshnaidm: It's failing to delete subnets that are already deleted due to port conflicts, which makes no sense.15:09
pabelangerdprince: sure, once we are back online of course15:09
sshnaidmbnemec, and the port still exists?15:10
bnemecsshnaidm: No, the ports were deleted correctly.  It's nonsense.15:10
shardybnemec: is convergence enabled?15:12
sshnaidmbnemec, hmm.. curiouser and curiouser15:12
* shardy isn't sure if we're discussing undercloud or overcloud heat here15:12
bnemecshardy: Shouldn't be.  This is a Mitaka cloud.15:12
shardybnemec: ah15:12
bnemecRetrying stack-delete enough times did eventually make it go away.15:13
*** yamahata has joined #tripleo15:14
*** dtrainor has quit IRC15:15
*** dtrainor has joined #tripleo15:15
sshnaidmbnemec, are you going to tune there performance and shut it down?15:15
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Set Redis VIP on all nodes  https://review.openstack.org/36612815:16
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Use Redis VIP when deploying with keepalived  https://review.openstack.org/36491715:16
bnemecsshnaidm: I wasn't planning to.  dprince didn't want to add more variables to the outage, which is fair.15:16
sshnaidmbnemec, btw, I run this script for deleting stack right now, it may clash with you try if we deleted it in the same time15:16
*** aufi has quit IRC15:17
bnemecsshnaidm: I've been picking random stacks from the middle of the list, so if you start at the top we should be pretty safe. :-)15:17
sshnaidmbnemec, pity, I would restart it completely..15:17
bnemecWe still need to do that at some point.15:17
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Fix aodh auth url to remove suffix  https://review.openstack.org/36511715:17
derekhsshnaidm: bnemec also we were still seeing db connection problems up until sshnaidm stoped the te_broker, so something needs to be restarted15:18
*** thrash is now known as thrash|biab15:18
*** ebarrera has quit IRC15:18
bnemecderekh: Oh look, my last stack-delete failed on a mysql connection error.15:19
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: Work around deletion of _member_ role assignments on upgrade  https://review.openstack.org/30735215:19
*** pradk has joined #tripleo15:20
*** oshvartz has joined #tripleo15:20
*** zigo_ is now known as zigo15:20
*** fragatina has joined #tripleo15:20
derekhsshnaidm: bnemec so we've aready restarted haproxy and heat-engine so far, I'd say either mysql is next or a complete restart, after which monitor open files, tcp connections etc... and figure out whats leaking15:21
derekhdprince: ^15:21
bnemecYeah, strangely there don't appear to be a huge number of db connections open.15:21
EmilienMjistr: you take https://bugs.launchpad.net/tripleo/+bug/1620657 ?15:22
openstackLaunchpad bug 1620657 in tripleo "OpenConrail loadbalancer plugin value is incorrect" [High,New]15:22
dprincederekh: ack. FWIW if we are doing a complete restart may as well let bnemec change the CPU settings15:22
bnemec| Max_used_connections                                         | 891                                    |15:22
openstackgerritMerged openstack/tripleo-common: Add List Roles Action  https://review.openstack.org/33028315:22
bnemecThat's really not very many.15:22
derekhbnemec: it may also be the size of the heat db tables aswell, causing things to timeout,15:22
derekhbnemec: ya15:22
derekhdprince: yup, makes sense15:22
dprincederekh: could we just delete/recreate the heat db tables?15:23
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton  https://review.openstack.org/34699515:23
bnemec5.4G Sep  6 14:11 raw_template.ibd15:23
derekhdprince: maybe, its crazy enough that it just might work ;-)15:23
derekhbnemec: http://paste.openstack.org/show/567125/15:24
derekhbnemec: we ran heat-manage purge on it earlier but it did nothing15:24
sshnaidmbnemec, we tried heat-manage purge , but it just didn't work15:24
shardyWe do have a new enough heat to have https://bugs.launchpad.net/heat/+bug/1524387 fixed, right?15:26
openstackLaunchpad bug 1524387 in Ubuntu Cloud Archive liberty "heat-manage purge_deleted broken" [Undecided,In progress] - Assigned to Seyeong Kim (xtrusia)15:26
shardyThat was fixed during mitaka, so I assume so15:27
*** fragatina has quit IRC15:27
bnemecOuch, lots of db connection errors in the heat logs.15:27
shardytherve: ^^ Hey are you aware of any more recent heat-manage purge_deleted bugs that may not have made it onto mitaka yet?15:27
shardyI recall we've had a few issues but am failing to find all-the-bugs atm15:28
derekhshardy: looks like we have that fix15:28
sshnaidmwe have openstack-heat-engine-6.0.1-0.20160722173744.656ac73.el7.centos.noarch15:29
bnemecmongo is eating 20 GB of memory on the controller15:29
derekhbnemec: ya, I was gonna bring that up after this15:29
shardybnemec: do we even need it?15:29
bnemecWe shouldn't.15:30
EmilienMstop ceilo and mongo15:30
EmilienMwe don't need it15:30
bnemecWe aren't using ceilometer.15:30
shardyor zaqar15:30
bnemecI bounced mongo, which freed up the memory.15:31
dprinceI stopped ceilo last week15:31
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move role deployment steps into overcloud.yaml  https://review.openstack.org/36576315:31
shardydprince: ^^ I think that reworks things into /post.yaml like you wanted15:31
dprinceshardy: looking15:32
openstackgerritMerged openstack/tripleo-common: Fix pep8 errors when running with python3  https://review.openstack.org/36572415:33
openstackgerritMerged openstack/tripleo-heat-templates: Cleanup haproxy service from explicit services enablement  https://review.openstack.org/36280015:33
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure  https://review.openstack.org/36391815:34
dprinceshardy: one quick comment on passing just 'step_config' into post.yaml15:34
derekhOk, we gotta make a decision here ;-),15:34
derekh1. restart mysql15:34
derekh2. restart controller15:34
derekh3. delete and recreate the heat database15:34
derekhthat the order I reckon we should try thing in, thoughts?15:34
derekhbnemec: sshnaidm dprince shardy ^15:35
dprinceshardy: I think it would be better to just pass role_data in...15:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Hook internal TLS flag to apache-based services  https://review.openstack.org/36607515:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776515:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add VIP names to allNodesConfig  https://review.openstack.org/36589515:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add parameters for internal TLS  https://review.openstack.org/36594215:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service  https://review.openstack.org/35643015:35
shardydprince: ack, yup true15:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add bootstrap_node and vip_data to hierarchy for all roles  https://review.openstack.org/36604915:35
* shardy seds things15:35
sshnaidmderekh, +115:35
bnemecderekh: Sounds reasonable15:35
dprincederekh: if we can easily control the resources I'd go for bouncing 1 and 3 first15:36
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485915:36
dprincederekh: if that doesn't work then follow up with #2 and let bnemec make his change15:36
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: DO NOT MERGE - TESTING BARBICAN  https://review.openstack.org/35326215:36
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add Barbican to the overcloud - WIP  https://review.openstack.org/35245815:36
dprincederekh: to the BIOS15:36
derekhdprince: easily contol the resources?15:36
dprincederekh: sorry, like can we make sure any reasources created by those heat stacks get cleaned up elsewhere?15:37
dprincederekh: nova and neutron stufff that gets created...15:37
derekhI should be able to do "systemctl restart mariadb" right ? there is no galera magic going on is there?15:37
derekhdprince: ahh, ok, ya we should be able to do that I think15:37
dprincederekh: correct, without pacemaker systemd is fine15:38
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Update OpenContrail loadbalancer plugin value  https://review.openstack.org/36621515:38
*** matbu is now known as matbu|bbl15:38
EmilienMjistr: I took it ^15:39
jpichI'd like to help self-triage the bugs I open at least, is there a process to get the right Launchpad permissions?15:39
bnemecjpich: I think shardy needs to add you to the tripleo drivers team.15:41
EmilienMjpich: let me add you, a sec15:41
shardyjpich: I'll add you, sec15:41
jpichbnemec, EmilienM, shardy: Thanks!15:41
shardyjpich: done15:41
EmilienMhe was faster :(15:42
bnemecderekh: dprince: I need to leave shortly for an appt.  If we do delete the heat dbs, someone please grab a list of the stacks beforehand so we know what we need to clean up.15:42
*** pcaruana has joined #tripleo15:42
bnemecThat will at least make our lives easier.15:42
derekhbnemec: will do15:42
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart: Add config parameter to collect-logs ci-script  https://review.openstack.org/36439415:43
derekhits still restarting15:44
*** thrash|biab is now known as thrash15:44
EmilienMshardy: can you define "tripleo ongoing" milestone?15:44
shardyEmilienM: it's for stuff unrelated to the release deliverables, like constantly fixing CI15:44
dprincederekh: I stopped openstack-heat-api-cloudwatch, and all the ceilo* processes last week15:45
derekhmysql taking its time restarting, I guess its flushing data to disk, gonna wait it out15:46
dprincederekh: ack, we could stop nova-api to make it easier on it perhaps15:47
*** bvandenh has quit IRC15:47
dprincederekh: and neutron-server. Those two APIs get queried a lot15:47
*** leanderthal is now known as leanderthal|afk15:48
derekhdprince: done and done15:48
* dprince likes watching the haproxy broadcast messages as derekh restarts things15:48
dprincederekh: I think mariadb is restarted?15:50
dprincederekh: systemd indicates active since 2 minutes ago or so...15:50
derekhdprince: yup, we're back, the other two services are now running again too15:50
derekhsshnaidm: now lets see if we can delete things again15:50
* derekh restarted heat-engine also for the laugh15:51
*** bana_k has joined #tripleo15:52
dprincederekh: going to clear out the heat stacks too? Or wait?15:52
dprincederekh: I mean the heat database?15:53
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart: Add config parameter to collect-logs ci-script  https://review.openstack.org/36439415:53
derekhsshnaidm: dprince I'm wondering if the db problems cause resources to get created that heat doesn't have record of, so I'm going to first try and manually delete everything outside of heat15:55
derekhthe delete the stacks which should have nothing left in them15:55
dprincederekh: ack15:56
sshnaidmderekh, it's what bnemec's script does actually15:56
sshnaidmderekh, but if there is faster way to delete everything..15:56
pabelangerderekh: shardy: EmilienM: something wrong with triple-test-cloud-rh1? We are getting exceptions when nodepool is deleting nodes15:57
sshnaidmpabelanger, db was restarted15:58
pabelangersshnaidm: what was the issue?15:58
sshnaidmpabelanger, ovb is not functional now, there are a lot of them actually..15:59
sshnaidmpabelanger, if shortly - environments are failed to create, heat mostly fails to delete and create stacks16:00
pabelangersshnaidm: so you believe there is issues with heat running on tripleo-test-cloud-rh1?16:01
shardyderekh: if heat makes the call to  create something then can't update the resource record with the ID, we could potentially leak resources16:02
sshnaidmpabelanger, yes, not sure it's all fault of heat, but the issues we see firstly on its actions16:02
derekhshardy: ok, I'm deleting eveything now and bypassing heat to see how that goes16:03
dprincederekh, sshnaidm, bnemec: before and after restart memory usuages here: http://paste.openstack.org/show/567202/16:03
derekhdprince: sshnaidm bnemec FYI what I'm running, very slow ---> neutron port-list | grep baremetal- | awk '{print $2}' | xargs -t -n 1 neutron port-delete16:04
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move role deployment steps into puppet/post.yaml  https://review.openstack.org/36576316:07
shardydprince: ^^ addressed your comments I think16:07
shardywfm locally too16:07
*** mcornea has quit IRC16:07
*** abregman has quit IRC16:08
*** dsariel has quit IRC16:13
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: keystone: configure credentials  https://review.openstack.org/36624016:13
derekhsshnaidm: dprince bnemec also running this ->> nova list | grep  baremetal- | awk '{print $4}' | xargs -n 1 -t nova delete16:13
derekhthese are going to take ages16:14
derekhback in a bit16:14
*** derekh is now known as derekh_afk16:14
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Handle image location based on CI_ENV and REL_TYPE  https://review.openstack.org/35979616:15
jistrEmilienM: ok thanks :))16:16
jtomasekd0ugal: I dont' cherry pick anything at the moment, although there is a bunch of patches up that fixes various stuff16:16
jistrre the lbaas value BZ16:16
jtomasekd0ugal, florianf: re deploy action, we can call whatever is prefered. In my patch I called deploy action directly because calling workflow did not bring any additional value16:17
*** nyechiel_ has joined #tripleo16:17
jtomasekOriginally I thought deploy workflow would run until the deployment finishes16:17
sshnaidmderekh_afk, if no ports are left, we can try stack deletes16:17
jtomasekshardy: sorry for missing the meeting:/16:18
d0ugaljtomasek: Yeah, I think the deploy will do that eventually - it can't now because of token timeouts.16:20
d0ugaljtomasek: I'd guess the deploy workflow is a big candidate for adding extra checks, steps etc. so please do use the workflow16:21
jtomasekd0ugal: yes, makes total sense16:21
*** rhallisey has quit IRC16:21
jtomasekd0ugal: I'll update my GUI patch to run workflow16:22
d0ugaljtomasek: Thanks!16:22
*** rhallisey has joined #tripleo16:22
jtomasekjrist: do you need help with GUI status?16:23
jristjtomasek: a short summary would be great. what patches you think we need16:23
jtomasekd0ugal, shardy: I think I can put together a list of non GUI patches that I see as an absolute requirement for RC1 GUI16:23
jtomasekjrist: ok, I can do that16:23
*** yamahata has quit IRC16:23
*** zoli|mtg is now known as zoli|gone16:23
d0ugaljtomasek: I think we also need a list of GUI patches that are needed.16:24
*** zoli|gone is now known as zoli_gone-proxy16:24
jtomasekd0ugal: yep, I'll put it together and send to ML16:25
d0ugalI'll be sending the Mistral status tomorrow morning16:25
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Handle image location based on CI_ENV and REL_TYPE  https://review.openstack.org/35979616:25
jtomasekd0ugal: thanks for that 'output' mistral patch!16:25
*** dbecker has joined #tripleo16:25
jtomasekd0ugal: I am so glad it went in... now we need to make sure it is included in tripleo RC1 mistral16:26
d0ugaljtomasek: no worries, I'm glad it was welcome upstream. I thought there would be come debating.16:26
*** akshai has quit IRC16:26
*** dtantsur is now known as dtantsur|afk16:28
*** akshai has joined #tripleo16:29
*** jlinkes has quit IRC16:31
derekh_afksshnaidm: yup, about half way through deleting all the ports,16:34
derekh_afksshnaidm:  all but one of the instances are gone16:34
derekh_afksshnaidm: then I'll delete all the subnets and networks16:35
derekh_afksshnaidm: and finially we can delete the stacks16:35
derekh_afksshnaidm: at least that how I hope it will go....16:35
*** derekh_afk is now known as derekh16:35
*** nyechiel_ has quit IRC16:38
*** fragatina has joined #tripleo16:42
*** fragatina has quit IRC16:43
*** lucasagomes is now known as lucas-dinner16:44
*** fragatina has joined #tripleo16:44
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Override allNodes redis_vip with correct RedisVirtualIP value  https://review.openstack.org/36579116:47
*** weshay is now known as weshay_bbiab16:47
*** pradk has quit IRC16:47
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Remove *ExtraConfig parameters from overcloud.yaml  https://review.openstack.org/36579216:48
*** fragatina has quit IRC16:52
*** jcoufal_ has joined #tripleo16:58
*** hogepodge has quit IRC16:58
*** jpich has quit IRC16:59
*** trown is now known as trown|lunch17:00
*** anshul has joined #tripleo17:00
*** jcoufal has quit IRC17:01
*** fzdarsky is now known as fzdarsky|afk17:01
*** hogepodge has joined #tripleo17:03
*** bana_k has quit IRC17:03
*** florianf has quit IRC17:04
*** ohamada has quit IRC17:05
shardyhttps://review.openstack.org/#/c/363188 needs a +A if anyone has a moment17:05
*** yamahata has joined #tripleo17:06
*** florianf has joined #tripleo17:10
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements  https://review.openstack.org/36150117:12
*** pradk has joined #tripleo17:16
*** lblanchard has quit IRC17:18
*** bana_k has joined #tripleo17:20
therveshardy, Mitaka branch, no. Mitaka release, very possible17:21
shardytherve: Ok, I think we've gone with the delete-the-world approach now, but if there's any specific patches you can identify we could check and perhaps confirm it's not an unfixed problem17:22
*** pkovar has quit IRC17:24
therveshardy, Nevermind, I was thinking of liberty17:25
therveMitaka ought to be good17:25
*** lblanchard has joined #tripleo17:27
shardytherve: Ok, thanks, something of a mystery then, unfortunately we didn't have a chance to properly debug it17:28
*** akshai has quit IRC17:30
*** fragatina has joined #tripleo17:32
*** fragatina has quit IRC17:32
*** fragatina has joined #tripleo17:33
shardyIs there a fix for the "Duplicate entry for Environment: ['name']]" error on plan creation?17:34
shardyd0ugal: ^^ perhaps you know if you're still around17:34
*** paramite has quit IRC17:36
*** jaosorior has quit IRC17:36
*** florianf has quit IRC17:40
openstackgerritMerged openstack/tripleo-common: Better namespace the list of TripleO actions  https://review.openstack.org/35768217:43
*** chem has quit IRC17:49
*** chem has joined #tripleo17:49
*** dbecker has quit IRC17:51
openstackgerritSteven Hardy proposed openstack/tripleo-common: Wire in jinja templating for custom roles  https://review.openstack.org/36246517:53
*** anshul has quit IRC17:53
*** fragatina has quit IRC17:54
*** florianf has joined #tripleo17:54
*** fragatina has joined #tripleo17:55
*** akshai has joined #tripleo17:57
*** crinkle_ is now known as crinkle18:07
*** athomas has quit IRC18:07
bnemecshardy: therve: I don't think we wiped the heat db, so it's still going to need cleaning.18:07
bnemecAnd even if we didn't, we'd still need to fix the problem because we can't have the whole cloud blowing up once a month. :-)18:07
bnemec*even if we did18:08
shardybnemec: sure, we just haven't done enough debugging yet to identify the problem18:08
shardyI was just hoping we'd be missing some recent fix, sounds like we're not18:08
*** trown|lunch is now known as trown18:12
openstackgerritMerged openstack/tripleo-heat-templates: Add missing reference to ExternalPort for DVR  https://review.openstack.org/36318818:13
derekhbnemec: still deleting resources btw18:14
derekhbnemec: I guess you are too cause I've noticed my loops are trying to delete things that are already delete18:14
bnemecderekh: Yeah, I still see a bunch of delete failures in the heat logs though. :-(18:15
bnemecderekh: No, I'm actually not.  Maybe sshnaidm is?18:15
sshnaidmderekh, bnemec yeah, I'm trying do delete stacks too18:15
derekhbnemec: I havn't even gone near heat yet, I've deleted all ports am nearly finished deleting subnets and nets18:15
derekhbnemec: sshnaidm and I've delete all nova instances18:15
sshnaidmderekh, yeah, I saw, only ports and subnets are left18:16
sshnaidmderekh, actually subnets and networks, I see ports are already done18:16
derekhsshnaidm: yup, ports are all deleted, subnets nearly finished18:17
*** pradk has quit IRC18:18
sshnaidmabout 40 stacks remain18:18
openstackgerritSteven Hardy proposed openstack/python-tripleoclient: Allow and render overcloud.j2.yaml template  https://review.openstack.org/36573518:21
*** fragatin_ has joined #tripleo18:22
sshnaidmmaybe it's better to have rh2 cloud working, even with this ntp issue18:23
sshnaidmit's just 10 minutes of job, not critical at all, and it could support and back up CI18:23
*** fragatina has quit IRC18:25
*** shardy has quit IRC18:27
*** rhallisey has quit IRC18:28
sshnaidmI still see in logs "One or more ports have an IP allocation from this subnet", although no ports at all, wtf18:29
sshnaidmmaybe it's neutron which database we should purge18:30
*** rhallisey has joined #tripleo18:30
openstackgerritMerged openstack/tripleo-heat-templates: Add constraint to prohibit balance-tcp from BondInterfaceOvsOptions  https://review.openstack.org/35507318:31
openstackgerritMerged openstack/tripleo-heat-templates: Update OpenContrail loadbalancer plugin value  https://review.openstack.org/36621518:31
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: Generate Keystone credentials for overcloud  https://review.openstack.org/36628718:35
*** pradk has joined #tripleo18:35
bnemecsshnaidm: Actually, I think the rh2 ntp problem is fixed.18:38
bnemecWe need to make sure the flavors in rh2 are all updated since we changed some of them.18:39
bnemecBut other than that I think we could turn it back on.18:39
sshnaidmbnemec, they are not afaik18:39
bnemecI have a patch up to start using it again: https://review.openstack.org/35316318:39
sshnaidmbnemec, should I delete the old flavor and create a new one, right?18:41
bnemecsshnaidm: I just updated both of them.  They should be right now.18:41
bnemecsshnaidm: But yes, that's what you would do.  Or you use Horizon like I did and it does it for you. :-)18:42
bnemecI'm going to drop WIP on the re-enable patch.18:42
sshnaidmbnemec, undercloud should be with 4 cpus18:43
bnemecsshnaidm: Whoops, fixed.18:43
sshnaidmbnemec, great, so maybe we'll enable rh2 when we are fixing rh118:44
bnemecsshnaidm: Yeah, I dropped WIP on the infra patch to re-enable it.18:45
bnemecpabelanger: ^It would be good to get rh2 running again while we address the issues in rh1.18:46
pabelangerbnemec: I wanted to finish the slip between ovb / nodepool. Clean up any existing issues before moving it back into production.18:49
*** weshay_bbiab is now known as weshay18:51
*** mwhahaha has quit IRC18:53
*** mwhahaha has joined #tripleo18:54
sshnaidmbnemec, pabelanger maybe let's continue here19:00
sshnaidm<pabelanger> sshnaidm: Which is causing warnings to appear in nodepool.o.o.  If you create a new project, openstackzuul, which shared network access to the openstack-nodepool project, we'd update nodepool.o.o to use that19:00
sshnaidmderekh, bnemec, can we do this now? I don't know about possible problems it may appear19:01
pabelangerI also updated the review19:01
*** tzumainn has quit IRC19:03
sshnaidm2016-09-06 19:01:41.909 5804 ERROR heat.engine.stack [-] Unexpected exception in delete19:04
sshnaidmvery informative19:04
*** jprovazn has quit IRC19:05
sshnaidmderekh, stackw are done, ooff19:05
sshnaidmbnemec, what have we decided about restart of server? are you gonna tune it?19:05
derekhsshnaidm: ya, I'm creating a env now to see how it goes19:07
bnemecsshnaidm: I think we had decided not to unless it's necessary.19:07
sshnaidmderekh, restarting services there19:07
bnemecIf we do end up restarting then yes, I'll change the tuning on the system.19:07
derekhsshnaidm: restarting what?19:07
bnemecIt should be a safe operation.  If anything, I expect it to help withour current issues.19:07
bnemec*with our19:08
sshnaidmderekh, sorry, nova , neutron and heat19:08
sshnaidmderekh, done19:08
*** fzdarsky|afk is now known as fzdarsky19:08
*** tzumainn has joined #tripleo19:09
*** akshai has quit IRC19:09
derekhsshnaidm: ok, gonna create a couple of test envs now to see how they go19:10
sshnaidmderekh, at least delete worked :)19:10
derekhsshnaidm: bnemec btw this is how I'm testing it on the te-broker19:11
derekh[root@te-broker openstack-virtual-baremetal]# /opt/stack/tripleo-ci/scripts/te-broker/create-env derekh1 5 ce28ba92-5060-4301-9727-4d99933b868719:11
derekhce28ba92-5060-4301-9727-4d99933b8687 is the uuid of a test instance I created19:11
*** pcaruana has quit IRC19:11
sshnaidmI see it does good19:12
*** antipsychiatry has joined #tripleo19:17
*** fragatin_ has quit IRC19:17
*** fragatina has joined #tripleo19:18
beagleshey guys, is there a good diagram that illustrates how network isolation is hooked up?19:18
derekhsshnaidm: | baremetal_derekh1 | 5ccdc577-31cf-40fe-ae15-cc1dcc016d67 | Stack CREATE completed successfully | CREATE_COMPLETE    | 2016-09-06T19:15:15 |19:18
derekhsshnaidm: deleting it again, then I think  we can try re-enabling the te-broker19:19
sshnaidmderekh, so..let's try broker?19:19
sshnaidmderekh, ok19:19
derekhsshnaidm: fire ahead19:19
*** pradk- has joined #tripleo19:20
sshnaidmderekh, started19:20
*** pradk has quit IRC19:21
*** pradk- is now known as pradk19:22
sshnaidmderekh, running a job, let's see..19:24
*** tesseract- has quit IRC19:25
*** akshai has joined #tripleo19:25
*** tzumainn has quit IRC19:27
derekhsshnaidm: actually we should stop it for a minute again and restart geard very quickly19:27
derekhwill explain why in a second19:28
sshnaidmok, sure19:28
* bnemec predicts a lot of pain spinning up 50 testenvs at once19:28
bnemecAlthough I guess the queue is empty, so maybe not.19:29
derekhsshnaidm: bassically geard has a bunch of jobs queued for ci tests that were killed long ago19:29
derekhsshnaidm: restarting it will clear that queue19:29
sshnaidmderekh, aa, that's why it was creating so many envs19:30
*** mcornea has joined #tripleo19:30
derekhsshnaidm: yup19:30
derekhsshnaidm: it would have went all the way up to 8019:30
derekhsshnaidm: I'm deleting the ones it created and we can start again19:30
sshnaidmderekh, ok19:30
derekhbnemec: ya, the zuul queue is pretty much empty so once we restart there shouldn't be a big rush19:31
*** coolsvap_ has quit IRC19:32
*** mbound has joined #tripleo19:32
sshnaidmderekh, do you delete them with "heat stack-delete"?19:34
derekh/opt/stack/tripleo-ci/scripts/te-broker/destroy-env <envnum>19:34
derekhsshnaidm: I started using that script but am also doing heat stack deletes to speed things up19:35
sshnaidmderekh, can you use "destroy-env"?19:36
derekhsshnaidm: yup19:38
derekh/opt/st.... test sshnaidm do you see this ?19:38
*** rcernin has joined #tripleo19:41
* derekh notes people don't see his message when they start with a "/"19:41
derekh<derekh> /opt/stack/tripleo-ci/scripts/te-broker/destroy-env <envnum>19:42
derekh<derekh> sshnaidm: I started using that script but am also doing heat stack deletes to speed things up19:42
derekh<derekh> /opt/st.... test sshnaidm do you see this ?19:42
openstackgerritMerged openstack/tripleo-quickstart: Handle image location based on CI_ENV and REL_TYPE  https://review.openstack.org/35979619:43
*** rbowen is now known as DrBacchus19:44
bnemecActually I did see the ones that started with /19:45
derekhbnemec: ok, looks like I jumped to the wrong conclusion19:47
*** weshay is now known as weshay_mtg19:47
bnemecI do run into that occasionally though.19:47
derekhso many ports take ages to delete :-(19:47
bnemecParticularly when I try to /rant. :-)19:48
*** rcernin has quit IRC19:48
bnemecYeah, and the apis in this cloud are just _slow_.19:49
*** pradk has quit IRC19:49
bnemecI'll be very curious to see if that's still the case once we fix the performance profile.19:49
sshnaidm /me can see19:49
bnemecIt seems like the default cpu scaling doesn't scale fast enough to handle our workload properly.19:50
*** akuznetsov has joined #tripleo19:52
*** akuznetsov has quit IRC19:55
derekhsshnaidm: te_broker and geard started up again, rechecking something19:56
*** antipsychiatry has quit IRC19:57
bnemec\o/ finally got rid of the 1292 instance that has been stuck deleting for about a month.19:57
*** akuznetsov has joined #tripleo19:58
*** davidlenwell has quit IRC19:59
*** florianf has quit IRC20:00
derekhbnemec: did you do anything special beside keep trying? I tried a force-delete earlier but it didn't seem to work20:01
bnemecderekh: I did a nova reset-state on it first, then tried regular delete again.20:01
bnemecI figured we needed to clear it out since we'll be resetting the testenv counter here. :-)20:03
bnemecderekh: So are we back up and running?20:04
bnemecAh, I see testenvs creating.  We must be.20:04
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Don't send metrics to graphite in failed jobs  https://review.openstack.org/36511420:05
bnemecHere goes nothing. :-)20:05
derekhbnemec: yup, fingers crossed we're back in business20:11
*** akuznetsov has quit IRC20:11
*** davidlenwell has joined #tripleo20:12
*** ChanServ sets mode: +v davidlenwell20:12
derekhFirst job got an env20:13
*** jayg is now known as jayg|g0n320:14
derekhand the second20:14
*** jpena|away is now known as jpena|off20:15
bnemecsix more just went create complete20:18
bnemecAlthough I have to admit 6.5 minutes to create one of these simple stacks seems excessive.20:19
bnemecKeyError: 'provision-1'20:26
bnemecHmm, not good.20:26
bnemecIt claimed that the heat stack completed, but apparently it didn't.20:27
openstackgerritSteven Hardy proposed openstack/tripleo-common: Wire in jinja templating for custom roles  https://review.openstack.org/36246520:28
bnemecYeah, we've still got problems.  Three jobs have already failed on problems setting up the testenv.20:30
derekhbnemec: stop it with the good news20:30
openstackgerritMerged openstack/tripleo-heat-templates: Change NeutronWorkers default to result in previous behavior  https://review.openstack.org/36448320:31
derekhbnemec: I think is does seem better but not good enough, wanna stop it and do the heat db delete or controller reboot so?20:32
*** radeks has quit IRC20:34
bnemecderekh: I dunno.  Heat says it created all the things successfully.  Maybe there's a race between when neutron says the port is created,  nova says the port is attached, and when it actually happens.20:38
bnemecI'm trolling logs to try to figure out what might have happened.20:38
*** ayoung has quit IRC20:41
derekhand the other one failed on a different error,20:41
*** Goneri has quit IRC20:41
bnemecYeah, I see a 404 in there too.20:43
bnemecWhich interestingly is an error I saw on Friday as well when trying to list the instances from the command-line.20:43
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: Generate Keystone credentials for overcloud  https://review.openstack.org/36628720:44
*** pradk has joined #tripleo20:44
bnemecBut why would a simple nova list 404?20:44
* bnemec shrugs20:44
derekh2016-09-06 20:17:02.059 19033 INFO nova.osapi_compute.wsgi.server [req-34de622f-2a14-421d-b223-6a58cb730e0b ba119eef29ce49f5b8697f4d63948e3c b79291658f384b7ebbc9019b6349e5c9 - - -] "GET /v2.1/b79291658f384b7ebbc9019b6349e5c9/servers/ HTTP/1.1" status: 404 len: 240 time: 1.527310120:45
derekhit 404'd alright20:45
*** weshay_mtg is now known as weshay20:47
EmilienMdprince: please review https://review.openstack.org/#/c/366240/20:47
EmilienMdprince: requirement for Newton20:47
EmilienMdprince: /me working on overcloud now20:47
*** bfournie has quit IRC20:48
dprinceEmilienM: looking20:48
dprinceEmilienM: keystone makes me sad20:49
EmilienMdprince: and me?20:49
dprinceEmilienM: so much backwards incompatability20:49
EmilienMi'm super sad20:49
EmilienMthat is terrible20:49
EmilienMand this is not finished20:49
EmilienMthe overcloud part is crazy20:49
dprinceEmilienM: thanks for looking into these20:50
EmilienMwe have no choice20:50
EmilienMwe can't bump keystone otherwise20:50
*** tzumainn has joined #tripleo20:50
EmilienMweshay, dmsimard: where is /etc/gnocchi ? https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-minimal_pacemaker-109/overcloud-controller-0/etc/20:51
EmilienMor maybe you don't install it?20:51
EmilienMdamn wrong windo20:51
*** fzdarsky has quit IRC20:53
trownEmilienM: should we recheck on https://review.openstack.org/#/c/366240/ or is OVB jobs known issue?20:53
EmilienMovb jobs are down20:53
EmilienMmy patch pass the undercloud-multinode patch, we can go ahead IMHO20:54
EmilienMbut we can also wait, no opinion20:54
trownk, if that somehow passed multinode and it is actually broken, then the multinode job is very broken20:55
trownso lets go dor it20:55
trownfor it even20:55
*** oneswig has joined #tripleo20:55
derekhbnemec: rabbit is using a lot of RAM aswell, I wonder if its worth a restart /me is kind of just trying random things now20:56
EmilienMtrown: thanks20:57
EmilienMtrown: we need that work to be done so we can unpin keystone20:57
* trown is a bit confused about how it is ok to pin keystone in the first place... but gets that it is crunch time20:58
bnemecderekh: Worth a shot.20:58
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: Generate Keystone credentials for overcloud  https://review.openstack.org/36628720:59
EmilienMtrown: FYI that is my work for overcloud ^21:00
EmilienMtrown: highly WIP now.21:00
EmilienMtrown: but I expect to finish it tonight21:00
trownEmilienM: k I will check it out tomorrow21:00
derekhbnemec: done21:02
*** trown is now known as trown|outtypewww21:02
derekhbnemec: sshnaidm I gotta run, will pick up on this tomorrow,21:02
bnemecderekh: Okay, have a good one.21:03
derekhbnemec: let us know if you find anything, or if you want to to try the reboot send intructions about the cpu scaling thing (or if you decide to do it yourself that fine by me)21:04
bnemecderekh: Yeah, I think I'll hold off for now.  I don't see any more env failures since those first few.21:04
derekhbnemec: ok, ttyl21:05
*** derekh has quit IRC21:05
*** lblanchard has quit IRC21:09
*** rhallisey has quit IRC21:12
*** dprince has quit IRC21:16
openstackgerritMerged openstack/diskimage-builder: Revert "Revert "Pre-install pip/virtualenv packages""  https://review.openstack.org/34193421:21
*** mcornea has quit IRC21:23
*** jcoufal_ has quit IRC21:29
*** rhallisey has joined #tripleo21:34
*** mbound has quit IRC21:40
*** pradk has quit IRC21:43
*** rhallisey has quit IRC22:00
*** oneswig has quit IRC22:02
*** cdearborn has quit IRC22:08
*** Goneri has joined #tripleo22:10
openstackgerritMerged openstack/instack-undercloud: keystone: configure credentials  https://review.openstack.org/36624022:11
*** jeckersb_gone is now known as jeckersb22:17
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Configure Keystone credentials  https://review.openstack.org/36640022:26
EmilienMok this is the overcloud version ^22:26
stevebakerbnemec: so ah, what do you think about deleting all the elements in tripleo-image-elements which are not used?22:27
*** Goneri has quit IRC22:29
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Expose parameter to enable combination alarms  https://review.openstack.org/36374822:30
*** jeckersb is now known as jeckersb_gone22:31
EmilienMI would need some review on https://review.openstack.org/#/c/364916/ https://review.openstack.org/#/c/366128/ https://review.openstack.org/#/c/364917/ and https://review.openstack.org/#/c/365117/22:34
EmilienMit will fix Aodh and Gnocchi in TripleO22:34
EmilienMweshay: ^22:34
EmilienMgate-tripleo-ci-centos-7-scenario001-multinode-nv is now green22:34
EmilienMwhich means Telemetry is working !22:35
*** rajinir has joined #tripleo22:36
openstackgerritSteve Baker proposed openstack/tripleo-image-elements: Make os-collect-config element package driven  https://review.openstack.org/36640322:37
openstackgerritSteve Baker proposed openstack/tripleo-image-elements: Make os-apply-config package driven  https://review.openstack.org/36640422:37
openstackgerritSteve Baker proposed openstack/tripleo-image-elements: Make os-refresh-config element package driven  https://review.openstack.org/36640522:37
EmilienMtrown|outtypewww: ^22:38
stevebakerEmilienM: do you know if this makes Aodh use keystone v3? https://review.openstack.org/#/c/365117/22:38
openstackgerritSteve Baker proposed openstack/python-tripleoclient: Replace agent elements with package python-heat-agent-puppet  https://review.openstack.org/36640622:39
EmilienMstevebaker: i'm not sure if we need to set a auth_plugin specific to keystone v322:39
EmilienMwe use 'password' everywhere22:39
*** yamahata has quit IRC22:40
stevebakerEmilienM: we have a heat trusts Aodh bug which this might help with https://bugzilla.redhat.com/show_bug.cgi?id=1367138#c1622:41
openstackbugzilla.redhat.com bug 1367138 in openstack-heat "Autoscaling with trust notifier doesn't work" [Unspecified,New] - Assigned to sbaker22:41
EmilienMstevebaker: we set auth_type = password I confirm22:43
EmilienMdoes it matter?22:43
stevebakerEmilienM: in heat?22:44
openstackgerritSteven Hardy proposed openstack/python-tripleoclient: Get template contents from plan, not local path  https://review.openstack.org/36573522:45
EmilienMstevebaker: in aodh22:45
stevebakerEmilienM: ah, I don't actually know. I've not looked into that bug yet.22:47
*** yamahata has joined #tripleo22:49
*** akshai has quit IRC22:53
*** rlandy is now known as rlandy|bbl22:55
*** lblanchard has joined #tripleo23:01
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: Add libffi-dev to bindep.txt  https://review.openstack.org/36641323:01
EmilienMpabelanger: ^23:01
pabelangerEmilienM: looks right23:02
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: Generate Keystone credentials for overcloud  https://review.openstack.org/36628723:03
*** ayoung has joined #tripleo23:10
*** dbecker has joined #tripleo23:20
*** akshai has joined #tripleo23:22
*** maeca2 has joined #tripleo23:23
*** maeca1 has quit IRC23:25
openstackgerritMerged openstack/diskimage-builder: fix systemd resource deadlock  https://review.openstack.org/36579723:33
*** akshai has quit IRC23:33
EmilienMI am going to self approve https://review.openstack.org/#/c/366413/ because CI is broken for tripleoclient23:38
EmilienMexcept if anyone around can look23:39
EmilienMbnemec, thrash ^23:39
thrashEmilienM: looking23:49
EmilienMthrash: see my comment23:49
EmilienMis it ok?23:49
thrashEmilienM: lgtm23:49
openstackgerritSteve Baker proposed openstack/tripleo-image-elements: Remove unused elements  https://review.openstack.org/36641723:51

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!