Friday, 2016-09-23

openstackgerritMerged openstack/instack-undercloud: Ensure that the default plan was created successfully
*** rajinir has quit IRC00:05
*** gfidente is now known as gfidente|afk00:05
*** yamahata has quit IRC00:09
*** limao has joined #tripleo00:23
*** rbrady has quit IRC01:09
dtrainorI have a stuck deployment.  There's nothing happening on either of my one Control or one Compute Overcloud instances.  No obvious errors.  It's just... stuck.  They're stuck on ComputeNodesPostDeployment and ControllerNodesPostDeployment, respectively.  I've exhausted my troubleshooting options.01:12
*** dmacpher-afk is now known as dmacpher01:21
*** dprince has joined #tripleo01:22
*** dprince has quit IRC01:22
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: Add swift proxy for ceilometer middleware
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: Switch puppet-tripleo to use puppet-openstack_spec_helper
*** bana_k has quit IRC01:33
*** bfournie has joined #tripleo01:55
*** kjw3 has joined #tripleo02:04
*** kjw3 has quit IRC02:19
*** thrash is now known as thrash|g0ne02:35
openstackgerritMerged openstack-infra/tripleo-ci: Use low-memory-usage.yaml in ci
*** rajinir has joined #tripleo02:56
openstackgerritMerged openstack/python-tripleoclient: Remove the get_hiera_key function
openstackgerritMerged openstack/tripleo-common: Updated from global requirements
openstackgerritMerged openstack/instack-undercloud: Update default VM memory
*** david-lyle has quit IRC03:04
*** rbrady has joined #tripleo03:05
*** ayoung has joined #tripleo03:26
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Undercloud install with tripleo-quickstart
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore
*** fultonj has quit IRC03:36
*** jrist has joined #tripleo03:59
*** pgadiya has joined #tripleo04:06
*** coolsvap has joined #tripleo04:08
*** limao has quit IRC04:26
*** limao has joined #tripleo04:28
*** limao has quit IRC04:31
*** limao has joined #tripleo04:34
*** rajinir has quit IRC05:05
*** jaosorior has joined #tripleo05:10
*** bana_k has joined #tripleo05:12
*** ebalduf has quit IRC05:14
*** limao has quit IRC05:32
*** limao has joined #tripleo05:37
*** absubram has quit IRC05:47
jaosoriorbandini: hey dude! how's it going?05:56
bandinijaosorior: hola! it's friday, nothing can bring me down ;) how about you?05:59
jaosoriorhaha all good here05:59
jaosoriorcoffee coffee coffee05:59
matbubandini: even another new upgrade blocker bug ? :)06:00
bandinimatbu: ahaha I collect those like folks collect pokemons ;)06:02
bandinimatbu: this is my last one from last night btw06:02
openstackLaunchpad bug 1626628 in tripleo "M/N Upgrade - major-upgrade-pacemaker times out" [Critical,New]06:02
*** ianw is now known as ianw_pto06:02
matbubandini: lol06:03
matbubandini: hmm progress now you hit pcs issue :)06:04
bandinimatbu: i think it is heat. we never get to pcs commands06:08
*** yamahata has joined #tripleo06:09
bandiniI can reproduce it 100% it seems06:12
*** mbozhenko has joined #tripleo06:14
*** rcernin has joined #tripleo06:15
*** rasca has joined #tripleo06:15
*** saneax-_-|AFK is now known as saneax06:25
*** tobias_fiberdata has joined #tripleo06:27
*** bana_k has quit IRC06:31
mariosmatbu: bandini jaosorior o/06:34
marioswe made it to friday, now just keep calm and it will finish soon enough06:34
* marios cries06:34
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Customize undercloud and overcloud with virt-customize
*** aufi has joined #tripleo06:39
matbumarios: o/06:40
*** florianf has joined #tripleo06:41
matbumarios: don't cries, it's a light week for you next week ? :)06:41
jaosoriormarios: coffee will make it all better dude06:42
*** tremble has joined #tripleo06:42
mariosmatbu: :( i think that makes it worse. i mean there is so much stuff going on my head hurts06:43
*** jprovazn has joined #tripleo06:43
openstackgerritMerged openstack/diskimage-builder: Allow ramdisk-create to run without $USER set
*** cylopez has joined #tripleo06:47
*** mcornea has joined #tripleo06:52
*** jlinkes has joined #tripleo06:57
*** tzumainn has quit IRC07:05
*** dsariel has joined #tripleo07:09
openstackgerritDougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names
*** gfidente|afk is now known as gfidente07:13
gfidentematbu, curiosity, have you attempted upgrade with ceph again?07:14
matbugfidente: nop not yet07:15
matbugfidente: i think chem (not online) does07:15
*** panda|zZ is now known as panda07:16
gfidenteit looks like the version of the heatclient we have07:16
gfidentewe install07:16
matbugfidente: you mean regarding this issue ?07:18
openstackLaunchpad bug 1623506 in tripleo "Unknown Property batch_create in SoftwareDeploymentGroup for CephMon upgrade" [Undecided,Fix released] - Assigned to Giulio Fidente (gfidente)07:18
*** jbadiapa has joined #tripleo07:18
gfidenteso that should be sorted now07:18
gfidentebut now I needed
gfidenteand it's not there07:20
mariosgfidente: good morning man. thanks very much for looking at puppet-manila I just couldn't get to it yesterday was planning on digging today (not sure when exactly)07:21
gfidentemarios, I think I'll test if that change fixes it or not07:21
gfidenteas it might not be the root cause07:22
mariosgfidente: sure, i know tbarron was also looking, hopefully he will get a run in today too07:22
gfidentethough I saw db sync failing on the controller so I suppose it was just trying to run it before the db was created07:22
jaosoriormarios, gfidente if you have time can you check this commit out?
mariosjaosorior: ack07:22
gfidentejaosorior, that's cool07:24
gfidentebut is it setting the servername to which the virthost responds to?07:24
*** flepied has quit IRC07:24
gfidenteI think that if haproxy is calling it by ip we need to add serveralias too?07:25
jaosoriorgfidente: so, at the moment haproxy is using the IP to point to that server07:25
jaosoriorgfidente: so, right now it will route with the IP and port that the virtualhost is using07:26
gfidentejaosorior, so if I remember correctly httpd will see if string matches the virtualhost name and if it doesn't, it will serve the first virtualhost07:27
jaosoriorif there are collisions in the IP, then it will route using the servername07:27
gfidenteyeah so don't we need to use the IP as servername/serveralias?07:27
jaosorior"It is important to recognize that the first step in name-based virtual host resolution is IP-based resolution."07:28
jaosoriorgfidente: not really. we already have the IP set07:28
gfidenteah the virtualhosts are set as IP:name not *:name07:28
gfidenteack, thanks07:29
*** zoli_gone-proxy is now known as zoliXXL07:29
jaosoriorgfidente: it is only when we want to access the server via the hostname that we run into trouble07:29
jaosoriorwhich is what that fix wants to address07:29
jaosoriorgfidente: thanks for checking it out dude :D07:29
zoliXXLgood morning07:30
gfidenteso do you guys have any idea why on a fresh tripleo.sh07:30
gfidenteI don't get a change for heatclient dated 25th of aug?07:30
jaosoriorgfidente: maybe it's cause heat is not gotten from delorean-current07:31
gfidenteyeah I was looking into that07:31
gfidentewe don't install clients from -current07:31
jaosoriorshould we?07:31
gfidentenot sure, we don't get heat from -current either apparently07:32
gfidentebut still does that mean delorean did not promote these from a month now?07:32
*** jpena|off is now known as jpena07:33
jaosoriorgfidente: I have no idea07:33
jaosoriorgfidente: would need to ask in #rdo I guess07:33
gfidenteyep going :)07:34
*** ebarrera has joined #tripleo07:35
*** jpich has joined #tripleo07:37
gfidentejaosorior, apparently heatclient is pinned to some old tag07:39
gfidentesee #rdo07:39
jaosoriorgfidente: oh :/07:43
*** yolanda has quit IRC07:44
*** ohamada has joined #tripleo07:47
*** yolanda has joined #tripleo07:47
*** shardy has joined #tripleo07:49
*** hjensas has joined #tripleo07:50
jpichAll the client patches are failing with a UT failure - is anyone looking into this? I don't see any open bug07:53
* jpich looking07:53
*** athomas has quit IRC07:55
* jpich filed
openstackLaunchpad bug 1626885 in tripleo "Client unit tests failing: "TypeError: 'Mock' object is not iterable"" [Critical,Triaged] - Assigned to Julie Pichon (jpichon)07:56
matbujpich: hey thanks07:57
matbujpich: i looked briefly yesterday07:57
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Stacks and Resources data storing in app state
matbujpich: cause one of my review was hitting this failure07:57
jpichmatbu: Yeah, that's what got me started too :)07:58
jpichmatbu: Did you find anything? I can reproduce after recreating my tox env, that's about as far as I got07:58
jpichIf you had a fix on the way feel free to reassign!07:59
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Add metricd workers support in gnocchi
matbujpich: no i don't try to fix yet, i was just making sure that it wasn't my review which break it :)07:59
matbui'll try to see if i can08:00
jpichProbably should have just renamed my tox env to compare versions more easily, blergh08:00
jpichmatbu: Cool, cheers :)08:00
*** athomas has joined #tripleo08:01
shardybandini: Hey, quick sanity check re custom-roles and ha-lite when you get a moment08:03
shardybandini: the DB and rabbit services are still managed by pacemaker in Newton, and that means they are tied to the controller, right?08:04
shardyI'm being asked if it's possible to define e.g a "database" role, and just deploy the DB services on those nodes08:04
shardybut I think that won't be possible until we've got the composable HA stuff worked out?08:05
d0ugalAt the moment if a workflow fails and doesn't post to Zaqar the CLI hangs forever. I have a patch that adds timeouts, it works well enough but does anyone have any idea how I can figure out what a sensible default timeout is for each of the commands?08:05
gfidenteshardy, curious what are the expectations, to have two different pacemaker clusters one for rabbit and one for database nodes08:05
shardyd0ugal: can you put a log line with some timing into the client, then look at CI results for a few days to figure out the upper bound?08:06
gfidenteshardy, or to deploy the database role withouth pacemaker?08:06
*** mbozhenko has quit IRC08:06
shardyd0ugal: then double it or something?08:06
gfidenteshardy, looks like the 2nd scenario could be worked out fixing the non-pacemaker roles for galera08:06
d0ugalshardy: Sure, that seems fine - but say, for example, introspection could take a really long time in the real world if you have enough nodes.08:06
d0ugalshardy: so maybe that one should never timeout?08:07
d0ugalor maybe 4 hours as then we will hit token issues anywya.08:07
bandinishardy: correct yes. until composable ha you cannot really split out pacemaker services out08:07
shardyd0ugal: that makes me think the client should never be making synchronous calls to the workflow?08:07
d0ugalshardy: it doesn't.08:07
shardye.g perhaps we should poll the queue rather than block on a websocket?08:07
d0ugalshardy: hrm, maybe.08:08
shardyd0ugal: my thinking is you could then provide incremental status to users and just let them decide if it's stuck08:08
d0ugalshardy: Isn't the result the same? How long do we poll for?08:08
shardyvs giving no output for 4 hours08:08
d0ugalshardy: I see - but what output would you give? "Polling for messages... none found" over and over.08:09
shardyd0ugal: isn't the problem that when a workflow fails, we should always get a failure notification?08:09
d0ugalshardy: Yeah, so I guess polling Mistral for the workflow status makes sense.08:09
openstackgerritmathieu bultel proposed openstack/python-tripleoclient: Keystone credentials and CephClusterFSID needs to be set with the overcloud password
d0ugalThis would all make much more sense if the CLI didn't block - then we could check a few things.08:10
shardygfidente: I'm not clear on the requirement around HA here, the only request is how do we scale e.g DB nodes beyond 3 nodes08:10
shardygfidente: I assume they would want a separate pacemaker cluster, but the non-pcmk approach might be a workable interim solution I guess08:10
*** masco has joined #tripleo08:10
shardyd0ugal: Yeah, that's what I was getting at with the synchronous comment08:11
shardylike, show the status of the nodes during introspection vs doing nothing08:11
shardyI do wonder if we could wire that in via the workflow though?08:11
d0ugalshardy: Right, but really the workflow should be sending us all that status information - the only reason it doesn't is due to time constraints.08:12
d0ugalshardy: I'd like to do that for O.08:12
bandinishardy, gfidente: here is an initial wip for composable HA roles my plan is to write a spec and submit it for ocata08:12
shardyd0ugal: you could have a trigger introspection workflow, and a check for introspection complete workflow08:12
shardythe latter could be restarted on timeout08:12
shardy(same pattern for all long running calls)08:13
matbujpich: few hours ago a commit has fixed the failure08:13
shardyideally we'll fix the token reauth thing with mistral in ocata tho08:13
matbujpich: i comment the LP08:13
matbujpich: all UT are green in my local env08:13
shardybandini: thanks, will check it out08:13
jpichmatbu: Oh? That's great \o/08:13
shardybandini: my query is more about what is possible with Newton tho08:13
d0ugalshardy: okay, so rather than blocking and waiting for messages - poll the workflow status and then consume the messages.08:13
shardyd0ugal: Yeah, just throwing ideas around but perhaps that would help?08:14
shardyit'd be much easier to set a sane timeout on a thing that just checks status08:14
d0ugalshardy: Yeah, I think it would. It is less elegant but I can't think of any other ideas.08:14
bandinishardy: yeah so with newton you are basically limited to the number of pacemaker nodes you instal. you install 5 you get five rabbits and five galeras running there08:14
d0ugalshardy: Why would that be easier?08:15
d0ugalshardy: essentially the waiting for queue messages is just checking status too08:15
d0ugalshardy: and that is easy to timeout, I've already done it.08:15
shardybandini: Ok, I think we only reccomend max 3 controllers but that makes sense08:15
matbujpich: yep, a nice friday bug ;)08:15
*** akuznetsov has joined #tripleo08:16
shardyd0ugal: we can guestimate a reasonable time to wait for a check_$foo workflow to send a status message08:16
bandinishardy: that is correct yes. mainly due to a) none tested != 3 b) none proved it actually improves performance08:16
shardybecause a few GET calls shouldn't take a really long time, unlike creating some resource then waiting for an inteterminate amount of time for it to finish08:16
shardyd0ugal: e.g ironic node-list and nova list08:16
shardyI'd say anything more than, say, a minute for either and things are very broken, no?08:17
shardybandini: ack, yeah I was under the impression galera scale limitations meant it didn't necesarily improve performance, but have no data to back that up :)08:18
shardybandini: Ok, thanks for the confirmation!08:18
bandinishardy: yeah we should really gather some data on that eventually :)08:18
openstackgerritDougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names
jpichmatbu: Awesome, thanks. Of course d0ugal was already on top of it :)08:20
d0ugalshardy: jpich ?08:20
jpichd0ugal: Please open bugs when you fix big problems ;)08:20
d0ugalshardy: oops, that wasn't meant for you - sorry.08:20
d0ugaljpich: The failing unittest?08:20
jpichd0ugal: Yup :)
openstackLaunchpad bug 1626885 in tripleo "Client unit tests failing: "TypeError: 'Mock' object is not iterable"" [Low,Triaged] - Assigned to Julie Pichon (jpichon)08:21
d0ugalshardy: Okay, yeah, I think that makes sense. Trying to figure out how to change the CLI to do something like that now.08:21
jpichd0ugal: Just so people don't get too much of a surprise when they wake up to a red gate!!08:21
d0ugaljpich: heh, indeed, that is why I done a rushed fix yesterday08:21
d0ugaljpich: but I failed to realise it would need to be backported.08:22
jpichd0ugal: Tis ok. Wanna click that button or I can follow up with the backport?08:22
matbuyolanda: morning, i saw you merged :, do you know when the change would be apply with jjb ?08:23
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Change the level of mocking for the wait_for_stack_ready test
d0ugaljpich: ^08:23
yolandahi matbu , if that already was merged, it can take over half an hour to be applied08:24
jpichd0ugal: \o/ Thank you for always being on top of this kind of problems!08:24
matbuyolanda: k, thx08:25
d0ugaljpich: np, I just run head first into them quickly it seems08:26
*** akrivoka has joined #tripleo08:26
*** flepied has joined #tripleo08:28
*** dsariel has quit IRC08:29
*** absubram has joined #tripleo08:30
shardyjaosorior: Hey, I had a question re
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Stop plan creation when container exists
shardyjaosorior: do you think we still need global_config_settings now we have the stricter service_config_settings interface?08:30
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Remove deprecation warning for bulk introspection
shardyI remember we were going to use global_ for some stuff, but I don't see any actual usage of it in-tree atm08:30
*** absubram_ has joined #tripleo08:32
*** absubram has quit IRC08:34
*** absubram_ is now known as absubram08:34
*** absubram has quit IRC08:34
*** hewbrocca-afk is now known as hewbrocca08:35
jaosoriorshardy: I remember we talked about this before. I don't think we need that anymore. The service-specific configs are sufficient08:37
jaosoriorshardy: only thing is that for the haproxy use-case we needed a deep merge08:38
jaosoriorso we could get rid of that ^^08:38
openstackgerritMerged openstack/tripleo-heat-templates: Make apache-based services use network-dependent servername
*** florianf has quit IRC08:41
shardyjaosorior: ah, yeah that would be nice to clean up! :)08:43
*** chem has joined #tripleo08:43
shardyOk, I'll propose a patch removing global_config_settings as if we release with it, I guess we'll have to maintain it08:43
jaosoriorshardy: sounds good to me08:43
*** zoliXXL is now known as zoli|brb08:44
*** derekh has joined #tripleo08:44
*** dtantsur|afk is now known as dtantsur08:45
openstackgerritDougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names
*** paramite has joined #tripleo08:45
*** chem has quit IRC08:46
*** chem has joined #tripleo08:46
*** florianf has joined #tripleo08:47
jistrmerge? :)
*** b00tcat has joined #tripleo08:48
openstackgerritMerged openstack/diskimage-builder: elements: opensuse: Add support for openSUSE Leap
shardyjistr: done!08:49
shardyNice that the PostDeploySteps refactoring made this a little cleaner :)08:50
*** zoli|brb is now known as zoli08:50
jistryea :)08:50
gfidenteshardy, so service_config_settings is like global_08:50
gfidentebut it only distributes to the nodes where service_name is deployed?08:51
shardygfidente: yes, exactly08:51
gfidentewow :)08:51
*** cylopez has quit IRC08:56
*** cylopez1 has joined #tripleo08:56
hewbroccaMorning folks!08:56
shardy'morning hewbrocca !08:56
*** pkovar has joined #tripleo08:58
b00tcatanother silly question: if I don't specify a `-e` option to `openstack overcloud deploy`, what's the used env file?09:03
b00tcator are the default values used?09:03
*** openstackgerrit has quit IRC09:03
*** openstackgerrit has joined #tripleo09:03
*** yamahata has quit IRC09:03
shardyb00tcat: it only includes the "base" environment in that case, which is overcloud-resource-registry-puppet.yaml09:04
shardyb00tcat: the idea is you pass a series of -e options which then either override or add to what's in that base registry09:04
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Add the workflow name to the starting workflow message
*** pmannidi has quit IRC09:06
b00tcatgot it shardy , thanks again!09:06
* tbarron drinks coffee and reads the backlog09:08
tbarrongfidente: w.r.t. did you want to try something yourself on that beaker machine? Otherwise since I don't know fancy artefact injection tricks I'd tear down the overcloud, pull in that patch on top of, and redeploy09:14
Jokke_morning tbarron09:14
tbarronmatbu: Jokke_ ^^09:14
tbarronJokke_: morning it is :)09:15
gfidentetbarron, give me a sec and we do artifacts together09:15
tbarrongfidente: k, no rush, thanks09:15
* matbu brb09:15
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Remove hard-coded roles in EnabledServices output
openstackgerritGiulio Fidente proposed openstack/tripleo-puppet-elements: Install ceph-selinux in overcloud-{controller,cephstorage} images
openstackgerritDougal Matthews proposed openstack/tripleo-common: Use kwargs to pass in data and error to Mistral Result
*** limao has quit IRC09:24
*** limao has joined #tripleo09:26
d0ugalEvery time I see "WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils" it makes me a little mad09:28
gfidentetbarron, now?09:29
gfidenteJokke_, ^^?09:29
tbarrongfidente: Jokke_ I'm there09:31
tbarrongfidente: Jokke_ watching09:31
*** milan has joined #tripleo09:33
jpichd0ugal: Shouldn't it all be gone now though?09:37
*** akuznetsov has quit IRC09:37
d0ugaljpich: I feel like it should, but it isn't09:40
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Make the ini-based validations more robust
*** panda is now known as panda|afk09:43
jpichd0ugal: Maybe 'cause it's an old environment09:48
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Make the ini-based validations more robust
bandiniany heat pro that can tip me in the right direction on  how to debug
openstackLaunchpad bug 1626628 in tripleo "M/N Upgrade - major-upgrade-pacemaker times out" [Critical,New]09:49
d0ugaljpich: Yeah, maybe. I'm reinstalling anyway.09:49
therveshardy, I'm looking at heat memory issue09:50
therveshardy, Are you still using CFN as a signal transport everywhere?09:50
gfidentebandini, me worried about the cephmonupgrade part09:51
bandinigfidente: it could be that it is simply because it is the first resource there?09:52
gfidentebandini, or the script itself timing out09:52
gfidentedo you see any of the echo lines from the shell in the journal log?09:52
bandinigfidente: let me double check09:53
*** abehl has joined #tripleo09:53
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Make the ini-based validations more robust
*** tosky has joined #tripleo09:56
gfidentebandini, how many storage nodes you have?09:57
gfidentethis loop would run forver if OSDs < replica size09:57
gfidentewell not forver, 300secs09:57
shardytherve: Hi!09:57
*** florianf has quit IRC09:58
bandinigfidente: I have none09:58
shardytherve: thanks for looking into it - I tried the three patches from zaneb and it's not made a lot of difference09:58
openstackgerritDougal Matthews proposed openstack/tripleo-common: Use kwargs to pass in data and error to Mistral Result
openstackgerritMerged openstack/tripleo-heat-templates: No-op Puppet for upgrades/migrations according to composable roles
shardytherve: yes, we're still using the default CFN transport for signalling09:58
shardytherve: but we switched to swift tempurl for polling metadata09:58
therveshardy, Which ones are those patches?09:58
therveAh, ok09:58
shardytherve: ^^09:59
shardyI rebased on latest master and applied the two remaining patches09:59
therveCool, thanks09:59
shardypeak memory usage is slightly reduced, but not by much09:59
shardytherve: I can fairly easily reproduce my test now, do you want me to try with a different signal transport?10:01
therveshardy, No I was wondering if that could a reason for the change10:01
shardythe plan is to switch everything over to zaqar, but we've not quite got there yet10:02
skramajamcornea: thanks for the ServiceNetMap. now the host name is proper for the new role (compute-dpdk)10:03
skramajashardy: mcornea: but i am facing another issue at step 3 in the controller..10:04
jaosoriorshardy: by the way, have you come up with a solution for the remaining haproxy configuration that's outside the service profile? the service_certificate10:04
skramajapasted the log output. any clue?10:04
shardyskramaja: is that error happening on your new role, e.g the compute-dpdk node?10:05
shardyor the controller?10:05
skramajaits happening on the controller shardy10:06
shardyskramaja: Hmm, and this environment was all working fine before you started working with custom roles?10:06
mcorneaskramaja: I haven't seen this but it points that it can't run openstack project list so I'd check how keystone looks on controller nodes10:06
shardyyeah, it looks like keystone is broken10:06
shardybut adding the new compute role shouldn't have caused that (in theory at least..)10:06
skramajayes.. thats my confusion too.10:07
skramajaanyways still looking.. if you have any pointers it would make easier..10:07
shardyI'd check the status of keystone and try running it manually to get an idea of why it's not responding10:07
*** thrash|g0ne is now known as thrash10:07
*** yolanda has quit IRC10:09
*** florianf has joined #tripleo10:10
*** zoli is now known as zoli|lunch10:11
skramajaok shardy10:13
*** limao has quit IRC10:14
*** leanderthal|afk is now known as leanderthal10:17
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Add network gateway validation
*** panda|afk is now known as panda10:22
*** fultonj has joined #tripleo10:23
openstackgerritMerged openstack/tripleo-heat-templates: explicitly set fluentd service_provider
*** jprovazn has quit IRC10:23
shadowerHey folks, could I get these two validations reviewed? and
shadowerthe first one fixes a bug with a couple existing ones and a lot of others depend on the latter10:24
jpichshadower: Hello! When you have a chance sometime, do you think you could add a few thoughts/limitations/possible workarounds/words of warnings about validations in ?10:25
shadowerjpich: interesting. Yeah I'll do10:27
jpichshadower: Thank you!10:27
shardyshadower: lgtm, added to rc210:27
shardyshadower: do we have any CI coverage of validations yet?10:27
shadowershardy: thanks! And not yet but I'd like to make that happen in a few weeks (though we may want to discuss it at the summit first)10:28
*** yolanda has joined #tripleo10:30
*** kbyrne has quit IRC10:31
shardyshadower: ack, sounds good10:32
shardyI think a discussion on how we get validations generally better integrated (including CI) would be a good thing :)10:33
*** dtantsur is now known as dtantsur|bbl10:34
*** kbyrne has joined #tripleo10:34
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Validate the IP range for the control plane
*** kbyrne has quit IRC10:39
*** kbyrne has joined #tripleo10:39
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Validation to verify connectivity to overcloud
*** dprince has joined #tripleo10:58
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Composable Mistral services
*** rasca has quit IRC11:03
*** rasca has joined #tripleo11:04
*** lucas-afk is now known as lucasagomes11:05
thrashccamacho|afk: it's not creating the endpoint, which is why the pingtest is failing.11:11
*** ccamacho|afk is now known as ccamacho11:11
ccamachothrash ack let me it11:12
thrashor I should say, it's not creating the endpoint correctly.11:12
ccamachothrash ill deploy locally, as that service its almost good to go then, are you working on it?11:13
jaosoriorthrash: ccamacho: by the way, the keystone::auth parameters will change place (hopefully today) you will need to specify them in t-h-t in another section called service_config_settings11:13
thrashccamacho: yes, I'm working on it.11:14
thrashjaosorior: ack.11:14
jaosoriorthrash, ccamacho: if you need help debugging that endpoint issue I can take a look.11:14
ccamachojaosorior, can you paste that submission like?11:14
thrashjaosorior: it's creating the publicurl, but not admin and internal.11:15
thrashsee the paste above..11:15
ccamachothrash, then I wont push any update, just local tests and just give some feedback11:15
thrashccamacho: thanks11:15
jaosoriorthrash: undercloud or overcloud?11:15
thrashjaosorior: over11:15
jaosoriorthrash: pass the CR11:15
jaosoriorthrash: mistral has no network assigned11:16
jaosoriorthrash: you need to add it here
thrashjaosorior: ack11:17
jaosoriorthat's the issue. the endpoint map tries to fetch the value of MistralApiNetwork, and it fails. Thus, internal and admin are empty.11:17
jaosoriorpublic doesn't need the network, it uses the public ~VIP11:18
jaosoriorthrash: but it's good that it manifested there, you were gonna have a bunch of other issues too11:18
thrashjaosorior: thanks. that would have taken me forever to figure out. :)11:18
thrashjaosorior: like?11:19
thrashOr just because of that?11:19
jaosoriorbecause of that11:19
jaosoriorso, without that network mapping, probably the mistral_network variable wouldn't have been generated, or the nodes, or even the mistral_vip11:19
jaosoriorso to start with haproxy would have had a wrong configuration11:19
jaosorior(haproxy would have worked anyway, cause it will default to using ctlplane)11:20
*** dtantsur|bbl is now known as dtantsur11:20
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Move inclusion of ::manila::db::mysql in manila/api profile
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Activate StorageMgmtPort on computes in HCI environment
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: Composable Mistral services
*** hjensas has quit IRC11:27
*** hjensas has joined #tripleo11:29
*** hjensas has joined #tripleo11:29
*** yolanda has quit IRC11:33
*** zoli|lunch is now known as zoli11:33
*** zoli is now known as zoliXXL11:33
*** jeckersb is now known as jeckersb_gone11:34
*** yolanda has joined #tripleo11:34
openstackgerritMerged openstack/tripleo-heat-templates: Tolerate missing keys from role_data in service templates
*** jprovazn has joined #tripleo11:35
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton
openstackgerritJustin Kilpatrick proposed openstack/tripleo-quickstart: Add documentation for add-on roles
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Move keystone::auth into service_config_settings
*** rhallisey has joined #tripleo11:50
*** rbrady has quit IRC11:51
*** rbrady has joined #tripleo11:52
*** fultonj has quit IRC11:52
jaosoriordprince: hey man, could you take another look at this?
dprincejaosorior: yep11:55
*** jrist has quit IRC11:56
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use HCI Ceph in HA job
openstackgerritMerged openstack/tripleo-validations: Make the ini-based validations more robust
*** ccamacho is now known as ccamacho|lunch12:02
*** jpena is now known as jpena|lunch12:03
mcorneaEmilienM: shardy is being addressed by ? IIUC the patch resolves the situation when keystone runs on a different role but the bug is about the databases not being created on the db cluster12:04
openstackLaunchpad bug 1620595 in tripleo "When Heat API services run on a different node than controller the heat-dbsync fails due to incorrect mysql permissions" [High,In progress] - Assigned to Dan Prince (dan-prince)12:04
jaosoriormcornea: uhm... actually I think that's a bigger issue.12:05
jaosoriormcornea: to do the db-sync, for most services, there is a check if it's the "bootstrap node"12:05
mcorneajaosorior: from what I've seen the database gets created on the node where the service is running12:06
*** zoliXXL is now known as zoli|brb12:06
jaosoriorI'm not sure how the bootstrap node part will work on different roles12:07
jaosoriorshardy: how does the bootstrap node part work nowadays?12:08
*** jrist has joined #tripleo12:08
openstackgerritJiri Stransky proposed openstack/diskimage-builder: Fix grub installation for RHEL
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types
*** pkovar has quit IRC12:18
*** pkovar has joined #tripleo12:22
EmilienMmcornea: I'll look in a few12:23
mcorneaEmilienM: thanks12:23
*** hjensas has quit IRC12:24
shardyjaosorior: we set bootstrap node for all roles now, but we've not yet solved how to nominate a boostrap host if you deploy a service on multiple roles12:25
*** masco has quit IRC12:25
jaosoriorshardy: so we now trigger db-sync on all nodes that deploy a service?12:26
shardyjaosorior: No, only on the bootstrap node, exactly as before12:26
shardybut there's one bootstrap node per role (group of nodes)12:26
shardyso you can't deploy a service whidh does bootstrap things on two roles12:26
mandreSeing this in my keystone logs: SQL connection failed. -614 attempts left.12:27
shardyEmilienM: No, I think it's closely related but we may need to move some mysql things into service_config_settings12:27
mandresomeone messed his loop :)12:27
jaosoriorshardy: ok, that makes sense12:27
shardyjaosorior: I think we can rework it in ocata, to e.g look at $service_node_names and only use the first name in the list12:28
jaosoriorshardy: but... why don't we have a bootstrap node per service?12:28
shardyjaosorior: because nobody has written that patch yet ;)12:28
jaosoriorshardy: well, seems that would fix the bug mcornea mentioned12:28
*** jayg|g0n3 is now known as jayg12:29
jaosoriorI don't have cycles for that :(12:29
jaosoriorbut alright12:29
jaosoriorthanks for the explanation12:29
shardyjaosorior: maybe, isnt' the problem there that it's trying to do the db-sync on the wrong role?12:29
shardyI need to test it tbh12:29
jaosoriorshardy: well, if heat is deployed in a role that isn't the controller, how would it even do the db-sync? If that's what we explicitly check for12:31
openstackgerritmathieu bultel proposed openstack/python-tripleoclient: Keystone credentials and CephClusterFSID needs to be set with the overcloud password
openstackgerritMerged openstack/python-tripleoclient: Change the level of mocking for the wait_for_stack_ready test
*** zoli|brb is now known as zoli12:34
*** zoli is now known as zoliXXL12:34
openstackgerritMerged openstack/python-tripleoclient: Display error message when socket is closed
*** pgadiya has quit IRC12:34
openstackgerritMerged openstack/python-tripleoclient: Provide more information when 'node provide' fails
shardyjaosorior: The idea is it'd do the db-sync on the bootstrap node of $role_not_controller12:34
shardybut evidently something isn't wired in atm to allow that12:34
shardyprovided we have the connection string to reach the DB, I think it should work12:35
jaosoriorthat was my understanding12:35
shardybut something is missing from the configuration atm12:35
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use HCI Ceph in HA job
*** rcernin has quit IRC12:38
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use HCI Ceph in HA job
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Make the ovb-updates job work again
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton
gfidentebnemec, dah rebased
gfidentewanted to make depend on it12:43
*** pcaruana|afk| has joined #tripleo12:44
*** cylopez1 has left #tripleo12:45
mandrehello, none of my services on the overcloud can talk to mariadb terminated by haproxy,
*** fultonj has joined #tripleo12:47
mandrehere's my haproxy.conf:
*** pgadiya has joined #tripleo12:50
*** david-lyle has joined #tripleo12:56
*** tzumainn has joined #tripleo12:57
*** pcaruana|afk| has quit IRC13:00
EmilienMshardy: I'll let you +A or not13:00
*** pcaruana has quit IRC13:01
*** pcaruana has joined #tripleo13:02
beaglesshardy: what are your thoughts on when we move the l3 ha logic from the tht to puppet-tripleo? Better to do it now, or would it be better to hold onto it until Ocata? I'm good either way.13:02
*** rcernin has joined #tripleo13:06
openstackgerritBrad P. Crochet proposed openstack/puppet-tripleo: Add support for rabbit hosts to mistral
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: Composable Mistral services
EmilienMthrash: why not doing it via Hiera? ^13:12
thrashEmilienM: I was just following along with what every other service was doing... Am I doing it wrong then? :)13:12
*** jpena|lunch is now known as jpena13:13
EmilienMyes, we need to define it with hiera13:13
EmilienMin the composable service13:13
*** jaosorior has quit IRC13:13
thrashEmilienM: just curious, but why is literally every other service doing it that way? (I don't care, makes it easier for me)13:14
*** jaosorior has joined #tripleo13:14
EmilienMthrash: let me check but it's a mistake13:14
EmilienMthrash: we use hiera for parameters13:14
EmilienMjaosorior: fyi ^13:14
*** cdearborn has joined #tripleo13:14
thrashLiterally every other one.13:15
EmilienMit's bad13:15
EmilienMwe need to use hiera in THT13:15
EmilienMthat's how we configure services13:15
openstackgerritRyan Brady proposed openstack/tripleo-common: Port password generation from tripleoclient to tripleo-common
EmilienMin "puppet way"13:15
thrashEmilienM: shardy did it 26 days ago13:15
EmilienMbut again that's an error13:16
EmilienMpuppet profiles should only do "include ::myclass"13:16
EmilienMand THT should feed data with Hiera13:16
thrashEmilienM: I'm not arguing. I totally agree.13:16
thrashshardy: so, why did you do it that way, and not via hiera?13:17
EmilienMthrash: I'm working on a fix13:17
*** zoliXXL is now known as zoli|brb13:18
gfidentemarios, can you add the depends-on
gfidentefor the manila submission?13:19
gfidenteseems like we got it passing13:19
*** adarazs is now known as adarazs_brb13:19
gfidentedeployment failed on httpd failing to start but the manila part was in place13:19
EmilienMthrash: wait13:20
gfidentewhile we retry that I think depends-on will speed up things13:20
EmilienMI don't see rabbitmq_node_ips in THT13:20
thrashEmilienM: I was just seeing that myself...13:20
thrashBut yet, somehow, all the other services are getting it configured...13:20
*** myoung|gone is now known as myoung13:21
EmilienMthrash: go ahead with mistral13:21
EmilienMignore my -113:21
thrashEmilienM: ack13:21
mariosgfidente: ack incoming fantastic to hear13:21
openstackgerritBrad P. Crochet proposed openstack/puppet-tripleo: Add support for rabbit hosts to mistral
thrashEmilienM: it looks like mistral was just missed in
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add integration with Manila CephFS Native driver
gfidentemarios, wait wrong link13:23
mariosJokke_: added the depends on gfidente yes it should be right13:24
mariosJokke_: gfidente:
*** pkovar has quit IRC13:24
EmilienMthrash: so there is some magic in tht13:25
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Add a validation-specific Ansible output callback
*** rcernin has quit IRC13:26
mariosgfidente: ah i see you changed it a bit, removed the rabbit_hosts added bootstrap_node13:26
gfidentemarios, that's so scheduler can pick up rabbit hosts from different node13:27
gfidentebut the db sync issue was inclusion of ::db::mysql in scheduler role while it's ::api attempting ::sync13:27
mariosgfidente: right thanks I +2 - you had a good run with it right?13:28
gfidenteyes but I am fixing lint stuff yet13:28
gfidentesorry :(13:28
mariosgfidente: kk thanks np ping me when/if wipes the votes13:28
mariosgfidente: thank YOU gfidente but stop talking now thanks13:28
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Add a validation-specific Ansible output callback
*** kjw3 has joined #tripleo13:29
trownpanda: do you happen to know where in tripleo-ci we setup the undercloud for network isolation? I see but have no idea how eth2 is setup on OVB13:31
EmilienMgfidente: can you approve this quick patch?
*** jeckersb_gone is now known as jeckersb13:32
gfidentebnemec, so no reason for adding it back to HA except I wanted it combined with netiso13:32
EmilienMgfidente: same for
gfidenteEmilienM, ack13:32
*** pkovar has joined #tripleo13:33
EmilienMgfidente: thanks :)13:33
*** Goneri has joined #tripleo13:34
bnemecgfidente: Ah, I guess that makes sense.  Too bad. :-/13:34
gfidentebnemec, but13:34
bnemecRather not add more time to the ha job, but it's better than adding it to updates. :-)13:34
gfidentebut I agree if we could balance something out on nonha it'd be better13:34
gfidenteespecially because ceph doesn't care about ha/nonha13:34
bnemecgfidente: Yeah, but I don't know that I want to add net-iso to nonha.  It's the only job that tests without it right now.13:35
gfidentebnemec, now, out of the blue13:35
gfidentedo we really need the non-netiso scenario in tripleo? :)13:36
pandatrown: what is missing from those lines that you're not finding ?13:36
bnemecgfidente: Well, that's a valid question.13:36
*** rcrit has joined #tripleo13:36
bnemecAnd we do sort of cover it in the multinode jobs.13:36
trownpanda: well there is no eth2 on default quickstart undercloud, and if I make just an extra network, that does not work either :)13:37
trownpanda: in the RDO CI code we make a vlan for network isolation, so trying to figure out what tripleo-ci needs from undercloud13:37
EmilienMgfidente: why do you tag it rc2 ?
EmilienMdoes it fix an RC2 bug?13:37
pandatrown: I'm stealing bnemec today to talk more or less about this. I'm trying to understand how to setup network environment to work with VM and get the same tripleo-ci behaviour, want to join ?13:40
EmilienMgfidente: i'm find +2 if we're improving coverage13:40
trownpanda: yes please, I got nonha job working (with hacks) on top of quickstart undercloud, but ha job is causing me issues because network isolation13:40
*** adarazs_brb is now known as adarazs13:41
pandabnemec: whenever you feel comfortable, I have a fresh provisioned undercloud with some vm waiting to be deployed13:42
gfidenteEmilienM, so it's not really a bug13:42
gfidentebut if you want to do netiso and hci those two lines should be uncomented13:42
gfidentewe thought it would be better to uncomment them by default13:42
gfidenteand to land that with the release to not change the default after the release13:43
EmilienMgfidente: ok, fair enough, +213:43
gfidentehehe sorry should have been uncommented from the start13:43
trownpanda: ah looks like we need more than 1 extra network:
trownlooks like we need 6 overcloud networks to use the tripleo-ci method13:44
openstackgerritDmitry Tantsur proposed openstack/tripleo-quickstart: Remove really old cruft from instackenv.json.j2
dtantsurtrown, ^^13:45
trowndtantsur: cool, I guess we will find out if it works for old versions of ironic via CI :)13:45
dtantsurnow we need the same kind of clean up in instack-virt-setup, but I guess it may be harder...13:46
trowndo we?13:47
dtantsurwell, or whatever else generates instackenv.json13:47
*** akshai has joined #tripleo13:47
trownI mean if it doesnt break anything, cosmetic cleanup there seems a bit like lipstick on a bull13:47
dtantsurI particularly adore references to nova-baremetal there :)13:47
dtantsuroh, that's tripleo-incubator:
dtantsurI'm a bit scared to touch this code :D13:48
openstackgerritMerged openstack/tripleo-puppet-elements: Midonet element now complies with version 5.2
openstackgerritMerged openstack/tripleo-common: Add FS_TYPE xfs to RHEL and CentOS images
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Always configure ipv6 address with net-iso
pandatrown: how many do we have now ?13:51
mandrequick question about haproxy, do we enable it for all overcloud deployments, even if there is only one control node?13:51
trownpanda: by default we just make one overcloud network, and one external network, the RDO CI code uses single-nic-vlans to do netiso13:52
hewbroccaI thought we killed all the crap in incubator13:52
trownpanda: it looks like tripleo-ci uses a custom version of multi-nic13:52
trownpanda: testing a run now with 6 overcloud networks to see if it magically will "just work"13:53
openstackgerritDimitri Savineau proposed openstack/tripleo-heat-templates: Added support for pass-through iSER configuration
*** rcernin has joined #tripleo13:53
*** zoli|brb is now known as zoli13:53
*** zoli is now known as zoliXXL13:53
jaosoriormandre: yes13:53
gfidentematbu, marios what is going on with ?13:54
gfidentefrom what I can tell it *will* regenerate the keys on upgrade now13:54
mandrejaosorior: I do see a haproxy_enabled hieradata, does it mean it's optional?13:54
jaosoriormandre: it is. If you configure the overcloud with an external loadbalancer13:54
jaosoriormandre: why would you want to get rid of it?13:55
mariosgfidente: the assumption is it would be set on deploy and stored in the passwords file no? /me trying to finish something before scrum will look more later13:55
mandrejaosorior: just being generally curious :)13:55
mandrejaosorior: also, my overcloud services can't talk to mysql, i think it might be a misconfiguration of haproxy13:56
*** jeckersb is now known as jeckersb_gone13:56
jaosoriormandre: I can work with you on that13:56
jaosoriorwhat's up?13:56
matbugfidente: the goal was to manage those keys like the other password are managed13:57
mandrejaosorior: haproxy logs reported that mysql service was down, and it didn't allow connection13:57
mandrejaosorior: here's part of my haproxy.conf
jaosoriormandre: and I assume that mysql is indeed running, right?13:58
jaosoriormandre: I've seen it go down due to memory issues :/13:58
mariosjistr: bandini ha - i will fix it momentarily as I need to add an exception there for sahara13:58
mariosmatbu: ^13:58
mandrejaosorior: yes it is, and I can connect to it when not via haproxy13:58
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Add a validation-specific Ansible output callback
openstackgerritDmitry Tantsur proposed openstack/tripleo-incubator: Remove really old stuff from generated instackenv.json
bandinimarios: ack13:59
bandinimatbu: do you ever get to the stage of running the major-upgrade-pacemaker step?13:59
dtantsurI'm ready to risk ^^13:59
jaosoriormandre: undercloud or overcloud?13:59
mariosbandini: doing so on 10 packages still tho14:00
bandinimarios: 10 packages?14:00
mariosbandini: like we don't have latest things, even the change to postdeploy noop you hit yesterday14:00
jaosoriormandre: so, the configuration I'm seeing (I just deployed) is this:
trowndtantsur: there is no CI on that excercises that code :)14:00
mariosbandini: i mean osp10 ... upgrading osp9 to osp1014:00
jistrfolks how can i pull this into Newton RC2 for tripleo-common? I don't see a way to do it in the interface, neither to change the importance on it
openstackLaunchpad bug 1609020 in tripleo-common "Servers get rebuilt during a stack update following new image rollout" [Undecided,In progress]14:00
mandrejaosorior: it happens on the overcloud14:00
dtantsurtrown, no CI on tripleo-incubator? Oo14:00
jistrEmilienM: would you know? ^^14:00
mariosbandini: so is a little behind upstream14:00
gfidentematbu, marios agreed they are stored in that file and not recreated14:01
gfidentebut not on upgrade14:01
jaosoriormandre: quite strange that it's lacking some options for you. Did you modify it manually?14:01
trowndtantsur: CI might run on it, but it doesnt run instack-virt-setup, or use that instackenv.json from there14:01
bandinimarios: ah ok sorry I am slower then usual ;)14:01
gfidentebecause from previous deployment those were not dumped in that file, so they *will* actually be regenerated14:01
EmilienMjistr: do you have a patch?14:01
dtantsurtrown, ok, I'll withdraw this patch...14:01
trowndtantsur: but if quickstart CI passes on that change, the incubator one is probably safe14:01
mandrejaosorior: hmmm, I though I deployed using master puppet module14:01
dtantsurone more reason to not use tripleo-quickstart14:01
mandrejaosorior: maybe I have an older version?14:01
jaosoriorit's fine14:01
jistrEmilienM: yea that's the one14:02
jaosoriorit seems that it's the standard nonha config14:02
jaosoriorI just checked on CI14:02
pandadtantsur: hey!14:02
EmilienMjistr: if you feel like we need it, yes..14:02
jaosoriorand what we're deploying on CI for the nonha configuration is just like that too14:02
mandrejaosorior: right, I only have one control node14:02
mariosbandini: heh np bandini ... btw the migration handles the removal of sahara nicely, i mean i went looking for what we had to remove, the constraints and r esource definition14:02
mariosbandini: so all we have to do is stop it being started14:02
jistrEmilienM: upgrades are a no go without it, it rebuilds the whole cloud if we don't have it14:02
shardyjistr: We've been trying to track everything in the main tripleo LP project, so you can just mark the bug as affecting tripleo14:02
shardythen target it to rc214:02
mariosbandini: for the 'remove sahara' case... i mean14:02
bandinimarios: nice!14:02
jistrshardy: ah perfect, thanks14:02
EmilienMjistr: upgrade are automatically rc214:02
matbubandini: nop stuck somewhere on upgrade controller14:02
EmilienMdon't even ask :)14:02
jaosoriormandre: yeah, I deployed ha... that's why it's different14:03
jaosorioralright then14:03
jaosoriormandre: can you pass the log again?14:03
*** jeckersb_gone is now known as jeckersb14:03
mandrejaosorior: haproxy logs show
bandinimatbu: is it bug ? i.e. after networkdeplymend gives SIGNAL_COMPLETE nothing happens?14:03
openstackLaunchpad bug 1626628 in tripleo "M/N Upgrade - major-upgrade-pacemaker times out" [Critical,Triaged]14:03
jaosoriorconnection refused... what the hell14:04
jaosoriormandre: is your undercloud accessible somehow?14:04
matbubandini: yep14:05
*** ramishra has quit IRC14:05
mandrejaosorior: I suppose I can make it accessible to you but I need a moment then14:05
jaosoriormandre: sure14:05
matbubandini: /me looks the LP14:05
bandinimatbu: alrighty at least we're on the same issue as well, I am quite stuck on that atm14:05
*** jprovazn has quit IRC14:06
matbugfidente: if the FSID exist, it won't override, if not it will create one, right ?14:06
gfidentematbu, yes the problem is that on upgrade there is no FSID in the passwords file14:06
*** ramishra has joined #tripleo14:06
mandrejaosorior: the log says connection error when connecting to the mysql backup server, which is supposedly listening on
matbugfidente: yes, but (stupid question) why is a pb ?14:07
matbugfidente: i mean, the upgrade will generate a new FSID14:08
gfidentematbu, yeah the ceph osds are bound to an FSID14:08
mandrejaosorior: is just another address for my overcloud control node14:08
jaosoriormandre: I understood what the log says. It was just weird that it got a connection error14:08
gfidentematbu, so the ceph cluster loose all the data if that changes14:09
*** yamahata has joined #tripleo14:09
jaosoriormandre: anyway, lets check out what's up. Let me know when I can log in14:09
mandrejaosorior: according to galera.conf, mysql binds to overcloud-controller-014:09
mandrejaosorior: and overcloud-controller-0 resolves to, so it's not really surprising that we get a connection error when connecting on
jaosoriormandre: interesting14:12
jaosoriormandre: lets figure out why14:12
jaosoriormandre: I have the same thing. Except that for me it actually resolves to the internal_api network (probably cause of net-iso14:12
matbugfidente: ok, then, if we don't fsid and the stack is not new, we should not generate an uuid. we only handle this uuid on new deployment, do we want to store the FSID in the password file ?14:13
*** pgadiya has quit IRC14:14
jaosoriormandre: found the issue14:14
mandrejaosorior: already? nice :)14:14
gfidentematbu, right, probably adding it into passwords file is a good idea14:15
jaosoriormandre: this is wrong
shardy$package_manifest_name = join(['/var/lib/tripleo/installed-packages/overcloud_controller ...14:16
jaosoriormandre: lets try to fix it now14:16
shardycan anyone point me to where we build the list of packages?14:16
gfidentematbu, but we need some logic to pick it up from the passwords file (and generate if missing) only if stack is not new14:16
mandrejaosorior: what should it be binding on?14:16
matbugfidente: yep14:17
*** Goneri has quit IRC14:17
jaosoriormandre: I'm gonna get it to bind, still to a hostname, but one that's according to it's appropriate network14:17
jaosoriormandre: can you do facter -p $fqdn_internal_api ?14:17
jaosoriormandre: what's the result?14:18
jaosoriorwithout the $14:18
jaosoriorfacter -p fqdn_internal_api14:18
jaosoriormandre: also, can you file a bug?14:18
shardynvm, found it14:18
mandrejaosorior: facter -p fqdn_internal_api returns nothing14:19
*** ccamacho|lunch is now known as ccamacho14:20
openstackgerritMerged openstack/diskimage-builder: Shorten DHCP timeout in dhcp-all-interfaces
shardyDo we still need controller-config.yaml and controller-config-pacemaker.yaml?14:20
shardyseems like we can combine them into a common template as the only difference is the path for the package_manifest provider14:20
jaosoriormandre: ah... true.. that fact is only available in puppet-tripleo14:21
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add Removal of Sahara during upgrade
mandrejaosorior: but there is a internal_api_virtual_ip hieradata if that helps14:22
mariostosky: ^ cant find you in gerrit to add you on review14:22
toskymarios: uuuh14:22
toskybut I'm there14:22
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make mysql bind-address configurable
mariostosky: i am also in mad rush cos have scrum in 5 mins and didn't prep :)14:23
mariostosky: so fyi14:23
toskymarios: I would say that "for $service in " is an unrelated fix, but I would not complain14:23
mandrejaosorior: or even the cloud_name_internal_api hieradata that contains the hostname14:25
mariostosky: ack14:25
toskyno, but it's important to have anyway14:25
toskyotherwise services are not stopped14:25
EmilienMpanda: ipv6 job still timeouting at step514:25
mariostosky: i can split noproblem, just wanted to get it out14:26
mariostosky: will revisit later add a comment thanks14:26
toskyas I said, leaving it to you, no strong push there14:26
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make mysql bind-address configurable
b00tcathi - I'm testing some changes I did on the tripleo-heat-templates manifests and had some syntax error14:27
b00tcatso I SSHed into the overcloud controller and did `puppet apply /var/lib/heat-config/heat-config-puppet/*.pp`14:27
b00tcathowever I'm getting "Error: Could not find data item step in any Hiera data file and no default supplied at blablabla"14:28
toskyapart from that - argh, array manipulated as strings in bash14:28
b00tcatit is not enough to do `puppet apply` ?14:28
shardyb00tcat: we pass a "step" variable to puppet when the puppet apply is done via heat14:28
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Bind MySQL address to hostname appropriate to its network
jaosoriormandre: that should do the trick ^^14:29
shardyyou can hack the step you want into a file in /etc/puppet/hieradata, then do puppet apply if you want for debugging14:29
shardyremember to remove it afterwards ;)14:29
jaosoriormandre: I need a bug number to be able to get it to newton though14:29
openstackgerritMerged openstack/tripleo-heat-templates: Move keystone::auth into service_config_settings
shardyor pass the step variable on the CLI I guess14:29
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu
b00tcatshardy: let me check if I got you - so basically I can hardcode the step "I'm in" on the hieradata, then do puppet apply and everything will go smooth?14:33
*** pkovar has quit IRC14:33
shardyb00tcat: yes, you have to pass the step you want to test to puppet, then it will do the same as if heat applied it14:33
trownpanda: so far so good on just making extra networks to get tripleo-ci netiso working on top of quickstart undercloud, deploy  made it past the network config phase anyways14:34
b00tcatshardy: got it, thanks!14:34
pandatrown: your previously deploys hung on NetworkDeployment step ?14:35
pandaEmilienM: any guess on what is haning on step5 ?14:36
trownpanda: ya when I only added one extra network so there would be an eth2 on the undercloud it hung there14:36
EmilienMpanda: I'm looking14:36
trownpanda: but with 6 overcloud networks, it seems happy14:36
mandrejaosorior: sorry, was on the phone, I'm filling the bug right now14:37
jaosoriormandre: awesome14:37
pandabnemec: :(14:38
bandinipanda: you get something hung on NetworkDeployment step?14:41
pandabandini: yes, but mainly because I'm trying to apply trieplo-ci test environemtn file to a virt env generated by quickstart14:42
*** mah has joined #tripleo14:43
jaosoriorbandini: galera is still using pacemaker, right?14:43
mahHi all, I am using opnfv-apex which is based on tripleo.. but I got an issue stucked at the end (post install configuration) configuring neutron external network14:43
openstackLaunchpad bug 1627060 in tripleo "Can't connect to mariadb in nonha haproxy setup" [Undecided,New]14:44
jaosoriormandre: awesomeness. I'll use that14:44
trownpanda: is the config I used to setup quickstart env with 6 overcloud networks14:44
mahand I tried to do openstack user list , but got stucked too .. stucked at making authentication to httP://ip:5000/v2.0/tokens14:44
mahany help please ?14:45
mandrejaosorior: and I'll try your patch14:45
*** florianf has quit IRC14:45
jaosoriormandre: please do. Only thing is that it's incomplete. I'm not taking into account HA14:46
jaosoriormandre: so I gotta update it. But the nonha part should stay the same14:46
bandinijaosorior: yes14:46
mandrejaosorior: ack14:47
jaosoriorbandini: thanks14:47
bandinijaosorior: ;) np14:47
pandatrown: thanks. I plan o understand a bit more what is expected from networks ..14:48
trownpanda: so far it seems like, just having the interfaces connected14:49
*** Ryjedo has joined #tripleo14:49
trownpanda: since we are using a version of multi-nic and not single-nic-vlans, there is not really anything extra to setup14:49
trownpanda: but I still wouldnt mind sitting in with you and bnemec, networking is my weakest area14:50
pandatrown: do you know if undercloud is still acting as default route ?14:51
*** pkovar has joined #tripleo14:51
openstackgerritRyan Brady proposed openstack/python-tripleoclient: [WIP] Change tripleoclient to use password generation from workflows
trownpanda: ya,
trownpanda: and we set that on the undercloud interface in
jaosoriormandre: note that the t-h-t patch depends on a puppet-tripleo patch14:53
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make mysql bind-address configurable
mandrejaosorior: ohhh I totally missed that14:53
*** florianf has joined #tripleo14:54
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Bind MySQL address to hostname appropriate to its network
*** jlinkes has quit IRC14:56
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Bind MySQL address to hostname appropriate to its network
gfidentemwhahaha, not sure if you have a minute but have any clue why 4.5 is failing ?14:57
mwhahahagfidente: missing facts14:58
mwhahahagfidente: i'll find the specifics and comment14:58
gfidenteso is this something changed in puppet 4.514:58
gfidentethanks btw :)14:59
mwhahahayea in 4 it doesn't like it when vercmp is given an undef or a ''14:59
openstackgerritMartin Mágr proposed openstack/instack-undercloud: Revert "Deploy Monitoring on the undercloud with Puppet"
gfidentemwhahaha, eh I was looking into that but we seem to do versioncmp with operatingsystemmajrelease15:00
gfidentecan that ever be '' ?15:00
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton
mwhahahayea that's a fact so in rspec that has to be provided15:00
mwhahahanot sure why your change is triggering that15:00
*** milan has quit IRC15:01
mwhahahamight have been an apache change that broke stuff15:01
gfidenteright doesn't seem related to the change itself15:01
mwhahahayea let me propose a fix and i'll rebase that change on top of it15:01
mwhahahacause we'll probably have to backport the fix15:02
*** jistr is now known as jistr|call15:03
jaosoriorgfidente, bandini: Do you guys know if galera can take hostnames in the gmcast.listen_addr parameter?15:03
bandinidciabrin: ^15:04
bandinijaosorior: I'd be surprised if it could not, but maybe Damien knows for sure?15:04
bnemecEmilienM: panda: I just got a successful ping test on an ipv6 overcloud:
jaosoriorbandini: the documentation is a bit lacking15:04
EmilienMbnemec: woot15:04
pandabnemec: any significant changes from what we're testing ?15:05
EmilienMbnemec: nice :)15:05
EmilienMyeah, why does it timeout in CI?15:05
*** abehl has quit IRC15:05
EmilienMbnemec: are you running the same envs as in ?15:05
mahhi all, openstack commands stucks always here --> Making authentication request to
mahanyhelp ?15:05
dciabrinbandini, jaosorior checking at once15:06
bnemecpanda: EmilienM: I had to bump my overcloud memory to 7 GB.  I kept OOMing at 6 yesterday.15:07
bnemecThat's likely part of why CI keeps failing on thos patches.15:07
EmilienMbnemec: why doesn't it fail on regular ha job?15:07
EmilienMor at least not much15:08
EmilienMipv6 always timeout15:08
openstackgerritRyan Brady proposed openstack/tripleo-common: Port password generation from tripleoclient to tripleo-common
openstackgerritMartin Mágr proposed openstack/instack-undercloud: Revert "Deploy Monitoring on the undercloud with Puppet"
EmilienMbnemec: can you double check panda's patch to make sure we use the right environments?15:08
bnemecEmilienM: I've seen quite a few failures on the regular ha job too.15:08
bnemecIt's possible I have more cpus on my overcloud vms too, which would make it worse.15:08
bnemecYeah, I have 2 cpus per overcloud node, CI only has 1.15:10
bnemecBut I saw similar hangs when I had 6 GB overcloud nodes too.15:10
mwhahahagfidente: yea puppet ceph tests are hosed, i'll propose a fix15:10
bandinigfidente: here is my last finding15:11
bandinimatbu: ^15:11
EmilienMdprince: I see you assigned on
openstackLaunchpad bug 1620595 in tripleo "When Heat API services run on a different node than controller the heat-dbsync fails due to incorrect mysql permissions" [High,In progress] - Assigned to Dan Prince (dan-prince)15:12
EmilienMdprince: do you have WIP about the db_sync issue mcornea reported?15:12
*** leanderthal is now known as leanderthal|afk15:12
dprinceEmilienM: I will look at it. now WIP yet though but it will be similar to the keystone issue I think15:15
dprinceno WIP yet15:15
fultonjpuppet ceph tests are hosed, mwhahaha to propose fix15:16
dprinceEmilienM: we'll have to change the puppet manifests for this I think too15:16
dciabrinjaosorior, it looks like it can. I can bootstrap an ipv4 cluster with hostname in it, galera version 25.3.5-715:17
matbubandini: works, if you don't make the BlockStorageUpgradeDeployment ?15:17
*** saneax is now known as saneax-_-|AFK15:17
openstackgerritRyan Brady proposed openstack/tripleo-common: Separate Template Processing From Create/Update Plan
dciabrinbandini, jaosorior what's the issue with having ip values?15:17
jaosoriordciabrin: no issue15:18
jaosoriordciabrin: hope there's no issue with using hostnames either :D15:18
dciabrinjaosorior, haha :)15:18
*** rcernin has quit IRC15:19
*** lucasagomes is now known as lucas-hungry15:20
*** zaneb has quit IRC15:23
mahhi all, openstack commands stucks always here --> Making authentication request to
*** pkovar has quit IRC15:25
*** aufi has quit IRC15:25
bandinimatbu: ok found a workaround for the issue ;)15:25
bandiniI don't *why* it works, but we'll figure that out eventually15:26
*** jistr|call is now known as jistr15:26
*** bana_k has joined #tripleo15:27
*** jistr is now known as jistr|biab15:28
matbubandini: lol cool15:29
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart: Update minimal  create docs payload to use overcloud prep roles
*** ebarrera has quit IRC15:32
*** rajinir has joined #tripleo15:32
*** chem` has joined #tripleo15:33
openstackgerritmathieu bultel proposed openstack/python-tripleoclient: Keystone credentials needs to be set with the overcloud password
*** chem has quit IRC15:34
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: WIP Work around stuck get_param calls during major-pacemaker-upgrade
*** jistr|biab is now known as jistr15:36
*** mcornea has quit IRC15:38
gfidentebandini++ on
zoliXXLhave a good weekend15:41
gfidentemarios, jistr ^^ sounds like we want to tag tag tripleo/rc2 and merge quickly15:42
bandinigfidente: note that we have many more instances
gfidentebandini, right, of stuff we don't test in CI indeed :)15:43
*** zoliXXL is now known as zoli|gone15:43
gfidentethey all need fixing15:43
bandinigfidente: ack I will submit v215:43
bandiniI think a heat bug is in order as well15:44
*** zoli|gone is now known as zoli_gone-proxy15:44
bandiniit could barf instead of timing out15:44
gfidenteyou mean syntax validation?15:44
jistrgfidente, bandini: i'm a bit confused maybe, i thought we should be using get_attr for these?
bandinidon't you think?15:44
gfidentejistr, no we get list of servers as param there15:44
jistrah true just looked it up in docs...15:45
*** chem`` has joined #tripleo15:45
bandiniI will make a v2 after dinner. need to tackle the kids now15:46
gfidentebandini, it's friday!15:46
jistri know get_param is for things passed in, get_resource is for resources, but never sure about get_attr :)15:46
bandinigfidente: :) amen to that15:46
*** chem` has quit IRC15:46
jistrif get_resource behaved similarly as get_param then perhaps we wouldn't need get_attr at all15:46
shardyjistr: get_attr always gets outputs from resources :)15:46
shardyjistr: where it gets confusing I guess is that outputs from nested stacks are referenced as attributes of the owning resource (in the parent stack)15:47
gfidentebut the 'resource' is local to the parent in that case though15:48
gfidentelocal went in quotes , not resource15:48
shardyYeah, the resource is a reference to the nested stack, so you can see it's outputs (as attributes)15:48
shardyyou can't replace get_attr with get_resource, because get_resource always returns the ID of the resource15:48
shardye.g the nova server ID15:48
*** absubram has joined #tripleo15:48
*** yamahata has quit IRC15:49
shardyI suppose you could extend it to allow {get_resource: [foo, some_attr]}15:49
*** tremble has quit IRC15:49
jistr^ that's exactly what i meant :)15:49
shardyjistr: aha15:49
jistrand then get_attr is superfluous15:49
*** flepied has quit IRC15:49
* shardy is running out of brain power, it's friday ;)15:49
jistrthat's what confuses me always... we have get_param which is powerful enough to fetch the outputs too, but get_resource isn't, and instead of that functionality we have get_attr15:50
jistrand i never remember which one is the more powerful one :)15:50
*** dtantsur is now known as dtantsur|afk15:50
shardyjistr: Yeah, well get_param and get_attr both support path based lookups15:51
*** chem`` has quit IRC15:51
shardyso you have a valid point :)15:51
gfidentebut in that yaml we had to use _param15:52
mariosgfidente: reading back15:52
gfidenteso jistr is complaining on friday15:52
mariosgfidente: for which one? manila?15:52
pandashardy: so, ben was able to make a test pass that is failing without apparent reason in CI just increasing the amount of RAM on the overcloud, so the ram problem may affect overcloud nodes too ...15:53
mariosgfidente: why the rush? i mean yes we want to land it but is upgrades related so has exception. it still has WIP?15:53
gfidentemarios, it blocks upgrade attempts15:53
jistrtrue maybe we could un-wip :D15:54
mariosbandini: is this the fix for the heat timetout15:54
dtrainorAre any and all resources and parameters changeable using *ExtraConfig using t-h-t?  Should that be the expectation when poking and prodding templates?15:54
gfidentedon't say you took me seriously there15:54
jistrgfidente: yes i hear you :D15:54
gfidentebecause I don't expect you to15:54
bandinimarios: yes15:54
jistri'm complaining every day :D15:54
bandinijistr: that's the spirit ;)15:54
gfidentethat's what I meant15:54
jistrBNaaS bad news as a service15:54
mariosbandini: this is fantastic cool stuff15:55
gfidentenow nobody please15:55
gfidentego talking about conferencing as a service15:55
*** myoung is now known as myoung|lunch15:55
mwhahahagfidente, fultonj: proposed fixes for puppet-ceph and rebased the osd change so it should pass now15:56
gfidentemwhahaha, which one is it? was it a backward compat issue of 4.515:56
gfidenteor something actually wrong in puppet-ceph we didn't notice before?15:56
mwhahahagfidente: puppetlabs-apt release 2.3.0 that now uses lsbdistrelease fact which we were not defining in the tests15:57
mwhahahait's not actually a functional issue just tests15:57
Jokke_any chance to get review eyes on this ?15:57
bnemecMan, if we get BNaaS I can retire early.15:57
bnemecThat is Ben Nemec as a Service, right? ;-)15:57
*** bnemec is now known as beekneemech15:58
beekneemechAnd hey, it's Friday!15:58
*** jpena is now known as jpena|off15:58
*** Goneri has joined #tripleo15:58
*** jpena|off is now known as jpena15:59
Jokke_beekneemech: indeed :(15:59
*** ebarrera has joined #tripleo16:04
*** dprince has quit IRC16:05
gfidentenow guys16:06
gfidenteI will miss you during the weekend16:07
gfidentebut I think I am leaving for today16:07
gfidentethough I migh send you some review requests on a late night saturday16:07
gfidentewho knows16:07
gfidentepretend everything is urgent for rc216:07
gfidenteto add some last minute bugs16:07
gfidenteas I am used to16:07
hewbroccagfidente: yessss moar bugz16:09
mariosgfidente: have a good one faidentee16:09
*** rasca has quit IRC16:12
*** hewbrocca is now known as hewbrocca-afk16:12
*** bana_k has quit IRC16:14
*** bkopilov has quit IRC16:14
*** ohamada has quit IRC16:15
*** fultonj has quit IRC16:15
jaosoriormandre: so my commit fails cause the hostnames used in the multinode gate get too long :(. I'll fix it on monday.16:16
mandrejaosorior: right now, I'm having issue with the upload-puppet-module script that doesn't copy anything so I couldn't actually test your patch16:17
*** gfidente has left #tripleo16:18
*** gfidente has joined #tripleo16:18
*** jistr is now known as jistr|afk16:18
jaosoriormandre: so upload-puppet-module script is broken it seems16:19
jaosoriorI worked around it by deleting this from upload-swift-artifact:16:19
EmilienMgfidente: lol16:19
gfidenteEmilienM, oh you you16:19
mandrejaosorior: yeah, I'm currently looking into it16:19
mandrejaosorior: good to know16:19
gfidentewhy you hate me for the puppet openstack integration comment16:20
jaosoriorI deleted that one16:20
jaosoriorand only used the second one16:20
EmilienMgfidente: where?16:20
jaosoriorgetting it off the if16:20
gfidenteI get it's different purpose from tripleo-ci16:20
gfidenteand politically not nice to share ci16:20
jaosoriormandre: anyway, that should get it working16:20
jaosoriorI gotta go now16:20
jaosoriorhappy weekend everyone!16:20
mandrejaosorior: cool, will try it16:20
gfidentebut we'd be repeating lots of similar studd16:20
mandrejaosorior: enjoy16:20
gfidenteEmilienM, anyway, we can talk another day16:20
*** jaosorior has quit IRC16:21
EmilienMgfidente: oh I see16:21
EmilienMgfidente: I agree it would be nice to share things but I don't see how here16:22
EmilienMgfidente: the scenario job is still a good start :)16:22
gfidentehow is the problem agreed16:22
openstackgerritRyan Brady proposed openstack/tripleo-common: Port password generation from tripleoclient to tripleo-common
*** gfidente has quit IRC16:25
EmilienMwe have 2 patches to help with puppet tripleo unit testing, please review them and
*** yamahata has joined #tripleo16:29
*** lucas-hungry is now known as lucasagomes16:29
openstackgerritMerged openstack/puppet-tripleo: certmonger: improve orchestration for puppet4
EmilienMayoung: taking it here since it's about tripleo. If you plan to do the tripleoclient thing, make sure upgrade work. My initial patch was not working for upgrades, I missed it. See
openstackLaunchpad bug 1626422 in tripleo "Failed to validate nested template: Property error: resources[10].properties: Property KeystoneCredential0 not assigned" [Critical,In progress] - Assigned to mbu (mat-bultel)16:33
EmilienMmatbu is working on it:
EmilienMso I guess we'll need this patch before doing fernets16:33
ayoungEmilienM, good to know.  I was thinkg more like 1. Make sure we could enable Fernet for a new install and then 2.  make it default, to include upgrades.  Is that a reasonable path16:34
ccamachoGuys enjoy the weekend!!!!16:35
ayoungEmilienM, also, limiting it to 2 credentials might be problematic.16:35
ayoungEmilienM, I was thinking through this.  Credentials are different from Fernet Keys.  Fernet is always 2 (or so) but credentials might be an evergrowing list16:36
EmilienMayoung: I see16:37
ayoungEmilienM, there is always one active credential that you use to encrypt new passwords, but for old ones, you need to make sure that they are all decryptable.16:37
EmilienMayoung: again, it was a first iteration16:37
ayoungI'm going to discuss that in upstream Keystone at the tsummit16:37
EmilienMI guess we need to improve it in Ocata16:37
ayoungI think we want a tool in keystone-manage to update all the passwords to the latest keys so we can dump old ones16:37
EmilienMwhat is worries me is how to transfer the key to all keystone servers/containers16:38
EmilienMit would be cool if we could use swift as a backend storage16:39
EmilienMdirectly in keystone16:39
EmilienMinstead of local file16:39
ayoungEmilienM, I'm discussing that, too.  It will be similar for both Fernet and credentials16:39
*** myoung|lunch is now known as myoung16:41
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu
ayoungEmilienM, I can see a solution that uses PKI, but it will take some coding.  It would also take each of the Keystone instances producing an X509 cert that would be used as part of an encryption process. I want to discuss with the Keystone team, and maybe we can come up with something less heavyweight16:41
EmilienMin any way, I don't think storing certificates on the machine is good16:42
EmilienMwe should use external storage to make it accessible from all keystone services16:42
EmilienMswift has proven to be stable and secured, why not using it?16:42
*** b00tcat has quit IRC16:43
ayoungEmilienM, So, the way that PKI works is that each machine has to have its own secure storage for Private Keys.  They never leave that machine. But the certificate can and should leave the machine, as it only has public information.16:43
ayoungThat is actually the case for far larger uses than Keystone, to include all of the HTTPS/TLS work that jaosorio is doing.16:44
*** paramite has quit IRC16:44
ayoungIt makes the workflow really painful, though, and that is why I don't want to do it if we don't have to16:45
EmilienMI'm afk for lunch, we can talk later16:45
ayoungEmilienM, the problem with Swift is that to access it, the overcloud keystone server needs a credential that is accepted in the undercloud16:45
ayoungWe don't have that today.  So the alternative is insecure16:46
*** fultonj has joined #tripleo16:47
*** ccamacho is now known as ccamacho|out16:49
mahhi all, at deployment opnfv-apex which is based on tripleo ,, the post instal configuration stuck at Configuring Neutron external Network ... any solutions ?16:50
trownpanda: beekneemech, should IPv6 in CI be using instead of the packaged net-multiple-nics-v6.yaml
*** dmacpher is now known as dmacpher-afk16:51
beekneemechtrown: Yes, see
beekneemechWe just never updated the updates job for ovb.16:52
trownbeekneemech: awesome giving that a go16:53
beekneemechUnfortunately all of the ipv6-based jobs are failing for some reason right now. :-/16:53
*** jpena is now known as jpena|off16:54
*** derekh has quit IRC16:56
*** abregman has joined #tripleo16:56
trownbeekneemech: k, I am just testing the IPv6 net-iso part (not updates), and will provide feedback on the patch16:56
*** bana_k has joined #tripleo16:56
*** jistr|afk is now known as jistr16:59
*** ccamacho|out has quit IRC17:05
*** trown is now known as trown|lunch17:06
openstackgerritRyan Brady proposed openstack/tripleo-common: Port password generation from tripleoclient to tripleo-common
*** athomas has quit IRC17:22
*** ebarrera has quit IRC17:27
*** jpich has quit IRC17:29
*** tosky has quit IRC17:35
*** akshai has quit IRC17:35
*** florianf has quit IRC17:36
*** akshai has joined #tripleo17:37
jristhonza: do you have a trick for testing this or have you actually gotten a full deploy to work17:37
honzajrist: i have a full deploy working and just refresh the page when i change the code, sorry17:38
honzajrist: it's using the heat api so i don't think you can really fake it17:38
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: get_param calls with multiple arguments need brackets around them
jristhonza: I'm jealous.17:42
jristhonza: got a screenshot?17:42
honzajrist: one sec17:43
*** jbadiapa has quit IRC17:44
openstackgerritMerged openstack/tripleo-common: Remove the old, deprecated Mistral action names
openstackgerritMerged openstack/instack-undercloud: Make puppet-stack-config.pp Puppet 4 compliant
openstackgerritMerged openstack/tripleo-heat-templates: Remove hard-coded roles in EnabledServices output
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove the get_hiera_key function
jristgood stuff17:47
jristno validations!17:47
*** florianf has joined #tripleo17:47
honzajrist: *shrug*17:48
jristwait a minute17:48
jristhow'd you get the deployment to pass17:48
jristif you have no nodes17:48
openstackgerritRyan Hallisey proposed openstack/tripleo-common: Remove references to overcloud-without-mergepy
honzaI think it was deployed via CLI17:49
* jrist grumbles17:49
*** egafford has joined #tripleo17:49
*** flepied has joined #tripleo18:02
*** trown|lunch is now known as trown18:06
*** rhallisey has quit IRC18:08
EmilienMcan someone review this quick puppet4 fix?
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu
*** florianf has quit IRC18:12
openstackgerritJohn Eckersberg proposed openstack/tripleo-puppet-elements: Add puppet-qdr module
openstackgerritJohn Eckersberg proposed openstack/tripleo-puppet-elements: Add qpid-dispatch-router to overcloud-controller element
dtrainorI've been battling with a deployment that just stalls at "Running /var/lib/heat-config/hooks/puppet < /var/lib/heat-config/deployed/51333d0a-cbc9-4c9a-8f68-9799d4bdf302.json" for both my Compute and my Controller nodes.18:27
dtrainorI found this and this but I can't see how they would apply18:28
openstackLaunchpad bug 1488366 in tripleo "TripleO Deployments get stuck sometimes until a timeout is reached" [High,Fix released]18:28 bug 1262189 in openstack-puppet-modules "Overcloud Deployment fails: either hangs or raises resources.ComputeNodesPostDeployment.resources.ComputePuppetDeployment.resources[0]" [High,Closed: errata] - Assigned to ichavero18:28
dtrainorThe hanging does in fact happen on the ComputeNodesPostDeployment and ControllerNodesPostDeployment resources.18:29
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: scenario001: deploy Ceph
*** rcarrillocruz has quit IRC18:40
*** rcarrillocruz has joined #tripleo18:45
*** jeckersb is now known as jeckersb_gone18:49
beaglesdtrainor: I ran into that once... but I think I had some things out of sync. I couldn't see anything obviously wrong, it just wasn't going anywere.18:56
beaglesor anywhere rather18:56
dtrainori may have stumbled upon something here18:56
beaglesdtrainor: it was also with some stuff I was working on. I rebased, updated my images and then it worked...18:57
beaglesdtrainor: not at all helpful I know18:57
dtrainorhehe no worries18:58
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: Composable Mistral services
dtrainori think it has to do witht he order in which templates are applied, coupled with a (seemingly) infinite timeout18:59
openstackgerritMerged openstack/puppet-tripleo: Add support for rabbit hosts to mistral
openstackgerritMerged openstack/puppet-tripleo: Add FQDN testcase in swift proxy profile rspec tests
*** david-lyle_ has joined #tripleo19:10
EmilienMbeekneemech, slagle: can you guys review and please? easy patches19:12
EmilienMbeekneemech: also that one, if the format is ok for you19:12
*** david-lyle has quit IRC19:13
beekneemechEmilienM: I'm curious, why use tee when you're going to throw away stdout anyway?19:14
EmilienMbeekneemech: i don't have an answer, I was using sudo in Puppet CI and copy pasted this command. We could also redirect I guess19:14
beekneemechEmilienM: Ah, copy pasta from sudo makes sense.  I don't really care, I was just wondering if I missed something.19:15
EmilienMbeekneemech: I'm wondering why we had this csv format19:15
EmilienMis it something we can exploit?19:16
* EmilienM afk19:17
beekneemechEmilienM: No idea.  I assumed we copied whatever devstack was doing with dstat.19:17
*** r-mibu has quit IRC19:17
*** r-mibu has joined #tripleo19:17
*** abregman is now known as abregman|afk19:18
openstackgerritMerged openstack/puppet-tripleo: Add in rspec-puppet-facts support
openstackgerritMerged openstack/python-tripleoclient: Stop plan creation when container exists
slagleEmilienM: does the FF / rc bugs only, not apply to puppet-tripleo?19:36
slagleor is puppet-tripleo already branched?19:36
*** jeckersb_gone is now known as jeckersb19:41
*** absubram has quit IRC19:41
mwhahahano i don't believe it's branched yet19:42
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu
*** akrivoka has quit IRC19:55
*** absubram has joined #tripleo19:56
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu
*** absubram_ has joined #tripleo19:57
*** absubram has quit IRC20:01
*** absubram_ is now known as absubram20:01
*** david-lyle_ is now known as david-lyle20:06
*** lucasagomes is now known as lucas-afk20:09
*** absubram has quit IRC20:10
*** rajinir has quit IRC20:15
*** akshai has quit IRC20:15
*** akshai has joined #tripleo20:20
*** jayg is now known as jayg|g0n320:23
*** mburned is now known as mburned_out20:38
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Add FixedIPs parameter to from_service.yaml
openstackgerritMerged openstack-infra/tripleo-ci: dstat: improve output to track high cpu process
openstackgerritMerged openstack/puppet-tripleo: Switch puppet-tripleo to use puppet-openstack_spec_helper
*** trown is now known as trown|outtypewww21:09
*** kberger has quit IRC21:13
*** kberger has joined #tripleo21:13
*** pradk has quit IRC21:14
*** thrash is now known as thrash|g0ne21:21
*** myoung is now known as myoung|afk21:21
*** coolsvap has quit IRC21:22
openstackgerritAlex Schultz proposed openstack/puppet-tripleo: Add aodh profile rspec testing
*** abregman|afk has quit IRC21:31
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Include ceilometer in swift proxy pipeline
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo: Add swift proxy for ceilometer middleware
*** cdearborn has quit IRC21:45
*** rlandy has quit IRC21:49
*** fultonj has quit IRC21:52
*** yamahata has quit IRC21:57
*** yamahata has joined #tripleo21:58
*** kberger has quit IRC22:02
*** jrist has quit IRC22:08
*** akshai has quit IRC22:25
*** Ryjedo has quit IRC22:51
*** jrist has joined #tripleo23:00
openstackgerritSarath Kumar Sankaran Kutty proposed openstack/os-net-config: Add bonding_options attribute to NFVSwitch
*** akshai has joined #tripleo23:03
*** jrist has quit IRC23:03
*** jrist has joined #tripleo23:04
*** jrist has joined #tripleo23:04
*** saneax-_-|AFK is now known as saneax23:18
*** abehl has joined #tripleo23:37
*** akshai has quit IRC23:41
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Add ssh-keygen systemd service to debian / ubuntu

Generated by 2.14.0 by Marius Gedminas - find it at!