Wednesday, 2016-08-03

*** rlandy has quit IRC00:12
*** bana_k has quit IRC00:20
*** bana_k has joined #tripleo00:31
*** sarath has quit IRC00:34
*** apetrich has quit IRC00:40
*** apetrich has joined #tripleo00:42
*** dprince has quit IRC00:44
*** limao has joined #tripleo00:45
*** limao has quit IRC00:47
*** limao has joined #tripleo00:47
*** limao has quit IRC01:00
*** limao has joined #tripleo01:00
*** ramishra has quit IRC01:04
*** bana_k has quit IRC01:10
*** akshai has joined #tripleo01:23
*** gfidente has quit IRC01:24
*** akshai has quit IRC01:35
*** saneax is now known as saneax_AFK01:37
*** fzdarsky_ has joined #tripleo01:58
*** fzdarsky has quit IRC02:02
*** coolsvap has joined #tripleo02:19
*** ramishra has joined #tripleo02:21
*** tzumainn has quit IRC02:26
*** ramishra has quit IRC02:26
*** ramishra has joined #tripleo02:28
*** yamahata has quit IRC02:28
*** r-mibu has quit IRC03:24
*** r-mibu has joined #tripleo03:24
*** Nakato has quit IRC03:27
*** colonwq is now known as colonwq_afk03:32
*** Nakato has joined #tripleo03:32
*** ayoung has quit IRC03:48
*** links has joined #tripleo04:10
*** masco has joined #tripleo04:29
*** julim has quit IRC05:02
*** jaosorior has joined #tripleo05:03
*** yamahata has joined #tripleo05:05
*** saneax_AFK is now known as saneax05:09
openstackgerritMerged openstack/puppet-tripleo: Run local CA trust before haproxy deployment
jaosoriormarios: Hey dude, if you get a chance to give the undercloud certmonger review another check Here's a commit that actually uses the code and it's passing CI
*** mbound has joined #tripleo05:16
*** bana_k has joined #tripleo05:19
*** yamahata has quit IRC05:19
*** mbound has quit IRC05:21
*** bana_k has quit IRC05:22
*** mburned has quit IRC05:27
*** liverpooler has joined #tripleo05:28
*** rcernin has joined #tripleo05:28
*** mburned has joined #tripleo05:30
*** liverpooler has quit IRC05:30
*** liverpooler has joined #tripleo05:31
mariosjaosorior: ack05:34
jaosoriormarios: Anything needing review on your side?05:35
mariosjaosorior: do not worry i will hassle you at the apropriate time ;)05:36
jaosoriormarios hahaha alright then, no problem05:37
*** fragatina has joined #tripleo05:45
*** fragatina has quit IRC05:51
*** fragatina has joined #tripleo05:52
*** julim has joined #tripleo05:57
bandinimorning folks06:00
*** aufi has joined #tripleo06:03
openstackgerritBabu Shanmugam proposed openstack/tripleo-heat-templates: OVN heat templates
*** oshvartz has joined #tripleo06:10
*** leanderthal|afk is now known as leanderthal06:12
jaosoriorbandini: que onda wei06:13
*** karthiks|afk is now known as karthiks06:13
bandinijaosorior: ;) todo bien y tu?06:13
bandinistill looking into that aodh failure, it's a bit weird06:14
jaosoriorbandini: service_manage is set to false. Might it be an issue with the puppet manifest?06:14
bandinidunno, still looking06:15
*** florianf has joined #tripleo06:23
*** mburned has quit IRC06:24
*** bana_k has joined #tripleo06:27
*** mburned has joined #tripleo06:30
*** hewbrocca-afk is now known as hewbrocca06:31
*** bootsha has joined #tripleo06:34
*** milan has joined #tripleo06:36
*** pgadiya has joined #tripleo06:36
*** panda|Zz is now known as panda06:44
*** tesseract- has joined #tripleo06:45
*** paramite has joined #tripleo06:48
*** tremble has joined #tripleo06:57
*** tremble has joined #tripleo06:57
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Refactor endpoint env variable population
*** jbadiapa has joined #tripleo07:05
*** jaosorior is now known as jaosorior_brb07:15
*** jpena|off is now known as jpena07:16
*** zoli_gone-proxy is now known as zoliXXL07:19
*** ohamada has joined #tripleo07:19
*** pcaruana has joined #tripleo07:21
*** tremble has quit IRC07:22
*** links has quit IRC07:23
*** mcornea has joined #tripleo07:24
*** hewbrocca is now known as hewbrocca-afk07:25
*** tremble has joined #tripleo07:26
*** tremble has joined #tripleo07:26
*** anshul has joined #tripleo07:27
*** ebarrera has joined #tripleo07:29
*** jpich has joined #tripleo07:35
*** tremble has quit IRC07:36
*** links has joined #tripleo07:36
*** tremble has joined #tripleo07:40
*** ebalduf has quit IRC07:41
*** bana_k has quit IRC07:45
*** hewbrocca-afk is now known as hewbrocca07:52
*** Goneri has joined #tripleo07:54
hewbroccaCould we get a final ack for paramite on please?07:55
*** Goneri has quit IRC07:56
*** Goneri has joined #tripleo07:56
*** devvesa has joined #tripleo07:56
*** gfidente has joined #tripleo07:57
*** gfidente has quit IRC07:57
*** gfidente has joined #tripleo07:57
*** shardy has joined #tripleo07:59
*** jistr is now known as jistr|training07:59
*** radek_ has quit IRC08:01
openstackgerritJulie Pichon proposed openstack/tripleo-common: Mistral workflow for creating a deployment plan from JSON input
*** dsariel has joined #tripleo08:02
gfidenteshardy, fwiw, we still didn't get to the bottom of the mitaka/ci/tripleoclient failure08:05
gfidenteyesterday sshnaidm was looking into that as well and he was able to deploy with the RPM08:05
gfidentewonder if we aren't using tripleoclient from a virtualenv08:06
gfidenteas it's only failing in ci08:07
*** coolsvap is now known as coolsvap_08:08
gfidenteso... still going through tripleo-ci looking for possible causes08:08
*** zoliXXL is now known as zoli|brb08:08
*** limao_ has joined #tripleo08:10
shardygfidente: Hmm, Ok.  That is strange, as you can see the RPM get installed :(08:11
gfidenteyeah and the RPM is good08:11
shardygfidente: maybe we could ask someone with CI access to take a look when the test is actually running later08:11
shardyor is there anything we can push as a WIP tripleo-ci patch which proves the version?08:12
*** jaosorior_brb has quit IRC08:12
*** limao has quit IRC08:13
*** jaosorior_brb has joined #tripleo08:13
gfidenteshardy, well I could try to print the plugin version in overcloud_deploy08:14
gfidentebut to do so I'd have to use a depends_on08:14
gfidentein which case we will *want* a non-rpm version of tripleoclient08:14
gfidentewell I could trigger this to see if the tht submission passes though08:15
sshnaidmgfidente, shardy I ran a patched CI version with discovering version of tripleoclient, I didn't find anything other than installed08:16
sshnaidmalso didn't find installation of tripleoclient within the scripts other than "yum install", and of course delorean building08:17
gfidentesshnaidm, hey, so the same version which passes locally, doesn't work in ci?08:17
shardysounds like we do need access to a CI testenv to debug this08:18
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient: DO NOT MERGE: test tripleoclient in mitaka/ci
sshnaidmgfidente, not really, I just tried tripleo-ci in master08:18
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: We don't need to set a default for the CephX keys and cluster FSID
*** oshvartz has quit IRC08:19
sshnaidmgfidente, tripleo-ci doesn't have branches, should be same for all08:19
*** snecklifter has joined #tripleo08:20
shardywell we do use different codepaths in for stable branches08:20
gfidentesshnaidm, right but yesterday you tested the tripleoclient RPM to deploy the overcloud, correct?08:20
sshnaidmgfidente, maybe there is possible to do a few dependent patches - tripleoclient with version printing, tripleo-ci patch which run client to detect the version, and then the tht patch for mitaka, if all of them will run, maybe we'll discover something?08:20
sshnaidmgfidente, yes, it succeeded08:21
gfidentesshnaidm, yeah I tried the RPM and worked for me too08:21
*** oshvartz has joined #tripleo08:21
gfidentesshnaidm, so what I did for now is just pushing a fake tripleoclient update to stable/mitaka and make tht depends on it08:21
gfidenteif we checkout tripleoclient from stable/mitaka branch for the depends_on it should be overriding any local version and the tht submission pass08:22
gfidentebut let's add the version number too08:23
*** jaosorior_brb is now known as jaosorior08:23
sshnaidmgfidente, the version number is just for being sure we use particular client, not something installed in venv08:24
gfidentesshnaidm, yep, ideally we want the git commit would08:25
sshnaidmgfidente, you will see in logs when it's built by delorean08:26
*** kbyrne has quit IRC08:29
shadowerjtomasek: ping08:29
jtomasekshadower: pong08:30
shadowerjtomasek: question about the legacy API+UI: can you run a group of validations at once? And if so, how is that done? Does the UI run all validations individually or does it issue 1 API call to run them all?08:31
shadowerI've noticed something in the way the legacy API handles group validations and I wonder if it's going to be an issue08:31
*** fzdarsky_ is now known as fzdarsky08:32
* jtomasek checks08:32
*** dculik has quit IRC08:33
jtomasekshadower: GUI uses /stages/${stageId}/run08:33
shadowerjtomasek: right, thanks08:34
shadowerjtomasek: that would be a problem, but it turns out, the API just runs all the validations individually for that request so we're good08:34
jtomasekshadower: ok08:34
* shadower highfives himself from 6 moths ago08:34
shadowerjtomasek: and fwiw it's good that the UI uses that API call -- that's why we put it there08:35
snecklifterhi folks, I've managed to get this passing CI, any chance of a review:
snecklifterI'm not the owner but have a vague interest in it getting committed, thanks08:35
jtomasekshadower: haha08:36
shardysnecklifter: sorry, we can't merge that in it's current state, it needs to be refactored into the composable services format, which I think jaosorior already commented on08:37
*** noshankus has joined #tripleo08:40
*** zoli|brb is now known as zoli08:42
*** zoli is now known as zoliXXL08:42
*** electrofelix has joined #tripleo08:43
shardysnecklifter: let me know if any help is needed, I can probably cut/paste it pretty quickly if someone else is willing to test the resulting patch08:45
noshankusHi guys - I'm trying to run through a http proxy... it seems to be failing at the stage while trying to resolve pbr: "Download error on [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:765) -- Some packages may not be found!" I have set env vars for http_proxy and https_proxy (both pointing to a _http_ address) - any ideas?08:45
shardynoshankus: have you tested other requests which use https_proxy to confirm it's not a proxy issue?08:47
noshankusYep - for example, the git clone beforehand worked: "git clone /root/.quickstart/tripleo-quickstart"08:47
shardynoshankus: but how about e.g a pip install, as that appears to be where it's failing?08:48
noshankusI can also wget both http and https urls successfully... if I run the command "python install egg_info --egg-base /root/.quickstart" manually... it seems to work08:48
*** dculik has joined #tripleo08:48
*** radek_ has joined #tripleo08:49
*** bootsha has quit IRC08:49
noshankusYep - pip install works...08:49
shardynoshankus: Ok, may have to ask trown|outtypewww or larsks later when they come online, not sure what else to suggest08:49
*** jistr|training is now known as jistr08:49
jaosoriorhey guys, more or less how long does it take for a change in a component (in my case mistral) to be taken into account in the tripleo gates? I need a fix that was merged there to verify if a tripleo patch works08:50
noshankusI'm using latest CentOS-7-x86_64-Everything-1511.iso and latest from master...08:50
gfidenteshardy, sshnaidm I might have found the issue with the tht change08:50
noshankus@shardy - ok thanks :)08:51
shardyjaosorior: whenever current-tripleo gets promoted08:51
shardyjaosorior: if you need to test it before then, you can post a WIP patch to tripleo-ci that adds the package to the delorean-current whitelist in then use a Depends-On08:52
sshnaidmgfidente, cool! what is it?08:52
shardygfidente: nice!08:52
gfidenteit's not nice08:52
gfidenteyou'll want to kill me08:52
gfidentelet me update08:52
jaosoriorshardy: Alright, will check that out. Thanks man08:52
* shardy waits to see the diff08:52
jaosoriorjistr: Hey dude, if you get time, could you check out the certmonger patch for the undercloud?  this other commit is actually testing it in CI
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient: Generate CephX keys and FSID regardless of CephStorageCount
sshnaidmare any DIB people here?08:55
jistrjaosorior: ack will review that08:55
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: We don't need to set a default for the CephX keys and cluster FSID
sshnaidmgfidente, so it didn't work, when no ceph storage?08:57
shardyd0ugal: Hey, what's the deal with the deployment/template mistral patches?08:57
gfidentesshnaidm, yeah because client wasn't generating the key in that case08:57
sneckliftershardy: thanks for comments, reviewing now08:57
shardyI expected to see confirmation of my debugging/fix and a new series posted?08:57
shardydid you and rbrady confirm my findings?08:57
d0ugalshardy: Yeah, I did.08:59
d0ugalshardy: been working on the CLI, I'll push your fix now...08:59
shardyd0ugal: ack, cool - I just wanted to ensure I'm testing the latest stuff locally08:59
openstackgerritDougal Matthews proposed openstack/tripleo-common: Adds parameters actions
openstackgerritDougal Matthews proposed openstack/tripleo-common: Adds Deployment Action & Workflow
openstackgerritDougal Matthews proposed openstack/tripleo-common: Adds action for template processing
*** oneswig has joined #tripleo08:59
d0ugalshardy: ^08:59
shardyd0ugal: thanks, ping me when there's a new CLI patch and I'll pull/test/review09:00
shardylooks like we're super-close now :)09:00
d0ugalshardy: Yup, will do!09:00
jaosoriorshardy: can you check out this revert to make swift the glance backend again?
*** mgould|afk is now known as mgould09:02
*** shardy is now known as shardy_mtg09:02
shardy_mtgjaosorior: will do09:03
*** dculik has quit IRC09:05
openstackgerritMartin Mágr proposed openstack/tripleo-specs: Availability monitoring support
sshnaidmdoes anybody know when can I find DIB related folks?09:08
*** TSCHAK has quit IRC09:11
*** dculik has joined #tripleo09:18
*** akrivoka has joined #tripleo09:20
*** kbyrne has joined #tripleo09:21
*** dtantsur|afk is now known as dtantsur09:24
mariosshardy_mtg: o/ fyi haven't forgotten about my AI to look into the upgrades blueprint... it is on my todo list today (just got sidetracked with a support thing but tdb how much time that will take). i will ping you later once i've looked09:28
openstackgerritChristian Schwede proposed openstack/tripleo-common: Install the upload-* scripts with tripleo-common
*** electrofelix has quit IRC09:40
*** electrofelix has joined #tripleo09:40
*** shardy_mtg is now known as shardy09:43
shardymarios: ack, thanks!09:43
*** zoliXXL is now known as zoli|lunch09:47
*** bootsha has joined #tripleo09:49
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/tripleo-quickstart: [WIP] undercloud reboot
*** bootsha has quit IRC09:55
*** yamahata has joined #tripleo09:56
*** tremble has quit IRC09:56
*** Jaison has joined #tripleo09:58
*** links has quit IRC09:59
*** tremble has joined #tripleo09:59
*** tremble has quit IRC09:59
*** tremble has joined #tripleo09:59
*** bootsha has joined #tripleo09:59
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Disable zaqar services by default.
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Disable zaqar services by default.
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Add 'openstack overcloud plan list' command
openstackgerritDougal Matthews proposed openstack/tripleo-common: Generate reference documentation for Workbooks
d0ugaljpich: Any idea what happens with your create workflow if the plan already exists?10:03
jpichd0ugal: "A container with this name already exists" error, I think10:04
d0ugaljpich: okay, thanks. I think I am going to have to use this in the deploy command10:05
*** radek_ has quit IRC10:05
jpichd0ugal: How so?10:06
d0ugaljpich: The deploy command accepts templates as an arg, so I need to create a plan on the fly10:06
shardywhat happens if you do a deploy, then delete the stack?10:08
shardywe don't have an overcloud delete command10:08
shardysounds like we may need one, to delete the stack, then delete (or at least rename) the plan10:08
*** Goneri has quit IRC10:08
shardyI would probably prefer we used a timestamp or something as a suffix, or can we use container versioning?10:09
d0ugalshardy: You can have an undeployed plan, so it would just become that I guess?10:09
shardyd0ugal: I'm just thinking if I do overcloud deploy --templates foo10:09
*** fragatina has quit IRC10:09
shardythen we'll try to create a new container called "overcloud"10:09
shardybut if I do that twice, there will already be a container with that name10:09
d0ugalshardy: The container name matches the stack name10:10
shardywhich always defaults to overcloud10:10
d0ugalThat's how the workflows link them at the moment10:10
*** chem has joined #tripleo10:10
d0ugalshardy: Sure, it will default to that.10:10
*** yamahata has quit IRC10:11
d0ugalshardy: so I am now at the point of getting a CREATE_FAILED and the CLI handles that correctly!10:11
d0ugalTrying to figure out what went wrong now.10:11
shardyopenstack overcloud stack failures list10:12
jpich"overcloud delete" sounds like a scary operation10:12
shardyjpich: why?  Folks do openstack stack delete atm10:12
shardyonly that will only delete the heat stack, not anything in swift10:13
d0ugaljpich: heat stack-delete overcloud - much the same thing.10:13
d0ugalshardy: is that a real command?10:13
shardyd0ugal: yes10:13
d0ugalI don't seem to have it :/10:13
jpichshardy, d0ugal: If everything worked fine, does it remove the stuff that got properly deployed?10:13
shardyd0ugal: pull the latest heatclient, maybe we've not yet released it10:13
d0ugaljpich: Yup10:14
* shardy will chase tagging one10:14
jpichd0ugal: Scary ;)10:14
d0ugalshardy: oh, it's in heat client. Works when I remove "overcloud" :)10:14
*** tremble has quit IRC10:14
*** sanjayu|AFK has quit IRC10:14
*** sanjayu|AFK has joined #tripleo10:15
d0ugal(or rather, move it to the end)10:15
shardyd0ugal: I think we should add that same thing to tripleoclient, so we basically run that by default whenver the overcloud deploy fails10:15
d0ugalOkay, seems I am missing keystone_signing_certificate. /me digs10:15
*** tremble has joined #tripleo10:16
*** tdasilva has quit IRC10:16
d0ugalWe really need the keystone certs to be created in the workflow10:19
d0ugalI guess for now I'll hack it in and we can move it over10:19
d0ugalI have no idea how the UI will do this otherwise10:19
d0ugalhrm, there is actually no good way for me to do this from the CLI10:19
d0ugalsince I can't pass the deploy workflow a env file10:20
d0ugalshardy: ^10:20
*** radek_ has joined #tripleo10:20
shardyd0ugal: well, we'll have to fix that ;)10:20
shardyagree the cert stuff needs to move into an action (as does the os-cloud-config endpoint stuff)10:20
d0ugalI'll see if I can hack it in for now to find out what else is broken10:21
d0ugalthen I can fix bits like this individually10:21
shardyd0ugal: do you need any help or are you happy pushing on this?10:21
*** spredzy has quit IRC10:22
d0ugalshardy: I'm happy pushing it, but help would be good.10:22
shardyseems like we need two new actions, an adjustment to the workflow, and an interface for user provided environments to the templates workflow10:22
d0ugaland we need to figure out what else is broken :)10:22
rookEmilienM: passed CI minus the non-voting centos-7-undercloud, which seems to have failed very early...10:23
d0ugalshardy: Maybe the idea is that env files are added during the plan creation.10:23
jaosoriorjistr: thanks for your comments on the certmonger patch dude. So I could actually work on getting HAProxy to always be there as follow up work. I just left it the way it was before cause that's what the previous behaviour was.10:23
*** spredzy has joined #tripleo10:23
*** Goneri has joined #tripleo10:24
*** tosky has joined #tripleo10:25
shardyd0ugal: I suppose it could go into either the plan generation or the templates processing10:26
d0ugalshardy: for flexibility we might need both.10:26
shardyprobably depends if we want to persist the ssl stuff in swift10:26
d0ugalWe could really do with Barbican for some of this...10:27
rookshardy hey! long time --- curious if you have time today to give me the state of composable roles in OOO? (or maybe there is a email/etherpad I should be following)10:27
shardywell lets pick one to start with - want me to hack a new action into the plan generation?10:27
jaosoriorjistr: By the way, note that we still support passing the certificate. In that case, certmonger wouldn't be tracking anything.10:28
shardydon't want to overlap with your efforts but happy to help as I need this working asap10:28
d0ugalshardy: Sure, that would be great. So it will just create the certs etc. when the plan is created?10:28
jaosoriorso certmonger only comes into play if we set the "generate_service_certificate" flag10:28
jaosoriorin the undercloud10:28
* d0ugal isn't sure what the "etc" part is :)10:28
shardyd0ugal: Yeah I'll drop a yaml file into the plan, then we just need to include it10:28
shardyrook: Hi!10:28
*** limao_ has quit IRC10:29
shardyrook: sure, basically we've landed nearly all of the composable services now10:29
rookMost excellent!10:29
d0ugalshardy: Great.10:29
shardyrook: we don't yet have landed support for custom roles, but I'm working on it10:29
shardythis stuff d0ugal and I are discussing is the next step towards it10:29
d0ugalshardy: btw, the deploy workflow ends before the stack operation is finished - it doesn't wait. that might be an issue for the endpoint work.10:29
rookcomposable roles vs custom roles?10:29
shardyrook: composable services means you can move services between existing roles, custom roles is when you want e.g a completely new group of nodes, e.g "ContrailController" or something10:30
*** rodrigods has quit IRC10:30
shardyd0ugal: yeah, that needs fixing too10:30
rookah ha!10:30
*** rodrigods has joined #tripleo10:30
shardyrook: I've been planning a blog post about it, will try to do it later today10:30
rookno karma bot fail10:31
shardyrook: You can also see the most recent deep dive, which was about puppet but touched on the composable stuff too10:31
rookshardy thanks! We are getting Mitaka numbers now (at scale) so we will have Liberty-> Mitaka-> Newton numbers (mem usage, time, etc)10:32
rookShould be pretty cool.10:32
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Migrate the deploy command to use the Mistral workflows
shardyrook: nice!  How is liberty-mitaka looking?10:33
shardyI would assume roughly the same?10:33
d0ugalshardy: ^ still far from complete, but that updated patch will now at least not error out (well, it might if the stack create works!)10:33
shardyd0ugal: thanks, will pull it now10:33
rookshardy so far, mitaka looks worse for some odd reason.10:34
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Migrate the deploy command to use the Mistral workflows
shardyrook: Hmm, Ok, well be happy to help look into that if you can provide access to data10:34
rookshardy however, I haven't personally looked at it.10:34
rookshardy this is someone on my team - once we have the data, ill be sure to send it off your way for sure.10:35
shardyI expect the newton numbers to look quite a lot better, but very keen to get confirmation on that10:35
rookme too10:35
rookshardy I will personally work on that when it lands in RDO.10:35
shardyrook: nice, thanks!10:35
*** social has quit IRC10:36
d0ugalshardy: We probably need a version of this that doesn't write to a file.
rookNow will we have a custom role for OpenShift :P10:37
shardyd0ugal: Yeah, for now I was going to pass it a temporary file10:37
shardywe can fix it later to return a map10:37
*** tdasilva has joined #tripleo10:38
*** tosky has quit IRC10:39
*** tosky has joined #tripleo10:39
*** tosky has quit IRC10:39
*** tosky has joined #tripleo10:40
*** tosky has quit IRC10:41
*** social has joined #tripleo10:41
*** tosky has joined #tripleo10:43
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Use certmonger for automatic cert generation
jaosoriorjistr: addressed your comment of using reload instead of restart ^^10:44
*** milan has quit IRC10:48
*** Jaison has quit IRC10:49
*** tosky has quit IRC10:49
mariosgfidente: jaosorior can you please revisit
jaosoriormarios: I did10:51
mariosgfidente: jaosorior shardy when you get a chance please. we have a green run at wdyt?10:51
mariosjaosorior: thanks10:51
mariosjaosorior: i haven't forgotten your reviews man, i got pulled into some other thing. which is ongoing but quiet for now, iso i'll try do some reviews10:52
gfidentethis can be uses against you in the future10:52
gfidenteI will :)10:52
gfidenteI was more concerned about the manila_config cleanup though10:53
mariosgfidente: there are a few things that are given in life, and one must learn to deal with them. You taking advantage of this situation is a give. I was waiting after you typed 'so' i was just expecting you to say 'NO" and torment me for a while10:53
*** links has joined #tripleo10:53
mariosgfidente: i am alsmost disappointed that you didn't10:53
openstackgerritDougal Matthews proposed openstack/os-cloud-config: Extract the keystone cert generation into a function
*** tosky has joined #tripleo10:54
mariosi .. i can't bring myself to say it...10:54
marios"thanks gfidente"10:55
gfidentemarios,  ahahah10:55
*** bvandenh has joined #tripleo10:56
openstackgerritMerged openstack/puppet-tripleo: Fixup nit in manila pacemaker profile, duplicate variable
*** oneswig has quit IRC11:06
jaosoriormarios: regarding the certmonger patch. I actually changed to using reload in patch #37
jaosoriorthat's pretty much the only change11:07
mariosjaosorior: ack yeah man i had seen it but already wrote the comment so posted it whilst reviewing v37 agan 'properly'11:07
marioslgtm, comments there11:08
*** kbyrne has quit IRC11:11
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Reinstall python-requests in bootstrap script
*** kbyrne has joined #tripleo11:12
jistrjaosorior: thanks for the replies, +2'd11:13
jaosoriormarios, jistr: I'm trying again an update from old style auto generate to the certmonger approach11:15
mariosjaosorior: nice thanks11:16
gfidenteshardy, so it's green now , it just needed :(11:20
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Don't assume an eth1
*** lucasagomes is now known as lucas-hungry11:21
shardygfidente: ack, thanks, approved!11:22
gfidente shardy sorry ;(11:22
shardyno worries! :)11:22
*** rhallisey has joined #tripleo11:25
*** tremble has quit IRC11:25
*** yolanda has quit IRC11:27
*** tremble has joined #tripleo11:28
*** yolanda has joined #tripleo11:31
jaosoriorjistr, marios: crap... so with the addition of the anchor I did; doing an upgrade from old-style auto generate to the certmonger approach can fail (certmonger executes some stuff pretty randomly). For some reason puppet disregards the notify I had in puppet-tripleo that should execute the CA code before the haproxy code. So it starts doing keystone commands before tha CA is trusted, and thus the SSL11:32
jaosoriorverification fails. This isn't an issue with FreeIPA, or if it's a new deployment. But yeah, this case is an issue. I need to figure out a way for puppet to respect that ordering. I marked the commit as WIP again until I figure this out. Sorry about it.11:32
mariosjaosorior: ack thanks for checking good thing jistr brought it up then11:33
jistrjaosorior: no worries thanks for verifying :)11:33
jaosoriorhad tested it, but it was before I added the anchor. I'll let you guys know when I come up with a solution and have tested the full matrix again (new deployment, update, and FreeIPA)11:34
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Don't assume an eth1
thrashzaneb: ping... got this when trying to scale up with your patch for the image id thing: Resource CREATE failed: BadRequest: resources.NovaCompute: Invalid imageRef provided. (HTTP 400)11:35
*** bootsha has quit IRC11:35
mariosjaosorior: nobody likes your excuses jaosorior i mean look at gfidente. fully joking obviously man thanks for checking and the fact you did and -1d it is why i was happy to say and stand by " I also think jaosorior is sane enough"11:36
mariosfor what *thats* worth anyway11:36
*** aviramb has joined #tripleo11:37
*** bootsha has joined #tripleo11:38
gfidentemarios, right I noticed on manila you were already at patchset #91 and didn't want you to hit 10011:38
mariosgfidente: so in my defence it was already weeel into double digits by the time i came to it11:39
gfidenteyou started11:39
gfidenteI just replied11:39
mariosgfidente: what did i ever do to you gfidente :( i will have to report you to hr11:40
gfidentelet me see if I can find a picture11:40
*** dculik has quit IRC11:44
*** sshnaidm is now known as sshnaidm|afk11:44
*** coolsvap_ is now known as coolsvap11:45
*** dprince has joined #tripleo11:45
gfidenteor better
*** limao has joined #tripleo11:47
*** morazi has joined #tripleo11:50
openstackgerritChristopher Brown proposed openstack/tripleo-heat-templates: Enable Neutron LBaaS Integration
*** fultonj has joined #tripleo11:51
*** bfournie has quit IRC11:52
jaosoriormarios, jistr: actually, I just noticed that the try I did did not include the latest puppet-tripleo11:54
jaosoriorso the notify I did wasn't in the package I used for puppet-tripleo, for some reason11:54
jistrah ok, so there's still a chance that everything's fine11:55
jaosoriorjistr: Yes11:55
jaosoriorI'm trying to build the puppet-tripleo package with what's in master11:55
jaosoriorbut it's the package build is failing using tripleo.sh11:56
*** tzumainn has joined #tripleo11:56
jistrperhaps you could try the swift artifacts .tar.gz approach11:56
jaosoriorjistr: Can that work with testing the undercloud?11:57
jaosoriorthought it was an overcloud thing11:57
jistrjaosorior: of course it can't, i'm confused :))11:57
jistrjaosorior: so i think in this case you could just update it in-place11:57
jistrif the RPM building gives you lemons11:58
mariosjaosorior: ack, theres a bunch of things that review depended on and that's just the last few revisions since i looked so i'm not surprised but yeah take your time and verify would be good11:58
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Add ceph_pacemaker config
jaosoriormarios: Yep, verifying again11:59
*** limao has quit IRC11:59
openstackgerritBrad P. Crochet proposed openstack/tripleo-common: Change image names to match what the rest of the system expects
openstackgerritBrad P. Crochet proposed openstack/tripleo-common: Remove the delorean-repo element from CentOS images
openstackgerritBrad P. Crochet proposed openstack/tripleo-common: Remove epel element from base yaml ramdisk agent image
*** limao_ has joined #tripleo12:00
*** zoli|lunch is now known as zoli12:01
*** zoli is now known as zoliXXL12:01
pandathere's something during undercloud install that modifies /etc/sudoers ?12:02
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Make $TRIPLEO_ROOT overrideable
jaosoriorEmilienM: sup dude12:03
*** radek_ has quit IRC12:04
*** milan has joined #tripleo12:05
*** weshay has joined #tripleo12:07
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart: Run dstat on the virthost and the undercloud
*** ccamacho is now known as ccamacho|lunch12:11
openstackgerritSteven Hardy proposed openstack/os-cloud-config: Add the option to dump keystone pki data to a string not a file
*** dculik has joined #tripleo12:13
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart: Run dstat on the virthost and the undercloud
openstackgerritSteven Hardy proposed openstack/tripleo-common: Add action to generate PKI certs in plan
shardyd0ugal: sorry for the delay, I think that'll do it12:15
shardyI only added it to the default plan creation workflow for now12:15
jaosoriorshardy: I thought we weren't using keystone PKI tokens anymore12:15
jaosoriorAnd the plan was to move to fernet12:16
shardywe still generate the certs and pass them in, but I actually think the default is still uuid12:16
d0ugalshardy: np, I'll check it out shortly. Just going to grab some lunch now.12:16
jaosoriorshardy: I think the recommendation is not to use PKI tokens12:16
shardyjaosorior: that is just moving some existing logic from tripleoclient to mistral12:16
*** limao_ has quit IRC12:17
shardyjaosorior: feel free to investigate, it may be that we need to adjust defaults and deprecate that stuff12:17
shardyjaosorior: for now, the deployment fails without it12:17
*** limao has joined #tripleo12:17
jaosoriorshardy: I think we should be deprecating it. But yeah, we can ask ayoung when he's online12:17
d0ugalthat would be even better :)12:17
jaosoriord0ugal, what?12:17
*** chem has quit IRC12:18
d0ugaljaosorior: deprecating it :)12:18
*** ramishra has quit IRC12:18
shardyrip it all out and make my last hours work pointless :D12:18
*** jpena is now known as jpena|lunch12:18
*** chem has joined #tripleo12:18
* d0ugal will bbiab12:18
jaosoriorshardy lol, sorry about that dude12:19
*** oshvartz has quit IRC12:19
EmilienMslagle: why did you do a recheck on ?12:19
EmilienMslagle: the patch failed, and it was expected12:19
EmilienMalso see
shardyjaosorior: see KeystoneCACertificate in ./puppet/services/keystone.yaml12:19
EmilienMslagle: it demonstrates well that puppet modules are installed from source when needed12:19
shardyI'm fine with deprecating it if that's the reccomended approach, was only joking12:19
shardyit's not actually PKI is it12:20
shardyjaosorior: we generate KeystoneCACertificate and KeystoneSigningKey12:21
jaosoriorshardy: shardy, no worries, lets just consult ayoung when he's online. He's the keystone guru12:21
slagleEmilienM: the multinode job failed b/c of a different error and never made it to OC deploy12:22
slaglebut, yea, i see that the uc job did fail on the right thing12:22
EmilienMslagle: no, oc12:22
EmilienMslagle: but yeah I'll let the jobs finish to make sure it works everywhere12:22
EmilienMshardy, slagle, bnemec: fyi, I got an undercloud upgrade working in CI from Mitaka to newton12:23
slagleEmilienM: ok, i'm confused12:23
slagleEmilienM: first, the console log you linked me is from an ovb ha job12:23
*** adarazs is now known as adarazs_brb12:23
EmilienMslagle: right, so we're good, we see the Puppet failure, which was wanted12:23
slagleEmilienM: the point of this though is to make sure the multinode job fails12:24
slagleon the puppet failure12:24
EmilienMslagle: ok12:24
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bridge and Port
slagleand it failed on a different thing, which is the bug i filed and fixed and then rechecked12:24
EmilienMslagle: sorry I was confused why you rechecked; I'll make sure it works in multi node too :-)12:24
slagleassuming the recheck gets it on to a different cloud (rax)12:25
openstackgerritMerged openstack/python-tripleoclient: Generate CephX keys and FSID regardless of CephStorageCount
EmilienMslagle: it doesn't matter12:25
jaosoriorshardy: By the way, if we are defaulting to UUID tokens, what are we using the keystoneCA and signing keys for?12:25
*** tremble has quit IRC12:25
*** bfournie has joined #tripleo12:26
*** tremble has joined #tripleo12:27
slagleEmilienM: it does matter12:27
slagleEmilienM: due to the other unrelated bug12:27
*** jayg|g0n3 is now known as jayg12:28
slagleEmilienM: everything will fail on ovh and osic clouds right now12:28
shardyjaosorior: we don't actually specify the token format in services/keystone AFAICT, so it's probably from when keystone or puppet-keystone had pki as a default?12:28
shardyIf we can remove them now, lets do that12:28
hewbroccaI know keystone guys would be really happy if we got rid of UUID tokens12:29
EmilienMslagle: damn12:29
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: Stop all services before upgrading undercloud
EmilienMbnemec: I took over, and I tested and upgraded mitaka to newton successfuly ^12:30
*** dculik has quit IRC12:30
shardyhewbrocca: AFAICS we're just using the keystone default12:30
*** limao has quit IRC12:31
*** adarazs_brb is now known as adarazs12:34
jaosoriorhewbrocca: which is UUID AFAIK12:34
*** sshnaidm|afk is now known as sshnaidm12:34
shardyaccording to that all the cert stuff is deprecated12:35
dprinceshardy: this is what I'm using for my signalling fix12:35
dprinceshardy: A hook which invokes os-apply-config.12:35
shardyso I wonder if we can reasonably remove it12:35
shardygiven that we don't actually offer any parameter that enables operators to choose pki12:35
shardythey might get broken if they're forcing the token format via hieradata12:35
shardybut that's true for almost any hiera override I think12:35
shardydprince: nice, will check it out!12:36
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Add new pacemaker configs
jaosoriorjistr, marios: Fuck yeaaaah! it does work with ugprades! Just tried it again on a freshl deployment, making sure the puppet-tripleo dependency was there.12:37
jistrjaosorior: neat. So i think we can un-wip and land12:37
*** radek_ has joined #tripleo12:37
jistronce CI results come in of course12:37
jaosoriorjistr, marios: sure. Either way, I also re-ran the non-ha, so it should again show that it works on the first run.12:38
openstackgerritSteven Hardy proposed openstack/puppet-tripleo: Remove keystone PKI cert generation
mariosjaosorior: yeah maybe add some info as a comment on what/how you tested12:38
jaosoriormarios: will do12:39
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Remove keystone PKI related parameters
shardyd0ugal: Ok, that's the other way to fix it ;)12:41
*** rlandy has joined #tripleo12:41
*** lucas-hungry is now known as lucasagomes12:43
shardydprince: the only weird thing about that is o-a-c isn't operating on input data from the deployment?12:44
dprinceshardy: yep, we don't need it I think though12:44
shardySure, for this case, but it makes it kind of broken as a generic heat-config hook12:44
dprinceshardy: it is weird. The point was this solves signalling issues for good here12:44
shardydprince: yeah, I'd like to see us deploy hiera via a hook too12:45
dprinceshardy: would you rather tie in signalling to 99-refresh-completed which is already quite a mess?12:45
dprinceshardy: I was about to do that... but this seemed easier since it is all plummed and ready to go12:46
shardydprince: No, I'd really like to see 99-refresh-completed die in a fire ;)12:46
dprinceshardy: right, so this is a step towards that12:46
dprinceshardy: except that we do still need os-refresh-config12:46
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart: Run dstat on the virthost and the undercloud
dprincethe os-refresh-config element that is, which is where 99-refresh-config lives12:47
*** julim has quit IRC12:48
dprinceshardy: So right now this would send 2 signals I think, unless you removed that file12:48
shardydprince: Yeah, that's the part I'm unsure on - if you have a group: os-apply-config SoftwareConfig, this hook will run, but what deploys the data passed to the config?12:48
dprinceshardy: the data is deployed by os-collect-config12:49
dprinceshardy: os-collect-config writes metadata. All of it12:49
dprinceshardy: and os-apply-config gets at it by using the /var/lib/os-collect-config/os_config_files.json list12:49
shardyyeah, I meant we have no way of knowing which deployment acts on which piece of metadata12:49
dprinceshardy: we already have that with os-apply-config12:50
dprinceshardy: to me the point here is signalling12:50
shardyYeah, but it's not what any other heat hooks do12:50
dprinceshardy: it isn't12:50
dprinceshardy: but group: os-apply-config in our heat templates is the only case that doesn't abide by proper signal handling IMO12:51
dprinceshardy: and this would fix that12:51
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bond
shardydprince: well, it wouldn't really fix it, because if you run 10 group: os-apply-config concurrently and one of them fails you have zero idea which one was actually broken12:52
shardyone will just randomly fail12:52
shardyit fixes the transport though, that's true12:52
gfidentebreaking news, we are now gating on ceph/jewel in master and ceph/hammer in stable/liberty12:52
shardydprince: FWIW I envisaged a hook like this which *only* worked with the data passed to the deployment12:52
shardyso we can break the oac global merging of configs thing12:53
shardywe'd have to rename it e.g group: heat-os-apply-config I guess12:53
dprinceshardy: sure, so I think that isn't called os-apply-config12:53
dprinceshardy: but we could switch to it, sure12:53
*** tosky has quit IRC12:53
shardydprince: Ok, cool, just throwing some ideas around, I'm OK with this as an interim measure if it's needed12:54
shardyideally I'd like us to stop using os-apply-config completely ;)12:54
dprinceshardy: well, this gives me signals12:55
*** tosky has joined #tripleo12:55
dprinceshardy: its an idea. If people don't like it we can go a different direction12:55
shardydprince: sure, I think it's better than what we have currently as it fixes some transports, and gives us an error path12:55
dprinceshardy: but, the short term would likely be adding more to 99-refresh-completed12:56
shardydprince: yeah, I guess if we land this we can then disable the signalling in there12:56
dprinceshardy: I wasn't sure on that actually. People may will still depend on that12:57
dprinceshardy: like Magnum or something12:57
dprinceshardy: lots of groups use os-apply-config I think as an element12:57
shardydprince: we could have it check the installed hooks and not signal if the o-a-c one is installed12:57
dprinceshardy: so maybe we keep both elements, and one "provides" the os-apply-config requirement12:58
dprinceshardy: yep, I thought of that too12:58
shardythe duplicate signal will be ignored, but they generate confusing events12:58
dprinceshardy: have the user just choose one over the other seems cleaner. DIB already has a "provides" feature so I think we could use it to satisfy the requirement12:58
*** rhallisey has quit IRC13:00
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton
EmilienMslagle: when you have time, could you give feedback on ^ please?13:01
slagleEmilienM: yep will do13:04
EmilienMslagle: thanks, also bnemec & shardy when you have time13:04
*** akrivoka has quit IRC13:05
*** rhallisey has joined #tripleo13:05
EmilienMslagle: when we have the upstream undercloud upgrade CI job in place, I would like to continue with overcloud. Did you already start something on this side?13:07
slagleEmilienM: not for upgrades13:08
slagleEmilienM: i'm doing some testing of HA locally13:08
EmilienMslagle: do you mind if I start looking at it?13:08
slagleEmilienM: at what? upgrades?13:08
slagleno. go4it13:09
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bond
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bridge and Port
EmilienMslagle: I am a bit confused about current status of baremetal jobs, are we supposed to have the old upgrade job back soon?13:09
slaglewhat baremetal jobs?13:10
EmilienMslagle: I plan to start small and upgrade a single node overcloud13:10
slagleor do you just mean ovb?13:10
EmilienMslagle: you know, the old upgrade job we had the last months13:10
EmilienMwhich was more an "update" job13:10
slaglewe should be able to add all the old jobs back once we bring back full capacity on rh113:10
EmilienMok, when is it scheduled?13:10
slagleEmilienM: when do you want to schedule it? :)13:11
EmilienMtonight !13:11
slagleEmilienM: if you can check the jobs that have been running on rh1 and see if they are stable13:11
slagleif they look good, then propos a patch to increase the workers on rh1 in project-config13:11
*** ramishra has joined #tripleo13:11
slaglethen look at adding the jobs back13:11
*** oshvartz has joined #tripleo13:12
*** jcoufal has joined #tripleo13:12
slagleEmilienM: these are the patches:13:14
slagleEmilienM: can you push on those?13:15
slagleif we can try and move those forward...13:15
slaglethat is the road to getting rh1 back13:15
*** jpena|lunch is now known as jpena13:16
EmilienMslagle: sure thing, /me looks now13:16
*** ccamacho|lunch is now known as ccamacho13:17
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Enable net-iso for ovb jobs
EmilienMthis is a blocker iiuc ^13:18
*** akrivoka has joined #tripleo13:19
EmilienMslagle: can you look ? name is ok?13:21
*** jdob has quit IRC13:22
*** jdob has joined #tripleo13:23
dtantsurfolks, do you know if ooo-quickstart tears down the previous undercloud by default?13:23
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Add minimal_pacemaker config
*** pgadiya has quit IRC13:25
openstackgerritGabriele Cerami proposed openstack/diskimage-builder: secure_path in sudoers: deal with possible quotes
*** fragatina has joined #tripleo13:28
mariosshardy: and magic
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/tripleo-quickstart: [WIP] undercloud reboot
pandathis blocked one of my undercloud installs, don't know why that option is surrounded by quotes now. ^13:31
shardymarios: thanks! :)13:32
openstackgerritKarthik S proposed openstack/tripleo-heat-templates: Configure the pci_passthrough_whitelist via THT
*** noshankus has quit IRC13:33
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Environments Listing and Updating using Mistral Actions
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart: Run dstat on the virthost and the undercloud
*** jdob has quit IRC13:36
*** jdob has joined #tripleo13:37
*** lblanchard has joined #tripleo13:40
jaosoriordtantsur: I don't know. What I do is actually do a destroy and then undefine of the undercloud domain. And then I run the again13:40
dtantsurhmm, not very handy13:40
jaosoriordtantsur: not really. If you figure it out let me know. Would save me some time too13:41
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart: Run dstat on the virthost and the undercloud
*** radek_ has quit IRC13:47
openstackgerritJeff Peeler proposed openstack/tripleo-common: Change input for UpdateCapabilitiesAction
*** fragatina has quit IRC13:51
*** ebalduf has joined #tripleo13:53
*** akshai has joined #tripleo13:54
jaosoriordprince: Does zaqar use some sort of api-paste configuration?13:56
dprincejaosorior: I don't think so. Don't rememver13:58
jaosoriordprince: Also, I tried doing a simple curl to zaqar to see how the version json-home behaves and I got an error straight:
jaosoriordprince: This is in a clean deployment with the latest undercloud13:59
jaosoriorI haven't put anything in front of zaqar yet13:59
jaosoriorI'm working on adding TLS for it in the undercloud13:59
dprincejaosorior: Now I feel like I've seen this before13:59
dprincejaosorior: I think it may have been from a too recent autobahn library or something14:00
dprincejaosorior: using the packages?14:00
*** fragatina has joined #tripleo14:00
*** krot_sickleave is now known as krotscheck14:00
*** akshai has quit IRC14:00
dprincejaosorior: when I hit this the package didn't work at all14:00
openstackgerritMerged openstack-infra/tripleo-ci: Don't assume an eth1
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/tripleo-quickstart: [WIP] undercloud reboot
dprincejaosorior: I had to go and add policy.json, and request that RDO build the autobahn dep.14:01
jaosoriordprince: Yes, using packages14:01
dprincejaosorior: so, I've seen it but I can't remember what exactly I did to solve it14:01
jaosoriordprince: policy.json is present in /etc/zaqar14:01
dprincejaosorior: other than I ended up not needing to fix anything14:01
dprincejaosorior: right policy.json isn't related. Just saying I used to pip install it because it was broken otherwise14:02
dprincejaosorior: and that was when I hit this error14:02
*** akshai has joined #tripleo14:02
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Replace hard-coded regionOne with parameter references
jaosoriordprince: Anybody I could ask about this?14:03
hewbroccaHey folks, just a little workflow love for paramite spec and ops-tools will be off to the races14:06
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton
hewbroccaalready got a lovely +2 from mr slagle14:07
dprincejaosorior: it has to do with falcon14:07
jaosoriorseems so14:07
dprincejaosorior: there was a revert for this, sec14:07
slagleEmilienM: looks like both your test CI patches failed at the right place this time14:07
dprincejaosorior: 9cfa5bc8ca1b31845848a0d3b5db0ee06885473914:08
dprincejaosorior: I filed a bug for this and it was a dup14:08
dprincejaosorior: memory is coming back to me now :)14:08
dprincejaosorior: looking for links...14:08
EmilienMslagle: ok, I'll double check and abandon them, so we know we're good now :)14:08
*** tbonds has joined #tripleo14:08
openstackLaunchpad bug 1583358 in zaqar "Zaqar tempest test failed" [High,Fix released] - Assigned to Fei Long Wang (flwang)14:10
pabelangerEmilienM: slagle: how is tripleo-ci this morning? Any problems stopping us from increasing capacity on -rh1?14:10
dprincejaosorior: and this was the commit that fixed it e6d975fd1009c70a1d2b3fbe699895345220562e14:10
slaglepabelanger: EmilienM was going to check14:11
dprincejaosorior: double check that... and perhaps if you need more help reach out to wanghao who seems familiar in this territory14:11
openstackgerritChristopher Brown proposed openstack/tripleo-heat-templates: Enable Neutron LBaaS Integration
EmilienMpabelanger: we have some blockers apparently14:12
EmilienMpabelanger: AFIK we need first14:12
openstackgerritMerged openstack/tripleo-ui: Add prepublish script to bundle dependencies
*** ayoung has joined #tripleo14:12
slagleEmilienM: why is that a blocker?14:13
jaosoriordprince: Actually it seems that I have the proposed fix as part of the package14:13
EmilienMslagle: required for
dprincejaosorior: link?14:13
ayoungjaosorior, once you are done with this, can we discuss TLS/SSL?14:13
jaosoriordprince: No link, just checked the source code14:14
jaosoriorayoung: if it's fast. I gotta go soon :/14:14
dprincejaosorior: sorry, what I mean is look to see if there have been any recent changes there14:14
ayoungjaosorior, OK14:14
dprincejaosorior: when I saw issues with this that was the fix...14:14
jaosoriorayoung: what's up?14:14
jaosoriordprince: It does seem to me that the packaged version of falcon is 0.3.014:15
dprincejaosorior: this happend 2 months back, thus my memory fails me a bit here14:15
ayoungjaosorior, so I need to set up TLS with Hostname support14:15
ayoungI was looking at and a couple things are not clear14:15
ayoungI did a deploy using   CloudName:
ayoung    DnsServers:
ayoungbut modified for mine, and the auth URL is still not the FQDN14:15
slagleEmilienM: ok. why is a blocker?14:15
jaosoriorayoung: you gotta use CLOUDNAME instead of IP_ADDRESS in the enable-tls.yaml14:16
*** ramishra has quit IRC14:16
EmilienMslagle: you told me so14:16
EmilienM1h ago14:16
ayoungjaosorior, ah, so not just in the overall setup?  I had not yet done TLS...OK, that makes sense.  Next question14:16
EmilienMand I'm still figuring why14:16
slagleEmilienM: i did not say that was a blocker14:16
ayoungif I set things up using self signed, can I swap in differn certs afterwards?14:16
EmilienMslagle: ok, good. Thanks.14:17
slagleEmilienM: it was one of a few patches14:17
jaosoriorayoung: Also remember to have a certificate that has the the fqdn you would use, and not the ip address (I've messed that up too many times)14:18
jaosoriordprince: I think it's related to having an old falcon version14:18
ayoungjaosorior, yes, that is what I am trying to do.  I was hoping to deploy with FQDN without TLS first to test things out, and then to roll in TLS, but looks like I need to go all the way to make it work?14:18
ayoungjaosorior, so I set:14:18
ayoung  CloudName: openstack.ayoung-dell-t1700.test14:19
ayoung  CloudDomain: ayoung-dell-t1700.test14:19
ayoung  DnsServers:
jaosoriorayoung: You can do it if you override the EndpointMap to use CLOUDNAME instead of IP_ADDRESS, which is the default14:19
jaosoriorso you need something similar to enable-tls.yaml but without the TLS bits (which is the ports and the https protocols)14:19
ayoungjaosorior, I was also hoping to not have to copy the entire template directory, but rather just the templates I need to modify14:19
d0ugalshardy: With do we not need something in the deployment workflow to make it include the environment?14:20
jaosoriorayoung: You only need one environment file14:20
jaosoriorto override that value14:20
*** jeckersb is now known as jeckersb_gone14:20
ayoungjaosorior, and that needs to just have the same values as enable-tls.yaml does in the sample...let me try14:20
ayoungjaosorior, what is that about a root cert?14:21
jaosoriorayoung: Not the same values. If you're not doing TLS, then you don't need any of the certs and CA stuff14:21
EmilienMpabelanger: honestly, I'm not familiar enough atm to know the road to RH1, I started to look at the patches this morning14:21
jaosoriorand you need to switch the ports to the defaults (not the overridden ports that we use for TLS), and also set all the ports to http instead of https14:21
EmilienMpabelanger: I would ask bnemec probably14:21
dtantsurhmm, now is it normal for ooo-quickstart to get stuck on "Prepare the undercloud for deploy" for an hour?14:21
ayoungjaosorior, I do want to do TLS, just step by step14:22
jaosoriorooh, alright14:23
jaosoriorayoung: Then in that case just copy enable-tls.yaml14:23
jaosoriorand add the cert and key there14:23
*** links has quit IRC14:23
ayoungjaosorior, so the part that says14:23
ayoung  OS::TripleO::NodeTLSData: ../puppet/extraconfig/tls/tls-cert-inject.yaml14:23
jaosoriorthe root ca you need to add to another environment14:23
slagledtantsur: fyi, i dont know if quickstart people answer in this channel. they have their own channel apparently. you might try there. or encourage them to join here14:23
ayoungcan I edit that and make it an absolute URL?14:23
jaosoriorayoung: Yes14:23
ayounger path14:23
dtantsurslagle, oh, really? do you know the channel name?14:23
jaosoriorthe root CA you add here
jaosoriorthat's another environment14:24
* dtantsur is really surprised, provided that we're pushing for quickstart to become the default entrance to tripleo14:24
slagledtantsur: oooq maybe? or ooo-q14:24
ayoungok, so I have a bunch of files I am maintainitng in a git repo in /home/stack/deployment.  I'll add a copy of enable-tls in there with my values14:24
slagledtantsur: yea, well, not everyone is14:24
bnemecEmilienM: pabelanger: tripleo-ci looks to be in good shape today, and I see seven passed jobs on rh1 so I'd say let's go for it.14:24
slagledtantsur: for reasons such as these :)14:24
ayoungjaosorior, should be enough to get me rolling.  I'll bug you more tomorrow.  I still need to make it work with the CA cert issued by IPA.14:24
pabelangerbnemec: EmilienM: okay, let me get a review up in openstack-infra14:25
bnemecWhen I looked yesterday the failed jobs on rh1 all appeared to be legit, either due to problems with the patch or the ceph bug.14:25
jaosoriorayoung: awesome! hope it goes well :)14:25
jaosoriorI gotta go now14:25
shardyd0ugal: Yes, but I'm currently testing as an alternative14:25
jaosoriorhave a good day guys!14:25
*** jaosorior has quit IRC14:25
dtantsurslagle, hmm, I see.. it would be nice to get it sorted.. thanks, #oooq worked14:25
slagledtantsur: it is sorted from my perspective :)14:25
d0ugalshardy: oh, I guess I could try just setting them to an empty string for now14:26
EmilienMpabelanger: and
dmsimardHey, are specs rendered somewhere ? mentions but that's a dead link.14:26
shardyd0ugal: Yeah, it looks like we need to remove them, then switch to fernet but I wouldn't get blocked on it14:26
shardyAFAICS we don't use them anymore14:26
dtantsurslagle, as a person a bit aside of tripleo processes, I'm really confused by these numerous ways of setting up a virtual environment..14:27
dmsimardslagle: okay, I'll submit a patch to fix the link :)14:27
slagledtantsur: we have 1 documented way14:27
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add support for deploying RGW on controllers. (Composable)
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add support for deploying Ceph RGW role
slagledtantsur: i was just working on a patch to update it since it broke when we removed epel repos14:28
*** ramishra has joined #tripleo14:28
*** yamahata has joined #tripleo14:28
slagledtantsur: tripleo quickstart was supposed to replace that 1 way. but docs were never updated, so afaic it's not replaced14:29
dtantsurslagle, I've heard people saying things like "tripleo-docs is hopelessly broken". was it referring to this epel problem?14:29
shardyalso CI doesn't use it, which is a reason many of did not switch to quickstart14:29
slagledtantsur: no. and i don't subscribe to "hopelessly broken" fud14:30
slaglei'm sure there are problems14:30
slaglewe fix them14:30
dtantsurcool. because I've used the docs successfully for quite some time14:30
slagleor we can fix bugs that are filed against the docs14:30
dtantsurso I was just as surprised14:30
shardydtantsur: if quickstart doesn't work for you, and you want to automate some steps from the docs (and get pretty close to what we run in CI) try this:14:30
shardythose are notes of how I set my env up, many folks do similar14:30
dtantsuraha, now :) this is the 3rd way, but I think someone was working on changing docs to use it14:31
shardydtantsur: No, it's just a script we use in CI14:31
shardyI was trying to be helpful and get things working for you :\14:31
shardythe steps are supposed to match the docs, but invariably docs rot, so we need to auto-generate them from CI14:32
dtantsurthanks, shardy, I'm not blaming you at all, just trying to figure out the whole picture14:32
slagledtantsur: is copy/paste commands from the docs. that's what it is there for14:32
jristI know everyone is very busy but can we get some eyes on ?14:33
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Add 'openstack overcloud plan delete' command
openstackgerritDavid Moreau Simard proposed openstack/tripleo-specs: Update rendered specs URL
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Add cinder-backup profiles
shardyjrist: the assumption of plan names always matching stack names seems bogus to me, so we'll probably end up having to fix that14:37
shardyI can imagine folks wanting to upload multiple plans, then deploy overclouds with the same name, changing plan version while testing14:38
*** myoung is now known as myoung|mtg14:38
shardyI guess we can land it as-is, but it seems to be an indication we've not really thought through the operator workflow around plans yet?14:38
jristrbrady: ^14:38
openstackgerritJames Slagle proposed openstack/tripleo-incubator: Don't require EPEL
shardySimilar with the hacky introspection of overcloud.yaml in the roles listing14:39
openstackgerritMerged openstack/tripleo-validations: Documentation how validations work
shardylike, it's fine as a first step but it's going to break pretty soon14:39
shardyjrist: I'd probably prefer we added a PlanName parameter to overcloud.yaml, then always pass in the name of the actual plan used when creating/updating a stack14:40
rbradyshardy, jrist: this has been used int he previous implementation and was included in the spec for discussion.  I have no problem changing it, but the concept has had opportunity for discussion but seemed maybe it was missed14:41
shardyrbrady: sure, I guess it was missed - I'm just pointing out that as someone who regularly uses TripleO, this won't work for me14:42
shardymaybe it's fine as a first revision tho14:42
shardylike, already today I've been having to manually delete the overcloud plan then do swift upload of my local templates14:43
shardyit's be much nicer to enable multiple plans and just switch between them14:43
shardyopenstack overcloud deploy --plan overcloud_dev12314:43
shardykinda similar to what we used to do with Tuskar14:43
rbradyshardy: is this something you've recently discovered? (the fact that this would not work for you)14:44
EmilienMshardy, slagle: do you know why delorean would fail to build tripleoclient when using depends-on? see
shardyrbrady: Yes, since I've been using the WIP mistral based deployment patches14:44
*** ramishra has quit IRC14:44
shardythey've only been semi-working since yesterday14:44
shardyjrist: what's the UI flow around this, does it assume only ever having one plan and one overcloud?14:45
jristjtomasek: can you comment on ^ - I believe that is *not* the case14:45
jristespecially considering OpsTools14:46
*** ramishra has joined #tripleo14:46
shardyrbrady: I'm not trying to be difficult btw, it's expected that we'll iterate on these interfaces until folks are comfortable14:46
jristthat sounds like a +2 to me? :)14:47
jristshardy: :)14:47
rbradyshardy: I'm not thinking that you personally are trying to be difficult.  I am frustrated that this detail was in 2 previously approved specs and was never discussed14:47
rbradyshardy: I could have implemented it the way you're suggesting earlier...being more efficient, etc14:48
*** fragatina has quit IRC14:49
*** myoung|mtg is now known as myoung14:49
*** fragatina has joined #tripleo14:49
openstackgerritMartin André proposed openstack/tripleo-validations: Validation to detect if undercloud disks use 512e
shardyrbrady: To be fair, this isn't directly addressed by the spec14:50
shardy"The deletion of a plan will be implemented in a Workflow to ensure there isn't an associated stack before deleting the templates,"14:50
shardythere's no mention of plan or stack multiplicity14:50
*** ramishra has quit IRC14:50
shardyrbrady: I'm cool to just land it if it's going to cause frustration - I'll post patches revising it for you consideration afterwards14:51
* jtomasek reads back14:51
shardyrbrady: FWIW my confusion may stem from the fact that we documented things like listing plans14:51
shardyif you can't select different plans, I'm not sure why we'd do that14:52
rbradyshardy: in this case, it's removing code.  I can take some time away from deployment to change this14:53
openstackgerritMartin André proposed openstack/tripleo-validations: Add network gateway validation
*** ramishra has joined #tripleo14:53
shardyrbrady: Don't take any time away from deployment, let's get that working and fix this later14:53
larskshey all.  what consumes tripleo-puppet-elements?14:53
shardyI'm just trying to have an open discussion having spent the last two days digging into this trying to help14:54
rbradylarsks: large purple monsters14:54
hewbroccaThe triple owl14:54
openstackgerritMartin André proposed openstack/tripleo-validations: Port the network environment validation
shardypurple owls?14:54
larsksSeriously though, what process actually applies those?14:54
hewbroccad-i-b, right?14:54
larsksBut what *passes* those to dib?14:55
openstackgerritLucas Alvares Gomes proposed openstack/instack-undercloud: Ironic: Configure iPXE to work with UEFI
rbradyshardy: I have added blueprints for the remaining work(that I know of) associated with tripleo workflows and actions and linked them to the primary bp:
shardylarsks: we use the puppet-modules one during image building to include puppet modules in the overcloud images14:55
shardylarsks: the hiera one is used on the overcloud nodes to deploy hieradata14:56
shardyso, again, it's built into the image via DiB14:56
*** Ryjedo has joined #tripleo14:56
larsksshardy: where does the image building happen? I am trying to track down an issue with the images used by tripleo-quickstart, and it's not clear to me whether those images use these elements...14:57
shardylarsks: tripleoclient calls DiB when you do openstack overcloud image build14:57
jtomasekshardy, rbrady: isn't the plan only something that acts as a name for a set of templates uploaded to swift? I thought we support the usecase shardy specified14:57
larsksshardy: thanks.14:57
shardylarsks: there is another tripleo-build-images tool too, but we're not using it upstream yet14:58
jtomasekshardy, rbrady I see now, so the heat stack name matches plan name...14:58
rbradyjtomasek: the plan consists of the swift container and all of its contents and the mistral environment which contains deployment specific data14:58
jtomasekis taht what the problem is?14:58
shardyjtomasek: the only problem is the plan delete workflow assumes a stack called "overcloud" can only ever be deployed from a container called "overcloud"14:59
jtomasekshadower: I see, hmm14:59
*** pradk_ has joined #tripleo14:59
jtomasekshardy: is 'overcloud' stak name somewhat mandatory from the TripleO deployment standpoint?15:00
shardyif we refine that slightly, we'll make it cleaner for folks wishing to do stuff like dev testing or staging workflows switching between different plans15:00
shardyjtomasek: No, it's completely configurable15:00
shardyAnyway, I didn't mean to make a big issue, just wanted to share some ideas15:00
jtomasekshardy: ok, yeah, your point makes sense15:01
*** fragatina has quit IRC15:02
pabelangerokay, so is a thing now. I have a patch up to start using tarballs.o.o for our cached images15:04
pabelangerrunning in the CI now15:04
*** paramite has quit IRC15:06
pabelangerlarsks: I think you're tripleo-quickstart-core, have a few minutes to review some patches?
*** akshai has quit IRC15:09
larskspabelanger: sure.15:09
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE TESTING
larskspabelanger: just need to finish a call first...15:09
pabelangerlarsks: great, thanks15:09
*** dprince has quit IRC15:10
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add cinder-backup service as role
*** ebarrera has quit IRC15:15
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: undercloud/upgrade: yum update of instack dependencies
EmilienMbnemec: I took over the tripleoclient patch, let me know if it's a problem and I can revert my change and do a separacted patch.15:15
*** julim has joined #tripleo15:16
EmilienMbnemec: I'm doing good progress15:16
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Retry stack-delete in case of failure
openstackgerritPaul Belanger proposed openstack/tripleo-quickstart: [WIP] Stop using data_files
*** bootsha has quit IRC15:17
*** akshai has joined #tripleo15:18
openstackgerritMartin André proposed openstack/tripleo-validations: Add a dynamic ansible inventory script
dtantsurfolks, I'm seeing a strange output of "introspection bulk start", something might be slightly wrong about the new workflows:
* dtantsur is looking whom to ping about it15:20
openstackgerritPaul Belanger proposed openstack/tripleo-quickstart: [WIP] Stop using data_files
*** hewbrocca is now known as hewbrocca-afk15:23
openstackgerritPaul Belanger proposed openstack/tripleo-quickstart: [WIP] Stop using data_files
*** bana_k has joined #tripleo15:26
openstackgerritPaul Belanger proposed openstack/tripleo-quickstart: [WIP] Stop using data_files
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Remove unused parameter in sahara
*** leanderthal is now known as leanderthal|afk15:37
pandahow do I access the overcloud nodes if the ssh key is not working ?15:39
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Remove unused parameter in sahara
*** jpeeler has quit IRC15:43
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Add Aodh composable roles
*** zoliXXL is now known as zoli|gone15:45
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: Implement pre_upgrade hook when deploying undercloud
mariosshardy: do you think you're ready to reinstate your +2 ? green ci and green run at - fyi when/if you have a moment thanks!15:50
*** zoli|gone is now known as zoli_gone-proxy15:50
openstackgerritRyan Brady proposed openstack/tripleo-common: Adds Plan List Action
*** aufi has quit IRC15:54
EmilienMbnemec, shardy: I just sent an email to ML about undercloud upgrade CI job status15:58
*** akshai has quit IRC15:59
*** ramishra has quit IRC15:59
EmilienMbnemec, slagle: you disagree on where to run systemctl stop openstack-*16:00
slagleEmilienM: i don't like that logic living in tripleoclient16:00
openstackgerritRyan Brady proposed openstack/tripleo-common: Adds Plan List Action
EmilienMslagle suggested me to move it to instack-undercloud but it looks like bnemec wants to keep it in tripleoclient16:00
slagleEmilienM: tripleoclient should not be the source of truth on how to "upgrade an undercloud"16:00
EmilienMI agree we should use isntack-undercloud16:00
bnemecslagle: Doing it in instack-undercloud means instack-undercloud has to know how to upgrade itself.16:01
bnemecWhich it can't because it's already running at that point.16:01
bnemecUnless we want to play the re-exec game.16:01
bnemecWhich we do not.16:01
slaglebnemec: no....16:01
slaglewe still need the yum update i-u in tripleoclient16:02
slaglebut the systemctl stop openstack* should be in i-u16:02
slaglewould that work?16:02
bnemecslagle: But why?  Now we've split the upgrade logic.16:02
slaglei don't consider upgrading i-u itself part of the logic to upgrade an undercloud16:03
slagleit's a precondition16:03
slagle"you need the latest versin of this package installed"16:03
slaglethat package contains the logic to upgrade the undercloud16:03
slagleconsider what would happen when we need to systemctl stop something else16:04
slaglehow would they get the latest tripeloclient to do that if it was in tripleoclient?16:04
slaglethey would have to have the latest16:05
slagleso just make all the "logic" in i-u and have tripleoclient make sure you have the latest of i-u16:05
*** jistr is now known as jistr|biab16:06
EmilienMI tend to agree with slagle's proposal, and that's what I'm currently implementing16:07
*** Ryjedo has quit IRC16:08
EmilienMbut we need to find a concensus16:08
bnemecTBH, I want to go back to PS 2 of my client patch.16:08
bnemecStop all, upgrade all, install undercloud16:08
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: Implement pre_upgrade hook when deploying undercloud
EmilienMbnemec: we CANT upgrade all16:09
EmilienMI already explained why...16:09
EmilienMbecause puppet already upgrade all later16:09
bnemecEmilienM: Where?16:09
shardymarios: -1!16:09
bnemecWhat does that have to do with it?16:09
EmilienMbut puppet needs to upgrade OpenStack packages first16:09
shardyonly joking, approved :)16:09
EmilienMso it can trigger things in catalog16:10
EmilienMlike dbsync at the right time, etc16:10
*** Ryjedo has joined #tripleo16:10
EmilienMif you run yum update -y before puppet run, I suspect some future issues with Puppet catalog16:10
EmilienMbnemec: but we can try it, if you think it's better16:10
EmilienMthe question is, where? in tripleoclient?16:11
bnemecThe db sync is fixed, and we need to sanely handle packages being updated outside puppet anyway.16:12
bnemecBecause people can and will do that.16:12
EmilienMok, let's say we update all before16:12
EmilienMI'm ok with it16:12
EmilienMI just need a concensus on where to do it16:12
EmilienMi-u or tc16:12
*** saneax is now known as saneax_AFK16:12
bnemecThis would have to be the client.16:13
mariosshardy: /me will be telling his grandkids about the day manila landed16:13
slagleso hardcoding a list of service names in the client?16:13
openstackgerritPaul Belanger proposed openstack/tripleo-quickstart: [WIP] Stop using data_files
mariosshardy: thanks!16:13
EmilienMmarios: lol16:13
bnemecslagle: No, all of openstack.  Plus neutron because they're a grandfathered in snowflake.16:13
openstackgerritOpenStack Proposal Bot proposed openstack/os-cloud-config: Updated from global requirements
EmilienMwe can start by this proposal, i'm fine with it16:14
EmilienMwe need to start somewhere anyway16:14
slaglebnemec: and rabbit? haproxy? httpd? :)16:14
EmilienMslagle: puppet should take care of it later16:14
EmilienMbnemec: did you figure WHY we need those "stop" ?16:14
EmilienMwhy Puppet catalog hangs?16:14
bnemecEmilienM: It's because the OpenStack service conf files aren't forward compatible.16:14
*** Ryjedo has quit IRC16:15
EmilienMslagle: no mariadb is the same in mitaka & newton16:15
EmilienMslagle: 10.x16:15
bnemecslagle: Shit, that's a good point.  We could hit the same problem in another service. :-(16:15
openstackgerritOpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements
*** dtantsur is now known as dtantsur|afk16:15
EmilienMwe can start by openstack-only services now..16:16
openstackgerritMerged openstack/tripleo-heat-templates: Enable Manila integration - as a composable controller service
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements
EmilienMbnemec, slagle: or better, I could even solve it in Puppet catalog16:17
EmilienMin the same catalog, stop all services before anything else in catalog16:17
*** Goneri has quit IRC16:18
EmilienMbut we have to consider that everytime a user would run puppet on the undercloud, services would stop/start16:18
EmilienMis it fine?16:18
snecklifterrhefner: hi, I've sort of hijacked
bnemecEmilienM: I think that's fine.  The services will start and stop anyway, right?16:18
EmilienMbnemec, slagle: in my proposal, we keep and I need to patch undercloud puppet manifest to stop services before. Is it fine?16:19
bnemecYeah, I'm convinced that this is right now.16:20
*** anshul has quit IRC16:20
EmilienMslagle: ?16:21
rhefnersnecklifter: hi, I saw some of your commits - thanks! I was pulled off of that for a little while and it's on my plate to resume later this week16:21
*** jistr|biab is now known as jistr16:22
snecklifterrhefner: ah ok, looks like latest build has stalled but hopefully its not far off :)16:22
rhefnersnecklifter: perfect!16:23
*** devvesa has quit IRC16:23
*** Ryjedo has joined #tripleo16:23
*** Ryjedo has quit IRC16:23
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Implement aodh profiles
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Add Aodh composable roles
*** ccamacho is now known as ccamacho|out16:26
*** fultonj has quit IRC16:26
*** Ryjedo has joined #tripleo16:27
*** tremble has quit IRC16:27
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Fix mistral profile steps
*** fultonj has joined #tripleo16:28
*** ohamada has quit IRC16:29
slagleEmilienM: i really don't like that16:29
jpichshardy, rbrady: Once I have a better understanding of how it needs to change I'm happy to try and give a hand with iterating over the plan deletion etc actions, if that helps. I just created a couple of patches to call these from the client today so I have a moderate understanding of what they do now, if nothing else! Just let me know.16:29
slaglewhy the reinstall?16:29
slagleEmilienM: we keep making everything slower and slower16:29
slagleEmilienM: i probably need to just review the spec16:29
shardyjpich: thanks!  I've taken the bug for now, but will happily pass it on if you've got bandwidth to do it16:29
*** akshai has joined #tripleo16:30
shardyjpich: lets get the basic workflow landed, then we can iterate on it16:30
EmilienMslagle: yes16:30
bnemecslagle: The spec probably needs an update now though.16:30
slagleEmilienM: that's going to probably be 100+ packages to reinstall16:30
*** jpena is now known as jpena|off16:30
slagledefinitely not a fan of that16:30
bnemecIt was written based on my assumption that we could do a mass update, which isn't true.16:30
EmilienMslagle: why 100+? where do you see that?16:30
slagleEmilienM: on an uc, how many get removed when you run that yum remove?16:31
slaglei won't guess, you can tell me :)16:31
EmilienMslagle: the tripleo packages?16:31
EmilienMos-, tripleo-* etc ,16:31
EmilienMaround 2016:31
EmilienMI have logs16:31
EmilienMone sec16:31
openstackgerritMerged openstack/tripleo-quickstart: Add minimal_pacemaker config
*** oshvartz has quit IRC16:31
bnemecEmilienM: I'm also concerned that we aren't properly updating the dependencies doing it this way.16:32
bnemecI left a comment on the client review.16:32
*** akshai_ has joined #tripleo16:32
EmilienMslagle: ok more but really fast
*** tremble has joined #tripleo16:32
slaglesure, in CI with a local mirror16:32
slaglei'll just review the patch16:32
EmilienMslagle: less than ONE minute to remove/install tripleo deps16:33
shardyweren't we talking about just stopping services, not reinstalling packages?16:33
*** dprince has joined #tripleo16:33
EmilienMI'm about to drop off for lunch, feel free to review and give insights. we need to find concensus here16:33
*** snecklifter has quit IRC16:33
*** Ryjedo has quit IRC16:34
slagleEmilienM: which is why we have a spec right?16:34
jpichshardy: Sure! Subscribed to , I'll keep an eye on it16:34
openstackLaunchpad bug 1609454 in tripleo "Workflows Assuming a Single Plan is Associated with a Single Stack" [Medium,Triaged] - Assigned to Steven Hardy (shardy)16:34
slagleEmilienM: we might consider letting that play out.16:34
EmilienMsure, I'll let the spec make progress16:35
* EmilienM afk16:35
*** akshai has quit IRC16:35
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Composable Mistral services
*** rcernin has quit IRC16:38
*** dsariel has quit IRC16:39
*** ramishra has joined #tripleo16:43
*** lucasagomes is now known as lucas-dinner16:45
*** devvesa has joined #tripleo16:46
*** devvesa has quit IRC16:47
openstackgerritSteven Hardy proposed openstack/puppet-tripleo: Remove keystone PKI cert generation
*** Ryjedo has joined #tripleo16:50
*** aviramb has quit IRC16:52
*** oshvartz has joined #tripleo16:52
shardyd0ugal: Hey, are you looking at adding support for user environments to the deploy workflow?16:52
*** krotscheck is now known as kro_focused16:53
shardyd0ugal: I've hit a barrier with testing because .tripleo/environments/puppet-modules-url.yaml no longer works16:53
*** Ryjedo has quit IRC16:55
d0ugalshardy: I am trying, but not having much luck figuring out how to fit it in16:57
*** Ryjedo has joined #tripleo16:58
d0ugalrbrady: Had you thought how users would provide extra environment files?16:58
shardyseems like we either need to push them to the swift container, then reference them in the environment list defined in the mistral environment, or put the whole content in the environment16:59
shardyare there limits on the size of mistral environments?16:59
d0ugalshardy: Not that I can see, it is just a text field.16:59
*** pcaruana has quit IRC17:00
shardyit still might be cleaner to push them to swift I guess, then all template stuff is in one place17:00
d0ugalYeah, I think so17:00
d0ugalwe really need all the crud operations for plans17:00
shardyProcessTemplatesAction looks like it will do the right thing if we add the swift urls to the environment "environments" list17:01
d0ugalshardy: right, so add it to swift and then add it to the list of envs to be processed17:01
shardyI'm not sure how we differentiate between environments that are already in the plan data, vs ones that are defined elsewhere17:02
shardymaybe just compare the dirname17:02
d0ugalshardy: yeah, it gets tricky as users at the moment can give us a path to an environment anywhere17:02
d0ugalwe might need to do something to give them a unique name17:03
shardyopenstack overcloud deploy --templates /tmp/tht/ -e /tmp/tht/environments/enable_foo.yaml -e ./override_foo.yaml17:03
*** jbadiapa has quit IRC17:03
*** tesseract- has quit IRC17:03
shardyd0ugal: probably just creating a "user_environments" subdir in the container would be enough?17:03
d0ugalshardy: if they are both called enable_foo.yaml it becomes more fun :)17:03
shardyd0ugal: the environment path needs to be fully qualified and match either a swift URL or something in the files map17:04
shardyd0ugal: Yeah, it'd be something like push user_environments/enable_foo.yaml17:04
shardyto swift17:04
d0ugalWhy do I feel like this is going to be the source of weird edge cases :)17:05
shardythen add path: http:<swifturl prefix>/user_environments/enable_foo.yaml to the mistral env17:05
*** openstackgerrit_ has joined #tripleo17:06
shardyd0ugal: actually, do mistral environment-get overcloud17:07
shardythere are no prefixes17:07
shardyso they must be added elsewhere, because we do download overcloud-resource-registry-puppet.yaml from swift (ref bug breaking deployment yesterday)17:07
*** openstackgerrit_ has quit IRC17:08
shardyah, yeah, we join them in the ProcessTemplatesAction17:08
d0ugalshardy: I'm going to have to call it for today.17:10
d0ugalI think I can get something together quickly for tomorrow17:10
shardyd0ugal: WHAT?17:10
shardyOnly joking, have a great evening :)17:10
* shardy needs to do likewise soon17:10
d0ugalshardy: I am not a force of nature like you :)17:10
*** mgould is now known as mgould|afk17:11
*** akshai_ has quit IRC17:11
*** Ryjedo has quit IRC17:12
*** yamahata has quit IRC17:15
*** Ryjedo has joined #tripleo17:22
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Set Coordination url for central agent
*** bootsha has joined #tripleo17:26
openstackgerritPaul Belanger proposed openstack/tripleo-quickstart: [WIP] Stop using data_files
*** Ryjedo has quit IRC17:27
*** fzdarsky is now known as fzdarsky|afk17:32
*** abehl has joined #tripleo17:33
* bnemec has a fruit fly problem in his office17:33
bnemecWhich I don't understand.  I'm very careful not to leave open food lying around for exactly that reason.17:34
*** abehl has quit IRC17:34
openstackgerritMerged openstack/tripleo-quickstart: Fix E226 missing whitespace around arithmetic operator
openstackgerritMerged openstack/tripleo-quickstart: Use ansible 2.1 importing of boilerplate code
openstackgerritMerged openstack/tripleo-quickstart: Fix H102 Apache 2.0 license header not found
openstackgerritPaul Belanger proposed openstack/tripleo-quickstart: [WIP] Stop using data_files
openstackgerritJames Slagle proposed openstack/tripleo-docs: Document additional NODE vars
*** shardy has quit IRC17:44
*** bana_k has quit IRC17:45
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart: Fix wrong naming of image's name in release.
rbradyshardy, d0ugal: I think one thing to think about is how environments are chosen (UpdateCapabilities).  We need to update that code to ensure it provides any additional envs from either swift or stored in the mistral environment.variables['environments'] list17:48
rbradyd0ugal: I had been thinking of adding an action to support adding environment data to a plan, but it has not been requested as a feature that I know of yet17:49
openstackgerritJames Slagle proposed openstack/tripleo-docs: Document additional NODE vars
EmilienMslagle: what is your suggestion about ? I'm not sure to understand17:52
*** milan has quit IRC17:53
*** sarath has joined #tripleo17:53
socialEmilienM: why it's part of tripleo client?17:53
*** yamahata has joined #tripleo17:53
*** julim has quit IRC17:53
slaglehere we go :)17:54
EmilienMsocial: that's what we are discussing right now17:54
EmilienMslagle: so should we remove the UpgradeUndercloud function?17:54
sarathdprince, bnemec: could you kindly review the os-net-config PR at
EmilienMerr, class17:54
slagleEmilienM: i'd like to know why17:54
slagleEmilienM: i asked in the review17:54
slaglewhy the remove/install17:54
slaglei admit i probably do not fully understand all the issues, as I've not yet read the spec17:55
slaglehowever, my initial thinking is to upgrade the packages that provide whatever code would already be running/loaded when instack-install-undercloud is executed17:55
slagleessentially, what i-u can't upgrade itself w/o re-exec'ing17:56
socialEmilienM: shouldn't it at max take care of installing/upgrading instack-undercloud and puppet,opm&deps upgrade should be present in instack-undercloud?17:56
slagleand then let i-u upgrade everything else17:56
socialEmilienM: and rest update in puppet run17:56
*** julim has joined #tripleo17:56
slagleEmilienM: and i-u can upgrade "everything else" however it wants. via puppet, etc17:57
*** bana_k has joined #tripleo17:58
slaglenow...the other use case that we've said we want to support is if someone externally runs "yum update", and then rerun openstack undercloud install17:59
slagleit should work17:59
slaglei do not thing we necessarily have to immediately solve both of those in the same patch17:59
EmilienMI have to admit I'm really confused right now18:00
socialslagle: you don't resolve external yum update that's all18:00
slaglei dont understand what you're saying18:02
slagleit doesn't work now? it will never work?18:03
openstackgerritBen Nemec proposed openstack/tripleo-specs: Undercloud Upgrade
*** tosky has quit IRC18:09
*** akrivoka has quit IRC18:10
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Add 'openstack overcloud plan list' command
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Add 'openstack overcloud plan delete' command
*** jdob has quit IRC18:11
*** jdob has joined #tripleo18:11
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton
EmilienMbnemec, slagle: ok: I pushed over ^ to come back to my PoC without tripleoclient or i-u changes.18:14
EmilienMso in the code, we know what we need to and decide where to put it later18:14
*** Ryjedo has joined #tripleo18:15
*** yamahata has quit IRC18:18
*** yamahata has joined #tripleo18:19
*** mcornea has quit IRC18:19
*** jdob has quit IRC18:20
*** fzdarsky|afk is now known as fzdarsky18:20
*** jdob has joined #tripleo18:20
*** bootsha has quit IRC18:22
*** jpich has quit IRC18:22
*** electrofelix has quit IRC18:22
*** Ryjedo has quit IRC18:23
EmilienMDevice "eth1" does not exist.18:25
EmilienMon internap cloud18:25
EmilienMslagle: do we still have this bug in master CI? I thought it was fixed in instack-undercloud18:25
*** bootsha has joined #tripleo18:26
ccamacho|outEmilienM, slagle mind to review this? its about to remove an unused param, a harmless submission.18:27
*** rwsu has quit IRC18:27
*** oshvartz has quit IRC18:28
EmilienMccamacho|out: +218:28
*** jaosorior has joined #tripleo18:29
*** jaosorior has quit IRC18:30
openstackgerritDan Sneddon proposed openstack/tripleo-specs: Spec: undercloud-ntp-server - Add NTP services to Undercloud
EmilienMI see the eth1 issue on ovh too18:32
gfidenteccamacho|out, sounds like we don't enable sahara by default18:32
gfidenteis this intended?18:32
gfidentethrash, do you know anything about ^^ ?18:37
gfidentewe had in the default setup in mitaka18:37
ccamacho|outgfidente, not sure  (::sahara::service::api::enable its defaulted to true)but from what I remember they were all enabled by default, which I18:37
gfidenteccamacho|out, oh I meant we don't enable the role on the controllers18:37
ccamacho|outthink at the end we should provide a naked overcloud18:38
openstackgerritMerged openstack-infra/tripleo-ci: Add opstools candidate repo
*** ayoung has quit IRC18:38
*** ccamacho|out is now known as ccamacho18:39
ccamachogfidente, you are right there.. its not enabled18:39
ccamachommmm need to get the submission to see if it was tested.18:39
ccamachoill take notes18:39
rookEmilienM re the glance-api patch for # of workers. Leaving a empty string will make glance-api use the default # of cores.18:41
rookEmilienM: is THT doing something eles if workers= ?18:42
*** oshvartz has joined #tripleo18:42
ccamachogfidente, nope it wasnt enabled, /me taking notes and Ill test it tomorrow18:42
EmilienMrook: look the logs, you'll see workers = 118:43
ccamachogood catch18:43
thrashgfidente: it's supposed to be18:43
EmilienMrook: it sounds like it take the default in puppet-glance18:43
EmilienMpradk: hey about are you sure about the format?18:43
thrashgfidente: I thought we had a patch to fix that tho18:43
EmilienMredis://password@host:port ?18:43
gfidentethrash, oh right I remember that18:44
gfidenteso it needs rebase and such18:45
thrashgfidente: yep18:45
thrashI'll work on that.18:45
ccamachothrash++ nice, Ill add it to my queue for tomorrow then18:45
pradkEmilienM, yes format is correct.. my local install in progress.. i can confirm if coordination works once thats done18:45
rookEmilienM: i see nothing in puppet-glance that would set it to one, unless the # of vCPUs is 118:46
*** julim has quit IRC18:46
rookEmilienM puppet-glance defaults to # of cores.18:46
gfidentetty tomorrow guys18:46
rookjust like every other service.18:47
rookor most other services*18:47
EmilienMpradk: ack18:47
EmilienMrook: the default value in puppet-glance is the number of processors18:47
EmilienMrook: right !18:47
EmilienMso I guess it takes it18:47
rookEmilienM: does our CI deploy things with > 1vCPU? if so, then you are right, this patch isn't good.18:48
rookhowever, if they deploy with 1vcpu then we are fine.18:48
*** bootsha has quit IRC18:48
*** bootsha has joined #tripleo18:49
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: Split Sahara pacemaker roles into separate services
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: Add Sahara services to ControllerServices list
EmilienMrook: /me reading logs18:55
EmilienMrook: I confirm, 118:55
*** cwolferh has quit IRC18:56
sarathdsneddon: os-net-config IVS cleanup patch as you had requested:
openstackgerritMerged openstack/instack-undercloud: Use certmonger for automatic cert generation
rookEmilienM: looking at the nova logs, it they only started a single worker too18:57
*** paramite has joined #tripleo18:59
gfidentethrash, I think the dependency goes in the other way around no?19:01
gfidenteit's which depends on
openstackgerritMerged openstack/tripleo-heat-templates: We don't need to set a default for the CephX keys and cluster FSID
*** cwolferh has joined #tripleo19:02
thrashgfidente: huh?19:04
thrashgfidente: nope19:04
gfidentewell we only enable sahara in
gfidentebut we deploy it in pacemaker19:05
thrashgfidente: 336119 has some bug fixes19:05
thrashgfidente: are you mixing up the puppet submission with the THT submission for the split?19:05
gfidenteyeah but fixes for the non-pacemaker scenario we don't test19:05
thrashgfidente: I don't follow...19:06
gfidenteno my point is that will enable sahara, but gate is doing pacemaker scenario, so it'll try to run sahara without the fix in
gfidenteand it will probably fail because of that19:06
*** fultonj has quit IRC19:08
thrashwe can squash them.19:08
thrashgfidente: if that helps. :)19:08
gfidentesquash or invert the order19:09
thrashgfidente: I know I can't invert the order. 327722 will not work without the fixes in 33611919:09
gfidenteat which point will always pass because we don't test it19:09
thrashwhich is also why we inverted it in the first place.19:10
thrashdidn't want to merge something that we knew wasn't tested.19:10
gfidenteright but it'll be tested by
thrashgfidente: I'll just squash it. It will make it easier.19:12
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: Add Sahara services to ControllerServices list
thrashgfidente: there you go.19:18
gfidentelet's merge it :)19:18
*** tbarron has joined #tripleo19:20
pabelangerbnemec: EmilienM: slagle: tripleo-test-cloud-rh1 online now, jobs stopping on tripleo-test-cloud-rh2. 43 jobs currently coming online19:20
*** radek_ has joined #tripleo19:21
*** maeca1 has joined #tripleo19:28
bswartz^ I'm trying to run tripleo-quickstart and it's failing with permission denied creating a directory19:28
tbarron^^ marios if you are there we're trying to get off the ground for the netapp/manila plugin now that merged (w00t!)19:29
*** tesseract- has joined #tripleo19:31
rookEmilienM: so we should be good to push that lil patch19:34
*** jcoufal has quit IRC19:34
pabelangerslagle: EmilienM: bnemec: still waiting for new nodes to launch in tripleo-test-cloud-rh1.  Been 20mins in building start, anybody able to see what is going on ?19:36
*** tesseract- has quit IRC19:37
bnemecpabelanger: I see environments spinning up.  A lot of slaves in the BUILD status.19:38
pabelangerbnemec: 36, been about 24mins now19:38
*** dsneddon has quit IRC19:39
*** bootsha has quit IRC19:39
bnemecpabelanger: Yeah, it seems a bit slow.  It's possible the controller is freaking out a bit about trying to spin up so much at once.19:39
*** dsneddon has joined #tripleo19:39
*** rwsu has joined #tripleo19:40
pabelangerokay, we'll see what happens19:40
bnemecpabelanger: I'm seeing quite a bit of glance and swift activity on the controller.  We may be bottlenecked getting the images cached on all the compute nodes.19:42
pabelangerbnemec: ya, that is likely the issue. Our image is 5GB19:43
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add cinder-backup service as role
*** ayoung has joined #tripleo19:45
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add undercloud job in cistatus
*** Goneri has joined #tripleo19:51
dsneddonsarath, Thank you very much!19:58
*** dprince has quit IRC19:58
*** maeca2 has joined #tripleo20:01
pabelangermetrics_to_graphite "" #Dan's temp graphite server20:01
pabelangerWho is Dan in that comment?20:01
pabelangerthat is from tripleo-ci?20:01
pabelangerdprince I suspect20:01
*** maeca1 has quit IRC20:04
bnemecpabelanger: Yeah, dprince20:06
bnemecI'm seeing a bunch of 404's on the bmc and ipxe images in the rh1 glance logs.20:06
bnemecWhich I don't understand because some of the envs stood up fine.20:06
d0ugalrbrady: every CLI user provides environments when deploying I think - so it should be requested :)20:08
pabelangerbnemec: thanks, I'll ping him tomorrow about it. I'd like to understand what its use-case is and how to replace it for something in openstack-infra20:08
bnemecpabelanger: It's just for tracking how long various parts of the CI deployments take over time.  If there's an infra place we can do that instead I think everyone would be fine with it.20:09
*** Goneri has quit IRC20:10
pabelangerbnemec: right, talking in openstack-infra about it, it seems subunit2sql with stackviz could provide the same info20:10
*** coolsvap has quit IRC20:11
slaglepabelanger: is there a concern with keeping rh2 in nodepool longer?20:17
EmilienMpabelanger: we use it in puppet CI20:17
slaglepabelanger: i don't feel that the pressure to get it removed is coming from the tripleo side20:17
bnemecHey, the rh1 slaves have come up.20:17
slaglepabelanger: we have the rack for as long as we reasonably need accourding to weshay20:18
weshayI'm a nice guy.. what can I say ;)20:19
rbradyd0ugal: do you want a register a blueprint for it and document in the summary how the CLI provides this to the operator so we can duplicate it with an action or workflow?20:20
pabelangerslagle: that is a question for derekh, as I understood it the hardware was getting reclaimed20:20
pabelanger31 nodes in tripleo-test-cloud-rh1 ATM20:21
*** tremble has quit IRC20:21
weshaypabelanger, slagle I suspect when the rdocloud comes up we can get it back20:21
pabelangerEmilienM: right, tripleo-ci should be using that too over graphite20:21
d0ugalrbrady: I can look into doing that tomorrow, but if you want it now, just look at the -e arg for `openstack overcloud deploy`20:21
pabelangerweshay: I can ping ggilies about that tonight20:22
slagleok. well i guess if rh1 is coming up fine, we can keep it shut off20:22
slaglebut i think we'll need a couple of days with things running at normal capacity to confirm that20:23
EmilienMand since couple of days means Friday, I suggest to wait Monday.20:23
*** sarath has quit IRC20:23
pabelangeryes we can do that20:24
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: add temp new release to verify centos7-new
pabelangerslagle: infact, all jobs in check-tripleo are running right now20:24
EmilienMpabelanger: I  have a new experimental job request:
*** bootsha has joined #tripleo20:28
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Import CI THT environments from tripleo-ci
openstackgerritJames Slagle proposed openstack/tripleo-docs: Add all puppet modules to includepkgs
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Import CI THT environments from tripleo-ci
openstackgerritThomas Herve proposed openstack/tripleo-common: Ignore changes to images in nova servers
pabelangerhow is overcloud-full.qcow2 build today?20:36
pabelangeractually, where does the build happen20:37
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Remove test-environments from tripleo-ci
slaglepabelanger: it's built on the undercloud20:38
EmilienMbnemec, slagle: you're probably going to -2 me but this is a start:
*** jayg is now known as jayg|g0n320:38
EmilienMfeedback is welcome20:38
slaglepabelanger: during the CI job. if we determine we can't use the cache20:38
*** paramite has quit IRC20:38
slaglei think the periodic job always builds as well20:38
bnemecActually they're all building images right now because of the switch over from source puppet to packaged puppet.20:40
bnemecIf we updated the cache since that change was made we can try undoing that though.20:41
openstackgerritMerged openstack/puppet-tripleo: Remove unused parameter in sahara
ayoungbnemec, slagle I'm trying to do SSL as per and something seems to be missing, but I can fix it by hand.  However, it leaves the overcloud deploy in a 1/2 done state.  How do I say "continue deploying from where it was broken?"20:43
slaglei dont think you can, for various reasons20:44
slagleit's not going to do the post deployment endpoint creation if it's already failed20:45
slagleand it's also likely to blow away your manual fix, unless you made that fix on the template side20:45
bnemecActually, there's a CLI switch to force postconfig now:
slaglei'll be a hot dog20:46
bnemecI wanted it on by default.20:46
bnemecBut I got shouted down as usual.20:46
bnemecNot that I'm bitter or anything. :-P20:47
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: add temp new release to verify centos7-new
slaglewhere is the patch to move all this to puppet anyway?20:49
EmilienMcan we enable again the tripleo upgrade job?20:49
EmilienMnow we have more nodes20:49
bnemecI'm betting it doesn't work.20:49
bnemecnet-iso is broken20:49
ayoungslagle, so, what is happening is that the cert is not getting added to the list of trusted certs on the controller.  I can add that by hand, and I would not expect anything to undo that20:49
bnemecAnd I think net-iso is used in the upgrade job.20:50
EmilienMbnemec: why is it broken? what is broken?20:50
bnemecayoung: You should not have to do that.20:50
ayoungshould I just re-run openstack overcloud deploy20:50
slagleayoung: ok20:50
ayoungbnemec, I know, but Juan is not back until morning20:50
bnemecEmilienM: I don't know, but Ceph refuses to start on my net-iso re-enablement patch.20:50
ayoungand he and I will work through it then.  I think the rest of the deploy is OK, would like to be able to keep working on it20:50
slagleayoung: yes, but you need to pass --force-postconfig as well20:50
ayoungslagle, ++20:50
slaglewhen you rerun20:51
ayoungI can do that...I have that technology!20:51
slaglesince what I told you yesterday worked, i would expect that today this will blow up20:51
EmilienMbnemec: I don't mind if it fails in CI at least we can see the failure online and not on your devenv20:51
ayoungit fails very late in the process.  THe funny thing is that the remote systems are OK with the cert, it is just controller calling itself via https that fails, which is a strange path20:51
bnemecEmilienM: Have at:
bnemecI'm actually in the process of trying to reproduce it locally so I can debug.20:52
EmilienMbnemec: do you remember what disabled the upgrade job? was it in project-config?20:52
bnemecEmilienM: I would assume so.20:52
EmilienMok so let's get net-iso back first, and then we'll enable upgrade job I guess20:53
EmilienMbnemec: I'm looking at logs20:53
EmilienMError: /Stage[main]/Ceph::Osds/Ceph::Osd[/srv/data]/Exec[ceph-osd-activate-/srv/data]/returns: change from notrun to 0 failed: Command exceeded timeout\20:54
EmilienMI thought we were running Jewel20:54
EmilienMwe need to change that path on Jewel, otherwise we'll have SElinux alerts20:54
EmilienMnot sure it's related to the patch though20:54
EmilienMbnemec: wow you're adding a lot of templates..20:55
EmilienMwhile I'm moving them away from tripleo CI20:55
bnemecEmilienM: I have no idea, but I'm pretty sure they're all failing basically the same way.20:55
gfidenteEmilienM, which job is printing that?20:55
EmilienMI thought we agree to move them20:55
gfidentecause jewel is running now from an entire day20:55
bnemecEmilienM: I just -1'd the t-h-t patch. :-P20:56
gfidenteso I am surprised it would pop up only now20:56
bnemecgfidente: I'm assuming it's because that patch turns back on net-iso.20:56
gfidentebnemec, but the error seems unrelated though20:56
EmilienMbnemec: how do you want to branch the templates without moving them?20:57
EmilienMthe worker-config-mitaka-and-below.yaml is a proof that we need to move them20:57
EmilienMhave bash conditionals for every release is a terrible idea20:57
EmilienMwe have stable branches20:57
pabelangerslagle: is it python-tripleoclient that builds it?20:57
gfidenteEmilienM, bnemec oh command exceeded timeout20:58
gfidentethen yes looks like OSDs couldn't reach the MONs20:58
gfidenteit's not the selinux thing20:58
bnemecEmilienM: These are templates that users write. If we're breaking them then we should absolutely be feeling that pain.20:58
bnemecBecause all of our users are too.20:58
gfidenteEmilienM, in ci centos runs with selinux in permissive mode anyway20:59
gfidenteEmilienM, which is why we aren't blocked on the selinux/path problem20:59
EmilienMbnemec: I don't think we can maintain backward compatibily in templates for life20:59
EmilienMbnemec: worker-config-mitaka-and-below.yaml is a proof, again.20:59
bnemecEmilienM: That's fine.  When we break compatibility, we'll appreciate that we've just caused our users pain because we'll have to deal with it too.20:59
bnemecThis is a feature, not a bug IMHO.20:59
pabelangerslagle: looks like this is the code that does it:
EmilienMbnemec: can you look my comments on and tell me if I can go ahead?21:04
gfidenteweshay, do we have to do anything in quickstart to cover ?21:04
weshaygfidente, last I heard this morning ceph deployments were failing this morning21:07
weshaygfidente, sec21:07
bnemecEmilienM: That's fine.  I thought that's what we had agreed on in the ML discussion.21:07
gfidenteweshay, in upstream ci ?21:07
EmilienMbnemec: cool21:07
*** liverpooler has quit IRC21:08
gfidenteweshay, and is rdo using quickstart?21:10
gfidenteso yes they will fail until we got ^^ those repos21:10
*** fzdarsky has quit IRC21:10
weshaygfidente, I can answer in a bit21:10
gfidenteinto quickstart21:10
*** lblanchard has quit IRC21:12
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Import CI THT environments from tripleo-ci
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Remove test-environments from tripleo-ci
EmilienMbnemec: done ^21:13
EmilienMwe should really converge the tools here21:14
* bnemec points out that he already wrote, tested, and proposed that tool:
bnemecAnd got a resounding "meh" in response.21:15
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton
bnemecAlthough now that we're adding a bunch of other repos, it may have to be renamed tripleo-repo.21:19
EmilienMbnemec: the tool looks cool, why not using it in our CI?21:20
* EmilienM off for a few hours from now21:23
*** Ryjedo has joined #tripleo21:24
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: add temp new release to verify centos7-new
openstackgerritBob Fournier proposed openstack/tripleo-heat-templates: Add management network to list of networks to ping
bnemecEmilienM: There's a bunch more work required to import it to gerrit, package it, and integrate it into the flow.  I didn't want to do all of that without some indication that it was something people thought we should do.21:27
*** radek_ has quit IRC21:29
SlowerI think the "Enable latest RDO Trunk Delorean repository only for the TripleO packages" step is wrong now in the install docs21:30
Slowerthe includepkgs list does not include all the puppet packages21:30
pabelangerany suggestions on my error with python-tripleoclient?21:31
pabelangerERROR: Element 'overcloud-full' not found in '/usr/share/tripleo-puppet-elements:/usr/share/instack-undercloud:/usr/share/tripleo-image-elements:/usr/share/openstack-heat-templates/software-config/elements:/usr/bin/../share/diskimage-builder/elements'21:31
pabelangerI am not sure which project provides overcloud-full21:31
*** rhallisey has quit IRC21:32
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: add temp new release to verify centos7-new
*** bootsha has quit IRC21:35
bnemecpabelanger: It's in instack-undercloud:
openstackgerritDan Sneddon proposed openstack/os-net-config: Disable IPv6 Autoconf When use_dhcpv6=False
*** chem has quit IRC21:39
rlandybnemec: Running tripleo-quickstart install on OVB from master, I get  "Connection timed out (" when getting agent.kernel. Introspection gets stuck - repeating the error message over. Running a mitaka install on the same env works. Any thoughts on what's going wrong with master?21:40
*** rwsu has quit IRC21:40
*** dbecker has quit IRC21:42
*** tbonds has quit IRC21:43
weshaygfidente, sorry back21:44
bnemecrlandy: Not really.  It looks like it's working in CI on master.  I don't tend to run it locally because I rarely need introspection data.21:44
rlandybnemec: thanks  - I am wondering if I need bigger instances for master than mitaka - master passed on real baremetal hardware21:46
bnemecrlandy: Introspection shouldn't require much in the way of resources.  It's just collecting data, not actually deploying anything.21:47
EmilienMbnemec: for we are fucked, because the templates are not in THT stable branches21:48
*** bootsha has joined #tripleo21:49
*** limao has joined #tripleo21:49
*** limao has quit IRC21:50
*** limao_ has joined #tripleo21:50
gfidenteweshay, so I think we need to bring those repo changes into quickstart21:51
weshaygfidente, k.. the centos ceph hammer is in.. at least it is in our rdo tests internal.. thanks I'll double check at the ci.centos level21:52
weshaygfidente, thanks!21:53
pabelangerbnemec: thanks, took 20mins but it is working21:53
pabelanger$ openstack overcloud image build --type overcloud-full --elements-path="/home/centos/instack-undercloud/elements:/home/centos/tripleo-puppet-elements/elements:/home/centos/tripleo-image-elements/elements:/home/centos/heat-templates/hot/software-config/elements"21:53
gfidenteweshay, I am not sure where to add those changes21:54
weshayimage building.. np21:54
weshayI got it21:54
gfidenteok unfortunately centos/jewel isn't finished yet but that is what we need on master21:54
gfidenteadd me on the reviews21:54
pabelangerbnemec: EmilienM: Who should I be talking to about gate jobs for python-tripleoclient?21:54
weshaygfidente, ack21:55
*** morazi has quit IRC21:56
*** rlandy has quit IRC21:59
bnemecpabelanger: I'm not sure there's anyone specific.  What are you looking to add?22:00
*** rwsu has joined #tripleo22:03
*** florianf has quit IRC22:03
pabelangerbnemec: I think we could add a job to build overcloud-full for example22:06
bnemecpabelanger: We could, but it still has to be run through CI to make sure it works before we publish it.22:07
*** ebalduf has quit IRC22:08
pabelangerbnemec: I think there is a few use cases. 1st being, can we build an overcloud-full properly with all the dependency.  Which I think could happen in python-tripleoclient. Then the 2nd would be, can tripleo-ci use that image properly22:09
pabelangerthe 2nd would be for the purpose of using cached artifacts for CI jobs, to make things faster22:10
bnemecpabelanger: We already do all of that though.22:11
bnemectripleoclient changes get tripleo-ci run on them, which rebuilds the image.22:12
bnemecPatches that can't affect the image (like tripleo-heat-templates) use a cached image.22:12
bnemec(well, they did before we shut it off while we transition to packaged puppet modules)22:12
*** rwsu has quit IRC22:13
pabelangerRight, I am not sure how some of this works in tripleo-ci, but up until recently it was only running on tripleo-test-cloud-rh1.  My thinking is, if we reorg some of the things we do in tripleo-ci, into external jobs.22:14
pabelangerit also means we don't need to setup devstack everytime to test a patch to python-tripleoclient in the gate22:15
pabelangeras an example:22:16
pabelangeropenstack overcloud image build --type overcloud-full --elements-path="/home/centos/instack-undercloud/elements:/home/centos/tripleo-puppet-elements/elements:/home/centos/tripleo-image-elements/elements:/home/centos/heat-templates/hot/software-config/elements" --debug22:16
pabelangerfails for me locally22:16
pabelangerand I don't know why22:16
*** fultonj has joined #tripleo22:18
*** Ryjedo has quit IRC22:23
*** rwsu has joined #tripleo22:24
*** Ryjedo has joined #tripleo22:24
*** Ryjedo has quit IRC22:25
*** Ryjedo has joined #tripleo22:25
*** Ryjedo has quit IRC22:25
*** Ryjedo has joined #tripleo22:26
*** Ryjedo has quit IRC22:26
*** Ryjedo has joined #tripleo22:27
*** Ryjedo has quit IRC22:27
*** Ryjedo has joined #tripleo22:28
*** Ryjedo has quit IRC22:28
*** Ryjedo has joined #tripleo22:29
*** julim has joined #tripleo22:37
*** bfournie has quit IRC22:38
*** rwsu has quit IRC22:40
*** Ryjedo_ has joined #tripleo22:40
*** Ryjedo has quit IRC22:40
*** Ryjedo_ is now known as Ryjedo22:40
*** rwsu has joined #tripleo22:45
*** limao has joined #tripleo22:48
*** limao_ has quit IRC22:51
*** limao has quit IRC22:51
*** limao has joined #tripleo22:51
*** limao_ has joined #tripleo23:12
*** limao has quit IRC23:16
*** limao_ has quit IRC23:26
*** tobias_fiberdata has joined #tripleo23:30
slaglepabelanger: before we publish images, we want to make sure they work. which is why that happens as part of tripleo-ci23:31
slagleand we want to gate changes to tripleoclient on the condition that the built image has been used successfully by tripleo-ci23:32
slaglei guess you're saying just move the publishing part as part of the publish job on tripleoclient?23:33
*** Ryjedo_ has joined #tripleo23:33
slaglei'm not sure how those jobs work, but as long as we're publishing the same qcow2 that was used in tripleo-ci, it wfm23:34
*** Ryjedo has quit IRC23:35
*** gfidente has quit IRC23:37
tobias_fiberdataNice, you have updated the documentation with mitaka repo! Very nice23:37
tobias_fiberdataNow i'll go sleep :p23:37
*** tobias_fiberdata has quit IRC23:37
pabelangerslagle: Ya, I'll have some code tomorrow and possible experimental job23:38
slaglethe last time i looked at the publish jobs...i couldn't see a way to tie it in23:38
pabelangerslagle: the biggest benefit, it could become voting once stable23:38
*** Ryjedo_ has quit IRC23:38
*** gfidente has joined #tripleo23:38
pabelangerwhich is something you couldn't do running them under tripleo-test-cloud-rh123:38
*** bootsha has quit IRC23:38
*** gfidente has quit IRC23:38
*** gfidente has joined #tripleo23:38
slagletrove for instance, just publishes an image on a commit to their elements23:39
slaglebut i dont know what tests that said image actually works23:39
pabelangerya, a few projects are building images on commit23:39
pabelangerand then publishing in the post pipeline23:39
pabelangerright now, I'd just like to figure out the buildimage part23:40
pabelangerthen have the discussion on how to publish them, if needed23:40
pabelangernow to find some food23:41
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton
*** tobias_fiberdata has joined #tripleo23:45
*** jrist has quit IRC23:49
*** maeca2 has left #tripleo23:55
*** tobias_fiberdata has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!