Tuesday, 2016-09-20

*** dtrainor has joined #tripleo00:00
ayoungrcrit,  what is the hack to get an overcloud redeploy to work?  Aside from deleting the overcloud stack, we had to do something else?00:04
ayoung    openstack stack delete overcloud --yes00:04
ayoung    swift delete overcloud00:04
ayoung    mistral environment-delete overcloud00:04
ayoungthere it is00:04
*** sshnaidm has quit IRC00:19
*** r2bit has quit IRC00:29
*** limao has joined #tripleo00:29
*** limao has quit IRC00:29
*** limao has joined #tripleo00:29
*** r2bit has joined #tripleo00:30
openstackgerritOpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements  https://review.openstack.org/37281400:32
openstackgerritMerged openstack/tripleo-common: Fix upload-swift-artifacts work new OSclient  https://review.openstack.org/36291500:35
*** dmacpher has quit IRC00:48
openstackgerritMerged openstack/diskimage-builder: yum-minimal: Disable excludes when installing pkg manager  https://review.openstack.org/37280800:57
*** rbrady_ has joined #tripleo01:02
*** rbrady has quit IRC01:02
*** saneax is now known as saneax-_-|AFK01:15
*** bana_k has quit IRC01:28
*** sshnaidm has joined #tripleo01:33
*** dmacpher has joined #tripleo01:37
*** mburned` is now known as mburned_out01:37
*** thrash is now known as thrash|g0ne01:55
*** tzumainn has quit IRC02:05
openstackgerritBrad P. Crochet proposed openstack/puppet-tripleo: Add zaqar profiles  https://review.openstack.org/33168102:10
*** limao has quit IRC02:49
*** limao has joined #tripleo02:49
*** david-lyle has quit IRC03:03
*** limao_ has joined #tripleo03:08
*** limao has quit IRC03:11
*** coolsvap has joined #tripleo03:22
*** dtrainor has quit IRC03:35
*** rbrady_ has quit IRC03:40
*** jcoufal has quit IRC04:12
*** yamahata has joined #tripleo04:21
*** masco has joined #tripleo04:25
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Terminate Zaqar websocket endpoint in HAProxy  https://review.openstack.org/36032904:52
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Enable TLS for Zaqar's websocket endpoint  https://review.openstack.org/36035004:52
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Make HAProxy terminate OpenStack internal/admin endpoints  https://review.openstack.org/36036604:53
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Always deploy HAProxy  https://review.openstack.org/36036804:53
*** pgadiya has joined #tripleo04:54
*** jaosorior has joined #tripleo04:55
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-docs: Move undercloud SSL admonition to SSL file  https://review.openstack.org/37250805:07
*** jroll has quit IRC05:09
*** jroll has joined #tripleo05:09
*** jroll has quit IRC05:09
*** jroll has joined #tripleo05:09
*** jaosorior has quit IRC05:09
*** jaosorior has joined #tripleo05:10
*** akshai has joined #tripleo05:11
openstackgerritMerged openstack/python-tripleoclient: Pass the timeout to the deploy workflow  https://review.openstack.org/37018605:14
*** akshai has quit IRC05:19
*** saneax-_-|AFK is now known as saneax05:19
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: DO NOT MERGE - TESTING BARBICAN  https://review.openstack.org/35326205:21
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add Barbican to the overcloud - WIP  https://review.openstack.org/35245805:21
openstackgerrityolanda.robla proposed openstack/tripleo-validations: Port the network environment validation  https://review.openstack.org/34158605:26
openstackgerrityolanda.robla proposed openstack/tripleo-validations: Add jsonschema validation for os-net-config data  https://review.openstack.org/35134505:26
*** bnemec has joined #tripleo05:27
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore  https://review.openstack.org/36117305:30
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore  https://review.openstack.org/36544905:30
*** limao_ has quit IRC05:33
*** limao has joined #tripleo05:33
jaosoriord0ugal: ping05:33
*** yolanda has joined #tripleo05:34
*** florianf has quit IRC05:41
*** florianf has joined #tripleo05:41
openstackgerritMerged openstack/tripleo-ui: Migrate Deploy action to Mistral  https://review.openstack.org/35712505:51
*** oshvartz has joined #tripleo05:56
openstackgerritSaravanan KR proposed openstack/os-net-config: Fixed nic numbering issue of DPDK nics after the nic has bound  https://review.openstack.org/36435406:00
openstackgerritSaravanan KR proposed openstack/os-net-config: Add mac address to the DPDK mapping file  https://review.openstack.org/37001206:00
openstackgerritavnish proposed openstack/os-cloud-config: Update home page link in cfg file  https://review.openstack.org/37290406:04
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Work around deletion of _member_ role assignments on upgrade  https://review.openstack.org/30735206:07
*** fzdarsky has joined #tripleo06:07
*** ccamacho has quit IRC06:07
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Work around deletion of _member_ role assignments on upgrade  https://review.openstack.org/30735206:08
*** rasca has joined #tripleo06:11
*** shardy has joined #tripleo06:12
jaosoriorshardy: hey dude, have you been looking into the bug where mistral can't execute the action plan_deploy due tripleo-common being unable to retrieve a local file?06:19
*** dmacpher has quit IRC06:20
openstackgerritMerged openstack/diskimage-builder: Add libselinux-python to yum-minimal  https://review.openstack.org/37183406:23
*** ccamacho has joined #tripleo06:24
shardyjaosorior: Yes, I tried to reproduce it yesterday but so far have failed to do so, create of a stack works fine for me, including with optional and locally defined extra environments & templates06:26
shardyThere are two bugs:06:26
openstackLaunchpad bug 1623552 in tripleo "heatclient resolves paths for types and get_file calls that then don't make sense in swift" [Critical,Triaged]06:26
ccamachomorning all!06:26
openstackLaunchpad bug 1624727 in tripleo "Could not fetch contents for file:///home/stack/tripleo-heat-templates/puppet/post.yaml" [Critical,Triaged]06:26
shardyjaosorior: I suspect they are both the same issue, and that the problem is we have some brokenness in the plan update workflow which isn't yet fixed06:27
*** mcornea has joined #tripleo06:27
shardytry doing a openstack plan delete overcloud06:27
jaosoriorshardy: you gotta do a re-deploy06:27
shardythen do a clean deploy06:27
jaosoriorthat's how I got that issue06:27
ccamachojaosorior, there is another issue in liberty but this time when creating overcloud images https://bugs.launchpad.net/tripleo/+bug/162546306:27
openstackLaunchpad bug 1625463 in tripleo "Package dependency issue in liberty deployments" [Undecided,New]06:27
jaosoriordoing a deploy works, but re-deploying breaks with that06:27
shardyjaosorior: You mean do overcloud deploy, then another overcloud deploy without deleting?06:28
shardyor clean install of the undercloud?06:28
jaosoriorshardy: first option06:28
openstackgerritavnish proposed openstack/tripleo-validations: modify the home-page info with the developer documentation  https://review.openstack.org/37293006:28
shardyjaosorior: Yeah, that's what I'm saying06:28
jaosoriorshardy: so as a workaround for the existing issues with the workflows06:28
jaosoriorI delete the plan06:28
jaosoriorthen delete the swift container06:28
shardywe know that is broken, d0ugal has patches up but they've not yet landed06:28
jaosoriorthen try to deploy again06:28
*** jprovazn has joined #tripleo06:29
jaosoriorand that's how I get that "could not fetch contents for file://..." error06:29
shardyjaosorior: ack, pls can you comment on https://bugs.launchpad.net/tripleo/+bug/1623552 if that works06:29
openstackLaunchpad bug 1623552 in tripleo "heatclient resolves paths for types and get_file calls that then don't make sense in swift" [Critical,Triaged]06:29
jaosoriorshardy: yes; I'm aware there are fixes... I was just telling you how I had stumbled upon that06:29
ccamachojaosorior, I confirm that issue from yesterday (second deployment fails "delete the plan" and then the post.yaml error)06:29
jaosoriorccamacho: exactly06:30
openstackgerritavnish proposed openstack/tripleo-incubator: Update home page link in cfg file  https://review.openstack.org/37293306:30
ccamachojaosorior** but, like 2 days ago didnt hit the post.yaml issue only the plan one06:30
shardyI'm confused then06:30
shardyI did repeated redeploys all afternoon yesterday06:30
shardyand it works fine, provided you do a stack-delete && plan delete06:30
jaosoriorshardy: I did a fresh deploy yesterday evening. And it doesn't work for me06:30
jaosoriorshardy: stack delete?06:31
jaosoriorno no06:31
shardyYes, my sequence is openstack overcloud deploy --templates -e $stuff06:31
jaosoriorI'm deleting the swift container and the plan06:31
*** yolanda has quit IRC06:31
shardythen heat stack-delete overcloud06:31
ccamachoI think there was a revert somewhere which introduced that :P06:31
shardythen openstack overcloud plan delete overcloud06:31
shardyI can do that cycle as many times as I want, no errors06:32
ccamachoas as shady said I wasnt having that issue also, only since a fresh install from yesterday06:32
jaosoriorshardy: yeah, I think that works06:32
shardyyou shouldn't delete the swift container manually06:32
shardyyou have to use plan delete overcloud06:32
jaosoriorshardy: I tried getting deploying to work06:32
shardyor the mistral environment remains which messes things up06:32
shardyI raised a bug about that already06:32
jaosoriorshardy: alright, so here the thing is that I was told that to be able to do an update (or overcloud deploy, again) I needed to delete the container, and the mistral environment06:33
jaosoriorand then I should just do an overcloud deploy (with the stack still existing)06:33
jaosoriorand now I'm stumbling upon that bug06:33
shardyjaosorior: Yeah, or do plan delete, which is the same thing06:33
shardyNo, you should delete the stack too06:33
jaosoriorshardy: plan delete requires that the stack be deleted06:33
jaosoriorso then we're talking about different things06:33
jaosoriorI don't want to do a complete redeploy every single time06:34
shardyall the update stuff is busted until the fixes from d0ugal lands06:34
shardybecause we don't render the j2 templates06:34
shardythat's while post.yaml is missing06:34
jaosoriorok, that makes sense. Then I had wrong information06:34
jaosoriorthanks for clarifying06:34
shardyjaosorior: You have two choices, either do a full overcloud redeploy, or pull the fixes listed in https://bugs.launchpad.net/tripleo/+bug/162472706:35
openstackLaunchpad bug 1624727 in tripleo "Could not fetch contents for file:///home/stack/tripleo-heat-templates/puppet/post.yaml" [Critical,Triaged]06:35
shardyeither way, feedback on the bugs would be good06:35
shardyhopefully d0ugal can confirm how we unblock those fixes today06:35
jaosoriorshardy: I can pull the fixes. Got a list?06:35
*** bana_k has joined #tripleo06:35
shardyin the bug I just linked06:35
jaosoriorok, will pull those in06:36
ccamacho*** Currently both deployments Mitaka and Liberty are broken... https://bugs.launchpad.net/tripleo/+bug/1625463  https://bugs.launchpad.net/tripleo/+bug/162531506:37
openstackLaunchpad bug 1625463 in tripleo "Package dependency issue in liberty deployments" [Undecided,New]06:37
openstackLaunchpad bug 1625315 in tripleo "Can't deploy Mitaka overcloud on CentOS" [Undecided,New]06:37
ccamachojust reported in RDO06:37
ccamachothere is some dep issue when building the overcloud iamges06:37
ccamachogoing for some coffee http://cdn77.sadanduseless.com/wp-content/uploads/2014/05/bird-with-arms16.jpg06:39
jaosoriorshardy: by the way, do you know more or less how often the tripleo changes get pulled in RDO?06:41
*** jbadiapa has joined #tripleo06:43
*** dciabrin|away is now known as dciabrin06:46
jaosoriorshardy, d0ugal also, this patch https://review.openstack.org/#/c/371347/ which is listed in the fixes has a depends-on on an abandoned patch on tripleoclient, what's up?06:46
d0ugaljaosorior: pong06:48
* d0ugal reads up06:48
jaosoriord0ugal: I can test out the fixes, but I'm a bit confused on which to pull06:50
*** pcaruana has joined #tripleo06:50
d0ugaljaosorior: Right, which problem are you hitting?06:51
d0ugalA bunch landed yesterday.06:51
jaosoriord0ugal: ok, I'm re-running undercloud install just to make sure I have everything06:52
jaosoriord0ugal: but yeah, I was still having issues with doing an update to a running overcloud06:55
shardyd0ugal: we're trying to figure out the needed fixes for the broken plan update stuff06:55
shardyas I think that's the main cause of the file::/// bugs06:55
d0ugalshardy: Right, I don't think there is a complete fix yet06:56
*** zoli_gone-proxy is now known as zoliXXL06:56
d0ugalshardy: I am thinking about just deleting the plan and re-creating.06:56
d0ugalshardy: because I have no idea what is going on.06:56
shardyd0ugal: can we at least land the workflow fix to run ProcessTemplates on plan update?06:56
d0ugalWe really screwed up by not planning the update workflow06:56
d0ugalshardy: I think so, yes.06:56
shardyhttps://bugs.launchpad.net/tripleo/+bug/1624727 will be fixed by that AFAICS06:56
openstackLaunchpad bug 1624727 in tripleo "Could not fetch contents for file:///home/stack/tripleo-heat-templates/puppet/post.yaml" [Critical,Triaged]06:56
d0ugalshardy, jaosorior: https://review.openstack.org/#/c/371027/06:56
shardythe post.yaml is just missing because we delete all the files, then fail to run j2 afterwards06:57
jaosoriord0ugal: already had my +2 :P06:57
d0ugaljaosorior: oops :)06:57
*** tremble has joined #tripleo06:57
d0ugalah, I need to move that change out of the chain06:57
d0ugal1 min06:57
shardyd0ugal: yeah, that was my next question ;)06:57
shardyI commented on https://review.openstack.org/#/c/371347/ (head of the chain) yesterday, and I'm still not exactly clear why that depends on the mistral fix06:58
openstackgerritDougal Matthews proposed openstack/tripleo-common: Add template processing to the update plan workflow.  https://review.openstack.org/37102706:58
shardyis it because we're returning a gigantic data blob from the action?06:58
d0ugalshardy, jaosorior: https://review.openstack.org/#/c/371027/ now doesn't depend on anything06:59
*** skramaja has joined #tripleo06:59
d0ugalshardy: when the default plan creation works we hit the mistral bug.06:59
d0ugalshardy: at the moment it is broken so the workflow ends quickly06:59
jaosoriorok, pulled the latest stuff and now I can't do an overcloud deploy (not even a starting one)06:59
jaosorior u'message': u"No connection adapters were found for 'file:///home/stack/tripleo-heat-templates/puppet/manifests/overcloud_controller_pacemaker.pp'",06:59
*** skramaja has quit IRC06:59
d0ugalshardy: otherwise it is still running when some of the CLI commands are called, that causes the error.06:59
d0ugaljaosorior: huh, you found the error shardy was looking for :)06:59
shardyd0ugal: Ok, thanks for confirming07:00
d0ugalshardy: so because I fixed the default plan at the same time as adding templating processing (I was rushing) both got reverted to fix the CI issues.07:00
shardyjaosorior: Ok, so that's the bug reported in https://bugs.launchpad.net/tripleo/+bug/1623552 and I cannot reproduce it07:00
openstackLaunchpad bug 1623552 in tripleo "heatclient resolves paths for types and get_file calls that then don't make sense in swift" [Critical,Triaged]07:00
d0ugaljaosorior: Which environments are you passing in?07:01
jaosoriorshardy: have you tried using manifests from your home directory and not the ones that are packaged?07:01
shardyyeah, please update the bug with the tripleo-common, tripleoclient and heatclient version you have, and the exact CLI call you're making07:01
shardyjaosorior: Yes, I always deploy a local tree, not the packaged one07:02
shardybut I normally do it from /tmp/tripleo-heat-templates07:02
jaosoriord0ugal: passing in this openstack overcloud deploy \07:02
jaosorior    --templates tripleo-heat-templates \07:02
jaosorior    --libvirt-type qemu --control-flavor oooq_control --compute-flavor oooq_compute --ceph-storage-flavor oooq_ceph --timeout 90 -e tripleo-heat-templates/environments/network-isolation.yaml -e tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e $HOME/network-environment.yaml -e tripleo-heat-templates/environments/puppet-pacemaker.yaml --control-scale 3 --neutron-network-type vxlan07:02
jaosorior--neutron-tunnel-types vxlan --ntp-server pool.ntp.org07:02
shardyOk, so relative paths are one difference here07:02
d0ugalshardy: do you ever deploy anything outside your t-h-t directory? i.e. --templates somewhere -e somewhere-else?07:02
*** leanderthal|afk is now known as leanderthal07:02
shardymine are all absolute07:02
shardyd0ugal: yes, I tested that yesterday, and it works for me07:02
shardyI passed in a different firstboot script via an extra template & environment07:03
d0ugalperhaps we can make all the paths absolute first?07:03
shardyd0ugal: well, first lets figure out a reproducer07:03
d0ugalTrue :)07:03
shardyI was just pointing out I'm driving the CLI differently, it may be unrelated07:03
jaosoriorshardy: well, to be fair I'm also passing a bunch of other environment files07:03
jaosoriorpassing in names and such07:04
jaosoriorbut, nothing related to the stuff that it spits out as an error. That one is included in the pacemaker one07:04
*** bana_k has quit IRC07:04
shardyjaosorior: Yeah, I just need some way to reproduce it - can you try again with absolute paths for everything?07:04
shardyparticularly tripleo-heat-templates/environments/puppet-pacemaker.yaml07:04
* d0ugal is waiting for his undercloud to finish installing07:05
* shardy is waiting for his coffee to kick in07:05
jaosoriorwill do07:06
*** radeks has joined #tripleo07:07
*** zoliXXL is now known as zoli_gone-proxy07:07
jaosoriorshardy: it worked07:09
d0ugalSo it turns out I am just bad at path munging :)07:10
shardyI've reproduced locally with relative paths07:11
shardyIt blows up with heatclient trying to read local files inside the mistral action07:11
shardyd0ugal: do you want to take a look?  I guess we must be missing the entries for relative paths from the files map07:12
*** flepied1 is now known as flepied07:12
shardyand/or we're doing heatclient stuff we shouldn't do in the mistral action07:12
d0ugalshardy: Yeah, I can try and have a look - but I'm not really sure how I've missed them.07:12
d0ugalshardy: I suspect we are doing something wrong :)07:13
shardyd0ugal: Ok, I have an appointment shortly, but if you've not cracked it by the time I return, I'll be happy to help07:13
shardynow it's reproducable we can iterate faster :)07:13
shardyI suspect we can probably simplify things in tripleoclient, with two calls to heatclient07:13
shardyone which grabs all the local files, and another which resolves the uploaded container07:14
shardywill give that a try later07:14
*** zoli_gone-proxy is now known as zoliXXL07:14
d0ugalshardy: Yeah, probably, that would be good. I'll see what I can learn while you are out.07:14
*** chlong has quit IRC07:14
*** zoliXXL is now known as zoli|trng07:15
*** hjensas has joined #tripleo07:19
*** aufi has joined #tripleo07:19
*** shardy has quit IRC07:19
mariosjistr: hey morning... i took objection to 'stalled' :) https://etherpad.openstack.org/p/ocata-tripleo07:19
*** rcernin has joined #tripleo07:19
mariosjistr: had to go looking for your reviews saw you abandoned them07:19
jaosoriortherve: thought the error was transient. Had stopped already.07:23
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Add an optional timeout when waiting for websocket messages  https://review.openstack.org/36425207:23
thervejaosorior, After 4 builds with the same error we should get a clue :p07:24
jaosoriorCan someone check out these two commits? https://review.openstack.org/#/c/360329/  https://review.openstack.org/#/c/360350/07:24
jaosoriortherve: hadn't seen that it was 4 builds.07:24
openstackgerritDougal Matthews proposed openstack/tripleo-common: Return the result of create_plan in create_deployment_plan workflow  https://review.openstack.org/37134807:25
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fix the default plan creation  https://review.openstack.org/37134707:25
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fix the default plan creation  https://review.openstack.org/37134707:25
openstackgerritDougal Matthews proposed openstack/tripleo-common: Remove the unused service_host arg from node registration  https://review.openstack.org/32603607:27
*** yamahata has quit IRC07:27
jtomasekshardy, d0ugal, EmilienM: I've added proposal for GUI/CLI/Validations Summit session with a link to temporary etherpad https://etherpad.openstack.org/p/ocata-tripleo07:33
*** dsariel has joined #tripleo07:33
jtomasekshardy, d0ugal, EmilienM: I'll continue updating the etherpad and then move the information to ocata etherpad. Feel free to add stuff/comment07:33
*** jpich has joined #tripleo07:33
*** ohamada has joined #tripleo07:34
jaosoriormarios: hey dude, can you check out these two CRs?  https://review.openstack.org/#/c/360329/  https://review.openstack.org/#/c/360350/   they've been there for a while, but I was missing a configuration in tripleo-ci before that would work.07:35
d0ugaljtomasek: cool, sounds good :) Who is going from the Validations/CLI side of things?07:35
mariosjaosorior: will do in a bit ... sorry yesterday the one you asked me to loko at merged by the time i got to it, but they are in the list ...07:36
jaosoriormarios: no biggie, thanks dude07:36
*** jpena|off is now known as jpena07:36
*** ebarrera has joined #tripleo07:37
jtomasekd0ugal: looking at the list it is probably you for CLI07:37
jtomasekd0ugal: and shadower for validations07:38
d0ugaljtomasek: heh, well, I can be there to represent but don't sign me up for any work :)07:42
*** jlinkes has joined #tripleo07:44
jtomasekd0ugal: ack07:45
d0ugaloh, I almost forgot I need to run to the dentist. Back in a bit07:46
* d0ugal -> dentist07:46
*** rdopiera has joined #tripleo07:48
rdopieraI wonder if any of you, gentlepeople, could point me to where the overcloud horizon's configuration is defined?07:48
rdopiera(preferably in rhel downstream)07:49
jaosoriorrdopiera: you mean this? https://github.com/openstack/puppet-horizon/blob/master/templates/local_settings.py.erb07:49
bandinirdopiera: look at tripleo-heat-templates/puppet/services/horizon.yaml (that is the entry point), then look at puppet-tripleo/manifests/profile/base/horizon.pp and then puppet-horizon07:50
rdopierabandini, jaosorior: perfect, thank you07:53
rdopieranow just to find where it is packaged...07:54
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375007:55
*** b00tcat has quit IRC07:57
*** milan has joined #tripleo07:58
jistrmarios: hi, yea i did. I only had the original approach that got -2'd.07:59
*** athomas has joined #tripleo08:00
jaosoriorjistr: hey man, when you get time can you review these  https://review.openstack.org/#/c/360329/  https://review.openstack.org/#/c/360350/ ?08:01
*** hewbrocca-afk is now known as hewbrocca08:01
*** b00tcat has joined #tripleo08:02
jistrjaosorior: should the bugs be targeted at rc2? otherwise we can't merge them08:03
*** chem``` has joined #tripleo08:08
*** limao has quit IRC08:11
*** fzdarsky has quit IRC08:14
*** pmannidi has quit IRC08:15
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: RabbitMQ threads should be configured dynamically  https://review.openstack.org/37302708:17
*** chem``` is now known as chem08:18
*** fzdarsky has joined #tripleo08:19
*** chem has quit IRC08:20
*** chem has joined #tripleo08:21
*** radeks has quit IRC08:23
*** gfidente has joined #tripleo08:24
*** yolanda has joined #tripleo08:24
*** cylopez has joined #tripleo08:26
*** abehl has joined #tripleo08:29
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: RabbitMQ threads should be configured dynamically  https://review.openstack.org/37302708:33
*** derekh has joined #tripleo08:34
*** skramaja has joined #tripleo08:42
*** social has joined #tripleo08:42
*** pkovar has joined #tripleo08:47
*** electrofelix has joined #tripleo08:52
*** dsariel has quit IRC08:55
*** akrivoka has joined #tripleo08:58
jaosoriorjistr: yes08:59
jaosoriorjistr: I tagged it for rc2 now, sorry about that09:01
jistrjaosorior: ok thanks. I thought it was important for rc2 by the sound of it but wanted to double check :)09:02
tobias-fiberdatajaosorior, have you experienced that your undercloud does not reboot correctly and mount the LVM volumes after installing mitaka undercloud?09:04
jaosoriortobias-fiberdata: I haven't :/09:04
tobias-fiberdatacause this is weird. it doesnt mount up the LVM for home09:04
tobias-fiberdataroot and swap is working fine09:05
tobias-fiberdatabut perhaps it's a centOS and lvm problem.09:05
hewbroccatobias-fiberdata: you're probably hitting the lvm automation bug09:05
tobias-fiberdatahewbrocca, we have gotten that twice09:05
tobias-fiberdataboth times when installing mitaka undercloud09:05
hewbroccaIf you have more than two volumes on your undercloud host, it will bite you09:06
tobias-fiberdataphysical volumes?09:06
tobias-fiberdatagot three09:07
tobias-fiberdataso basically09:07
tobias-fiberdataremove the swap will solve it09:07
hewbroccaThe container guys put in an LVM conf hack to *not* automatically mount volumes,09:07
hewbroccabecause it breaks Docker09:07
hewbroccaTurns out that is a problem :(09:07
tobias-fiberdatait is for sure09:07
hewbroccaIt's fixed in Newton, I believe, with a regex in lvm.conf09:07
tobias-fiberdatait is fixed there09:07
hewbroccayou could grab the regex from Newton and add it to the undercloud lvm.conf, I guess09:08
*** pkovar has quit IRC09:09
hewbroccaor just change lvm.conf to behave normally, since you're presumably not using a containerized compute node09:09
*** limao has joined #tripleo09:14
*** fzdarsky has quit IRC09:19
tobias-fiberdatahewbrocca, we remove swap to solve it09:19
jaosoriormarios: yep, it's being added to that list here https://review.openstack.org/#/c/360350/6/instack_undercloud/undercloud.py09:22
jaosoriorhad to be added in a different way since it's not using http/https, but a websocket-specific protocol09:23
*** akrivoka has quit IRC09:23
openstackgerritDougal Matthews proposed openstack/tripleo-common: Port password generation from tripleoclient to tripleo-common  https://review.openstack.org/36815009:25
mariosjaosorior: ack thanks missed it there (didn't scroll down enough :)09:26
jaosoriormarios: haha it's cool. Thanks for checking it out dude09:28
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Add `openstack overcloud plan deploy`  https://review.openstack.org/37308909:29
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Pass the timeout to the deploy workflow  https://review.openstack.org/37309009:29
*** skramaja_ has joined #tripleo09:30
*** skramaja has quit IRC09:30
*** skramaja_ is now known as skramaja09:30
*** dsariel has joined #tripleo09:32
jaosoriorHey guys, can we merge this documentation patch? https://review.openstack.org/#/c/348038/09:34
*** akrivoka has joined #tripleo09:35
*** stendulker has joined #tripleo09:38
*** pkovar has joined #tripleo09:38
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Remove the interactive gate from full-deploy.sh  https://review.openstack.org/37309809:42
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Remove roles-deploy.sh and fix full-deploy.sh  https://review.openstack.org/37236709:45
*** akrivoka has quit IRC09:46
*** chem has quit IRC09:50
*** shardy has joined #tripleo09:53
*** panda|Zz is now known as panda09:55
*** akrivoka has joined #tripleo09:58
*** tosky has joined #tripleo10:03
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart: add tempest back into the quickstart-extras workflow  https://review.openstack.org/37274810:04
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Customize undercloud and overcloud with virt-customize  https://review.openstack.org/37011410:06
jaosoriorshardy: hey, It seems that we have an issue with the nova_metadata endpoint10:08
jaosoriorshardy: the yaql function that generates the <service>_vip doesn't take it into account because it's not a service10:08
jaosoriorit's part of nova-api10:09
jaosoriorso haproxy is misconfiguring it, because it expects nova_metadata_vip to exist10:09
*** limao has quit IRC10:09
jaosoriorthe same issue is with nova_metadata_node_ips10:09
jaosoriorshardy: so we have two options: 1) create a metadata role (which doesn't import any puppet stuff, just does hiera config) 2) add metadata manually like we do for keystone10:10
jaosoriorshardy: what do you think?10:10
shardyjaosorior: Hmm, yeah I knew this was an issue when refactoring, because the nova_api service actually deploys two different things10:10
openstackgerritCarlos Camacho proposed openstack/tripleo-image-elements: Add stable release to image elements  https://review.openstack.org/37313110:11
shardyI'd probably prefer breaking out the nova metadata hiera into a new service template, even if we know it can't be deployed without nova_api10:11
shardythe problem, IIRC, was that the nova_api puppet module expects a list of enabled APIs in hiera10:11
shardyso we'd need to leave that hard-coded in nova_api referencing the n-api and the metadata api10:12
openstackgerritCarlos Camacho proposed openstack/tripleo-image-elements: Add stable release to image elements  https://review.openstack.org/37313110:12
shardyjaosorior: FWIW the special-casing around keystone is something I'd like to fix, so ideally it'd be good to avoid perpetuating that sort of special-casing10:13
*** SlickNik has quit IRC10:13
*** chem has joined #tripleo10:14
shardyjaosorior: ^^10:15
shardythat is the problem10:15
shardyI guess we could pass in two booleans for nova_api and nova_metadata_api to puppet-tripleo10:15
shardyand have the nova_api profile combine them to create the right list10:16
shardyOr, for now, accept that is hard-coded and that both service templates must be deployed together10:16
*** SlickNik has joined #tripleo10:17
shardyd0ugal: Hey, just about to dig into the file::// stuff, unless you've made some progress?10:18
jaosoriorshardy: ok, I'll do the metadata template10:18
d0ugalshardy: not yet, I had a dentist appointment which I almost forgot about10:19
d0ugalshardy: so I am looking into it now also, but I've not made any solid progress.10:19
shardyd0ugal: ack, I'll dig into it and we can compare findings to decide the best fix10:19
d0ugalSounds good10:20
openstackgerritIan Wienand proposed openstack/diskimage-builder: Merge remote-tracking branch 'origin/master' into merge-branch  https://review.openstack.org/37313410:22
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Customize undercloud and overcloud with virt-customize  https://review.openstack.org/37011410:22
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add nova-metadata template  https://review.openstack.org/37314110:27
jaosoriormcornea: lets see if I didn't mess it up ^^10:27
mcorneajaosorior: checking10:27
shardyd0ugal: so, the problem definitely seems to be with _process_and_upload_environment - we need to remove the mangling of the resource_registry prefixes I think, and figure out a different way to push any missing files to the plan10:33
*** radez has quit IRC10:34
shardyd0ugal: file:///tmp/tripleo-heat-templates/puppet/services/pacemaker/rabbitmq.yaml gets mangled to user-files/b06f4f7c785fbf79c1207ab7897711fa-rabbitmq.yaml10:34
shardywhich can't be right, because it's already in the plan10:34
*** radez has joined #tripleo10:34
* shardy starts commenting out code10:34
*** adarazs is now known as adarazs_lunch10:35
*** rwsu has quit IRC10:35
openstackgerritLucky samadhiya proposed openstack/tripleo-common: update homepage with developer documentation page  https://review.openstack.org/37315010:37
*** jaosorior is now known as jaosorior_lunch10:37
*** mburned_out is now known as mburned10:38
d0ugalshardy: Right, makes sense.10:39
d0ugalshardy: I was always suspicious of that code :)10:40
shardyd0ugal: I think we're moving the entire files map into moved_files, I'm hacking up a reworked version trying to simplify that a bit10:40
tobias-fiberdatahow do we configure normal interfaces aswell as the bonds with vlan? What type should it be? just "type: bond"?10:41
*** rodrigods has quit IRC10:41
*** rodrigods has joined #tripleo10:41
tobias-fiberdatato be short10:42
shardytobias-fiberdata: https://github.com/openstack/os-net-config/blob/master/etc/os-net-config/samples/bond.yaml10:42
tobias-fiberdatawe want one bond to connect to the bridge, aswell as one bond without attaching it10:42
shardyyup, you just configure the members type: interface inside the type: *bond10:42
tobias-fiberdatacan someone review our controller.yaml and compute.yaml? I'll paste it in a sec10:44
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Use the hexdigest of the path to make the filename unique in swift  https://review.openstack.org/37315910:45
tobias-fiberdatathis is just the resource part10:46
tobias-fiberdataour deployment fails when configuring the storageport10:46
shardyd0ugal: I don't think there's any way we can properly fix this without dowloading the plan contents, after the j2 rendering is done10:46
*** rhallisey has joined #tripleo10:47
shardythen running heatclient template_utils.get_template_contents again pointing at the local tree to pull in any missing files we didn't get by uploading the --templates10:47
shardyany other solution will break as soon as we reference another file inside a .j2.yaml template10:48
shardyOn a positive note, that will remove almost all of the template mangling code from tripleoclient10:48
shardyI suppose the alternative is say referencing files outside of the plan in a j2 template is not allowed10:51
shardy(which would probably be better for the UI)10:51
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Print the execution ID when starting Mistral workflows  https://review.openstack.org/37316210:53
d0ugalshardy: It would be better to keep all the files in one directory10:57
d0ugalshardy: That is where all of these problems come from, the CLI is almost too flexible :)10:57
*** fzdarsky has joined #tripleo10:58
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Remove the environments from Mistral when removing from Swift  https://review.openstack.org/37316710:58
*** pkovar has quit IRC11:00
*** jaosorior_lunch is now known as jaosorior11:03
jaosoriormcornea: thanks for checking the patch out11:03
mcorneajaosorior: thank you, it looks good11:03
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fix the default plan creation  https://review.openstack.org/37134711:04
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Verify that the Deployment Plan creation was successful  https://review.openstack.org/36924711:04
*** thrash|g0ne is now known as thrash11:05
jaosoriorHey guys, any +A for these two? https://review.openstack.org/#/c/360329/  https://review.openstack.org/360350 ??11:07
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Use a released version of tripleo-common  https://review.openstack.org/37317511:08
d0ugaljaosorior: I think you can +A if it has two other cores and was just waiting for CI11:08
d0ugaljaosorior: I do for my own patches, anyway +A'ed yours.11:08
jaosoriord0ugal: thanks dude11:09
jpichd0ugal, jaosorior: FWIW came across that doc that agrees with d0ugal on this -> http://specs.openstack.org/openstack/tripleo-specs/specs/policy/expedited-approvals.html#self-approval11:13
jaosoriorshardy: is upload-puppet-modules still working for you? I think there's an issue in the underlying upload-swift-artifacts11:16
jaosoriorjpich: I see :O11:16
*** stendulker has quit IRC11:16
shardyjaosorior: Yeah it's been working fine for me provided you have my fix that changed the default container name11:16
jaosoriorI downloaded the latest tripleo-common today11:17
jaosoriorand it's not working for me11:17
jaosoriorbut it seems to be related to how it gets the swift keystone endpoint11:17
shardyOk, I've not tested it today, but it was working for me yesterday11:17
jaosoriorshardy: same here, it was working yesterday11:17
* shardy been too busy making tripleoclient explode in exciting ways11:17
*** dsariel has quit IRC11:20
*** karthiks has quit IRC11:28
*** osp has quit IRC11:30
tobias-fiberdatajaosorior, shardy how long does a deployment take usually?11:31
tobias-fiberdatathing 3controller HA and 2 compute11:31
tobias-fiberdatathink 3controller HA and 2 compute*11:31
tobias-fiberdatano SSD in the servers just SAS and the undercloud is even SATA XD11:32
jaosoriorit can take a while. I'm only using virtual and it can take about an hour or so11:32
*** skramaja has quit IRC11:32
tobias-fiberdataso far so good11:33
tobias-fiberdatawe removed some conf11:33
jaosorioryou're using baremetal, so it should be faster tahn my deployments11:33
shardyjaosorior: what's your hardware?11:33
shardytwo node deployments on a desktop box with an SSD take about 20mins for me11:33
tobias-fiberdataif it's getting stuck ont hsi11:33
tobias-fiberdataUpdateDeployment         | b21a72ad-687e-49b7-8ed6-76258618b188 | OS::Heat::SoftwareDeployment                    | CREATE_IN_PROGRESS11:33
jaosoriorshardy: got no SSD, and I'm talking about 3 controllers and one compute11:33
tobias-fiberdatawhat should i look for?11:33
shardyjaosorior: ack, yeah I've not got enough ram for that setup atm11:34
jaosoriorshardy: how much do you have?11:34
tobias-fiberdatai think our setup are enough :D 3x 32GB ram for controller and 256GB each for two computes11:34
tobias-fiberdataso far11:34
shardytobias-fiberdata: that tries to do a yum update, I'd check the boxes can reach the repos11:34
shardyjaosorior: 32G11:35
jaosoriorshardy: that's exactly what I have11:35
shardyI can run the undercloud and ~4nodes11:35
jaosoriorI run the undercloud, 4 nodes and even FreeIPA on the host11:35
shardyjaosorior: How much memory allocated to the VMs?11:35
shardyI use 8G to avoid swapping or deployments take forever11:35
jaosoriorwell, it's swapping like there's no tomorrow11:36
shardyjaosorior: Yeah, that's why the deployment takes an hour ;)11:36
jaosoriorI put 7 in the undercloud11:36
*** adarazs_lunch is now known as adarazs11:36
*** masco has quit IRC11:36
jaosoriorshardy: well, gotta deploy FreeIPA anyway for the work I'm doing11:36
jaosoriorso I have no choice11:36
*** saneax is now known as saneax-_-|AFK11:38
jaosoriormcornea: do you still have the environment set up where you tested the nova_metadata patch?11:40
mcorneajaosorior: I should have it soon, currently it's broken, I'm testing some network templates changes11:40
jaosoriormcornea: what are you testing out?11:40
jaosoriormcornea: can you check if the value nova_metadata_network is being set?11:41
*** dprince has joined #tripleo11:41
pandain HA deployment, which controller is usually the default location for the endpoint ?11:41
EmilienMhello tripleoers11:41
EmilienMd0ugal: ok cool11:42
mcorneajaosorior: doing a deployment separate network nodes and trying to build network templates for them and the deployment is currently stuck, I need to debug what went wrong there11:42
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: swift: normalize memcache servers IP addresses  https://review.openstack.org/37273411:42
mcorneajaosorior: I'll check it once I get the deployment going11:42
jaosoriorpanda: which endpoint?11:42
jaosoriorEmilienM: hey dude, how's it going?11:43
EmilienMpanda: FYI https://review.openstack.org/#/c/372903/ and https://review.openstack.org/#/c/372734/11:43
EmilienMjaosorior: thanks, good11:43
EmilienMjaosorior: and you?11:44
jaosoriorpretty good, dude. Work, work, work11:44
*** lucas-afk is now known as lucasagomes11:44
*** jpena is now known as jpena|lunch11:44
*** pkovar has joined #tripleo11:44
jaosoriorthe usual11:44
tobias-fiberdatashardy, now i lost connection to the machines.11:45
tobias-fiberdatai guess there's a timeout on several hours on this step?11:45
pandajaosorior: keystone11:45
jaosoriorEmilienM: does this https://review.openstack.org/#/c/372734/2/manifests/profile/base/swift/proxy.pp mean that we can there only use IP addresses? wouldn't want to restrict to that. I'm hoping for Ocata to be able to use FQDNs there11:45
EmilienMjaosorior: no you can use hostnames11:46
jaosoriorpanda: that would be in the bootstrap node. I guess it would be usually controller-0?11:46
EmilienMjaosorior: the normalize function does nothing if the content is not an ipv611:46
jaosoriorah, I see now11:46
jaosoriorwait up11:47
jaosoriorbut, wouldn't it fail?11:47
jaosoriorEmilienM: IPAddr.new(ip) throws an error if it's not an IP11:47
jaosoriorand then you ignore it11:47
jaosoriorI see now11:47
EmilienMjaosorior: no it doesn't11:47
EmilienMchem wrote that code and we test that you can give an hostname11:47
jaosoriorEmilienM: yeah dude, I saw now that the Argument error is catched and ignored11:48
EmilienMsee the spec https://review.openstack.org/#/c/372903/2/spec/functions/normalize_ip_for_uri_spec.rb11:48
pandaEmilienM: so the last run on my patch in ipv4 is failing because it needs the latesta patch sets from the ones you showed me ?11:48
chemEmilienM: hostname to normalize_ip_for_uri ?11:48
EmilienMpanda: not sure why ipv4 failed tbh11:48
pandajaosorior: ok, thanks11:48
jaosoriorEmilienM: so it's good11:48
jaosoriorEmilienM: +1ed it11:48
chemEmilienM: that shouldn't be any issue11:48
EmilienMchem: yes11:48
chemEmilienM: if it can't get to ipv6, it just return the string11:49
jaosoriorchem: yeah, that's what I just realized now that I gave another read to the code11:49
jaosoriorit should be fine11:50
*** bfournie has quit IRC11:51
*** trown|outtypewww is now known as trown11:52
shardytobias-fiberdata: if the network config is messed up and the machines are unresponsive, heat will sit there until the deployment timeout is hit11:52
shardyby default that's 4 hours11:52
shardysounds like you'll need console access to debug the network config?11:53
*** rbrady has joined #tripleo11:54
*** coolsvap is now known as coolsvap_11:55
*** coolsvap_ has quit IRC11:55
*** spredzy has quit IRC11:55
*** coolsvap_ has joined #tripleo11:55
*** bfournie has joined #tripleo11:55
*** bfournie has quit IRC11:55
*** limao has joined #tripleo11:55
openstackgerritMerged openstack/puppet-tripleo: Terminate Zaqar websocket endpoint in HAProxy  https://review.openstack.org/36032911:55
*** egafford has quit IRC11:56
tobias-fiberdatashardy, how do i get inside the machines without networkaccess to them? because i assume there's no password11:57
*** spredzy has joined #tripleo11:57
shardytobias-fiberdata: when I debug issues like this I modify the image with virt-customize (or one of the other libguestfs tools) to set a known root password or inject a backdoor user11:58
tobias-fiberdatawe'll be updating or removing the stack11:59
tobias-fiberdatai guess11:59
tobias-fiberdatalooks like the network part is messed up11:59
shardyyeah, it's hard to debug networking without either a dedicated nic you're not messing with or primary console access11:59
shardytobias-fiberdata: if you don't want to modify the images, you could also inject a user with cloud-init via a Firstboot script12:01
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375012:01
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fetch internal certificates for HAProxy based on network  https://review.openstack.org/36654812:02
*** rwsu has joined #tripleo12:03
*** ccamacho is now known as ccamacho|lunch12:04
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fetch internal certificates for HAProxy based on network  https://review.openstack.org/36654812:04
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fetch internal certificates for HAProxy based on network  https://review.openstack.org/36654812:05
openstackgerritSteven Hardy proposed openstack/python-tripleoclient: Always use absolute path for tht_root  https://review.openstack.org/37322012:05
jaosoriorshardy: don't we have to apply the same logic for environments?12:06
shardyd0ugal, gfidente: Hey that is the simplest initial fix for https://bugs.launchpad.net/tripleo/+bug/1623552 I can find12:06
openstackLaunchpad bug 1623552 in tripleo "heatclient resolves paths for types and get_file calls that then don't make sense in swift" [Critical,In progress] - Assigned to Steven Hardy (shardy)12:06
gfidenteshardy, ack looking into it12:07
openstackgerritMerged openstack/tripleo-quickstart: Remove roles-deploy.sh and fix full-deploy.sh  https://review.openstack.org/37236712:07
shardyjaosorior: as mentioned in the commit message, this is probably the first of several fixes12:07
shardybut this solves the specific issue reported in the bug, which is due to this comparison:12:07
*** skramaja has joined #tripleo12:07
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fetch internal certificates for HAProxy based on network  https://review.openstack.org/36654812:07
shardywe're comparing an absolute path inside the files map with a relative tht_root12:08
shardyso this gets us past that problem at least12:08
EmilienMjaosorior: when you file a bug, please also set the importance12:08
openstackgerritMarios Andreou proposed openstack/puppet-tripleo: Fixup manila-cephfs native backend defaults  https://review.openstack.org/36676012:08
EmilienMjaosorior: it helps release managers to prioritize triage etc12:08
*** jeckersb is now known as jeckersb_gone12:08
shardyI'm looking into the remainder of the fix, but I think this is valid regardless of any further rework12:08
jaosoriorEmilienM: will do12:09
EmilienMshardy: I added summit topic to https://etherpad.openstack.org/p/tripleo-meeting-items12:09
jaosoriorEmilienM: changed a bit your submission for the _member_ role fix. What do you think?12:13
jaosoriorEmilienM: https://review.openstack.org/#/c/307352/12:13
EmilienMjaosorior: I saw the launchpad email I have not looked at code yet, but thanks for your help12:13
EmilienMjaosorior: looking now12:13
shardyEmilienM: ack, thanks12:13
shardyEmilienM: would you like to chair the meeting today?12:13
shardyget into the new-PTL routine since the Newton release is nearly done ;)12:14
EmilienMshardy: heh ! newton is not out yet \o/12:15
EmilienMbut yeah, I'll do it12:15
shardyhehe, ack, thanks :)12:15
EmilienMjaosorior: the diff lgtm. Again thanks for your help12:15
jaosoriorEmilienM: no biggie12:15
jaosoriorglad to help12:15
mariosshardy: EmilienM is "tripleo/rc2" the right topic... i ask cos i only see one patch at https://review.openstack.org/#/q/topic:tripleo/rc212:15
EmilienMjaosorior: let me know if I can also help12:15
EmilienMmarios: if you use it I'll send you pancakes from Canada, using gerrit topic will really help us to track last blockers12:16
mariosshardy: EmilienM i want to tag the puppet-tripleo side for https://review.openstack.org/#/c/358525/ (<-- is tagged with "tripleo/rc2", puppet-tripleo dependency isn't)12:16
*** jlinkes_ has joined #tripleo12:16
EmilienMshardy: can you review https://review.openstack.org/#/c/307352/ please?12:16
EmilienMmarios: please do12:16
mariosEmilienM: ack thanks (except now i cant stop thinking of pancakes)12:17
EmilienMmarios: I can give you the recipe :P12:18
*** jcoufal has joined #tripleo12:18
mariosEmilienM: i need pancake as a service12:18
shardyEmilienM: lgtm, I've not approved though as I was a co-author12:19
*** shardy is now known as shardy_lunch12:20
*** jlinkes has quit IRC12:20
EmilienMjaosorior: I'm doing a recheck to try getting upgrade job working12:23
EmilienMI'm curious to see if upgrade actually works12:23
jaosoriorEmilienM: I haven't seen it pass in a long time12:24
EmilienMbut yeah it sounds like http://logs.openstack.org/52/307352/19/check/gate-tripleo-ci-centos-7-nonha-multinode-updates-nv/9840f79/console.html#_2016-09-20_07_27_49_87191112:24
EmilienMthe heat stack is created12:24
EmilienMdamn wrong job12:24
EmilienMgate-tripleo-ci-centos-7-undercloud-upgrades-nv is actually green12:24
EmilienMso it works12:24
*** mburned is now known as mburned_out12:24
jaosoriorso which was red?12:25
EmilienMthe overcloud update12:25
EmilienMbut it's a transient issue12:25
jaosoriorI see12:25
*** jayg|g0n3 is now known as jayg12:25
*** bfournie has joined #tripleo12:27
*** bfournie has quit IRC12:27
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Add devmode.sh and document Third Party CI  https://review.openstack.org/36000712:28
d0ugalshardy_lunch: Just testing your patch12:29
*** mburned_out is now known as mburned12:29
*** mburned is now known as mburns_mtg12:29
*** ayoung has quit IRC12:30
*** fultonj has joined #tripleo12:32
*** bfournie has joined #tripleo12:33
*** bfournie has quit IRC12:33
d0ugalDoes POST_FAILURE mean the job timed out?12:35
d0ugalAlways seems to be about 2 hrs 50 mins12:35
*** jprovazn has quit IRC12:36
EmilienMand that's a bug12:36
EmilienMbecause our jobs should timeout themselves in tripleo-ci12:36
d0ugalEmilienM: k, thanks.12:36
EmilienMd0ugal: which job?12:38
d0ugalEmilienM: Just on this patch: https://review.openstack.org/#/c/36815012:39
d0ugalEmilienM: all of them :) So I think there is an issue with the patch12:39
d0ugalrbrady: You around?12:39
rbradyd0ugal: yep12:39
d0ugalrbrady: I'm trying to figure out if https://review.openstack.org/#/c/368150 has a valid failure or not12:40
d0ugalrbrady: Was going to try and run it manually, have you?12:40
* rbrady reads comments12:40
EmilienMd0ugal: i'll dig a bit12:40
d0ugalEmilienM: Thanks12:40
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375012:41
rbradyd0ugal: I did, but only to generate the passwords that matched what was done in the client before.  I had another issue that wouldn't allow me to create a stack in my env12:41
d0ugalrbrady: ah, okay - so there might be a conflict between the CLI passwords and Mistral passwords I guess12:42
mcorneajaosorior: hiera nova_metadata_network12:43
jaosoriormcornea: awesome! so that patch works well then12:43
rbradyd0ugal: I'll rebuild my env and try it again.  will paste both sets of params12:43
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: swift: normalize memcache servers IP addresses  https://review.openstack.org/37273412:43
EmilienMjaosorior, chem, panda: I added unit tests ^12:44
mcorneajaosorior: yep, thanks for the fix12:44
EmilienMthe won't pass because depends-on doesn't work for unit tests in puppet-tripleo, it's a long story but i'll fix it later12:44
EmilienMthough it works for all other puppet openstack modules12:44
*** rwsu has quit IRC12:44
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types  https://review.openstack.org/36367412:45
*** jlinkes__ has joined #tripleo12:45
jaosoriorEmilienM: the unit tests look pretty good! Could you add one with FQDN just to have it there (and kinda document it informally)12:46
EmilienMjaosorior: yes sir12:46
d0ugalrbrady: k, thanks - I'll wait and try manually later if you can't.12:46
EmilienMjaosorior: will do in 5, pancakes now12:46
*** jpena|lunch is now known as jpena12:46
*** jlinkes__ has quit IRC12:47
*** jlinkes__ has joined #tripleo12:47
jaosoriorduuuude pancakes12:49
*** jlinkes_ has quit IRC12:49
jaosoriorEmilienM: pics or it didn't happen12:49
rookshardy_lunch: heya! are custom roles baked?12:50
*** shardy_lunch is now known as shardy12:50
shardyrook: pretty much, I've not yet had time to write docs/blog about how to use it tho12:50
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: Composable Zaqar services  https://review.openstack.org/33168212:50
shardyrook: I'm considering one final patch to make the user interfaces easier following some feedback from mcornea12:50
shardyrook: I'm planning said patch and docs/blog this week12:51
b00tcatis the Puppet version always 3.x on the images that are built for the overcloud? I always got a 3.x but wanted to make sure12:52
mcorneashardy: btw, I closed this one as not a bug: https://bugs.launchpad.net/tripleo/+bug/1625132 but it would be great if we could have the ResolveNetwork covered automatically as well12:53
openstackLaunchpad bug 1625132 in tripleo "Invalid /etc/hosts files on nodes using custom roles with network isolation" [Undecided,Invalid]12:53
*** Goneri has joined #tripleo12:53
*** jeckersb_gone is now known as jeckersb12:53
shardymcornea: ack, thanks12:53
rookshardy: ok - any rough txt file on how to play with it?12:54
shardyrook: No, sorry, been so busy I've not written anything, but like I said it's on the TODO list (near the top)12:54
shardyrook: mcornea may have some notes as he's been playing with it12:54
shardythere's some copying/sedding of files which ideally I'd like to remove before the final release12:55
rookno shardy12:55
rookthanks for the infomation12:56
mcornearook: I can share what I'm using, I haven't done any notes yet12:56
ccamacho|lunchfolks I'm getting issues deploying liberty/mitaka in my local env. mind to check this fix https://review.openstack.org/#/c/373131/  to close (https://bugs.launchpad.net/tripleo/+bug/1625463 and https://bugs.launchpad.net/tripleo/+bug/1625315)12:56
openstackLaunchpad bug 1625463 in tripleo "Package dependency issue in liberty deployments" [Undecided,In progress] - Assigned to Carlos Camacho (ccamacho)12:56
openstackLaunchpad bug 1625315 in tripleo "Can't deploy Mitaka overcloud on CentOS" [Undecided,In progress] - Assigned to Carlos Camacho (ccamacho)12:56
rookmcornea: cool thanks!12:56
*** ccamacho|lunch is now known as ccamacho12:56
*** david-lyle has joined #tripleo12:56
mcornearook: https://gist.github.com/mcornea/72232edfccd9e3088373553d4c57d232 : i'm deploying 2 separate network nodes for running the neutron agents13:01
*** rwsu has joined #tripleo13:03
*** jlinkes_ has joined #tripleo13:03
rooksweet mcornea!13:03
tobias-fiberdatashardy, there's a bug in cloud-init, do you know what version of cloud-init it is in the images that boot?13:04
tobias-fiberdatashardy, atleast for injecting passwords13:04
EmilienMjaosorior: no pancakes this morning but doing my own bread :) https://www.flickr.com/gp/134083237@N03/19S3J713:05
rbradyshardy, dprince: need a discussion about state of parameter generation for newton13:05
jaosoriorEmilienM: Hey! not bad!13:05
EmilienMjaosorior: that picture is from January, but I do this thing every week :)13:06
dprincerbrady: sure13:06
*** jlinkes__ has quit IRC13:07
*** pgadiya has quit IRC13:07
*** cdearborn has joined #tripleo13:08
*** liverpooler has quit IRC13:08
rbradydprince: there's a single password retrieved from a file in the CLI in the ~/undercloud_passwords.conf file13:09
*** liverpooler has joined #tripleo13:09
rbradydprince: undercloud_ceilometer_snmpd_password, and it's stored in SnmpdReadonlyUserPassword13:09
*** jaosorior has quit IRC13:09
rbradydprince: this is generated in instack-undercloud I think13:09
rbradydprince: the mistral action is never going to see the file13:10
*** jaosorior has joined #tripleo13:10
dprincerbrady: can we just generate it somewhere in our workflows?13:10
dprincerbrady: perhaps even a generic mechanism to do more parameters like this in the future if we need it13:10
*** hjensas has quit IRC13:11
d0ugaldprince: doesn't it need to match the value from instack?13:11
dprinced0ugal: it would, for those upgrading I suppose13:11
shardyIf upgrading with an overcloud in place we could just read it from heat13:12
shardythe tricky one is upgrading an undercloud if the operator deleted the overcloud first I think13:12
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Make apache-based services use network-dependent servername  https://review.openstack.org/36558513:12
d0ugalshardy, dprince: How would you feel about mistral accessing hiera? (I'm not sure how I feel about that)13:13
d0ugalbecause it is there too13:13
dprinced0ugal: the "execution" workflow would give you this. Just a simple script...13:14
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Hook internal TLS flag to apache-based services  https://review.openstack.org/36607513:14
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add flag for internal TLS  https://review.openstack.org/36594213:14
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service  https://review.openstack.org/35643013:14
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add option to specify Certmonger CA  https://review.openstack.org/36864313:14
rbradydprince, shardy, d0ugal: here's where it is in instack-undercloud https://github.com/openstack/instack-undercloud/blob/e42b80e69da8d491aa3b98963b2da99733319763/elements/undercloud-install/os-apply-config/root/tripleo-undercloud-passwords#L513:14
dprinced0ugal: if that is all we've got I don't feel too bad about it13:14
d0ugaldprince: it would be the simplest solution :)13:14
dprinced0ugal: I actually feel fine about it13:14
dprinced0ugal: good even13:14
dprinced0ugal: I'd rather grab it from an API, but if we can't then this works13:15
d0ugalrbrady: https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/utils.py#L452-L46213:15
d0ugalrbrady: so that could be a fallback13:15
d0ugalI assume the mistral action can access that, but I've not tested it13:15
shardy+1 it feels kind of hacky, but at this point I think we have to shoot for expediency over elegance ;)13:15
d0ugalshardy: Yeah, that is my thinking :(13:15
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart: Add settings to general config  https://review.openstack.org/37114413:15
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart: Remove settings from release configs  https://review.openstack.org/37111413:15
openstackgerritOpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements  https://review.openstack.org/37331513:16
rbradyd0ugal: I thought there was a std action in mistral to execute a script, but I'm not seeing it.13:16
d0ugalrbrady: std.ssh maybe?13:17
shardyit'd be pretty scary if they allowed arbitrary scripts to be run on the mistral server by default ;)13:17
d0ugalrbrady: that what you might be thinking of anyway13:17
rbradyd0ugal, shardy: I'll port the get_hiera_key function over for now13:17
*** limao_ has joined #tripleo13:18
d0ugalrbrady: Sounds good, and it might be a good idea to add an ocata bug for this.13:18
d0ugalI think we need to revisit the password storage stuff in general13:18
d0ugalsince mistral envs are plain text in the database AFAIK13:19
rbradyd0ugal: like moving password storage to barbican for ocata?  sounds good.13:19
d0ugalrbrady: If we are allowed to add Barbican, sure :)13:19
shardytobias-fiberdata: it's 0.7.5 atm in my image, but you can check with yum install libguestfs-tools && virt-inspector -a overcloud-full.qcow2 | grep -A10 cloud-init13:19
shardy(or an xpath query to virt-inpsector instead of the grep)13:20
*** limao has quit IRC13:21
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375013:21
*** lblanchard has joined #tripleo13:21
tobias-fiberdatadoes it look ok? http://paste.openstack.org/show/582243/13:21
*** dsariel has joined #tripleo13:22
tobias-fiberdatathen i have a environmentfile i start with13:22
tobias-fiberdatai'll guess we try13:22
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375013:22
*** egafford has joined #tripleo13:24
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove the get_hiera_key function  https://review.openstack.org/36736713:24
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart: Add settings to general config  https://review.openstack.org/37111413:28
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375013:31
*** tzumainn has joined #tripleo13:31
*** cylopez has left #tripleo13:33
*** limao_ has quit IRC13:36
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: swift: normalize memcache servers IP addresses  https://review.openstack.org/37273413:36
EmilienMpanda: I had to make sure we return an array ^13:36
*** limao has joined #tripleo13:36
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove unused code found by vulture  https://review.openstack.org/37333013:36
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types  https://review.openstack.org/36367413:36
*** openstackstatus has joined #tripleo13:37
*** ChanServ sets mode: +v openstackstatus13:37
openstackgerritMerged openstack/tripleo-common: Add template processing to the update plan workflow.  https://review.openstack.org/37102713:39
openstackgerritMerged openstack/instack-undercloud: Work around deletion of _member_ role assignments on upgrade  https://review.openstack.org/30735213:39
EmilienMjaosorior: what is prio of https://bugs.launchpad.net/bugs/1625448 ?13:40
openstackLaunchpad bug 1625448 in tripleo "The undercloud's Zaqar Websocket endpoint always plaintext" [Undecided,In progress] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles)13:40
*** gfidente has quit IRC13:42
-openstackstatus- NOTICE: OpenStack Infra now has a Twitter bot, follow it at https://twitter.com/openstackinfra13:45
*** hrybacki is now known as hrybacki|mtg13:45
openstackgerritSaravanan KR proposed openstack/python-tripleoclient: Remove the reference to overcloud-without-mergepy file  https://review.openstack.org/37333613:49
*** yamahata has joined #tripleo13:50
EmilienMtripleo meeting in 10 min, please add off items on https://etherpad.openstack.org/p/tripleo-meeting-items13:50
*** jprovazn has joined #tripleo13:50
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fix the default plan creation  https://review.openstack.org/37134713:51
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Verify that the Deployment Plan creation was successful  https://review.openstack.org/36924713:51
*** dmacpher has joined #tripleo13:52
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart: Remove settings from release configs  https://review.openstack.org/37111413:53
*** rajinir has joined #tripleo13:54
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Update the Mistral action names  https://review.openstack.org/36651913:55
jaosoriorEmilienM: High. it's plaintext right now13:55
EmilienMjaosorior: lol13:55
EmilienMjaosorior: indeed :)13:55
pandaEmilienM: shouldn't the return array have been resolved in 372903 ?13:55
openstackgerritDougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names  https://review.openstack.org/36652913:56
jaosoriorEmilienM: can you review the bug report for this? https://review.openstack.org/#/c/373141/13:57
*** limao has quit IRC13:57
*** limao has joined #tripleo13:58
EmilienMpanda: no because it can return a string if we pass a string when using the function13:58
jaosoriorEmilienM: the whole problem is that it's included in nova-api. Since that's the case haproxy doesn't find nova_metadata_vip and nova_metadata_node_ips because they are not generated (since it's not a service) and then gets configured wrong13:58
EmilienMjaosorior: yes13:58
jaosoriorEmilienM: talked about it with shardy, that there are two solutions13:58
jaosorior1) add a new serviec profile13:58
EmilienMmeeting in 2 min13:58
EmilienMok I see, because of the service_name13:58
jaosorior2) hardcode nova_metadata_vip, network and node IPs13:58
jaosoriorEmilienM: yeah :/13:59
EmilienMcan't you use the same vip?13:59
openstackgerritBrad P. Crochet proposed openstack/puppet-tripleo: Add zaqar profiles  https://review.openstack.org/33168113:59
EmilienMok meeting13:59
openstackgerritHonza Pokorny proposed openstack/tripleo-ui: When deploy finishes, show overcloud info  https://review.openstack.org/37076513:59
jaosoriorEmilienM: we can, but then conifiguring the service_ip_map won't work13:59
jaosoriorshardy: mind checking this out? https://review.openstack.org/#/c/373141/14:00
*** jlinkes__ has joined #tripleo14:00
EmilienMjaosorior: meeting now14:00
openstackgerritAlejandro Andreu proposed openstack/tripleo-puppet-elements: Midonet element now complies with version 5.2  https://review.openstack.org/37335214:01
*** jlinkes_ has quit IRC14:04
openstackgerritDan Prince proposed openstack/tripleo-specs: Single process Heat driven Undercloud installer  https://review.openstack.org/35211314:04
*** abehl has quit IRC14:06
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Configure heat engine to use convergence  https://review.openstack.org/33389014:08
*** oshvartz has quit IRC14:08
*** saneax-_-|AFK is now known as saneax14:15
*** ayoung has joined #tripleo14:15
*** hjensas has joined #tripleo14:18
*** hjensas has quit IRC14:18
*** hjensas has joined #tripleo14:18
*** limao_ has joined #tripleo14:18
*** bfournie has joined #tripleo14:20
*** limao__ has joined #tripleo14:20
*** limao has quit IRC14:22
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates: Fix nested list for enabled_services  https://review.openstack.org/37006914:23
*** limao_ has quit IRC14:23
*** gfidente has joined #tripleo14:25
*** gfidente has quit IRC14:25
*** gfidente has joined #tripleo14:25
*** zoli|trng is now known as zoli14:26
*** tosky has quit IRC14:28
tobias-fiberdatashardy, apperently it's turning off all interfaces?14:30
tobias-fiberdatashardy, managed to inject a user/password14:30
tobias-fiberdataand all my interfaces were down14:30
*** tosky has joined #tripleo14:33
openstackgerritMerged openstack/tripleo-ui: Add Role Detail  https://review.openstack.org/36799314:35
*** fultonj has quit IRC14:37
shardytobias-fiberdata: you can run os-net-config manually on the CLI mode (it also has a noop mode), which should show you why14:38
tobias-fiberdatashardy, alright14:39
*** fultonj has joined #tripleo14:39
openstackgerritMerged openstack/instack-undercloud: Enable TLS for Zaqar's websocket endpoint  https://review.openstack.org/36035014:42
hewbroccawhoa, TLS for zaqar14:42
*** limao__ has quit IRC14:42
hewbroccajaosorior: nice one dude14:42
tobias-fiberdatashardy, loads of questions, but how do i do the manually os-net-config?14:43
tobias-fiberdataseirously we have been having an openstack-plattform for 2years but i feel like a freaking noob when it comes to this14:43
jaosoriorhewbrocca: thanks14:43
*** limao has joined #tripleo14:44
jaosoriortobias-fiberdata: I can relate; openstack feels that way always14:44
jaosoriortobias-fiberdata: by the way, the weekly meeting is on-going on #openstack-meeting-alt so people might be a bit busy. It's only 15 min left though14:45
*** dtrainor has joined #tripleo14:45
tobias-fiberdatathat's okey14:45
shardytobias-fiberdata: os-net-config -c /etc/os-net-config/config.json14:45
tobias-fiberdatashardy, thanks alot man14:45
shardyoptionally add the --verbose and --noop flags14:46
tobias-fiberdatayou are a diamond14:46
tobias-fiberdatathis explains everything14:47
tobias-fiberdatayou are really a diamond shardy taking your time helping out! ofc, everyone else should have credit aswell14:47
sshnaidmderekh, matbu  I see upgrades job runs on "centos-7-2-node-internap-mtl01-4410463" and it doesn't seem like rh1 cloud14:48
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: TESTING - DO NOT MERGE  https://review.openstack.org/37338614:48
sshnaidmderekh, how do we move it to run on rh1?14:48
jaosoriorhewbrocca: and I actually have TLS for all endpoints in haproxy working. But I have to wait for Ocata to merge that.14:50
hewbroccaToo bad14:51
openstackgerritMerged openstack/python-tripleoclient: Use the hexdigest of the path to make the filename unique in swift  https://review.openstack.org/37315914:52
jeckersbany clue why this newton overcloud deployment just kinda silently fails? - https://paste.fedoraproject.org/431495/14743831/14:57
jeckersbin the end there's just no stack14:57
jeckersbnot sure why, don't see an error anywhere14:57
jaosoriorjeckersb: does mistral execution-list print any error?14:57
*** hrybacki|mtg is now known as hrybacki14:57
jaosoriorEmilienM: It seems that scenario001 is broken :/14:58
d0ugaljeckersb: do you have a heat stack? $ openstack stack list14:58
jeckersbjaosorior: yep!14:59
jeckersbd0ugal: nope no stack14:59
jaosoriorjeckersb: can you do mistral execution-get <id> ?15:00
* jeckersb is unfamiliar with the new mistral stuff so is totally lost15:00
jaosoriorthat will give you more info15:00
EmilienMjaosorior: is it voting?15:00
jaosoriorEmilienM: it's not15:00
EmilienMso :)15:00
EmilienMI know it's broken, something with gnocchi15:00
jaosoriorI see15:00
EmilienMI never took the time to investigate since RC1 and 2 things15:00
jaosoriorI wasn't aware it's a known issue.15:00
EmilienMit's on my list15:00
* d0ugal needs to finish his debugging tripleo and mistral blog post15:01
*** limao has quit IRC15:01
d0ugaljaosorior, jeckersb - that error can probably be ignored.15:01
bnemecjeckersb: It looks like this is a script and you're doing || true on the deploy command?15:01
*** limao has joined #tripleo15:01
bnemecThere's probably a failure in stderr or something.15:02
*** chem has quit IRC15:02
d0ugaljaosorior, jeckersb - if that is the only one, ignore it. Everyone will be getting that now as the default plan creation is broken. (see https://review.openstack.org/371347)15:02
jeckersbbnemec: yeah this is from quickstart, running the overcloud-deploy.sh15:02
EmilienMjaosorior: +2 on metadata, you win15:02
*** limao has quit IRC15:02
jeckersbYaqlEvaluationException: Can not evaluate YAQL expression: ...15:03
*** limao has joined #tripleo15:03
jeckersblooks like the root cause15:03
jaosoriorEmilienM: well, for Ocata we could attempt to split nova-api and have a metadata-specific manifest15:03
jaosoriorshardy: can you review https://review.openstack.org/#/c/373141/ ?15:04
*** bandini has quit IRC15:04
EmilienMjaosorior: ok15:04
EmilienMjaosorior: please file a bug and track it in ocata-1 - thx a ton15:05
*** bandini has joined #tripleo15:05
shardyYeah, earlier we discussed it, and there is coupling in puppet-nova here:15:05
derekhsshnaidm: here is where the job is defined http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n1395715:05
shardyprobably not high priority, but we might look at breaking that in ocata so the seperate templates work better with the underlying puppet abstractions15:05
derekhyou switch to OVB, change the node from centos-7-2-node too tripleo-centos-715:05
jaosoriorEmilienM: for puppet-nova?15:06
EmilienMjaosorior: probably15:06
sshnaidmderekh, yeah, and rename it maybe15:06
EmilienMshardy: right15:06
*** yamahata has quit IRC15:07
derekhsshnaidm: yup15:07
EmilienMjaosorior, shardy: also, we could deploy nova-api with medatata service only and have osapi in wsgi with apache15:07
derekhsshnaidm: and remove multinode from the type15:07
sshnaidmderekh, yeah, sure15:07
shardyYeah, there's probably a few valid reasons for wanting these split up15:07
shardyI approved the patch, we can revisit in ocata15:08
EmilienMshardy: ++15:08
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Make apache-based services use network-dependent servername  https://review.openstack.org/36558515:08
mwhahahaso what is the solution to the overcloud-without-mergepy.yaml thing, i just tried a fresh HA deploy and got a 404 on that.15:10
shardymwhahaha: what version of tripleo-common are you running?15:11
shardyand did you do any overcloud deploy commands before seeing the 404?15:11
*** ebarrera has quit IRC15:13
shardymwhahaha: Ok, that should work, but you'll have to do an openstack stack delete overcloud && openstack overcloud plan delete overcloud between deploy (of overcloud) attempts until you get https://github.com/openstack/tripleo-common/commit/2d87e9640427f12eb5604f290fe38eae00f402ee15:13
jaosoriorEmilienM: done https://bugs.launchpad.net/tripleo/+bug/162567915:13
openstackLaunchpad bug 1625679 in tripleo "no manifest for deploying nova-metadata specific resources" [Low,New]15:13
*** oshvartz has joined #tripleo15:13
sshnaidmderekh, matbu https://review.openstack.org/#/c/373406/15:13
shardymwhahaha: what does swift download overcloud overcloud.yaml give you?15:13
EmilienMjaosorior: thx15:13
mwhahahashardy: http://paste.openstack.org/show/582265/15:14
shardymwhahaha: also check /var/log/mistral/mistral-server.log for errors around the "jinja2 rendering overcloud.j2.yaml" logs15:14
mwhahahashardy: no errors15:15
shardymwhahaha: Ok, so the j2 rendering worked, the deploy should just use that, it shouldn't care about the overcloud-without-mergepy.yaml15:15
shardy(thats the old deprecated name we support for backwards compat and should probably remove)15:15
*** lblanchard has quit IRC15:15
*** jlinkes__ has quit IRC15:16
*** lblanchard has joined #tripleo15:16
shardymwhahaha: I'd tail the mistral logs, then delete the heat stack (if there is one, probably there isn't), and do openstack overcloud plan delete overcloud15:16
shardythen do openstack overcloud deploy --templates while looking at the mistral logs for errors15:16
shardymwhahaha: also, you may need this patch if you're using relative paths:15:17
mwhahahai saw some zaqar errors15:17
mwhahahai'm not, i litterally just followed the examples from the docs15:17
shardybut I'd expect it to fail on some file other than overcloud* if that's the problem15:17
jaosoriormwhahaha: actually, there is a zaqar bug. You need this patch which just merged today https://review.openstack.org/#/c/369285/15:18
shardymwhahaha: ack, just giving a list of the current known issues ;)15:18
derekhbnemec: slagle: btw, we don't even seem to be getting up to 50 simultanious jobs at the moment, we're in the mid 20's15:18
hewbroccaderekh: I was asking dbecker earlier15:18
*** sshnaidm is now known as sshnaidm|afk15:18
hewbroccais *all* our traffic going through the bastion host?15:18
derekhhewbrocca: yes15:19
derekhhewbrocca: no15:19
shardyderekh: top marks for clarity ;)15:19
derekhhewbrocca: all our *outside* traffic15:19
derekhshardy: thanks15:19
*** lucasagomes is now known as lucas-hungry15:21
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: swift: normalize memcache servers IP addresses  https://review.openstack.org/37273415:21
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types  https://review.openstack.org/36367415:22
hewbroccaderekh: Is there any reason we're running public API traffic through the bastion host?15:23
derekhhewbrocca: when the rack was setup origionally (2+ years ago), that how it was setup and given to us, the public link is only connected to the bastion15:24
hewbroccaderekh: interesting15:25
hewbroccaMaybe we don't care, but it seems odd15:25
*** leanderthal is now known as leanderthal|afk15:26
*** pkovar has quit IRC15:28
derekhhewbrocca: I'm sure its caused problems over time, I'm not sure how hard it would be to change, pabelanger started a email about changing it last week15:29
hewbroccaoh really15:29
*** yolanda has quit IRC15:29
derekhhewbrocca: http://lists.openstack.org/pipermail/openstack-dev/2016-September/103344.html15:30
*** zoli is now known as zoli|gone15:31
*** ramishra has quit IRC15:31
openstackgerritSaravanan KR proposed openstack/python-tripleoclient: Remove the reference to overcloud-without-mergepy file  https://review.openstack.org/37333615:31
*** zoli|gone is now known as zoli_gone-proxy15:31
*** tremble has quit IRC15:32
*** fzdarsky has quit IRC15:33
*** akshai has joined #tripleo15:33
*** jbadiapa has quit IRC15:38
*** jaosorior has quit IRC15:40
*** dprince has quit IRC15:42
*** aufi has quit IRC15:43
slagleEmilienM: since we have "include ::keystone::cron::token_flush15:43
slaglein the undercloud puppet manifest...15:43
slaglewhere should we see the cron job defined?15:43
slaglebecause I dont see it anywhere15:44
EmilienMunder the keystone user15:44
EmilienMyou should see it15:44
slagleoh ok, let me check15:44
slaglei was checking root and stack15:44
*** skramaja has quit IRC15:45
slagleEmilienM: ok, i see it under the keystone user, thanks15:45
EmilienMslagle: https://github.com/openstack/puppet-keystone/blob/master/manifests/cron/token_flush.pp#L8015:46
EmilienMwe create the crontab under keystone user to make sure logs can be written correctly15:46
EmilienMotherwise root would override logs and keystone will crash15:47
EmilienMbecause logs need to be owned by keystone15:47
d0ugalHow do I get to a point where two patches that depend on the other can land? CI fails with either on their own, but together it passes. https://review.openstack.org/#/c/369247/ and https://review.openstack.org/#/c/371347/15:47
EmilienMd0ugal: damn15:47
EmilienMlet me look15:47
bnemecCan I get +2a on https://review.openstack.org/#/c/365114/ ?15:48
EmilienMd0ugal: it means the patches are not backward compatible15:48
EmilienMbnemec: boom15:48
d0ugalEmilienM: in what way?15:49
*** derekh has quit IRC15:49
EmilienMbnemec: how can we say no to so much green :)15:49
EmilienMd0ugal: well, https://review.openstack.org/#/c/371347/ seems backward incompatible15:49
bnemecEmilienM: Thanks15:49
d0ugalEmilienM: it depends on https://review.openstack.org/#/c/369247/ (and vice versa)15:50
d0ugalI guess I could split it into a series of three patches...15:50
EmilienMd0ugal: but still backward non compatible15:50
EmilienMwe have the same problem sometimes in puppet/THT15:50
EmilienMwe have to split in 3 patches15:50
bnemecd0ugal: I've had to do that in the past.15:50
EmilienMto have a transition period15:50
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Remove one last reference to openstackclient.tests  https://review.openstack.org/37343515:50
bnemecDammit, has that still not merged?15:51
bnemecOh, stable/newton15:51
EmilienMjpich is our stable/newton guardian15:52
EmilienMrbrady, d0ugal, jpich, shardy: have you seen https://review.openstack.org/#/c/370423/ ?15:55
*** ebarrera has joined #tripleo15:55
EmilienMmy goal is to prevent tripleo CI breakages when patching mistral15:55
d0ugaloh, nice15:56
EmilienMI'm not sure mistral PTL saw it :P15:56
d0ugalProb not15:56
d0ugalEmilienM: will it be voting?15:56
EmilienMd0ugal: yes (lol)15:56
rbradyEmilienM: that's great15:57
*** dprince has joined #tripleo15:58
*** ramishra has joined #tripleo15:59
trownslagle: do you know of any workaround for that password prompt issue in osc?16:00
*** rasca has quit IRC16:01
slagletrown: pressing Enter16:02
trownfound that one :P16:02
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120916:03
*** jcoufal_ has joined #tripleo16:04
trownhmmm I suppose `printf '\n' | tripleo.sh` would also work16:04
pandaEmilienM: we should add Closes-Bug: #1618491 to https://review.openstack.org/36367416:05
openstackbug 1618491 in tripleo "Add IPv6 tests in CI for newton" [Wishlist,In progress] https://launchpad.net/bugs/1618491 - Assigned to Gabriele Cerami (gcerami)16:05
EmilienMpanda: yes, please not now, jobs are running :)16:05
pandaEmilienM: yep, that's why I said it, and not done it.16:06
pandabandini: *prod*16:06
EmilienMpanda: thanks :)16:06
*** jcoufal has quit IRC16:08
shardyrbrady: Hey, did you start any patches that decouple process templates from the j2 rendering?16:11
shardyrbrady: I'd like to add a -r somecustom_roles_data.yaml to tripleoclient16:11
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Verify that the Deployment Plan creation was successful  https://review.openstack.org/36924716:11
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Ensure that the default plan was created successfully  https://review.openstack.org/37344616:11
shardybut that implies creating the plan, then updating it with a new roles_data, then rendering the plan16:11
shardytrying to figure out the least hacky way to do it16:11
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fix the default plan creation  https://review.openstack.org/37134716:12
*** mah has joined #tripleo16:12
rbradyshardy: I did, but thought this would be better suited for after the release in the start of Ocata, https://review.openstack.org/#/c/370868/16:12
mahhi all, anybody tried before fake pxe in his deployment ?16:12
shardyrbrady: ack, yeah probably best to wait, I was looking for a quick way to add it16:13
shardyI guess we'll document cp -r in the docs for Newton ;)16:13
rbradyshardy: it's up to you.  if you want to add this, I could rebase the patch and add a better commit message16:14
shardymah: Yes, I've used it, you'll need to modify the undercloud ironic configuration but basically it works16:14
mahI am using OPNFV-Apex which is based on TripleO16:15
rbradyshardy: I was just worried that it's yet another patch that needs to land.  while the current code is less efficient, it's not causing exceptions that I know of16:15
mahand adjusted it to use the fake pxe16:15
mahand it receive the pxe ips16:15
shardyrbrady: If you can rebase it and update the commit msg that would be good, then I'll base my testing on it for early ocata16:15
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Ensure that the default plan was created successfully  https://review.openstack.org/37344616:15
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Verify that the Deployment Plan creation was successful  https://review.openstack.org/36924716:15
mahand then after provisioning , the nodes are shutdown automatically16:15
rbradyshardy: ack16:15
shardyrbrady: The main issue is we document overridding packaged tripleo-heat-templates, which is going to be impossible with custom roles for newton16:16
*** ohamada has quit IRC16:16
mahand then I power them up but it gave me black screen with cursos flashing at the left corner16:16
shardyas you need to copy the tree somewhere, modify roles_data.yaml, then upload to the plan16:16
shardyrbrady: I guess this is advanced functionality so that will have to do for this release16:16
*** dsariel has quit IRC16:17
shardymah: are these uefi boxes?16:17
mahby the way the configuration should be legacy first or UEFI first ?16:18
shardyOk, I'd maybe chat with the ironic folks, as I know some fixes related to uefi landed quite recently16:18
shardyand I guess you're working on an older fork/branch of openstack?16:18
d0ugalhah, well, I think I confused Depends-On :/16:18
shardymah: when I last tested I couldn't get uefi boot to work, I had to set the box to boot in legacy mode in the bios16:18
shardyI think that is fixed now, but cannot personally confirm it16:19
shardylucas-hungry may have some additional information16:19
mahok I will adjust them to leagacy and see16:19
mahI have to enable CSM16:20
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: Tripleoclient leaks temporary files  https://review.openstack.org/37344916:20
*** lucas-hungry is now known as lucasagomes16:22
lucasagomesshardy, mah hi there... Yes we recently landed some changes in instack-undercloud and also puppet-ironic to configure UEFI by default16:24
lucasagomesshardy, mah by default I mean, just to have it configured... if you wanna use UEFI for deployment you still have to set the capability "boot_mode:uefi" in Ironic and Nova flavor16:25
*** florianf has quit IRC16:26
shardylucasagomes: thanks for the info!16:26
shardymah: ^^16:26
mahthanks for the info Lucas16:27
mahI would like to know why when it is powered off automatically, and I power them up again , it does not show me the login  and shows black screen !!16:28
*** saneax is now known as saneax-_-|AFK16:29
shardymah: probably because the bootloader isn't installed correctly16:29
mahbootloader which is configured at the jumphost ?16:30
lucasagomesmah, so the instance is marked as active ? And the node is powered off ?16:30
lucasagomesso you power it on again and you don't see anything in the console  ?16:30
mahI am using opnfv-apex16:30
*** bana_k has joined #tripleo16:31
* lucasagomes googles16:31
mahso at the overcloud deployment , I power on the nodes , then they receive the ips .. and get configured ... after some time  .. I found all the nodes are powered off ,, I think that happen by the jumphost16:31
mahbecause usually in opnfv-apex after provisioning the overcloud nodes, the nodes should be restarted16:32
mahsince we use fake pxe16:32
mahso they are powered off16:32
mahand I power them up again16:32
mahat this time it found the same ip it has assigned before and get for me a black screen16:33
lucasagomesmah, oh if you use fake_* driver ironic can't power control it, so it's unlikely that ironic is doing something here16:33
lucasagomesmah, for the console you may want to add console=ttyS0 to the kernel cmdline (in the bootloader) maybe ?16:34
*** trown is now known as trown|lunch16:35
mahso why it is powerd off16:35
shardylucasagomes: FWIW I saw similar when I last tested a uefi box16:36
shardythe ramdisk shuts the node off after silently failing to fully install the bootloader16:36
shardythen you turn it on and no bootloader16:36
shardyflashing cursor16:36
lucasagomesshardy, oh right, and you were using the overcloud-full disk image ?16:37
shardywhich is why I suspect mah may need the uefi fixes, or to enable legacy boot mode16:37
openstackgerritAna Krivokapic proposed openstack/tripleo-ui: Extract Mistral action and workflow names into constants  https://review.openstack.org/36668516:37
shardylucasagomes: yeah16:37
mahI am now using leagcy mode16:37
openstackgerritimain proposed openstack/tripleo-common: Create new docker command hook.  https://review.openstack.org/31272316:37
lucasagomesshardy, gotcha, yeah UEFI in Ironic is complicated cause we don't actually test it in the gate16:38
mahbut at the first time before it power off16:38
lucasagomesshardy, mah just today I created this: https://bugs.launchpad.net/ironic/+bug/162561616:38
openstackLaunchpad bug 1625616 in Ironic "[RFE] Support for UEFI deployment in DevStack / Gate tests for UEFI deployments" [Wishlist,New] - Assigned to Lucas Alvares Gomes (lucasagomes)16:38
mahit shows to me that I can log in16:38
lucasagomeswhich I'm hoping to improve soon16:38
lucasagomesshardy, but I will take a look at this problem then16:38
shardylucasagomes: sounds good, thanks!16:38
lucasagomesmah, and what is the node provision state in Ironic ?16:39
mahI did not see it as the deployment was not finished yet there16:39
lucasagomesif the bootloader fails to install ironic should have detected it and failed the deployment as well16:39
lucasagomesmah, ironic node-show <node uuid>16:40
lucasagomesyou can run it while it's deploying16:40
mahand how to get uuids16:40
mahironic node-list16:40
mahok will check16:43
rookmcornea: so looking through this gist you sent for custom roles... you still pass --control-scale 316:44
mcornearook: yes, I had 3 controllers + 2 network nodes, for the network nodes I used NetworkerCount: 2 in network-environment.yaml16:45
*** jpich has quit IRC16:48
mariosgfidente: please revisit your +2 https://review.openstack.org/#/c/366760/ when you have 1 minute thanks16:48
mahnow it is showing the power state is on16:51
mahand provisioning state is deploying16:51
openstackgerritBrent Eagles proposed openstack/tripleo-docs: Remove references to balance-tcp  https://review.openstack.org/37259016:52
openstackgerritHonza Pokorny proposed openstack/tripleo-ui: When deploy finishes, show overcloud info  https://review.openstack.org/37076516:53
dprinceshardy: It would be good to get this in finally https://review.openstack.org/#/c/312723/16:55
openstackgerritMerged openstack/tripleo-specs: Simplify the TripleO Workflow  https://review.openstack.org/23819216:55
*** rcernin has quit IRC16:55
Slowerdprince: thanks16:55
shardydprince: it's still being proposed to the wrong repo?16:56
dprinceshardy: I don't think steve baker wanted it in heat-templates16:57
shardyI commented on that back in july16:57
Slowershardy: It's going to transmute into something tripleo specific16:57
dprinceshardy: which causes us some packaging issues16:57
dprinceshardy: perhaps I misunderstood16:57
dprinceshardy: I'd actually be happy for it to live in heat-templates FWIW16:57
SlowerIt's unclear to me what the best thing to do here is16:57
dprinceshardy: it makes a nice subpackage then off of os-collect-config per stevebaker's latest work16:57
SlowerIt's going to become more and more tripleo specific16:57
beaglesshardy: was there a bug around the nova metadata ip thing that jaosorior brought up earlier today... it might also be breaking DVR. Basically the metadata IP being used on the compute node is that of the compute node itself not the controller.16:57
shardyI just wanted us to use a generic heat hook that runs docker, vs a special snowflake tripleo thing16:57
shardyif we have to go with tripleo-common for now I guess that's fine16:58
shardyI just would have preferred a more generic approach16:58
beaglesshardy: looking to see if it is something else bunging up the works though... it's kinda weird16:58
shardybeagles: Yeah, I think we landed the fix16:58
beaglesshardy: oh sweet16:58
beaglesshardy: I'll update and try again16:59
* beagles hopes it wasn't the fix that broke it :)16:59
lucasagomesmah, so this machines you are deploying, can you check what is the boot device order ?16:59
shardydprince: stevebaker said "I'd really like to see hook-docker.py live in heat-templates"16:59
shardyso AFAICT he and I are agreeing16:59
dprinceshardy: well then I agree with you all too :)17:00
shardyI don't want to stop progress on this either, but I gave this feedback a long time ago17:00
mahthe device order is nic then the disk Centos17:00
*** dsneddon has joined #tripleo17:00
lucasagomesmah, I wonder if the fake_ driver can also change the boot device, therefore after the installation it may still want to PXE boot the node (and you are using local boot, which should go trough the disk instead)17:00
dprinceSlower: can we move this over real quick?17:00
Slowerwe could yes17:00
Slowerfwiw it makes building the container a pita17:00
Slowerso 6 of one..17:00
dprinceSlower: actually, I think it'll be fine to build the container17:01
dprinceSlower: because we'll have a proper package for this17:01
dprinceSlower: step 1) Add it to heat-templates17:01
dprinceSlower: step 2) create a subpackage like this https://review.rdoproject.org/r/#/c/2106/17:01
shardybeagles: actually, it's still gating : https://review.openstack.org/#/c/373141/17:01
Slowerdprince: gotcha17:01
dprinceSlower: step 3) update the tripleo containers to simply install that package. Done!17:01
dprinceSlower: super clean I think17:02
beaglesshardy: ah cool.. thanks17:02
Slowerdprince, shardy: ok I'll move it over17:02
*** mcornea has quit IRC17:02
shardydprince, Slower: thanks guys, sorry to push the issue17:02
*** hewbrocca is now known as hewbrocca-afk17:02
Slowerno worries17:02
shardyhopefully it will be worth the effort to keep things clean in the long run17:02
Slowerya probly not ;-)17:03
* Slower runs and hides17:03
lucasagomesmah, so, check ironic node-show <node uuid>... in the "properties" field, do you see a key called "capabilities" ? If so, do you see a capability called "boot_option:local" ?17:03
mahI have it network as first place in boot order17:03
shardySlower: You don't think it's worth using a common heat community maintained hook?17:03
lucasagomesmah, right, and after the deployment Ironic would be trying to set it to boot from the HD, but as you are using the fake driver none will happen17:04
Slowershardy: I don't think anyone else is going to use it honestly17:04
lucasagomesmah, so once you power the node on again it will try to PXE boot, that's why i think it's not booting for you17:04
lucasagomesmah, see if you have that boot_option:local capabilities set17:04
Slowerbut you never know17:04
mahboot_option where ? in the jumphost node or the other nodes ?17:05
lucasagomesmah, in the ironic nodes17:05
shardySlower: tbh I think just getting the packaging sorted out for free is reason enough17:05
lucasagomesmah, "ironic node-show <node uuid>"17:05
shardyref the comment from stevebaker17:05
* shardy notes we may have to buy stevebaker beer17:05
Slowershardy: true17:05
*** panda is now known as panda|dinner17:06
*** jpena is now known as jpena|off17:06
openstackgerritMerged openstack/instack-undercloud: Add sysctl preflight install check  https://review.openstack.org/37260517:06
mahcapabilities is boot_option:local17:06
lucasagomesmah, right... so I think that's the problem17:07
lucasagomessince you are using the fake_ driver ironic won't be able to set the boot mode of the node to boot from the HD17:08
lucasagomesit will be no-op17:08
mahso how to solve it ?17:08
*** sai is now known as sai|mtg17:08
lucasagomesmah, you have to options: 1. change the boot mode manually after the deployment 2. always boot from the network17:08
lucasagomesmah, if you awnna go with #2 (I think it will be better) you can do:17:08
mahI already using 217:09
mahto boot from the PXE17:09
maheven before the deployment17:09
lucasagomesmah, but ironic is not, ironic still thinks it can change the boot device17:09
lucasagomesbut it's "fake"17:09
lucasagomesit's a mock up driver17:09
mahyes I know17:09
lucasagomesmah, so to tell ironic to always boot from network do: ironic node-update <node uuid> remove properties/capabilities17:09
mahnow we have two options to boot from network or to boot from HD17:09
mahI do the sequence to be always from the network17:10
mahmmmm you mean to change it to the ironic17:10
lucasagomesmah, if you have that capabilitity set in ironic matching with the flavor ironic will assume you want to boot from the local disk17:10
mahnot for the machines17:10
lucasagomesand not from the network17:10
lucasagomesmah, yes exactly17:11
lucasagomesmah, so use that ironic command and also remove the same capability from the nova flavor17:11
mahI think it is easier to change the node boot mode :)17:11
mahand let it from HD17:11
lucasagomesmah, yeah that's option #1 :-)17:12
mahyes will give it a try :)17:12
lucasagomesbut it's less automated since you will have to change it every time you deploy17:12
lucasagomesand move it back boot from network for deploying17:12
lucasagomesthat said, fake_ drivers are just testing drivers they are not meant for production17:12
lucasagomesbecause of ^ this things17:12
mahyes I know17:12
mahbut I will not deploy million of times :D .. it is just one time every some monthes :)17:13
dprinceSlower: the bad news is I can't +2 anything in the heat-templates repo17:13
lucasagomesmah, cool17:13
dprinceSlower: and it does seem a bit slower to land things there sometimes. It happens17:14
lucasagomesmah, cool, ok so go ahead and change the boot mode and see if it works17:14
dprinceSlower: so... press on17:14
*** pcaruana has quit IRC17:14
*** milan has quit IRC17:17
mahI think it works :) ...it opens  for the login and continued configuration of the interfaces there :)17:17
mahthanks a lot Lucas17:17
*** liverpooler has quit IRC17:20
*** akrivoka has quit IRC17:23
Slowerdprince: ok17:23
*** liverpooler has joined #tripleo17:24
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: Make puppet-stack-config.pp Puppet 4 compliant  https://review.openstack.org/37347717:24
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120917:25
*** bana_k has quit IRC17:27
*** liverpooler has quit IRC17:27
lucasagomesmah, cool no worries17:27
lucasagomesshardy, ^17:27
*** bkopilov has quit IRC17:28
shardythanks for the help lucasagomes!17:28
*** bkopilov has joined #tripleo17:30
*** yamahata has joined #tripleo17:31
dsneddonlucasagomes, Are you there? I'm looking for the blueprints which cover multi-tenant bare metal. I looked on blueprints.launchpad.net with no results, not sure where to look next.17:32
lucasagomesdsneddon, hi there. Right, yeah Ironic does not use blueprints anymore17:33
lucasagomeswe use only RFE's (wishlist bugs) and specs17:33
lucasagomesdsneddon, lemme dig the links for ya17:33
dsneddonlucasagomes, Thanks!17:33
lucasagomesdsneddon, https://trello.com/c/HWVHhxOj/1-multi-tenant-networking-network-isolation17:34
dsneddonlucasagomes, Perfect, thanks. I've been asked to provide some feedback for Nuage, so they can make sure their solution works with multi-tenant bare metal networking.17:35
lucasagomesdsneddon, gotcha, I replied to a BZ just today17:35
lucasagomesmaybe that's why17:35
lucasagomesdsneddon, see https://bugzilla.redhat.com/show_bug.cgi?id=128616417:35
openstacklucasagomes: Error: Error getting bugzilla.redhat.com bug #1286164: NotPermitted17:36
dsneddonlucasagomes, Ah, thanks for the context.17:36
lucasagomesdsneddon, np, it's a guess but I think that was what triggered it (see comment 13)17:36
openstackgerritMerged openstack/python-tripleoclient: Always use absolute path for tht_root  https://review.openstack.org/37322017:38
*** bana_k has joined #tripleo17:40
EmilienMshardy: can we backport https://review.openstack.org/#/c/373220/ ?17:41
shardyEmilienM: yes, I'll propose it now17:42
*** saneax-_-|AFK is now known as saneax17:43
openstackgerritSteven Hardy proposed openstack/python-tripleoclient: Always use absolute path for tht_root  https://review.openstack.org/37348717:44
*** akshai has quit IRC17:44
*** akshai has joined #tripleo17:44
EmilienMshardy: thx17:44
*** cwolferh has quit IRC17:44
EmilienMshardy: I'll approve it if CI pass well17:44
shardyEmilienM: ack, thanks!17:45
*** akshai has quit IRC17:45
*** rbowen has quit IRC17:45
*** mburns_mtg is now known as mburned_out17:46
*** rbowen has joined #tripleo17:46
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: Make puppet-stack-config.pp Puppet 4 compliant  https://review.openstack.org/37347717:46
*** mburned_out is now known as mburns_mtg17:47
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120917:47
*** tosky has quit IRC17:47
openstackgerritJohn Eckersberg proposed openstack/tripleo-puppet-elements: Add puppet-qdr module  https://review.openstack.org/37348817:48
openstackgerritJohn Eckersberg proposed openstack/tripleo-puppet-elements: Add qpid-dispatch-router to overcloud-controller element  https://review.openstack.org/37348917:48
*** trown|lunch is now known as trown17:49
thrashEmilienM: finally got scenario 2 to pass on https://review.openstack.org/#/c/331682/32 !17:50
openstackgerritMerged openstack/tripleo-heat-templates: Add nova-metadata template  https://review.openstack.org/37314117:50
thrashEmilienM: it's not done with jobs yet, but according to zuul it's passed17:50
trowndoes this trace look familiar to anyone? http://paste.openstack.org/show/582292/17:51
trowntrying to make a tripleo-ci/deploy.sh deploy work on a quickstart virt setup17:51
EmilienMthrash: oh nice :D17:52
EmilienMthrash: nice work17:52
EmilienMtrown: ouch, sounds like some problem in heat.conf with authtoken params17:53
trownEmilienM: hmm... I wonder actually if it is something SSL related, I got tripleo.sh to work from the same setup, but now trying to go one step further and just plug deploy.sh directly17:55
trownso a developer could run EXACTLY what runs in CI, rather than needing to find all the relevant things to set that we do in deploy.sh17:56
*** sai|mtg is now known as sai17:57
openstackgerritMerged openstack/python-tripleoclient: Updated from global requirements  https://review.openstack.org/37331517:57
openstackgerritMerged openstack-infra/tripleo-ci: Don't send metrics to graphite in failed jobs  https://review.openstack.org/36511417:57
*** akshai has joined #tripleo17:57
EmilienMtrown: nice17:57
EmilienMtrown: can I see all heat logs and config?17:58
trownEmilienM: heat.conf http://chunk.io/f/afc1ca0a439e4660b20e3c28c7fa6cfd17:59
trownheat-engine.log http://chunk.io/f/b57a8b26572648d6ba698ee0ee0fe7b618:00
EmilienM[keystone_authtoken] looks good18:00
EmilienMoh and keystone logs18:00
trownkeystone logs http://chunk.io/f/d5825c6d5a124e74a9d29f95c6544abc18:01
EmilienMand grep 22ba3ecc-21f5-4005-9af2-cbb7c2f68d9d18:01
EmilienMI don't see the request in keystone logs18:02
EmilienMa sec18:02
openstackgerritAlex Schultz proposed openstack/instack-undercloud: Use correct password for keystone bootstrap  https://review.openstack.org/37264018:02
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates: Use correct password for keystone bootstrap  https://review.openstack.org/37263518:02
EmilienMtrown: keystone is running?18:03
EmilienMI guess yes18:03
EmilienMI see lot of valid requests18:03
EmilienMshould not it be https?18:03
EmilienMoh but we're running ssl termination right18:03
EmilienMtrown: it's undercloud right?18:04
*** mburns_mtg is now known as mburned_out18:04
trownEmilienM: ya... I think it must be SSL related, as I am starting with nonha config first and that is the main thing that is different in deploy.sh18:05
trownEmilienM: I can keep digging on it, dont want to waste your time on this18:05
*** shardy has quit IRC18:05
EmilienMi thought we gated on ssl in some jobs18:06
trownwe do, the nonha job has SSL enabled18:06
EmilienMfor undercloud also?18:06
EmilienMiirc yes18:06
*** mburned_out is now known as mburns_mtg18:06
trownEmilienM: ya18:07
dsneddonEmilienM, Just to confirm, if SSL is enabled on the undercloud, then the HTTPS listener for the Admin API will be "", right?18:09
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: Make puppet-stack-config.pp Puppet 4 compliant  https://review.openstack.org/37347718:09
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120918:10
EmilienMdsneddon: it should be yes18:10
*** athomas has quit IRC18:14
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120918:14
*** Ryjedo has joined #tripleo18:21
mwhahahahow do you use the j2.yaml templates18:27
mwhahahalike what's the correct way to invoke them18:28
openstackgerritRyan Brady proposed openstack/tripleo-common: Separate Template Processing From Create/Update Plan  https://review.openstack.org/37086818:29
*** akuznetsov has joined #tripleo18:31
openstackgerritRyan Brady proposed openstack/tripleo-common: Separate Template Processing From Create/Update Plan  https://review.openstack.org/37086818:32
*** cwolferh has joined #tripleo18:35
openstackgerritMerged openstack/python-tripleoclient: Remove one last reference to openstackclient.tests  https://review.openstack.org/37343518:38
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Ceilometer Wsgi Mitaka->Newton upgrades  https://review.openstack.org/36000418:40
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Ceilometer Wsgi Mitaka->Newton upgrades  https://review.openstack.org/36000418:44
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: Composable Mistral services  https://review.openstack.org/32343618:45
thrashEmilienM: so now that it's passed on the zaqar, I should take off my -1 workflow, yah?18:48
thrashEmilienM: on the scenario002 patch18:48
trownEmilienM: btw figured it out... was SSL, need https://github.com/openstack-infra/tripleo-ci/blob/master/toci_instack.sh#L180-L18218:48
EmilienMthrash: yes but we are in feature freeze now18:48
thrashEmilienM: no worries.18:48
EmilienMthrash: so we want land it until we cut stable/newton18:48
EmilienMtrown: oops18:49
thrashEmilienM: word18:49
trownEmilienM: works fine in CI... just not when trying to use deploy.sh outside of CI18:49
EmilienMtrown: right, that's bad, we need to fix that18:50
*** yamahata has quit IRC18:50
EmilienMbnemec: did we stopped to use extraconfig/all_nodes/swap-partition.yaml in CI?18:51
*** gfidente has quit IRC18:55
EmilienMdprince: do you know how the j2 templates can be used? It sounds like the swap template doesn't work18:57
EmilienMmwhahaha: do you have logs?18:57
openstackgerritMerged openstack/python-tripleoclient: Add `openstack overcloud plan deploy`  https://review.openstack.org/37308918:57
openstackgerritMerged openstack/python-tripleoclient: Pass the timeout to the deploy workflow  https://review.openstack.org/37309018:57
bnemecEmilienM: I'm not sure.  I thought we stopped doing that, but I haven't looked recently.18:57
dprinceEmilienM: is there a bug?18:58
mwhahahaEmilienM: http://paste.openstack.org/show/582306/18:58
mwhahahai don't know if it's a bug because i don't know how to use them18:58
* dprince checks18:58
mwhahahait doesn't seem to be the same way as previous18:58
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Remove test-environments/swap-partition.yaml  https://review.openstack.org/37350018:58
mwhahahait's one of those things where i don't know if it's me or the system :D18:59
bnemecEmilienM: I don't see anything about swap in http://logs.openstack.org/81/331681/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/b970470/console.html#_2016-09-20_16_13_27_91479618:59
*** akshai has quit IRC18:59
*** panda|dinner is now known as panda18:59
EmilienMthe log is really hard to read honestly18:59
mwhahahawhat you don't like reading raw json dumps?19:00
*** gfidente has joined #tripleo19:00
dprincewe added Swap partition in 94f580370d1104fb0f0d076cd3c3fd572cd961e819:01
mwhahahaso swap-partition used to work19:02
*** akshai has joined #tripleo19:02
mwhahahathis is related to shardy's AllConfigs backwards compatibility break19:02
mwhahahathat change switched it to a j2.yaml from just a .yaml19:02
dprinceand removed it w/ the j2 templates19:02
dprincemwhahaha: yes19:02
mwhahahathe change mentions docs to be written19:02
mwhahahai assume they have not been written19:02
mwhahahawhich is why i asked, is there something i missing or is it broken19:03
dprincemwhahaha: I think it now lives at extraconfig/all_nodes/swap-partition.yaml19:03
*** jpeeler has quit IRC19:03
mwhahahalike if i just call that file the j2 automagically gets converted into that?19:03
mwhahahalike i don't know how this stuff works19:03
dprincemwhahaha: it is now an extraconfig template19:04
mwhahahai have no idea what that means19:04
EmilienMI agree this is not easy to understand when you haven't worked on this feature19:04
*** lblanchard has quit IRC19:04
dprincemwhahaha: http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/extra_config.html19:05
mwhahahayea i followed that19:05
dprincemwhahaha: before we had composable services this was an interface in heat we used to customized things19:05
mwhahahaso what's the way to do it with composable services?19:05
mwhahahalike fundamentally i understand now it's a j2 template that needs to have the role information populated19:06
mwhahahabut i have no idea how to do that now19:06
*** kjw3 has quit IRC19:06
dprincemwhahaha: the j2 template gets rendered into a swift container as a normal heat template19:06
mwhahahaby what19:06
dprincemwhahaha: so, swift list overcloud19:06
mwhahahacause that doesn't seem to be working19:07
dprincemwhahaha: our workflow tooling does this19:07
mwhahahahow do j2 templates get rendered19:07
dprincemwhahaha: we now use Mistral to generate the plan, and drive the high level deployment19:07
mwhahahabut mistral is puking on the j2 formatting19:07
dprincemwhahaha: all of this enables the CLI and API to follow the same deployment paths19:07
mwhahahaso is that a bug or am i providing the wrong info19:07
dprincemwhahaha: it is working for me, so it could be environmental19:07
dprincemwhahaha: like perhaps something in your environment is old or outdated19:08
dprincemwhahaha: the workflow stuff just landed in the last week or so19:08
dprincemwhahaha: so it is quite new19:08
mwhahahai installed yesterday19:08
mwhahahausing current19:08
dprincemwhahaha: I think we fixed some crucial bugs this morning (if you use current)19:09
*** ebarrera has quit IRC19:09
dprincemwhahaha: you might re-spin it or review the recently closed LP's19:09
mwhahahak i can retry19:09
dprincemwhahaha: sorry, I'm not being super helpful19:09
mwhahahaso to confirm, lines 1-4 of http://paste.openstack.org/show/582306/ are correft right?19:09
mwhahaha1-5 actually19:09
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: Make puppet-stack-config.pp Puppet 4 compliant  https://review.openstack.org/37347719:09
*** lblanchard has joined #tripleo19:10
dprincemwhahaha: I think that should work19:10
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120919:10
mwhahahak i'll respin and see19:10
mwhahahathat's what i wanted to confirm is that it should still work if i just s/.yaml/.j2.yaml19:11
EmilienMmwhahaha: it sounds like the old template not using j2 format won't work anymore19:11
EmilienMor at least the swap one19:11
mwhahahai'm trying to use the new way19:12
mwhahahawith the j2 file and mistral isn't having any of that19:12
EmilienMmwhahaha: I agree that's a poor user experience19:12
dprincemwhahaha: it still needs to say .yaml19:15
dprincemwhahaha: j2.yaml is the new one. correct19:15
mwhahahawhat still needs to say .yaml19:15
*** r-mibu has quit IRC19:17
*** r-mibu has joined #tripleo19:17
openstackgerritMerged openstack/python-tripleoclient: Use a released version of tripleo-common  https://review.openstack.org/37317519:17
openstackgerritMerged openstack/python-tripleoclient: Print the execution ID when starting Mistral workflows  https://review.openstack.org/37316219:17
mwhahahaso looking in the mistral logs it says it rendered extraconfig/all_nodes/swap-partition.j2.yaml19:18
mwhahahadprince: so i guess the question is how do i reference what it rendered?19:19
dprincemwhahaha: I think it is just this extraconfig/all_nodes/swap-partition.yaml19:20
mwhahahalet me try that19:20
dprincemwhahaha: that is what gets rendered into the swift container19:20
dprincemwhahaha: swift list overcloud19:20
mwhahahaso that's the difference is that it's no longer a local reference it must be the rendered reference19:20
mwhahahathat really should have been documented first19:20
mwhahahano that gets assumed as a local path19:21
mwhahahaCould not fetch contents for file:///home/centos/extraconfig/all_nodes/swap-partition.yaml19:21
dprincemwhahaha: use the full path to the file19:22
rcritI'm doing cloud-init on the overcloud install to do some pre-configuration and since CentOS ships cloud-init 0.7.5 it fails to run because the ConfigDriveNet doesn't work in that version19:22
dprincemwhahaha: with just .yaml as the suffix19:22
rcritI have a bug opened but it has gotten no traction yet, https://bugs.centos.org/view.php?id=1146219:22
mwhahahadprince: so swift lists extraconfig/all_nodes/swap-partition.j2.yaml19:23
rcritdon't know if anyone else has run into this19:23
mwhahahareferencing the full path with just 'swap-partition.yaml' doesn't work either19:23
dprincemwhahaha: my swift also lists extraconfig/all_nodes/swap-partition.yaml19:23
mwhahahathat's the old file19:24
openstackgerritMerged openstack/tripleo-quickstart: Remove settings from release configs  https://review.openstack.org/37111419:24
dprincemwhahaha: I think you've got enough to file a bug on this now19:24
dprincemwhahaha: I don't have an enviornment setup or I'd try it for you19:24
mwhahahak i'll write something up19:24
EmilienMI'm tryin gto reproduce it too19:25
openstackgerritMerged openstack/python-tripleoclient: Remove the environments from Mistral when removing from Swift  https://review.openstack.org/37316719:26
openstackgerritMerged openstack/python-tripleoclient: Throw an error/exception when interactive overcloud update fails  https://review.openstack.org/37074219:26
*** akshai has quit IRC19:28
*** jpeeler has joined #tripleo19:28
*** jpeeler has joined #tripleo19:28
*** electrofelix has quit IRC19:29
*** paramite has quit IRC19:32
*** akshai has joined #tripleo19:34
mwhahahaEmilienM, dprince: https://bugs.launchpad.net/tripleo/+bug/162578319:36
openstackLaunchpad bug 1625783 in tripleo "unable to leverage swap-partition extraconfig" [Undecided,New]19:36
EmilienMmwhahaha: setting to critical19:36
*** dbecker has quit IRC19:37
*** hjensas has quit IRC19:37
mwhahahatrying with todays packages and we'll see19:37
gfidentemwhahaha, dprince I think that's fixed by19:37
dprincegfidente: ack, looks like it19:38
*** dbecker has joined #tripleo19:44
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: certmonger: improve orchestration for puppet4  https://review.openstack.org/37351319:54
*** andrey-mp has joined #tripleo19:55
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120919:55
openstackgerritGiulio Fidente proposed openstack/tripleo-puppet-elements: Install ceph-selinux in overcloud-{controller,cephstorage} images  https://review.openstack.org/37351419:55
andrey-mpHi! I've trying to deploy undercloud from master. when I run 'openstack undercloud install' it asks me about some password? What password I should enter???19:55
*** akuznetsov has quit IRC20:00
andrey-mpalso, I've got an error right after undercloud installation - "Error: Must pass keystone_password to Class[Glance::Api]" what it means??? how to deploy undercloud from master???20:04
*** jcoufal_ has quit IRC20:09
EmilienMthe press enter bug is pretty interesting :)20:09
EmilienMandrey-mp: press enter20:09
andrey-mpEmilienM: yeah, I did it. but I faced with nex bug )20:10
trownsecond one I have not seen20:11
andrey-mpi filed it - https://bugs.launchpad.net/tripleo/+bug/162579620:11
openstackLaunchpad bug 1625796 in tripleo "Can't install undercloud on centos7 from master" [Undecided,New]20:11
mwhahahasounds like old puppet modules20:12
sshnaidm|afkandrey-mp, there is bug about it20:12
mwhahahacause keystone_password is not required anymore20:12
andrey-mpsshnaidm|afk - good news )20:13
mwhahahathe requirement was moved to glance::keystone::authtoken20:13
andrey-mpmwhahaha: I do not control puppets, right?20:14
mwhahahadepends on how you installed it20:14
andrey-mpmwhahaha: i must enable latest repos and run undercloud install?20:14
andrey-mpmwhahaha: from this doc - http://docs.openstack.org/developer/tripleo-docs/installation/installation.html#installing-the-undercloud20:14
*** lucasagomes is now known as lucas-afk20:15
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Ensure that the default plan was created successfully  https://review.openstack.org/37344620:15
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Verify that the Deployment Plan creation was successful  https://review.openstack.org/36924720:15
mwhahahain a condensed form -^20:15
*** jprovazn has quit IRC20:16
andrey-mpmwhahaha: i have similar script but without enabling puppet source and ceph20:16
mwhahahasure, adjust as necessary, but make sure you're repos are correct and you installed the priorities plugin20:17
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: certmonger: improve orchestration for puppet4  https://review.openstack.org/37351320:17
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120920:18
*** Goneri has quit IRC20:18
*** mah has quit IRC20:19
andrey-mpmwhahaha: thanks, will try this20:19
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Ensure that the default plan was created successfully  https://review.openstack.org/37344620:20
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Verify that the Deployment Plan creation was successful  https://review.openstack.org/36924720:20
EmilienMdprince: I reproduced mwhahaha's bug20:21
sshnaidm|afkandrey-mp, https://bugs.launchpad.net/tripleo/+bug/162152420:22
openstackLaunchpad bug 1619274 in tripleo "duplicate for #1621524 openstack complete triggers password prompt" [High,In progress] - Assigned to James Slagle (james-slagle)20:22
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fix the default plan creation  https://review.openstack.org/37134720:22
andrey-mpalso I have another question - I've tried to install Mitaka version. undercloud installed ok. then I tried to install overcloud with three controllers - and all controllers failed. all had error from *-manage modules. for example cinder-manage couldn't create some table with error 'column already exists'. looks like that each controller tried to create DB for all services. when I'm wrong?20:22
dprinceEmilienM: okay, using the latest python-tripleoclient?20:22
EmilienMlet me check20:22
dprinceEmilienM: with shardy's "Always use absolute path for tht_root" fix...20:23
EmilienMlet me see20:23
EmilienMlet's see in code20:23
*** Ryjedo has quit IRC20:24
trownI dont see 7833303 as a short sha in tripleoclient repo20:24
EmilienMdprince: without20:24
EmilienMlet me checkout20:24
trownoh because it is only on stable/newton20:25
EmilienMI'm using delorean-current20:25
trownso that tripleoclient is from stable/newton branch20:25
EmilienMhis patch was backported or not yet20:25
*** Ryjedo has joined #tripleo20:26
trownjust saying the version you gave points to it being built from that commit ^20:26
EmilienMnot yet20:26
trownwhich is a merge commit on stable/newton20:26
EmilienMso I approved the backport20:26
EmilienMand now testing it again on my box20:26
EmilienMtrown: using quickstart this time fyi20:26
EmilienMworks pretty well20:26
trownEmilienM: how did you run it?20:27
EmilienMtrown: bash quickstart.sh --release master-tripleo-ci -T all $VIRTHOST20:27
EmilienMI'm a very simple guy20:27
trownnice, so using the master-tripleo-ci mode20:28
trownI think that is pretty key to reproducing tripleo-ci20:28
EmilienMnow having another trace http://paste.openstack.org/show/582314/20:28
*** myoung is now known as myoung|bbl20:28
EmilienMswift list overcloud doesn't show up the overcloud.yaml20:29
EmilienMdprince: ^20:29
EmilienMmaybe I need to destroy the plan before20:29
trownwe probably need a `openstack overcloud delete` command now... there are more steps than just `openstack stack delete overcloud`20:30
EmilienMno, I had to run:20:30
EmilienMopenstack overcloud plan delete overcloud20:30
EmilienMmhh I have the same error as initially with the wrong generation20:31
EmilienMshould I remove something in the swift containers?20:34
*** andrey-mp has left #tripleo20:36
slagletry a swift delete overcloud20:41
slaglesaw d0ugal mention yesterday that it might be required20:41
*** jayg is now known as jayg|g0n320:43
bnemecMerge please? https://review.openstack.org/#/c/37111520:45
dprincebnemec: +A20:48
slagleyou actually want to test image building with tripleoclient? :)20:48
bnemecdprince: Thanks20:49
bnemecTake it easy, enter key.20:49
bnemecslagle: Crazy, right? :-)20:49
*** dprince has quit IRC20:49
*** egafford has quit IRC20:50
*** oneswig has joined #tripleo20:56
*** yolanda has joined #tripleo20:58
*** gfidente has quit IRC21:05
*** ccamacho has quit IRC21:06
*** dsariel has joined #tripleo21:06
EmilienMslagle: thanks21:06
EmilienMlet's try again21:07
*** lblanchard has quit IRC21:08
EmilienMI see HA jobs failing more often than the last 3 days21:08
EmilienMlot of timeouts on overcloud with heat engine consumming memory21:08
EmilienMhave you noticed the same?21:08
*** mburns_mtg is now known as mburned_out21:14
*** jeckersb is now known as jeckersb_gone21:20
*** yamahata has joined #tripleo21:28
EmilienMslagle: yes give me asec21:32
EmilienMslagle: http://logs.openstack.org/34/372734/5/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/93e9821/logs/postci.txt.gz#_2016-09-20_18_51_22_00021:32
EmilienMService[heat-engine]: Could not evaluate: Cannot allocate memory - fork21:32
stevebakerEmilienM, slagle: is there a bug for swiftclent logging its 404s at the start of overcloud deploy?21:33
openstackgerritMerged openstack/python-tripleoclient: Always use absolute path for tht_root  https://review.openstack.org/37348721:34
EmilienMstevebaker: not afik21:34
EmilienMbut I probably missed it21:34
EmilienMmaybe in https://launchpad.net/tripleo/+milestone/newton-rc221:34
EmilienMstevebaker: good morning!21:34
stevebakerEmilienM: good afternoon!21:34
*** panda is now known as panda|Zz21:35
*** cwolferh has quit IRC21:38
*** akshai has quit IRC21:39
slaglei don't see one21:39
stevebakerI'll raise one, I fixed something similar in heat21:39
bnemecstevebaker: That was definitely a recent problem.  I don't know if it's fixed in newer versions of the client.21:40
bnemecThere was also an issue doing tripleoclient builds until lately, so even if we merged a fix it may not have gotten into the dlrn repos.21:40
stevebakerbnemec: this is what I did https://review.openstack.org/#/c/347111/21:41
*** dsariel has quit IRC21:44
*** dtrainor has quit IRC21:51
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: Convert $step to an integer  https://review.openstack.org/37354121:54
*** fultonj has quit IRC21:54
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120921:55
openstackgerritMerged openstack-infra/tripleo-ci: Don't use cached images in tripleoclient changes  https://review.openstack.org/37111522:00
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: Convert $step to an integer  https://review.openstack.org/37354122:03
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120922:03
*** oneswig has quit IRC22:07
*** bfournie has quit IRC22:09
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates: Set Neutron's metadata_ip to the nova metadata VIP  https://review.openstack.org/37354422:13
*** cwolferh has joined #tripleo22:17
*** cdearborn has quit IRC22:21
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: add tempest back into the quickstart-extras workflow  https://review.openstack.org/37274822:28
openstackgerritMerged openstack/tripleo-heat-templates: RabbitMQ threads should be configured dynamically  https://review.openstack.org/37302722:40
openstackgerritMerged openstack/puppet-tripleo: Fixup manila-cephfs native backend defaults  https://review.openstack.org/36676022:42
*** Guest38034 has quit IRC22:51
*** Guest38034 has joined #tripleo23:01
openstackgerritMerged openstack/python-tripleoclient: Update the Mistral action names  https://review.openstack.org/36651923:03
openstackgerritMerged openstack-infra/tripleo-ci: Remove support for legacy rh1  https://review.openstack.org/34791823:03
openstackgerritMerged openstack/diskimage-builder: Document source glean installs in simple-init  https://review.openstack.org/36456823:05
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart: DONT MERGE: POC: new devmode  https://review.openstack.org/36200023:09
*** limao has quit IRC23:09
*** pmannidi has joined #tripleo23:09
*** TSCHAK_ has joined #tripleo23:09
*** TSCHAK has quit IRC23:12
*** TSCHAK has joined #tripleo23:20
*** TSCHAK_ has quit IRC23:22
*** Goneri has joined #tripleo23:33
*** rdopiera has quit IRC23:35
openstackgerritMerged openstack/diskimage-builder: don't configure 'lo' for dhcp  https://review.openstack.org/36752723:37
*** rdopiera has joined #tripleo23:37
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120923:41
*** mburned_out is now known as mburns_mtg23:41
*** yamahata has quit IRC23:42
*** noslzzp has quit IRC23:43
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types  https://review.openstack.org/36367423:46
*** Goneri has quit IRC23:52
*** rajinir has quit IRC23:55
*** dmacpher has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!