Thursday, 2016-01-21

openstackgerritSteve Baker proposed openstack/tripleo-common: Implement ViewManager list_software_deployments
openstackgerritSteve Baker proposed openstack/tripleo-common: show_software_deployment
openstackgerritSteve Baker proposed openstack/tripleo-common: list_stack_failures
*** cwolferh has joined #tripleo00:06
*** absubram has quit IRC00:07
*** dsneddon has quit IRC00:11
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Remove zero length files
*** dsneddon has joined #tripleo00:15
*** dsneddon has quit IRC00:15
*** dsneddon has joined #tripleo00:16
*** weshay has quit IRC00:19
*** dmacpher has quit IRC00:21
*** rasca has quit IRC00:22
*** rasca has joined #tripleo00:23
*** apetrich_ has joined #tripleo00:42
*** apetrich has quit IRC00:45
*** morazi has quit IRC00:48
*** cwolferh has quit IRC01:00
*** cwolferh has joined #tripleo01:00
*** dmacpher has joined #tripleo01:05
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Fix rpmlint script-without-shebang errors
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Remove shebangs from non-execute files
*** dmacpher has quit IRC01:29
*** penick has quit IRC01:33
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Remove shebangs from non-execute files
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Remove shebangs from non-execute files
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Fix rpmlint script-without-shebang errors
*** shivrao has quit IRC01:40
*** sthillma has quit IRC01:52
*** tiswanso has joined #tripleo02:01
*** tiswanso has quit IRC02:03
*** tiswanso has joined #tripleo02:04
*** rhallisey has quit IRC02:28
*** cwolferh has quit IRC02:32
*** julim has quit IRC02:32
*** yamahata has quit IRC03:00
*** masco has joined #tripleo04:10
*** regebro is now known as Guest9511004:15
*** rebrego has joined #tripleo04:15
*** Guest95110 has quit IRC04:15
*** rebrego is now known as regebro04:15
openstackgerritMatthew Thode proposed openstack/diskimage-builder: add support for package-installs to gentoo
*** trozet has joined #tripleo04:21
*** tzumainn has quit IRC04:31
*** rasca has quit IRC04:35
*** rasca has joined #tripleo04:36
*** rasca has quit IRC04:41
*** rasca has joined #tripleo04:41
*** coolsvap|away is now known as coolsvap04:49
EmilienMcould someone look please?04:52
EmilienMrun keystone in a wsgi process on the undercloud -> passing CI and already +204:52
*** stendulker has joined #tripleo04:56
*** david-lyle has quit IRC05:02
*** david-lyle has joined #tripleo05:03
*** Marga_ has quit IRC05:07
*** cwolferh has joined #tripleo05:08
*** Marga_ has joined #tripleo05:08
*** rasca has quit IRC05:11
*** rasca has joined #tripleo05:13
*** shivrao has joined #tripleo05:18
*** rlandy has quit IRC05:28
*** tiswanso has quit IRC05:31
*** yuanying has joined #tripleo05:32
*** yuanying_ has quit IRC05:36
*** xinwu has quit IRC05:42
*** jaosorior has joined #tripleo05:47
*** xinwu has joined #tripleo05:52
*** xinwu has quit IRC06:08
*** rasca has quit IRC06:10
*** rasca has joined #tripleo06:11
*** ayoung has quit IRC06:16
*** anande has joined #tripleo06:18
*** spredzy has quit IRC06:18
*** spredzy has joined #tripleo06:18
*** liverpooler has quit IRC06:19
*** anande has quit IRC06:20
*** derekh has joined #tripleo06:31
openstackgerritMerged openstack/tripleo-common: Adds a simple overcloud tenant vm ping test to
derekhAnybody want to add a second +2 to the ping test?
* derekh has done too much to the patch to merge it himself06:33
*** Marga_ has quit IRC06:37
*** xinwu has joined #tripleo06:37
*** jprovazn has joined #tripleo06:38
*** yamahata has joined #tripleo06:41
*** dmacpher has joined #tripleo06:47
*** rwsu has quit IRC06:52
*** trozet has quit IRC07:03
openstackgerritDerek Higgins proposed openstack/tripleo-common: WIP: split overcloud setup out of pingtest
derekhafazekas: I've propsed a patch to split out the setup stuff from that pingtest, still a WIP but we can discuss the details there
derekhafazekas: on the train at the moment, will work on it more today07:10
*** shivrao has quit IRC07:11
afazekasderekh, for foobar reason I can't get a working install since last night. What are the common issues with reusing the `root` vm ? Do you know about some `unbreaking`  change from the prev days ?07:13
afazekasI should have the `not use tmpfs` for images07:13
*** ukalifon has joined #tripleo07:13
afazekasBTW, can we create swap on the instack vm ?07:13
derekhafazekas: the vm I gave you? you can do what ever you want with it, like this
derekhafazekas: maybe you are hitting yesterdays bug, one sec07:14
derekhafazekas: have you got this
afazekas/tmp/swap ???07:15
afazekasisn't tmp already tmpfs ?07:15
afazekasyou need /swap to be clear07:16
afazekasderekh, also consider swapping the bs with the count07:16
derekhnope, at least I didn't think it was, I'll double check07:16
afazekasderekh: even if it is not tmpfs at the moment, not recommended07:17
derekhafazekas: will do07:17
*** alop has quit IRC07:19
afazekasyee, it is really not a tmpfs :)07:19
*** rcernin has joined #tripleo07:20
derekhafazekas: yup  , although maybe I should move it incase it ever becomes one again07:20
mariosderekh: thanks
mariosderekh: been trying to get back to that but i keep getting pulled under :/07:21
derekhmarios: no problem, just gotta get the one that wires it up merged now07:23
afazekasderekh, on the `root` vm, might be better to just to change the flavor, and extending it with some swap07:23
derekhafazekas: ahh yes, your talking about the undercloud, wouldn't that be a change to the undercloud image we build? That instance isn't started by nova, the undercloud image is copies to /var/lib/libvirt/... and we call a virsh start07:30
derekhafazekas: yes it would be better, the mkswap patch is just to find out if it help things before spending more time on it.07:30
afazekasderekh: root vm = what is the jenkins slave, for the undercloud and to the other it is image build part can help, also if you have cloud init (we have) if the domain created with preformed swap disk it is supposed to jump on it07:33
afazekasAs I remember there is bz request for using swap on the overcloud vms (or just changing the memory over commit ratio to <=1.0 on n-cpu nodes)07:34
*** jcoufal has joined #tripleo07:35
derekhafazekas: "What are the common issues with reusing the `root` vm ?"07:39
derekhafazekas: so reusing that VM in the passed, the problems have been fairly obvious, things like "mkdir " failing because the directory already exists the second time round07:40
afazekasBTW: some magic from devstack, we might use it or something similar
afazekasone older variant also indicated the highest resource user process07:52
*** jaosorior has quit IRC07:53
afazekasit is based on dstat (similar to iostat and vmstat), and easir to decode the output than the sa107:53
*** liverpooler has joined #tripleo07:54
*** liverpooler has quit IRC07:55
*** jtomasek has joined #tripleo07:55
*** liverpooler has joined #tripleo07:55
*** dshulyak_ has joined #tripleo07:56
*** ayoung has joined #tripleo08:01
*** bvandenh has joined #tripleo08:05
*** jaosorior has joined #tripleo08:06
*** fgimenez has joined #tripleo08:06
*** fgimenez has quit IRC08:06
*** fgimenez has joined #tripleo08:06
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Add some swap to the undercloud
*** tosky has joined #tripleo08:09
*** bvandenh has quit IRC08:13
mariosderekh: was that "Error: Could not prefetch mongodb_replset provider 'mongo'" resolved in ci? (thinking if i should recheck reviews yet)08:19
derekhmarios: yup we pinned the mondo puppet module, EmilienM is working on the fix08:19
mariosderekh: thx08:20
jaosoriormarios: Well, it seems to me that it's working08:20
mariosjaosorior: yeah some reviews that ran yesterday08:21
marios(are failed and am going to recheck)08:21
derekhmarios: wait08:22
derekhmarios: well go ahead08:22
mariosderekh: k, i only rechecked one08:22
derekhmarios: just we said at the meeting that we wouldn't merge anything else without the ping test08:22
mariosderekh: ah ack sorry missed that thx08:23
derekhso if you want to not have to recheck again you should wait untill this is merged08:23
mariosderekh: right thanks08:24
*** bvandenh has joined #tripleo08:24
derekhmarios: I hadn't +2'd bcause I submitted the last patchset, but I guess since both of us are core, one of us can08:25
*** aufi has joined #tripleo08:25
mariosderekh: yeah i mean, i didn't do it because i am listed as owner...08:27
*** mkovacik has joined #tripleo08:27
*** dmacpher has quit IRC08:27
derekhmarios: ok I'm going to merge it, it was just waiting on ci up to no08:27
mariosderekh: but i guess it changed enough for it to be ok08:27
derekhrace ya08:28
mariosderekh: ok am going to merge sorry was haing a closer look at the diff08:29
openstackgerritMerged openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the heredoc
derekhmarios: cool08:31
*** gfidente has joined #tripleo08:32
*** gfidente has quit IRC08:32
*** gfidente has joined #tripleo08:32
mariosderekh: thanks to you and bnemec for sorting it out08:32
derekhI wouldn't be surprised if we now hit more failed CI tests, but at least they are for reasons we need to fix....08:32
derekhmarios: no prob08:32
*** paramite has joined #tripleo08:35
*** shardy has joined #tripleo08:39
openstackgerritafazekas proposed openstack-infra/tripleo-ci: [WIP] trying to add one block node
*** mcornea has joined #tripleo08:43
mariosjaosorior: can you check
jaosoriormarios: Will do, I'm currently checking out this one though
*** paramite is now known as paramite|afk08:48
mariosjaosorior: sure just think it is a rebase nit (removes the HTTP_X_FORWARDED_PROTO)08:49
jaosoriormarios: Taking a quick look it seems to me like yeah, that config shouldn't have been deleted08:50
*** mkovacik has quit IRC08:50
mariosjaosorior: k, just pointing it out cos i know you added that stuff08:50
jaosoriormarios: Thanks dude08:51
mariosnp :)08:52
*** derekh has quit IRC08:52
jaosoriormarios: this commit seems alright, but just a quick question. Why add the nova_ipv6 to the structuredconfig instead of using the nova::use_ipv6 that's already there?08:53
jaosorioryou could probably still get that using hiera, could you?08:53
mariosjaosorior: that is a good question. i saw it was actually setting something in the nova puppet class and wasn't sure it'd be available like 'normal' hiera data. i should check though08:54
mariosi think i still have a v6 beaker box... checking now08:55
mariosjaosorior: yeah :/08:57
*** bvandenh has quit IRC08:57
*** dtantsur|afk is now known as dtantsur08:58
mariosjaosorior: well it *is* in the normal hieradata... i wonder if we can pick it up like normal though... i mean08:58
marios[root@overcloud-controller-0 ~]# grep 'nova.*ipv6' /etc/puppet/hieradata/controller.yaml08:58
mariosnova::use_ipv6: true08:58
mariosnova_ipv6: true08:58
mariosjaosorior: yeah, should be fine, example08:59
jaosoriormarios: yeah, would be nicer if it's possible :D08:59
jaosorioraaah, there you go08:59
mariosjaosorior: yeah it was one of the many things that were fixed over the weekend, hurredly08:59
mariosjaosorior: thanks!08:59
jaosoriormarios: Everything else looks good in that CR09:00
mariosjaosorior: tx, if you add a comment i can push a revision in a sec09:00
mariosjaosorior: i can add a comment, just thought you may want to cos you thought of it09:01
*** nico_auv has joined #tripleo09:02
*** mbound has joined #tripleo09:02
*** bvandenh has joined #tripleo09:10
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Fixup the memcached servers string in nova.conf for v6
*** paramite|afk is now known as paramite09:12
*** devvesa has joined #tripleo09:14
*** jistr has joined #tripleo09:16
afazekasI have tried to run a minimal tempest ^ , 4G memory is simply not enough for an operating controller09:19
shardySimple patch needs a +A, passing CI:
afazekasAre there any chance to reconfigure the domains with 6G memory at least ?09:20
jaosoriorshardy: This one also :D
*** derekh has joined #tripleo09:23
openstackgerritMerged openstack/python-tripleoclient: Apply --min-tmpfs 5 to agent ramdisk build too
* afazekas FYI: something like this should be enough to run tempest:
* afazekas
*** bvandenh has quit IRC09:30
shardyjaosorior: Yeah, need to get that one to pass the HA job tho and/or fix the issues causing that job to fail so often :(09:32
shardyafazekas: Are you planning to run tempest as part of an upstream CI run?09:33
*** bvandenh has joined #tripleo09:34
shardyafazekas: we're hitting various OOM issues, so increaing the defaults may be a good idea, but the other challenge is the time taken to run the tests09:35
shardye.g we're regularly hitting the infra timeouts on HA/ceph jobs already09:35
shardyderekh can probably advise if we have capacity to increase memory defaults any further09:36
jaosoriorshardy: Funky thing is that patch isn't even being used, it's just the definition of a resource in puppet. So I'm pretty puzzled that it was failing, but gonna recheck it.09:36
shardyjaosorior: it's probably failing for an unrelated reason, if you check the undercloud logs you'll probably find something got killed by the OOM killer :(09:37
shardyout of memory killer09:38
jaosoriorI see09:39
shardywe're running with not-really-enough RAM and no swap, everywhere09:39
afazekasshardy: yes, i am planing09:39
shardyafazekas: cool, better testing (and help with upstream CI) will be awesome, but I wonder about the runtime09:40
shardyhow long does tempest take to run?09:40
*** jcoufal has quit IRC09:40
afazekasshardy: it depends on the performance of the cluster09:40
afazekastempest it self is parallel by default09:40
afazekas40m..80m is full run with small clusters09:41
shardyafazekas: Ok, I don't think we'll have capacity for that duration on any of our current tests09:41
afazekasit can be limited to smaller test set , like just the scenarios09:41
shardybecause we're hitting timeouts already09:42
shardyYeah, I think it'll need to be limited to something which can run in less than about 10 minutes, and even then it'll probably only work on the nonha job09:42
*** bvandenh has quit IRC09:42
afazekasI know, I frequently crying about installation speed :)09:42
shardyafazekas: well, it's not just the speed, it's the performance of our CI09:42
shardyanyway, just so long as you're aware before committing too much time to it :)09:43
shardyhopefully we can work out a solution09:43
afazekasshardy: do not think is it significantly faster on bigger machine, one cpu user process, which is cpu bound will not be faster anywhere else09:44
jaosoriorMaybe it's time to start separating undercloud CI from overcloud CI? like, use a pre-built undercloud when running tests related to t-h-t for instance09:44
afazekasshardy: the task description was not involved full runs :)09:44
shardyjaosorior: Yeah, there may be optimisations like that, or not building images for every change09:45
afazekasshardy: I'll try to cut to 4..12 test to the voting jobs, and try to do full run on the periodic ones09:45
shardyafazekas: ack, sounds good, thanks! :)09:46
jaosoriorshardy: Yeah, that would probably help too09:46
shardyjaosorior: the problem is, what happens when a t-h-t change depends-on a puppet-* change09:46
shardythen you have to build the images, or change CI to use DeployArtefacts and patch in the new modules09:46
jaosorioruhm... yep, there would be a lot more logic required in the C I09:47
afazekasshardy: BTW, who know the hw spec of the BM nodes which hosting the overcloud vms?09:49
shardyafazekas: derekh will know09:51
afazekasIs I remember one gearman worker was sized to much more small VM than the jobs actually uses, so reserving less, but bigger nodes seams possible09:52
derekhafazekas: shardy on the run time, we're initially hopeing to get tempest into our periodic jobs, so increasing the run time with tempest shouldn't be an issue09:53
derekhi.e. is wont effect the time devs have to wait on feedback09:53
shardyderekh: ack, Ok thanks09:54
derekhafazekas: yes, your correct, each testenv has a load of nodes we don't use, 10 I think09:54
derekhafazekas:  but we just did that so that we wouldn't need to rebuild the nodes if we wanted to add anything09:55
*** kbyrne has joined #tripleo09:55
derekhafazekas: and beacuase we were using the devtest_testenv script to build it, it create lots of nodes incase developers wanted to use them09:56
derekhafazekas: there is no way we could actually use all 10 nodes in the testenv, we are already overcomited as it is09:56
derekhafazekas: some number09:56
derekhafazekas: each test env host has 64G RAM , and supports 4 testenvs09:57
afazekasderekh, not just 3 env ?09:58
afazekasyes it is over-committed  anyway09:58
*** tosky has quit IRC09:59
derekhafazekas: if a host get 4 HA tests at the same time, that (4G * 5 ) 20G by 4 , 80G commited , we are relying on the fact that some of the jobs on a testenv will be HA and some nonha, also that each job will be at a different s=tage in the test09:59
afazekasderekh: do you have stats about how many env are reserved ?09:59
derekhafazekas: I think it 4 per Host, I'll check now one sec09:59
afazekaslest time I hear 3 :)10:00
afazekasderekh, do they have available DIMM slot?10:00
openstackgerritMerged openstack/instack-undercloud: Remove Py33 support
*** aufi has quit IRC10:01
afazekasusually dmdecode can tell10:01
derekhafazekas: 4 test envs per host on rh1 ,
derekhafazekas: not sure about spare dims checking10:01
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates: Make External Load Balancer templates work with IPv6 [DO NOT MERGE]
*** openstackgerrit has quit IRC10:02
*** openstackgerrit has joined #tripleo10:02
openstackgerritMerged openstack/tripleo-heat-templates: Let Puppet update all packages on non-controllers
afazekasderekh: can you show also a /proc/cpuinfo ?10:03
*** rasca_ has joined #tripleo10:03
*** pblaho has quit IRC10:03
*** electrofelix has joined #tripleo10:03
*** aufi has joined #tripleo10:04
*** yamahata has quit IRC10:04
*** rasca has quit IRC10:04
derekhafazekas: cpuinfo
openstackgerritMerged openstack/instack-undercloud: Remove answers file support
*** paramite has quit IRC10:07
*** aufi has quit IRC10:08
*** jaosorior has quit IRC10:09
*** aufi has joined #tripleo10:09
derekhafazekas: If I'm reading this correctly we have 24 slots, and we're using 8 of them (8 x 8G10:09
*** jaosorior has joined #tripleo10:10
*** akrivoka has joined #tripleo10:10
*** mkovacik has joined #tripleo10:10
*** jaosorior has quit IRC10:10
*** jaosorior has joined #tripleo10:11
*** athomas has quit IRC10:12
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack-infra/tripleo-ci: Remove unused urllib
afazekasderekh: how many cpu core/thread , is the machines expected remain in service after the warranty expired ?10:13
*** dtantsur is now known as dtantsur|brb10:15
afazekasderekh, the CPU type is not exposed to the guest, it is rumored to using cpu pass-trough can make things faster10:17
*** xinwu has quit IRC10:19
*** athomas has joined #tripleo10:19
derekhafazekas: are you talking about exposing it to the jenkins node, or the testenv nodes (probably both)10:19
*** xinwu has joined #tripleo10:20
afazekas 'host-passthrough' the rumors from here10:20
derekhafazekas: to be honest my preference would be to redeploy the whole thing with OVB soon10:20
*** xinwu has quit IRC10:20
afazekasdisadvantage: live migration to inferior host will not work  :)10:20
derekhafazekas: not sure about plans after warranty, the topic has never come up but its a question worth asking10:21
*** paramite has joined #tripleo10:21
derekhafazekas: I wouldn't worry about live migration, for this rack, ci jobs can be restarted, far simpler10:22
afazekasderekh: I am asking for nr host cpu core, does it worth to invest memory into this machines10:22
afazekasObviously the controller nodes will use more and more memory when we get more and more service10:23
*** aufi has quit IRC10:23
afazekasThe only thing what may mitigate it is to reducing the number of service worker processes10:23
*** rasca_ has quit IRC10:24
*** aufi has joined #tripleo10:24
derekhafazekas: each host has 2 X ( 6 cores, 12 threads),
derekhafazekas: I should find out how long we intend the machines to stick around before we looking into more memory10:25
*** rasca has joined #tripleo10:26
afazekasderekh: for running mainly install test is it still ok to extend with memory10:27
derekhafazekas: to extend what with memory? I'm not sure if your talking about the testenv hosts, the jenkins nodes, the undercloud, or one of the overcloud nodes.10:29
afazekasderekh: other memory saving thing can be, using less memory with != controller (or undercloud) vms . for example with tempest we usually using this flavor for vms nova flavor-create m1.nano 42 64 0 110:29
*** StevenK has quit IRC10:29
afazekasderekh: testenv10:29
afazekaswith m1.nano you can spin up a `lot of` vm even on 2G memory compute node10:30
derekhafazekas: yup, sounds like a good flavor to use10:31
openstackgerritMerged openstack/python-tripleoclient: Sahara integration
derekhafazekas: back in a bit, got a meeting10:32
*** paramite has quit IRC10:49
openstackgerritJaume Devesa proposed openstack/python-tripleoclient: image build: add --builder-extra-args
*** StevenK has joined #tripleo10:53
*** tosky has joined #tripleo10:58
*** fgimenez_ has joined #tripleo10:59
*** fgimenez has quit IRC10:59
*** fgimenez_ is now known as fgimenez11:00
*** paramite has joined #tripleo11:01
*** aufi has quit IRC11:01
*** aufi has joined #tripleo11:03
openstackgerritMerged openstack/tripleo-heat-templates: Rename validate tox env to linters
*** regebro has quit IRC11:11
*** mgould has joined #tripleo11:18
*** shadower has joined #tripleo11:19
gfidenteshardy, I saw your comment on
gfidenteI think it'd be indeed nicer to do it that way11:24
*** pblaho has joined #tripleo11:25
shardygfidente: cool - I felt bad as it looks nearly ready to land, but I think it may simplify things somewhat11:25
shardyin general, we've been pretty bad at copying everything everywhere for network stuff, so it'd be good to start getting away from that and parameterizing things where possible11:26
*** regebro has joined #tripleo11:27
hewbroccashardy: code reuse!11:31
*** jcoufal has joined #tripleo11:34
*** olap has joined #tripleo11:36
shardyjistr: Hey did you get to the root-cause of your no valid host errors yesterday?11:36
shardyI've built a fresh environment and getting similar issues11:37
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: SSL/Cinder: enable ssl_header_handler filter
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable SSL middleware for cinder
jaosoriorJust a rebase ^^11:39
jistrshardy: hi, yes i did, it was most probably OOM issues (root cause is probably heat-engine eating memory). The first problem i saw in ironic-conductor logs was OOM error, and from that point on various issues appeared. Most probably OS started killing things. I restarted heat-engine to clear up memory, lowered it to 2 workers. Restarted ironic-conductor to re-register it with ironic-api which got me rid of ironic errors, then i was getting11:41
jistrneutron errors (in nova-compute.log) because ovs agent was down (probably killed too), so to make sure i restarted all neutron and nova services too.11:41
gfidenteshardy, so I want to update the external-lb thing to make it work first11:42
shardyjistr: ack, OK thanks for the info11:42
gfidentebut then I'll try to rewrite things using both the format param AND the str_split11:42
gfidenteupdating those same patches11:42
shardygfidente: Ok, sounds great11:42
shardyif we need to backport that patch to kilo, it's fine to do the str_split in a separate patch, or I can rebase on to it11:43
*** stendulker_ has joined #tripleo11:43
*** stendulker has quit IRC11:47
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Make External Load Balancer templates work with IPv6
*** stendulker_ has quit IRC12:00
*** fgimenez has quit IRC12:05
*** fgimenez has joined #tripleo12:05
*** fgimenez has quit IRC12:05
*** fgimenez has joined #tripleo12:05
*** coolsvap is now known as coolsvap|away12:07
*** dtantsur|brb is now known as dtantsur12:13
gfidentejistr, marios I noticed the tunnelidranges and vniranges parameters are not wired from top into controller and compute templates anymore12:16
gfidentedo you know if that was done on purpose?12:16
mariosgfidente: not something i did afaicr... who does git blame?12:17
jistrhmm i don't recall that12:17
gfidenteI am not sure because the lines are actually missing so git blame doesn't say much12:17
gfidenteI'll wire those back in and add a bunch of people to see if anything comes up12:17
gfidenteI am not sure if more are missing12:19
gfidenteshardy, I wonder if we shouldn't remove the default for things supposed to come from the parent stack?12:19
gfidenteand the description! :)12:20
jistrgfidente: so you mean that there exist top level params but they aren't wired to the node params?12:21
gfidentejistr, yeah12:21
jistrgfidente: i just went through the code and it looks like this wiring never existed. We missed that because if you pass it via parameter_defaults, it works anyway.12:22
gfidenteuhm like never never?12:22
gfidenteI remember client in 7x was passing those as parameter12:22
mariosah right :/ there is some way by doing sthing like git log -p but can't figure out how to get the commit or change id yet12:23
mariosgfidente: ^^ (about the deletion)12:23
jistrnever never, at least upstream. The patch which introduces the params to overcloud-without-mergepy.yaml only adds them to parameter list, doesn't use them anywhere else in the file.12:23
shardygfidente: IMO it depends which templates, e.g when we move to composable roles we'll probably stop passing nearly all parameters from, and instead rely on parameter_defaults for per-service parameters12:23
shardyif we're sure the parent will always pass the value, removing the default is probably fine tho12:24
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Wire NeutronTunnelIdRanges and NeutronVniRanges to to node templates
mariosgfidente: yeah like git log -p puppet/controller.yaml should help you find it at least12:24
*** thrash|g0ne is now known as thrash12:24
jistrmarios, gfidente: be0d3f3520037c4e2e9fb4f9a235171aaad7e25312:25
marios(jistr i used that commit id to get the review)12:26
mariosjistr: but this adds them... gfidente are they removed?12:26
marios14:16 < gfidente> jistr, marios I noticed the tunnelidranges and vniranges parameters are not wired from top into controller and compute templates anymore12:26
gfidentemarios, nah they're not wired12:27
gfidenteand we didn't notice I think12:27
jistrmarios: i think what gfidente meant is that they're only in the param list but not wired further here
jistr(passed to compute/controller)12:27
* marios palmface12:28
jistrif you pass it via parameter_defaults, it doesn't change actual behavior though12:28
gfidenteshardy, so I am going to try a mass delete of default in controller and compute yaml12:28
jistrso that's why we probably didn't notice12:28
mariosjistr: yeah cos they are wired into the controller/compute itself12:28
shardygfidente: I personally am -1 on that, for the reason I just mentioned (composable roles)12:29
gfidentebut again, the client was using parameters: initially12:29
shardyall those parameters are going to move into per-service templates, where the defaults will be needed12:29
shardyand all parameters for services will be removed from overcloud.yaml12:29
shardygfidente: however, I'm +1 on removing any defaults where no sane default actually exists :)12:30
gfidenteshardy, yeah I get that with composables we'll add back the default12:30
shardygfidente: Ok, cool12:32
gfidenteshardy, but my point is to sanitize where it is needed and where it isn't12:32
gfidenteand fail if we miss it12:32
shardygfidente: ack, that makes sense, I'm just hoping it doesn't make the refactoring for composable roles harder12:32
shardyI guess we'll copy the parameter definitions from overcloud.yaml instead, which should be OK12:33
* shardy goes to lunch12:33
jistrgfidente: what about removing the top-level params instead? everybody who used parameter_defaults will have it working still, and everybody who used parameters never had it working anyway12:34
gfidentejistr, that is my point for using always parameter_defaults12:35
gfidentelike we're doing now12:35
jistryea +1. i'll mention it on the review then12:35
gfidenteas long as we take one route12:36
gfidenteand to be honest, that would also cleanup the templates from lines which are just passing variables along, so I'd like that too12:37
gfidentejistr, comments, I'd like us to act into one of the two directions12:46
*** trown|outttypeww is now known as trown12:48
jistrgfidente: is there something else unwired though? or do you mean even for the params which are wired?12:48
gfidenteI mean for all12:49
gfidentea proper cleanup of a 1000lines!12:49
jistryea, this probably makes sense to be done while extracting the composable roles12:51
jistrremoving top level params was problematic when CLI used parameters instead of parameter_defaults, but i think nowadays we don't have to worry about that12:52
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: [WIP] OpenContrail integration
afazekasderekh, I guess the nested guest support is disabled on testenv, however the n-cpu is configured to use 'kvm' instead of 'qemu'13:05
derekhafazekas: yes, nested kvm is disabled, we (only yesterday), fixed that config to set it to qemu
afazekasnova.conf [libvirt].virt_type  is expected to be configured by  the user, or is it based on auto detection ?13:07
afazekasderekh: if the host at least el 7.1 nested guest can be enabled  (unofficial)13:08
afazekasderekh, thx, I'll apply that change13:09
*** jtomasek has quit IRC13:09
*** rhallisey has joined #tripleo13:10
derekhafazekas: last time I saw that tried (not sure what version it was) the host kernels were core dumping a lot, would be worth trying again when doing a redeploy13:10
*** akuznetsov has joined #tripleo13:11
*** tzumainn has joined #tripleo13:16
*** jayg|g0n3 is now known as jayg13:16
*** dprince has joined #tripleo13:17
*** weshay_xchat has joined #tripleo13:18
*** weshay_xchat is now known as weshay13:18
*** fgimenez has quit IRC13:19
afazekasderekh: It can made the 'heat-slow' named job 'heat-fast' :)13:19
*** fgimenez has joined #tripleo13:19
*** fgimenez has quit IRC13:19
*** fgimenez has joined #tripleo13:19
derekhafazekas: ;-)13:20
afazekasIf by accident machines get replaced >=haswell cpu also recommended for 'heat-even-faster' ;-)13:23
*** lblanchard has joined #tripleo13:33
EmilienMjistr: can you look please ?13:36
dprincejistr: too13:39
*** masco has quit IRC13:39
jistralright :)13:40
jistryea i think UC is fine with httpd as there's no HA arch to be worried about13:40
EmilienMderekh: good news, the mongodb should work13:41
EmilienMderekh: looking at zuul, gate-tripleo-ci-f22-nonha is green when unpinning13:41
derekhEmilienM: great, thanks13:42
EmilienMderekh: so should be ok (after job results)13:42
EmilienMjistr: well, FYI keystone under wsgi is not an option anymore in Mitaka13:42
*** julim has joined #tripleo13:42
jistrhmm though what was the result re yesterday's discussion about not landing anything unless it moves the pingtest forward? (i just approved and un-approved the change :) )13:43
EmilienMjistr: why don't you merge it?13:43
mariosjistr: i think that is all done now derekh ?13:43
mariosderekh: the toci change was what we were waiting for?13:43
* marios driveby comment back to rebases13:43
mariosjistr: (this landed this morning
* jistr checks recent CI logs for pingtest13:48
openstackgerritMerged openstack/instack-undercloud: Clean out os-refresh-config on every run
EmilienMjistr: that's an undercloud patch13:49
EmilienMI don't get why we don't move forward with that13:49
jistrEmilienM: yea i was just going to say "pingtest *is* in CI, this would probably not cause overcloud to become unpingable all of a sudden"13:50
jistrEmilienM: i'm just doing "measure twice, cut once" (not sure if that's just a czech saying or a more global one :) )13:51
hewbroccajistr: I've certainly heard it in English13:51
trownjistr: definitely global, and definitely appreciated13:52
shardyderekh: Hey, I'm trying to build oslo.utils with --delorean-build, and it's failing because map-project-name only maps from oslo-utils to the rdopkg python-oslo-utils13:52
openstackgerritBrad P. Crochet proposed openstack/instack-undercloud: Clean out os-refresh-config on every run
shardyderekh: do you think a patch to map-project-name adding a --upstream option would be OK?13:52
shardyOr, I can special-case it inside tripleo.sh13:52
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Set NeutronMetadataProxySharedSecret
shardybasically I want package["upstream"], not just "name"13:53
openstackgerritMerged openstack/instack-undercloud: run keystone in a wsgi process
shardySorry, I just realized apevec actually wrote that script, I should probably ask him ;)13:54
derekhjistr: As I understand it we're ok to merge now that the ping test is running, of course you'll need to recheck if your CI results are from before the ping test13:54
*** akuznetsov has quit IRC13:54
jistrderekh: yeah though the nature of the change wasn't overcloud related, as EmilienM said13:55
*** egafford has left #tripleo13:55
*** egafford has joined #tripleo13:56
*** weshay has quit IRC14:01
*** shardy has quit IRC14:03
*** shardy has joined #tripleo14:05
marios[m@m openstack-tripleo-heat-templates]$ git log | grep "Use KeystoneAdminApiVirtualIP for heat keystone domain admin setup"  Use KeystoneAdminApiVirtualIP for heat keystone domain admin setup14:11
*** trozet has joined #tripleo14:13
openstackgerritEthan Gafford proposed openstack/python-tripleoclient: Trove integration
*** weshay_xchat has joined #tripleo14:18
*** morazi has joined #tripleo14:20
*** tiswanso has joined #tripleo14:22
*** tiswanso has quit IRC14:26
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: [WIP] OpenContrail integration
*** rlandy has joined #tripleo14:27
*** fgimenez has quit IRC14:27
*** fgimenez has joined #tripleo14:30
*** fgimenez has quit IRC14:30
*** fgimenez has joined #tripleo14:30
*** jtomasek has joined #tripleo14:31
*** tiswanso has joined #tripleo14:32
openstackgerritMarius Cornea proposed openstack/tripleo-heat-templates: Currently the vnc server on the compute nodes binds on While this works with IPv4 addresses it breaks connectivity with IPv6 addressing. This change allows the vnc server on the compute nodes to listen on the address assigned on the network speci
*** absubram has joined #tripleo14:37
*** egafford has quit IRC14:41
openstackgerritMarius Cornea proposed openstack/tripleo-heat-templates: Allow the vnc server to bind on IPv6 address on computes Currently the vnc server on the compute nodes binds on While this works with IPv4 addresses it breaks connectivity with IPv6 addressing. This change allows the vnc server on the compute nod
*** egafford has joined #tripleo14:42
*** akuznetsov has joined #tripleo14:46
*** egafford has quit IRC14:48
shardyHey all, can we land this change, which adds some swap to the undercloud?14:50
shardya bunch of jobs have been failing CI because the OOM killer randomly kills heat-engine processes14:50
*** liverpooler has quit IRC14:50
shardyuntil we work out a more permanent solution, some swapping seems less-bad14:51
shardydevvesa: ^^ FYI14:51
therveshardy, Is that a recent issue?14:53
openstackgerritMarius Cornea proposed openstack/tripleo-heat-templates: Allow the vnc server to bind on IPv6 address on computes
shardytherve: Yes, we increased the vcpu count for the undercloud, now we're running out of ram because of lots more workers14:54
therveshardy, Ah yeah, I thought that it may be related to that. Reducing the number of workers is a potential solution too.14:54
shardytherve: but, we needed the workers in some cases, e.g there were RPC timeouts w/heat because we were attempting to use a single heat-engine process for huge tripleo stacks14:55
shardytherve: Yeah, there may be some tuning we can do, but the main reason for the CPU count increase was to increase the number of workers ;)14:55
shardybasically, we need more memory14:55
afazekasshardy: we can add swap creation as pre deployment script/hook as temporary workaround14:56
*** julim has quit IRC14:56
*** akuznetsov has quit IRC14:56
afazekasAs I remember the templates has this option14:56
shardyafazekas: It's the undercloud that needs it14:56
shardyafazekas: the patch I referenced above adds some swap prior to attempting the overcloud deploy14:56
slagleshardy: it lgtm, i'm happy to land it14:57
afazekassorry, I already +1 ed it, but we will need more mmeory on the controllers soon (if not now)14:57
shardyafazekas: ack, yeah that's a separate issue - atm we just need to get CI jobs which were working back to a reliable state14:58
shardyslagle: thanks14:58
afazekasshardy: for the monitoring  we could add a similar dstat script as devstack uses14:58
shardyafazekas: Yeah, something like that would be helpful14:59
openstackgerritMerged openstack-infra/tripleo-ci: Add some swap to the undercloud
*** rpothier has joined #tripleo15:01
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Generate a unique DeployIdentifier on updates
*** julim has joined #tripleo15:02
afazekasshardy, what is the right mailing list to speak about triplo-ci infra (rh1) ?15:04
shardyafazekas: openstack-dev with the [TripleO] subject, although I think only derekh and dprince have access to the actual infrastructure servers15:09
openstackgerritDougal Matthews proposed openstack/tuskar: Retire the Tuskar codebase.
derekhshardy: afazekas the people on this list have access,
derekhshardy: afazekas although in reality only a small subset of people use the access and know where things are15:12
shardyderekh: thanks, looks like that list should be pruned down to three ;)15:12
afazekasderekh, is jog0 still around ?15:12
derekhshardy: yup probably15:13
openstackgerritDougal Matthews proposed openstack/tuskar-ui: Retire the Tuskar UI codebase.
*** akuznetsov has joined #tripleo15:13
afazekasderekh, may be memory is broken, but as I remember nowadays he does something else15:14
derekhafazekas: yup, a bunch of people can be taken from the list, I'll submit a patch now, not sure if the script that syncs access hadn't delete thought15:15
derekhdprince: do you know ? ^15:15
openstackgerritDougal Matthews proposed openstack/python-tuskarclient: Retire the Tuskar Client codebase.
dprincederekh: I think the script adds, but I seem to remember deleting being manual15:15
dprincederekh: Will and Kambiz can help us w/ that on the bastion15:16
dprincederekh: I would just email them to double check...15:16
derekhdprince: I'm going to submit a patch to remove people from the list that are no longer on tripleo, then ping one of them about it when its merged15:16
dprincederekh: cool, that is the correct way to proceed here I  think15:17
*** lazy_prince has joined #tripleo15:19
openstackgerritDerek Higgins proposed openstack/tripleo-incubator: Remove admin no longer working on tripleo
derekhdprince: shardy afazekas ^15:19
openstackgerritDerek Higgins proposed openstack/tripleo-incubator: Remove admins no longer working on tripleo
*** akuznetsov has quit IRC15:23
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: OpenContrail heat templates
*** gchamoul has quit IRC15:24
EmilienMjistr: can you land please ?15:25
*** akuznetsov has joined #tripleo15:25
jistrEmilienM: landing15:25
dprincederekh: fire when ready15:25
openstackgerritMerged openstack/tripleo-common: Revert "Pin the puppetlabs mongodb puppet module"
EmilienMjistr: you on fire!15:27
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: OpenContrail heat templates
openstackgerritSteven Hardy proposed openstack/tripleo-common: --delorean-build handle oslo.* package builds
*** dprince has quit IRC15:35
*** akuznetsov has quit IRC15:35
*** mbound has quit IRC15:37
*** tiswanso has quit IRC15:38
*** tiswanso has joined #tripleo15:39
*** egafford has joined #tripleo15:40
openstackgerritMerged openstack/tripleo-heat-templates: Don't write CLOUDNAME to the hosts file.
*** masco has joined #tripleo15:47
openstackgerritNicolas Hicher proposed openstack/tripleo-heat-templates: OpenContrail heat templates
*** aufi has quit IRC15:57
*** shardy has quit IRC15:57
*** shardy has joined #tripleo15:59
*** jdob has joined #tripleo16:03
*** gchamoul has joined #tripleo16:08
*** rcernin has quit IRC16:09
*** lazy_prince has quit IRC16:10
*** yamahata has joined #tripleo16:10
*** regebro has quit IRC16:10
*** mbound has joined #tripleo16:13
*** mkovacik has quit IRC16:14
*** regebro has joined #tripleo16:14
*** slagle has quit IRC16:14
openstackgerritDmitry Tantsur proposed openstack/python-tripleoclient: Install python-hardware package on the agent image by default
afazekasderekh,dprince: I started to log how many testenv is in use, how long sample history can be considered as big enough to say what is the avg usage ?16:16
EmilienMayoung: the rebase on failed...16:17
EmilienMayoung: I'll update the patch16:17
ayoungEmilienM, yeah, there were a slew of lint errors, but I couldn't get lint to run on my home system.  tox was just horribly anti-social when I tried.16:18
openstackgerritJohn Trowbridge proposed openstack/tripleo-common: Update pingtest for newer heatclient
derekhafazekas: probably a few weeks, but we need to allow for peak times of the day at peak times of the cycle, so average usage doesn't tell us much16:18
*** bnemec has quit IRC16:18
*** trozet has quit IRC16:19
*** trozet has joined #tripleo16:19
*** egafford1 has joined #tripleo16:19
*** egafford1 has quit IRC16:19
derekhafazekas: for example we know most of the time most of the testenvs arn't in use, but we need to be able to deal with peaks in order to ensure a backlog doesn't develop16:20
afazekasderekh: I can weight it, but probbaly we can also decrees the number of env in use we can make faster each env ..16:20
*** bnemec has joined #tripleo16:20
*** egafford has quit IRC16:22
*** jdob has quit IRC16:25
*** jdob has joined #tripleo16:25
*** regebro has quit IRC16:27
*** regebro has joined #tripleo16:29
openstackgerritBrad P. Crochet proposed openstack/tripleo-common: Use the new yaml syntax in CI
openstackgerritBrad P. Crochet proposed openstack/tripleo-common: Build image files from definitions in yaml
*** jdob1 has joined #tripleo16:33
*** marios has quit IRC16:34
openstackgerritDmitry Tantsur proposed openstack/instack-undercloud: Update puppet configuration for ironic-inspector
dtantsurI've no idea about puppet, so this ^^ might be terribly wrong16:35
*** jdob has quit IRC16:38
*** olap has quit IRC16:39
*** paramite has quit IRC16:42
*** regebro has quit IRC16:43
*** rcernin has joined #tripleo16:49
trowndtantsur: it is not terribly wrong... we just want to put those values in the hiera16:50
*** masco has quit IRC16:51
*** bnemec has quit IRC16:52
*** Marga_ has joined #tripleo16:52
*** marios has joined #tripleo16:54
dtantsurtrown, oh, that's how it is done! thanks. do you know why some values are set in puppet-stack-config.pp ?16:54
trowndtantsur: legacy I think16:54
dtantsurgot it, will fix16:54
trowndtantsur: anything new should go in hiera16:54
EmilienMjistr: could we land please?16:56
* jistr the lander16:58
jistroof CI run Dec 1016:58
*** davidlenwell has quit IRC16:58
jistri don't suppose it's going to break but 1.5 months is a lot, i'm going to recheck to make sure16:58
EmilienMjistr: sure16:59
EmilienMjistr: yeah I have a lot of old patches16:59
EmilienMI'm trying to get rid of themm16:59
*** davidlenwell has joined #tripleo16:59
*** ChanServ sets mode: +v davidlenwell16:59
openstackgerritDmitry Tantsur proposed openstack/instack-undercloud: Update puppet configuration for ironic-inspector
dtantsurtrown, thanks, done ^^17:00
*** bnemec has joined #tripleo17:02
EmilienMjistr: same for
EmilienMjistr: and
EmilienMat least some feedback17:04
jistrthe 2nd one is merged17:04
EmilienMoh yeah17:04
EmilienMsorry my gertty was not updated17:04
*** dtantsur is now known as dtantsur|afk17:06
*** Marga_ has quit IRC17:07
*** coolsvap|away has quit IRC17:07
*** Marga_ has joined #tripleo17:08
*** fgimenez has quit IRC17:09
*** ukalifon has quit IRC17:09
*** fgimenez has joined #tripleo17:12
*** olap has joined #tripleo17:18
openstackgerritBen Nemec proposed openstack/tripleo-common: Use where possible
*** dcain has joined #tripleo17:23
*** jistr has quit IRC17:24
*** fgimenez has quit IRC17:24
*** dprince has joined #tripleo17:33
*** shivrao has joined #tripleo17:34
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Allow the vnc server to bind on IPv6 address on computes
*** shadower has quit IRC17:41
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Allow vncproxy to work with ssl enabled
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Configure keystone public_endpoint
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Don't write CLOUDNAME to the hosts file.
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Enable TLS in loadbalancer if cert path is detected
*** stevebaker has quit IRC17:43
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Allow the vnc server to bind on IPv6 address on computes
openstackgerritOpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements
*** rwsu has joined #tripleo17:48
*** shivrao_ has joined #tripleo17:48
*** ukalifon1 has joined #tripleo17:49
*** davidlenwell has quit IRC17:50
*** shivrao has quit IRC17:51
ayoungEmilienM, OK, now I am ready to test things again
*** shivrao_ has quit IRC17:53
*** bvandenh has joined #tripleo17:54
*** davidlenwell has joined #tripleo17:55
*** ChanServ sets mode: +v davidlenwell17:55
*** egafford has joined #tripleo17:58
*** marios has quit IRC17:59
egaffordbnemec, dprince: Any notion of how likely it might be to get reviews and merges on,, and in the near future?18:00
egaffordTrying to determine whether to wait on the Trove patches to hit master, or to go ahead and switch my dev env over to stable/liberty to start backporting Sahara before Trove hits (if it is likely to be a significant while longer.)18:00
egaffordAs always, any answer is sensible; just looking for information.18:00
*** akrivoka has quit IRC18:00
egafford (puppet elements) should probably be a slam dunk; heat templates and tripleoclient may be a bit dodgier, but they follow form from Sahara's patches very directly.18:01
*** Marga_ has quit IRC18:03
*** derekh has quit IRC18:03
dprinceegafford: I will try and look at the trove stuff again shortly18:06
*** bvandenh has quit IRC18:07
egafforddprince: Cool; as noted, you've got to do what you've got to do. I just want to know the most efficient path for me to take in re: dev cycles. I'll hold off on stable/liberty for the moment, then. Thanks!18:08
*** jaosorior has quit IRC18:09
*** devvesa has quit IRC18:10
*** jaosorior has joined #tripleo18:10
*** slagle has joined #tripleo18:13
bnemecIt looks like something is breaking the undercloud install right now.  I'm looking into it.18:17
*** eggmaster has quit IRC18:17
*** trown is now known as trown|lunch18:22
Slower_shardy: can you look at ?18:23
Slower_shardy: should be good to go..18:23
*** Slower_ is now known as Slower18:23
Slowerwe've been using it for weeks.. :)18:23
*** stevebaker has joined #tripleo18:28
*** athomas has quit IRC18:31
*** tosky_ has joined #tripleo18:31
*** alop has joined #tripleo18:32
*** tosky has quit IRC18:32
*** tosky_ is now known as tosky18:32
*** mgould has quit IRC18:32
*** shivrao has joined #tripleo18:33
shardySlower: ack, done18:34
*** jaosorior_ has joined #tripleo18:36
openstackgerritMerged openstack/tripleo-common: Heat Docker agent built with Dockerfile
*** electrofelix has quit IRC18:39
*** penick has joined #tripleo18:39
*** athomas has joined #tripleo18:39
Slowershardy: woot!18:39
Slowerdprince: what do you think of this one?  I like your hack better but.. :)18:42
*** sthillma has joined #tripleo18:43
*** ukalifon1 has quit IRC18:43
*** ukalifon has joined #tripleo18:43
Slowerdprince: it does cut about 3mins off the deploy time18:44
*** Marga_ has joined #tripleo18:44
*** gfidente is now known as gfidente|afk18:45
dprinceSlower: probably fine for now I think18:47
dprinceSlower: I've got a new idea for the "hack" BTW. One that I think everyone will find acceptable18:47
dprinceSlower: essentially we can create a custom puppet package "provider" that creates the default directories and users for us18:48
dprinceSlower: that is like 90% of the issue, there are still a few exceptions though18:48
EmilienMayoung: nice18:49
*** ukalifon has quit IRC18:51
Slowerdprince: ah good idea!18:51
dprinceSlower: haven't had the cycles yet to pursue it, but I'll get back to it18:52
openstackgerritPaul Belanger proposed openstack/diskimage-builder: Remove shebangs from non-execute files
openstackgerritBen Nemec proposed openstack/instack-undercloud: Explicitly decode to utf-8
*** tosky has quit IRC19:01
*** xinwu has joined #tripleo19:02
*** Marga_ has quit IRC19:11
*** yamahata has quit IRC19:12
*** Marga_ has joined #tripleo19:16
*** alop has quit IRC19:19
*** alop has joined #tripleo19:19
bnemecln: failed to create symbolic link ���/usr/local/bin/dib-python���: File exists19:22
bnemecAha.  Smoking gun.19:22
bnemecWe should still merge the utf-8 fix at some point so we can debug this stuff.19:23
bnemecOh, and now I see why I couldn't reproduce this.  I didn't build my undercloud image with DIB, so the link wasn't created already.19:24
slagleeasy review if someone could take a look,
*** trown|lunch is now known as trown19:26
openstackgerritBen Nemec proposed openstack/diskimage-builder: Force dib-python symlink creation
bnemecslagle: dprince: ^ will be needed to unblock CI.19:28
trownah that is why I dont hit it with RDO too... was wondring that19:28
*** pradk has quit IRC19:29
dprincebnemec: fire when ready, thanks19:29
bnemecInteresting.  The CI runs on that patch never got to the point of installing the undercloud.19:30
bnemecThat should _not_ have merged.19:31
*** ukalifon has joined #tripleo19:31
*** eggmaster has joined #tripleo19:33
bnemeclol, the zuul eta for the nonha job has dropped to about 40 minutes because the jobs have been failing so fast.19:35
*** xinwu has quit IRC19:35
bnemecdprince: I'm thinking I'll just merge that as soon as it gets past the point where the other jobs were failing.  It can't really make the CI situation worse. :-)19:36
*** gfidente|afk has quit IRC19:39
openstackgerritMerged openstack/tripleo-heat-templates: Add update yaml backward compatibe with PublicVirtualIP on ctlplane
*** yamahata has joined #tripleo19:41
dprincebnemec: ++19:43
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Add update yaml backward compatibe with PublicVirtualIP on ctlplane
bnemecAway she goes.  Now we just have to wait for the infra gate to catch up.19:53
*** ayoung has quit IRC19:56
*** jcoufal has quit IRC19:59
*** ukalifon has quit IRC20:06
*** eggmaster has quit IRC20:07
*** xinwu has joined #tripleo20:10
*** ukalifon has joined #tripleo20:12
*** gchamoul has quit IRC20:13
*** gchamoul has joined #tripleo20:20
*** Marga_ has quit IRC20:22
*** Marga_ has joined #tripleo20:22
*** ukalifon has quit IRC20:34
*** ryansb has quit IRC20:40
*** tiswanso has quit IRC20:42
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Manage keystone initialization directly in t-h-t manifests
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: endpoint_map: bump nova api from 2.0 to 2.1
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: puppet: specify server lists for mongodb_conn_validator provider
*** ryansb has joined #tripleo20:44
*** ryansb has joined #tripleo20:44
egaffordIs the recommended dev env spinup tool for stable/liberty?20:48
egaffordSeeing some fairly early errors without modification; figured I'd check.20:49
*** mkovacik has joined #tripleo20:49
bnemecegafford: I would say yes, although I haven't actually tried it myself.20:50
* bnemec intends to work on the stable branch at some point, when everything stops breaking20:51
egaffordbnemec: How do you build stable/liberty envs?20:51
bnemecegafford: I don't.20:51
bnemecI've only done backports, and I rely on CI to tell me if I broke something.20:51
egaffordOkey doke.20:51
*** yamahata has quit IRC20:52
*** lblanchard has quit IRC20:54
egaffordbnemec: So... does openstack/os-cloud-config have a stable/liberty branch?21:05
egaffordIf not, what?21:05
bnemecegafford: No, that's supposed to be strictly backwards compatible.21:05
egaffordbnemec: Okey doke.21:06
*** yamahata has joined #tripleo21:06
egafford<super downstream concern in public>Will the master be pulled into RHOS 8 again prior to release?</sdcip>21:06
EmilienMhaving this CI error:
openstackgerritEthan Gafford proposed openstack/puppet-tripleo: Sahara integration
EmilienMis it a known bug?21:10
EmilienMdprince, bnemec ^21:10
bnemecegafford: I _think_ so, but you probably want to talk to somebody downstream about it to know for sure.21:10
bnemecEmilienM: Yes, a fix is in the gate, but was failing the check job last I looked so it may be a while. :-/21:11
*** prometheanfire has joined #tripleo21:11
bnemecI'm looking for a way to cherry-pick it into CI, but that all go blown away in the move to tripleo-common, so I'm not sure where to put it.21:11
prometheanfiregate sucking?
bnemecprometheanfire: Yes21:11
dprincebnemec: can you like the patch again?21:11
bnemecA bad change got merged.21:11
prometheanfirek, that's what I thought21:11
bnemecdprince: Or do you want the broken one?21:12
dprincebnemec: nah, that was fine. Not sure why I failed to see that in my reviewed patches. :)21:13
dprinceEmilienM: ^^ that was the issue21:13
bnemecdprince: I'm seeing a lot of failures today in the dib gate too.  It may be tricky to get this to merge. :-/21:14
openstackgerritEthan Gafford proposed openstack/tripleo-puppet-elements: Sahara integration
bnemecBleepity bleep bleep bleeping bleep.21:14
bnemecMy undercloud just died building an image.21:15
*** absubram has quit IRC21:15
bnemecApparently even 12 GB is no longer enough to build in memory safely.21:15
*** dshulyak_ has quit IRC21:18
trownbuilding images directly on the undercloud is one of the more inefficient things in our workflow21:20
openstackgerritEthan Gafford proposed openstack/tripleo-heat-templates: Removing Sahara password default
trownit is pretty much always going to be faster to build from the host machine21:20
bnemectrown: Yeah, unfortunately it's also about the only way to ensure all the things are in sync right now.21:23
bnemecWe used to build images on the host, but unless you start over from scratch and reinstall the host from the latest delorean every time you end up with the images being out of date.21:23
trownbnemec: ya that part seems odd to me... it seems like what is on the host and what is in the image do not need to be coupled21:25
trownfor RDO for instance, I am building images with the liberty release rpm installed on the host... even for mitaka delorean images21:26
*** dshulyak_ has joined #tripleo21:26
bnemectrown: Yeah, that's where dprince's deploy from swift artifacts spec comes in.  It should decouple the image from the thing being deployed a lot more.21:26
*** mbound has quit IRC21:28
trownbnemec: ya what I am talking about is a bit different... ie if you ensure the image has the up to date correct repos, it shouldnt matter what is on the host21:28
trownthe host just needs some semi-recent version of dib and tripleoclient21:29
trownand those are not even really hard reqs, they are only required to try to follow our workflow21:30
trownreally what we want from an overcloud image is an image with certain repos and packages installed... that is a pretty trivial problem compared to the pretty complex machinery in our workflow21:31
trownthe ramdisk is slightly more complex, but I am not sure even that has to be21:31
bnemecI guess downstream already does kind of work that way.  You export a bunch of env vars that tell the included elements how to set up the repos.  The only client dep is that the client has to include the appropriate elements.21:33
*** jayg is now known as jayg|g0n321:33
dprincetrown: In my dev environment I always build on a "host" machine. I'd like to get to the point where the undercloud was treated more like a black box21:36
*** julim has quit IRC21:36
dprincetrown: you can actually build your images elsewhere21:36
trowndprince: ya for RDO the images (including an undercloud.qcow2) are getting built in CI21:36
dprincetrown: tripleoclient currently does a local ramdisk copy to a web root21:36
trownso whenever a delorean repo is promoted, it pushes new images21:37
*** penick_ has joined #tripleo21:37
*** penick has quit IRC21:37
*** penick_ is now known as penick21:37
trownI have thought of proposing a spec for tripleo, but am not sure it fits in our CI21:37
dprincetrown: a spec to build images outside of the workflow?21:38
trowndprince: a spec to build an undercloud.qcow2 with overcloud images built in21:38
dprincetrown: I'd actually like to see us go the other way21:39
trowndprince: since we also install most packages twice, and package installs are the thing that takes the longest in both the image building and undercloud install21:39
dprincetrown: I would love a pre-built undercloud.qcow (with packages installed). This is actually what we used to do with seed.qcow in devtest.sh21:40
dprincetrown: that sort of utility model is quite nice, but I don't think the overcloud images belong inside of it21:40
bnemecIt's possible we could pre-populate the dib yum cache based on what is already installed on the build system though.21:41
dprincetrown: those could be pre-built elsewhere too, and if you were say only testing a puppet-nova patch you wouldn't have to rebuild them necessarily21:41
trowndprince: ya, it is pretty nice to just virt-customize in a simple fix to a pre-built overcloud image21:41
bnemecAlthough that wouldn't hugely help CI or me because we both have squid configured to avoid re-downloading packages anyway.21:41
trowntakes on the order of a minute, vs 20 minutes to build images21:41
dprincetrown: I'm keen on customizing a stock image if it saves us this sort of time, very keen21:42
*** jprovazn has quit IRC21:42
dprincetrown: for undercloud we might need to customize21:43
*** shardy has quit IRC21:43
dprincetrown: for overcloud though I think simply building the packages, and tarballing the puppet modules and we could just deploy them in CI with the swift artifacts deployment mechanism21:43
bnemecWe might want to point CI at a fully updated base cloud image.  I know in my image builds I lose a bunch of time updating 3 or 400 packages that are just part of the base image.21:44
trownbnemec: ya I made that optimization for RDO CI21:44
bnemecMaybe leave like the nonha job pointing at the base image just to make sure it works.21:44
trownit shaves a good 5 minutes21:44
trownsince the base cloud image needs like 200 package updates21:45
*** trozet_ has joined #tripleo21:45
*** trozet has quit IRC21:47
trowndprince: what do you think we would need to customize on the undercloud? assuming the undercloud is just a cloud image with packages and repos installed21:47
*** trozet_ is now known as trozet21:47
trownie `openstack undercloud install` has never run21:47
dprincetrown: I think cracking the image, or splicing into it would be fine21:48
dprincetrown: I've used DIB for this sort of thing before (like with the old seed)21:48
dprincetrown: once upon a time I had patches posted to do that. They weren't well received but we could revive those. It really is quite simple w/ DIB21:49
dprincetrown: or any virt-customize, libguestfs sort of approach would be fine too21:49
trowndprince: I am pretty partial to libguestfs, but I get that is not the default for tripleo21:49
bnemecDoes guestfs work in vms?21:50
dprincebnemec: I think so21:50
trownbnemec: it does21:52
trownI use it from the undercloud all the time21:52
openstackgerritSteve Baker proposed openstack/tripleo-common: Convenience methods in base model class
openstackgerritSteve Baker proposed openstack/tripleo-common: Implement ViewManager list_software_deployments
openstackgerritSteve Baker proposed openstack/tripleo-common: Implement ViewManager list_stack_failures
openstackgerritSteve Baker proposed openstack/tripleo-common: Implement ViewManager show_software_deployment
*** dsneddon is now known as dsneddon_afk21:58
trowndprince: how could a pre-built undercloud.qcow2 be used in our CI? assuming it had a delorean server on it to build packages under test21:58
openstackgerritBen Nemec proposed openstack/python-tripleoclient: Remove ntp element from overcloud images
bnemecslagle: ^ fixes the non-sticky puppet ntp.conf problem.22:00
dprincetrown: download the qcow2, crack it -> install new packages and puppet modules, etc. Then deploy it as normal to the testenvs22:00
dprincetrown: I think we'd build the delorean packages on the Jenkins slave right?22:00
dprincetrown: this could be done ahead of time....22:01
trowndprince: hmm, and the qcow2 would be created in a periodic job?22:01
dprincetrown: sure, we aren't doing this sort of thing yet, but infra does22:01
trowndprince: ok, I will try to get a spec started for this, I am doing this in RDO right now22:01
dprincetrown: like nodepool creates its own pre-cached version of things and copies over an /opt/stack blob for each job with all the refspecs set accordingly22:02
stevebakertrown: hey, are there undercloud images being generated and put somewhere?22:02
*** penick has quit IRC22:02
trownstevebaker: ya is where they end up22:02
dprincetrown: we could just use them from RDO as a test too22:02
stevebakertrown: oh, sweet22:03
trownstevebaker: they are just overwritten, so stable matches delorean current-passed-ci22:03
trownstevebaker: and testing is used during the CI test run itself22:03
stevebakertrown: is there any boot-time config needed?22:03
dprincestevebaker: hey, just noticed your patches above. What would you think if we implemented those via Mistral workflows instead?22:04
trownstevebaker: the undercloud.qcow2 is just packages and repos, so you still have to create an isntackenv.json and ssh keys22:04
dprincestevebaker: the Mistral API would let us expose the output to python-tripleoclient and a UI just as well22:04
trownstevebaker: I have some ansible I have been working on for both the image creation and consumption22:05
stevebakertrown: but the undercloud services are running?22:05
trownstevebaker: nope, you have to run the undercloud install, but it is super fast because the packages are all installed22:05
stevebakertrown: ok, cool22:06
bnemecHow is CI so broken?!22:06
bnemecThere hasn't been a dib patch that passed tripleo-ci in two days because of package build failures.22:06
openstackLaunchpad bug 1536807 in tripleo "TOCI failing to build package on dib patches" [Critical,Triaged]22:08
stevebakerdprince: these methods just do get and list calls, then process and return the results. I'm not sure that would fit into a workflow API. I'm not against mistral handling these in the future but I would *really* like these commands to get into liberty to help users debug and manage their overclouds22:08
stevebakerdprince: there will be some corresponding tripleoclient changes which call these tripleo-common methods22:09
*** jaosorior_ has quit IRC22:09
dprincestevebaker: right, looking at the initial spec I see you got some feedback on this already22:09
dprincestevebaker: I probably would've just landed them in python-tripleoclient first22:10
stevebakerdprince: yeah, I don't mind. These methods would be useful to a UI too22:10
dprincestevebaker: I realize these are simple things, but Mistral already has native support for interacting with openstack API's quite nicely22:10
stevebakerdprince: but isn't the idea of a workflow that it performs a change on the target? these are just observing commands22:11
dprincestevebaker: there are discussions underway to use Mistral in place of the TripleO API.22:11
dprincestevebaker: a workflow takes inputs, and provides outputs22:12
dprincestevebaker: This is my prototype work to make python-tripleoclient call a Mistral workflow to do the deployment for us:
prometheanfirebnemec: let me know when ci is working?22:13
dprincestevebaker: I've got some example custom actions here:, but many of the stock "OpenStack" actions are supported natively by Mistral22:13
*** pradk has joined #tripleo22:13
stevebakerdprince: I would bet you beer that mistral wouldn't be able to implement this sort of logic without some custom python in the mix
dprincestevebaker: probably more than you wanted to chat about right now... but the main reason I suggest these to you is that the UI may want to run your code too, and in that case I would propose perhaps just having both tripleoclient and the UI use the same codepath (API)22:14
dprincestevebaker: custom python is totally fine for Mistral. Stevedore's entrypoints load the custom actions quite nicely22:15
dprincestevebaker: but I'll take your bet ;)22:15
dprincestevebaker: because I think I win either way22:15
bnemecprometheanfire: Yeah, there's a whole chain of fail going on right now.  There are multiple things broken in dib. :-(22:16
prometheanfireya, have a patch in neutron in the same position22:16
prometheanfiregate is probably over 36 hours now22:16
stevebakerdprince: ok, so those stevedore things would still be contributed by tripleo-common, so we're still future-proofing ourselves22:17
dprincestevebaker: yep, no patches to Mistral required
dprincestevebaker: we can get a "stable" API by simply versioning our workflows22:18
dprincestevebaker: tripleo.v1.<workflow_name>22:18
*** penick has joined #tripleo22:18
* bnemec needs to finish writing his email about this whole Mistral thing22:19
bnemecLet's just say I have concerns.22:19
egaffordWait, are we integrating Mistral as a strict dependency of TripleO?22:22
stevebakerDan has gone to dinner22:22
*** dshulyak_ has quit IRC22:22
*** rpothier has quit IRC22:28
*** olap has quit IRC22:30
*** weshay_xchat has quit IRC22:40
*** trown is now known as trown|outttypeww22:43
*** ayoung has joined #tripleo22:58
*** egafford has quit IRC22:58
NobodyCamgood morning / afternoon / evening / night TripleO folks23:00
NobodyCamany one encountered this error today?23:00
NobodyCamCould not fetch URL There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) - skipping23:00
bnemecNobodyCam: Yes.  It's killing all of the dib functests right now. appears to fix it.23:01
bnemecNote that there are two other problems in dib that also need to be resolved before anything should be merging there too.23:02
openstackLaunchpad bug 1536807 in tripleo "TOCI failing to build package on dib patches" [Critical,Triaged]23:02
bnemecAnd another one I haven't actually opened a bug for but is killing all of the TOCI jobs in other projects too.23:02
NobodyCamoh pinning to 7.1.2 not 8.0.123:03
NobodyCambnemec: :( ugh23:03
NobodyCambnemec: Thank you for the pointer(help) :)23:05
openstackgerritMerged openstack/diskimage-builder: Use pip 7 for ironic
NobodyCamlol wow what timing :)23:06
bnemecNow to find out if it actually fixes the problem. :-)23:06
bnemecAlthough it's the only patch that has passed the functest CI in two days, so I feel pretty safe saying it will.23:06
*** xinwu is now known as xinwu_23:07
openstackgerritBen Nemec proposed openstack/python-tripleoclient: Remove ntp element from overcloud images
*** dcain has quit IRC23:13
*** trozet has quit IRC23:31
*** mcornea has quit IRC23:39
*** dprince has quit IRC23:40
*** rcernin has quit IRC23:47
*** isq_ has quit IRC23:48
*** isq has joined #tripleo23:56

Generated by 2.14.0 by Marius Gedminas - find it at!