14:00:22 #startmeeting tripleo 14:00:28 Meeting started Tue Aug 9 14:00:22 2016 UTC and is due to finish in 60 minutes. The chair is shardy. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:29 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:31 The meeting name has been set to 'tripleo' 14:00:34 #topic rollcall 14:00:38 Hi all, who's around? 14:00:40 o/ 14:00:40 <_coolsvap_> o/ 14:00:41 hello 14:00:41 HI 14:00:44 o/ 14:00:47 o/ 14:00:47 hi 14:00:48 o/ 14:00:49 heya 14:00:50 hi \o 14:00:52 o/ 14:00:54 hi 14:00:59 o/ 14:01:00 o/ 14:01:04 o/ 14:01:17 HEllo! 14:01:21 o/ 14:01:34 o/ 14:01:58 #topic agenda 14:01:58 * one off agenda items 14:01:58 * bugs 14:01:58 * Projects releases or stable backports 14:01:58 * CI 14:02:01 * Specs 14:02:03 * open discussion 14:02:05 o/ 14:02:07 o/ 14:02:09 #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:02:24 Seems we've already got several one-off items, please add any remaining ones to the end of the list 14:03:00 #topic one-off agenda items 14:03:03 hi 14:03:30 rkolli added one but doesn't seem to be here? 14:03:34 shardy: hi, i am proxing for rkolli 14:03:41 SumitNaiksatam: Aha, hi! 14:03:42 he was planning to be here 14:03:50 meanwhile i will try to fill in 14:04:10 so we were looking for guidance on how to make progress with: #link https://review.openstack.org/#/c/352661 14:04:10 So, I'm wondering what the related patch is, which requires this new hieradata? 14:04:19 #link https://review.openstack.org/#/c/352661 14:04:26 shardy: right 14:04:29 #link https://review.openstack.org/#/c/351358 14:04:50 SumitNaiksatam: basically the issue is, we don't allow feature backports, and we're trying to land everything in the new composable services format to master 14:05:23 shardy: so 352661 is essentially a backport of 351358 14:05:30 what you're proposing seems low-risk, but for master it's a poor fit for our new architecture, and for mitaka I'm wondering if it implies a follow-up patch that backports a feature? 14:05:57 shardy: i believe there are no follow up patches to be upstreamed 14:06:12 ah rkolli is here 14:06:16 SumitNaiksatam: Ok, so this is to enable an out-of-tree integration via ControllerExtraConfigPre? 14:06:16 o/ 14:06:22 shardy: right 14:07:03 shardy: we wondering if #link https://review.openstack.org/#/c/352661 can be treated as perhaps a bug and accepted? 14:07:13 in stable/mitaka 14:07:21 SumitNaiksatam: does the out of tree template contain logic to calculate hieradata, or is it just writing some static hiera keys? 14:07:54 shardy: good question, i havent seen that part of the code, so I will have to defer to rkolli 14:08:26 SumitNaiksatam: Ok, well I'll add some comments to the review, if it's just some key/value pairs there's a way to do it without this backport 14:08:29 shardy: i believe rkolli is having some logistical difficulties with connecting 14:08:40 shardy: ah okay 14:08:42 SumitNaiksatam: lets chat about it later in #tripleo when I've added those comments 14:08:47 Steven, the out of tree template writes the out the values provided by env file to the hiera data 14:09:14 rkolli: Ok, I think you may be able to use the ControllerExtraConfig and ComputeExtraConfig parameters instead 14:09:23 it's hard to comment when we can't see the out of tree code 14:09:34 if you can post it somewhere it would help provide more specific advice 14:09:54 I will share the out of tree code after this meeting 14:10:13 rkolli: Ok, thanks - let's follow up on the review and/or in #tripleo 14:10:16 shardy: and as far as #link https://review.openstack.org/#/c/351358 is concerned its a firm no go? 14:10:43 SumitNaiksatam: If we have to we probably can do that, but I'd like to discuss the other options first, if that's OK? 14:10:56 shardy: good point, thanks! 14:11:47 The next item is re SR-IOV and DPDK, who wants to give more details on that one? 14:11:51 i would like to give an update on the SR-IOV and DPDK features. 14:11:52 on SR-IOV, we have uploaded all the patches, and it is pending for reviewer. 14:12:12 we have done the integration of all patches, tested and demoed to QE team on the DPDK automation, execept 2 WIP (depends on openvswitch patch), all the patches (including os-net-config) has been uploaded and it is in review state. 14:13:05 karthiks: Ok, thanks for the update, we can push on reviewing those 14:13:18 karthiks: was there a dependency for SR-IOV on custom roles? 14:13:19 Thanks.... 14:13:43 I was wondering how you worked around that, did you just enable it for all compute nodes? 14:14:08 as of now we are considering uniform cluster .... 14:14:08 shardy: yes.. we are enabling for all compute nodes with 14:14:40 assumption as uniform compute nodes.. 14:14:47 beagles: can you help with reviewing these please? IIRC you had a lot of good feedback on the specs 14:14:56 shardy, absolutely 14:15:01 skramaja, karthiks: OK, sounds like a good first step, thanks for the update 14:15:09 let's see if we can get it landed for newton-3 14:15:25 anything else on this before we move on? 14:15:27 :) 14:15:43 Nothing shardy 14:15:48 ok, thanks! 14:16:08 jaosorior: you're next re the network range 14:16:21 #link https://review.openstack.org/#/c/343443/ 14:16:42 shardy: Right, so we finally set up a deprecation message for the 192.0.2..0/24 range in the undercloud 14:16:51 but would be nice to come up with some other default. 14:17:07 so I went ahead and pushed a commit in tripleo-quickstart. So at least an alternative is there as reference. 14:17:10 What do you guys think? 14:17:36 jaosorior: is there any way we can switch the instack-undercloud default, but have it look at the local box so existing deployments aren't modified on update? 14:17:58 I'm not opposed to the quickstart fix, but there are a lot of non-quickstart users, so it feels like fixing it in the wrong place? 14:18:39 The problem is we can't know whether a configuration was using the old default. 14:18:42 o/ 14:18:52 I too prefer to wait for a converged solution 14:19:01 so I thought that at least new users (using quickstart) could start using the new default. So that would be a good start IMO 14:19:09 but yeah, bnemec pretty much pointed out the problem 14:19:28 There's a deprecation message telling people not to do it, next cycle we can change it in instack-undercloud. 14:19:37 bnemec: couldn't the undercloud install look at br-ctlplane and have a conditional based on that? 14:19:44 a special-case for backwards compatibility? 14:20:15 bnemec: just changing it is still going to break folks on upgrade isn't it, even if we warned them? 14:20:24 shardy: Maybe 14:20:32 AFAIK yeah (I tried) 14:20:42 but it's been a while 14:20:42 shardy: The warning tells them to explicitly configure 192.0.2.0 if they need to keep using it. 14:20:45 192.0 is not a legal range to use 14:20:54 its for documentation only. 14:20:56 And yet it works fine. 14:21:00 In most cases. 14:21:02 bnemec, no it does not 14:21:07 ayoung: Yeah, we know that, the problem is folks are already using it 14:21:09 bnemec, we would not be discussing it if it did 14:21:22 it works in *some cases. 14:21:34 THe issue is that we don't know what will or won't work 14:21:34 ayoung: If it didn't work we would not be discussing it. Because we wouldn't have been able to use it for so long. 14:21:37 we need to ask the user 14:21:52 That's not the point anyway. 14:21:57 We're discussing how to change it. 14:22:11 can we go IPv6? 14:22:20 dumb question, but lets throw it out there 14:22:26 it makes the clash a lot less likely 14:22:31 shardy: I can maybe look into the br-ctlplane thing. It's possible that would work. 14:22:51 Anyway, here's the proposed range in reference of tripleo-quickstart https://review.openstack.org/#/c/343443/ . If people are OK with that, it would be a good start, and then we can start figuring out how to fix the undercloud to have a different default and respect previous configurations. 14:22:54 bnemec: ack, sounds good - we could have the workaround trigger a very noisy warning too I guess 14:23:09 It's probably going to be a little hacky, but that's probably okay. 14:23:26 I don't _think_ a lot of people are going to be using this in production anyway, but I've seen crazier things. :-) 14:23:34 lol 14:23:45 can we try and get Tripleo to use IPv6 for all purely internal communication? Is that a well enough tested path that we can at least pursue it? 14:23:53 I think best result would be to make undercloud not change pre-existing deployments and or on upgrade, but switch to a new default for new installs 14:24:05 Ok, sounds like we have a plan, folks can comment on the quickstart patch and bnemec will look into an instack-undercloud fix/workaround 14:24:15 great! 14:24:36 gfidente: not switching on upgrade implies having a n upgrade specific env file (or similar) to have /specify the old defaults 14:24:39 awesome 14:24:45 ayoung: it's not a well tested path AFAIK 14:24:55 marios, I was thinking about undercloud just *not* applying the changes if pre-configured 14:25:17 gfidente: k, lets talk more offline then thanks 14:25:31 shardy, so, I think that changing the defaults, while they will work for us, we are basing them on what we have in our organization, whcih is, I suspect, the same organization fro most developers here 14:25:42 Hmm, good point. Does this affect the net-iso template defaults too? 14:25:53 That's a little scarier. 14:25:56 ayoung: it's just a default, but we chose a bad default 14:26:05 shardy, and anything we propose as a default is going to trip someone up. I would like to make the "query for network ranges" be part of the workflow 14:26:39 ayoung: sure, there's probably a few ways we could be smarter about this during the undercloud configuration phase 14:26:51 ayoung: want to switch to your Federation update? 14:26:55 shardy, sure 14:26:57 (ayoung) Federation PoC Update and lessons learned 14:26:58 this is good news 14:27:18 OK, so Federation means that we have an external applicaiton (KeyCloak in mycase) that has the user database 14:27:59 we have to configure Keystone to accept redirects from Horizon, and redirect to Keyscloak, user signs in, and then the reverse the redirecets (keycloak to keystone, keystone to horizon) 14:28:11 this has been a journey, and I got a proof of concept to work last night. 14:28:31 I am sure there are things I need to do better, and probably some changes coming to puppet modules 14:28:41 the big lessons learned: 14:29:14 1. All of the external facing interfaces need to use TLS. This includes the Auth URL that Horizon uses to talk to Keystone. It was not expoesed in the past, but will be with Fedration 14:30:06 2. We need to use FQDNs as the normal way to deploy. A Keystone server, for example, should not thinkg of its ServerName as controller-0 but rather opensack (or whatever the cloudname is) 14:30:31 #2 is coming soon :D 14:30:40 Also, huge thanks to many people that helpd out... shardy EmilienM dprince all provided huge degrees of help 14:30:55 a big lesson learned is that developers should work with things in HA configuration 14:31:10 ayoung: thanks for the update, good that you got a postive outcome :) 14:31:10 since that is the expected deployemnt, you get something working without HA, it might be meaningless 14:31:19 shardy, blog post with details. 14:31:29 shardy, Oh, and I also got dynamic policy to work to. 14:31:37 It was a good week for Tripleo/Keystone 14:31:48 ayoung: Ah, I'm drafting a blog post for you about the DeployArtifacts interface 14:31:50 nice! 14:31:56 did you get that working anyway? 14:32:11 shardy, I did. I only did Keystone, but it would work for all of the services. 14:32:29 ayoung: Ok, well that's good news - I'll post the blog anyway as it's nearly done 14:32:35 shardy, I might want to use the same approach for dpleoying the metadata for Fedraion 14:33:06 its a cluster of files. The one thing I need to figure out is how to force a yum install of a module required first, or everything will reak 14:33:06 Ok, anything else on this before we move on? 14:33:07 brek 14:33:11 * ayoung done 14:33:31 shardy just added a blueprint draft to check what people think about that idea, we can discuss it later on #tripleo (not getting time from meeting) 14:33:36 ccamacho: You're next - the naked overcloud! :) 14:33:49 not sure if we have enough time 14:33:53 Scandalous! 14:33:54 ccamacho: so, this is a followup from last week, where we discussed enabling less services by default? 14:34:03 yeahp 14:34:24 I'm +1 on that, but as we discussed last week, we do need a CI matrix that provides coverage of services not enabled by default 14:34:28 its the idea in a blueprint 14:34:31 or they'll just be constantly broken 14:35:01 ack, Ill keep elaborating on that 14:35:28 ccamacho: re actually enabling them, which I see you mention in the blueprint, see https://review.openstack.org/#/c/330414/ 14:35:44 The next part of the blueprint should be defining this coverage matrix 14:35:50 ramishra has heat patches for that, so we'll avoid having to map stuff to OS::Heat::None 14:36:13 folks can help by pulling and testing his patches (it's also on my todo list) 14:36:22 ccamacho: as an FYI, there is this kind of matrix in puppet CI https://github.com/openstack/puppet-openstack-integration#description 14:36:28 shardy: have a link to how that works (not using OS::Heat::None) 14:36:33 Ill add a note to my todo 14:36:40 https://review.openstack.org/#/c/351092/ 14:36:57 marios: it's the spec link I just posted, and ^^ is the main patch implementing it 14:37:01 thanks shardy 14:37:08 thanks 14:37:19 Ok, I think that's all of the one-off items 14:37:30 #topic bugs 14:37:50 Can somebody drop a link to ccamacho's spec? 14:37:58 https://bugs.launchpad.net/tripleo/?orderby=-id&start=0 14:38:23 https://blueprints.launchpad.net/tripleo/+spec/deploying-a-naked-overcloud 14:38:35 #link https://bugs.launchpad.net/tripleo/+milestone/newton-3 14:38:46 Does anyone have specific bugs to raise this week? 14:39:01 ccamacho: excellent proposal 14:39:09 I think other than the sahara issues EmilienM mentioned earlier in #tripleo, the bug backlog looks OK 14:39:32 we still have a lot of stuff to land for newton-3 though, so please prioritize reviews to help burn down that list 14:39:33 shardy: right, and jaosorior has a fix 14:39:52 shardy: https://review.openstack.org/#/c/352746/ 14:39:57 ccamacho: Is there an actual spec? blueprints are really awkward to review (thus the spec process). 14:40:04 EmilienM: Ok, thanks for the update 14:40:06 * bnemec apologizes for being stuck on the previous topic 14:40:20 bnemec, I'll change it then to a spec 14:40:38 well, it's a spec-lite 14:41:02 #topic Projects releases or stable backports 14:41:24 So I didn't have time to propose stable branch releases last week, but I'll do that later today 14:41:42 and as mentioned, we're running out of time for newton-3, so please review all-the-things :) 14:41:57 anyone have anything else to raise re releases or backports? 14:42:48 #topic CI 14:43:01 shardy: matbu and I are working on upgrade / update jobs 14:43:07 So, who can give an update re CI seeing as derekh isn't around? 14:43:21 EmilienM: excellent, I saw the ML discussion, sounds good 14:43:28 last I checked, everything was working in tripleo-test-cloud-rh1 14:43:29 FYI, we renamed upgrades -> updades jobs 14:43:35 the main question I have is how do we do it within the infra timeout 14:43:38 I also created: http://grafana.openstack.org/dashboard/db/nodepool-tripleo-test-cloud 14:43:49 so people can visualize what the cloud is doing 14:43:52 shardy: +1 14:44:03 What's the status re RH1, do we have full capacity there now? 14:44:08 shardy: right now, we're still testing on multinode-nonha (aio overcloud), so i don't think we'll hit it 14:44:14 pabelanger: nice! :) 14:44:27 right now it is setup to launch 50 nodes 14:44:31 shardy: Not completely. I think we're still being limited to 50 envs on the infra side. 14:44:34 Yeah, that. :-) 14:44:36 not sure what the actually hardware is 14:44:48 The cloud is configured for up to 80 right now. 14:45:07 That said, I'm a little concerned by the amount of CPU usage on the controller right now. 14:45:09 I also raised a discussion again about the future of tripleo-test-cloud-rh1, if people would like to get up to speed on it: http://lists.openstack.org/pipermail/openstack-dev/2016-August/101043.html 14:45:29 generate reaction is in favor of both, moving to 3rd party CI and running more openstack jobs on the hardware 14:46:35 pabelanger: thanks for that, I remember there was some heated debate on that topic last time ;) 14:47:05 shardy: indeed 14:47:08 Ok, anything else re CI before we move on? 14:47:45 I'll continue my topic on the ML 14:47:57 pabelanger: ack, sounds good, thanks! 14:48:02 #topic Specs 14:48:08 #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:48:33 getting near to the end of the cycle and there's still some stuff to land there, pls review if you have any free cycles 14:49:10 shardy: I posted a new spec for the undercloud heat idea as well 14:49:16 shardy: https://review.openstack.org/#/c/352113/ 14:49:20 dprince: nice, just spotted that 14:49:30 I liked the demo, will review the spec 14:49:33 perhaps more of an 'O' thing 14:49:45 dprince: Yeah, I think so, but good to start the discussion now 14:50:27 Related to that, I've already started deferring anything not started in launchpad to Ocata 14:50:59 please update the implementation status of your blueprints that are targetted to newton-3, so we can get a clear view of what's going to land and what might slip 14:51:26 Anything else re specs to raise? 14:52:10 #topic Open Discussion 14:52:47 I had one thing - a few folks have been asking me for RFEs related to tripleo, particularly heat features we need or would like 14:52:51 do you guys also own the triple-quickstart project? 14:53:09 I'll probably start a ML thread with an etherpad linked so we can track the wishlist there 14:53:09 tripleo-quickstart* 14:53:27 bswartz: There are a subset of TripleO folks who maintain it, yes 14:53:41 but it's supposed to be stable, right? 14:54:02 define: stable 14:54:02 I tried it and had poor luck -- I didn't know if I should try something else or hack away at it 14:54:03 bswartz: it should be stable. Any issue ? 14:54:24 bswartz: ask us on #tripleo at the end of the meeting 14:54:34 panda: it fails nondeterministically 14:54:37 okay thanks 14:54:52 bswartz: It is supposed to be stable, but it isn't what we run in upstream CI right now 14:55:08 yup, lets discuss in #tripleo to figure out the issues, thanks panda! 14:55:24 Anyone else have anything to mention? 14:56:29 Ok then, if that is all, we can wrap up early and head back to #tripleo 14:56:33 thanks everyone! 14:56:34 thx shardy ! 14:56:37 thanks guys!!! 14:56:41 #endmeeting