20:00:00 #startmeeting heat 20:00:00 Meeting started Wed Jan 29 20:00:00 2014 UTC and is due to finish in 60 minutes. The chair is stevebaker. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00:01 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 20:00:05 The meeting name has been set to 'heat' 20:00:05 #topic rollcall 20:00:12 o/ 20:00:12 hi 20:00:13 o/ 20:00:14 \o 20:00:14 hi all! 20:00:15 hi 20:00:17 Hi 20:00:17 o/ 20:00:19 hi 20:00:23 hello 20:00:31 howdy 20:00:37 hey 20:01:08 no actions last week 20:01:25 #topic adding items to the agenda 20:01:27 #link https://wiki.openstack.org/wiki/Meetings/HeatAgenda#Agenda_.282014-1-29.29 20:02:03 any extra topics? 20:02:41 alright then 20:02:44 #topic Remove the intention of ever having an XML API 20:02:49 +1 20:03:02 sdake: I haven't even said anything yet ;) 20:03:03 +2 20:03:10 stevebaker: jasond` wanted to discuss https://bugs.launchpad.net/heat/+bug/1274201 20:03:10 +2/+A then ;) 20:03:11 stevebaker: https://bugs.launchpad.net/heat/+bug/1274201 20:03:29 o/ 20:03:38 o/ 20:03:51 so nova have decided to remove XML support from their new v3 API, and nobody seems to mind 20:03:53 +1 for burning XML 20:04:08 xml had its time - and that time has passed 20:04:12 +2 lets kill xml, except for the CFN API where it already works 20:04:15 apparently java binds nicely to json, so that is a non-issue 20:04:17 +1 20:04:28 +1 20:04:29 * samkottler waves 20:04:42 stevebaker: so, this one time I wanted to add something to the agenda and it hadn't been created yet. So I copied the last week's and left it there because I didn't know if we moved them to another location? And ever since then we have an infinitely-extending list of old agendas 20:04:58 samkottler: o/ 20:05:13 all this means for us is that there is a *bunch* of zaneb TODOs which can be deleted :D 20:05:33 zaneb: what is the agenda item to add? 20:05:51 OK, let's delete them. Because we already decided not to do a native XML api over a year ago 20:06:14 stevebaker I think zaneb's agenda item is cleaning up the agenda page on the wiki :) 20:06:15 #agreed XML can die in a fire 20:06:22 stevebaker: maybe add an agenda item about removing old agendas from the agenda page ;) 20:06:34 zaneb: I'd rather have a vote on that 20:06:44 i think zaneb kind of ninja added it 20:06:47 but vote wfm 20:06:56 that was a joke 20:06:56 stevebaker: I'm just pointing out that it was all my fault 20:06:59 so sorry 20:07:11 * zaneb returns to his corner 20:07:17 aaaaaaanywho 20:07:25 #topic Policy on deprecating cfn features in the HOT spec 20:07:26 stevebaker, zaneb: I think we should have previous agenda and current agenda. 20:07:49 so this was being discussed just now in #heat 20:07:54 wha? 20:08:43 basically some breaking changes were introduced into hot, and either a database migration needs to be introduced, or we need to support a schema with 'String' and 'string' 20:08:47 so HOT is basically a hacked up prototype on top of the CFN code 20:09:14 as we create a proper implementation for it, some things that were never proper HOT but which worked will break 20:09:28 ahhh 20:09:34 and andrew_plunk has found the first example of that 20:09:41 I'm of the ilk that documentation trumps accidental implementation 20:09:55 AFAIK we have never yet declared HOT to be stable (in fact I specifically said it wasn't when we released Havana) 20:09:57 we aren't locked into hot yet 20:10:15 i think we agreed icehosue would be the first stable version 20:10:18 I agree and said as much in #heat 20:10:19 zaneb: what was the problem andrew_plunk found? 20:10:21 but we should consider that decision carefully 20:10:25 it is still painful to introduce breaking changes without some kind of migration 20:10:30 tspatzier: https://review.openstack.org/#/c/67844/1/tempest/cli/simple_read_only/heat_templates/heat_minimal_hot.yaml 20:10:33 Also I feel that the HOT in Havana was discouraged to even be used, and thus I don't think we should try too hard to coddle early adopters.. but we should try hard to make it clear that this is the policy. 20:10:37 the issue is for people who have templates in the db that Heat now cannot read 20:10:37 the hot schema used to support the type 'String' 20:10:48 tspatzier ^^ 20:10:54 tspatzier: it was type: String vs string 20:10:59 Can we say that the "migration" for any changes up to icehouse will be deleting and re-creating your stack? 20:11:12 stevebaker wfm 20:11:15 imagine people with production heat 20:11:22 stevebaker: you won't be able to delete your stack either 20:11:23 also it wouldn't hurt to just poll the openstack@ mailing list to find out if anybody's world will implode if we break havana's HOT. 20:11:26 zaneb, andrew_plunk : ah yes, that was the thing I actually changed in tempest 20:11:43 zaneb: and fixing any issues which result in undeletable stacks 20:11:50 andrew_plunk: why would they be using HOT in production, when we specifically said it was unstable? 20:11:54 would that only be for templates using hot? 20:11:56 or cfn as well? 20:12:10 Anyone can submit templates against our hot 20:12:14 chmouel: only hot, cfn should be stable 20:12:22 SpamapS: we had a tempest test failing and andrew_plunk and tims have both had problems, so it's safe to say the answer will be 'yes' 20:12:23 you can submit whatever syntax you want. 20:12:35 against our heat* 20:12:38 chmouel: only HOT 20:12:54 stevebaker I don't like that delete and re-create approach. The community for Heat is large, even if people are only using it for experimental purposes. 20:13:23 zaneb: hm. Tempest tests I'm not worried about.. that is something we have direct control over. Andrew and Tim can chime in on whether it was world-imploding or just inconvenient. 20:13:25 #link https://wiki.openstack.org/wiki/ReleaseNotes/Havana#Initial_support_for_native_template_language_.28HOT.29 20:13:47 SpamapS: it's drop-your-db-and-start-again inconvenient 20:13:50 kebray it seems clear we had a developer consensus that hot was not prime time for havana 20:13:52 I think it is pretty standard in software development that if you introduce a breaking change, you migrate over the data that you are breaking 20:13:59 So there have been some code pieces that accepted "legacy" spelling, and we only kept samples in heat-templates in one format. What happened now was that one of those tolerant code pieces got removed. 20:14:02 kebray what do you suggest, supporting something we did not agree we would support? 20:14:04 sdake I'm not arguing that point. 20:14:05 zaneb: that's a bug, isn't it? 20:14:31 I would opt for removing all such pieces and force everyone to stick to the hot specification 20:14:43 sdake I'm suggesting that, when possible, we minimize the breaking change or provide a migration path. I think that can be done in this case, no? 20:14:49 SpamapS: well, you have your template stored in the database. it was valid at the time you created the stack, but now it's not, so Heat can't load it 20:15:13 kebray if an exception is made once, it sets a precedence that an exception can be made later 20:15:21 this is the problem with standards in general 20:15:30 when we say we are going to support forward migration, we better mean it :) 20:15:41 we said no such thing previously 20:15:43 as much as I wish we could ignore this problem, I think kebray is right 20:15:45 this would be the first migration which needs to load the template data-structure, change something, and save it again. If someone did the tooling to make that easy then that sounds like a useful thing anyway 20:15:47 zaneb: right, that seems like a bug in Heat. 20:15:52 so let's talk about possible solutions 20:16:03 1) add a DB migration to fix up existing templates 20:16:10 hi 20:16:39 2) continue accepting broken templates up until a certain date. If people don't update by then, they're hosed 20:17:01 +1 on 1) if somebody is prepared to write the migration 20:17:13 3) bump the version date of HOT. only accept the cfn stuff for the older version 20:17:14 2 will require people to migrate their database anyway, so I vote for #1 20:17:41 we'll need a volunteer to sign up for this one though 20:17:44 I volunteer for the migration 20:17:45 +1 on (1) as well. 20:17:57 andrew_plunk: nice one 20:18:01 so just that I understand the migration 20:18:20 the idea is the hot spec will still be "string" but the database will be modified on first-run of heat-engine to change String to "string" ? 20:18:20 make me think we should add a template validaton job to heat-templates jenkins gate 20:18:37 sdake: we go through the database, and fix any existing HOT templates in there 20:18:39 sdake: it will just be a migration script, like the others 20:18:41 1++ plus in 2, people uploading broken templates might complain 20:18:49 correct sdake: I think we should check for other types too 20:19:01 sdake: yes, but as stevebaker said 20:19:04 cool that works for me - in that case no exception is made 20:19:25 we are just fixing a bug via an extra tool rather then providing exceptions to the standard 20:19:46 andrew_plunk: yeah, just convert all type: values to lower case 20:19:48 sdake cool.. works for me too. I'm happy. 20:19:55 sounds good zaneb 20:19:58 #topic Rackspace authentication is broken 20:20:02 #link https://bugs.launchpad.net/heat/+bug/1274201 20:20:16 There some more of those issues in the current HOT code. E.g. in a resource you can use 'type' or 'Type', or 'properties' or 'Properties'. We should fix this also ASAP 20:20:31 tspatzier: ++ 20:20:34 So, I added some comments to the bug, jasond` can you summarize your position? 20:20:38 I suppose this also means standalone heat on havana openstack is also broken? (or is keystone v3 complete in havana?) 20:20:48 do you have a migration path towards v3 keystone (or something compatible?) 20:21:03 so, rackspace uses v2 of its keystone-like API. we just need a way to support v2 along with v3 20:21:19 shim? 20:21:22 stevebaker: only until the keystone bugs I fixed get backported to stable/havana 20:21:24 zaneb: I can open a bug and do this change. Should be easy. This would be another thing that the migration script would fix then, right? 20:21:32 i don't know when rackspace might upgrade to v3 20:21:50 Havana has v3 keystone, it's just the RAX $thing_not_keystone doesn't 20:21:52 shardy: OK 20:22:00 stevebaker Rackspace does have a plan to support v3... but, it's gonna be a bit. 20:22:20 kebray: keystone are deprecating v2 for Juno 20:22:26 shardy: (it's called the identity service) 20:22:33 so we need to migrate to v3 regardless, and we need the v3 features now 20:22:43 shardy yep.. understood.. but, that doesn't mean that large public clouds make the switch the day Juno code is ready. 20:22:52 tspatzier: ideally the migration should happen in the same patch, so maybe it is better in a separate script. Copying parts of the first script should make it easy (if not efficient) 20:23:17 probably worth making keystone pluggable. 20:23:22 kebray: but most public clouds will upgrade their entire cloud, no, e.g Icehouse Heat and Icehoue keystone? 20:23:34 shardy I think that is not always the case unfortunately 20:23:44 In phases sometimes. 20:23:45 +1 SpamapS 20:23:51 zaneb: ok, I guess I'll synch up with andrew_plunk on the first migration script he will be writing 20:23:52 shardy some faster moving projects will update more often like neutron and heat 20:23:54 kebray: I think this is partly the tension of the heat which is fully integrated with the current development cycle, and the heat which is deployed on older/different clouds 20:24:02 sounds good tspatzier 20:24:02 shardy don't we all wish things were that easy :-) 20:24:10 thing is, I think we can work with the keystone v3 in havana, right? 20:24:14 sdake: well, the question is how many OpenStack releases back should we be targeting support for? 20:24:34 zaneb good question, I believe long ago we said 1 release back 20:24:40 sdake: we test zero releases back, so there is a clue 20:24:47 kebray: well we do have some compatibility with older keystone, but we do need v3 auth 20:24:55 that maybe we will have a bad time 20:24:57 if it is not tested, it is broken ;) 20:24:58 so as long as Icehouse Heat works with havana keystone, we've covered the most common pure-OpenStack upgrade scenarios. 20:25:08 zaneb: grenade? 20:25:25 SpamapS: no thanks 20:25:26 is keystone v2 still supported when icehouse is cut, yes? 20:25:34 kebray: yes 20:25:34 SpamapS: I just had one 20:25:39 SpamapS: Havana keystone will work, but there are a couple of pending bugfix backports (mentioned in my commit messages) 20:25:43 SpamapS: doesn't grenade that only test upgrades? 20:25:44 keystone v2 is deprecated in icehouse iirc 20:25:52 so, why would we force use of v3 in icehouse? 20:26:04 sdake ok.. I think it's important we sort that out. 20:26:08 stevebaker: yeah I think it upgrades everything 20:26:17 shardy can you confirm v2 is deprecated in icehouse? 20:26:18 kebray: because the features we need don't exist in v2 keystone 20:26:27 kebray: because v2 doesn't do everything Heat needs. 20:26:28 (the api in keystone) 20:26:34 sdake: Yes, there are lots of noisy warnings in the log to that effect 20:27:08 so if someone wrote a v2 shim, should it live in contrib or heat? 20:27:18 shardy: you said deprecated in Juno... did you mean removed in Juno? 20:27:24 clients are supposed to be forward compatible, so I guess the issue comes down to we are using features which are not in the v2 api which are in the v3 api 20:27:26 I don't think we should just tell v2-only alternative keystone backends to stuff it. I do think the onus is on those users to keep that support alive. 20:27:35 shardy SpamapS right.. so, new feature needs v3... but, that feature shouldn't be a requirement to run Heat or for Heat to work until v2 is depricated. 20:28:10 deprecated means it will be removed soon typically 20:28:51 ok, so lets focus on the actual problem, which I think comes down to programming time left before i3 20:29:03 shardy is there time to add v2 and v3 support to icehosue heat? 20:29:05 2014-01-29 20:28:41.793 2230 WARNING keystone.openstack.common.versionutils [-] Deprecated: v2 API is deprecated as of Icehouse in favor of v3 API and may be removed in K. 20:29:11 (if your the only guy doing the work) 20:29:22 eg, to make it backwards compatible 20:29:36 sdake: I've just been removing the hybrid v2/v3 support, because supporting both is a total mess 20:30:19 sdake: My proposal is to make the heat_keystoneclient wrapper pluggable so there could be a contrib v2 client 20:30:32 I get that it is architecturally ugly to have v2 and v3 in the same codebase 20:30:45 and I can make the domain features optional via the config file (defaulted to on), like trusts 20:30:50 so this theoretical v2 cient would model the v3 api then shardy? 20:31:27 sdake: exactly the same interfaces, but raise NotImplemented exceptions for all the trusts/domains stuff 20:31:46 personally I don't want to maintain it though, it's too much effort to test properly 20:32:07 kebray: It looks like we're angling for volunteers again ;) 20:32:12 so shardy just to be clear, I think your pretty swamped just getting v3 rolling and basically don't have bandwidth to deal with a v2 plugin 20:32:24 sdake: exactly 20:32:46 kebray I think there is a path forward, but as stevebaker points out, in need of a volunteer :) 20:32:51 If someone wants to independently maintain a v2 plugin, in contrib or /deprecated or something, that's fine 20:32:59 stevebaker not sure who exactly, but we have to stay on v2 for some period of time.. it's not an option. You don't just upgrade a large public cloud (all services) all at once over night :-) 20:33:21 but look at the heat_keystoneclient.py code in Havana and tell me we want a common client for both versions - we really don't 20:33:24 so, someone at Rackspace will pick it up if no one else does. 20:33:36 kebray: we need a v2 shim, but we *really* need icehouse heat to not require admin users to launch stacks 20:34:00 ya, admin for wait conditions really makes heat look sloppy 20:34:01 stevebaker: Note those to requirements are mutually exclusive 20:34:09 hopefully rax can sort that out in their further updates 20:34:14 agreed an admin user shouldn't be required to launch stacks... iirc, this revolves around admin API capabilities for doing a GET on all stacks. 20:34:19 kebray: doesn't the admin requirement make v2 unworkable for you anyway? 20:34:37 or does rax heat not support waitconditions? 20:35:14 kebray: How awesome is identity services btw? Like, does it actually order pizza for you, or does it just call an API that orders pizza for you? (trying to figure out why nobody at Rackspace is working on deploying _keystone_) 20:35:15 shardy the way the admin requirement landed in patches, you are correct that v2 is unworkable, because the patches landed as requiring v3 trusts (if I understand correctly) 20:35:28 kebray: The issue is heat creates keystone users for certain resources, and for Icehouse I want them to be in a separate heat specific domain, which requires v3 keystone 20:35:38 SpamapS hehe.. there is history.. but, I won't derail this meeting with it. 20:35:51 then heat can create the users, instead of the user (so they won't need admin) 20:36:26 shardy I think your approach is correct, and rax is just up against a specific business problem they need to solve 20:36:35 kebray: this has nothing to do with trusts 20:36:40 kebray: I heard some other large public OpenStack provider has a keystone work-alike too.. ;) 20:36:51 the answer isn't "find another way" the answer is "v2 plugin auth with notimplemented raises" 20:36:55 shardy yep, that's the best go-forward solution, agreed. But, in general I want to bring Heat (and HOT) to customers soon... but, HOT isn't ready for customers, which mean Heat isn't ready for customers. So, I want what I can't have from the beginning :-) 20:37:13 kebray: the admin requirement didn't land in patches, you've always needed admin to create a stack with certain types of resources in it 20:37:17 kebray: This is to do with WaitConditionHandle and ScalingPolicy resources (and User resources associated with ec2 credentials) 20:37:21 "Sometimes the wanting, is better than the having." -- George Clooney 20:37:38 zaneb ok.. I think I'm confusing my streams of terminology. 20:37:39 kebray: the _fix_ for that landed in patches, but requires keystone v3 20:37:40 i'd rather have then want personally :) 20:38:11 zaneb: The fix for that is still in the process of landing ;) 20:38:18 I dont think its worth beating up on kebray any more, keith do you agree that the solution outlined is workable on your end? 20:38:18 ok, lets move on 20:38:24 but the admin thing has nothing do do with trusts 20:38:41 trusts are about not storing the user password, and working with token-only auth 20:39:02 sdake I'll get my experts to explain it to me in lamen terms. We'll bring back to the mailing list if needed. 20:39:07 shardy: right, but both trusts and the admin fix require v3 keystone? 20:39:10 kebray sounds good 20:39:23 #topic Autoscaling 20:39:29 need more minerals 20:39:35 zaneb: Yes, both require v3 keystone, which is why I'm saying "lets just use that" 20:40:04 radix: therve just wanted to know the plan, but can't attend this meeting 20:40:07 ideally, yeah 20:40:19 I realize that it's difficult for Rax due to divergence from openstack auth, but from an upstream perpective, we really have to focus on what works with recent keystone 20:41:02 shardy I think everyone is in total agreement - kebray just has a specific business problem to sort out and there is a path to get there 20:41:16 I think joesavak was saying that auth v3 on RAX should be available sometime this year and v2 and v3 can be both exposed 20:41:21 AUto scaling.. what _is_ the plan exactly? 20:41:26 sdake: yup 20:41:28 sdake: is there though? 20:41:31 shardy, departure from keystone isn't the problem. We are just on keystone v2 instead of v3. 20:41:35 chmouel: maybe this will be... motivating 20:42:03 stevebaker: :) 20:42:07 zaneb my understanding of the proposal is to make a v3 api that only does v2 things and raises not implemented when v3 things are used 20:42:07 kebray: Ah, but it is keystone? 20:42:19 shardy it is a keystone compatible contract. 20:42:26 zaneb and rax will have to sort that problem out since we dont have their auth system in devstack :) 20:42:30 kebray: lol 20:43:23 sdake: we can do that, but e.g. rax will not be able to use, say, autoscaling because it requires the user to be admin 20:43:48 zaneb understood, autoscaling and waitconditions are out because of the limitations of rax auth 20:43:49 which, it turns out, is not some tenant-local role but gives you access to the whole world 20:43:59 sdake you do have keystone v2 in devstack, yes? that's the concern... not Rax auth, as if something works against keystone v2, it'll work at Rax. 20:44:31 kebray I see, but t he issue is heat is broken as is against the v2 api - that is why the v3 api was invented :) 20:44:32 so, Autoscale? 20:44:33 No openstack project promises indefinite backwards comaptibilty, that's the whole point of the coordinated release and stable brances 20:44:37 branches 20:45:24 kebray: not without admin, which is why I'm so keen to fix this problem :) 20:45:44 kebray: I think that the only reason keystone vs. rax auth matters is because if you were using keystone you'd probably have upgraded by now. I recognise that I'm probably preaching to the choir here ;) 20:46:00 shardy: so, if we have v3 in place and a v2 shim, in the near future, what would be broken for folks using the v2shim? 20:46:12 autoscaling and wait conditions 20:46:25 the same stuff that is broken currently 20:46:31 unless you're an admin user 20:46:43 andersonvom: Anything using SignalResponder resources (WaitConditions, AutoScaling) and User Resources which are used for cfn-hup and cfn-push-stats agents 20:47:34 #topic PTL is on leave, nobody notices 20:47:35 zaneb, even the folks that run keystone implementation have differing auth needs and extend it.. we are certainly not the only ones who are doing that! 20:48:08 I'm away for next week's Projects meeting, and Heat meeting. Can anybody volunteer to cover? 20:48:48 stevebaker: can do 20:49:45 zaneb: cool, thanks. There is a short 1-1 with ttx 90 minutes before the Projects meeting too, which takes place in #openstack-relmgr-office 20:50:07 ok 20:50:23 how much pain should I expect? ;) 20:50:45 ttx: zaneb will be your torture monkey next week for Heat 20:51:08 its not too bad 20:51:33 since its so early in i-3 we can maintain our delusion ;) 20:51:47 #topic Open discussion 20:51:49 tbh I think it's time for mass panic 20:52:21 I'd like to cut a heatclient release, but there are many pending reviews https://review.openstack.org/#/q/status:open+project:openstack/python-heatclient,n,z 20:52:51 +1 stevebaker! 20:52:52 8 minutes, anything else to raise? 20:53:16 Hi, I'm new, nice to be here 20:53:16 * stevebaker needs to try out the heatclient requests change 20:53:28 cmyster: hi 20:53:42 cmyster: welcome! 20:53:54 Note to anyone with broken master heatclient : https://review.openstack.org/#/c/69821/ 20:53:54 stevebaker: please :) 20:54:32 Question: Are we planning to call HOT stable when Icehouse is cut? 20:54:43 anywho, I'll start taking over (slowly) automation of, well, hopfully everything. but I have loads to learn still. 20:55:10 cmyster: welcome :) 20:55:11 kebray: yes 20:55:19 thanks again shardy 20:55:34 stevebaker: in that case, definitely time for panic :D 20:55:48 stevebaker what are the biggest risks (if any) to that happening? 20:56:01 besides zaneb panicking. 20:56:02 kebray: I mean it will continue to evolve during Juno, but we'll be caring *much* more about backwards compat 20:56:18 k. 20:56:31 stevebaker: I was just going to ask what stable means. I think you just answered. 20:57:05 kebray: biggest risk is that we can't get to plugin template formats, which will allow us to maintain multiple versions easily 20:57:12 we should spend feature freeze checking that it is actually possible to write real templates with HOT, and fix bugs as we find them 20:57:56 stevebaker: we could use a lot more examples in heat-templates 20:58:43 shardy: I'd prefer a template writing guide, but I've ranted enough about that. Maybe we'll have time to write one 20:59:28 stevebaker: there already is one. I guess we'll have to extend it based on new features like software orchestration, get_file etc. 20:59:47 tspatzier: link please ? 20:59:56 ... but samples are still the thing most people start with 21:00:13 tspatzier: yes, I mean extending the current guide with many recipe-style mini tutorials on how to achieve specific things with the best HOT style 21:00:26 cmyster: http://docs.openstack.org/developer/heat/template_guide/ 21:00:29 #link http://docs.openstack.org/developer/heat/template_guide/hot_guide.html 21:00:29 #endmeeting