17:59:51 <stevemar> #startmeeting keystone 17:59:52 <openstack> Meeting started Tue Sep 6 17:59:51 2016 UTC and is due to finish in 60 minutes. The chair is stevemar. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:59:53 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:59:55 <openstack> The meeting name has been set to 'keystone' 17:59:59 <samueldmq> hi all 18:00:09 <jaugustine> o/ 18:00:25 <dolphm> o/ 18:00:26 <NishaYadav> o/ 18:00:32 <shaleh> \o 18:00:46 <gagehugo> hey 18:01:04 <stevemar> #agenda: https://etherpad.openstack.org/p/keystone-weekly-meeting 18:01:08 <stevemar> o/ 18:01:09 <browne> o/ 18:01:15 <bknudson> hi 18:01:27 <ayoung> Holla 18:01:43 <henrynash> hi 18:01:43 <stevemar> ayoung: i moved your thing to the end of the agenda OK 18:01:59 <ayoung> stevemar, not really, no. 18:02:02 <topol> o/ 18:02:15 <stevemar> ayoung: we will get there by 2:05 :) 18:02:16 <ayoung> We break the other projects, it is kindof like the worst thing to do 18:02:52 <stevemar> just push the pause button for 5 minutes 18:03:00 <stevemar> #topic Announcements 18:03:06 <stevemar> rderose is now core :) 18:03:13 <raildo> rderose, congrats :) 18:03:14 <henrynash> yeah! 18:03:15 <lbragstad> ++ 18:03:16 <knikolla> congrats! 18:03:17 <stevemar> thanks rderose for your hard work 18:03:23 <bknudson> nice 18:03:23 <rderose> yeah! thanks guys :) 18:03:24 <gagehugo> congrats! 18:03:29 <gyee> rderose, u da man! 18:03:37 <rderose> gyee: :) 18:03:43 <henrynash> another rabbit bites the dust? 18:03:48 <henrynash> just kiddin 18:03:50 <rderose> hahaha 18:03:58 <stevemar> there are many other great contributors helping make keystone awesome, you're all great in my eyes 18:04:08 <henrynash> stevemar:++ 18:04:19 <ayoung> We just lost another productive community member to the burden of code reviews. 18:04:20 <stevemar> rderose: thanks for all your coding and reviewing 18:04:38 <rderose> stevemar: thanks man, really appreciate all of your support 18:04:43 <dolphm> ayoung: ++ 18:04:45 <gyee> ayoung, then try to write less code damn it! 18:05:00 <stevemar> #topic Release status 18:05:36 <stevemar> we're in the RC period, so only critical bug fixes should be merging now 18:05:45 <stevemar> docs and tests are always OK to merge 18:06:09 <stevemar> please go and play with the latest master level and find bugs :) 18:06:17 <stevemar> especially in the PCI support :D 18:06:27 <ayoung> https://bugs.launchpad.net/keystone/+bug/1619758 should be on that list 18:06:27 <openstack> Launchpad bug 1619758 in OpenStack Identity (keystone) "Credential Encryption breaks deployments without Fernet" [Undecided,In progress] - Assigned to Adam Young (ayoung) 18:06:27 <dolphm> and rolling upgrades :) 18:06:39 <stevemar> dolphm: yes that too! 18:06:50 <stevemar> dolphm: have prometheanfire test it for postgres :S 18:07:05 <ayoung> where is the bug list? 18:07:23 <stevemar> ayoung: the only one targeted for rc1 is yours 18:07:48 <stevemar> RC bugs will be tracked here: https://bugs.launchpad.net/keystone/+milestone/newton-rc1 18:07:49 <dolphm> #link https://launchpad.net/keystone/+milestone/newton-rc1 18:08:13 <ayoung> dolphm, thanks 18:08:17 <stevemar> i encourage everyone to go and triage some of the open keystone bugs 18:08:25 <stevemar> make sure we are not missing any 18:08:31 <stevemar> that should go into newton 18:08:55 <stevemar> alright, done all the organization stuff 18:09:05 <stevemar> #topic credential encryption breaking the world 18:09:14 <stevemar> some background on this 18:09:29 <stevemar> actually, someone else want to take this one? dolphm? it's a lot of typing 18:09:32 <stevemar> :P 18:09:36 <dolphm> ayoung: ? 18:09:42 <stevemar> we broke tripleo 18:09:48 <bknudson> did grenade break or did a change have to be made to devstack? 18:10:00 <lbragstad> a change was made to both devstack and grenade 18:10:02 <dolphm> we broke everyone that is not following our upgrade release notes and attempting to upgrade anyway 18:10:15 <stevemar> bknudson: we upgraded grenade https://github.com/openstack-dev/grenade/blob/master/projects/10_keystone/from-mitaka/upgrade-keystone 18:10:15 <ayoung> yeah, firedrill 18:10:45 <lbragstad> it fails because if there are credentials in the backend when the migration is run - it will attempt to encrypt them 18:10:49 <ayoung> So...distribution of keys is hard 18:10:53 <bknudson> do you need to do this even if you don't have any credentials? 18:10:58 <lbragstad> bknudson no 18:10:58 <stevemar> in an attempt to make keystone more secure, by encrypting credentials, we are *forcing* an extra upgrade step for mitaka -> newton, you must run `keystone-manage --credential_setup` 18:11:04 <henrynash> dolphm: so an offline migration fails? 18:11:04 <lbragstad> bknudson it's conditional if you have credentials 18:11:12 <ayoung> doing it right is pretty much beyond the scope of what Tripleo should be trying to do between now and release 18:11:19 <lbragstad> bknudson https://github.com/openstack/keystone/blob/b47f10290ed83415149f3d2ab6b0dc64646e578a/keystone/common/sql/data_migration_repo/versions/003_migrate_unencrypted_credentials.py#L26 18:11:29 * stevemar waves at EmilienM 18:11:33 <dolphm> lbragstad: the error in bug 1619758 is not super helpful, btw 18:11:33 <openstack> bug 1619758 in OpenStack Identity (keystone) "Credential Encryption breaks deployments without Fernet" [Undecided,In progress] https://launchpad.net/bugs/1619758 - Assigned to Adam Young (ayoung) 18:11:40 <dolphm> should fix that with a better message 18:11:40 <EmilienM> o/ 18:11:48 <ayoung> To use the Credentials backend now requires a setup of the keys to encrypt the credentials 18:12:04 <stevemar> ayoung: yes, as it should have been from day 1 18:12:04 <lbragstad> dolphm sure - we can catch that and report something more useful 18:12:05 <ayoung> and, this key needs to be in sync across all keystone servers that talk to the same database 18:12:30 <lbragstad> dolphm bug report? 18:12:32 <dolphm> stevemar: ++ we should never have been storing secrets in plaintext anywhere, but since we're already in that business, we need to correct our behavior ASAP 18:12:39 <dolphm> lbragstad: https://bugs.launchpad.net/keystone/+bug/1619758 18:12:39 <openstack> Launchpad bug 1619758 in OpenStack Identity (keystone) "Credential Encryption breaks deployments without Fernet" [Undecided,In progress] - Assigned to Adam Young (ayoung) 18:12:40 <stevemar> lbragstad: https://bugs.launchpad.net/keystone/+bug/1619758 18:12:41 <ayoung> dolphm, that is what happens when you record bugs after close of business on a Friday evening 18:13:07 <lbragstad> dolphm want me to just tack the fix on to that? 18:13:12 <stevemar> dolphm: what are the OSA guys doing? 18:13:27 <dolphm> ayoung: i'm referring to needing to encrypt things, and not let deployers continue to store secrets in plain text using keystone 18:13:38 <stevemar> lbragstad: you can make the error message improved related to the bug, not close it though 18:13:42 <dolphm> stevemar: running credential_setup ? 18:13:43 <ayoung> EmilienM, check me on this, the breakage was due to Tempest run, rigjht? 18:13:47 <lbragstad> stevemar yep 18:13:49 <lbragstad> stevemar I can do that 18:13:55 <lbragstad> stevemar i'll leave a comment 18:13:59 <stevemar> lbragstad: thx 18:14:15 <lbragstad> stevemar andymccr is working on it for OSA 18:14:24 <lbragstad> i already had a conversation with him about it this morning 18:14:32 <EmilienM> ayoung: tempest could not validate keystone without credentials 18:14:35 <dolphm> stevemar: OSA is not broken because they're not attempting to upgrade to master or newton-rc1 yet 18:14:41 <stevemar> ayoung: yes, so we had to decide... if someone is not encrypting their credentials, do we break them on upgrade, or when they are using the API 18:14:49 <dolphm> stevemar: at least, not in their gate or anything 18:14:53 <ayoung> stevemar, the answer is "Neither" 18:14:55 <ayoung> but... 18:15:08 <lbragstad> dolphm andymccr was hitting some issues on some tests he is working on locally 18:15:41 <ayoung> So, lets get out of the way the real sin is in storing and retrieving credentials, and encrypting them is a bandaid (albeit necessarY) on a sucking chest wound. 18:15:52 <ayoung> Yes, unencrypted is worse 18:16:24 <ayoung> so, we should have no-opped this from the beginning, and given people at least one dev cycle to catch up 18:16:30 <dolphm> ayoung's patch provides a no-op credential provider, that basically doesn't require any configuration, and he's proposing it as the default. that would let operators switch to the fernet-based credential encryption provider later, but its probably broken by rolling upgrades now (?) 18:16:30 <ayoung> the dropped the no-op 18:17:07 <ayoung> now...I would be happy with having no-op as an option, but not the default option, as that would only require a single config change 18:17:14 <ayoung> we can live with that if necessary 18:17:17 <bknudson> can we add a warning to say to switch it? 18:17:23 <ayoung> but, that is just me being selfish 18:17:33 <dolphm> bknudson: i'd suggest marking the no-op as deprecated immediately to do exactly that 18:17:46 <lbragstad> the no-op thing will only work if there aren't any credentials stored in the backend 18:17:46 <ayoung> dolphm, that makes sense 18:17:52 <lbragstad> when the upgrade takes place 18:18:02 <samueldmq> dolphm: ++ 18:18:06 <bknudson> I like the deprecated no-op as a solution. 18:18:22 <stevemar> bknudson: but will the noop driver work? 18:18:31 <ayoung> passes unit tests 18:18:35 <stevemar> i think lbragstad has doubts 18:18:39 <bknudson> nobody knows if anything works. 18:18:46 <ayoung> apevev said it failed due to a different problem 18:18:47 <dolphm> ayoung: the rolling upgrade process migrates plaintext data in the database to be encrypted -- that's not something operators can postpone 18:18:49 <ayoung> something about entrypoints 18:18:59 <lbragstad> it won't work if someone has credentials stored in plaintext and they upgrade 18:19:11 <dolphm> lbragstad: ++ 18:19:15 <stevemar> right, the upgrade will fail 18:19:18 <lbragstad> if they do that and then switch to the noop provider they will be return cipher text to the end user 18:19:20 <ayoung> lbragstad, won't that migration fail if there is no key? 18:19:39 <dolphm> ayoung: yes, you can't upgrade without running credential_setup first 18:20:21 <bknudson> are there grenade tests with credentials? 18:20:34 <dolphm> "In order to upgrade successfully to Newton, deployers must encrypt all credentials currently stored before contracting the database. Deployers must run keystone-manage credential_setup in order to use the credential API within Newton, or finish the upgrade from Mitaka to Newton. This will result in a service outage for the credential API where credentials will be read-only for the duration of the upgrade process. Once 18:20:34 <dolphm> the database is contracted credentials will be writeable again. Database contraction phases only apply to rolling upgrades." http://docs.openstack.org/releasenotes/keystone/unreleased.html#upgrade-notes 18:20:41 <ayoung> so noop really is not going to help, then, is it 18:20:55 <lbragstad> bknudson now - but after the upgrade grenade will exercise the credential api 18:20:58 <lbragstad> bknudson no* 18:21:02 <dolphm> ayoung: i was hoping it would, but lbragstad is right -- it's only useful if you don't have any credentials anyway 18:21:29 <lbragstad> bknudson the credential api was failing to do stuff because it didn't have any keys to encrypt with 18:22:09 <ayoung> dolphm, so I don't think there are any easy answers here. 18:22:31 <stevemar> ayoung: are tripleo users upgrading? 18:22:49 <dolphm> ayoung: i'm afraid to ask, but what's the complexity blocking tripleo from dropping fernet keys on disk prior to upgrading? 18:22:51 <ayoung> stevemar, they will, so we need to cover that, too 18:23:08 <ayoung> dolphm, good question, here it goes: 18:23:26 <dolphm> ayoung: there's no reason to *require* credential_setup or any syncing if you can distribute them any other way 18:23:32 <dolphm> them = fernet keys 18:23:36 <ayoung> Tripleo does not have an easy way of distributing the credentials in a way that is not world readable 18:23:44 <stevemar> dolphm: i think EmilienM is looking into that https://bugs.launchpad.net/keystone/+bug/1619758/comments/3 18:23:44 <openstack> Launchpad bug 1619758 in OpenStack Identity (keystone) "Credential Encryption breaks deployments without Fernet" [Undecided,In progress] - Assigned to Adam Young (ayoung) 18:23:50 <ayoung> if it were just a single entry, then hiera can hide it from the world 18:23:58 <dolphm> ayoung: how are service user passwords and things distributed? or database passwords? 18:24:10 <ayoung> but the fact that it is a directory can only, at the moment, make use of a crude mechanism 18:24:13 <EmilienM> yes I'm currently working on tripleoclient to generate the keys for managing keystone credentials 18:24:25 <EmilienM> so we can use puppet to put the files on keystone servers 18:24:34 <ayoung> IN general, Tripleo is Heat generating Hiera which is used to configure puppet 18:24:39 <stevemar> dolphm: the complexity is unknown :( 18:24:45 <ayoung> so if the puppet modules can't do soemthing, we have limited tools available 18:24:59 <ayoung> which means poor EmilienM is responsible for making everythign work 18:25:11 <antipsychiatry> Destroy " Israel" !!!!!!! They are antichrist !!!!!!!!!!!!!!!!!!! They are synagogue of Satan!!!!!!!!!! 18:25:57 <ayoung> the keystone tooling assumes it is executed on the keystone node, but we need to generate the keys on the undercloud and then deploy them in sync to all the other nodes. 18:26:38 <ayoung> hence my Proof of concept using the undercloud's keystone to generate the Repo for the overcloud. 18:26:48 <stevemar> thanks infra <3 18:27:13 <fungi> (any time) 18:27:19 <dolphm> ayoung: you should be able to generate them on either cloud, or outside the cloud, doesn't matter. it's a single line of python to generate a key 18:27:26 <lbragstad> ayoung if you do use the keystone tooling, is there a way to sync them after they are generated? 18:27:36 <lbragstad> like how osa does it with fernet keys? 18:27:49 <ayoung> So, changes to fix tripleo usually require 1) Puppet changes and 2)triple-heat-template changes. In this case, I think a third change to generate the Keys might also be required 18:27:54 <dolphm> i'm pretty sure mfisch has public puppet code out there to illustrate fernet key configuration and distribution without using keystone-manage at all 18:28:10 <lbragstad> #link https://github.com/pyca/cryptography/blob/master/src/cryptography/fernet.py#L46 18:28:29 <ayoung> lbragstad, not a good way. 18:28:33 <dolphm> lbragstad: don't even need to install cryptography! 18:28:43 <ayoung> Possibly sufficient to get through gate, but not one I would support live 18:28:45 <lbragstad> nope just need standard lib stuff 18:28:52 <ayoung> yeah, that is the easy part 18:28:58 <lbragstad> i believe EmilienM was working on something using that 18:29:10 <lbragstad> generating the keys and storing them in puppet somewhere? 18:29:35 <stevemar> while EmilienM hacks on his PoC, are there any other avenues we should look at ? 18:29:51 <stevemar> revert the credential encryption? :) 18:30:03 <ayoung> its not an impossible task, just one we should not expect the installers to deal with post Milestone 3 18:30:28 <stevemar> i agree the timing is terrible, which is why i put the revert on the table 18:30:38 <ayoung> stevemar, so, I'd be OK with a No op solution 18:31:22 <stevemar> what about a noop and a fernet, but another command to actually do the encrypting of all credentials? 18:31:39 <ayoung> lbragstad, what if we skip the encryption until the no-op provider is disabled, and do it as an offline operation 18:31:43 <dolphm> stevemar: the problem there is that a database migration is required 18:31:45 <ayoung> stevemar, that 18:31:57 <dolphm> stevemar: and we can't do database migrations outside of the upgrade process 18:32:08 <dolphm> it's not a standalone, at-will operation like credential rotation is 18:32:12 <dolphm> or fernet key rotation 18:32:17 <lbragstad> right 18:32:34 <bknudson> how about don't encrypt/decrypt if there are no keys? 18:32:43 <ayoung> dolphm, so, if the no-op provider is enabled, skip the encryption. If the no-op provider is enabled, and encryption has not happened, report an error 18:32:49 <stevemar> bknudson: oh, like at all? 18:33:08 <stevemar> lbragstad: dolphm is that possible? 18:33:15 <ayoung> do we drop the old column? 18:33:18 <bknudson> right, currently it's an error if there are no keys, so the proposal would be to allow no keys to work (don't decrypt/encrypt) 18:33:25 <lbragstad> ayoung yes 18:33:36 <dolphm> bknudson: that'd require either rewriting our migration history or introducing a new migration after the one that ayoung is struggling with to re-introduce a plaintext column or something 18:33:43 <dolphm> ayoung: yes 18:33:51 <ayoung> ok...what if we put the keys in the database or have a default key that gets removed later? 18:34:00 <ayoung> encrypt, but poorly 18:34:04 <dolphm> keys in the database defeats everything 18:34:08 <ayoung> agreed 18:34:15 <ayoung> and a default key in the config file does, too 18:34:21 <dolphm> ayoung: hardcode your upgrade to use a null key 18:34:27 <dolphm> ayoung: then credential rotate later 18:34:30 <dolphm> #solved 18:34:33 <Anticimex> (as a deployer, hashicorp vault seems better than puppet / puppet-db for distributing secrets - or similar) 18:34:33 <ayoung> but...adding a key to the config file that we generate is probably easier than managing the repo... 18:35:10 <breton> the situation with requiring keys is also bad because the keys will be in the backend in the next cycle (i hope) 18:35:22 <ayoung> Anticimex, we are not even using the puppet mechanism here 18:35:45 <EmilienM> this is the link of my PoC https://review.openstack.org/366287 (it's only a start and unit tests will fail but save the URL) 18:36:27 * lbragstad starred 18:36:50 <dolphm> ayoung: EmilienM: seriously, hardcode tripleo to drop two credentials keys on disk that are just a bunch of nulls 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=' 18:37:03 <EmilienM> I also need to patch puppet-keystone and tripleo but I'll do it after my terrible dentist apt :( 18:37:14 <dolphm> ayoung: EmilienM: finish the upgrade, and then at your leisure when running newton, you can rotate new keys in with credential_setup 18:37:20 <ayoung> dolphm, that is essentially what EmilienM is saying, only actually generating a key instead. Its the file managment that is the fireddrill here 18:37:26 <dolphm> or in a future release when you solve the syncing problem 18:37:36 <ayoung> but if Tripleo has to do it, everyone will have to, and keystone will have broken the world 18:37:54 <dolphm> ayoung: i assume the complexity in the file management is in coordinating the contents of the files? 18:38:00 <bknudson> it would be great if tripleo was reporting CI failures 18:38:02 <dolphm> not in writing a file with a known value 18:38:04 <lbragstad> i imagine osa is going to use the same mechanism that it uses for fernet rotation 18:38:22 <ayoung> dolphm, probably more important is to identify the existence of certain files and get them to the servers. 18:38:35 <ayoung> the contents are always in sync across all nodes 18:38:57 <dolphm> ayoung: /etc/keystone/credentials-keys/0 should always contain AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= 18:39:00 <dolphm> ayoung: /etc/keystone/credentials-keys/1 should always contain AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= 18:39:00 <bknudson> how do you re-encrypt the credentials with the new keys? 18:39:14 <dolphm> bknudson: keystone-manage credential_rotate does the db work to do that 18:39:36 <breton> the requirement to have the keys before keystone starts bugs me 18:40:00 <dolphm> bknudson: or keystone-manage credential_migrate? (help, lbragstad) 18:40:08 <breton> i already tried to address it in https://review.openstack.org/362785 with low-priority 18:40:50 <lbragstad> bknudson keystone-manage credential_migrate assumes you have keys on disk in order to re-encrypt your credentials 18:41:15 <lbragstad> bknudson https://gist.github.com/lbragstad/ddfb10f9f9048414d1f781ba006e95d1#encrypted-credential-key-management 18:41:18 <dolphm> breton: that's only for the fernet token provider? 18:41:25 <lbragstad> explained in more detail there ^ 18:41:26 <EmilienM> ayoung: sorry I have to leave now, I'm back in 1h30 18:41:27 <dolphm> breton: not for the fernet credential provider 18:41:55 <breton> dolphm: yes. But i guess it would have to be done for credential provider when the keys move to backend. 18:42:27 <bknudson> lbragstad: neat 18:43:44 <stevemar> bknudson: will your time be able to handle credential encryption? 18:43:51 <ayoung> so we could run the encrypt with a null key 18:43:56 <stevemar> bknudson: i assume you guys are using fernet tokens now anyway 18:44:02 <ayoung> er 18:44:04 <ayoung> upgrade 18:44:13 <bknudson> stevemar: arrrsula supports fernet tokens. 18:44:23 <bknudson> not sure what's going to happen with this change. 18:44:34 <stevemar> bknudson: i'm assuming you'll use the same mechanisms then 18:44:39 <bknudson> nobody has reported any failures 18:44:40 <dolphm> ayoung: does that make sense? 18:45:10 <dolphm> ayoung: it lets you postpone the hard part, and we can document it as a minimum viable upgrade solution 18:45:15 <dolphm> (in keystone) 18:45:24 <ayoung> dolphm, I think you misunderstand what I am saying 18:45:38 <ayoung> we could have keystone operate using the null key until one is provided 18:45:51 <ayoung> if the key database is not populated 18:46:23 <ayoung> and deprecate that, too 18:46:45 <ayoung> dolphm, we just need to give people breathing room, not maintain the no-change upgrade forever 18:47:00 <lbragstad> why not just have tripleo drop two null keys on disk as /etc/keystone/credential-keys/0 and /etc/keystone/credential-keys/1 ? 18:47:06 <dolphm> i'm a little lost on how why supporting insecure upgrades needs to be keystone's responsibility 18:47:07 <ayoung> If we had landed this first thing in the cycle, I would be right there with you 18:47:22 <ayoung> it is purely the timing that leads me to want to compromise here 18:48:02 <breton> lbragstad: what would be the contents of the null keys? AAA...=? 18:48:22 <dolphm> breton: literally AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= 18:48:28 <lbragstad> breton yeah 18:48:29 <dolphm> breton: it's the null byte base64 encoded 18:48:38 <dolphm> well, several null bytes :) 18:48:39 <ayoung> The insecurity was Keystone's in the first place. A Credential Database is a scary thing, and it should never have existed, certainly not the way it was implemented. The encryption of credentials with a single key is still medicocre at best. 18:48:43 <lbragstad> or if there is a syncing mechanism in place they could be randomly generated 18:49:26 <breton> dolphm: lbragstad: :( why should tripleo know such implementation details about keystone? 18:49:31 <ayoung> It is not just Tripleo. It is all of the installers that are going to be messed up buy this. Tripleo was just the one that reported it 18:50:02 <bknudson> we already knew about it because of the grenade change 18:50:32 <lbragstad> breton i thought we considered encryption keys as configuration 18:50:43 <dolphm> breton: this is something we can document "if you'd like to opt-out of securely storing credentials in your database, then put these two well-known keys in these two files on disk to operate insecurely" 18:51:49 <ayoung> dolphm, I know it seems simple, but until you've worked through the mechanisms, it is a change that has significant impact 18:52:14 <ayoung> "here are two file that have cryptographic info in them, but use defaults..." 18:52:18 <dolphm> yeah, it seems like this is an option that could have been implemented before this meeting was over 18:52:45 <ayoung> what I did in my POC for Fernet tokens would also work for the credentials. it just would be world readable 18:53:26 <ayoung> We broke it, we own it. 18:54:02 <dolphm> ayoung: if you're implying that keystone broke tripleo, i don't think that's fair to say at all 18:54:18 <ayoung> dolphm, I am saying keystone broken Tripleo. 18:54:36 <dolphm> tripleo broke itself by attempting upgrades entirely blindly. we write release notes, documentation, examples, configuration docs, etc, for a reason 18:54:38 <ayoung> and doing so is OKish early in the development cycle 18:54:59 <ayoung> dolphm, Keystone broke every installer out there that uses this stuff 18:55:22 <ayoung> We essentially deprecated no-op with no lead time and no transition time 18:55:36 <ayoung> The end state, with encryption, is probably worth it 18:55:49 <dolphm> i've literally never heard of anyone attempting to upgrade without at least reading release notes 18:56:05 <ayoung> dolphm, it wasn't an upgrade 18:56:07 <ayoung> it was an install 18:56:18 <ayoung> that install used the credentials basckend 18:56:25 <stevemar> the install is fine, it's trying to use the credentials backend that failed 18:56:31 <ayoung> and that fails without the keys in place 18:56:33 <stevemar> it's tempest that fails 18:56:47 <stevemar> its the tests *after* install 18:57:17 <ayoung> As I said, it is probably the right thing to do, in the absolute sense. Just that the timing sucks 18:57:19 <dolphm> ayoung: an installation of master? 18:57:32 <ayoung> dolphm, yep 18:57:39 <bknudson> IBM public cloud process is going to automate upgrading. We're not going to have people reading release notes on every change. 18:57:46 <ayoung> ++ 18:58:03 <bknudson> at least that's the plan 18:58:06 <breton> why... do we write them then 18:58:20 <dolphm> bknudson: i'm all for automating it, but expect it to fail and to require maintenance to keep it working when changes to your moving target require additional steps 18:58:22 <breton> git rm -fr release_notes 18:58:28 <stevemar> breton: so when upgrades fail, we can point bugs to them 18:58:59 <lbragstad> some folks rely on them for their upgrade process 18:59:01 <bknudson> dolphm: yes, we'll have tests in place and do expect to do maintenance when the tests fail. 18:59:13 <stevemar> for most deployments, it'll be a one-time step 18:59:23 <breton> 1 minute left 18:59:25 <stevemar> that's the point of the scripts in grenade 18:59:44 <stevemar> tripleo is unfortunately, not a routine deployment of openstack 19:00:04 <stevemar> otherwise it would be a 1 line command 19:00:09 <stevemar> #endmeeting