17:58:09 #startmeeting trove 17:58:09 Meeting started Wed Jul 8 17:58:09 2015 UTC and is due to finish in 60 minutes. The chair is SlickNik. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:58:10 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:58:12 The meeting name has been set to 'trove' 17:58:43 It's still a couple of minutes to the hour, so giving folks a few more minutes to trickle in 17:59:15 The agenda for the meeting is at: 17:59:23 o/ 17:59:24 #link https://wiki.openstack.org/wiki/Meetings/TroveMeeting 17:59:29 heyo 17:59:38 o/ 18:00:09 o? 18:00:29 lolz thats a head scratcher 18:01:11 hah, okay — let's get started. 18:01:25 #topic Trove pulse update 18:01:31 #link https://etherpad.openstack.org/p/trove-pulse-update 18:01:32 ol 18:01:51 o/ 18:02:05 hello everyone 18:02:46 Fewer reviews this week (sort of expected given the long weekend, and holiday in Canada). 18:03:19 Thanks for reviews for those who were able to take the time to do that. 18:03:36 Is the trove pulse update not loading for anyone else or is it just me? 18:03:47 worked for me 18:03:52 i am able to see that 18:03:53 But I think we need to step them up to keep the momentum going. 18:03:54 just stuck on loading screen for me 18:03:58 working for me. 18:04:01 edmondk: try in a different browser 18:04:16 edmondk: Ctrl + Shift + R 18:04:21 safari worked 18:04:26 might need to clear the caches for chrome 18:04:48 (or command + shift + R on a mac) 18:05:04 i had a similar problem and deleted the user/chrome/cache dir and it started worked 18:05:08 working* 18:05:18 all good we can continue :) 18:05:22 Okay, getting back on topic :P 18:05:24 works for me 18:05:59 Let's do more reviews this week to ensure that we're not falling behind with our momentum. 18:06:35 holiday is over 18:06:41 ./ 18:06:44 And keep an eye out for that review count to ensure it's closer to the average next week. :) 18:07:38 Okay, that's all I had on this — any other comments / concerns regarding this? 18:07:43 vacations might have taken its toll on reviews 18:08:00 it shud be better next week, i guess 18:08:05 sushilkm: ++ 18:08:45 okay, let's move on. 18:08:52 #topic Add root-disable api 18:09:06 may I paste-bomb? 18:09:12 #link https://review.openstack.org/#/c/189837/ 18:09:29 amrith: sure 18:09:33 The issue with 'root-enabled' (the flag) is that in reality it means, "a root user was enabled on this instance at some time, or on some instance that was used to generate a backup and then subsequently launch this instance". 18:09:33 Today, without the root-disable feature, one can follow these steps. 18:09:33 1. create an instance 18:09:33 2. enable root 18:09:33 3. login as root 18:09:35 4. create a new user and grant that user all privileges 18:09:37 5. delete the root user 18:09:39 6. backup the instance 18:09:41 7. launch a new instance 18:09:43 The new instance would indicate root-enabled is false. 18:09:45 A peculiarity in how ACL's are controlled on MySQL further means that I can ... 18:09:47 1. create an instance 18:09:49 2. enable root 18:09:51 3. login as root 18:09:53 4. create a new user and grant that user all privileges 18:09:57 5. do some nefarious thing as either new user or root user 18:10:00 6. login as the new user (session A) 18:10:01 7. login as the root user (session B) 18:10:03 8. on session A, drop root user 18:10:05 9. backup the database 18:10:07 What we need (I think) is a mechanism that will attach the root-enabled status on an instance to the backup and restore that state when a backup is restored. After posting comments in the review, I was writing a BP to propose this and was going to submit code once the BP (approach) was agreed to. But I got scooped ;( 18:11:04 FWIW, I believe that the fix in https://review.openstack.org/#/c/199635/ doesn't solve the issue. 18:11:59 * SlickNik reads paste bomb 18:12:04 lol 18:12:14 so my first thought is that we know what instance the backup came from and should be able to tell if that instance had root enabled in the past and mark the new instance with root enabled. 18:12:37 cp16net, can't be sure that the instance still exists 18:12:38 that sounds like it would resolve the issue right? 18:12:54 but we keep track of that in the database right? 18:12:56 I could extend this to say take a backup, delete the old instance, ... 18:13:17 no guarantee that we will keep track of deleted instances forever. 18:13:24 that's an admin cleanup thing 18:13:39 we should (I believe) persist the root enabled state into the backup 18:13:46 or as you say store it in the database 18:13:48 amrith / craig: that's a possibility. Another option would be to mark this in the metadata, so that we can set the root-enabled flag when we do the restore based on the backup metadata. 18:13:48 with the backup 18:13:48 we'd need to store the root-enabled in the backup table as well 18:13:55 so extend the schema for the backup list 18:13:58 and store root enabled there 18:14:05 peterstac ++ 18:14:32 amrith: backups already have a metadata field in the table, if I'm recalling correctly. 18:14:38 SlickNik, yes, that would work too, as long as the user can't muck with the swift meta-data 18:14:58 FWIW, the reason I suggest that https://review.openstack.org/#/c/199635/ does little is that we can extend my second repro to have session A delete root user and session B delete the new user. 18:15:16 peterstac, HUH? 18:15:53 you are suggesting that the one doing trove create is also going to mess with the metadata? 18:16:03 well, we can't prevent against that. 18:16:25 we have a root_enabled_history table for every instance that root was enabled on 18:16:25 no matter where you store the data (unless you run with the modified remote.py ;)), the person running trove create can muck with it. 18:16:43 cp16net ++ 18:16:43 if we look at that table and find it (even if the instance was deleted) we know 18:17:22 i verified that enabling root and backing up an instance and creating a new one still shows the instance as root enabled 18:17:40 cp16net, yes 18:17:43 i deleted all the root users and ran a backup 18:17:53 and it failed because no root user was available 18:17:57 cp16net we could also keep track of when and who enabled/disabled the root access. It would also make it datastore agnostic. 18:18:09 umm its sorta like that 18:18:43 root_enabled_history has id(instance) user(user id) created(time it was enabled) 18:18:50 I verified that if you create a new user (with super-user privileges) and then delete the root user, the new instance created from the backup will show as root-enabled=false, even though you can still log in as the super-user 18:18:52 so SlickNik ... procedurally, do we want to spend the meeting finding the solution, or do we want to propose that someone (I volunteer) write up a BP for this and then we review it that way 18:18:56 it doesnt care about datastore 18:18:58 https://review.openstack.org/#/c/199635/ helps eradicating the problem as it finds any user with access more than USAGE created for anyone except localhost 18:19:04 (^^^ with Duk) 18:19:09 which could only be created using a root-access 18:19:16 So — here's where we are at. 18:19:27 * amrith waits for SlickNik to speak 18:19:29 sushilkm, that's very mysql-specific though 18:19:40 We've identified a hole that exists with this, and want to plug it. 18:19:59 And we've also identified customers and users who want a root-disabled API. 18:20:08 22cp16net: Well, but it should help us to avoid having to mess with datastore (mysql-specific) stuff on the GA as we have to do now... 18:20:54 We're acknowledging that we have a security issue here (existing, and possibly easier to find with the root-disable API). 18:21:39 Let's open up a bug to fix the issue and discuss the design of how we're going to fix it as part of that bugfix. 18:22:26 And we can unblock work on blueprint of adding the "root-disable" API. 18:22:49 SlickNik, sounds good 18:22:51 * amrith waits for EOF from SlickNik 18:22:52 yeah i think there was some great convos that came out of the root-disable API 18:23:11 It would be good to get both of these to land in Liberty, but I don't see why both need to land as part of the same blueprint. 18:23:32 18:23:37 SlickNik, ++ 18:23:52 #agreed 18:23:59 Happy to volunteer to land the code for this proposal (have some of it going, duk and co have framework to test it in) 18:24:24 I appreciate the unblocking of the disable root BP. and agree that it would be advisable for both to land in liberty 18:24:27 Okay, sounds good to me. 18:24:35 I think we have a good plan going forward. 18:24:35 in a way that the latter can be back ported if required. 18:24:55 I'll update the spec with this info immediately after this meeting. 18:25:04 thx 18:25:28 And open a bug for the issue we've identified so that we can continue discussion on that front.. 18:25:59 #action SlickNik to open a bug for the root-enable security hole on restore. 18:26:33 #action SlickNik update root-disable BP with plan of action from meeting. 18:27:04 Moving on. 18:27:17 #topic Trove-compatible images creation automation 18:28:38 recalling the conversation from last week. 18:29:37 vkmc was going to split out the image-elements from trove-integration to a new repo. 18:29:47 ./ 18:30:09 And come up with a script to build guest images independently of running redstack. 18:30:43 indeed, something like what Sahara is doing with their images 18:31:10 https://github.com/openstack/sahara-image-elements 18:31:14 I don't understand the solution/problem 18:31:17 we have elements 18:31:24 Once this was done, the next step would be to refactor redstack to simply pull in and use this repo and use this script so that the image building wouldn't have live solely in redstack. 18:31:27 we can put (in trove-integration) a script to build images 18:31:34 heck, you can run redstack build-image 18:31:50 or you can use the command in the dev-doc that we have. 18:32:01 so I'm not sure how a new repo improves the situation. 18:32:24 so, IMO the problem with trove-integration right now is that in order to create images for trove, you need to get the whole script for trove-integration 18:32:57 sorry, what does "whole script for trove-integration" mean? 18:33:08 plus, trove-integration is usually considered to be useful to deploy development environments, not real deployments 18:33:48 with that I mean, development environment setup script, image creation script and integration tests 18:33:52 all in one 18:34:05 * amrith waits for vkmc to finish 18:34:21 when maybe you only want a way to create images, to use in your production environment 18:35:09 we already have the docs amrith write for that, but certainly is not enough 18:35:42 we always get people in -trove asking things related to the image building 18:36:01 and that's pretty much it 18:36:10 ./ 18:36:19 so, a couple of things 18:36:30 do you think the issue is that trove-integration is hard for people to use 18:36:37 is it that http://docs.openstack.org/developer/trove/dev/building_guest_images.html is hard to read? 18:36:45 is it that trove should offer production scripts? 18:36:52 is it that trove should offer production elements? 18:36:58 is it that we need better documentation? 18:37:12 unfortunately, to none of those do I think the answer is "new project/repository" 18:37:20 amrith: why do you need a lot of variables related to redstack but not needed for image creation? 18:37:21 so I must be missing the question (I think) 18:37:22 is it that when users ask how to create their images, a vital part on trove, they don't have a place to go 18:37:44 they have the docs, yes... but still they find issues 18:37:54 amrith: please note that a separate repository dedicated to image creation is something already successfully used elsewhere (Sahara) 18:38:21 ok vkmc, I agree that there's an opportunity to improve the docs; we spoke about doing a presentation about that at summit 18:38:26 and should try again for Tokyo. 18:38:34 whereas the docs cover the major part, they don't cover things specific to the different distributions 18:38:38 and write better docs, and help people learn how to write elements. 18:38:39 amrith / vkmc: I can also understand an argument made for separation of concern. Right now our image-creation is tightly coupled with our development environments (e.g. passing of env variables regarding where the source code lives on the devstack box, etc.). If we want the elements to be reusable in a manner that lets users create images that they can use, it makes sense to decouple this further. 18:39:11 tosky, what variables? I don't get what you are saying. 18:39:13 It would be nice to refactor this so that the dev-stuff is done in a separate element that gets mixed in only for redstack. 18:39:28 so... if for a customer to use trove requires this customer to write their own elements 18:39:37 then I think we might face an adoption issue 18:39:43 SlickNik, you raise a great question, which is probably masked by this solution; should Trove be in the business of offering production elements. 18:39:53 I don't profess to have an answer; the quesiton however is the one at hand 18:40:05 amrith: 18:40:07 export HOST_USERNAME 18:40:08 export HOST_SCP_USERNAME 18:40:10 amrith: etc etc etc 18:40:30 you don't have to export all of those things if you don't want to. redstack build-image will do it for you. 18:40:45 the reason you have to set them is because the elements need them. 18:41:03 if you don't want to, you can write a wrapper around dib. 18:41:16 I think we can seperate it by saying trove-image-elements hosts standard elements for installing datastore specific stuff and trove integration provides elements specific for dev environment 18:41:22 that way people will be less confused 18:41:25 amrith: the element that vkmc prepared don't need them, and I'm sure REDSTACK_SCRIPTS is not needed by dib 18:41:55 so great ... are we suggesting that trove should 'support' production ready elements? 18:42:07 Ultimately, don't the user's operators need access to the nova instances? How would trove supplied "production elements" support that? 18:42:17 I don't think trove should support production ready elements 18:42:19 vgnbkr ++ 18:42:28 we are suggesting to have a way to easily build production ready images, with clean elements 18:42:39 again, this is something which is working fine in Sahara 18:42:45 vgnbkr, not sure what you meant 18:42:53 how many different 'databases' does Sahara support? 18:43:05 trove should provide elements that will allow somebody to install datastore sepcific packages but not put dev burden on them 18:43:23 vkmc, if something breaks in the nova instance underneath the trove instance, how would the operator fix it? 18:43:25 saurabhs, that's exactly my point 18:43:47 vgnbkr: I don't think the suggestion is that you supply _all_ the elements for a production image. If I understand the suggestion correctly, you'd be supplying most of them which an operator can extend on top of. 18:43:53 vgnbkr, we are not aiming to solve that 18:43:56 to all ... I would like to just point out that a solution was proposed, and today we're still having a somewhat challenging time stating what the problem is. 18:44:07 by moving datastore specific elements to a different repo we will decouple elements for dev and real use 18:44:24 for building production grade images people will still need other stuff on their images 18:44:36 saurabhs, by real use, you mean 'production-ready' ? 18:44:39 vgnbkr: Today, the elements are coupled with the dev-env, so you'd have to rewrite them if you wanted to build on top of some of those pieces for a non-development image. 18:44:44 amrith, not really, I think the problem is clear: we don't have a clear, automatic way to create images 18:44:44 saurabhs, suppose I propose a change to trove and it breaks one of those "production elements" then who gets to fix it? 18:45:14 vkmc, we have a clear way to create redstack images. we have NO way to generate production images. 18:45:16 amrith: who ever broke it :) 18:45:24 amrith: Sahara regularly tests the elements producing images; the solution is... testing, of course 18:45:24 so saurabhs that means Trove just expanded. 18:45:27 I am not suggesting that we make a copy of current image lements 18:45:49 tosky, the answer to my question was "1". There is one database you support. Trove today supports close to a dozen. 18:45:52 I am saying strategically move datastore specific stuff out to a seperate repo 18:46:04 and keep dev specific elements in trove integration 18:46:06 Seems to me that if the user can figure out all the other things they will need to do on top of production versions of the few images we provide, making the production versions would be trivial. 18:46:21 and when building elements for dev env we will combine elements from both 18:46:25 oh, great, so saurabhs is proposing splitting the trove repository into datastore specific ones ;) 18:46:45 amrith: not sure where I said that :) 18:46:56 amrith: Sahara supports 4 plugins, two different version of Hadoop for each of them, on multiple distributions 18:47:06 saurabhs, "strategically move datastore specific stuff out to a seperate repo" ;) 18:47:19 yeh but not per datastore :) 18:47:22 Why doesn't someone just make a SourceForge project for whatever database they want to support. Then there could be different types of "production images" for different types of deployments. 18:47:28 in total we're publishing more than dozen of images generated by sahara-image-elements script 18:47:31 saurabhs, you can buy me a beer when we meet ... 18:48:07 I am saying lets move elements which install/configure datastore (all the datastore) into a seperate repo that way its useful for building images for more than dev envs 18:48:37 saurabhs, and once again, that is what I wanted to mean 18:48:39 let trove integration have elements that allow things like rsync 18:48:54 vkmc: then we both agree 18:49:00 saurabhs++ 18:49:01 I'm fine with this as long as a trove developer isn't on the hook to make sure that changes are made and tested in this 'production ready trove CI'. that will do doubt follow. 18:49:03 We already decided not to split the guest agent into a separate repo because of gerritt issues - why would we do so for datastore images? 18:49:55 amrtith: today if a dev breaks mongodb element who is suppose to fix it? I think responsibility will be same when we move to a different repo 18:50:12 no, but the current elements are development only. 18:50:25 and therefore the bar is lower. 18:50:25 vgnbkr: The issue there was python packaging and circular dependencies. In this case dib-elements don't pose an issue (the way sahara does image elements today shows that this is a viable approach). 18:50:33 saurabhs, today the only elements that gate are mysql 5.6 on ubuntu 18:50:50 :) we want people to use trove or not or use it only for development purposes. 18:51:01 saurabhs, people do use trove 18:51:07 and people do build elements and images 18:51:21 I just want to know where we draw the line on the responsibility of the trove developer 18:51:37 if you stand up a new repo and you have a bunch of differen tproduction ready elements there 18:51:49 amrith: so how the real world testing is done, if only dev images are tested? 18:51:50 is it the trove developers responsibility to keep them updated 18:52:07 amrith / saurabhs: FWIW (anecdotal evidence alert) I've had multiple conversations with folks whose top problem is that building trove images is too hard given the current process. 18:52:19 SlickNik, I agree 18:52:40 but I don't know how this proposed solution does anything to address it! 18:52:45 amrith: and moreover, how different are the dev images from at least a simple model of production images? 18:52:58 amrtith: how about we take it from a different angle lets not call it prod ready elements. lets call them independant elements which don't require trove integration elements but will produce the image in a such a way that datastore specific package and configuration is there 18:53:02 and most of them are of the opinion that moving to a sahara like model (where the image-elements are separated out) so that they don't have to understand the intricacies of redstack would improve this quite a bit. 18:53:10 anyway, maybe this needs to be an in-person conversation. at mid-cycle? 18:53:26 so in this new repo will there be an "easier" script or will it basically be all the same it is today in redstack? 18:53:29 amrith: this proposed solution gives you a simple set of files which you don't need to dig from a bigger set, like (sorry for repeating it) sahara-image-elements 18:53:34 the major difference will be in where you get the trove code - is it from apt-get or git - is it a release version or HEAD etc 18:53:40 somebody can go on the top of that and install stuff specific for their production or even non production use 18:53:54 cp16net: please check https://github.com/openstack/sahara-image-elements 18:53:59 cp16net: The proposal was that there would be an easier script (ala sahara) — although I'll let vkmc confirm. 18:54:00 abramley, good point 18:54:00 cp16net: the idea is the same 18:54:08 and also what we do about users / ssh keys etc in the guests - since we expect certain things in the dev environment that you would never do in production 18:54:10 cp16net, yes, the idea is to make it easier 18:54:35 so whilst the database specific elements may look the same in dev vs production - the trove guest agent elements will be different 18:54:48 abramley: make those artifacts part element that only gets mixed in when test images are being built. 18:55:06 part of an element* 18:55:06 SlickNik, may I propose that we treat this just as we would any other 'feature' and BP it with a clear description of the usage, the use case, ... 18:55:29 if this is really so simple a solution, I don't believe writing it up as a BP should be that hard? 18:55:38 cp16net, a simpler script, easier to scale 18:55:42 SlickNik, so we'd maintain a second elements dir with dev stuff in it? 18:55:44 amrith, ++ 18:55:52 and oh by the way, a link to sahara elements repo isn't the answer. trove is just a bit different 18:55:55 amrith++ 18:55:57 If you understand what you need to do, it only takes a few minutes to deal with the trove-integration images. It's the months you need to put in to figure out all the other stuff. 18:56:09 s/images/elements 18:56:16 peterstac: no, ultimately we'd refactor redstack to call out to this other dib script once it's all set up so we're not maintaining two versions. 18:56:44 time check ... 4 minutes to go. 18:56:45 amrith: they are closer than many other projects; provide instances with $something, plugins vs datastores 18:57:07 regarding this specific issue, they are really really close, for what I've been able to grasp 18:58:14 Sure, we can have a BP to explain what we're doing in this case, but I don't want to block progress on this. 18:59:14 I think that simplifying the image-building process is something that we _really_ need to address (based on user feedback), and that this is a step in the right direction. 18:59:25 I'll propose the spec, we can continue discussing over it, if that is ok for all :) 18:59:36 vkmc: ++ Thanks! 18:59:43 vkmc ++ 18:59:49 vkmc++ 18:59:51 thanks :) 18:59:52 sounds good 18:59:57 We're out of time for today's meeting! 18:59:58 i'm still wrapping my head around it 19:00:03 :) 19:00:12 #topic Open Discussion 19:00:16 thnx :) 19:00:16 cp16net, don't hurt urself ;) 19:00:25 Any items? 19:00:31 (for open discussion) 19:00:33 * cp16net thinks i pulled a brain muscle 19:00:37 who's coming to mid-cycle ;) 19:00:45 o/ 19:00:46 We might have to take them to the #openstack-trove room 19:00:47 * amrith planning beer deliveries 19:00:49 o/ 19:00:52 :) thats hilarious 19:01:09 * amrith wanders off to #openstack-trove 19:01:17 i heard you like trove so i put some trove peeps in yo meeting room 19:01:18 amrith: I have registrations from evite, I'll send you numbers. 19:01:26 lol 19:01:31 #endmeeting