21:01:16 #startmeeting Reddwarf 21:01:17 Meeting started Tue Mar 12 21:01:16 2013 UTC. The chair is vipul. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:18 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:20 present 21:01:21 The meeting name has been set to 'reddwarf' 21:01:28 agenda: 21:01:30 #link https://wiki.openstack.org/wiki/Meetings/RedDwarfMeeting 21:01:34 \o 21:01:35 Here. 21:01:37 feel free to update as necessary 21:01:40 hello 21:02:08 #topic action items 21:02:19 http://eavesdrop.openstack.org/meetings/reddwarf/2013/reddwarf.2013-03-05-21.59.html 21:02:27 ok first one.. didn't update the snapshots wiki yet 21:02:41 #action vipul ot update Snapshots wiki to handle deleted in swift use case 21:02:55 i swear i was going to do it today 21:02:57 other stuff came up 21:02:58 oh well 21:03:06 ok sounds good 21:03:08 next.. SlickNik 21:03:12 Second one, I'm still working on Security Groups. 21:03:18 vipul: btw- robertmyers, imsplitbit and myself have a meeting tomorrow to look over that wiki article. 21:03:26 Cleaning up, extending the client — writing int tests. 21:03:37 grapex: awesome.. we've started some initial impl so appreciate any feedback 21:03:42 grapex, I was talking about it with him 21:04:01 I should have a patch out for review by end of tomorrow. 21:04:03 vipul: Cool - I noticed there was a commit for the schema. That may be premature, it's hard for me to review schema changes w/o the associated code. 21:04:21 I'd appreciate it if I can get multiple eyeballs on it after that. :) 21:04:25 #agree with grapex 21:04:36 #agree 21:04:37 SlickNik: cool 21:04:52 Although if several people need ot sync up on work haivng it merged would make it easier. I think the typical approach is to create a branch that everyone works from, then one person does a gerrit review for the whole thing on completion 21:04:53 #action SlickNik still working on Security-Groups. Hope to have patch up for review by Wed/Thu. 21:05:08 grapex: regarding schema change.. i think we should strive for smaller patches.. makes things easier to reivew 21:05:13 but i see the point about not having enough context 21:05:31 so maybe we need to find a good balance.. so maybe schema change + models code? 21:05:35 and then API later? 21:05:44 vipul: Yes, if the models code had some kind of unit tests on it 21:05:52 Enough just to show how it's used 21:06:03 #agreed 21:06:05 Yeah, I like that. Schema + models has a bit more meat to it. 21:06:06 sorry if i missed it was there docs around the schema changes? 21:06:06 vipul I think the models code would def help 21:06:15 so that we could see the path forawrd? 21:06:20 cp16net: https://wiki.openstack.org/wiki/Snapshot-design 21:06:22 :p 21:06:32 yea we tried to get that captured in the wiki design doc 21:07:08 ok cool it looks like there is more there than when i last looked at it 21:07:09 thanks 21:07:13 np 21:07:22 ok we'll get another review that's got more meat 21:07:35 So the other thing regarding schema changes for a feature like snapshot- if throughout testing we discover there have been any mistakes, we'll we need to add additional migration files to tweak the schemas? Or is the plan to just change the one migration file created for that feature? 21:08:06 grapex: that's a good point.. 21:08:18 grapex: i would be OK with changing the same migration.. 21:08:24 yeah i would rather it be a single change set at that point 21:08:25 the only thing is we shoudl coordinate to make sure it's not deployed 21:08:34 I think it would be best to just wrap it on one 21:08:44 or update the original 21:08:46 vipul: Me too, but internally at Rack I'd need to make sure I didn't put that code in the release until the feature was under wraps to keep ops from any headaches 21:08:46 maybe that can be communicated during the review.. 21:09:04 one works as long as it isn't already deployed 21:09:05 grapex: yea, shoudl be fine until we get other people deploying it 21:09:10 Well, headaches is an exageration, it wouldn't be that bad really, just wouldn't want to run the migration before the feature is used. 21:09:11 i know adding that once its deployed you cant change those migration scripts 21:09:15 cause ti will be a pain to fix... yeah what grapex said 21:09:16 * cp16net been there.... 21:09:37 * imsplitbit done that 21:09:40 yea, so maybe the rule of thumb is change existing, and if anyone has concerns that can be xpressed in review 21:09:48 at which point we can split it out to a new migration 21:09:55 grapex: I'd don't think you guys would want a half feature (i.e. only models code) in release anyhow. But yes, that's something to be wary about. 21:10:01 vipul: Sounds good. 21:10:31 grapex: another question regarding the shcema 21:10:43 do you guys do constraints on your migrations? 21:10:55 or something that you handle out of sqlalchemy or not do at all 21:11:21 i think most of openstack there really aren't constraints expressed in the migrate scripts 21:11:35 vipul: We actually just use the sqlalchemy stuff directly and 80% have no issues 21:11:50 constraints are a pain because existing data can set them off. 21:11:52 20% we talk to Ops and do it by hand I think. 21:11:58 Oh, I see. 21:12:20 jcooley: agree, so maybe we don't do constraints in reddwarf either 21:12:30 I think once I tried to add non-null constraints and ran into problems. 21:12:57 ok action 21:13:00 Seems like a good idea honestly if we could get them to work. 21:13:02 yep, worse is there are NULL values and someone applies a non-NULL constraint. then you crash. 21:13:03 i dont remember either 21:13:24 happens frequently when you add columns to a table. :) 21:13:39 for simple things like unique constraints, we might want to add those... 21:13:43 just no fkey, etc 21:14:05 I think FK constraints are goodness too for things that make sense (like ids) 21:14:05 as long as those constraints as defined when the table is first created not added later 21:14:10 i did add unique constraint in the quota table 21:14:11 Yes, agreed 21:14:47 You'd run into most problems when trying to add constraints to pre-existing data. 21:15:00 yep.. 21:15:05 ok next action was mine as well 21:15:15 #action vipul to submit snapshots API to database-api docs 21:15:17 yes 21:15:19 did not complete 21:15:31 next item: grapex 21:15:36 did you learn how to use action? 21:15:47 heh 21:15:51 No, I forgot. :( 21:15:54 that was hubcap's action, I think… :) 21:16:00 to "teach" grapex... 21:16:04 As for that decorator, I did some thinking and closed the blueprint. 21:16:10 slacker... 21:16:29 I was thinking, adding a decorator like that defeats the purpose of gating on tests. 21:16:47 decorator to only run tests against certain versions? 21:16:54 vipul: yes 21:16:58 i hope we solved the client issue 21:17:03 by making it pull directly from github 21:17:07 It's frustrating, but if we disable tests on brand new features we could check in tests that fail to work and not realize it until we tick up the client version later. 21:17:18 grapex: maybe the change we make to reddwarf's pip-requires will be enough to keep the client in sync 21:17:42 It seems a better solution is to keep them in Gerrit purgatory until everything is ready to go. 21:17:44 I've been trying to check in client changes first.. 21:17:49 Right 21:17:55 esp vipul: I think the git change vipul merged will really help us. 21:17:57 esp: yea if we stick to that pattern, seems like we'll be ok 21:18:15 it sucks for the person submitting the feature but will ultimately benefit the wider team 21:18:25 #agree 21:18:40 ok last item: SlickNik 21:18:43 I'm not sure I'm a fan of the decorator myself, so as much as I don't like gerrit purgatory, I think that might be the better option here. 21:18:48 trimming redstack 21:19:14 I looked into refactoring redstack, and started on it... 21:19:27 SlickNik: Nothing more fun than refactoring bash eh? 21:19:41 Grapex, let's just say It's a little hairier than I thought :) 21:19:42 Better than refactoring awk 21:19:52 it woudl be good to know some of the 'other' functions in redstack are used by anyone 21:19:56 datsub180b: Or batch files. 21:20:10 like start-deps, etc 21:20:14 I'll probably have done after this sec-groups stuff (hopefully by end of this week) 21:20:28 vipul: We end up using some functions for internal things, mostly billing testing. 21:21:03 And personall I call stop and start-deps frequently when making changes to Nova. 21:21:11 The issue is that there is a lot of code that is "slightly" different… So I'm trying to find most of so I can de-dupe and reduce the footprint.. 21:21:20 grapex: we should trim if we can anywhere.. maybe we need them though 21:21:34 SlickNik: do you have a bug/bp for that? 21:21:41 Yes, I have a bug…one sec... 21:21:49 maybe we can get grapex to comment on that bug with stuff they use or don't 21:21:52 #link https://bugs.launchpad.net/reddwarf-integration/+bug/1154170 21:22:26 #action SlickNik working on trimming down redstack to use local.sh https://bugs.launchpad.net/reddwarf-integration/+bug/1154170 21:22:47 awesome 21:22:48 thanks! 21:23:14 ok that's it for action items... 21:23:21 np. 21:23:21 #topic limits updates 21:23:36 most of the limits stuff is merged 21:23:37 so I'm in the middle of some fixin' 21:23:45 should be done between today and tomorrow. 21:24:04 sweet 21:24:10 I had to refactor view code in the API and Client to get it working with XML 21:24:15 esp: Nice. 21:24:33 good timing on the xml tests grapex 21:24:41 yeah should be less crusty.. 21:24:42 probably wouldn't have been caught til much later 21:24:48 esp: lol 21:25:01 vipul: Thanks Vipul, we've had some issues lately at Rack so I figured I'd flip them on. 21:25:03 xml is soo 90's 21:25:22 do other openstack projects support xml? 21:25:24 lol, yeah, there are many "Enterprise Professionals" in suits who will be happy ours works. :) 21:25:31 just curious, not sure what the stance is there 21:25:31 vipul: In theory. ;) 21:25:44 They all do, I'm just kidding 21:25:54 remember that talk we went to about improving support though? 21:26:00 Apparently there are some issues 21:26:01 btw 21:26:08 I should probably talk about this a bit 21:26:10 yea it seems like everyone wants to drop support 21:26:11 oh yeah... haha 21:26:25 but obviously that's a huge change so they can't 21:26:27 xml support in nova no one could agree it sounded like 21:26:39 yeah 21:26:41 Well the other projects have XML and JSON views that are very different 21:26:51 esp totally, I feel retro when I use it 21:26:54 where as most of our views "just work" in XML with a small bit of serialization configuration 21:27:04 I feel like we've lucked out 21:27:18 imsplit: yep. listening to Toto as I code it. 21:27:31 oh wait, that's more 80's 21:27:34 yeah 21:27:34 grapex: Yea, it's nice not having to maintain two separate views 21:27:39 but funny nonetheless 21:27:55 I'm all for use JSON or go home 21:27:55 by the way, I'm testing this out internally before I subject you guys to it, but I think it may be possible to use a tool like XmlLint or Json schema in conjunction with the client. I have code that I'm running now that extends the client to save the request / response text into files 21:28:14 but i think esp found some assumptions that were made.. like the depth of the tree 21:28:27 my next goal is to run xmllint and json schema against it to make sure the old XSD is accurate... whether it is is a bit of a mystery now. :) 21:28:56 grapex: probably a good idea. 21:29:00 vipul: True. For something like limits that was taken from Nova, it may require work to make the views correct in both. 21:29:23 grapex: so would that be permanent in rdclient? or just a one time thing to test accuracy against xsd 21:29:30 if I get this working I'm not sure where we'd run it... I might just write up publicly how to do it so when we have real mode in CI on Jenkins somewhere we can kick it off. 21:29:34 but the 80's was such a great decade. ET, Nintendo, Thundercats and no Justin Beaver 21:29:45 LOL 21:29:46 It would use rdclient but the extra code is in the tests. Its just one class, pretty slim. 21:29:58 esmute: true, there were some good moments in there. 21:30:30 esmute: I think you might have had a better childhood than me. 21:30:40 grapex: Yea, it may be good to have it part of CI but may not be necessary to have validation in the client? 21:30:43 esmute: The really sad thing is that if they made a Thundercats movie today Jerry Bruckheimer would direct and Justin beaver would be the main star. 21:30:55 who's justin beaver 21:30:57 lol 21:30:57 vipul: Yeah, don't worry, I won't burden the client code / repo with that code. 21:31:05 oh no god no 21:31:29 evil twin of justin beiber? 21:31:41 vipul: Anyway, I'll keep you guys up to date on how it goes. 21:31:45 lol 21:31:51 grapex: ok cool 21:31:52 You don't know Justin Beaver? Dam…. 21:32:06 haha. Where is the auto-spell-correcting bot here? 21:32:20 grapex action? 21:32:29 esmute: Ha! I spelled it that way too. And yet, I feel no shame in getting it wrong... :) 21:32:30 good call cp16net 21:32:42 Grapex, that sounds good… 21:33:10 (both the xml and no shame parts) 21:33:15 oh sorry, hub_cap didn't 'teach' you about action 21:33:40 * grapex Look into validating the API with XmlLint and Json schema. 21:33:48 #action grapex to look into xml validation 21:33:54 done 21:33:54 vipul: Thanks. :) 21:34:05 #topic Security Groups update 21:34:18 Already covered this earlier. 21:34:33 k cool.. 21:34:41 Expect a patch up for review soon. 21:34:50 nice 21:35:05 (by Thursday more concretely) 21:35:08 #action Snapshots Update 21:35:12 whoops 21:35:19 #topic Snapshots Update 21:35:30 #link https://wiki.openstack.org/wiki/Snapshot-design 21:35:44 grapex, imsplitbit please review ^^ 21:35:51 appreciate feedback on that.. 21:36:13 swift client is about wrapped up 21:36:21 went with the good ol' fake 21:36:38 vipul: We're looking today, we'll have a meeting tomorrow over it. 21:36:46 realizing that being new to python and making some big changes doesn't make good chemistry 21:37:05 To recap some of the discussion i had with imsplit bit, we're supporting point in time backups and recovery to 'that' point in time. Not supporting point-in-time recovery (restore to 5 mins ago) 21:37:32 recovery is to a new instance only 21:37:41 vipul: This is for snapshots? 21:37:44 yes 21:37:47 so the user just can't pick the "snapshot" they want and say restore? 21:37:55 juice: exactly 21:38:05 juice: they get exactly what the snapshot contained 21:38:13 Ok, cool 21:38:18 in a new instance 21:38:19 So btw 21:38:35 in the end, this is going to be a call to the guest that will upload a file to swift, right? 21:38:37 how much storage do they get in swift and what happens when a snapshot will exceed those contents? 21:38:51 grapex: yes 21:38:53 yes grapex, that's the idea... 21:38:53 grapex: Yea, so we'll zip+upload 21:39:01 juice: we'll need to look into chunking 21:39:10 juice: i believe swiftclient already does this for you 21:39:11 chunking the post 21:39:17 if > 5 gigs 21:39:18 for performance? 21:39:22 ah 21:39:23 juice: thats a good point because you may have to chop up the zip 21:39:25 which i think is the max file size 21:39:37 object file size 21:39:50 if you guys add a fake method to the guest in fake mode, you should be able to write even the integration tests soon. In fake mode I think all the API will do is say "sure!" when you ask it to upload a a snapshot 21:39:57 yea think that's the max limit in swift? could be wrong 21:39:58 yes 5gb is the max size (which isn't the same for folks like was) 21:40:07 vipul: sorry if this is in the wiki, but is the idea to use the action stuff hub_cap was looking at? 21:40:32 at least it is stated as such and I have tried on occasion to exceed it when storing season one of walking dead 21:40:48 grapex: No, currently we don't have the framework for that in place 21:41:01 grapex: we could come back aorund and add the hooks into what hub_cap submit? 21:41:02 I wonder if we can do something cool with incremental block delta storage. (Might have to come in later). 21:41:28 vipul: I think so 21:41:46 The action stuff he showed me, which is in Nova now, looks like it might solve a lot of the issues we've had with asynchronous actions so far 21:42:05 oh yeah that would be awesome 21:42:25 vipul: Too bad he's been moving, as soon as he get's back I'll let him know to sync up with you guys. 21:42:48 grapex: Ok, yea i need to spend more time looking into the nova feature, i knnow he tried to explain it last meeting i think 21:43:36 #action hub_cap to sync up with vipul on the nova (instance_actions/task_actions) 21:43:40 grapex: I think there is also some open questions around myISAM.. which we will support.. but it's just a binary copy of the files (wht xtrabackup) 21:44:14 also means your db is locked 21:44:19 during that copy 21:44:34 which is something we don't have to worry about when you do xtrabckup against innodb 21:44:51 just stuff to think about when you guys review the spec 21:45:10 vipul: Ok, we'll look into it and get back to you sometime tomorrow or Thursday. 21:45:26 ok cool 21:45:33 I'll print out that wiki article and read it with some fine cognac and a slice of cheese. 21:45:42 #action annashen to amend patch to include models + tests w/migrate script 21:46:04 grapex: you might need some tequila just for good measure 21:46:31 ok anything with snapshots? 21:46:34 if not... 21:46:39 nope, sounds good. 21:46:42 vipul: Some shots to go along with snapshots? 21:46:46 heh 21:46:49 grapex: I think I would read more wiki articles if I did it your way :) 21:47:02 heh 21:47:08 #topic Releasing python-reddwarfclient 21:47:20 so i put this in there just cuz there was some confusion around how to release 21:47:23 and the new process 21:47:57 SlickNik made a patch to openstack-ci to support auto-releasing of python-reddwarfclient on a git tag 21:48:06 #link https://wiki.openstack.org/wiki/Release-python-reddwarfclient#Releasing_python-reddwarfclient 21:48:08 I remember seeing that 21:48:17 Btw, thanks SlickNik! 21:48:19 I tried to capture what i know of the process ^ 21:48:54 vipul: So as to frequency: I feel like the client almost never becomes backwards incompatable 21:49:02 Seems to be the case 21:49:03 gist of it is.. anyone in reddwarf-core can push a tag to gerrit 'git tag 1.1.1', 'git push --tags gerrit' 21:49:10 How would you all feel to a release following the completion of each new feature? 21:49:35 There's two things that come into effect when you tag a changeset in git and push it to gerrit... 21:49:47 how do we put the fire of someone accidentally tagging a busted release with a numeric version 21:49:52 i assume I'm going to do that twice 21:50:19 datsun180b: We write their name on the board or add a line to it. At five strikes we call their parents. 21:50:20 i guess gerrit will catch it first? 21:50:36 hmm..so can we post tag? 21:50:47 grapex: what if batman does it 21:50:57 lol 21:50:59 datsun180b: i think you may need to push twice to replace the 'bad' version 21:51:01 1. If the tag matches an alpanumeric regex of the form d*.d*.d*.d*, it is a pre-release and will make a tarball and upload it to tarballs.openstack.org 21:51:31 gotcha 21:51:38 oh thats awesome 21:51:40 (where is alpha/beta/a/b/c/d/e/g 21:51:53 tags should be treated as immutable fwiw 21:52:18 you cannot garuntee that a tag that is moved will update sanely everywhere 21:52:32 that's very important 21:52:37 2. If the tag matches a release version such as 0.1.2, it's treated as a release and will tarball and also upload to pypi... 21:52:58 clarkb: Will it also push the docs to PyPi? 21:53:35 Right now it doesn't push the docs, grapex. 21:53:43 no docs. there are docs jobs that do other things 21:53:48 SlickNik: Ok. 21:53:58 you can have a job to do that in theory though 21:54:11 clarkb: what's the relationship between the version='xx' in setup.py and git tag 'version' 21:54:23 rtfd and docs.o.o are currently supported 21:54:32 is there one that takes precedence? 21:54:43 vipul you shouldnt use a hard coded on anymore 21:55:11 using oslo you get dynamic versions based on git tags. mordred knows all about it because he wrote it 21:55:18 Ah yes.. ok 21:55:31 so we should probably setup.py from olso in too.. 21:55:49 grapex/clarkb, there might be some figuring out to do with credentials for the pypi upload, so we might need to flush that out when we're ready to release to pypi... 21:55:50 #action vipul to look into bringing in setup.py from oslo into python-reddwarfclient so version remains dynamic 21:56:11 SlickNik: Ok. 21:56:20 clarkb SlickNik vipul: Thanks for all your work on this. 21:56:59 yup thanks guys 21:57:08 np, anytime! 21:57:10 +1 21:57:16 thanks clarkb! 21:57:38 ok anything else about versioning/releasing client? 21:58:08 btw, reddwarf doens't use either, it pulls tarball from github directly 21:58:12 I'll update that wiki page with the actual release/pre-release regexp locations for reference… 21:58:25 yes, please do SlickNik, thanks 21:58:45 #topic Open Discussion 21:59:04 anything else we need to cover 21:59:06 #action update the release wiki page with regexps for release/pre-release 21:59:16 #action ^^^ SlickNik 21:59:32 nope, nothing else from my end. 21:59:40 wow look at that we might make time 22:00:04 vipul: We are on fire today! 22:00:15 w00t 22:00:24 hellz yea! 22:00:30 have a great day ev1 22:00:31 #endmeeting