Thursday, 2015-02-12

openstackgerritBrian Curtin proposed stackforge/python-openstacksdk: Add the Type resource for the Volume service
*** pm90_ has quit IRC00:22
openstackgerritBrian Curtin proposed stackforge/python-openstacksdk: Add the Volume type for the Volume service
openstackgerritBrian Curtin proposed stackforge/python-openstacksdk: Add the Snapshot type for the Volume service
*** pm90_ has joined #openstack-sdks00:49
*** sigmavirus24 is now known as sigmavirus24_awa01:43
*** etoews has joined #openstack-sdks02:12
*** etoews has quit IRC02:56
*** terrylhowe has quit IRC03:48
*** etoews has joined #openstack-sdks03:53
*** terrylhowe has joined #openstack-sdks03:54
*** etoews has quit IRC03:57
openstackgerritBrian Curtin proposed stackforge/python-openstacksdk: Add the Type resource for the Volume service
*** pm90_ has quit IRC04:25
*** ayoung has quit IRC04:34
*** etoews has joined #openstack-sdks05:53
*** etoews has quit IRC05:58
*** etoews has joined #openstack-sdks06:54
*** etoews has quit IRC07:03
*** Miouge has joined #openstack-sdks07:35
*** terrylhowe has quit IRC07:50
*** stevemar has quit IRC08:04
*** Miouge_ has joined #openstack-sdks08:05
*** Miouge has quit IRC08:06
*** Miouge_ is now known as Miouge08:06
*** openstackgerrit has quit IRC08:21
*** openstackgerrit has joined #openstack-sdks08:21
*** chlong has quit IRC08:42
*** etoews has joined #openstack-sdks09:01
*** etoews has quit IRC09:05
*** Miouge has quit IRC09:37
*** Miouge has joined #openstack-sdks09:43
*** Miouge has quit IRC09:48
*** Miouge has joined #openstack-sdks10:00
*** etoews has joined #openstack-sdks11:02
*** etoews has quit IRC11:07
*** terrylhowe has joined #openstack-sdks11:59
*** etoews has joined #openstack-sdks12:03
openstackgerritMerged stackforge/python-openstacksdk: Introduce the Volume service
*** etoews has quit IRC12:07
*** pm90_ has joined #openstack-sdks12:35
*** tellesnobrega_ has joined #openstack-sdks12:36
*** pm90_ has quit IRC12:46
*** pm90_ has joined #openstack-sdks12:46
*** tellesnobrega_ has quit IRC12:58
*** etoews has joined #openstack-sdks13:04
*** etoews has quit IRC13:08
*** tellesnobrega has quit IRC13:11
*** tellesnobrega has joined #openstack-sdks13:37
openstackgerritJulien Danjou proposed stackforge/python-openstacksdk: telemetry: add support for Gnocchi metric
openstackgerritJulien Danjou proposed stackforge/python-openstacksdk: telemetry: add support for Gnocchi capabilities
openstackgerritJulien Danjou proposed stackforge/python-openstacksdk: telemetry: add support for Gnocchi metric
openstackgerritJulien Danjou proposed stackforge/python-openstacksdk: telemetry: add support for Gnocchi capabilities
*** pm90_ has quit IRC13:40
*** sigmavirus24_awa is now known as sigmavirus2414:01
openstackgerritMerged stackforge/python-openstacksdk: telemetry: add support for Gnocchi capabilities
*** tellesnobrega has quit IRC14:51
*** tellesnobrega has joined #openstack-sdks14:54
*** tellesnobrega_ has joined #openstack-sdks14:54
*** tellesnobrega_ has quit IRC14:55
*** ayoung has joined #openstack-sdks14:56
*** etoews has joined #openstack-sdks14:58
*** stevemar has joined #openstack-sdks15:15
openstackgerritTerry Howe proposed stackforge/python-openstacksdk: Dynamically load services
*** pm90_ has joined #openstack-sdks15:43
*** pm90_ has quit IRC15:44
briancurtinterrylhowe: how will vendor plugins be able to override services with this entry_points model? the way it looks, there can only ever be one openstack.object_store, for example16:04
briancurtinoh, maybe I need to provide my own module loader that first finds and then overrides them with
dstufftentry points in general allow overriding16:11
dstufftI dunno if python-openstacksdk is doing something special about that16:11
briancurtinoh wait, i might have screwed something up (a period where i needed a colon), trying again16:12
dstufftbriancurtin: note: that depending on how you load the entrypoints you may or may not get a reasoanble behavior if you have two things named the name16:15
*** pm90_ has joined #openstack-sdks16:16
*** pm90_ has quit IRC16:16
*** pm90_ has joined #openstack-sdks16:17
briancurtinit seems like it's working as a last-one-wins. i changed the rackspace plugin to install the object_store entry_point to rackspace.object_store, and now even if i create a connection to my devstack, "openstack.object_store" points to that rackspace module16:17
dstufftis this using stevedore?16:19
dstufftso yea, stevedore just returns the last one16:20
dstufftentry points are not a stevedore thing, they are a setuptools thing, stevedore just wraps them16:21
dstufftentry points themselves will give you all the results for a specific group + name16:21
dstufftand it's up to the program to decide which one(s) to use16:21
dstufftstevedore makes the decision it should be the last one it finds16:21
briancurtindstufft: so in SDK we have and then my plugin has -- in the end, if you say you want to use rackspace, your openstack.object_store should be cloud files from the plugin (actually rackspace.cloud_files), but16:22
briancurtinif you don't provide a plugin your openstack.object_store is the underlying openstack.object_store16:22
briancurtin*** (actually rackspace.object_store)16:22
briancurtinis that something we can actually do here?16:23
dstufftLet me make sure I understand, if you have python-openstacksdk and racksapcesdk you're going to end up with two things providing an openstack.service named "object_store", the one that compares with python-openstacksdk and the one that comes with rackspacesdk (and if you also install, say hpsdk you'll get a third one).16:25
dstufftSo do you want python-openstacksdk to _always_ use the built in one unless otherwise specified by the person using python-openstacksdk, or do you want it to try and automatically pick one to use16:26
briancurtinyep, or at least that's how i think it works in entry_points land. i think if you specify nothing in particular, as in you don't create your connection with a vendor plugin (ie, use the built-in keystone), you should get the builtin. if you create a connection and pass in a plugin name, you should get the default services + any services your plugin overrides16:28
briancurtinshould be what you get. so in this case, creating with rackspace should get everything openstack provides with the addition of our object_store taking over for the default16:28
dstufftthere is no built in16:28
*** andreluiz has quit IRC16:29
*** andreluiz has joined #openstack-sdks16:30
dstufftthe APIs to list, "object_store" is going to just list everything that provides that in the order they appear on sys.path16:30
dstufftI believe if you named it "arackspace-sdk-plugin" instead of "rackspace-sdk-plugin" then you'd get the racksapce ones by default16:31
briancurtini dont think this is going to work16:32
briancurtinor, i think i can make it work but just not by relying 100% on entry_points to be teh way. i might try this custom module_loader thing to load the defaults, and then load the plugin, and inside UserPreference do the override16:34
sigmavirus24Yeah I think UserPreferences is the place this will need to be done16:34
briancurtinthat way entrypoints is giving us everything and we are determining the order based on what we need16:34
dstufftso entry points gives you 2 basic discrminators16:35
sigmavirus24briancurtin: I wonder if something like object_store.from_vendor('rackspace') would be a decent way to do it...16:35
dstufftyou can discriminate based on group + name, so like your group is "" and the name is "object_store" or you can discrimnate on distribution name16:36
dstufftso if you want to keep the group + name being + "object_store", you'd need to discreminate based on distribution name16:36
briancurtinsigmavirus24:  i dont think so. there should be one place to specify in the code where you're working with a plugin - when connecting. what you get back at that point should work with what that plugin specifies16:37
dstufftso you could easily have python-openstacksdk call pkg_resources.get_entry_map("python-openstacksdk", "") which would give you a dictionary of things like {"object_store": ""}16:38
dstufftand have people pass in something like plugin="racksapce-sdk-plugin"16:38
dstufftto load from rackspace16:38
*** tellesnobrega_ has joined #openstack-sdks16:39
*** tellesnobrega_ has quit IRC16:39
dstufftif you switched it so your "group name" was something like "", then you could have names like "default", "rackspace", etc. So instead you'd do something like pkg_resources.iter_entry_points("", "default") which would give you a list of (likely 1, unless someone else also installed something with that name) or something like pkg_resources.iter_entry_points("
dstufftject_store", "rackspace") with the same "a list of likely 1 unless two things provide a thing named rackspace". This would mean you could do plugin="rackspace" instead of needing to use the whole distribution name16:41
dstufftbut if you want to be able to look up a unique thing, you need to either use the distribution name or the the entry point name, the distribution name is guaranteed unique but may have extra stuff you need to type (like rackspace-sdk-plugin) the entry point name is probably unique and gives you the ability to have a shorter name (or multiple shorter names) but may have collisions16:43
dstufftif that all makes sense?16:43
dstufft(you can of course mix these, by having something like openstack.alias as a group name and rackspace = "some function that just returns rackspace-sdk-plugin" and use the alias to turn "rackspace" into "rackspace-sdk-plugin" and then use that to call pkg_resources.get_entry_map("racksapce-sdk-plugin", "")16:46
briancurtinit sort of makes sense. im going to try and handle this within this UserPreference class and maybe come back to it16:49
dstufftbriancurtin: Sure, fwiw if it's easier for you, you can tell me how you want the API for end users to look/work and I can translate that into something reasonable for entry points too16:51
*** tellesnobrega_ has joined #openstack-sdks16:52
briancurtindstufft: this change makes it work (now to just make some of that less boilerplate-ish, but not bad overall)17:03
dstufftbriancurtin: hmm, won't that make it hard to use plugins from two different python packages? If you're relying on overriding the module loader in a package specifier user preferences?17:05
briancurtini dont know what you mean by plugins from two different packages17:06
dstufftbriancurtin: what if you want to use Rackspace's object store with Foobar's Compute17:07
briancurtinyou would have to create two different Connection objects, unless you made a megaplugin that wanted to merge those two services17:08
briancurtinyeah, you would just create a UserPreference that specifies those different services17:10
briancurtinwhere you would just do service["compute"] = foobar.compute.compute_service (pseudocode, but sort of close)17:10
briancurtinso my original thing about two different Connections is wrong17:11
dstufftbriancurtin: Maybe It's not an importnat use case, I dunno. Seems like it'd be better to not rely on each vendor to ship their own user preferences, I think it'd be saner to let the user pass args into the openstacksdk's user preferences to instruct it to use different "backends" for a particular service17:12
dstufftsec, I'll show you what i mean17:13
dstufftbriancurtin: this will mean that openstack.user_preferences.UserPreference() will use everything from python-openstacksdk by default, if you do something like openstack.user_preference.UserPreference(provider="rackspace-sdk-plugin"), it'll load everything it can from rackspace-sdk-plugin and fall back to the default ones, or you can do something like openstack.user_preference.UserPreference(provider={"object_store": "rackspace-sdk-plugin"17:31
dstufft, "compute": "other vendor"}) and it'll load those specific ones from those vendor's sdk's and everything else from default17:31
dstufftbriancurtin: I dunno though, it feels like it should be possible to mix and match vendors without the end users needing to subclass things and just by providing the right arguments to something, and in that vein it feels like each vendor having their own user preferences at all should be discouraged? It's user preferences not vendor preferences?17:36
briancurtini dont know, i have to think about it. i am 100% against anyone having to identify things by "rackspace-sdk-plugin", though. right now that would make Connection(auth_plugin="rackspace", preference=UserPreference(provider="rackspace-sdk-plugin"), ...) the way to get started and that's a no-go17:39
briancurtini have to leave for a bit, will catch up later17:41
dstufftbriancurtin: we can add some indrection so users don't have to explicitly type in "Rackspace-sdk-plugin" fwiw (the auth plugin is a thing too, probably needs some thought about how to handle that in general)17:42
dstufftsee ya!17:42
*** bknudson has quit IRC17:47
*** tellesnobrega_ has quit IRC17:53
*** tellesnobrega_ has joined #openstack-sdks17:58
utahconcan you pass a keystone v2.Token() to session like you can v2.Password18:12
utahconI am getting a Bad Request, and I am not sure why.18:12
utahconI fail back to using .Password() with a username and password, and it works fine.18:12
utahconCould my token be bad? Or can you just not do that in v2?18:13
*** bknudson has joined #openstack-sdks18:13
briancurtinIn what library are you passing this token?18:18
*** tellesnobrega_ has quit IRC18:31
utahconbriancurtin: v2.Token() and session.Session() are from keystoneclient18:33
*** tellesnobrega_ has joined #openstack-sdks18:35
*** terrylhowe has quit IRC18:44
briancurtinutahcon: you might try #openstack-keystone (some ksc people are in here but more are in there)19:03
utahconthanks briancurtin19:09
dstufftdtroyer: ping19:13
dtroyerdstufft: yo19:14
dstufftdtroyer: I'm told you have opinions about how OSC might be modified to use python-openstacksdk19:15
dtroyerunpopular ones most likely19:15
*** katco has joined #openstack-sdks19:15
katcoi need some assistance with parsing the storage wadl file. i'm not that familiar with the format, but the wadl file seems to contradict the specification? is anyone familiar?19:16
dtroyerdstufft: was there something specific?19:18
dstufftdtroyer: Well I'm planning on generating some CRs to start moving OSC in that direction but I wasn't privy to past discussions on that so I figured i'd reach out incase you had any specific ideas or input about it19:19
dtroyerok.  I am in no hurry to adopt the sdk yet,  first off, I'm not sure that the auth bits align with ksc and we can't really go backward on the additional ksc plugins because people are already using it19:22
*** terrylhowe has joined #openstack-sdks19:22
dtroyerOSC really doesn't need the high-level stuff nor the object model stuff.  I have a low-level basically procedural API layer that does what OSC needs and attempts to suss out the common operational bits.19:22
dstufftdtroyer: is "low-level basically procedural API layer" something you've built on top of the existing python-wahteverclients or are you talking to the APIs directly?19:37
dtroyerI'm replacing the existing client libs.  they have to go.19:38
dtroyerIt builds on KSC's Session:
dtroyerand is incomplete19:38
dstufftdtroyer: Ok, so you're building on top of KSC's Session to build your own API that talks to the Openstack servers directly?19:39
dstufftdtroyer: So question then, how is that going to handle vendor specific things?19:40
briancurtindtroyer: is that low level thing something that could become a low level thing within SDK, and then SDK's higher levels build on that and you build OSC using it? or is it not like that?19:40
dtroyerat this point I'm not very worried about vendor-specific things… they can today be osc plugins and work fine, I have a few myself19:41
dtroyerbriancurtin: I porposed that very thing long ago and there was little interest in it19:41
dtroyerhonestly after the amount of time that Transport took for minor wrapper to requests.Session, I didn't have the time/energy to push it hard19:42
briancurtinlike a lot of things back then, i dont think we understood a lot about how this was going to grow. at least i didnt. i'm certainly not interested currently in both of us attempting to rid the world of python-*client while simultaneously duplicating library code19:43
dstufftit seems like having two different combined "talk to the openstack API" thing seems like a duplication of efforts and if they have incompatible ways for vendors to extend them that means every vendor will need to support both of them or things get kinda confusing from an end users stand point about which tools they should use with which cloud. Is there some way to unify the two approaches? (Forgive me, I'm still ramping up on the SDK so19:45
dstufft maybe this has been discussed already)19:45
dtroyerbriancurtin: agreed.19:45
dtroyerdstufft: it is a duplication, and probably mostly my fault for continuing in the direction I was already headed.  But there were other priorities (KSC auth) and still some other things I need to get into OSC.  user's don't care what is under the hood at this level if it works, except for the dependency nightmare of installing osc19:48
dstufftpython-openstackclient describes itself as "a command-line client for the OpenStack APIs." while python-openstacksdk describes itself as "a collection of libraries for building applications to work with OpenStack clouds", so it seems like it would make sense for python-openstackclient to consume python-openstacksdk, then code for talking to the servers (and whatever vendor extensions that entails) only needs to live in one location.19:48
dstufftdtroyer: So I guess my question is, at a high level, ignoring the details and what is supported where, how do you feel about python-openstackclient consuming the python-openstacksdk?19:49
dtroyerdstufft: that was the original intent19:49
dstufftif openstacksdk provides what you need, is that something you'd be interested in? If yes, what do you think it'd need for you to be willing to say "this is a thing I can use in python-openstackclieint"19:49
dtroyeragain, that is exactly why I got involved from the beginning of the sdk.  and still something I want to happen.  but OSC doesn't need the overhead of most of what is in the SDK today.  My priority code-wise is to eliminate the other client libs, they are nothing but a headache.  but right now AIUI, sdk can't do that for me yet, even if I use the object model19:51
dstufftdtroyer: So let me break apart this statement into two things, you're talking about the overhead of using objects instead of just calling functions?19:52
dtroyerthe overhead of making every resource used in the API objects.  both execution-wise and conceptually.19:53
dtroyerI see it as complexity that brings very little return in this particular application19:53
dtroyerfor others, it is valuable19:53
dstufftIs that a deal breaker for you? Or is that something you'd not mind if python-openstacksdk let you drop all of the other client libs?19:54
dtroyerit's a high bar.  other than 'it exists', what is the right reason to just absorb the complexity?19:56
katcois anyone here responsible for the WADL specs? e.g.
briancurtinkatco: #openstack-doc19:57
katcobriancurtin: hm. ty. i am generating a client based off these, that is not more appropriate here?19:58
briancurtinkatco: oh, that part of it is perhaps relevant, but i dont think anyone uses or is responsible for those in here, at least as far as i know19:58
katcobriancurtin: gotcha. thanks for pointing me in the right direction. i'll take a peek over there and pop back in if it becomes relevant here19:59
dtroyerkatco: looks like #openstack-cinder would know about the source of those docs19:59
katcodtroyer: i tried in there, but it looked like a pretty busy discussion was happening :)19:59
dstufftdtroyer: I'm not sure I have an opinion on it yet specifically or not as I'm not sure I fully understand your objection, From the python-openstacksdk side of things I'd be most concerned about what presents the best API usability for someone using the API, that does increase the work that the sdk (via Python) has to do in order to say complete request, but without benchmarks I can't say if it's a meaningful amount of work or not.20:00
katcodtroyer: i'll narrow my question and ask again20:00
katcodtroyer: ty20:00
*** Miouge has quit IRC20:03
dtroyerdstufft: I think we (openstack or python in general, not sure) have a tendency to over-engineer things in ways that do not add value.  A CLI is basically a translation pipe between stdin/stdout and a REST endpoint, I don't see the value in building an internal model to just turn around and dump it out again.  it isn't like an intermediate compiler stage that needs to do transforms and optimizations before generating code.  A20:05
*** briancurtin has quit IRC20:07
dtroyeranyway, deal-breaker #1 is KSC auth compatibility.  The now-external-to-ksc plugins would need to work, or be re-implemented, to do any change.  auth comes first.  then we can look at say implementing the network commands using sdk and eliminate neutronclient.20:07
dstufftdtroyer: so first off, I certainly agree that openstack loves to over-engineer things :)20:08
dstufftit may be the case that the object model itself isn't useful for python-openstackclient, I think my question there would be, is it hurtful to python-openstackclient? and if it is hurtful to python-openstackclient is it hurtful to other applications? Is it beneificial to other applications? If it's hurtful maybe sdk can layer itself or otherwise expose a "lighter" API if we understand why it's hurtful.20:14
dstufftThere are some obvious benefits (I think?) to having osc consume the sdk in reduction of effort both for the people making both tools and for vendors wanting to support them. So I i'd love to figure out how can bridge the gap :) It seems like KSC auth is an important step so I'll see about poking that first.20:14
*** briancurtin has joined #openstack-sdks20:17
dtroyerI don't have the tools to quantify harmful in a performance aspect, but against the turn-around times of a REST call, that's probably not useful, other than the overall load times for all of the required modules.  But I do think that unnecessary complexity is harmful.  compare neutronclient to any of the others.  there's a reason they are ready to dump it wholesale, and part of that is they way over-did the object inheritanc20:22
dtroyerthat is one reason there is a lot of seemingly duplicated boilerplate in the OSC command classes.  it is simpler to maintain and adjust and not break other bits.20:23
terrylhoweneutronclient does actually return dicts though dtroyer most of the other clients return a resource class of some sort20:29
dtroyerterrylhowe: yes, an in some cases that's good and others not so good.  but the code structure complexity remains in either case20:31
terrylhoweI’ll agree that the way neutronclient does their commands is overly complex20:31
dstufftdtroyer: sure, it's always a good thing to cutout unnecessary complexity, the key question is whether it's unnecessary for python-openstackclient or whether it's unnecessary for *everyone* (or most everyone). Like most abstractions it's going to be more or less useful in certain situations and the key is finding the balance between being an architecture astronaut and just telling people use a HTTP library (and I don't know that the sdk20:40
dstufft hits that balance correctly, I'm just starting to really look at it and form my opinions on it!)20:40
*** tellesnobrega_ has quit IRC20:41
*** pm90_ has quit IRC21:20
*** pm90_ has joined #openstack-sdks21:23
*** pm90_ has quit IRC21:29
*** sigmavirus24 is now known as sigmavirus24_awa21:55
briancurtinterrylhowe: while redoing those volume changes, i wanted to improve so it yields resources directly (rather than returning a list that Resource.list then turns into resource objects), which will make non-paginated but still list-returning REST APIs look the same. however, i broke Resource.find...but then i also found that it doesn't actually21:56
briancurtinseem to work for anything i try. are there services that respect these query parameters?21:56
*** pm90_ has joined #openstack-sdks21:58
terrylhowehmm, find was workign for me22:03
terrylhoweoh, after the change find didn’t work22:03
terrylhowewhat is the motivation to have page call yield?  I thought it was kind of nice to have something low level where you don’t have to worry about iterating22:04
briancurtinterrylhowe: i need something that can work with only one page of data and i dont want to have to put this code everywhere (at the end): - or, what i had previously done in the proxy for that  (at the top) -
briancurtinType.list by itself will just be that infinite loop, but turning Resource.list into a general purpose method that can handle paginated but also non-paginated responses is far too hard and penalizes with extra calls22:08
briancurtinand this behavior appears in a bunch of other places - it was originally reported as a problem with parts of network (where we saw that /networks can be paginated but it can also be disabled)22:10
briancurtinterrylhowe: so could stay as is, but I need something higher-level than it but that isn't Resource.list...and we're running out of names for what that is :)22:14
terrylhowelet me look at those changes I had marked as WIP briancurtin and look at some notes I had for things I wante to do in this area22:15
briancurtinok cool22:15
terrylhoweseems like I had some thoughts and I don’t recall what they are right now22:15
openstackgerritTerry Howe proposed stackforge/python-openstacksdk: allow paging when supported
terrylhowebriancurtin: ^^ I’m not thrilled with another flag to set, but that works nicely22:24
terrylhowedefinitely a WIP22:24
briancurtinthat might work. will think about it22:28
terrylhowethe one thing about it is it would be nice if the user could turn on/off paging at the service level instead of manually like that on the resource briancurtin22:30
terrylhowemaybe some kind of session.allow_paging(cls.service) call22:31
briancurtini wonder if service level is too heavy of a hammer. volume is currently entirely unpaginated, but i could see /types (which is small) being ok unpaginated, but /volumes (maybe being large) being paginated. i would always want to call /types as unpaginated, but i may still want /volumes paginated22:33
*** tellesnobrega_ has joined #openstack-sdks22:39
*** chlong has joined #openstack-sdks23:17
*** tellesnobrega_ has quit IRC23:39
*** openstack has joined #openstack-sdks23:42
*** bknudson has quit IRC23:42
*** ayoung has quit IRC23:57

Generated by 2.14.0 by Marius Gedminas - find it at!