Tuesday, 2017-04-18

*** TheJulia has quit IRC02:26
*** gouthamr has quit IRC02:27
*** TheJulia has joined #openstack-shade02:29
*** gkadam has joined #openstack-shade03:15
*** gkadam is now known as gkadam-brb03:15
*** gkadam-brb is now known as gkadam03:46
*** jamielennox is now known as jamielennox|away05:12
*** jamielennox|away is now known as jamielennox05:17
*** yfried has joined #openstack-shade06:21
yfriedhi, any core members around?06:42
*** ioggstream has joined #openstack-shade07:22
*** ioggstream has quit IRC07:32
*** jamielennox is now known as jamielennox|away07:34
*** ioggstream has joined #openstack-shade07:56
*** jamielennox|away is now known as jamielennox08:28
*** openstackgerrit has quit IRC08:33
*** openstackgerrit has joined #openstack-shade08:51
openstackgerritMonty Taylor proposed openstack/os-client-config master: Add ability to pass in user_agent  https://review.openstack.org/45255008:51
mordredyfried: sup?09:01
yfriedmordred: hi09:08
yfriedmordred: how are you online so early?09:08
openstackgerritMonty Taylor proposed openstack-infra/shade master: Pass in app_name information to keystoneauth  https://review.openstack.org/45626809:11
mordredyfried: sometimes one wakes up early in the morning for no good reason09:11
yfriedmordred: good for me :)09:11
yfriedmordred: we have a problem with shade dep09:11
yfriedmordred: I wonder if you could somehow help me09:12
yfriedmordred: https://github.com/openstack-infra/shade/commit/15d64f2af5d9f45ba91bc7f4f5f48402ab8c339b09:12
yfriedmordred: we were hit by this09:12
yfriedmordred: we are looking for a reliable way to depend on shade, but it keeps breaking us09:13
yfriedmordred: you guys ususually won't see it in shade or in ansible, but we are hit when argparse is evaluating dependencies09:13
yfriedpkg_resources.ContextualVersionConflict: (pbr 2.1.0 (/tmp/ir-venv-Bt1WP4k/lib/python2.7/site-packages), Requirement.parse('pbr!=2.1.0,>=2.0.0'), set(['openstacksdk']))09:14
yfriedmordred: is there a way to make shade dep imprevious to this kind of stuff?09:14
mordredyfried: hrm - well - lemme look real quick09:15
mordredyfried: (we just started following global-requirements in shade so it's a new process to us)09:16
yfriedmordred: I try to be patient, but my users aren't so accomodating09:17
mordredyfried: oh - are you depending on shade master?09:17
yfriedmordred: no09:17
yfriedmordred: I'm pining shade (currently 1.19.0)09:17
yfriedmordred: and testing every release before bumping it09:18
yfriedmordred: I'm doing the same with ansible09:18
yfriedmordred: but we (and you also, though you might not see this) are being hit by clients bump becuase shade have uncapped deps09:19
mordredI mean - I ask because that patch just landed a few hours ago09:19
yfriedpython-novaclient>=7.1.0 # Apache-2.009:19
mordredyah. well, _that_ I'm working on fixing as quickly as humanly possible09:19
yfriedmordred: yes, and we had to do the same for our req.txt untill we have a shade release that fixes it09:19
yfriedfor example, if you have a novaclient release that breaks, we are breaking top09:20
yfriedtoo09:20
mordredyes. this is why we're removing the deps on novaclient09:20
yfriedmordred: novaclient is one example :)09:20
mordredbut I'm confused - are you saying the issue is that you need https://github.com/openstack-infra/shade/commit/15d64f2af5d9f45ba91bc7f4f5f48402ab8c339b in a release?09:20
mordredyfried: I know - but the client libs are the source of almost all of these problems for shade09:21
mordredso the solution is removing their deps :)09:21
yfriedmordred: we got hits by other clients, as well as oslo09:21
yfriedmordred: I'm saying that we had the same issue you did and had push the same fix09:21
mordreddhellmann: ^^ btw, when you wake up, let's dig in to the above09:22
yfriedmordred: how can you remove clients dep from shade09:22
yfried?09:22
mordredyfried: we are switching to making direct rest calls09:22
mordredshade has already dropped glanceclient, heatclient, magnumclient, etc. morgan is almost done removing keystoneclient, I'm working on novaclient, we have someone working on cinderclient ... hoping to be done soon09:23
yfriedmordred: like tempest? really?09:23
mordredyes. the python client libraries are awful09:23
yfriedmordred: what about cinder?09:23
mordredall of them09:23
mordredwe will literally not use any of them soon09:23
yfriedsorry, I meant Neutron09:23
mordredyup. that's coming soon09:24
yfriedmordred: why not join hands with tempest on this?09:24
mordredbecause tempest and shade have opposite needs09:24
yfriedmordred: I'm asking about Neutron specifically, as I know it's a risky one09:24
mordredneutron is actually the easiest one because neutronclient has almost no logic in it - it's one of the most direct passthrough libs09:25
yfriedmordred: for now, can you plz cap the clients?09:25
yfriedmordred: to the currently working version?09:25
mordredno, I cannot. it is literally forbidden by openstack policy09:25
mordredand there is nothing I can do about that09:25
mordreddhellmann: ^^09:26
yfriedmordred: ok09:26
mordredbut I can make a release that contains that requirements fix if you like09:26
yfriedmordred: would be appreciated09:26
yfriedmordred: might I suggest a periodic gate that tests for this breakage?09:26
yfriedmordred: we have one running every 1hr on our latest and stable branches09:27
yfriedmordred: sending emails in case of breakage so we can fix it before it hits users09:27
mordredyah - I'll try to get that working today09:27
yfriedmordred: I would like to subscribe to such shade early warning system09:28
mordredyfried: can you give me a quick example of something that causes the error you saw? I just installed latest shade from pip then imported it and it worked fine - so I want to make sure I'm testing what's breaking you09:29
openstackgerritMonty Taylor proposed openstack-infra/shade master: Remove dead ImageSnapshotCreate task  https://review.openstack.org/45754309:32
yfried    from pkg_resources import load_entry_point09:33
yfriedwe have a CLI entry point09:36
yfriedit fails on     load_entry_point('infrared', 'console_scripts', 'infrared')()09:36
yfriedhttp://paste.openstack.org/show/606985/09:37
yfriedmordred: ^09:37
mordredyfried: thanks. I have made a script locally that also breaks09:40
yfriedmordred: let me review when you are ready. plz09:41
mordredyfried: this: http://paste.openstack.org/show/606987/ shows the error -although it also shows an error on a conflict with requests too09:47
mordredyfried: I'll work with dhellmann when he's up to figure out a better way to catch this a little more broadly (the issue is that openstacksdk released with a cap but other things don't have matching releases with the same cap)09:47
mordredof course, the 'funny' part is that shade doesn't even use openstacksdk - so the fact that openstacksdk made a release and it broke us is just super sadmaking09:48
mordred(and will be fixed by removing more depends - this time ironicclient, which depends on python-openstackclient which depends on openstacksdk - so maybe I should put ironicclient higher on the list to remove)09:49
openstackgerritMerged openstack-infra/shade master: Upgrade list volumes tests to use requests-mock  https://review.openstack.org/45673609:50
mordredyfried: sorry it's taking so long to remove the depends and that it's causing you issues - I very much am ready to be done fixing this problem :)09:51
yfriedmordred: that's good to know09:54
yfriedmordred: do you have a storyboard I can track about this?09:54
yfriedmordred: also, I have several ppl who might like to contribute to this effort (not making any promises)09:55
yfriedmordred: in the past, tempest had the same effort so we helped a little.09:55
yfriedmordred: If you have a task list ppl can pull from I might be able to lighten your load a little09:56
mordredyfried: awesome. we do have a storyboard for it: https://storyboard.openstack.org/#!/worklist/195 - but I need to break it out into more tasks (that list isn't super useful right now is it?)10:01
mordredyfried: so I'll do that and then let you know about it10:01
yfriedmordred: tnx10:01
yfriedmordred: I'd like to publish this so I'll wait for you to break it down and let my ppl know they can help10:02
yfriedmordred: storyboard always fails on the first try. I have to log out and log back in :(10:02
yfried400: GET /api/v1/users/preferences: Invalid input for field/attribute user_id. Value: 'preferences'. unable to convert to int10:03
mordredyfried: yah. I wish I knew why that was the case :(10:03
mordredand awesome - thanks!!!10:03
mordred(I'm going to work this morning on getting nova security group transitioned, because novaclient v8 doesn't have security group support anymore and I'd love to get that fix into this morning's release10:04
mordredthen I'll do storyboard breakdown - then likely try to get ironicclient done so that we can drop a bunch of depends10:04
openstackgerritMonty Taylor proposed openstack-infra/shade master: Remove dead ImageSnapshotCreate task  https://review.openstack.org/45754310:05
openstackgerritMonty Taylor proposed openstack-infra/shade master: Pass in app_name information to keystoneauth  https://review.openstack.org/45626810:05
openstackgerritMonty Taylor proposed openstack-infra/shade master: WIP Transition nova security groups to REST  https://review.openstack.org/45754810:05
yfriedmordred: could you plz explain why tempest rest clients can't work for you?10:06
mordred2 reasons10:08
mordredfirst - tempest clients are designed to test that openstack clouds work properly - which they should be - it's very important10:08
mordredbut shade clients need to handle it for the user when the clouds _don't_ work properly and hide it10:08
mordredso they have opposite desires in life10:09
mordredfor instance, in tempest-lib's create_flavor is this line:10:09
mordred        self.validate_response(schema.create_get_flavor_details, resp, body)10:09
mordredthat's _awesome_ - but it's not the thing we want/need in shade - we have to be very permissive and work around people doing evil things10:09
mordredthe second is much smaller, which is one of the main things we've learned with the client libs is that a wrapper library doesn't add much value for us- making the rest calls just using keystoneauth is easier than using another library10:10
mordredoh - also - tempestlib is not built on top of keystoneauth10:13
mordredit does its http/auth itself - which is bad for shade/ansible - we need to be able to use all of the keystoneauth auth plugins that exist - again, I think tempest made the right choice in _not_ using that so that they can be an independent validation that things work, but for us we need to consume keystoneauth10:14
*** ioggstream has quit IRC11:13
*** rcarrillocruz has quit IRC11:34
*** yolanda has quit IRC11:44
*** yolanda has joined #openstack-shade11:44
*** ioggstream has joined #openstack-shade11:48
*** yolanda has quit IRC11:49
*** yolanda has joined #openstack-shade11:50
*** yolanda_ has joined #openstack-shade11:56
*** yolanda has quit IRC11:58
*** yolanda_ has quit IRC12:19
*** yolanda_ has joined #openstack-shade12:19
*** gouthamr has joined #openstack-shade12:31
*** yolanda_ has quit IRC12:31
*** yolanda_ has joined #openstack-shade12:31
*** yolanda_ has quit IRC12:43
*** yolanda_ has joined #openstack-shade12:44
*** gkadam has quit IRC12:44
*** Matias has joined #openstack-shade12:53
*** yolanda_ has quit IRC13:04
*** yolanda_ has joined #openstack-shade13:05
*** yolanda_ is now known as yolanda13:05
rodsmordred I can switch from cinder to ironic if it has a higher priority14:11
mordredrods: sure! ironic definitely is responsible for pulling in a pile of dependencies that we don't actually use14:13
mordredrods: but honestly removing both are important, so whichever one sounds more exciting :)14:14
rodsok, I'll stick with cinder to avoid context switching :)14:17
mordredcool. I'll rock on the ironicclient stuff after I finish this lovely nova-network security group support (I do so much love ensuring features almost nobody is probably using work)14:21
rodseheh :)14:21
openstackgerritRosario Di Somma proposed openstack-infra/shade master: Use REST for cinder list volumes  https://review.openstack.org/45766514:30
mordredwoot!14:31
rodsok, gonna switch on same paid work and get back on this later in the evening :)14:32
rodsmordred thx for helping me to get up and running14:33
rods*some14:33
mordredrods: thanks for the help!14:34
openstackgerritMonty Taylor proposed openstack-infra/shade master: Transition nova security group tests to REST  https://review.openstack.org/45754815:18
openstackgerritMonty Taylor proposed openstack-infra/shade master: Remove dead ImageSnapshotCreate task  https://review.openstack.org/45754315:18
openstackgerritMonty Taylor proposed openstack-infra/shade master: Pass in app_name information to keystoneauth  https://review.openstack.org/45626815:18
openstackgerritMonty Taylor proposed openstack-infra/shade master: Replace nova security groups with REST  https://review.openstack.org/45769015:18
mordredShrews: if you have a sec ^^ I'd love to get those landed and get a release cut15:18
mordredShrews: there's a dependency issue yfried is running in to due to openstacksdk having released wiht a pbr pin but we haven't yet done so as well15:19
clarkbmordred: you shouldn't pin pbr15:20
mordredclarkb: I agree15:20
mordredthere was an exclusion that went out in global-requirements updates15:20
Shrewsmordred: did you fix the py35 fails in that iteration?15:32
mordredShrews: I did15:32
Shrewscool. i'll look again15:32
mordredShrews: oh - I'm going to push up a follow up on that - I covered over an issue and it can be fixed better15:41
Shrewsmordred: addressing Jens' comment?15:42
mordredShrews: yup15:43
slaweqih15:48
Shrewsslaweq: hello. welcome15:56
slaweqShrews: hello15:56
slaweqsorry for my last message, it was typed by mistake :)15:57
*** rcarrillocruz has joined #openstack-shade16:05
*** rcarrillocruz has quit IRC16:07
*** rcarrillocruz has joined #openstack-shade16:07
*** yfried has quit IRC16:12
*** rcarrillocruz has quit IRC16:18
*** rcarrillocruz has joined #openstack-shade16:18
mordredShrews: you're going to LOVE the next two patches16:26
openstackgerritMonty Taylor proposed openstack-infra/shade master: Actually fix the app_name protection  https://review.openstack.org/45771716:33
openstackgerritMonty Taylor proposed openstack-infra/shade master: Futureproof keystone unit tests against new occ  https://review.openstack.org/45771816:33
mordredShrews: please enjoy the wonder that is https://review.openstack.org/45771816:33
mordredShrews: I accidentally stumbled on that while testing the app_name patch16:34
samueldmqmorning shade16:37
mordredmorning samueldmq16:38
mordredsamueldmq: enjoy https://review.openstack.org/457718 ... I think you'll "like" it16:38
samueldmqmordred: o/ I am taking a look at restification ...16:38
* samueldmq looks16:38
mordredsamueldmq: it's the result of http://git.openstack.org/cgit/openstack/os-client-config/commit/?id=01ff292e078206e487751228be4a7062ba0c6048 fwiw16:38
Shrewsmordred: that's special16:41
mordredShrews: right?16:41
mordredShrews: I mean, luckily _all_ of that is getting better - but yay16:41
samueldmqmordred: I got what your patch does16:43
samueldmqmordred: just did not what changed in the specific bits on how ksc is constructed16:43
mordredsamueldmq: we _used_ to pass "kwargs['interface'] = plugin.AUTH_INTERFACE" to ksc16:45
mordredbecause there was a time sometime in the past when not doing that confused ksc16:45
mordred(plugin.AUTH_INTERFACE being a special singleton)16:45
samueldmqmordred: yes I see mock_session.get_endpoint.assert_called_with(interface=ksa_plugin.AUTH_INTERFACE) in the occ change16:46
mordredbut now that we actually pass the interface value like normal - it somehow changed how ksc did discovery :)16:46
mordredsamueldmq: now - _why_ that happens I have no idea :)16:47
samueldmqwow , I have no idea either :-)16:47
samueldmqmordred: commented on https://review.openstack.org/#/c/45771816:48
mordredbtw - I have learned when doing restification of nova things, that running the unittests under py35 is nicer because novaclient is stricter on data in 3.5 than in 2.716:48
samueldmqnice16:49
samueldmqmordred: btw how did you figure that out ? the keystone unit tests thing ... ?16:49
samueldmqmordred: is there a non-voting test job for shade in occ?16:49
mordredsamueldmq: here's the http trace for a test that uses that v2 codepath: http://paste.openstack.org/show/607032/16:51
mordredsamueldmq: I think the first comment you have is fine because of the way we construct the catalog and the client - I agree with the second one :)16:51
mordredsamueldmq: and we do - but only for devstack/functional tests ... we don't have anything that runs occ-master for shade unittests (which _most_ of the time should not matter)16:52
samueldmqmordred: OK so if it's a v2 client it will append /v2.0 to the url anyways (as per your paste)16:53
openstackgerritMonty Taylor proposed openstack-infra/shade master: Actually fix the app_name protection  https://review.openstack.org/45771716:54
openstackgerritMonty Taylor proposed openstack-infra/shade master: Futureproof keystone unit tests against new occ  https://review.openstack.org/45771816:54
openstackgerritMonty Taylor proposed openstack-infra/shade master: Transition nova security group tests to REST  https://review.openstack.org/45754816:54
openstackgerritMonty Taylor proposed openstack-infra/shade master: Replace nova security groups with REST  https://review.openstack.org/45769016:54
openstackgerritMonty Taylor proposed openstack-infra/shade master: Change versioned_endpoint to endpoint_uri  https://review.openstack.org/45773216:54
mordredShrews: fixed the python3 test failure in https://review.openstack.org/457548 - and learned that novaclient is stricter in python3 - which is awesome16:55
samueldmqmordred: FYI you just rebased https://review.openstack.org/#/c/457718 rather than updating it :)16:56
samueldmqmordred: what's the motivation for restification?16:56
samueldmqso that I make sure I fully understand what's behind the effort before doing it for keystone16:57
samueldmqas I see, we are moving from tasks (which use the clients) to rest calls16:57
samueldmqhmmm in order to assert the APIs are backward compatible instead of the python clients???16:58
mordredsamueldmq: yes - I fixed it in the followup change16:58
mordredsamueldmq: well - yes - we have a different worldview on api compat from the clients - so, for instance, novaclient v8 just removed support for nova-network16:59
mordredthis is a good thing, of course, it's a part of getting rid of nova-network16:59
mordredbut in shade, we still need to suppor that16:59
samueldmqmordred: no I think you forgot to add the change, https://review.openstack.org/#/c/457718/1..2 is empty :-)16:59
mordredsamueldmq: yah - I added it in a followup patch16:59
mordredhttps://review.openstack.org/#/c/457732/116:59
samueldmqmordred: ++17:00
samueldmqmordred: so you keep supporting nova-network in shade17:01
samueldmqvia the rest calls17:01
mordredsamueldmq: the other reason is that when shade depends on python clients, it makes it harder for deployers and users to use it, because they may not be running the most recent version of openstack, but shade would depend on the most recent version of client libs17:01
mordredso it made it hard for people to install/package17:01
mordredsamueldmq: that's right17:01
mordredand using requests_mock we can even continue to test it, at least from a regression perspective, even when we stop having devstack that can deploy it17:02
samueldmqmordred: makes sense,  and from now on (when we migrate all to rest) they'll just need to have the serviecs running17:02
mordredyup17:02
mordredexactly17:02
Shrewsmordred: i'm confused as to why 457543 fails functional-libs-nv17:04
samueldmqmordred: cool, I am taking notes, I may need to write details like this later in a thesis or something ;)17:04
*** ioggstream has quit IRC17:06
mordredShrews: looks like we just go lucky and had a devstack timeout17:07
mordredShrews: the next patch in the series passed functional-libs-nv in its run17:08
Shrewsyeah17:09
*** rods has quit IRC17:15
*** yfried has joined #openstack-shade17:30
openstackgerritSamuel de Medeiros Queiroz proposed openstack-infra/shade master: Move user list to REST  https://review.openstack.org/45774517:43
samueldmqmordred: ^ I think I am doing the right thing. I also left a couple of thoughts as comments...17:43
*** yfried has quit IRC17:46
*** ioggstream has joined #openstack-shade18:21
*** rods has joined #openstack-shade18:32
mordredsamueldmq: looks great - left some responses18:41
*** cdent has joined #openstack-shade18:58
samueldmqmordred: shouldn't it work with the unversioned endpoint ?19:01
samueldmqmordred: hmm it's not using the ksc/ksa version discovery thing, correct ? and doing direct HTTP calls instead19:01
samueldmqis that correct?19:01
*** ioggstream has quit IRC19:01
mordredI mean, it should be using ksa but then doing direct http calls19:02
mordredit's possible it's not doing _enough_ discovery via ksa19:02
mordredor that we need to do some different discovery19:02
samueldmqmordred: how does that work for the auth_url now ?19:02
samueldmqmordred: does it discover properly ?  I think so in that case19:02
samueldmqbut not for keystone API (not auth) calls19:03
mordredwell, auth_url works - that comes from config (we assume you must have an auth_url to start)19:03
mordredthen we do get the catalog, which we have fixtures for in tree19:03
mordredand I know _something_ does keystone version discovery after the catalog (since we have those calls to GET https://identity.example.com/)19:03
mordredBUT - self._identity_client may not be doing the keystone version discovery stuff it needs to yet19:04
mordredso that might be step one19:04
samueldmqmordred: so that could be done by calling something in ksa/ksc? not sure19:04
samueldmqksa is auth, and auth_url version discovery works properly already19:04
mordredsamueldmq: yah - not sure either. I have code in shade to do it for glance19:04
samueldmqnot sure it needs to know version discovery for the APIs19:05
mordredright. I agree - that starts to get weird19:05
samueldmqmordred: I will do that in shade and add jamielennox to the review, he will -3 me if that's wrong19:05
samueldmq:-)19:05
mordredI think doing it for now in a private method in shade is fine - and probably once we get one or two more done we may find that it's a pattern we _do_ want os-client-config to know how to do19:05
mordred"os_client_config.please_get_me_the_endpoint_from_version_discovery()" :)19:06
mordredsamueldmq: and yes he will :)19:06
samueldmqmordred: agreed, I will see what I can do there with the URL available + self.cloud_config.get_api_version('identity')19:07
mordredpatch bomb incoming19:13
openstackgerritMonty Taylor proposed openstack-infra/shade master: Actually fix the app_name protection  https://review.openstack.org/45771719:14
openstackgerritMonty Taylor proposed openstack-infra/shade master: Change versioned_endpoint to endpoint_uri  https://review.openstack.org/45773219:14
openstackgerritMonty Taylor proposed openstack-infra/shade master: Futureproof keystone unit tests against new occ  https://review.openstack.org/45771819:14
openstackgerritMonty Taylor proposed openstack-infra/shade master: Transition nova security group tests to REST  https://review.openstack.org/45754819:14
openstackgerritMonty Taylor proposed openstack-infra/shade master: Replace nova security groups with REST  https://review.openstack.org/45769019:14
openstackgerritMonty Taylor proposed openstack-infra/shade master: Remove extra unneeded API calls  https://review.openstack.org/45777519:14
mordredShrews: sorry - had to fix a broken test earlier in the stack - turns out it's really nice that we have  both functional and unit tests ... keeps us honest19:14
*** ioggstream has joined #openstack-shade19:41
morganmordred: damn i need to rebase my rebase now20:04
morganmordred: something changed and broke it again :P20:04
* morgan goes back to working on that.20:04
mordredmorgan: sorry bout that20:04
mordredmorgan: oh, also, samueldmq just started looking at switching list_users - y'all might be able to keystone-language at each other :)20:05
morganmordred: nah, really i just took too long20:05
samueldmqmordred: morgan o/20:06
samueldmqmorgan: I just started ... https://review.openstack.org/#/c/457745/20:06
morgansamueldmq: cool20:08
mordredmorgan: if you have a sec, jamielennox has blessed this already: https://review.openstack.org/#/c/452550/20:13
morganlooking20:13
mordredShrews: stack ending on https://review.openstack.org/#/c/457775/1 finally passes all the tests (sorry for the false starts earlier)20:17
morgan+220:17
mordredmorgan: woot! thanks!20:17
morganwitll +A as long as no one complains in the next few das20:17
morganys20:17
morganor feel free to self +A.20:17
morganbut LGTM20:17
mordredmorgan: maybe I'll even browbeat Shrews into approving https://review.openstack.org/#/c/452550/ :)20:18
morganhehe20:18
* mordred hands Shrews a box of brows20:18
openstackgerritSamuel de Medeiros Queiroz proposed openstack-infra/shade master: _discover_latest_version is private and not used  https://review.openstack.org/45778720:21
samueldmqmordred: ^ are there intentions on using this function in the future ?20:22
mordredsamueldmq: I imagine it was just there while starting to think about that topic20:23
Shrewsenough review browbeating, plz20:23
* samueldmq nods20:24
openstackgerritMerged openstack-infra/shade master: Pass in app_name information to keystoneauth  https://review.openstack.org/45626820:29
openstackgerritMerged openstack-infra/shade master: Remove dead ImageSnapshotCreate task  https://review.openstack.org/45754320:29
mordredShrews: that's all the browbeating I have for today20:29
mordredsamueldmq: and thank you!20:29
samueldmqnp20:30
samueldmqmordred and Shrews the only core-reviewers in shade?20:31
samueldmqI was trying to look up at LP but looks like shade is not using LP20:32
*** openstackgerrit has quit IRC20:33
mordrednope. we use the storyboard. infra are all cores on it too, as well as SpamapS - but it's been a while since SpamapS has done anything20:33
mordredso - you know, super happy to get people in a core-able place - we're a bit strict on it since it's a super-heavily used library in infra, so brekaing it can break the entire gate - but other than that we'd love to have more folks with the codebase in their head20:36
*** yfried has joined #openstack-shade20:41
samueldmqmordred: nice20:47
samueldmqmordred: when using the ksclient, where does the version come from?20:54
samueldmqI am trying to figure out if will need to support /v2.0 URLs at all for shade (the the APIs not auth)20:55
mordredsamueldmq: yes - we need to still support the v2.0 urls20:55
mordredsamueldmq: for now you can assume that the cloud_config.get_api_version('identity') is correct20:56
samueldmqmordred: what's the format of that ?20:56
samueldmqmordred: v3 / v2.0 ? 3 / 2 ?20:56
mordred3 / 220:56
*** openstackgerrit has joined #openstack-shade20:56
openstackgerritMerged openstack-infra/shade master: Transition nova security group tests to REST  https://review.openstack.org/45754820:56
mordredsamueldmq: sorry, I lied: http://git.openstack.org/cgit/openstack/os-client-config/tree/os_client_config/defaults.json20:57
mordredseems to be 2.0 or 320:57
samueldmqmordred: kk: if URL is unversioned: url = url + ('/v3' if cloud_config.get_api_version('identity') == '3' else '/v2.0')20:58
samueldmqmordred: I'm assuming it's okay to assume v2.0 if it's not 3 (2.0 or unspecified?)20:58
samueldmqanyways, impl specific, you'll look at the review anyways20:59
mordredit will always have a value - so it is safe to assume it has one20:59
samueldmqnice thanks20:59
mordredso we should never just append strings to the url - we want to consume the catalog and version discovery as much as possible20:59
mordredsamueldmq: and then, as we get better at version discovery, shade shoudl know what's the latest version it knows how to deal with, and if the user hasn't specified something different, we should try to use the latest version of the api we understand21:00
mordredthat's future though ... for right now, there will always be a "requested" version from shade's pov21:00
mordredjust mostly babbling about philosophy21:00
openstackgerritMerged openstack-infra/shade master: Replace nova security groups with REST  https://review.openstack.org/45769021:04
openstackgerritMerged openstack-infra/shade master: Actually fix the app_name protection  https://review.openstack.org/45771721:04
samueldmqmordred: ok, but for now we can just append the identity_api_version correct ?21:05
*** slaweq has quit IRC21:05
samueldmqmordred: I mean, there is no /v3.6 for example, so looking at the catalog is not useful I think21:06
samueldmqright now21:06
openstackgerritMerged openstack-infra/shade master: Futureproof keystone unit tests against new occ  https://review.openstack.org/45771821:06
openstackgerritMerged openstack-infra/shade master: Change versioned_endpoint to endpoint_uri  https://review.openstack.org/45773221:06
mordredright - but ... so we should use whatever the catalog returns to us, no?21:06
mordredlemme go get a few examples - it might make it easier to reason about21:06
samueldmqsure21:07
*** cdent has quit IRC21:07
mordredsamueldmq: ok. this is actually really good - it's super confusing so it's very clear I need to write a document21:14
samueldmqmordred: hehe kk21:16
*** yfried has quit IRC21:36
openstackgerritMerged openstack/os-client-config master: Add ability to pass in user_agent  https://review.openstack.org/45255021:41
*** gouthamr has quit IRC21:50
openstackgerritSamuel de Medeiros Queiroz proposed openstack-infra/shade master: Move user list to REST and build API URL  https://review.openstack.org/45774522:02
mordredsamueldmq, morgan: https://etherpad.openstack.org/p/shade-identity-versions22:06
mordredsamueldmq, morgan: that is half narrative and half pseudo-code22:06
morganhehe22:06
mordredbut I hope should explain (in bad words) what should happen - does it make sense?22:06
samueldmqmordred: I am looking, I also left another idea in that patch22:06
mordredsamueldmq: yah - so, the version in the catalog isn't necessarily latest available- and there are both versioned and unversioned endpoints in the catalog out in the wild :)22:08
pabelangerohai!22:09
pabelangerthis is likely for mordred, but if anybody else knows....22:09
mordredpabelanger: uhoh22:09
pabelangernodepool seems to be aggressive with NeutronFloatingipList against tripleo-test-cloud-rh1:  eg Manager tripleo-test-cloud-rh1 ran task NeutronFloatingIPList in 0.0846412181854s22:10
pabelangercurious what I could do to maybe see what is going on22:10
*** ianw_pto is now known as ianw22:12
mordredpabelanger: yah - it'll totally be doing floating ip list for each server22:12
mordredpabelanger: however, we've got a patch in trunk that should reduce that a little bit22:12
mordredso that we'll only ask for the floating ip when we're done with the server, and not on every poll22:13
pabelangerah, okay22:13
pabelangercool, then I'll hold off looking at it until we use latest shade22:13
mordredpabelanger: I think tl;dr - we're very close to making a release - probably first thing in the morning and it shoudl fix that problem22:13
mordredpabelanger: ++22:13
mordredsamueldmq: you may hate me by the end of that document ...22:14
samueldmqmordred: :-)22:14
mordredsamueldmq: but I _think_ I got all of the edge conditions down - sorry it's not the best structured22:16
samueldmqmordred: no problem, I've got the overall idea, need to re-read it though22:16
samueldmqmordred: is there any relationship between client.endpoint_override and identity_api_version today?22:17
mordredsamueldmq: yah - it's ... both complicated and disorganized22:17
mordredsamueldmq: no, not this instant - at the moment client is just mounted on whatever is in the catalog22:17
mordredwell - let me rephrase22:17
mordredfor keystone rest client- there is on relationship22:18
mordredfor keystoneclient, identity_api_version gets passed to the keystoneclient constructor22:18
mordredso there _is_ a relationship in released shade - but not in the _identity_client right now (which is one of the issues)22:18
samueldmqmordred: since we're dropping ksc, couldn't we drop that info too ?22:19
samueldmqand just discover what's available from catalog...22:19
samueldmqI mean, the less we get the user to specify the better22:20
mordredno - it's important info for the user to pass to us - sometimes clouds that people are using are broken in some way, so they need to be able to tell us "I don't care what the cloud says it can do, please do this other thing"22:20
samueldmqif we can figure that out22:20
mordredif we don't let the user tell us the cloud is wrong, the user can be broken with no recourse. BUT - we should also not require the user to tell us anything :)22:20
samueldmqmordred: like even if v3 isn't in the catalog, use v3 anyways22:21
samueldmq?22:21
mordredyah22:21
mordredfor instance, vexxhost right now has v2 in the catalog, but they have v3 and it works, so I can set identity_api_version to 3 to make use of it22:21
samueldmqok so the whole problem is the combinations between user explicitly specifying the version22:21
mordredbut if I don't set anything, I should get v2 like the catalog tells me22:21
mordredyup22:22
samueldmqand what comes in the URL of the catalog22:22
mordredyup22:22
samueldmqand does the URL in auth matter?22:22
mordredAND - the fact that some clouds put versioned and some clouds put unversioned in the catalog :)22:22
mordrednot for identity, no22:22
mordredthe auth_url should not matter for this at all22:22
samueldmqok so we completely ignore auth_url22:22
samueldmq++22:22
mordredyup22:22
samueldmqcool I think we are in the same page now22:22
mordredexcellent22:23
samueldmqoh I just CR -1 my own patch instead of W-122:23
mordredcool22:23
samueldmqmordred: I will update a patch by its own for the discovery thing and base the user list one on it22:24
mordredalso - welcome to the fun part of shade - solving tons of conflicting and overlapping problems! :)22:24
samueldmqso we can make sure it works22:24
mordredsamueldmq: sounds like a great plan to me22:24
clarkband doing it without making new problems22:24
mordredclarkb: yup22:24
samueldmqmordred: yay that's nice. I am looking forward to be doing the same for other projects too22:24
samueldmqI think this is a good way to have a good understanding of openstack overall22:24
mordredclarkb: btw - if you feel like reading that etherpad - I believe I've got everything22:24
mordredsamueldmq: ++22:24
samueldmqrather than just keystone.22:24
mordredsamueldmq: you will definitely wind up learning _way_ too many things22:25
samueldmqclarkb: o/22:25
samueldmqmordred: nice, I am looking forward to22:25
clarkbthe shade-identity-versions one?22:25
mordredclarkb: but it's possible you'll spot a logical flaw: https://etherpad.openstack.org/p/shade-identity-versions (it's also poorly organized)22:25
clarkbsamueldmq: ohai22:25
mordredyah22:25
mordredclarkb: I correct myself at the end - so don't be too bothered when it doesn't at first honor identity_api_version22:26
clarkbdo you think it should be stronger about rejecting things if the specified version is not useable? currently its a warning and tries to use latest if I read it right22:27
mordredI think I could go either way on that...22:28
clarkbI don't think one version of keystone is less secure than another right?22:29
clarkbif there is a case where ^ could be true then I think you'd want to fail hard but in this case its not about http vs https or passwd vs cert, its purely api version22:29
mordredI do not believe so, no - and also auth and CRUD are different22:29
mordredso identity_api_version does not affect which version you auth with22:30
clarkbwait what22:30
mordredthey're not related at all22:30
mordredauth version is that's controlled by auth_type and/or what auth parameters you have22:30
mordredso you can totally have auth_type: v3password and identity_api_version: 222:31
mordredor vice versa22:31
mordredand both totally work22:31
clarkbthat seems less that ideal particularly if the goal is to deprecate and remove v2 eventually22:31
mordred(I just verified on vexxhost that both combos work fine, fwiw)22:31
clarkbbasically it seems like unnecesary footgun22:32
mordredclarkb: yah - I agree. although from shade's pov my expectations are that we'll never be able to deprecate/remove v222:32
clarkbit works today until v2 goes away then you can only auth and you get all confused22:32
clarkbmordred: right I mean from cloud side. Say I am vexxhost user in that setup wti hshade and my config is that way. Then a year from now vexxhost upgrades and I stop working22:33
clarkbexcept I clearly see keystone is working for auth and then I go drinking :)22:33
mordredright - I do not think we should let shade break in that case22:33
mordredwhich is why I lean more towards "if you explicitly ask for v2 identity crud but it's not there, give you v3 crud and a warning"22:34
mordredsince we're going to be hiding the difference between _using_ those anyway22:34
mordredso list_users should still work in both cases22:34
mordredbut if you thought you were overriding what we'd discover, then we should at least let you know22:34
clarkbya22:35
mordredclarkb, samueldmq: tomorrow I will try to write up a pure narrative version of that etherpad generalized for all of the services22:37
mordredso it can be a document of "here's how versions work in shade"22:37
samueldmqmordred: nice, what should be part of shade's official docs22:38
samueldmqthat*22:38
mordredyup22:38
mordredI will write it as a doc patch22:39
samueldmq++22:39
samueldmqmordred: how does a shade user see those warnings ?22:39
samueldmqthey're just put on stdout as any other warnings correct?22:39
mordredsamueldmq: in _discover_image_endpoint, we do this:22:40
mordred            if warning_msg:22:40
mordred                self.log.debug(warning_msg)22:40
mordred                warnings.warn(warning_msg)22:40
mordredso we debug log it _and_ put it in to warnings.warn22:40
samueldmqwarnings.warn will go to stdout eventually correct22:40
samueldmqI just wanted to check we clearly communicate that to the user22:41
mordredyes. so warnings.warn is for the normal command line case - but self.log.debug is so that people running inside of service (like nodepool) can get the messages too22:41
mordred++22:41
mordredwe should likely also document what log things we log to22:41
mordredthere are a few specifically named ones that are there to allow users to turn them on or off22:41
mordredliek, "shade.request_ids" - we log request_ids to that :)22:42
samueldmqmordred: I was reading up your convo with clarkb and I saw you discussing about the user provided version not being usable22:42
mordredyah22:42
samueldmqhow do you determine it's not usable? user giving identity_api_version=3 might not be usable22:42
samueldmqif v3 doesn't exist at all in the cloud22:42
mordredright. that's exactly it22:42
samueldmqand shade could still interpret that as "I will do what the user says as the cloud might be broken"22:43
samueldmqdoes it try an API call? does it understand a 404 to define it's not usable ?22:43
mordredif the user gives identity_api_version = 3, but it doesn't exist in the cloud via discovery, we should grab 2 and use it instead and log a warning22:43
mordredbecause shade's job ultimately is to work around broken clouds :)22:43
mordredbut we try to hide the difference between versions for the most part22:44
samueldmqmordred: what is involved in "via discovery"?22:44
samueldmqmordred: looking up the catalog or trying an HTTP call?22:44
mordredsamueldmq: the / discovery document22:44
samueldmqmordred: so looking up the catalog?22:44
mordrednope, it's the root endpoint for the service22:44
mordredif the catalog has https://example.com/v322:44
samueldmqah, the / (root) URL. ok22:45
mordredandyou do GET https://example.com/v3 - you'll get back a {'version': { }} with some stuff22:45
mordredyah22:45
samueldmqand then you understand that exists22:45
mordredyah22:45
samueldmqif you do /v3 and gets 404 the user config is invalid22:45
mordredwell - you'd never get to getting /v3 just from the user config22:46
samueldmqso the service catalog is completely ignored/does not relate at all to this?22:46
mordredif you GET whatever is in the catalog, and the id is v2.0 and the user requested 322:46
mordredthen you know you have a versioned endpoint but it's for the wrong version22:46
mordredso you can remove an element from the end of the URL, then try again - and you should ge ta {'versions': 'values': []} document instead22:47
mordredthen you can look for a version that matches what the user requested22:47
samueldmqalways giving priority for the version the user asked for22:47
mordredand if you still can't find one that matches, you can warn the user and then pick the best one available22:47
mordredyah22:47
samueldmqdoes this relate at all to the service catalog stored in keystone ?22:48
mordredit does - in that we always _first_ try the entry in teh service catalog22:48
mordredbecause if the user didn't give us a version at all, we default to using what's in the catalog22:48
mordredor also what's in the catalog might be unversioned and we have to make some choices22:49
mordredbut what's in the catalog is always step one- unless the user has _also_ configured identity_endpoint_override22:49
* samueldmq nods22:49
mordredin which case we should fail if that url is bad - since the user was trying to be very specific and there is no way we can guess what else he meant22:49
samueldmqif identity_endpoint_override is invalid ?22:49
samueldmqthen we set to none and do discovery from 0 ?22:49
samueldmqand warn?22:49
mordredno - I think then we fail22:50
samueldmqmakes sense22:50
mordredwe shoudl always treat endpoint_override as authoritative22:50
samueldmqyes, and that's what we set at the end of the discovery22:50
mordredalso - it's a good mechanism for power users to tell us how to skip doing any discovery calls if htey want to optimize things22:50
mordredexactly22:50
samueldmqif it's set by the user, let's just use it22:50
samueldmqor error.22:51
mordred++22:51
mordredyup. that's right22:51
samueldmqok, I think I know enough to at least review your doc patch tomorrow :-)22:51
mordredwoot!22:51
samueldmqI'll mull it a bit more, and don't be surprised if I come with more questions :-)22:51
mordredhopefully that will be a review that is mostly "yes, that is exactly what I understood"22:51
samueldmqhehe22:52
mordredoh - yes- it's a rather complex topic22:52
mordredalthough, to be fair, it's a complex topic for all of openstack's users and many do not understand it - so if we can understand it fully, then we hopefully remove the pain from them :)22:52
samueldmq++22:53
samueldmqlet's make sure we write awesome docs, so they won't be afraid anymore22:53
samueldmqknow I understand that there is so much work to be done here in shade22:54
samueldmqnow* ( I'm tired)22:54
mordredI'm tired too :)22:54
samueldmqmordred: _identity_client.get_endpoint() already comes from the service catalog ?23:00
mordredsamueldmq: yes23:01
mordredsamueldmq: that's essentially keystoneauth.adapter.Adapter.get_endpoint23:01
samueldmqmordred: ++23:05
samueldmqmordred: I wrote a pseudocode at the end of that etherpad23:05
samueldmqmordred: I am reading up your description again to see if it matches23:05
openstackgerritSamuel de Medeiros Queiroz proposed openstack-infra/shade master: Move user list to REST and build API URL  https://review.openstack.org/45774523:11
samueldmqmordred: yes, I believe my pseudocode matches  yours, we're in the same page23:24
samueldmqwhat I did was to put all those questions/answers in a single code so I could process it better :-)23:25
*** gouthamr has joined #openstack-shade23:33

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!