| *** slaweq has joined #openstack-shade | 00:32 | |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: Use REST API for volume type_access and volume create https://review.openstack.org/460357 | 00:36 |
|---|---|---|
| rods | ^^ since I changed a test I'm keeping this ps small :) | 00:37 |
| *** slaweq has quit IRC | 00:37 | |
| *** gouthamr has joined #openstack-shade | 00:46 | |
| *** gouthamr has quit IRC | 01:42 | |
| *** gouthamr has joined #openstack-shade | 02:51 | |
| *** yolanda has quit IRC | 03:56 | |
| *** gouthamr has quit IRC | 04:01 | |
| *** yolanda has joined #openstack-shade | 04:04 | |
| *** yolanda has quit IRC | 04:10 | |
| *** gkadam has joined #openstack-shade | 04:12 | |
| *** slaweq has joined #openstack-shade | 05:20 | |
| *** slaweq has quit IRC | 05:26 | |
| *** slaweq has joined #openstack-shade | 06:27 | |
| *** slaweq has quit IRC | 06:32 | |
| *** cdent has joined #openstack-shade | 09:43 | |
| *** yfried has joined #openstack-shade | 10:47 | |
| *** cdent has quit IRC | 11:30 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Replace neutronclient with REST API calls in router commands https://review.openstack.org/460518 | 11:51 |
| mordred | rods: the gate robots did not like that :) | 12:06 |
| rods | yea :) | 12:06 |
| rods | looking into it | 12:06 |
| mordred | rods: ah! so - I think I see it - there is a weirdness with testtools.ExpectedException (that I do not like very much) | 12:13 |
| mordred | it matches exceptions _exactly_ - instead of matching them like python except: does | 12:13 |
| mordred | so the new code is throwing a more specific exception (OpenStackCloudURINotFound) which is a subclass of the more general exception those tests are written to check for (OpenStackCloudException) | 12:14 |
| mordred | and also the message is different now - so I think this is another case in which you get to change the test at the same time you change the code | 12:15 |
| mordred | you get all the fun with this one | 12:15 |
| rods | mordred lucky me :) | 12:16 |
| Shrews | i think ExepectedException matching the _exact_ exception is exactly what you want | 12:17 |
| Shrews | in a test scenario, you should KNOW which exact exception is being thrown | 12:17 |
| rods | mordred I'm running a devstack instance to run the functional tests against it, I was just running unit tests locally :) | 12:18 |
| Shrews | slaweq_: are you manually adding reviewers to your reviews? for example, https://review.openstack.org/460518 | 12:28 |
| Shrews | slaweq_: if so, that's not necessary | 12:28 |
| *** slaweq has joined #openstack-shade | 12:32 | |
| mordred | Shrews: yah - I guess you're right | 12:36 |
| Shrews | mordred: was there ever any doubt? :-P | 12:36 |
| slaweq_ | Shrews: yes, I was adding it manually | 12:37 |
| slaweq_ | Should I stop doing it? | 12:37 |
| *** slaweq has quit IRC | 12:37 | |
| Shrews | slaweq_: yeah. the folks that are interested in shade will already have things in place to get notified. adding _everyone_ just causes spam for those that don't actually review | 12:38 |
| slaweq_ | Shrews: ok, so sorry for that | 12:38 |
| * Shrews is very glad to see slaweq_ and rods getting involved in shade | 12:38 | |
| slaweq_ | I will not do it anymore | 12:38 |
| Shrews | slaweq_: no worries :) | 12:38 |
| *** gkadam has quit IRC | 12:41 | |
| mordred | Shrews: me too! | 12:44 |
| *** Guest94155 has quit IRC | 13:10 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: Replace neutronclient with REST API calls in router commands https://review.openstack.org/460518 | 13:54 |
| *** gouthamr has joined #openstack-shade | 14:31 | |
| *** slaweq has joined #openstack-shade | 14:33 | |
| *** slaweq has quit IRC | 14:38 | |
| rods | I might be missing something, I'm looking at this test here http://logs.openstack.org/57/460357/1/check/gate-shade-dsvm-functional-legacy/199378b/console.html#_2017-04-27_00_58_42_079351, looks like it is getting a 404 from cinder and is correctly raising a OpenStackCloudURINotFound here https://github.com/openstack-infra/shade/blob/master/shade/exc.py#L108 | 15:36 |
| rods | shouldn't OpenStackCloudURINotFound be caught here OpenStackCloudException and reraised as OpenStackCloudException with the correct message here https://github.com/openstack-infra/shade/blob/master/shade/operatorcloud.py#L2042 as the test is expecting https://github.com/openstack-infra/shade/blob/master/shade/tests/functional/test_volume_type.py#L107 | 15:37 |
| rods | mordred ^^ | 15:38 |
| rods | sorry for all the links pasting :) | 15:38 |
| rods | *be caught here https://github.com/openstack-infra/shade/blob/master/shade/_utils.py#L462 | 15:39 |
| mordred | rods: it should be directly re-raised here: https://github.com/openstack-infra/shade/blob/master/shade/_utils.py#L456 | 15:45 |
| mordred | rods: since OpenStackCloudURINotFound is a subclass of OpenStackCloudException | 15:45 |
| mordred | so the original REST GET call will generate OpenStackCloudURINotFound which we will no longer wrap (since we made it ourselves) | 15:46 |
| mordred | hrm - that said- ... I wonder if we should alter the message attribute of the exception in https://github.com/openstack-infra/shade/blob/master/shade/_utils.py#L456 before the raise as a way to give nicer error messages but still not mess up the tracebacks | 15:47 |
| mordred | we currently wrap exceptions thrown by the clients, which is pretty bad form since it makes the tracebacks harder to follow - but we also didn't want shade's users to need to catch exceptions from python-*client since we've always known we wanted to get rid of them | 15:48 |
| mordred | so one of the nice things about the move to REST is that we can get out of the game of intercepting and wrapping exceptions, since we can throw the right exception in the first place | 15:49 |
| mordred | but, because the call is something like "adapter.get('/foo')" - the error message that throws is a little more ugly - I didn't have a _great_ idea of what to do about that yet | 15:50 |
| rods | mordred yah I see, is there any doc about how to configure shade to run functional tests against a running devstack? I'd like to play a with it a little to get more familiar with the code | 15:52 |
| *** yfried has quit IRC | 15:58 | |
| Shrews | rods: You should only need a clouds.yaml with entries for 'devstack' and 'devstack-admin' (devstack outputs one somewhere...) and just run 'tox -e functional' | 15:59 |
| rods | Shrews ok, thx | 16:00 |
| Shrews | you can place the clouds.yaml in your ~/.config/openstack/ directory | 16:00 |
| mordred | yah - devstack should already make the clouds.yaml in /etc/openstack on the devstack machine | 16:00 |
| rods | cool, thx | 16:00 |
| *** yolanda has joined #openstack-shade | 16:09 | |
| *** slaweq has joined #openstack-shade | 16:10 | |
| *** slaweq has quit IRC | 16:12 | |
| *** gouthamr has quit IRC | 16:31 | |
| *** gouthamr has joined #openstack-shade | 16:40 | |
| *** slaweq has joined #openstack-shade | 17:13 | |
| *** slaweq has quit IRC | 17:18 | |
| *** slaweq has joined #openstack-shade | 17:47 | |
| slaweq | mordred: hello, can You have a look at result of tests in https://review.openstack.org/#/c/460518/ ? | 17:48 |
| slaweq | mordred: gate-shade-dsvm-functional-legacy are failing but I'm not sure how it could be connected to this patch | 17:48 |
| slaweq | IMHO it's not connected but maybe I'm wrong :) | 17:49 |
| mordred | slaweq: looking | 18:01 |
| *** yfried has joined #openstack-shade | 18:02 | |
| *** Matias has joined #openstack-shade | 18:05 | |
| mordred | slaweq: ah- I believe I know what that is, and no, it's not related to your patch | 18:05 |
| mordred | slaweq: lemme go poke a few folks | 18:05 |
| slaweq | mordred: thx, that's good information for me because I had no idea how I could fix it :P | 18:06 |
| *** yfried has quit IRC | 18:22 | |
| *** slaweq has quit IRC | 18:26 | |
| *** yfried has joined #openstack-shade | 18:27 | |
| *** yfried has quit IRC | 18:36 | |
| *** dhellmann has quit IRC | 18:38 | |
| *** dhellmann has joined #openstack-shade | 18:39 | |
| *** openstackgerrit has quit IRC | 18:48 | |
| *** slaweq has joined #openstack-shade | 18:50 | |
| *** slaweq has quit IRC | 19:00 | |
| *** cdent has joined #openstack-shade | 19:01 | |
| *** slaweq has joined #openstack-shade | 19:25 | |
| *** yfried has joined #openstack-shade | 20:06 | |
| *** cdent has quit IRC | 20:19 | |
| *** yfried has quit IRC | 20:41 | |
| *** ianw has quit IRC | 22:13 | |
| *** ianw has joined #openstack-shade | 22:18 | |
| *** purplerbot has quit IRC | 22:19 | |
| mordred | slaweq, rods, Shrews: I had an all-afternoon meeting so haven't been able to dig in to the gate failure | 22:39 |
| mordred | it's related toa recent devstack change, so has nothing to do with shade, but it'll likely be tomorrow before we can get it fixed | 22:39 |
| slaweq | mordred: ok, thx for info | 22:40 |
| slaweq | I will just wait | 22:40 |
| rods | mordred np, I'm able to reproduce the functional tests failure locally and working on fix them | 22:41 |
| clarkb | mordred: oh thanks for reminding me thats what i was going to look at more closedly after lunch. That and sdagues thing | 22:42 |
| mordred | clarkb: oh sweet - if you find the culprit that would be ossum | 22:42 |
| Shrews | mordred: this was the functional-legacy failure? devstack should NOT change in those | 22:42 |
| Shrews | unless something was backported | 22:43 |
| Shrews | seems unlikely for devstack | 22:43 |
| clarkb | Shrews: thats what I said | 22:43 |
| clarkb | hence my interest (and confusion) | 22:44 |
| mordred | yah. but - the keystone there seems to clearly be doing suburl and not port-based | 22:44 |
| mordred | so - yes - it seems something is wrong in a way that it should not be | 22:44 |
| Shrews | i like things being wrong in ways that they should be wrong | 22:44 |
| mordred | clarkb: we have BRANCH_OVERRIDE set in the legacy job - is there any thing else we're maybe doing wrong to tell devstack to give us newton? | 22:45 |
| clarkb | mordred: maybe that isn't working properly? | 22:47 |
| clarkb | or maybe things got backported | 22:47 |
| mordred | clarkb: I don't see any devstack backport patches | 22:48 |
| clarkb | ya I don't either | 22:49 |
| mordred | and the devstack-gate log looks like it ran setup_project with stable/newton | 22:49 |
| mordred | http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/logs/devstack-gate-setup-workspace-new.txt.gz#_2017-04-27_16_14_55_840 | 22:49 |
| clarkb | a595076 is what we chceked out | 22:49 |
| clarkb | which looks right | 22:50 |
| mordred | clarkb: oh - actually- the suburl is a red-herring- keystone has been deploying suburl for a while | 22:51 |
| clarkb | because of the wsgi switch | 22:51 |
| mordred | yah | 22:51 |
| clarkb | its odd to me that we would do it the way we did it though where http and port 5000 are apparently both valid | 22:51 |
| mordred | but these: 2017-04-27 16:18:46.167 | KEYSTONE_SERVICE_URI=http://198.72.124.194/identity | 22:52 |
| mordred | 2017-04-27 16:18:46.167 | KEYSTONE_SERVICE_URI_V3=http://198.72.124.194/identity/v3 | 22:52 |
| mordred | are not the endpoints that wound up in clouds.yaml | 22:52 |
| clarkb | http://198.72.124.194/identity_v2_admin is in clouds.yaml for all accounts | 22:53 |
| mordred | clarkb: yah - so - that's stupid and a bug - but I think it's not changed in a while | 22:58 |
| Shrews | mordred: the version of keystoneauth being used is the latest: http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/console.html#_2017-04-27_16_40_57_213748 | 23:00 |
| Shrews | that doesn't seem right | 23:00 |
| mordred | Shrews: yah - we don't cap libraries | 23:00 |
| clarkb | could it be that keystoneauth broke talking to old keystone? | 23:01 |
| clarkb | (I'd expect more screaming if it did though) | 23:01 |
| mordred | yah - that would break everything | 23:01 |
| Shrews | that's what i'm suspecting | 23:01 |
| mordred | clarkb: where do we log devstack gate hook scripts? | 23:02 |
| clarkb | mordred: in console.html iirc | 23:02 |
| mordred | yes. thanks | 23:02 |
| Shrews | mordred: i assumed k/a had a stable/newton branch, similar to the other libs. i guess not | 23:03 |
| mordred | clarkb, Shrews: it's making the connections to the service | 23:05 |
| clarkb | ok not firewall which makes sense because its localhost to loachost | 23:05 |
| clarkb | the magnum issue was it was from nested vm to localhost | 23:05 |
| clarkb | and that tripped iptables | 23:05 |
| mordred | like - it does version discovery properly here: http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/console.html#_2017-04-27_16_40_57_219285 | 23:06 |
| *** slaweq has quit IRC | 23:06 | |
| *** slaweq has joined #openstack-shade | 23:07 | |
| clarkb | ya it seems like its 404ing on the actual listings of users | 23:09 |
| *** slaweq has quit IRC | 23:11 | |
| mordred | it 404s on creating a service here: http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/console.html#_2017-04-27_16_40_57_227404 | 23:11 |
| clarkb | mordred: and if I grep for the request id in the keystone logs no match | 23:11 |
| mordred | I found 2 entries forthat | 23:12 |
| mordred | http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/logs/apache/keystone.txt.gz#_2017-04-27_16_40_57_091 | 23:12 |
| mordred | http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/logs/apache/keystone.txt.gz#_2017-04-27_16_40_57_131 | 23:12 |
| mordred | but they're not much help | 23:12 |
| clarkb | ya and no errors in that log | 23:13 |
| mordred | I also see: 198.72.124.194 - - [27/Apr/2017:16:40:57 +0000] "POST /identity/v2.0/OS-KSADM/services HTTP/1.1" 404 381 "-" "python-keystoneclient" | 23:14 |
| mordred | in the access.log | 23:14 |
| clarkb | mordred: what are the two strings after the request id in the keystone log? | 23:21 |
| clarkb | tokens/ | 23:22 |
| mordred | clarkb: I do not know | 23:22 |
| clarkb | also why is the auth uri the admin endpoint | 23:25 |
| mordred | because the clouds.yaml writing is dumb | 23:26 |
| clarkb | well its done that way all over though not just clouds.yaml | 23:26 |
| clarkb | anyways reading teh devstack log it appears that everything is using v3 | 23:26 |
| clarkb | and shade is using v2, so perhaps v2 isn't being allowed here for some reason? | 23:27 |
| clarkb | mordred: can you try forcing v3 and see what happens ? at the very least it might generate better errors we can trace in the logs | 23:27 |
| mordred | clarkb: https://review.openstack.org/460775 this is the weird thing with the clouds.yaml | 23:28 |
| mordred | clarkb: I can - although my main question then will be "what changed since yesterday" | 23:29 |
| clarkb | tempest stopped testing v2 looks like | 23:29 |
| clarkb | so it could've regressed without us noticing maybe | 23:29 |
| mordred | http://logs.openstack.org/27/460127/2/gate/gate-shade-dsvm-functional-legacy/da647ab/ | 23:31 |
| mordred | is a run from yesterday that worked | 23:31 |
| mordred | clarkb: the only thing that's different from a pip install perspective is nova :) | 23:32 |
| mordred | clarkb: and there is a point-release difference of mysql | 23:34 |
| mordred | in terms of apt packages | 23:34 |
| clarkb | what about keystone configs? | 23:34 |
| mordred | we dont seem to save tox dirs on these do we ... | 23:37 |
| clarkb | for tempest? no | 23:37 |
| *** slaweq has joined #openstack-shade | 23:38 | |
| mordred | well, I should add that to shade functional tests since we run them from tox | 23:38 |
| mordred | clarkb: I just move the dir into $WORKSPACE/logs and it gets picked up, right? | 23:39 |
| clarkb | I think only if the files have a recognized .txt/.log suffix? I'd have to look | 23:40 |
| mordred | bleh | 23:40 |
| clarkb | mordred: what I would do is grab reproduce.sh and then interact with it "locally" | 23:41 |
| clarkb | or just hold a node | 23:41 |
| clarkb | bceause this si really weird looking at configs I don't see diffs | 23:41 |
| clarkb | also try v3 | 23:42 |
| mordred | os-client-config changed | 23:42 |
| mordred | now - I have ZERO clue how that would result in this particular behavior | 23:42 |
| *** slaweq has quit IRC | 23:42 | |
| clarkb | doesn't it make the keysteonauth objects? | 23:43 |
| clarkb | and I bet its not tested against v2 | 23:43 |
| mordred | it does - and I'm sure it's the culprit | 23:43 |
| mordred | but I don't understand how making the objects could have changed such that this changed ... | 23:43 |
| mordred | but I now have something to investigate | 23:44 |
| clarkb | mordred: 01ff292e078206e487751228be4a7062ba0c6048 perhaps | 23:47 |
| *** openstackgerrit has joined #openstack-shade | 23:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add some more debugging to the post_test_hook https://review.openstack.org/460779 | 23:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: DNM block os-client-config 1.27.0 https://review.openstack.org/460780 | 23:47 |
| mordred | clarkb: yup. totally willing to bet | 23:48 |
| mordred | jamielennox: hey - so - remember this: https://review.openstack.org/#/c/450259/ ? | 23:49 |
| clarkb | and old keystone + keystoneauth rely on some side effecting behavior there is my guess | 23:49 |
| jamielennox | mordred: hmm | 23:49 |
| mordred | jamielennox: we're tracking down shade gate break that started yesterday and currently have a hunch that on old keystonev2 something about that is still needed | 23:50 |
| mordred | jamielennox: (os-client-config is the only thing different between a failing and passing test run) | 23:50 |
| mordred | jamielennox: however, http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/ is one of the fails if you wanna look at the rest interactions | 23:51 |
| jamielennox | mordred: the only reason i can see that failing is if the url in the catalog is bad | 23:51 |
| Shrews | but we have occ jobs that cross test with shade, and they seem to have passed: https://review.openstack.org/#/c/450259/ | 23:51 |
| mordred | Shrews: we use venv for our functional tets- that did not install os-client-config change into our tox venv :( | 23:51 |
| clarkb | it looks like get_endpoint defaults interface to None and not public (at least in the current checkout I have of keystoneauth | 23:52 |
| mordred | Shrews: (I'm just now realizing this hole in how we have that setup) | 23:52 |
| mordred | jamielennox: it winds up trying to POST to http://198.72.124.194/identity/v2.0/OS-KSADM/services | 23:53 |
| mordred | jamielennox: for which it gets a 404 | 23:53 |
| mordred | http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/console.html#_2017-04-27_16_40_57_225894 | 23:53 |
| Shrews | mordred: hmm, big hole | 23:53 |
| mordred | Shrews: yah | 23:53 |
| jamielennox | mordred: ok, i can see the problem, but not the why | 23:53 |
| mordred | Shrews: I'll fix that next :) | 23:53 |
| mordred | jamielennox: sweet! what's the problem? | 23:54 |
| jamielennox | so looking here: http://logs.openstack.org/18/460518/2/check/gate-shade-dsvm-functional-legacy/d22043c/console.html#_2017-04-27_16_43_18_420278 | 23:54 |
| jamielennox | looks like the auth_url is: GET http://198.72.124.194/identity_v2_admin | 23:54 |
| jamielennox | Making authentication request to http://198.72.124.194/identity_v2_admin/v2.0/tokens | 23:55 |
| mordred | yah - I just made a patch: https://review.openstack.org/460775 to change how that's getting written out | 23:55 |
| mordred | you think that might be the issue? | 23:55 |
| clarkb | it wasn't an issue before the occ release | 23:56 |
| clarkb | something to consider if other users are possibly impacted | 23:56 |
| mordred | well - it may be a combo thing | 23:56 |
| jamielennox | but then the catalog hasn't got the right admin interface? | 23:56 |
| mordred | the previous occ had a behavior that masked a broken config | 23:56 |
| jamielennox | cause the next step is REQ: curl -g -i -X GET http://198.72.124.194/identity | 23:56 |
| jamielennox | which is the non-admin | 23:57 |
| jamielennox | that OS-KSADM is only available on the keystone admin interface - which is not what you're hitting | 23:57 |
| mordred | jamielennox: I wish that was showing the catalog ... | 23:58 |
| jamielennox | mordred: was deemed to spammy | 23:58 |
| mordred | but yah - you're right - it's super strange that it's going from http://198.72.124.194/identity_v2_admin to http://198.72.124.194/identity | 23:58 |
| mordred | well - I can push up a patch that will print the catalog in that test :) | 23:59 |
| jamielennox | mordred: does it set interface='admin'? | 23:59 |
| jamielennox | cause it's defaulting to public | 23:59 |
| clarkb | the occ patch appears to have stopped setting interface entirely | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!