Tuesday, 2017-10-24

*** thorst has joined #openstack-powervm00:43
*** thorst has quit IRC00:44
*** thorst has joined #openstack-powervm01:33
*** thorst has quit IRC01:39
*** AlexeyAbashkin has joined #openstack-powervm02:57
*** AlexeyAbashkin has quit IRC03:02
*** k0da has joined #openstack-powervm06:50
*** AlexeyAbashkin has joined #openstack-powervm07:32
*** k0da has quit IRC08:39
*** k0da has joined #openstack-powervm08:51
*** thorst has joined #openstack-powervm09:56
*** thorst has quit IRC10:00
*** k0da has quit IRC10:11
*** thorst has joined #openstack-powervm12:04
*** esberglu has joined #openstack-powervm12:55
esberglu#startmeeting powervm_driver_meeting13:00
openstackMeeting started Tue Oct 24 13:00:16 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.13:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:00
*** openstack changes topic to " (Meeting topic: powervm_driver_meeting)"13:00
openstackThe meeting name has been set to 'powervm_driver_meeting'13:00
esberglu#topic In-Tree Driver13:02
*** openstack changes topic to "In-Tree Driver (Meeting topic: powervm_driver_meeting)"13:02
esbergluI have a couple nits to fix up for config drive13:03
esbergluThen returning focus to OVS13:03
esbergluNext step there is getting a working local.conf for IT13:03
*** edmondsw has joined #openstack-powervm13:04
esbergluNothing else from me13:04
esberglu#topic Out-of-Tree Driver13:05
*** openstack changes topic to "Out-of-Tree Driver (Meeting topic: powervm_driver_meeting)"13:05
esbergluI have to rework the HostCPUStats patches after seeing efried's comments13:06
efriedDo those comments make sense?13:06
efriedlet me know if you want to discuss later.13:07
esbergluefried: Yeah. I'll let you know if I have any questions13:07
edmondswI've several got patches up for translations13:08
edmondswefried I don't see any packaging manifests to update for pypowervm13:11
efriededmondsw May need to talk to minhn13:12
efriedCause the build is failing.13:12
efriedfor both RHEL and deb.13:12
efriedSo if the packaging manifests don't live in the pypowervm project, they'll need to be updated... wherever they do live.13:12
edmondswdid we lose esberglu?13:15
esbergluSorry responding to something else13:15
esbergluReady to move on?13:16
esberglu#topic Device Passthrough13:16
*** openstack changes topic to "Device Passthrough (Meeting topic: powervm_driver_meeting)"13:16
esbergluefried: You're up13:16
efriedSpecs are froze.  Everything that we need got approved.  Code is proceeding apace.13:17
efriedI wouldn't be at all offended if y'all reviewed my patches.13:17
efriedI got started on the series for granular resource requests: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/granular-resource-requests13:17
edmondswefried I would love to, but I don't see it happening any time soon with the way things are going13:18
efriedOther than that, I've been spending the majority of my time reviewing others' patches.13:18
edmondswI'm sure you've got things well in hand13:18
efriedGetting toward a decent understanding of how everything is working.13:18
efriedand keeping everyone honest.13:18
efriedStill waiting on requirements for the pike-based hack from jwcroppe13:19
efriedI imagine that's going to have to go through the whole UX design thing from pvc anyway13:19
efriedwhich will drive how the community side needs to look.13:19
efriedI'm almost thinking it may be more appropriate for pvc to carry this code in pvcos since we're gonna rip it all out anyway.13:20
efriedAnyway, that's about it.13:21
esberglu#topic PowerVM CI13:22
*** openstack changes topic to "PowerVM CI (Meeting topic: powervm_driver_meeting)"13:22
esbergluBeen in a holding pattern here mostly13:22
esbergluhsien enabled dev logs on one of the neos for the map lu bug13:23
esbergluJust need it to hit there again so he can take a look13:23
esberglu#topic Open Discussion13:25
*** openstack changes topic to "Open Discussion (Meeting topic: powervm_driver_meeting)"13:25
efriedGot a quick one for y'all13:26
efriedI see esberglu already hit it.13:26
esbergluedmondsw: recheck:powervm doesn't work13:26
efriedNoticed yesterday we don't get patchbot notifications for changes on stable branches.  I want that.13:26
esbergluIt needs to be the other order powervm:recheck13:27
edmondswesberglu doh13:27
esbergluefried: Yep good addition13:27
efriedY'all could review this one too: https://review.openstack.org/#/c/499367/13:27
efriedThen I could make that a regex and wouldn't have to remember to update it every release.13:28
efriedPeople were excited about that patch in IRC when I proposed it.  Then... nothing.13:28
esbergluefried: Will do13:28
efriedNothing else from me.13:28
esbergluMe neither13:29
*** openstack changes topic to "This channel is for PowerVM-related development and discussion. For general OpenStack support, please use #openstack."13:29
openstackMeeting ended Tue Oct 24 13:29:39 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)13:29
openstackMinutes:        http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-10-24-13.00.html13:29
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-10-24-13.00.txt13:29
openstackLog:            http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-10-24-13.00.log.html13:29
*** AlexeyAbashkin has quit IRC13:48
*** AlexeyAbashkin has joined #openstack-powervm13:50
*** tblakes has joined #openstack-powervm14:01
edmondswefried, I don't like the proposed solution to https://bugs.launchpad.net/nova-powervm/+bug/172408414:02
openstackLaunchpad bug 1724084 in nova-powervm "get_vnc_console in the driver raises an InternalError exception which isn't handled by the API" [Undecided,New] - Assigned to Tyler Blakeslee (tblakes)14:02
edmondswI understand you were talking to tblakes about this yesterday?14:02
efriededmondsw Yeah, talk to me.14:02
efriedmdrabe heads up14:02
edmondswthe right place to fix this is in nova... that expected_errors check they have is busted14:02
edmondswit needs to allow any HTTP 50014:03
efriededmondsw You're talking about a) convincing them of that, and b) getting it backported to pike.14:04
efriededmondsw They're also not going to allow an arbitrary exception to be serialized across the wire.14:04
edmondswefried I think (a) is extremely straightforward... it's very simple, as stated above14:05
edmondswserialized across the wire? Isn't this in the API, where it's going to return an HTTP response, not serialize anything?14:06
edmondswas for backporting to pike... if that doesn't go through, pvc can carry a patch14:06
efriededmondsw IIUC, the whole point of this bug is that we want the end user to see a useful message here.14:06
efriedThe driver raises the original exception (where we're making the change).  The API carries it across the wire.14:07
efriedAnd they're blocking any exception that doesn't come from nova.exceptions to obviate code injection, which seems reasonable.14:07
edmondsw1) this exception does come from nova.exception14:08
efriedNot the one we're trying to create here.14:08
edmondswefried https://github.com/openstack/nova/blob/master/nova/exception.py#L171914:10
edmondswbut that's really beside the point anyway14:10
edmondswthere's no injection issue here14:10
edmondswnova isn't trying to restrict what message can be returned, they're trying to restrict what HTTP error codes can be returned14:11
edmondswand HTTP 500 is something that should never be restricted... any API can raise that, at any time... that's intrinsic to HTTP14:11
edmondswand the message and error code are the only things that the API is going to send over the wire14:12
*** k0da has joined #openstack-powervm14:14
edmondswefried make sense?14:14
efriedThe exception raised by the driver gets wrapped in an HTTP exception and sent over the wire.  Then it gets deserialized on the other side, but only if it came from nova.exceptions.  Then, if it matches one of the three or four exceptions they're "expecting", it gets raised in a way where you can see the message; otherwise it gets raised in the generic and ugly way that we're trying to fix.14:14
efriedInternalError, which is what we have now, is subject to that last thing.14:14
efriedConsoleTypeUnavailable is copacetic, except that it's not really an appropriate exception for the error we're seeing.14:15
efriedSo we tried to subclass ConsoleTypeUnavailable, but got hammered by the can't-deserialize-external-exception thing.14:15
edmondswefried expected_errors is running in nova-compute, not in nova-api?14:15
efriedThat's not where the exception is being handled pretty-or-not.14:19
edmondswefried there has to be some way for a driver to raise HTTP 500 with a nicer-than-the-default error14:19
edmondswthat will require fixing expected_errors14:19
efriedThe driver doesn't raise 500s14:19
edmondswI know...14:19
edmondswlet me rephrase14:19
edmondswthere has to be some way for a driver to indicate that the API should return HTTP 50014:20
edmondswwith a nicer-than-the-default message14:20
efriedThis specific case is funneling through nova.api.openstack.compute.remote_consoles.RemoteConsolesController#get_vnc_console14:21
efriedBy the time we get there, we've gone through the API layer and deserialized the exception.14:22
efriedThat method is interpreting regular (non HTTP) exceptions and translating them to 4xx HTTP exceptions.14:23
edmondswefried meaning this is in nova-api after all, not nova-compute?14:23
efried"this" what?14:26
edmondswexpected_errors running14:26
efriedNo, the expected_exceptions decorator doesn't really have any effect here as far as I can tell.14:26
edmondswit's the key here...14:27
efriedBecause it's causing this to be raised as a 500 rather than a 400?14:27
edmondswit's what complains about the InternalError not being expected and raises a generic error instead14:27
edmondswand stripping the message, yes14:28
efriedtblakes yt?14:29
tblakesWhat's up?14:29
efriededmondsw and I have been talking about the exception thingy you've been working on.14:29
efriedThough at this point I'm not sure exactly what edmondsw is proposing.14:30
edmondswefried one sec and I'll show you what I'd propose...14:31
efriededmondsw You're saying we should add a generic exception to expected_exception and the nova.api.openstack.compute.remote_consoles.RemoteConsolesController#get_vnc_console handler that lets us raise a 500 with whatever message we want.14:31
efriedLike maybe let it handle InternalError?14:31
tblakesThat sounds like a good idea to have edmondsw show what he's proposing. I'm fine going with whatever solution you guys think is the best.14:32
edmondswefried tblakes I'm saying add this to expected_error: http://paste.openstack.org/show/624488/14:34
efriededmondsw tblakes So right now the bug is that we're getting this "Unexpected API Error" ?14:37
edmondswefried tblakes or this might be better... to ensure someone really is intentionally trying to return a message other than the default: http://paste.openstack.org/show/624489/14:37
edmondswefried right now, the bug is that the nice message is telling folks what kind of error they hit has been stripped off and replaced with a generic message14:38
efriedtblakes You have the setup to test this out easily, right?14:39
tblakesYeah I can test that out easily enough.14:40
edmondswtblakes now, taking a step back... why do we want to return a nice error message on an HTTP 500? Usually HTTP 500 means you need your admin to get involved and dig through logs, so there's no real reason to return a nicer message (and maybe reason not to)14:40
efriedIn this case, the console usually failed to open because it's already open.14:41
efriedAnd someone (the admin) has to go manually close it.14:41
efriedWe shouldn't need to dig through the logs for that.14:41
edmondswthen maybe HTTP 400 is a more appropriate code14:42
efriededmondsw I'm still not convinced this is the code path we're going through in the first place.14:45
efriedtblakes Can you try it out?14:45
tblakesYeah I'll try out edmondsw changes14:46
efriedtblakes if this makes it easier: https://review.openstack.org/#/c/514702/1/nova/api/openstack/extensions.py14:46
edmondswefried it's definitely going through that code... we're seeing a message that is only found in that code14:46
efriededmondsw Okay, that's what I asked earlier.  It would be nice if that were stated in the bug report.14:47
edmondswefried yeah, the LP bug is not well written... the pvc bug has more detail14:47
edmondswthe LP bug seems to have been written more with the fix in mind than the problem14:48
tblakesAfter updating with edmondsw change the error message says "Unable to open console." instead of the generic API exception message.14:48
efriedtblakes The message you're throwing from the driver?14:49
tblakesYeah that's the message you added to the InternalServerError.14:50
*** AlexeyAbashkin has quit IRC14:51
efriededmondsw btw, I was getting confused between expected_exceptions and expected_errors.  Compute manager's get_vnc_console method is decorated with the former, which doesn't seem to do anything.14:51
efriedThe latter must be buried somewhere in the API layer.14:51
edmondswah, ok14:51
efriededmondsw Okay, so what are we proposing here?14:52
edmondswefried and sorry, I should have realized I was privy to information from the pvc defect you didn't have just looking at LP14:52
edmondswefried I want to step back a second14:52
efriedA 400 means "go fix something and try again".  A 500 means "something unexpected is broken"14:52
edmondswefried so before https://github.com/openstack/nova-powervm/commit/0b91811519ce3e9d07b29412949a6fed97837835 this returned more info...14:52
edmondswthat removed the "Error is: %s" for some reason14:52
efriedYeah, that sucks.14:53
edmondswnow it just says unable to open console, and not why14:53
edmondswso if we really need to tell them why, we need to add that (or something) back14:53
edmondswthe next issue is what HTTP error code is appropriate here14:54
edmondsw400? 404? 409?14:54
edmondswor 500?14:54
edmondswprobably not 404 or 50014:54
edmondswI was kinda thinking 40914:55
edmondswbut maybe 40014:55
efried409 is more for an HTTP resource conflict, which this ain't.14:55
efriedAnd we don't even know for sure that it's for a console conflict, necessarily.14:55
efriedSo 400.14:55
efriedBut there's the rub14:55
edmondswtruth is, there isn't really a good HTTP code for this kind of thing (or a lot of rest errors)14:55
efriedSure it is14:56
efried400 means "Something broke that you can fix.  Fix it and try again."14:56
edmondswindicates that the server cannot or will not process the request due to something that is perceived to be a client error14:57
efried500 means "Something broke on the server side that shouldn't have broken.  It's either a bug or some kind of failure beyond your control or ours."14:57
edmondswnot exactly... 400 "indicates that the server cannot or will not process the request due to something that is perceived to be a client error". Would you call this a client error? I don't think I would14:58
tblakesI've got an appointment at 10:15 that I need to head to now, but I should be back by 11.14:58
efriedThey didn't close the console.14:58
efriedor whatever.14:58
edmondswsomeone didn't... not necessarily this client, right?14:58
edmondswbut whatever... I don't love 409 either... 400 is fine14:59
efriedSame thing if we tried to create an LPAR with a UUID that already exists14:59
efriedor attach a disk that's already attached somewhere else.14:59
*** AlexeyAbashkin has joined #openstack-powervm14:59
efriedDoesn't matter if it was *this* client.  By "client" it means "not the server".14:59
efriedOkay, so 400, but now we gotta convince nova that InternalError (or whatever we decide we want to push from our driver) should correspond to a 400.15:00
edmondswwell, not InternalError15:00
efriedRight, cause that sounds like 500.15:00
edmondswyep... but not ConsoleTypeUnavailable either15:00
efriedReally should be some kind of general ExceptionWeWantTheUserToSee15:00
efriedIt would be in the same philosophical category as BuildAbortException.15:01
efriedIn the sense that it gets treated specially.15:01
edmondswis this type of error specific to our driver? Nobody else is going to have this kind of issue where they can't open a console because one is already open?15:02
efriedNot sure that's relevant at the moment.15:02
efriedWe can make our specific snowflake exception a subclass of whatever this hypothetical passthrough exception is.15:02
edmondswthe reason I ask is because if other drivers have a similar issue, we can create a new nova.exception.ConsoleUnavailable15:02
efriedAnd pass that into expected_errors (and not modify expected_errors)?15:03
efriedThereby solving the specific issue but not the general one?15:03
edmondswand make ConsoleTypeUnavailable subclass that, then use ConsoleUnavailable instead of ConsoleTypeUnavailable in nova.api.openstack.compute.remote_consoles.RemoteConsolesController#get_vnc_console15:04
edmondswand then we don't have to change expected_errors at all15:04
efriedRight, so that's effectively the current solution.15:04
efriedExcept without involving nova.15:04
efriedtblakes and I did discuss doing something like ^15:04
efriedBut didn't think it worth the hassle of convincing nova it was necessary.15:05
efriedThey're pretty focused on blueprint work for queens.15:05
efriedBut if we want to throw it at the wall and see if it sticks...15:05
efriedSomebody's going to have to word that bug in a convincing way.15:06
edmondswlet's do that, and in the meantime we can push through tblakes change... though it still needs to add something to augment the message since we lost the necessary detail with https://github.com/openstack/nova-powervm/commit/0b91811519ce3e9d07b29412949a6fed9783783515:06
efriededmondsw His change has that in it.15:07
edmondswoh, yeah, it does... cool15:08
efriedSo tblakes just needs to address mdrabe's -115:08
*** AlexeyAbashkin has quit IRC15:10
edmondswefried did you and tblakes consider raising a webob exception?15:10
efriedFrom the driver?  No.15:10
edmondswyeah, probably don't want that going through serialization15:10
efriedIs there precedent for that?  I would be surprised.15:10
edmondswno idea15:10
edmondswok, I updated the review15:11
*** tjakobs has joined #openstack-powervm15:30
*** k0da has quit IRC15:53
*** k0da has joined #openstack-powervm16:06
*** AlexeyAbashkin has joined #openstack-powervm16:22
openstackgerritTyler Blakeslee proposed openstack/nova-powervm master: Raise catachable error when VNC terminal fails to open.  https://review.openstack.org/51383316:24
*** AlexeyAbashkin has quit IRC16:26
*** tblakes has quit IRC16:31
edmondswesberglu are we hitting more timeouts in the CI than usual?16:46
edmondswseems like it16:46
*** AlexeyAbashkin has joined #openstack-powervm16:55
*** AlexeyAbashkin has quit IRC16:59
openstackgerritMerged openstack/nova-powervm master: Raise catachable error when VNC terminal fails to open.  https://review.openstack.org/51383317:07
esbergluedmondsw: Seems to happen when a lot of runs are occurring. I was planning on setting some time aside next week to dig into those again17:26
*** k0da has quit IRC18:41
*** AlexeyAbashkin has joined #openstack-powervm18:55
*** k0da has joined #openstack-powervm18:58
*** AlexeyAbashkin has quit IRC19:00
edmondswesberglu can you take a look at https://review.openstack.org/514437 and https://review.openstack.org/514439?19:19
edmondswand 606819:19
esbergluedmondsw: Sure19:20
edmondswty sir19:20
edmondswefried when you have a minute... 606819:20
esbergluedmondsw: +2'ed all the removals19:22
esbergluHow do all of the .po files get generated?19:23
efriededmondsw Is that just a rebase?19:24
efriedNo edits, it appears?19:24
edmondswesberglu there's a command for that (don't recall exactly what offhand), and then we send them off for translation and then I checked in what we got back19:24
edmondswefried right, I removed the *-log-* files and that's about it19:24
edmondswefried esberglu can one of you +W https://review.openstack.org/#/c/514439/ and https://review.openstack.org/#/c/514437/ ?21:04
edmondswwould rather not do that on myself even thought you both +2ed21:04
esbergluedmondsw: Done. Is the ceil one stuck on CI21:05
edmondswesberglu tx, and yes21:05
edmondswesberglu been over 4 hours since I rechecked and no result yet21:05
*** thorst has quit IRC21:06
esbergluedmondsw: Looking. Haven't checked up on CI status this afternoon21:06
edmondswesberglu you just added another recheck? shouldn't we have waited for the results of my earlier recheck?21:06
esbergluedmondsw: Didn't realize that yours didn't go through yet21:07
*** thorst has joined #openstack-powervm21:07
esbergluedmondsw: Hmm weird looks like the nodepool nodes are getting removed from jenkins but not deleted from nodepool after the CI runs are completing.21:09
edmondswesberglu glad you checked :)21:10
*** thorst has quit IRC21:12
openstackgerritMerged openstack/nova-powervm master: Persist existing LPAR wrapper attributes in DefaultStandardize on resize  https://review.openstack.org/51134321:13
openstackgerritMerged openstack/networking-powervm master: remove log translation files  https://review.openstack.org/51443921:15
esbergluedmondsw: Relocating quick, will look into right after. Looks like it's a rabbitmq issue21:17
edmondswesberglu ack21:18
*** esberglu has quit IRC21:18
*** esberglu has joined #openstack-powervm21:19
openstackgerritMerged openstack/nova-powervm master: remove log translation files  https://review.openstack.org/51443721:21
*** esberglu has quit IRC21:24
openstackgerritMatthew Edmonds proposed openstack/networking-powervm master: update translations  https://review.openstack.org/51440321:29
*** thorst has joined #openstack-powervm21:32
*** thorst has quit IRC21:36
*** thorst has joined #openstack-powervm21:38
*** esberglu has joined #openstack-powervm21:39
*** thorst has quit IRC21:43
*** k0da has quit IRC21:51
*** AlexeyAbashkin has joined #openstack-powervm21:56
*** AlexeyAbashkin has quit IRC22:01
openstackgerritMatthew Edmonds proposed openstack/nova-powervm master: update translations  https://review.openstack.org/51439922:21
*** edmondsw has quit IRC22:29
*** edmondsw has joined #openstack-powervm22:29
*** edmondsw has quit IRC22:34
*** tjakobs has quit IRC22:36
*** thorst has joined #openstack-powervm22:43
*** thorst has quit IRC22:49
*** esberglu has quit IRC23:25
*** edmondsw has joined #openstack-powervm23:55
*** edmondsw has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!