Tuesday, 2017-06-20

*** openstack has joined #openstack-powervm00:17
*** edmondsw has quit IRC00:19
*** BorD_ has joined #openstack-powervm00:24
*** thorst has joined #openstack-powervm00:26
*** thorst has quit IRC00:27
*** tonyb_ is now known as tonyb00:57
*** smatzek has joined #openstack-powervm01:32
*** thorst has joined #openstack-powervm01:38
*** thorst has quit IRC01:56
*** thorst has joined #openstack-powervm01:56
*** esberglu has quit IRC02:00
*** thorst has quit IRC02:00
*** edmondsw has joined #openstack-powervm02:03
*** edmondsw has quit IRC02:08
*** chhavi has joined #openstack-powervm03:02
*** smatzek has quit IRC03:05
*** edmondsw has joined #openstack-powervm03:51
*** edmondsw has quit IRC03:56
*** thorst has joined #openstack-powervm03:57
*** thorst has quit IRC04:01
*** chhavi has quit IRC04:13
*** mdrabe has quit IRC04:40
*** tjakobs has joined #openstack-powervm05:06
*** tjakobs has quit IRC05:26
*** chhavi has joined #openstack-powervm05:38
*** edmondsw has joined #openstack-powervm05:39
*** edmondsw has quit IRC05:44
*** thorst has joined #openstack-powervm05:58
*** thorst has quit IRC06:02
*** k0da has joined #openstack-powervm07:05
*** k0da has quit IRC07:21
*** edmondsw has joined #openstack-powervm07:27
*** edmondsw has quit IRC07:32
*** thorst has joined #openstack-powervm07:58
*** thorst has quit IRC08:03
*** k0da has joined #openstack-powervm08:45
*** edmondsw has joined #openstack-powervm09:15
*** edmondsw has quit IRC09:20
*** thorst has joined #openstack-powervm09:49
*** thorst has quit IRC09:59
*** thorst has joined #openstack-powervm10:17
*** smatzek has joined #openstack-powervm10:55
*** edmondsw has joined #openstack-powervm11:03
*** edmondsw has quit IRC11:08
*** thorst has quit IRC11:16
*** thorst has joined #openstack-powervm11:47
*** thorst has quit IRC11:52
*** thorst has joined #openstack-powervm12:08
*** edmondsw has joined #openstack-powervm12:15
*** efried has joined #openstack-powervm12:21
*** jpasqualetto has joined #openstack-powervm12:27
*** kylek3h has joined #openstack-powervm12:31
*** dwayne has quit IRC12:33
*** efried has quit IRC12:34
*** esberglu has joined #openstack-powervm12:37
*** esberglu has quit IRC12:39
*** esberglu has joined #openstack-powervm12:39
*** jpasqualetto has quit IRC12:40
*** mdrabe has joined #openstack-powervm12:42
*** esberglu has quit IRC12:44
*** jpasqualetto has joined #openstack-powervm12:50
*** esberglu has joined #openstack-powervm12:51
*** efried has joined #openstack-powervm12:53
*** jpasqualetto has quit IRC12:55
*** jwcroppe has quit IRC12:55
-openstackstatus- NOTICE: restarting gerrit to address slowdown issues12:56
esberglu#startmeeting powervm_driver_meeting13:01
openstackMeeting started Tue Jun 20 13:01:24 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.13:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:01
openstackThe meeting name has been set to 'powervm_driver_meeting'13:01
esberglu#link https://etherpad.openstack.org/p/powervm_driver_meeting_agenda13:01
esbergluCreated an agenda etherpad. If everyone could start adding discussion items to topics prior to the meetings starting next week that would be great13:02
esbergluI will resend the event invites with the link included as well13:02
efriedLet's do a roll call.  That will determine what we can discuss today.13:03
efriedthorst edmondsw chhavi yt?13:04
efriedmdrabe ?13:04
chhaviyes, in call little slow13:05
efriedIt would be nice to have Jay here.13:05
efriedThe main thing I know of that we need to discuss is iSCSI code & testing.13:05
esbergluAlright lets get started13:06
esberglu#topic In-Tree Driver13:06
esberglu#link https://etherpad.openstack.org/p/powervm-in-tree-todos13:06
edmondswI believe the support matrix changes merged13:06
edmondswwhat is the status of the other half of the docs changes we need to make?13:07
edmondswthe hypervisors devref13:07
esbergluI've been putting that off to work CI. I guess I didn't think it was that time sensitive?13:07
esbergluIf it is I can get it done13:07
edmondswgetting the CI working again has definitely been higher priority, but I think we want those doc changes to be completed with Pike13:08
edmondsw"completed" to the level of the support we have in the Pike code, that is13:08
edmondswefried, I assume we have some time yet before they lock down that kind of change for Pike, yes?13:09
esbergluI'm out friday. So I will get a draft up before then and then fix any comments, etc. early next week. Sound good?13:09
edmondswsounds good13:09
efriedYes, we have time for pike.13:10
edmondswso we have a few weeks there13:11
efried7/27 is feature freeze, it would be nice to have this in before that.13:11
efriedthough if it does slip, this isn't really a feature per se.13:11
efriedI.e. no code impact.13:11
efriedSo not really crucial to get it in.13:12
edmondswthe string freeze may be the more important date13:12
edmondswsoft string freeze is the same week13:12
esbergluAnything else we need to discuss in-tree?13:14
edmondswI need to send a note to the ML about https://review.openstack.org/#/c/471773/ ... still going over how I want to word that13:14
efriedString freeze relates to translation, though, right?13:15
edmondswefried right... and docs would be translated, no?13:15
efriedWas about to ask.  I don't know.13:15
edmondswI assume yes13:15
edmondswnow that I think of it, though... maybe not?13:15
esbergluEither way lets just get it in by then13:15
edmondswefried did you have anything to talk about IT?13:16
edmondswso next topic13:16
esberglu#topic OOT Driver13:17
edmondsw#topic Out-of-Tree Driver13:17
edmondswanyone have anything to discuss OOT?13:17
esbergluI don't13:18
edmondswI have a change up for the driver to use CONF.host, like we changed IT, but I was waiting for the CI to pass before pushing that13:19
edmondswanybody else working on backporting IT changes to OOT?13:19
efriedI need to do some testing of the power-off changes.13:19
efriedThen merge the pypowervm side of that.13:19
efriedThen coordinate a new pypowervm release with the iSCSI stuff, which we need to go into a bit.13:20
efriedAccording to Hsien, the REST/core side of iSCSI stuff should be copacetic at this point.13:20
efriedchhavi was teed up to talk to Jay and see whether the latest dev nvl+pypowervm had been tested with the OOT nova-powervm changes.13:21
efriedBecause we don't want to cut a new pypowervm release only to test iSCSI and find we need another one right away.13:21
efriedSo those two - power-off & iSCSI - need to be tested out, then we cut pypowervm, then we funnel everything through g-r, then we merge the nova-powervm iSCSI change; and I'll also put up a change (IT & OOT) to use the new power-off method.13:22
edmondswwe should probably send an email since Jay hasn't been on the last few meetings13:22
efriedchhavi Did you get a chance to talk to Jay today?13:22
edmondswchhavi there are also still comments to address on https://review.openstack.org/#/c/467599/13:24
edmondswwe should probably move on13:26
edmondsw#topic CI13:26
edmondswesberglu, everything running now?13:26
esbergluCI is publishing and running, not passing13:26
chhaviyes I got the environment,13:26
chhavineed to load and verify13:27
efriedGreat chhavi.13:27
efriedDo you think you'll know by tomorrow?13:27
esbergluStable branches are passing13:28
esbergluOOT is failing 1 test13:28
esbergluOOT -
esbergluIT is failing a bunch, might have to do more local.conf work there13:28
esbergluIT -
esbergluAnd then some random stacks are failing with this13:29
esbergluSSLError -
esbergluPlanning on diving into those in that order today13:29
esbergluOther than that I have a bunch of small reviews for neo-os-ci and powervm-ci13:30
esbergluAnd I tested our config on devstack in the staging environment13:30
esbergluThe latest patch there is ready to go13:30
esbergluAdding devstack to the silent pipeline I'm talking about13:31
edmondswwhat does that mean, silent pipeline?13:31
esbergluIt won't publish any results, but it will run on every patch to devstack13:32
esbergluWe currently do that for neutron and ceilometer13:32
*** jwcroppe has joined #openstack-powervm13:32
esbergluThe idea is that it should help identify breaking devstack changes ideally before they merge but in reality probably just faster13:33
esbergluOther than that just working through13:34
esbergluas able13:34
esbergluThat's it for me, anyone have anything before we move on?13:34
*** efried has quit IRC13:35
mdrabeI'm still trying to figure out the 2.46.2 issue13:35
esberglumdrabe: You planning on working on that today at all?13:36
mdrabeYea I'm gonna run the bisect today to try and identify the commit13:36
mdrabeIf it is with Jenkins13:36
esbergluSweet. Just let me know when you need the env. back13:36
mdrabeYup yup13:37
esbergluNext topic13:37
esberglu#topic Driver Testing13:37
esbergluTalked a little bit above, anything further discussion needed?13:38
esbergluDoesn't sound like it13:39
esberglu#topic Open Discussion13:39
esbergluAny final thoughts?13:40
edmondswmdrabe thorst I need to talk to one or both of you about how nova-powervm gets pulled into PowerVC13:40
edmondswso fyi, I'll be pinging13:41
mdrabeedmondsw DM me13:41
*** efried has joined #openstack-powervm13:42
esbergluthanks for joining13:43
efriedHello?  What'd I miss?13:43
esbergluefried: Nothing13:43
esbergluUnless you had something else13:43
efriedLast thing I saw was the SSLError13:43
*** jpasqualetto has joined #openstack-powervm13:43
efriedI apparently got disconnected but my IRC client didn't see fit to let me know about it.13:43
efriedIf we're done, I'll catch up with the meetbot.13:44
edmondswnothing too significant... look for some CI patches that esberglu has proposed13:44
edmondswI think we're done13:44
esbergluefried: Just click the link at the end of the meeting if you want to reread, but nothing big13:44
openstackMeeting ended Tue Jun 20 13:44:53 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)13:44
openstackMinutes:        http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-06-20-13.01.html13:44
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-06-20-13.01.txt13:44
openstackLog:            http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-06-20-13.01.log.html13:44
edmondswefried do you know if it's possible to have nova accept both forms of its UIs... i.e. with and without tenant_id in the URL?13:46
efriedUI or URI?13:46
edmondswI actually meant API... typing while in another meeting13:47
efriedI don't know whether there's a level at which it will accept either form - though I imagine there probably is (deprecation cycle) - but note that if you're trying to guarantee one behavior or the other, you can specify a microversion.13:47
edmondswI don't know if they did microversions with that13:48
edmondswoddly enough13:48
efriedOh, I didn't get a chance to mention this, but I'll be attending bits of the TCC this week, so will have a bit less time on mainline work.13:48
efriededmondsw You could ask that question in #openstack-nova or in the ML and get a pretty quick answer, I would think.13:50
edmondswyeah, I just thought I'd try you first because I generally get no answer in #openstack-nova :)13:50
edmondswI'll ask there in a bit13:50
esbergluefried: edmondsw: thorst:
esbergluThis is failing because admin_client.list_agents() is only returning 1 agent14:27
esbergluAnd the test assumes that there are at least 2 agents, agents[0] and agents[1]14:28
esbergluIs this a bug there? I'm not sure what the dyn_agent is supposed to be referring to14:28
esbergluFigured I would see if any of you knew before I looked any deeped14:28
edmondswnot sure14:29
*** tjakobs has joined #openstack-powervm14:33
*** thorst has quit IRC14:37
*** dwayne has joined #openstack-powervm14:42
*** thorst has joined #openstack-powervm14:42
*** thorst has quit IRC14:47
efriedesberglu Yeah, I don't know what kind of agents we're looking for here.15:02
efriedHave you deeped yet?15:02
esbergluNo I was stacking a test node to look into the in tree stuff15:03
esbergluAbout to15:03
*** dwayne has quit IRC15:05
esbergluefried: edmondsw: Brought this up in openstack-neutron15:17
esbergluThey agreed that it was a bug with the test15:17
esbergluI'm going to open a bug for it, then we can disable that test until it gets resolved15:17
efriedesberglu Beaut.  Are you planning to propose the fix?15:19
efriedThat would be good exposure.15:19
esbergluefried: I'm gonna get IT CI fixed first. But yeah if no one from neutron gets there before that point15:20
*** thorst has joined #openstack-powervm15:33
*** thorst has quit IRC15:34
*** thorst has joined #openstack-powervm15:34
esbergluefried: edmondsw: 548415:34
efriedesberglu Please recheck https://review.openstack.org/#/c/474799/ when you think OOT is ready.  (edmondsw FYI)15:34
esbergluefried: Should be ready as soon as 5484 merges15:34
efriedesberglu Oh, if we were running more than one agent, it would work?15:35
efriedWe can run the pvm-sriov agent, no harm no foul.15:35
esbergluefried: Yeah we could probably just toss the sriov agent on15:35
efriedThat would be a local.conf change.15:35
edmondswesberglu I thought it was just a single test15:35
esbergluedmondsw: It's failing the resource_setup for that group of tests15:35
esbergluSo it's 515:35
efriedEven if it was, I think adding the sriov agent so we can keep the tests is a better workaround.15:35
edmondswesberglu oh, nm... it was in setup... right15:35
esbergluefried: I would like to test that out before we do it15:36
esbergluBut I think that should work15:36
edmondswesberglu how long will trying that take?15:37
esbergluSo I say we just merge this for now. Then we can remove if the sriov agent bit works15:37
efriedesberglu So you want to disable the tests for now, then test it the other way (with sriov enabled), then twiddle it?15:37
efriedright, okay.15:37
efried+2 with that caveat.15:37
esbergluedmondsw: I would need to spawn, stack, and run tempest. I could just run those 5 tests though. Maybe an hour?15:37
esbergluBut I would rather get IT working and come back to this15:37
efriedLong as we don't lose sight of it.15:38
efriedHow many tests that we've put in the "further investigation" part of the skip list have we actually come back and re-enabled?15:38
efriedI'm afraid of our CI tech debt curve trending ever upward.15:38
esbergluefried: Hopefully I should have time to start making progress with the addition of mdrabe15:39
esbergluAnd the recent changes essentially getting rid of redeploys15:40
edmondswefried I've been wondering if we should actually be disabling more in the CI, to improve performance15:40
edmondswefried basically, only test things that have anything to do with us15:40
efriedI don't think our runtime is long enough to be worth that, do you?15:40
edmondswe.g. we've got tests disabled for identity v2 API... I see no value in ever re-enabling those15:40
edmondswefried I don't know15:40
edmondswhow long is it?15:40
efriedMore to the point, how long is it compared to the other 3rd-party CIs.15:41
efriedAnd it's on par.15:41
efried45-90m, depending.15:41
edmondswyeah, that seems long to me, regardless of what others take15:41
edmondswI'm not saying we have to do that right now... but it's an idea worth mulling over15:41
efriedI would be okay disabling tests that never hit the compute driver.15:42
edmondswit wouldn't just help with perf... would also help with stability if we don't have to worry about those disabled things breaking us15:42
efriedNot sure how many of those there are.15:42
edmondswefried that's exactly what I'm saying15:42
efriedBut if it hits the compute driver at all, even if it's supposedly testing something unrelated, we should keep it.15:42
efriedOkay, sure, add it to the etherpad if it ain't there already.15:43
edmondswwill do15:43
edmondswI added to https://etherpad.openstack.org/p/powervm_ci_todos15:45
edmondswI just rechecked https://review.openstack.org/#/c/474799... let's see if it passes :)15:46
esbergluedmondsw: What you aren't confident after a week+ of failures ;)15:49
*** k0da has quit IRC15:54
*** efried has quit IRC15:56
edmondswesberglu yeah... I want to see it :)16:04
edmondswesberglu I'm curious about the answer to efried's question in 548016:05
esbergluedmondsw: I was gonna open one then add it to the comment before merging16:06
*** thorst has quit IRC16:09
*** mdrabe has quit IRC16:13
edmondswesberglu 5479... why does one file include devstack-pvm-ci-check and the other doesn't?16:14
*** mdrabe has joined #openstack-powervm16:19
esbergluedmondsw: Well we don't want devstack-pvm-ci-check to run every time there is a devstack patch16:22
esbergluedmondsw: I think what we really want is to16:22
esberglua) get rid of either pvm-ci-check or noop-check-communication, they are essentially the same thing16:23
esberglub) Only run whichever of those 2 we keep against the ci-sandbox project16:23
edmondswand you intend to do that separately, I assume16:24
esbergluedmondsw: I left a comment (a) explained but before I realized we needed (b). Yes, I think a separate patch is more appropriate16:25
*** thorst has joined #openstack-powervm16:25
edmondswesberglu ah, I see it16:26
*** jwcroppe has quit IRC16:32
mdrabeesberglu: Found the commit in Jenkins causing the problem ae402ba139df82f3d105b9e070444f1849543fce16:34
mdrabeI didn't see a bug in nodepool when I checked last week, so I think I'll have to create one, will probably follow up with a fix16:34
edmondswesberglu are the CI failures in https://review.openstack.org/#/c/474799/ something you can explain?16:35
edmondswthe driver wasn't able to connect to NovaLink16:35
edmondswso the compute service didn't come up properly16:35
*** thorst has quit IRC16:35
esbergluedmondsw: I'm confident that it's something in the patch. Hold on let me take a quick look16:36
edmondswesberglu I sure hope not, and I would have expected problems with the patch to look differently16:36
edmondswbut you may be right16:36
esbergluedmondsw: Well not 100% confident. But I haven't seen this anywhere else OOT in the last couple days. And the only thing that has changed OOT is log publishing and disabling those 5 tests16:37
esbergluSo unless this starts hitting other patches...16:37
edmondswI'll kick off a recheck and see if it happens again16:37
esbergluedmondsw: Ok. I'll keep an eye out for it too16:38
*** thorst has joined #openstack-powervm16:52
*** thorst has quit IRC16:55
*** thorst has joined #openstack-powervm17:00
*** jpasqualetto has quit IRC17:02
*** efried has joined #openstack-powervm17:02
*** jwcroppe has joined #openstack-powervm17:03
*** jpasqualetto has joined #openstack-powervm17:11
*** openstackgerrit has joined #openstack-powervm17:14
openstackgerritMerged openstack/ceilometer-powervm master: Fix html_last_updated_fmt for Python3  https://review.openstack.org/47486117:14
efriededmondsw I think in order to get any attention on https://review.openstack.org/#/c/472794/ you're going to have to either hit the ML or finish the UT and get rid of the WIP.17:18
efriedShould do it sooner rather than later; the probability of getting non-feature patches through is going to be decreasing steadily between now and feature freeze.17:19
efriedAnd yeah, yeah, I still owe you a comeback on https://review.openstack.org/#/c/471773/17:19
efriedI think last week I convinced myself that the algorithm you've worked up is roughly as simple as it's gonna get, which I'm pretty sad about.17:20
efriedI think there's a couple of things we can/should do to make the code itself smaller, like combining the exception cases.17:20
efriedWill comment accordingly.17:20
*** jpasqualetto has quit IRC17:24
*** jpasqualetto has joined #openstack-powervm17:24
esbergluefried: edmondsw: Enabling the sriov agent allowed those tests to pass. Patch will be up in a few17:29
*** jwcroppe_ has joined #openstack-powervm17:35
edmondswefried yeah, I need to get back to that17:36
*** jwcroppe has quit IRC17:37
esbergluedmondsw: efried: 548617:38
edmondswneeds a comment17:40
*** dwayne has joined #openstack-powervm17:41
efriededmondsw Comments on https://review.openstack.org/#/c/471773/17:42
*** jwcroppe_ has quit IRC17:43
edmondswefried tx17:43
esbergluedmondsw: Done17:43
esbergluNot just your patch17:46
edmondswesberglu unfortunately that doesn't mean it's not the same issue... that change was already merged in-tree17:47
edmondswefried ^17:47
*** efried has quit IRC17:47
edmondswtalking the CONF.host change17:48
esbergluedmondsw: Oh. That would make sense. Because this looks very similar to the current IT runs17:48
esbergluOh wait this is an IT run17:48
edmondswhttps://review.openstack.org/#/c/474799 is OOT17:48
esbergluedmondsw: Yeah still makes sense. I thought I linked an oot run in just now17:48
esbergluedmondsw: I'm gonna try a test run with that change reverted IT and see what happens17:49
edmondswesberglu yeah17:50
esbergluedmondsw: You have a link handy for the IT change?17:51
edmondswesberglu https://review.openstack.org/#/c/473975/17:52
*** dwayne has quit IRC17:54
*** thorst has quit IRC18:05
*** dwayne has joined #openstack-powervm18:14
*** chhavi has quit IRC18:18
*** jwcroppe has joined #openstack-powervm18:20
*** jwcroppe has quit IRC18:25
*** jwcroppe has joined #openstack-powervm18:38
*** jwcroppe has quit IRC18:44
*** thorst has joined #openstack-powervm18:58
*** mdrabe_ has joined #openstack-powervm18:59
*** mdrabe has quit IRC19:00
*** jwcroppe has joined #openstack-powervm19:03
*** jwcroppe has quit IRC19:06
*** thorst has quit IRC19:15
*** thorst has joined #openstack-powervm19:15
*** jwcroppe has joined #openstack-powervm19:18
*** thorst has quit IRC19:20
*** edmondsw has quit IRC19:29
*** jwcroppe has quit IRC19:37
*** k0da has joined #openstack-powervm19:48
*** edmondsw has joined #openstack-powervm20:00
*** jwcroppe has joined #openstack-powervm20:04
*** smatzek has quit IRC20:09
*** jwcroppe has quit IRC20:12
*** jwcroppe has joined #openstack-powervm20:13
*** k0da has quit IRC20:14
*** kylek3h has quit IRC20:16
*** jwcroppe has quit IRC20:17
*** mdrabe_ is now known as mdrabe20:18
*** efried has joined #openstack-powervm20:20
efriededmondsw I'm gonna +W https://review.openstack.org/#/c/468565/ k?20:31
edmondswefried please20:31
edmondswefried I'm trying to stack on my neo and got "Failed to update apt repos, we're dead now", any ideas?20:33
efriedUnstack.  Run apt-get update and whack away at it until it succeeds.  Then restack.20:33
efriededmondsw Is this on a system that successfully stacked before (since it was last reinstalled)?20:35
efriedokay.  Then your sources are probably busted.20:35
edmondswefried yeah, there are a bunch of warnings about things appearing more than once20:35
edmondswwould that do it?20:36
efriedBest bet is probably to replace stuff in /etc/apt/sources* with whatever's on your other neo that worked.20:36
efriedNo, duplicates shouldn't be breaking it.20:36
*** smatzek has joined #openstack-powervm20:38
esbergluedmondsw: In tree passed reverting the CONF commit20:39
efriedwhoah, I must have missed something.20:39
efriedWe talking about changing MTMS to CONF.host?20:39
esbergluefried: Yeah20:40
openstackgerritMerged openstack/nova-powervm master: Remove volume_group_vios_name  https://review.openstack.org/46856520:40
edmondswefried yeah... apparently that did break everything20:40
edmondswtrying to figure out why20:40
efriedDo you still have a CI failure for that?20:40
esbergluefried: One sec20:41
efried...as in, logs I can look at.20:41
edmondswfor one20:41
efriedYeah, I guess it would be anything.20:41
efriededmondsw Well, that one is OOT.20:41
efriedI would prefer apples:apples20:41
edmondswthey're the same, but here's an IT:
*** jwcroppe has joined #openstack-powervm20:46
*** jwcroppe has quit IRC20:51
*** jwcroppe has joined #openstack-powervm20:51
*** jwcroppe has quit IRC20:56
efriededmondsw esberglu Here's the problem: https://review.openstack.org/#/c/473975/1/nova/virt/powervm/host.py@5720:57
efriedIt would seem as though data['hypervisor_hostname'] needs to match something in the list get_available_nodes returns.20:58
edmondswefried how did you figure that out?20:59
efriedLogs & code20:59
edmondswefried I figured... I was thinking more specifically :)20:59
efriedJun 20 08:05:35.660790 powervm-ci-powervm-devstacked-15470 nova-compute[6673]: DEBUG nova.compute.resource_tracker [None req-8d9e66da-cf03-4a01-ba08-c79df4328516 None None] Hypervisor/Node resource view: name=8247-21L*212A5CA free_ram=205056MB free_disk=917.44GB free_vcpus=7 pci_devices=None {{(pid=6673) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:747}}20:59
efriedNote name=<mtms>20:59
efried ComputeHostNotFound: Compute host powervm-ci-powervm-devstacked-15470 could not be found.20:59
efriedFrom compute_node_get_all_by_host21:00
edmondswI missed that name=<mtms>21:00
efriedWhich looks in resources['hypervisor_hostname'], where resources comes from the aforementioned.21:00
efriedI'll put up a patch to confirm.21:00
edmondswefried patch will be IT, I assume?21:01
edmondswthen we can just add to the existing OOT patch21:01
efriedYeah, you can patch that up in parallel if you like21:01
esbergluOnce that merges all CI should be passing (how many times have I said that now?)21:01
edmondswnot rush on OOT since that hasn't merged yet21:01
edmondswI'll wait and see exactly what you do IT so it matches21:02
efriededmondsw I'm not convinced this is the right way to go.21:03
efriedJust gonna make sure it works.21:03
efriedI'm thinking it might be best to get one of those values from the other.21:04
efriedBut I'm not sure how exactly that would work.21:04
efriedAnyway, let's see if this flies first.21:04
edmondswefried that's what I would have done21:04
openstackgerritMerged openstack/networking-powervm master: Updated from global requirements  https://review.openstack.org/47294021:13
openstackgerritMatthew Edmonds proposed openstack/nova-powervm master: Use CONF.host for powervm nodename  https://review.openstack.org/47479921:14
esbergluefried: edmondsw: https://github.com/powervm/powervm-ci/blob/master/devstack/prep_devstack.sh#L245-L24921:15
esbergluRight now we have a count on discover_hosts but we don't fail if we reach the count21:15
esbergluThinking that perhaps somehow the run could go through without that command succeeding in the timeframe21:16
esbergluBut I would rather just fail there if we reach the timeout21:17
edmondswesberglu do we even need the loop now that --strict is there?21:17
esbergluedmondsw: Yeah21:17
esbergluBecause we restart the services right before this21:17
*** smatzek has quit IRC21:17
*** efried has quit IRC21:18
esbergluSo we hit an issue where the command was "succeeding" but only because the service wasn't all the way up21:18
edmondswI thought --strict blocked21:18
esbergluWait maybe not21:18
esbergluYeah that's why21:19
esbergluBecause the first discover_hosts might fail because the service isn't up21:20
esbergluSo we keep retrying21:20
esbergluUntil the service is up and it finds a host21:20
esbergluOr we hit the timeout21:20
*** jwcroppe has joined #openstack-powervm21:22
*** efried has joined #openstack-powervm21:22
efriedStupid conference wifi resets after 24h.  I lost some stuff there.21:27
efriedI've got the change finished up and ready.  Tempted to push it.21:27
efriededmondsw I'll wait for the results, I guess.  But here's the UT delta FYI: http://paste.openstack.org/show/613220/21:29
efriededmondsw Ugh, I should have caught this in the other change; should use self.flags from UT, not CONF.21:30
efriedMebbe I'll add that to mine.21:31
edmondswefried yeah, go ahead21:32
*** smatzek has joined #openstack-powervm21:32
edmondswI didn't even put the UT in the OOT patch, because it's essentially pointless and wasn't there before21:32
efriededmondsw "wasn't there before" is a good reason to put it in.21:32
*** thorst has joined #openstack-powervm21:32
edmondswefried "essentially pointless" was my reason for not21:33
efriedthorst Airport?21:33
thorstI'm inbetween connections21:33
efriedI dunno how pointless it is.  The UT I added proves we're getting it from CONF.host.21:33
edmondswefried I didn't mean the UT you did... I meant the one I had made IT21:35
*** kylek3h has joined #openstack-powervm21:38
edmondswefried I'm adding the UT you made for test_host to the OOT patch... it's a good change21:38
efriededmondsw Roger that.  Let's get the CI passing both IT & OOT before we merge this time :)21:39
edmondswefried shouldn't I be telling you that? ;)21:39
efriedIt wasn't *really* anyone's fault; it was a chain of miscues.  I asked for core reviews while our CI was still failing; then mriedem asked me about the CI and I told him it was failing for unrelated reasons (which it was at the time - it wasn't getting past devstack).  But then he merged it anyway, which was premature.21:40
edmondswno worries, man21:40
*** jpasqualetto has quit IRC21:40
esbergluefried: Trying to push it off on me huh? lol21:40
efriedHah.  No, I think mriedem is probably mostly to blame.21:40
esbergluGood call blame outside the team21:41
efriedBut I don't want to do *anything* that will encourage him to be even *less* eager to merge our stuff.21:41
efriedAs it is, if he notices, he'll be damn sure never to merge anything of ours again without our CI passing.21:41
efriedesberglu I missed the discussion about --strict while my wifi was resetting, caught up on eavesdrop.  I noticed that hole when you put up that change, but I figured you were okay letting it fail down the line.  That said, I would also be fine with you failing it right there, which would be easier to diagnose.21:42
esbergluefried: Yeah your comment is why I eventually circled back21:43
esbergluThe way it is now it will make it all the way through tempest with like test failures about cell mappings21:43
esbergluWhich I hate21:43
esberglu*like 50 test failures21:44
efriedesberglu Well, if we fail it right there, do we still get the syslogs so we'd be able to diagnose why the compute service didn't start?21:44
esbergluefried: Yep. The log output and publishing are post build steps that happen even on failure21:46
efriedGreat, then make it so.21:46
esberglu5488 is already up for it21:46
efriedBTW, I thought of a fairly easy way to isolate which tests would be candidates to skip because they don't hit our driver.21:46
esbergluefried: Sweet what's that21:47
efriedPut up a patch where init_host raises an exception.21:47
efriedSee what passes.21:47
efriedNot 100% sure if that would work.21:47
*** thorst has quit IRC21:47
efriedWell, I'm sure anything that still passes would be legit to skip.21:47
efriedBut I'm not sure that hits everything we would want to skip.21:48
esbergluefried: At least something to start from. We can probably come up with some similar patch ideas and use the intersections to get a pretty decent list going21:49
efriedYeah.  Anything more granular I would be concerned about missing tests we should have as we add function to our driver.21:49
edmondswefried these CONF.host issues had me thinking the same thing :)21:49
efriedYeah, looking at that log, with CONF.host bustage, the compute service is basically worthless, though it actually does *run*.  So anything that passes must not care about the comput host.21:50
efriedIt would be interesting to see the difference in passing tests between that and when the compute service just dies completely.21:51
esbergluefried: Weird your patch didn't get picked up by CI21:51
esbergluI ran a recheck21:51
efriedOh, then stop and let me post the final version.21:52
esbergluIt's going this time21:52
esbergluefried: Go for it. When a new patch comes in it automatically aborts jobs on previous patches21:52
efriedesberglu Done.21:52
efriededmondsw ^^ FYI21:52
esbergluefried: Oh you were just waiting to see if it worked so didn't want to kill it probably21:52
efriedCause I gotta bail here pretty soon.21:53
efriedSo I'm not going to be around to see the result tonight, but will check in a.m.21:53
esbergluefried: Sounds good have a good evening21:55
edmondswesberglu efried I got a lot further stacking after fixing apt sources, but now got this: [ERROR] /home/neo/devstack/functions-common:602 git call failed: [git clone git://git.openstack.org/openstack/horizon.git /opt/stack/horizon --branch master]21:55
efriedTry changing to github21:55
edmondswthat seems odd21:55
edmondswhmm... if I can figure out where that came from21:55
esbergluedmondsw: We've had issues with g.o.o clones in the past. Had to change a bunch of them to github for CI21:56
efriededmondsw stackrc:GIT_BASE=${GIT_BASE:-git://git.openstack.org}21:56
edmondswesberglu they're just inconsistent?21:56
esbergluDid you get some message about a bad index pack or something like that?21:56
efriedChange to github.opestack.org I think21:56
esbergluedmondsw: Yeah they would just randomly fail sometimes. Othertimes would be getting like 10kB/s dl21:57
esbergluAnd we haven't noticed the same issue with github21:57
efriedI don't think github.openstack.org is right - but that's the place to change it.21:57
*** thorst has joined #openstack-powervm21:57
edmondswefried not just github.com?21:58
efriedI don't remember.  esberglu should know21:58
esbergluyeah github.com21:59
*** smatzek has quit IRC21:59
edmondswesberglu if we've hit this a bunch, why don't we have this in the local.conf for the CI?22:00
efriedWe do22:00
efriedOr if not in the local.conf, it's somewhere.22:01
esbergluWe preclone all of the openstack repos as part of the image template build using github.com22:01
edmondswefried it's not in local.conf... and seems like that would be the place to put it22:01
esbergluAnd I think devstack just uses the remotes set up from that22:01
efriedshrug, I just know we have it in the CI somehow.  I could find the change set(s) that did it, if necessary.22:02
efriedBut not today.22:02
efriedI'm out.22:02
esbergluWith the exception of applying patches from openstack gerrit22:02
*** dwayne has quit IRC22:02
edmondswesberglu didn't follow22:03
edmondswwe preclone everything, even stuff like horizon?22:04
esbergluWe build the nightly image template. In that we clone all of the openstack projects (nova, neutron, nova-powervm, everything) so that we don't have to clone it on each ready node22:04
esbergluWe then pull in any differences before stacking22:04
edmondswI'm actually curious as to why horizon would get pulled down with our local.conf, since we shouldn't need horizon in a CI22:04
*** thorst has quit IRC22:05
*** thorst has joined #openstack-powervm22:05
openstackgerritMatthew Edmonds proposed openstack/nova-powervm master: Use CONF.host for powervm nodename  https://review.openstack.org/47479922:06
*** mdrabe has quit IRC22:06
esbergluedmondsw: Umm I think there might be some horizon tests that we run? I can look into though22:06
esbergluedmondsw: I'm also heading out. Circle back on this topic tomorrow?22:07
edmondswesberglu yep, have a good night22:07
esbergluYou too22:07
*** thorst has quit IRC22:07
*** esberglu has quit IRC22:07
*** esberglu has joined #openstack-powervm22:08
*** efried has quit IRC22:08
*** openstack has joined #openstack-powervm22:16
*** edmondsw has quit IRC22:23
*** esberglu has joined #openstack-powervm22:32
*** esberglu has quit IRC22:37
*** jwcroppe has joined #openstack-powervm22:43
*** jwcroppe has quit IRC22:43
*** tjakobs has quit IRC22:49
*** jwcroppe has joined #openstack-powervm22:52
*** dwayne has joined #openstack-powervm22:57
*** jwcroppe has quit IRC23:03
*** jwcroppe has joined #openstack-powervm23:04
*** jwcroppe has quit IRC23:09
*** tjakobs has joined #openstack-powervm23:52
*** tjakobs has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!