Tuesday, 2017-03-21

*** edmondsw has joined #openstack-powervm00:03
*** edmondsw has quit IRC00:08
*** waler has quit IRC01:39
*** waler has joined #openstack-powervm01:51
*** waler has quit IRC02:26
*** waler has joined #openstack-powervm02:26
*** esberglu has joined #openstack-powervm02:35
openstackgerritzhangyanxian proposed openstack/ceilometer-powervm master: Optimize the link address for fetching git code  https://review.openstack.org/44776202:35
openstackgerritzhangyanxian proposed openstack/ceilometer-powervm master: Optimize the link address for fetching git code  https://review.openstack.org/44776202:37
*** esberglu has quit IRC02:39
openstackgerritzhangyanxian proposed openstack/networking-powervm master: Optimize the link address for fetching git code  https://review.openstack.org/44776302:39
openstackgerritzhangyanxian proposed openstack/networking-powervm master: Optimize the link address for fetching git code  https://review.openstack.org/44776302:40
*** waler has quit IRC04:20
*** esberglu has joined #openstack-powervm04:24
*** esberglu has quit IRC04:29
*** waler has joined #openstack-powervm04:32
*** waler has quit IRC04:59
*** waler has joined #openstack-powervm05:00
*** esberglu has joined #openstack-powervm06:13
*** esberglu has quit IRC06:17
*** k0da has joined #openstack-powervm07:46
*** a1fisher has joined #openstack-powervm07:58
*** esberglu has joined #openstack-powervm08:02
*** esberglu has quit IRC08:06
-openstackstatus- NOTICE: Wiki is broken with database problems, we are working to resolve it08:32
*** ChanServ changes topic to "Wiki is broken with database problems, we are working to resolve it"08:32
*** k0da has quit IRC08:35
*** ChanServ changes topic to "This channel is for PowerVM-related development and discussion. For general OpenStack support, please use #openstack."08:40
-openstackstatus- NOTICE: Wiki problems have been fixed, it's up and running08:40
*** openstackgerrit has quit IRC09:03
*** k0da has joined #openstack-powervm09:16
*** esberglu has joined #openstack-powervm09:51
*** esberglu has quit IRC09:56
*** k0da has quit IRC10:46
*** smatzek has joined #openstack-powervm11:09
*** edmondsw has joined #openstack-powervm12:00
*** edmondsw has quit IRC12:05
*** jpasqualetto has joined #openstack-powervm12:08
*** jpasqualetto has quit IRC12:14
*** svenkat has joined #openstack-powervm12:25
*** jpasqualetto has joined #openstack-powervm12:26
*** edmondsw has joined #openstack-powervm12:26
*** kjw3 has quit IRC12:29
*** kjw3 has joined #openstack-powervm12:30
*** kylek3h has joined #openstack-powervm12:37
*** efried has joined #openstack-powervm12:52
*** esberglu has joined #openstack-powervm12:58
esberglu#startmeeting powervm_driver_meeting13:01
openstackMeeting started Tue Mar 21 13:01:45 2017 UTC and is due to finish in 60 minutes.  The chair is esberglu. Information about MeetBot at http://wiki.debian.org/MeetBot.13:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:01
openstackThe meeting name has been set to 'powervm_driver_meeting'13:01
esberglu#topic In Tree Driver13:02
efriedFour change sets are rebased and updated.13:02
esbergluYep looking at those is first on my list this morning13:02
esbergluMostly just rebasing or functional changes?13:03
efriedBottom two passed CI; top two failed.  I just submitted rechecks for 'em.  #4 failed one test, rebuild in error state or something.  #3 failed that same one and two more.13:03
esbergluYeah will talk about CI failures later in the meeting13:03
efriedActually almost no functional changes - a couple of method signatures and the way flavors are passed around.  The vast majority was UT changes.13:03
*** apearson has joined #openstack-powervm13:04
efriedSo I don't expect CI failures due to my changes.13:04
efriedFact that #2 succeeded supports that.13:04
esbergluYeah those failures are unrelated13:05
efriedCause the method signature changes started there.13:05
efriedSo today I plan to rebase the SSP change set.  Then jayasankar_ can retest that guy and we can see if there's still that weird glance bug to nail down.13:05
efriedSay, is jayasankar_ on this meeting notice?13:05
esbergluNo I will add him13:06
efriedAnd Nilesh too13:06
efriedI don't have anything else.13:06
esbergluI think jayasankar_ hit some issues testing in-tree, I haven't looked into yet but we may have an issue there13:07
esbergluGonna help him with that today if hes online13:07
esbergluThat's all I have for in-tree13:07
esberglu#topic Out of Tree Driver13:07
esbergluGuessing there isn't gonna be much to talk about here13:07
adreznecWe need to get that new pypowervm release out13:08
efriedI've still been backporting some of the changes from in-tree.13:08
esbergluRechecked all changes to the stable/* branches, so the backlog is cleared there13:08
adreznecCurrently our reqs issue is blocking an OSA SHA release13:08
efriedI'll put up a new OOT change set once I'm done with the SSP stuff.13:08
adreznecThe PTL come to talk to me about it yesterday13:09
efriedGuess you'll be on jfoliva's ass today.  Sh*t rolls downhill.13:09
adreznecI pinged Julio and I believe he and Dom merged the change, but getting that out today is my priority13:09
adreznecSo we can merge https://review.openstack.org/#/c/440811/13:09
efriedWe'll have to put up a new patch set on that.13:10
efriedbut yeah13:10
adreznecI'll drive that13:10
*** mdrabe has joined #openstack-powervm13:11
efried#action adreznec to hold the whip on jfoliva to get the new pypowervm release out.13:11
efried#action adreznec to close https://review.openstack.org/#/c/440811/13:11
*** tblakes has joined #openstack-powervm13:12
esberglu#topic CI13:13
esbergluWe are getting Http 500 Errors, it is hitting multiple tests13:14
esbergluI'm guessing that's what hung up your patches efried13:14
esberglu^ example13:14
esbergluI haven't had a chance to debug yet, been putting out other CI fires for the last week or so13:14
esbergluBut it's on my radar for today13:15
esbergluI know that it was hitting some of the "rebuild" tests as well13:15
esbergluThe other thing I wanted to talk about is how to determine which tests to run for in-tree and how/when we are going to put those out13:17
efriedunknown internal error with half a dozen blank lines in the <Message>13:17
efriedThat's gonna be fun to debug.13:17
efriedWe don't save off the REST logs, do we.13:17
esbergluWe could13:17
efriedBut I don't think my failures were the same, fwiw.13:17
efriedThe 500 showed up in the html report here, but the same wasn't happening in my failures.13:18
efriedAnyway, leave it to you to debug.  Let me know if you need help.13:18
esbergluSounds good13:18
esbergluAnyway back to the in-tree whitelist13:18
efriedCheck with apearson whether it's okay to publish REST logs publicly.13:18
esbergluK. The problem with the whitelist is that there isn't a good way to know which tests will be supported when we add certain functionality13:19
esbergluWhat I have been doing13:19
efriedesberglu One strategy is: with each new change set, make sure OOT is stable, then run the in-tree change set with the OOT config.13:19
efriedWhatever passes, makes the whitelist.13:19
esbergluYep that's what I have been doing13:19
efriedIt's simple and easy.  But it's not particularly scientific.13:19
efriedWhat we really should be doing (ugh) is inspecting each test and seeing if it *should* pass or fail.13:20
efriedIf it should fail and it passes, we should open a bug for that, and potentially fix the test.13:20
esbergluYeah, but that's a HUGE time investment13:20
efriedI suspect there are a number of tests that fit that category.13:20
efriedIf it should pass and it fails, likewise nail that down.13:20
efriedYes.  Possibly something jayasankar_ and nilesh could help out with.13:21
esbergluThe other thing related to this is when we are going to put changes out13:22
efriedto the in-tree whitelist?13:22
esbergluRight now we are pulling in the 1st patch for all in-tree runs13:22
esbergluOnce that 1st one merges, are we going to start pulling in the 2nd patch and test that for all in-tree?13:22
efriedWell, I don't think the methodology has to do with change-by-change so much as by when (at which change set boundaries) the whitelist changes.13:23
efriedThe whitelist may actually be the same for the first four or five change sets.13:23
efriedIs it?13:24
esberglupower on/off will have an associated whitelist change13:24
efriedOkay, so #1 and #2 have the same whitelist, then it changes for power on/off, then it stays the same again until SSP, I'm guessing.13:25
efriedor maybe console.13:25
esbergluYep I don't think any of the 4 spawn/destroy will change anything13:26
esbergluBut my issue is still present when we get to SSP13:26
esbergluWe can't change the whitelist until that change is in13:26
esbergluMeaning we won't get much CI volume on that change13:26
esbergluUnless we do what we are doing with PS1 right now and pull it in for every in-tree run13:26
efriedUnless we change our setup to pull in that guy13:26
efriedThen with the lower change sets we would have to figure out how to allow them to revert down the tree.13:27
*** smatzek has quit IRC13:27
efriedUntil they merge.13:27
efriedWhich is gonna be a while, way things are going.13:27
efriedSo maybe we should look into enabling that.13:27
efriedThen we could essentially move up the whitelist and the baseline change set any time we have a change set we consider stable.13:28
efriedRight now we're doing something like, "If the path from the current change set back to tip of master contains our baseline change set, don't apply our baseline change set."13:29
*** jpasqualetto has quit IRC13:30
esbergluOkay I like that going forward. For now I think we are stable through the 4th spawn delete (full flavor). So are we ready to move our baseline to that?13:30
esbergluAnd yeah that's what we are currently doing13:30
efriedWe would have to do that, but also, "If the path from our baseline back to the tip of master contains the current change set, don't apply our baseline change set."13:30
efriedYes, I'm happy to move our baseline to that once we have the above logic working.13:31
efriedBut not until then.13:31
efriedCause otherwise change sets 1-3 will fail.13:31
efriedThey won't fail the CI run - they'll fail on the git shuffle.13:31
esbergluOkay. I can look into that on staging, might bug you later if I have questions13:32
efriedCause you'll be trying to cherry-pick, say, #2 onto the tip of a chain that already contains #2.13:32
esbergluI think that --allow-empty gets past that no?13:32
esbergluor --keep-redundant13:32
esbergluI think13:32
efriedIt may.  But I'm not sure.  There's a more explicit way to do it.13:32
esberglu#action esberglu: Add CI logic for applying proper changesets13:34
efried    if git log --pretty=format:%H origin/${ZUUL_BRANCH}..HEAD | grep -q $commit; then13:34
efriedThat's what we're doing today.13:34
efriedThe new thing is going to be a little more complicated because we don't yet have $commit downloaded.13:34
efriedoo, it's even more complicated because we're theoretically looping through and cherry-picking multiple commits.13:35
efriedThis is gonna be fun.13:35
*** mdrabe has quit IRC13:36
efriedwe don't need to solve it here.13:36
efriedWe can actually do the git fetch and compare origin/${ZUUL_BRANCH} to FETCH_HEAD13:36
*** mdrabe has joined #openstack-powervm13:37
efriedAnd just skip the cherry-pick.13:37
efriedSo yeah, we don't need to solve it here.13:37
efriedBut that solves it.13:37
esbergluOkay. The only other thing I have for CI is OSA CI13:38
* efried backs away slowly13:38
esbergluCurrently just trying to get run_playbooks script to work13:39
esbergluWhich is what runs basically the entire set of ansible playbooks13:40
esbergluIt took forever to get that env. stable, but I think it finally is and I can start grinding through the failures there13:40
esbergluThat's all for me today. Any other topics/thoughts?13:41
adreznecAny specific issues on the OSA CI at this point?13:41
esbergluIt's failing to install pip13:41
adreznecThat... seems odd13:42
esbergluYeah. I've only tried the script once and didn't have time to debug yet13:42
esbergluSo it might be something trivial13:42
esbergluI'll keep you posted13:43
efriedesberglu #3 and #4 failed in-tree CI again.13:44
esbergluUgh. Same thing?13:44
efriedand test_multiple_create13:45
esbergluHttp 500 or not?13:45
efriednot from us.13:45
efriedLooks like might be neutron glitches.13:46
efriedWant me to recheck again, or leave 'em?13:46
esbergluLooks like the test_multiple_create is hitting all of the OOT runs since last night13:48
efriedis it new?13:49
efriedthe test, that is?13:49
efriedsorry, we shouldn't derail the meeting with this.  Is the meeting over?13:49
esbergluYeah I think so13:49
openstackMeeting ended Tue Mar 21 13:49:30 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)13:49
openstackMinutes:        http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-03-21-13.01.html13:49
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-03-21-13.01.txt13:49
openstackLog:            http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-03-21-13.01.log.html13:49
esbergluAnd no I don't believe that is a new test13:49
esbergluBut I'm checking now13:49
*** jpasqualetto has joined #openstack-powervm13:50
esbergluNope not new13:51
esbergluI will start debugging13:51
*** jpasqualetto has quit IRC13:52
*** smatzek has joined #openstack-powervm13:59
*** jpasqualetto has joined #openstack-powervm14:07
*** tjakobs has joined #openstack-powervm14:09
*** burgerk has joined #openstack-powervm14:22
esbergluefried: That test_multiple_create is failing with "Multiple Possible Networks Found" which has been very difficult to debug in the past14:25
esbergluStill looking14:26
esbergluBut nothing obvious in the logs so far14:26
*** k0da has joined #openstack-powervm14:42
*** openstackgerrit has joined #openstack-powervm14:48
openstackgerritgordon chung proposed openstack/ceilometer-powervm master: use new instance polling interface  https://review.openstack.org/44813414:49
*** mdrabe has quit IRC14:50
esbergluefried: I think the test_multiple_create just comes down to the way we have our net. configured14:51
*** tblakes has quit IRC14:55
*** mdrabe has joined #openstack-powervm14:58
*** tblakes has joined #openstack-powervm15:19
esbergluefried: You have any time to take a look?15:25
*** jayasankar_ has quit IRC15:49
efriedesberglu I could try...15:53
efriedWhat are we looking at?15:53
esbergluSo it's finding multiple possible networks and doesn't know which one to use15:54
esbergluWhich we have hit problems with before15:54
efriedI get a 404 on that link.15:54
efriedThere's an extra B in the URL15:55
esbergluOh theres an extra B in there15:55
efriedIs this in-tree?15:55
esbergluIt's hitting every run right now15:55
efriedAnd we've had multiple networks in the past when some other test has been creating them and something went wrong?15:56
esbergluYeah, we've hit issues with multiple possible networks found a few times15:56
esbergluLeading us to change our network config. to shared15:57
esbergluI think. Or maybe it was after we changed it to shared that we saw it15:57
esbergluThere is a section in the skip list with the comment "Failing becuase we are using shared networks"15:59
esbergluAnd I think they were tests also hitting this multiple networks issue16:00
esbergluAnd we tried to debug for a while and didn't get anywhere16:00
esbergluIn this case, I don't think there are other tests having issues. I think it is a timing thing and there are actually just multiple networks at that point in time16:02
esbergluAnd it doesn't know which to choose16:02
esbergluHard to say though, and I have been struggling to find anything useful in the logs16:03
efriedOthers must have this issue.  Maybe we can ask around on #openstack-qa?16:12
efriedI suppose attempting a run with concurrency=1 would take a nontrivial redeploy of something-or-other?16:12
esbergluActually I don't think it would. I could put up a patch to os_ci_tempest.sh and pull it in as part of the jenkins job without redeploying I think16:15
esbergluLet me confirm quick16:15
esbergluYeah I could just reload the jenkins config to get it in16:16
esbergluThink we should try it?16:18
*** a1fisher has quit IRC16:32
efriedesberglu If you're seeing that failure regularly, then yeah, I think it would be informative, if it's not too much trouble.  Of course, the run will take a lot longer, so it's not a viable long-term solution, but it should give us a better baseline for debugging.16:34
openstackgerritEric Berglund proposed openstack/nova-powervm master: DNM: ci check  https://review.openstack.org/32831516:43
esbergluefried: ^ Testing on that patch16:44
openstackgerritgordon chung proposed openstack/ceilometer-powervm master: use new instance polling interface  https://review.openstack.org/44813416:58
*** IRCFrEAK has joined #openstack-powervm17:50
*** IRCFrEAK has quit IRC17:52
*** adreznec has quit IRC17:55
*** adreznec has joined #openstack-powervm17:57
*** jpasqualetto has quit IRC19:04
*** jpasqualetto has joined #openstack-powervm19:16
openstackgerritgordon chung proposed openstack/ceilometer-powervm master: use new vnic polling interface  https://review.openstack.org/44824919:27
*** esberglu has quit IRC20:00
*** esberglu has joined #openstack-powervm20:00
*** esberglu has quit IRC20:05
*** esberglu has joined #openstack-powervm20:32
*** smatzek has quit IRC20:33
esbergluefried: Bad news. Still failing test_multiple_create with concurrency=120:34
efriedesberglu That's not actually bad news.  Means we don't have a timing problem with other tests.20:34
esbergluTrue. I'm looking into the test now to see what could be going wrong20:35
*** svenkat has quit IRC20:57
esbergluefried: Okay I think I see what's happening21:15
efriedtalk t ome21:15
esbergluSo you can see there they changed to call compute.create_test_server21:15
esbergluWhen it was calling self.create_test_server21:16
esbergluIt would eventually get to here21:16
esbergluWhich would set the network based on the fixed_network_name in the conf21:16
efriedDo we have such a thing?21:17
efriedI thought we did.21:17
esbergluBut there isn't any equivalent of that when calling compute.create_test_server21:17
efriedoh, yeah, sorry, I was backwards.21:17
*** apearson has quit IRC21:19
esbergluI think so. Making sure there isn't something else missing on our side21:21
efriedesberglu Other CIs aren't failing because.... they only have one network defined??21:36
efriedadreznec You got a problem with https://review.openstack.org/#/c/447762/ ?21:38
adreznecefried: Looking21:41
*** jpasqualetto has quit IRC21:41
efriedadreznec And should we do that for the other projects?21:41
efried(he did it for networking-powervm - added you)21:42
*** burgerk has quit IRC21:43
adreznecefried: I guess it's fine. Not really a big deal to me21:46
*** svenkat has joined #openstack-powervm21:46
adreznecWhy'd it fail CI?21:46
efriedYeah, I'm not seeing the problem with using http.21:46
*** tblakes has quit IRC21:46
efriedPresumably because of the thing esberglu is looking into.21:46
adreznecAh I see21:47
adreznecHadn't read backscroll21:47
*** jpasqualetto has joined #openstack-powervm21:57
*** tjakobs has quit IRC22:00
*** mdrabe has quit IRC22:04
*** k0da has quit IRC22:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!