Tuesday, 2017-02-07

*** smatzek has joined #openstack-powervm00:01
*** smatzek_ has joined #openstack-powervm00:03
*** svenkat has joined #openstack-powervm00:04
*** smatzek has quit IRC00:06
*** kriskend has quit IRC00:10
*** seroyer has quit IRC00:15
*** thorst_ has joined #openstack-powervm00:19
*** smatzek_ has quit IRC00:25
*** seroyer has joined #openstack-powervm00:36
*** seroyer has quit IRC00:49
*** dwayne has quit IRC00:54
*** kylek3h_away has joined #openstack-powervm01:06
*** thorst_ has quit IRC01:09
*** seroyer has joined #openstack-powervm01:44
*** thorst_ has joined #openstack-powervm02:48
*** thorst_ has quit IRC02:48
*** burgerk has joined #openstack-powervm03:00
*** thorst_ has joined #openstack-powervm03:04
*** thorst_ has quit IRC03:04
*** burgerk has quit IRC03:22
*** edmondsw has joined #openstack-powervm03:37
*** seroyer has quit IRC03:39
*** edmondsw has quit IRC03:41
*** thorst_ has joined #openstack-powervm03:59
*** thorst_ has quit IRC04:00
*** thorst_ has joined #openstack-powervm04:19
*** thorst_ has quit IRC04:19
*** nbante has joined #openstack-powervm04:59
*** svenkat has quit IRC05:08
*** thorst_ has joined #openstack-powervm06:20
*** thorst_ has quit IRC06:25
*** Jay2 has joined #openstack-powervm07:05
*** edmondsw has joined #openstack-powervm07:15
*** edmondsw has quit IRC07:19
*** Jay2 has quit IRC07:37
*** k0da has joined #openstack-powervm07:56
*** thorst_ has joined #openstack-powervm08:21
*** thorst_ has quit IRC08:26
*** svenkat has joined #openstack-powervm09:52
*** svenkat has quit IRC09:56
*** thorst_ has joined #openstack-powervm10:22
*** thorst_ has quit IRC10:26
*** svenkat has joined #openstack-powervm10:33
*** svenkat has quit IRC10:37
*** edmondsw has joined #openstack-powervm11:33
*** jpasqualetto has joined #openstack-powervm11:37
*** edmondsw has quit IRC11:37
*** svenkat has joined #openstack-powervm11:37
*** svenkat has quit IRC11:42
*** smatzek_ has joined #openstack-powervm12:08
*** nbante has quit IRC12:10
*** thorst_ has joined #openstack-powervm12:39
*** seroyer has joined #openstack-powervm12:44
*** svenkat has joined #openstack-powervm12:47
*** svenkat has quit IRC12:51
*** tblakes has joined #openstack-powervm13:09
*** svenkat has joined #openstack-powervm13:18
*** edmondsw has joined #openstack-powervm13:23
*** nbante has joined #openstack-powervm13:47
*** apearson has joined #openstack-powervm14:02
*** wangqwsh has joined #openstack-powervm14:02
thorst_driver meeting?14:02
efriedadreznec wangqwsh ?14:03
thorst_I think esberglu normally drives it14:04
efriedWe need him for some topics, but we can make progress while we wait.14:06
thorst_you want to kick off meeting?14:06
efried#startmeeting PowerVM Driver meeting14:06
openstackMeeting started Tue Feb  7 14:06:25 2017 UTC and is due to finish in 60 minutes.  The chair is efried. Information about MeetBot at http://wiki.debian.org/MeetBot.14:06
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:06
openstackThe meeting name has been set to 'powervm_driver_meeting'14:06
thorst_I still owe you a review...I know that14:06
efried#topic power on/off module refactor14:06
efried#action thorst_ to review14:07
efried#action efried to make sonar happy14:07
* thorst_ review all the things14:07
efriedThe UT I put up last night gives me a fairly high degree of confidence that what I have is backward compatible.14:07
efriedCause I wrote it against the old code first.14:07
efried#action esberglu to pull it into the CI env and give it a spin.14:08
efriedThat'll serve two purposes.  1) Make sure it works at all; and then 2) give us an easier way to make tweaks if we still see those power-off timeouts we were getting.14:09
thorst_dig it14:09
efried#topic In-tree change sets14:09
efriedI believe I have the bottom five properly rebased on one another, and I was in the process of going through the ovs-vif one when the power train left the station.14:10
efriedwow, that was WAY too many metaphors mixed in there.14:10
thorst_and the in-tree CI runs against them now?14:11
efriedRuns, yes.  We still get bitten by new tests seemingly daily.14:11
thorst_even though its a white list?14:11
thorst_how could we get bit?14:11
efriedThat's a great question.14:12
efriedMaybe I'm wrong.14:12
efriedBut we need esberglu to answer that.14:12
thorst_I bet we got clever with the whitelist14:12
thorst_and have some wildcard flags14:12
thorst_that pick up new tests14:12
thorst_I think mriedem really just wanted a couple tests in there...14:12
efriedNope, it's all UUIDs14:13
efriedLooking at the latest run, it didn't even get started.14:13
thorst_then we'll wait for esberglu14:13
efriedstack failed14:13
thorst_wonder if that's like Jay's14:13
thorst_but I haven't been keeping an eye on them14:13
*** mdrabe has joined #openstack-powervm14:14
thorst_OK - so I think I need to take some time investigating the state of the in-tree.14:15
efriedActually, I was looking at the OOT CI for the first in-tree change, which is still failing on duplicate conf options.  I don't see the in-tree having run at all.14:15
thorst_#action thorst to look over the in tree stuff a bit more now14:16
thorst_what else is there for today?14:17
*** Jay2 has joined #openstack-powervm14:18
efriedwangqwsh Ni you shama?14:18
efriedGuess we call it early, and fup with esberglu later.14:18
openstackMeeting ended Tue Feb  7 14:19:00 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:19
openstackMinutes:        http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-02-07-14.06.html14:19
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-02-07-14.06.txt14:19
openstackLog:            http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-02-07-14.06.log.html14:19
*** wangqwsh has quit IRC14:27
*** Jay2 has quit IRC14:28
*** wangqwsh has joined #openstack-powervm14:28
*** wangqwsh has quit IRC14:33
*** smatzek_ has quit IRC14:35
*** smatzek_ has joined #openstack-powervm14:55
*** smatzek__ has joined #openstack-powervm14:56
*** tjakobs has joined #openstack-powervm14:59
*** smatzek_ has quit IRC15:01
*** smatzek_ has joined #openstack-powervm15:02
*** seroyer has quit IRC15:02
*** smatzek__ has quit IRC15:06
*** tjakobs has quit IRC15:08
*** kriskend has joined #openstack-powervm15:13
*** Jay2 has joined #openstack-powervm15:15
*** seroyer has joined #openstack-powervm15:16
*** nbante has quit IRC15:37
*** mdrabe has quit IRC15:40
thorst_efried: good with me merging in https://review.openstack.org/#/c/408758 ?15:42
efriedthorst_ Yup.15:42
*** tjakobs has joined #openstack-powervm15:45
*** mdrabe has joined #openstack-powervm15:48
openstackgerritMerged openstack/nova-powervm master: First pass at imagecache  https://review.openstack.org/40875815:50
*** esberglu has joined #openstack-powervm15:53
efriedesberglu - here's the minutes http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-02-07-14.06.html and log http://eavesdrop.openstack.org/meetings/powervm_driver_meeting/2017/powervm_driver_meeting.2017-02-07-14.06.log.html from this morning.  Let me know when you're caught up.16:00
efriedesberglu Main thing to look at is: where's the in-tree CI run for https://review.openstack.org/391288 ?16:08
esbergluefried: I think you're confused about the in-tree CI. It's still running silent, so it isn't linking there16:08
efriedesberglu Oh.  So where do I go to look at it?16:08
esbergluI haven't turned it on like the OOT because it will fail every single patch that isn't our in-tree stuff16:09
esbergluUntil the 1st in-tree patch is merged16:09
esbergluI will link you some logs16:09
*** burgerk has joined #openstack-powervm16:10
*** openstackgerrit has quit IRC16:35
esbergluThose are all of the runs from the latest patchset16:35
thorst_efried: you still want me to hold off on the review for power, or rip it?16:39
efriedthorst_ Rip it.16:39
efriedthorst_ Still trying to satisfy sonar a bit, but shouldn't affect anything.16:41
thorst_I'm mostly good with it...I think there are some ways you can cut complexity16:45
thorst_of course you uploaded a new rev as I was commenting16:48
thorst_I think some of my comments are dupes16:48
thorst_no, don't think they are16:49
thorst_give it a look16:49
efriedDude, thorst_, you're worse than sonar.16:54
adreznecefried: Is that a compliment? Insult? Both at once?17:09
*** nbante has joined #openstack-powervm17:13
*** jpasqualetto has quit IRC17:18
*** efried has quit IRC17:28
thorst_a delay that long from efried means its an insult  ;-)17:28
*** jpasqualetto has joined #openstack-powervm17:35
*** esberglu has quit IRC17:38
*** efried has joined #openstack-powervm17:41
*** efried has quit IRC17:50
*** esberglu has joined #openstack-powervm17:52
*** efried has joined #openstack-powervm17:54
*** jpasqualetto has quit IRC17:56
efriedthorst_ Responses on the power change set.  Only one thing I'm going to act on right now; other issues are pending discussion.18:12
*** jpasqualetto has joined #openstack-powervm18:14
*** nbante has quit IRC18:16
thorst_efried: ok18:47
*** apearson has quit IRC19:00
*** apearson has joined #openstack-powervm19:01
esbergluefried: CI runs are going through with the power refactor. Results should come through within the hour19:19
*** jpasqualetto has quit IRC19:27
efriedesberglu noyce.  What did you have to do to get the patches to stack up right?19:35
esbergluPut the refactor change 1st19:35
efriedesberglu And remove 4761?19:36
efriedHope so, else results will be pretty unpredictable ;-)19:36
efriedesberglu Can I go look at the pypowervm code somewhere this is running?19:37
esbergluOne sec I'll pm19:37
*** jpasqualetto has joined #openstack-powervm19:44
*** apearson has quit IRC20:05
thorst_esberglu: there/20:07
thorst_I can move the VMs running our CI over to a SAN now20:07
thorst_should give us much higher throughput...better perf and all that good stuff20:08
thorst_but not sure when is a good time to do that20:08
thorst_will have same issue with morpheus.20:08
thorst_but I need you to start thinking about when you want me to do that.20:08
thorst_it means an hour or so of downtime for the VMs...maybe a bit more20:09
thorst_efried: you probably care about that too  :-)20:10
*** apearson has joined #openstack-powervm20:12
esbergluWhat issue with morpheus?20:15
*** jpasqualetto has quit IRC20:18
efriedadreznec thorst_ esberglu - Anybody remember what we do with pastebin et al when we see 'cannot import name deploy'?  Jay2 is hitting that.20:21
thorst_efried: uninstall pastebin20:21
thorst_pip install pastebin20:21
thorst_esberglu: morpheus is just slow20:21
esbergluI'm thinking we at least want to get the power-off stuff finished first20:27
mdrabeadreznec esberglu So any chance of getting the CI rebuilt for https://review.openstack.org/#/c/425780/?20:36
*** jpasqualetto has joined #openstack-powervm20:36
adreznecmdrabe: what do you need rebuilt for that exactly? newer novalink?20:37
efriedthorst_ esberglu - new rev of the power change set is up, with nontrivial changes.  Need new review & CI deploy, respectively.20:38
adreznecthorst_: esberglu ^^ so I assume we'd need things on near-latest for that mdrabe?20:38
mdrabeYea, the very latest, for something that merged hours ago20:38
adreznecOk, figured that would be the case. thorst_ efried can you think of anything that was changed from an API/core standpoint that would break say a Mitaka run?20:40
efriedoh, absolutely.  But I don't think mdrabe is looking for mitaka or even newton.20:40
adreznecBut if we need the latest novalink code to get these runs passing CI20:41
efriedWe always use the latest novalink code for master20:41
adreznecYou use the latest pypowervm20:41
efriedThe only reason we don't update it on every run is because we do the whole pre-preparing nodes thingy.20:42
adreznecBut the underlying novalink code is still
adreznecFrom REST down20:42
adreznecOne of the... complications... of the CI environment20:42
efriedIs it?  I didn't think so.20:42
adreznecI'm pretty sure it is... esberglu would know for sure20:43
adreznecOtherwise let me look20:43
efriedI thought we were updating the underlying novalink packages to dev levels every time they put in a feature we built on top of in e.g. pypowervm.20:43
efriedThat's the only thing that makes sense.20:43
efriedIt's just a question of how disruptive it is to get it in there.20:43
mdrabeIt would make sense for pypowervm20:44
adreznecii  pvm-rest-app                             ppc64el      The PowerVM NovaLink REST API Application20:44
adreznecii  pvm-novalink                            ppc64el      Meta package for all PowerVM Novalink packages20:44
efriedI'm not well-versed on the specifics, but it seems like there's at least three levels of "rebuild the CI env" depending on what we need to pick up.20:44
adreznecFrom at least 2 of the nodes20:44
mdrabeBut right now my change is failing in the CI with exactly what I expected, and it's b/c the REST level isn't right20:44
efriedSo is this one of the "really rebuild the whole shootin match" times?20:44
adreznecWe shouldn't have to do thart20:45
mdrabeYea, and it'll happen any time we've got dependant changes coming up the stack20:45
adreznecThe only part that really needs to change is to upgrade the novalink packages20:45
adreznecFrom ->
adreznecThe question is more what the impact of doing that is20:46
adreznecDo we break CI runs for mitaka, for example, by doing that?20:46
mdrabeI mean, it'd have to happen eventually no? Can't stay on forever?20:46
mdrabeIs it possible to keep a few nodes on and dedicate those to mitaka runs?20:47
adreznecesberglu: is there an "easy" way to bump staging to and see what happens to a stable/mitaka run?20:47
adreznecmdrabe: Not super easily right now... not a lot of flexibility in the jenkins/nodepool setup there IIRC20:47
adreznecDedicated node types were something that esberglu was looking at when we moved to zuulv320:48
esbergluWow picked the wrong time to grab a coffee. Let me get caught up quick20:49
thorst_if we're talking about a full rebuild...lets tie it to my 'fix the disks' work.20:51
adreznecesberglu: tsk tsk, you should know the CI requires constant vigilance!20:51
adreznecthorst_:  We shouldn't be20:51
thorst_we could take that on tomorrow even.20:51
adreznecAt least not for this20:51
esbergluI don't think we need a full rebuild? Pretty sure we can just update the packages right?20:52
adreznecThe question is what, if anything, that breaks20:52
esbergluWell looks like testing on staging is the way to go20:53
esbergluI will update it there20:53
adreznecSounds like a plan20:53
thorst_all in progress runs would likely be impacted20:53
thorst_but you could also do a rolling upgrade through the under cloud20:54
adreznecmdrabe: esberglu will let you know how it works on staging20:54
thorst_just put certain hosts in 'maintenance' (no new deploys there)20:54
thorst_and update that host when all the workload is off of it20:54
adreznecIf it goes well we could do a rolling upgrade like thorst_ is describing20:54
mdrabecool thnx guys20:54
thorst_mdrabe: I liked your blog post20:54
adreznecIf not we'll have to brainstorm a bit on how to handle things20:54
efriedWe don't run on mitaka anymore, I thought.20:56
mdrabethorst_ Now to get that support in nova20:56
efriedAnd it's *possible* that won't break newton.20:56
*** jpasqualetto has quit IRC20:57
esbergluWe are also using a newton undercloud. So this could potentially impact that as well right?21:06
thorst_efried: better not break newton21:06
thorst_I think you're just saying you haven't verified it21:06
thorst_so this is a good thing to test.21:06
esbergluDo I have to restart anything after moving from -->
adreznecesberglu: things should restart on their own I think21:12
*** smatzek_ has quit IRC21:29
*** seroyer has quit IRC21:33
*** seroyer has joined #openstack-powervm21:54
*** smatzek_ has joined #openstack-powervm22:00
*** thorst_ has quit IRC22:01
*** svenkat has quit IRC22:02
*** thorst_ has joined #openstack-powervm22:03
*** thorst_ has quit IRC22:08
*** Jay2 has quit IRC22:09
*** thorst_ has joined #openstack-powervm22:20
*** thorst_ has quit IRC22:25
*** smatzek_ has quit IRC22:26
*** edmondsw has quit IRC22:29
*** burgerk has quit IRC22:30
*** tblakes has quit IRC22:46
*** seroyer has quit IRC22:55
*** edmondsw has joined #openstack-powervm22:59
*** esberglu has quit IRC23:00
*** esberglu has joined #openstack-powervm23:00
*** esberglu has quit IRC23:01
*** edmondsw has quit IRC23:04
*** kriskend has quit IRC23:04
*** mdrabe has quit IRC23:09
*** esberglu has joined #openstack-powervm23:16
*** esberglu has quit IRC23:18
*** tlian has quit IRC23:26
*** apearson has quit IRC23:34
*** tlian has joined #openstack-powervm23:42
*** tjakobs has quit IRC23:49
*** seroyer has joined #openstack-powervm23:49

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!