Wednesday, 2017-06-28

*** k0da has quit IRC00:26
*** esberglu has joined #openstack-powervm01:36
*** esberglu has quit IRC01:36
*** esberglu has joined #openstack-powervm01:36
*** esberglu has quit IRC01:41
*** thorst has joined #openstack-powervm01:53
*** thorst has quit IRC01:57
*** tonyb_ is now known as tonyb02:40
*** jay1_ has joined #openstack-powervm03:21
*** jay1_ has quit IRC03:51
*** thorst has joined #openstack-powervm03:54
*** thorst has quit IRC04:00
*** k0da has joined #openstack-powervm04:50
*** k0da has quit IRC05:19
*** jay1_ has joined #openstack-powervm05:50
*** thorst has joined #openstack-powervm05:57
*** thorst has quit IRC06:02
*** k0da has joined #openstack-powervm07:00
*** thorst has joined #openstack-powervm07:58
*** thorst has quit IRC08:03
*** thorst has joined #openstack-powervm09:24
*** thorst has quit IRC09:29
*** k0da has quit IRC09:48
*** k0da has joined #openstack-powervm09:49
*** smatzek has joined #openstack-powervm10:59
*** jay1_ has quit IRC11:21
*** thorst has joined #openstack-powervm11:25
*** thorst has quit IRC11:29
efriededmondsw The CI setup scripts run pvmctl; and pvmctl/pypowervm need to be version synced.11:40
openstackgerritMerged openstack/nova-powervm master: Allow dynamic enable/disable of SRR capability
edmondswefried ah, right. Ok, so I think we at least need a comment that explains that12:03
efriededmondsw Well, I thought it through some more, commented.12:03
edmondswefried yes..12:05
edmondswefried I see you merged ... I thought we were thinking this didn't actually require any nova-powervm changes, in which case no need for a spec12:08
edmondswdid you think of something that needs to change?12:09
efriededmondsw I dig having a spec that says this didn't work before, and it's gonna work now.12:09
*** jpasqualetto has joined #openstack-powervm12:09
efriedThere's going to be a change required in nova-powervm - the requirements bump to pull in the new pypowervm - it's just that we're going to get it for free-ish from the bot.12:10
*** thorst has joined #openstack-powervm12:10
edmondswI don't think I agree with that, but I didn't -1 it yesterday, so my fault... it's done now12:11
*** esberglu has joined #openstack-powervm12:28
*** dwayne has quit IRC12:43
*** esberglu has quit IRC12:49
*** kylek3h has joined #openstack-powervm12:50
*** smatzek has quit IRC12:58
*** esberglu has joined #openstack-powervm13:05
*** mdrabe has joined #openstack-powervm13:15
*** smatzek has joined #openstack-powervm13:17
*** jay1_ has joined #openstack-powervm13:45
efriedesberglu I think the cherry-pick-rather-than-checkout change is busted.14:00
esbergluefried: Yeah I know why14:00
esbergluefried: Well at least I think I do14:02
esbergluzuul only reports a merge failure when it can't automatically merge (i.e. you can't just click the rebase button)14:02
esbergluBut I think the merge for cherry-pick may be different than that14:02
efriedesberglu I *just* rebased that one.14:02
efriedI think I may know why.14:03
efriedlet me look at that output again...14:03
efriedesberglu We're trying to cherry-pick the last commit in a series.14:04
efriedWithout applying the ones leading up to it.14:04
esbergluefried: Oh duh. We even brought that up in IRC before14:04
efriedIf I were doing that locally, I would do it as:14:04
efriedgit review -d <the_change>; git rebase <main_branch>14:04
esbergluSo without git review I think that would just be14:05
efried'cept would probably have to add logic to clean out commits in the series that have merged (possibly at a different patch set)14:05
esberglugit checkout <change>, git rebase14:05
efriedyeah, but to account for ^^, you may need some extra flaggage.14:06
efriedI actually don't know how to solve the thing with the already-merged commits earlier in the chain.14:07
esbergluefried: I think we can just use that --keep-redundant flag14:07
esbergluLet me look at the rebase doc14:08
efriedThat'll only work if the commit in the chain is up to date with what merged.14:08
esbergluSurely there is a way to accomplish that, I'll check it out after the kickoff meeting is over14:11
efriedesberglu Offhand, I think it will involve walking the commits backward until we get to the trunk, and somehow matching them up with the tip of the main branch.  By Change-Id?14:12
efriedesberglu In the meantime, prolly a good idea to revert that change.14:14
esbergluefried: 551614:15
efriedesberglu Updated commit message, and +214:17
esbergluefried: thx14:18
esbergluAll changes to the powervm-ci repo are instant14:19
efriedLoving that14:19
esbergluefried: Same14:19
efriedI'm sure I'll get over it eventually, but it gives me a thrill every time.14:20
*** dwayne has joined #openstack-powervm14:29
esbergluefried: It actually will hit runs that have started before the change merges. Because powervm-ci doesn't get pulled until after the stack14:33
esbergluWait no that isn't correct14:33
esbergluIt gets pulled right before the stack14:34
efried"This change will take effect half an hour ago."14:35
efriedThat's amazing.14:35
efriedBetter than quantum.14:35
esbergluefried: thorst: Where's the source for this
esbergluAh typo in my find command. Thanks15:14
*** k0da has quit IRC15:21
*** jay1_ has quit IRC15:24
*** mdrabe has quit IRC15:53
*** mdrabe has joined #openstack-powervm15:57
edmondswefried I just proposed 5518 for what I was saying yesterday in 551116:53
*** efried is now known as efried_afk16:56
edmondswesberglu efried what is sonar and why is it complaining about things that tox -epep8 allows?17:08
esbergluedmondsw: I believe its a set of rules like complexity checks, lines per file, etc. Not sure though17:12
edmondswesberglu yeah, that's exactly what it is... I guess my question would be better worded, why do we do this if it doesn't match tox -epep8?17:12
esbergluedmondsw: No idea17:13
edmondswthey should be consistent, so you don't have to wait for jenkins to fail to go address something.17:14
esbergluedmondsw: Yeah that makes sense to me. But perhaps there is a reason for it. I'd ask in the novalink channel someone there probably knows17:18
edmondswesberglu I hit this with pypowervm, so it's something we setup ourselves, not nova17:19
esbergluYeah novalink slack channel17:20
edmondswesberglu oh, sorry, I misread novalink as nova ;)17:20
*** k0da has joined #openstack-powervm17:21
*** efried_afk is now known as efried17:42
esbergluedmondsw: So what we really need to do regarding the convo in qa is use the devstack generated tempest.conf instead of our own.17:57
esbergluThere's already a TODO for it17:57
esbergluBut messing around with the local.conf files never goes smoothly17:58
edmondswesberglu ah, that explains it17:58
esbergluSo I've been putting it off since what we have is working17:58
esbergluIIRC the reason we had our own is that devstack didn't have enough conf options to do what we wanted with tempest17:58
esbergluIts getting closer, but there are still things missing17:58
esbergluSo not only will we have to update our local.conf, we will likely have to create devstack patches/bugs as necessary17:59
esbergluEventually I think the idea is to have a local.conf option for every tempest config setting17:59
edmondswesberglu couldn't we let devstack create tempest's conf, and then add/update it as well?18:02
edmondswopening devstack patches/bugs as well, sure, but in the meantime...18:02
esbergluedmondsw: We can just explicitly set tempest.conf values using the local.conf post config18:03
esbergluedmondsw: Here's what I will do. I will set up a stack and see what differs between their tempest.conf and ours18:04
esbergluThen set as much as possible using local.conf options18:04
edmondswesberglu +118:04
esbergluAnything that doesn't have one we stuff in the post config for now, open a devstack bug, and either fix ourselves or wait for one18:04
esbergluLike they said in qa, they are REALLY easy patches18:05
efriedthorst esberglu mdrabe Any y'all interested in reviewing ?18:20
efriedI won't blame you if you say no.18:20
esbergluefried: I've been slowly working through it. About halfway down18:20
efriedI'll hold off +W then.18:21
efriedDammit, how did we end up with a net LOC gain on this?18:21
esbergluefried: Really? Hasn't seemed that way so far18:22
efriedi know18:22
esbergluSure enough18:23
efriedI mean, I've found a few18:23
efriedBut way more removals18:23
thorstefried: you added 301, removed 34718:23
efriedoh, I was reading it backwards?18:23
thorstI think so18:23
esbergluSo did I lol. That makes more sense18:24
efriedYuh.  Derp.18:24
esbergluefried: Seeing some "except Exception:"18:34
esbergluThought we were trying to avoid that18:34
esbergluBut perhaps that isn't within the scope of this change18:34
efriedesberglu Yes, separate change TBD18:34
*** jay1_ has joined #openstack-powervm18:35
esbergluefried: Okay. 2 files left, but that's the only thing I have so far18:35
esbergluefried: LGTM18:36
efriedesberglu Thanks!18:36
esbergluefried: Np. It appeared to be much more daunting than it really was18:36
efriedYeah, lotta files but relatively low LOC count.18:36
edmondswefried good work there, tx18:38
efriedThanks for the thorough review18:38
edmondswesberglu I'm having trouble stacking, looks like a keystone version discovery issue... have you seen that before? I'm basically using the OOT local.conf18:43
mdrabe"except Exception:" is bad? I just wrote that line18:44
edmondswmdrabe not necessarily, but usually18:44
edmondswyou should catch specific exceptions that you expect, not everything, which could include both things you expect and things you didn't expect and didn't properly code to handle18:45
edmondswor things you shouldn't handle, like an interrupt18:45
mdrabeOh ok, thought there was some implicit behavior I wasn't aware of18:46
edmondswone example of a case where you *should* do except Exception is if you need to make sure something you didn't expect doesn't kill your process18:46
edmondswwith suitable logging, of course, so you can figure out what it was and start expecting it in future18:47
esbergluedmondsw: Looking now18:48
esbergluedmondsw: I've never ran devstack from a different directory than the other openstack projects18:53
esbergluNot sure if that matters at all18:53
edmondswwhere are the keystone logs?18:53
esbergluThe service isn't up yet so they don't exist18:54
edmondswesberglu well it's trying to talk to the service, so it better be up...18:54
esberglusudo journalctl -xea --unit devstack@18:54
esbergluThen tab will show you what's been started18:54
esbergluOnly dstat and etcd are up18:54
esbergluLooks like to me18:54
edmondswps -ef | grep keystone shows a bunch of wsgi processes for keystone18:55
*** k0da has quit IRC18:55
esbergluedmondsw:  Maybe you have to add18:57
esbergluenable_plugin keystone git://
edmondswwhy wouldn't we have had to add that in our CI then?18:58
esbergluI was just thinking the same18:58
edmondswI thought some of the base stuff like keystone you didn't need that for18:58
esbergluedmondsw: Yeah just spitballing here18:58
edmondswesberglu appreciated :)18:58
esbergluLet me think for a while. I'm sure I've seen that error before. Can't remember why (and this could very well be something different)18:59
esbergluBut it might come to me18:59
esbergluedmondsw: Perhaps something to do with WSGI_MODE=mod_wsgi?19:02
esbergluCould try commenting that out and restacking19:03
edmondswesberglu worth a shot...19:06
esbergluedmondsw: That would be nice if it is the case. Might shed some insight as to what we need to do to remove that from CI19:06
edmondswesberglu so commenting that line didn't fix the problem, but it did get keystone into journalctl so I can see the logs19:13
edmondswand it looks like "from paste import deploy" is getting an ImportError because it can't find deploy19:14
*** k0da has joined #openstack-powervm19:15
edmondswpip list shows that PasteDeploy 1.5.2 is installed...19:16
efriedThat thing again?19:17
efriedAlways pip & apt uninstall all paste crap before devstack.19:17
edmondswefried yay19:19
esbergluedmondsw: efried: Ahh that's where I've seen. I knew it was familiar. We have a step for that in the undercloud deploy19:20
esberglu"pip install pastedeploy --upgrade --force-reinstall"19:20
efriedUsually only see it first thing after fresh system install.19:20
efriedIf the CI node is installed from an image made after that's done, swhy you wouldn't have been seeing it lately.19:21
edmondswyeah, this is a fresh system... stacking again now, but I expect that will fix it19:23
openstackgerritMatt Rabe proposed openstack/nova-powervm master: WIP: Change NVRAM manager store to use uuid instead of instance object
efriededmondsw Until you hit the next thing.19:24
edmondswright... I expect that will fix it, not devstack :)19:24
esbergluYeah don't get your hopes too high when dealing with devstack, you'll very often be disappointed19:25
edmondswesberglu I forgot to put the WSGI_MODE line back, so we'll see how this works with that commented...19:25
edmondswI'm guessing it won't...19:25
esbergluedmondsw: In CI land you can still get it to stack. But it doesn't actually work19:26
esbergluSo you may get though and end up unstacking anyways19:26
edmondswesberglu do you remember what doesn't work, exactly?19:26
esbergluedmondsw: I just finished a stack and tempest run with that to see19:28
edmondswefried if this paste thing is a known issue, has anyone tried to get it fixed in the NovaLink setup?19:28
esbergluedmondsw: I can upload the logs to the logserver from that19:28
efriededmondsw I don't think so.  Not sure how simple it would be, though.  I suspect it gets pulled in when you install some other dependency, like python-dev or whatever.19:29
efriedAnd devstack seems to be the only thing that breaks on it - do you have any knowledge of pvc having trouble with it?19:30
efriedNothing else (like pvmctl) breaks)19:30
openstackgerritMatt Rabe proposed openstack/nova-powervm master: WIP: Change NVRAM manager store to use uuid instead of instance object
edmondswefried PowerVC will install everything it needs from debs, so it's not going to care if pip is messed up19:33
edmondswesberglu wow, that's a lot of red19:34
edmondswmore of a burnt orange, I guess19:34
esbergluedmondsw: Yep19:34
edmondswso... I don't want to bite that off today ;)19:34
esbergluIf you scroll fast enough it makes you dizzy19:34
esbergluedmondsw: You know anything about mod_wsgi/uwsgi19:35
edmondswa little... not too much19:36
esbergluedmondsw: The good thing is all of the errors are the same so once we figure it out we fix all of the orage19:37
*** jay1_ has quit IRC19:53
edmondswesberglu <sigh> just one devstack problem after another... now hitting VolumeTypeCreateFailed: Cannot create volume_type with name lvmdriver-1 and specs {}20:12
esbergluedmondsw: I once spent an pretty much entire internship fighting devstack. So I know the struggle. I'll hop on quick20:13
edmondswesberglu tx... this is with the WSGI_MODE uncommented again20:14
esbergluI miss screen...20:14
esbergluI find it hard to navigate the journalctl logs20:14
edmondswI tried running the command that failed, and it worked for me...20:14
edmondswtiming issue, maybe20:14
esbergluNot seeing anything in the cinder-volume logs20:16
esbergluedmondsw: Have  you been changing the local.conf before unstacking20:16
edmondswyeah, I think I did that with the WSGI_MODE change20:16
esbergluedmondsw: I don't think this issue is related to that. But I unstack also uses the local.conf20:17
edmondswmakes sense20:17
esbergluSo if you change it before unstacking it might not clean up properly20:17
esbergluedmondsw: Honestly I would just try again. That works sometimes. I've tried to find better ways to debug devstack, but often it just turns into20:18
esberglujust trying stuff until it works20:18
edmondswesberglu, this seems like a timing thing20:19
edmondswI'll just try again20:19
esbergluObviously still taking educated guesses, but trying to go through devstack to see whats going on is such a PITA its faster half the time20:20
edmondswdoesn't give me a warm fuzzy feeling about ever getting our CI stable...20:20
esbergluEh. Usually once you get something that works it works for  a while20:20
esbergluAnd if it breaks you at least have a timeframe for when the change was introduced20:20
esbergluBut since you are independent I wouldn't even know where to start looking20:20
edmondswbut if we're hitting things like this, where there isn't any change, it was just some intermittent problem...20:21
esbergluedmondsw: I mean we won't ever get to the point where CI is passing everything. I think that this would fall into the acceptable failure category20:21
esbergluas long as it isn't common20:21
esbergluDevstack is also getting much better. Used to be that devstack would break us about once a week20:25
efriedthorst Last chance at string cleanup patch20:41
thorstefried: I won't get to it today....  :-(20:42
thorstbut I trust you20:42
efriedOkay, merging.20:42
*** smatzek has quit IRC20:47
openstackgerritMerged openstack/nova-powervm master: Clean up log messages
esbergluOnce we get the generated tempest.conf going we should run against tempest as well20:54
esbergluNeed to check what kind of capacity that will take but I think we will be fine20:54
*** thorst has quit IRC21:12
*** kylek3h has quit IRC21:17
*** jpasqualetto has quit IRC21:27
*** esberglu has quit IRC21:27
*** thorst has joined #openstack-powervm21:46
*** thorst has quit IRC21:50
*** esberglu has joined #openstack-powervm21:52
*** esberglu has quit IRC21:57
*** k0da has quit IRC21:57
*** dwayne has quit IRC22:00
*** thorst has joined #openstack-powervm22:10
openstackgerritMatt Rabe proposed openstack/nova-powervm master: WIP: Change NVRAM manager store to use uuid instead of instance object
*** thorst has quit IRC22:17
*** thorst has joined #openstack-powervm22:17
*** thorst has quit IRC22:22
*** mdrabe has quit IRC22:23
*** dwayne has joined #openstack-powervm23:06
*** edmondsw has quit IRC23:19
*** smatzek has joined #openstack-powervm23:30
*** smatzek has quit IRC23:54

Generated by 2.15.3 by Marius Gedminas - find it at!