Monday, 2017-04-17

*** thorst has joined #openstack-powervm01:59
*** thorst has quit IRC02:00
*** thorst has joined #openstack-powervm02:31
*** thorst has quit IRC02:48
*** thorst has joined #openstack-powervm03:45
*** thorst has quit IRC04:04
*** thorst has joined #openstack-powervm06:02
*** thorst has quit IRC06:21
*** thorst has joined #openstack-powervm07:18
*** thorst has quit IRC07:23
*** thorst has joined #openstack-powervm09:20
*** thorst has quit IRC09:24
*** thorst has joined #openstack-powervm10:53
*** thorst has quit IRC10:57
*** smatzek has joined #openstack-powervm11:09
*** thorst has joined #openstack-powervm11:55
*** kylek3h has quit IRC11:56
*** edmondsw has joined #openstack-powervm12:14
*** kylek3h has joined #openstack-powervm12:30
openstackgerritMerged openstack/nova-powervm master: Check for presence of volume data before doing cleanup
*** mdrabe has joined #openstack-powervm12:35
efriedthorst Was this really worth a bug and a backport?12:50
efriedDid I misunderstand the fix?  Just avoiding a log message if there's no volume data?12:51
openstackgerritEric Fried proposed openstack/nova-powervm master: Get flavor from instance
openstackgerritEric Fried proposed openstack/nova-powervm master: Get flavor from instance
*** jpasqualetto has joined #openstack-powervm12:55
thorstefried: I think it was something Manas got a defect on...I didn't really see a reason not to port back13:02
thorstI'd rather he not carry patches13:02
*** jpasqualetto has quit IRC13:25
*** jpasqualetto has joined #openstack-powervm13:33
*** smatzek has quit IRC13:36
*** smatzek has joined #openstack-powervm13:57
*** esberglu has joined #openstack-powervm14:02
*** apearson has joined #openstack-powervm14:17
*** tjakobs has joined #openstack-powervm14:24
*** burgerk has joined #openstack-powervm14:41
*** dwayne has quit IRC14:53
openstackgerritEric Larese proposed openstack/networking-powervm master: [WIP] Exclude non-SEA vifs from list_vifs
efriedesberglu Rebased up to console in-tree.15:08
efriededmondsw Added you to reviews of bottom half dozen or so in-tree change sets.15:09
efriedIt's about as gentle an introduction to the community driver as yer gonna get ;-)15:09
edmondswefried :) tx15:09
efriedEasiest to review 'em in order.  If you load up any one of 'em, look on the RHS of the gerrit window and go bottom up.15:10
esbergluefried: Okay. I'm gonna let that last one run silently just as a sanity check. As soon as it looks okay, prod will go live15:11
esbergluAnd then we can go through and recheck the in-tree patches15:12
efriedesberglu Yup, sounds like the right plan.15:12
*** apearson has quit IRC15:28
*** apearson has joined #openstack-powervm15:30
esbergluefried: thorst: adreznec: CI is publishing15:49
efriedesberglu Good deal.15:49
esbergluefried: Want me to kick off rechecks for in-tree patches?15:50
efriedesberglu Sure.15:50
*** dwayne has joined #openstack-powervm16:18
esbergluefried: Want to talk CI session config quick?16:19
*** apearson has quit IRC16:33
*** apearson has joined #openstack-powervm16:35
*** k0da has joined #openstack-powervm17:10
openstackgerritEric Larese proposed openstack/networking-powervm master: [WIP] Exclude non-SEA vifs from list_vifs
openstackgerritEric Larese proposed openstack/networking-powervm master: Exclude non-SEA vifs from list_vifs
*** kjw3 has joined #openstack-powervm18:18
*** kjw3 has quit IRC19:08
efriedesberglu Sorry, missed that alert.  CI session config.  Yeah.19:11
esbergluefried: I ended up just putting another patch up if you haven't seen that yet19:11
efriedstand by19:11
efriedYeah, my view doesn't go bold because I introduced the change set.  And I check my email like twice a day ;-)19:12
esbergluEssentially the same thing as before, just doing the version comparison within prep_devstack and getting rid of the python package comparison19:13
efriedesberglu Does dpkg version compare work if pip installed?19:13
efriedcool beans.19:14
esbergluI haven't tested any of this on CI yet. About to deploy staging with if you don't have further comments19:15
efriedesberglu Where was the $pypowervm_branch variable getting set before??19:16
efriedThat line - `git checkout $pypowervm_branch` - was a no-op before :)19:16
efriedoh, I see it now, dah.19:17
efriedesberglu You don't still want to use --keep-redundant-commits ?19:19
efriedIf you don't, then we may end up having to maintain separate patch lists for each release.19:19
esbergluefried: I meant to add that. Will update19:20
efriedesberglu The CI is looking really good.19:20
efriedIt seems to be completing really fast - 33 minutes or so.  Is that... normal?19:21
esbergluWe doubled the memory for the instances19:21
*** kjw3 has joined #openstack-powervm19:22
efriedWow, okay.  I'm a little surprised that'd do it.19:22
efriedBut cool.19:22
efriedfreakin xenserver is still awful.19:23
efriedBut at least it takes 2h to run.19:23
esbergluYeah I'm not sure how much of a performance increase we would expect from upping mem. But there's nothing else that has happened recently that should impact run time19:25
efriedI dig it.19:25
efriedGift horse19:25
efriedThat whole thing.19:25
efriedthorst adreznec esberglu On jay's system, seeing a lot of this:19:38
efried    @mock.patch('nova_powervm.virt.powervm.disk.driver.IterableToFileAdapter')19:38
efriedwhoops, not that.19:39
efried2017-04-17 15:38:40.539 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: timed out from (pid=126938) _raise_timeout /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/
efried2017-04-17 15:38:40.539 DEBUG oslo.messaging._drivers.impl_rabbit [-] Timed out waiting for RPC response: Timeout while waiting on RPC response - topic: "<unknown>", RPC method: "<unknown>" info: "<unknown>" from (pid=126938) _raise_timeout /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabb19:39
efriedEvery second or two.19:39
efriedOne assumes that'll cause all kinds of problems, including inability to communicate between n-cpu and our pvm-sea agent.19:39
esbergluIs rabbitmq running?19:39
efriedaccording to service status, yeah.19:40
esbergluHmm. I think the only time I have seen that before when rabbitmq was down19:40
efriedmebbe I try restarting it.19:41
esbergluWorth a try19:41
efriedIt's also possible I'm not used to amqp logging at debug level.19:42
efriedLike, perhaps that timeout is normal?19:42
efriedI wouldn't have thought so, but...19:42
esbergluIs jay having other issues related to this? Or you just happened to notice the logging19:44
efriedWell, he's having issues where vif plug is failing.19:44
efriedHe's got debug = True in his nova.conf, which is blowing up the logs.  I'm going to turn that off and try a deploy and see if I notice anything... unusual.19:45
efriedthorst adreznec I think jay's issue is that the pvm-sea agent plug isn't getting the message back to nova anymore.19:56
efriedn-cpu just sits there waiting for it.  And eventually times out.19:56
efriedI'm seeing the provision happen on the agent.19:57
thorstefried: so I didn't see the provision actually go through on Jay's system.19:58
thorstI saw the request come into networking-powervm19:58
thorstbut not an actual up being sent back19:58
thorst(if I remember right)19:58
thorstefried: I stand corrected19:59
thorstso if you look in the neutron server...19:59
efriedWell, it's doing it in this case.  It's a new spawn.19:59
thorstthat is the thing that is supposed to take that active message20:00
thorstand then determine whether or not to send to nova20:00
thorstthe logging is done in neutron server though20:00
efriedokay, looking...20:00
efriedThe timestamp would be that last one, where it says sending to neutron, right?20:01
thorstaround then - yeah20:02
efriedThat looks nice to me.20:02
efriedBut what do I know?20:02
efriedThere is an error in the q-svc log about 13s earlier20:02
efriedq-svc is also claiming to have three agents running.  Is that right?20:03
thorstthat seems right.20:03
efriedour SEA & SR-IOV agents, and... what else?20:03
thorstmaybe L3?20:03
thorstoo....maybe the third agent is what is blocking it20:03
efriedWould I see that in screen?20:04
thorstthe message you'd want to look for is.... "Sending events:"20:04
thorstprobably, what are all the agents running there?20:04
efried 0$ shell  1$(L) dstat  2$(L) key  3$(L) key-access  4$(L) g-reg  5$(L) g-api  6$(L) n-api   7$(L) q-svc*  8$(L) q-dhcp  9$(L) n-cond  10$(L) n-crt  11$(L) n-sch  12$(L) n-novnc  13$(L) n-cauth  14-$(L) n-cpu  15$(L) placement-api  16$(L) horizon  17$(L) pvm-q-sea-agt  18$(L) pvm-q-sriov-agt20:04
thorstq-dhcp is the third20:05
thorstI don't think that one cares...20:05
thorstbecause that gets used when the VM boots20:05
thorst(and isn't using config drive)20:05
efriedIn q-svc init, I only see two mech drivers listed.20:05
efriedand the 'qos' extension driver20:05
thorstall seems correct20:05
efriedand 'port_security'20:05
thorstwelcome to neutron20:05
efriedthorst Okay, so next debug step?20:06
thorstIn that second paste you sent me, I don't see a "Sending events" debug message20:06
thorstso we probably need to pdb around the neutron.notifiers.nova code20:07
thorstmaybe pdb in there a bit...20:07
efriedI don't see "Sending events" anywhere in q-svc log.20:07
thorstyep...probably time to root around with pdb then20:12
thorstthat's the file I'd start with20:12
efriedthorst Remind me - import pdb; pdb.set_trace() ?20:23
thorstyes sir20:23
*** smatzek has quit IRC20:28
*** thorst has quit IRC21:02
*** smatzek has joined #openstack-powervm21:06
*** edmondsw has quit IRC21:10
*** dwayne has quit IRC21:19
*** burgerk has quit IRC21:30
*** smatzek has quit IRC21:31
*** esberglu has quit IRC21:34
*** esberglu has joined #openstack-powervm21:35
*** esberglu has quit IRC21:39
*** kylek3h has quit IRC21:45
*** esberglu has joined #openstack-powervm21:48
*** esberglu has quit IRC21:52
*** mdrabe has quit IRC21:56
*** apearson has quit IRC22:18
*** kjw3 has quit IRC22:30
*** tjakobs has quit IRC22:41
*** jpasqualetto has quit IRC22:42
*** kjw3 has joined #openstack-powervm22:46
*** edmondsw has joined #openstack-powervm22:49
*** edmondsw has quit IRC22:53
*** kjw3 has quit IRC22:55
*** kjw3 has joined #openstack-powervm23:03
*** k0da has quit IRC23:10
*** kjw3 has quit IRC23:17
*** thorst has joined #openstack-powervm23:37

Generated by 2.14.0 by Marius Gedminas - find it at!