Thursday, 2017-06-15

*** edmondsw has joined #openstack-powervm00:08
*** edmondsw has quit IRC00:12
*** mdrabe has quit IRC04:30
*** k0da has joined #openstack-powervm05:00
*** tjakobs has joined #openstack-powervm05:03
*** tjakobs has quit IRC05:10
*** edmondsw has joined #openstack-powervm05:32
*** edmondsw has quit IRC05:37
*** edmondsw has joined #openstack-powervm09:08
*** edmondsw has quit IRC09:18
*** smatzek has joined #openstack-powervm11:09
*** edmondsw has joined #openstack-powervm12:22
*** edmondsw has quit IRC12:24
*** jpasqualetto has joined #openstack-powervm12:24
*** edmondsw has joined #openstack-powervm12:25
*** jwcroppe has quit IRC12:42
*** esberglu has joined #openstack-powervm12:50
*** kylek3h has joined #openstack-powervm12:53
*** jwcroppe has joined #openstack-powervm12:54
*** mdrabe has joined #openstack-powervm13:20
*** mdrabe has quit IRC13:21
*** mdrabe has joined #openstack-powervm13:22
*** thorst has joined #openstack-powervm13:36
*** tjakobs has joined #openstack-powervm13:50
efriedesberglu You planning to add those to-dos from to our etherpad?13:51
esbergluefried: Yep13:52
esbergluefried: You have time to help me with devstack after the nova meeting?13:52
efriedesberglu Can try ;-)13:52
efriedesberglu Oh, after.  Yeah, for a bit; got a doc appt at 11 CT13:53
BorD__anyone familiar with loopmount on aix?  any way to know which .iso has been used for mounting?14:18
*** mdrabe has quit IRC14:43
*** mdrabe has joined #openstack-powervm14:43
esbergluefried: Let me know when you're ready to get filled in14:44
*** burgerk has joined #openstack-powervm15:04
*** mdrabe has quit IRC15:04
efriedesberglu Ight, let's do this.15:05
esbergluOkay so there are 2 different solutions that I have considered15:06
thorstesberglu: how bad would it be if I needed to reboot the host running nodepool, openstack underlay, etc...15:06
thorst(planning to update the underlying host...)15:07
esbergluthorst: I would just have to redeploy everything. Not a huge deal seeing as everything master is failing anyways15:07
thorsthow close are you to solving?15:07
thorstI obviously haven't been paying attention lately...  :-(15:07
esbergluthorst: But it would take away the vms I'm using to work through the devstack stuff.15:08
thorstok...I'll try to hold off a bit15:08
thorstmaybe next week we do it15:08
esbergluWe have it stacking but the networking isn't getting set up right. Was just about to dive in with efried15:08
efriedthorst If you're available, we could almost certainly use your help there.15:08
thorstwhat's the prob15:08
esbergluAnyway, 2 paths we could potentially take15:09
thorstand are we talking underlay or real VMs15:09
esbergluthorst: The AIO tempest instances15:10
*** tjakobs has quit IRC15:10
esberglu1) Right now we don't have devstack create networks. However with the current local.confs in review 545615:11
esbergluA private network is being created anyways.15:11
*** mdrabe has joined #openstack-powervm15:11
esbergluThen when we get to the post stack network creation, we can't create vlan networks only vxlan15:12
esbergluThe first path is to get prevent the initial private net from being created and get post stack network creation working15:12
esberglu2) The other option is to move back to having devstack create the networks for us15:13
esbergluThis will be more compatible with the devstack generated tempest.conf as it will set all the appropriate networking options for tempest15:13
esbergluBut we have had difficulties in the past using the devstack created networks15:13
thorstesberglu: hold up a sec...15:13
thorstI want to go back to 115:14
thorstwhy can't you create a VLAN network15:14
esbergluthorst: That's what I need help with15:14
thorstso I suspect the reason you can't create a VLAN network isn't because it pre-creates networks for you15:14
thorstI suspect they turned off VLAN15:14
thorstdo you have a node that is stacked that I can peak at?15:14
esbergluthorst: messaged you15:16
esbergluAnyway, I'll just finish explaining where #2 is at quick. If we have devstack create the networks, it creates 2 vxlan networks, both named private15:17
esbergluI haven't gone any further down that path15:17
esbergluthorst: There is this setting in /etc/neutron/plugins/ml2/ml2_conf.ini15:18
thorsteven a vxlan network needs a public (either VLAN or Flat) to route through...15:18
esberglucalled tenant_network_types15:19
esbergluThat was only getting set to vxlan15:19
thorsttenant_network_types...yeah.  But even then.15:19
thorstyour public networks aren't necessarily your tenant_networks15:19
thorstpulling down latest15:20
esbergluLatest what?15:20
thorstso first..we look for the type_drivers list15:22
thorstdoesn't overload we're fine15:22
*** efried is now known as help15:22
*** help is now known as efried-doc15:23
thorstesberglu: where do logs go now?15:28
esbergluthorst: You have to use journalctl15:29
esbergluthorst: journalctl -xea --unit devstack@n-cpu15:29
esbergluthorst: journalctl -xea --unit devstack@*15:29
esbergluFirst will give you that log15:29
esberglu2nd will give you all of the logs15:29
thorstwhat's the neutron server called?15:30
thorst(cause that doesn't show up)15:31
esbergluthorst: If you type sudo systemctl status devstack@15:37
*** tjakobs has joined #openstack-powervm15:38
esbergluAnd hit tab a few times you can see all the services15:38
thorst"InvalidInput: Invalid input for operation: physical_network 'default' unknown for VLAN provider network"15:40
thorstdo you have the physical networks set up properly?15:41
esbergluthorst: Not sure? How do I determine that?15:47
thorstI think you need in the neutron.conf a 'network_vlan_ranges=default:1:4095'15:47
thorsttry that, restart neutron-api and then we'll see if you can create a network then15:47
esbergluthorst: You mean the ml2 conf?15:49
esbergluYeah I see it there15:49
thorstI meant more neutron.conf15:50
thorstbut I guess try ml2.conf first15:50
esbergluthorst: Success!16:18
esbergluAnd I found the local.conf setting that will do that for us16:18
*** efried-doc is now known as efried17:21
edmondswesberglu so how are things looking now that you're past the network issue (I hope)?17:26
esbergluedmondsw: I ran tempest and only failed 2 tests master OOT. I'm trying a run on a fresh node now17:26
edmondswif you need more help, let me know17:27
edmondswefried you've asked me to rebase onto 2 separate and unrelated patches17:51
edmondswthat's not possible, is it?17:51
edmondswone of them would have to be based on the other, and since they're unrelated, that's not the case17:51
esbergluedmondsw: efried: thorst: 5456 is updated. Just waiting on tempest results from the fresh instance then we can get that in17:51
edmondswesberglu great, will look17:52
esbergluedmondsw: There's a couple TODOs in it, but at this point working is working17:52
efriededmondsw I'm sure it is possible, but I think you would have to have an intervening commit that merges the two of them together.  Usually when I do it, I base the whole chain linearly, even if it isn't strictly necessary at the bottom.17:52
efriedBut let me remind myself which ones I was talking about.17:52
edmondswefried one is so small I don't want to base the other on it... I just want to get it merged17:53
efriedthe conf.hosts one?17:54
edmondswefried the other is totally unrelated, and will probably take a while to get merged because I've not even gotten any feedback from my earlier irc question, so I don't really want that to be the root either17:54
edmondswwhich is my other question... should I try again on irc, or do you want to since that's your idea, or should one of us put something on the ML... ?17:55
efriedDah, you know, we should really get that CONF.hosts change passing our CI before we ask to get it merged.17:55
efried(Too late, though; I asked sdague and he already +2ed it.17:55
edmondswefried of course, but our CI isn't passing anything yet :)17:56
efriedYeah, but I definitely want it passing on that one before we merge it.17:56
edmondswhopefully soon17:56
edmondswthat's why I haven't bugged folks about it17:56
edmondswesberglu, why the \ in [[post-config|/$Q_PLUGIN_CONF_FILE]] ?17:56
edmondswsorry... /17:56
esbergluedmondsw: Uhh let me see if I can find the doc for it. There's a reason though17:59
edmondswinteresting... we didn't do that in other places, so it jumped out17:59
edmondswahh... k18:00
edmondswsounds like a devstack bug, actually ;)18:00
edmondswesberglu it occurs to me that we probably didn't need to get neutron working with the intree CI, since we don't have networking there yet :)18:01
edmondswbut if it's working now, no sense pulling it18:01
esbergluedmondsw: I guess that the local.conf should have Q_PLUGIN_CONF_FILE=etc/neutron/plugins/ml2/ml2_conf.ini18:01
esbergluBased on that doc18:01
esbergluedmondsw: Yeah I considered that as well. Was gonna see if this resolved IT too first18:02
edmondswso fix Q_PLUGIN_CONF_FILE and let's merge this!18:03
edmondswefried aren't there negative implications to recreating the Session if we don't really have to do that?18:05
efriededmondsw Like what?18:06
edmondswefried hmm... actually, I think all the ones I can think of have since been resolved18:07
esbergluedmondsw: efried: The qos tempest tests all failed. Other than that everything looks fine18:18
esbergluNot sure what changed in between runs, I used the same local.conf for the previous that only failed 2 tests18:19
esbergluThis run failed the same 2 tests18:19
efriedDo we do qos?18:19
esbergluThose additional 2 being identity tests18:20
esbergluefried: Yeah we've been running the qos tests18:20
edmondswesberglu link?18:20
esbergluedmondsw: dm'ed you. They don't get copied to the log server on the manual runs. I can collect them and add them to the log server manually if you want18:22
esbergluThe rest of the logs I mean18:22
efriederm, mriedem merged CONF.hosts.  So I realllly hope it passes the CI.18:43
efriededmondsw ^^18:43
edmondswefried hmm... yeah18:44
efriedMy fault for pushing for it before we had our CI going.18:44
edmondswwhy would they merge with our CI failing?18:44
edmondswa little surprised by that18:44
edmondswefried no worries... if you want to push something, push for comments on
efriedWhole lot easier to ask for attention on a one-liner18:51
edmondswno kidding :)18:59
edmondswesberglu have we done this?
esbergluedmondsw: Looks like the [agent] section was set correctly but not the [ml2] extension_drivers part19:19
esbergluPushing another run through, both IT and OOT19:19
* edmondsw crossing fingers...19:19
edmondswesberglu as for the identity tests... not sure what's going on there...19:21
edmondswbest guess I've been able to come up with is maybe tempest is multithreaded and some other test is trying to use the same user while it's locked, and thus extending the lockout period19:22
edmondswany idea if that's possible?19:22
edmondswwell, except that should show up as another test failing because of account lockout. Unless the test expected some kind of failure and ate it19:23
esbergluedmondsw: efried: Good news ml2 conf stuff also gets IT running the tempest tests19:33
edmondswesberglu woot!19:34
edmondswesberglu so are you updating 5456?19:34
esbergluI'm gonna hold on and see if adding qos to the extension_drivers worked first19:35
esbergluIn tree run just finished. Failed those same 2 identity tests, passed everything else20:04
*** thorst has quit IRC20:27
edmondswesberglu any thoughts on what I said about the identity tests above?20:36
esbergluedmondsw: I think it is more likely a configuration thing. That could be possible, tempest does run multiple tests at a time20:37
edmondswesberglu oh, I just realized that CONF.identity.user_lockout_duration is a tempest conf setting...20:41
edmondswyeah, that is set (defaults) to 5, which is wildly different from the default in keystone20:42
edmondswkeystone's default is 180020:42
edmondswin keystone/conf/security_compliance, called simply "lockout_duration"20:42
edmondswesberglu I wonder if we're supposed to be either a) disabling security_compliance testing (which probably isn't relevant to our CI) or b) changing one of those to match the other20:43
edmondswhmmm... looks like tempest is supposed to have security_compliance disabled by default (see tempest/ you didn't do something to enable that, did you?20:44
esbergluedmondsw: It is enabled in our tempest.conf20:45
edmondswboo... well, that's our problem20:45
esbergluI can add that in once this OOT run finishes20:45
*** smatzek has quit IRC20:46
edmondswesberglu looks like devstack enables that by default20:49
esbergluOnce this run finishes I can try running those few tests with (a) and (b)20:50
esbergluI can just reuse the environment if I'm only changing tempest.conf20:50
*** k0da has quit IRC20:51
*** thorst has joined #openstack-powervm20:59
*** jpasqualetto has quit IRC21:02
edmondswesberglu found the bug:
openstackLaunchpad bug 1693917 in tempest "test_user_account_lockout failed in gate because authN attempts took longer than usual" [Undecided,New]21:06
*** smatzek has joined #openstack-powervm21:06
edmondswesberglu a devstack change was merged yesterday, but the corresponding tempest change (which they don't seem to have realized is also necessary) hasn't merged yet21:06
edmondswI pinged lbragstad on the keystone channel, but I think he's probably afk21:07
esbergluAlright. I'll just temporarily disable until that bug is fixed. Probably something I can do21:07
edmondswI think the best solution for us is probably to disable security_compliance testing... KEYSTONE_SECURITY_COMPLIANCE_ENABLED=False in devstack local.conf21:08
esbergluWe could just set security_compliance = False in tempest.conf for now21:09
esbergluSetting all the tempest config options using devstack will be part of the auto generated tempest config thing21:09
esbergluIf we change that in our local.conf it won't actually do anything. Since we use our own tempest.conf not the one devstack makes21:10
edmondswoh, interesting21:10
esbergluI'll post a change for it21:11
*** smatzek has quit IRC21:22
esbergluefried: edmondsw: With the latest 5456 and the security_compliance change 5466 we should *fingers crossed* be passing21:22
esbergluthorst: ^21:22
esbergluI'm working on testing 5466 right now21:23
esbergluedmondsw: efried: 5466 needs a tempest change. One of the tests doesn't have a security_compliance check21:31
efriedesberglu You gonna propose it?21:32
efriedAnd disable it in the meantime?21:32
esbergluefried: Yeah21:32
esbergluI have to take off and take care of a few things. But if you guys are cool with the remove neutron legacy change we can merge. And then I can check back in tonight on the status21:32
esbergluAnd get that tempest fix up21:32
esbergluefried: edmondsw: ^21:33
efriedesberglu You've got +2s from me.21:35
*** burgerk_ has joined #openstack-powervm21:35
*** burgerk has quit IRC21:38
*** burgerk_ has quit IRC21:40
esbergluefried: One more change to 5466 updating the skip lists21:42
esbergluCan you merge once you +2 I have to head out now21:42
efriedesberglu Will do, but I don't see the patch up yet.21:43
esbergluShould be now. Thanks in advance21:43
*** esberglu has quit IRC21:43
*** esberglu has joined #openstack-powervm21:44
efriedesberglu Do we have a bug for the tempest change?21:45
efriededmondsw I made another delta on 5466, can you please approve?21:48
*** esberglu has quit IRC21:48
*** dwayne has quit IRC21:56
efriedeffit, I gotta run too; I'm going to merge that sucker.21:57
*** mdrabe has quit IRC22:00
edmondswefried sorry, had to step away. lgtm except I would have asked why we didn't use the uuid for the test in skip_tests22:05
efriedGood point.22:05
efriedHopefully that skippage will be short-lived.22:05
efriedI wouldn't mind putting up another rev with a bug number in it once that's opened against tempest; we can fix that then.22:05
efriedBut in the meantime, this should work, and I think esberglu wanted to get it in ASAP so we can start maybe passing.22:06
edmondswefried I'm looking at that test to see why security_compliance would even matter there...22:07
edmondswefried yeah, I don't think security_compliance should matter there... so I don't think we should have had to skip that test at all22:09
efriedoh, okay, well then we just need it in the skip section that says, "This is failing and we don't know why, so until we figure it out, we're skipping it so our CI can pass."22:09
edmondswefried I think it only failed because the other test ran first and locked the account. As long as the other test doesn't run, the account won't get locked, and this test should pass22:09
efriedor that.22:09
efriedLet's get some runs passing first; then we can nail that down.22:10
edmondswyep, agreed22:10
efriedI'm out.  Hasta maƱana.22:10
edmondswwe can clean this up tomorrow22:10
*** thorst has quit IRC22:14
*** kylek3h has quit IRC22:27
*** thorst has joined #openstack-powervm22:33
*** thorst has quit IRC22:34
*** esberglu has joined #openstack-powervm22:38
*** tjakobs has quit IRC22:39
openstackgerritMatthew Edmonds proposed openstack/nova-powervm master: Use for powervm nodename
*** esberglu has quit IRC22:43
*** esberglu has joined #openstack-powervm22:54
openstackgerritEric Berglund proposed openstack/nova-powervm master: DNM: ci check
openstackgerritEric Berglund proposed openstack/nova-powervm master: DNM: CI Check2
openstackgerritMatthew Edmonds proposed openstack/nova-powervm master: Remove volue_group_vios_name
*** edmondsw has quit IRC23:10
*** jwcroppe has quit IRC23:37
*** jwcroppe has joined #openstack-powervm23:38
*** jwcroppe has quit IRC23:42
*** thorst has joined #openstack-powervm23:53

Generated by 2.15.3 by Marius Gedminas - find it at!