| *** edmondsw has joined #openstack-powervm | 00:08 | |
| *** edmondsw has quit IRC | 00:12 | |
| *** mdrabe has quit IRC | 04:30 | |
| *** k0da has joined #openstack-powervm | 05:00 | |
| *** tjakobs has joined #openstack-powervm | 05:03 | |
| *** tjakobs has quit IRC | 05:10 | |
| *** edmondsw has joined #openstack-powervm | 05:32 | |
| *** edmondsw has quit IRC | 05:37 | |
| *** edmondsw has joined #openstack-powervm | 09:08 | |
| *** edmondsw has quit IRC | 09:18 | |
| *** smatzek has joined #openstack-powervm | 11:09 | |
| *** edmondsw has joined #openstack-powervm | 12:22 | |
| *** edmondsw has quit IRC | 12:24 | |
| *** jpasqualetto has joined #openstack-powervm | 12:24 | |
| *** edmondsw has joined #openstack-powervm | 12:25 | |
| *** jwcroppe has quit IRC | 12:42 | |
| *** esberglu has joined #openstack-powervm | 12:50 | |
| *** kylek3h has joined #openstack-powervm | 12:53 | |
| *** jwcroppe has joined #openstack-powervm | 12:54 | |
| *** mdrabe has joined #openstack-powervm | 13:20 | |
| *** mdrabe has quit IRC | 13:21 | |
| *** mdrabe has joined #openstack-powervm | 13:22 | |
| *** thorst has joined #openstack-powervm | 13:36 | |
| *** tjakobs has joined #openstack-powervm | 13:50 | |
| efried | esberglu You planning to add those to-dos from https://review.openstack.org/#/c/470999/ to our etherpad? | 13:51 |
|---|---|---|
| esberglu | efried: Yep | 13:52 |
| esberglu | efried: You have time to help me with devstack after the nova meeting? | 13:52 |
| efried | esberglu Can try ;-) | 13:52 |
| efried | esberglu Oh, after. Yeah, for a bit; got a doc appt at 11 CT | 13:53 |
| BorD__ | anyone familiar with loopmount on aix? any way to know which .iso has been used for mounting? | 14:18 |
| *** mdrabe has quit IRC | 14:43 | |
| *** mdrabe has joined #openstack-powervm | 14:43 | |
| esberglu | efried: Let me know when you're ready to get filled in | 14:44 |
| *** burgerk has joined #openstack-powervm | 15:04 | |
| *** mdrabe has quit IRC | 15:04 | |
| efried | esberglu Ight, let's do this. | 15:05 |
| esberglu | Okay so there are 2 different solutions that I have considered | 15:06 |
| thorst | esberglu: how bad would it be if I needed to reboot the host running nodepool, openstack underlay, etc... | 15:06 |
| thorst | (planning to update the underlying host...) | 15:07 |
| esberglu | thorst: I would just have to redeploy everything. Not a huge deal seeing as everything master is failing anyways | 15:07 |
| thorst | :-) | 15:07 |
| thorst | how close are you to solving? | 15:07 |
| thorst | I obviously haven't been paying attention lately... :-( | 15:07 |
| esberglu | thorst: But it would take away the vms I'm using to work through the devstack stuff. | 15:08 |
| thorst | ok...I'll try to hold off a bit | 15:08 |
| thorst | maybe next week we do it | 15:08 |
| esberglu | We have it stacking but the networking isn't getting set up right. Was just about to dive in with efried | 15:08 |
| efried | thorst If you're available, we could almost certainly use your help there. | 15:08 |
| thorst | what's the prob | 15:08 |
| esberglu | Anyway, 2 paths we could potentially take | 15:09 |
| thorst | and are we talking underlay or real VMs | 15:09 |
| esberglu | thorst: The AIO tempest instances | 15:10 |
| *** tjakobs has quit IRC | 15:10 | |
| thorst | k | 15:10 |
| esberglu | 1) Right now we don't have devstack create networks. However with the current local.confs in review 5456 | 15:11 |
| esberglu | A private network is being created anyways. | 15:11 |
| *** mdrabe has joined #openstack-powervm | 15:11 | |
| esberglu | Then when we get to the post stack network creation, we can't create vlan networks only vxlan | 15:12 |
| esberglu | The first path is to get prevent the initial private net from being created and get post stack network creation working | 15:12 |
| esberglu | 2) The other option is to move back to having devstack create the networks for us | 15:13 |
| esberglu | This will be more compatible with the devstack generated tempest.conf as it will set all the appropriate networking options for tempest | 15:13 |
| esberglu | But we have had difficulties in the past using the devstack created networks | 15:13 |
| thorst | esberglu: hold up a sec... | 15:13 |
| thorst | I want to go back to 1 | 15:14 |
| thorst | why can't you create a VLAN network | 15:14 |
| esberglu | thorst: That's what I need help with | 15:14 |
| thorst | so I suspect the reason you can't create a VLAN network isn't because it pre-creates networks for you | 15:14 |
| thorst | I suspect they turned off VLAN | 15:14 |
| thorst | do you have a node that is stacked that I can peak at? | 15:14 |
| esberglu | thorst: messaged you | 15:16 |
| esberglu | Anyway, I'll just finish explaining where #2 is at quick. If we have devstack create the networks, it creates 2 vxlan networks, both named private | 15:17 |
| esberglu | I haven't gone any further down that path | 15:17 |
| esberglu | thorst: There is this setting in /etc/neutron/plugins/ml2/ml2_conf.ini | 15:18 |
| thorst | even a vxlan network needs a public (either VLAN or Flat) to route through... | 15:18 |
| esberglu | called tenant_network_types | 15:19 |
| esberglu | That was only getting set to vxlan | 15:19 |
| thorst | tenant_network_types...yeah. But even then. | 15:19 |
| thorst | your public networks aren't necessarily your tenant_networks | 15:19 |
| thorst | pulling down latest | 15:20 |
| esberglu | Latest what? | 15:20 |
| thorst | code... | 15:20 |
| thorst | so first..we look for the type_drivers list | 15:22 |
| thorst | doesn't overload that...so we're fine | 15:22 |
| *** efried is now known as help | 15:22 | |
| help | nice | 15:23 |
| *** help is now known as efried-doc | 15:23 | |
| thorst | esberglu: where do logs go now? | 15:28 |
| esberglu | thorst: You have to use journalctl | 15:29 |
| esberglu | thorst: journalctl -xea --unit devstack@n-cpu | 15:29 |
| esberglu | thorst: journalctl -xea --unit devstack@* | 15:29 |
| esberglu | First will give you that log | 15:29 |
| esberglu | 2nd will give you all of the logs | 15:29 |
| thorst | what's the neutron server called? | 15:30 |
| thorst | q-api? | 15:30 |
| thorst | (cause that doesn't show up) | 15:31 |
| esberglu | neutron-api | 15:34 |
| esberglu | thorst: If you type sudo systemctl status devstack@ | 15:37 |
| *** tjakobs has joined #openstack-powervm | 15:38 | |
| esberglu | And hit tab a few times you can see all the services | 15:38 |
| thorst | "InvalidInput: Invalid input for operation: physical_network 'default' unknown for VLAN provider network" | 15:40 |
| thorst | do you have the physical networks set up properly? | 15:41 |
| esberglu | thorst: Not sure? How do I determine that? | 15:47 |
| thorst | I think you need in the neutron.conf a 'network_vlan_ranges=default:1:4095' | 15:47 |
| thorst | try that, restart neutron-api and then we'll see if you can create a network then | 15:47 |
| esberglu | thorst: You mean the ml2 conf? | 15:49 |
| esberglu | Yeah I see it there | 15:49 |
| thorst | I meant more neutron.conf | 15:50 |
| thorst | but I guess try ml2.conf first | 15:50 |
| esberglu | thorst: Success! | 16:18 |
| esberglu | And I found the local.conf setting that will do that for us | 16:18 |
| thorst | :-) | 16:54 |
| thorst | excellent | 16:54 |
| *** efried-doc is now known as efried | 17:21 | |
| edmondsw | esberglu so how are things looking now that you're past the network issue (I hope)? | 17:26 |
| esberglu | edmondsw: I ran tempest and only failed 2 tests master OOT. I'm trying a run on a fresh node now | 17:26 |
| edmondsw | if you need more help, let me know | 17:27 |
| edmondsw | efried you've asked me to rebase https://review.openstack.org/#/c/472794 onto 2 separate and unrelated patches | 17:51 |
| edmondsw | that's not possible, is it? | 17:51 |
| edmondsw | one of them would have to be based on the other, and since they're unrelated, that's not the case | 17:51 |
| esberglu | edmondsw: efried: thorst: 5456 is updated. Just waiting on tempest results from the fresh instance then we can get that in | 17:51 |
| edmondsw | esberglu great, will look | 17:52 |
| esberglu | edmondsw: There's a couple TODOs in it, but at this point working is working | 17:52 |
| efried | edmondsw I'm sure it is possible, but I think you would have to have an intervening commit that merges the two of them together. Usually when I do it, I base the whole chain linearly, even if it isn't strictly necessary at the bottom. | 17:52 |
| efried | But let me remind myself which ones I was talking about. | 17:52 |
| edmondsw | efried one is so small I don't want to base the other on it... I just want to get it merged | 17:53 |
| efried | the conf.hosts one? | 17:54 |
| edmondsw | yeah | 17:54 |
| edmondsw | efried the other is totally unrelated, and will probably take a while to get merged because I've not even gotten any feedback from my earlier irc question, so I don't really want that to be the root either | 17:54 |
| edmondsw | which is my other question... should I try again on irc, or do you want to since that's your idea, or should one of us put something on the ML... ? | 17:55 |
| efried | Dah, you know, we should really get that CONF.hosts change passing our CI before we ask to get it merged. | 17:55 |
| efried | (Too late, though; I asked sdague and he already +2ed it. | 17:55 |
| efried | ) | 17:55 |
| edmondsw | efried of course, but our CI isn't passing anything yet :) | 17:56 |
| efried | Yeah, but I definitely want it passing on that one before we merge it. | 17:56 |
| edmondsw | hopefully soon | 17:56 |
| edmondsw | agreed | 17:56 |
| edmondsw | that's why I haven't bugged folks about it | 17:56 |
| edmondsw | esberglu, why the \ in [[post-config|/$Q_PLUGIN_CONF_FILE]] ? | 17:56 |
| edmondsw | sorry... / | 17:56 |
| esberglu | edmondsw: Uhh let me see if I can find the doc for it. There's a reason though | 17:59 |
| edmondsw | interesting... we didn't do that in other places, so it jumped out | 17:59 |
| esberglu | edmondsw: https://docs.openstack.org/developer/devstack/configuration.html | 17:59 |
| edmondsw | ahh... k | 18:00 |
| edmondsw | weird | 18:00 |
| edmondsw | sounds like a devstack bug, actually ;) | 18:00 |
| edmondsw | esberglu it occurs to me that we probably didn't need to get neutron working with the intree CI, since we don't have networking there yet :) | 18:01 |
| edmondsw | but if it's working now, no sense pulling it | 18:01 |
| esberglu | edmondsw: I guess that the local.conf should have Q_PLUGIN_CONF_FILE=etc/neutron/plugins/ml2/ml2_conf.ini | 18:01 |
| esberglu | Based on that doc | 18:01 |
| edmondsw | yeah | 18:01 |
| esberglu | edmondsw: Yeah I considered that as well. Was gonna see if this resolved IT too first | 18:02 |
| edmondsw | so fix Q_PLUGIN_CONF_FILE and let's merge this! | 18:03 |
| edmondsw | efried aren't there negative implications to recreating the Session if we don't really have to do that? | 18:05 |
| efried | edmondsw Like what? | 18:06 |
| edmondsw | efried hmm... actually, I think all the ones I can think of have since been resolved | 18:07 |
| esberglu | edmondsw: efried: The qos tempest tests all failed. Other than that everything looks fine | 18:18 |
| esberglu | Not sure what changed in between runs, I used the same local.conf for the previous that only failed 2 tests | 18:19 |
| esberglu | This run failed the same 2 tests | 18:19 |
| efried | Do we do qos? | 18:19 |
| esberglu | Those additional 2 being identity tests | 18:20 |
| esberglu | efried: Yeah we've been running the qos tests | 18:20 |
| edmondsw | esberglu link? | 18:20 |
| esberglu | edmondsw: dm'ed you. They don't get copied to the log server on the manual runs. I can collect them and add them to the log server manually if you want | 18:22 |
| esberglu | The rest of the logs I mean | 18:22 |
| efried | erm, mriedem merged CONF.hosts. So I realllly hope it passes the CI. | 18:43 |
| efried | edmondsw ^^ | 18:43 |
| edmondsw | efried hmm... yeah | 18:44 |
| efried | My fault for pushing for it before we had our CI going. | 18:44 |
| edmondsw | why would they merge with our CI failing? | 18:44 |
| edmondsw | a little surprised by that | 18:44 |
| edmondsw | efried no worries... if you want to push something, push for comments on https://review.openstack.org/#/c/472794/ | 18:48 |
| efried | Whole lot easier to ask for attention on a one-liner | 18:51 |
| edmondsw | no kidding :) | 18:59 |
| edmondsw | esberglu have we done this? https://docs.openstack.org/mitaka/networking-guide/config-qos.html | 19:11 |
| esberglu | edmondsw: Looks like the [agent] section was set correctly but not the [ml2] extension_drivers part | 19:19 |
| esberglu | Pushing another run through, both IT and OOT | 19:19 |
| * edmondsw crossing fingers... | 19:19 | |
| edmondsw | esberglu as for the identity tests... not sure what's going on there... | 19:21 |
| edmondsw | best guess I've been able to come up with is maybe tempest is multithreaded and some other test is trying to use the same user while it's locked, and thus extending the lockout period | 19:22 |
| edmondsw | any idea if that's possible? | 19:22 |
| edmondsw | well, except that should show up as another test failing because of account lockout. Unless the test expected some kind of failure and ate it | 19:23 |
| esberglu | edmondsw: efried: Good news ml2 conf stuff also gets IT running the tempest tests | 19:33 |
| edmondsw | esberglu woot! | 19:34 |
| edmondsw | esberglu so are you updating 5456? | 19:34 |
| esberglu | I'm gonna hold on and see if adding qos to the extension_drivers worked first | 19:35 |
| esberglu | In tree run just finished. Failed those same 2 identity tests, passed everything else | 20:04 |
| *** thorst has quit IRC | 20:27 | |
| edmondsw | esberglu any thoughts on what I said about the identity tests above? | 20:36 |
| esberglu | edmondsw: I think it is more likely a configuration thing. That could be possible, tempest does run multiple tests at a time | 20:37 |
| edmondsw | esberglu oh, I just realized that CONF.identity.user_lockout_duration is a tempest conf setting... | 20:41 |
| edmondsw | yeah, that is set (defaults) to 5, which is wildly different from the default in keystone | 20:42 |
| edmondsw | keystone's default is 1800 | 20:42 |
| edmondsw | in keystone/conf/security_compliance, called simply "lockout_duration" | 20:42 |
| edmondsw | esberglu I wonder if we're supposed to be either a) disabling security_compliance testing (which probably isn't relevant to our CI) or b) changing one of those to match the other | 20:43 |
| edmondsw | hmmm... looks like tempest is supposed to have security_compliance disabled by default (see tempest/config.py)... you didn't do something to enable that, did you? | 20:44 |
| esberglu | edmondsw: It is enabled in our tempest.conf | 20:45 |
| edmondsw | boo... well, that's our problem | 20:45 |
| esberglu | I can add that in once this OOT run finishes | 20:45 |
| *** smatzek has quit IRC | 20:46 | |
| edmondsw | esberglu looks like devstack enables that by default | 20:49 |
| esberglu | Once this run finishes I can try running those few tests with (a) and (b) | 20:50 |
| edmondsw | https://github.com/openstack-dev/devstack/blob/master/lib/keystone#L125 | 20:50 |
| esberglu | I can just reuse the environment if I'm only changing tempest.conf | 20:50 |
| *** k0da has quit IRC | 20:51 | |
| *** thorst has joined #openstack-powervm | 20:59 | |
| *** jpasqualetto has quit IRC | 21:02 | |
| edmondsw | esberglu found the bug: https://bugs.launchpad.net/keystone/+bug/1693917 | 21:06 |
| openstack | Launchpad bug 1693917 in tempest "test_user_account_lockout failed in gate because authN attempts took longer than usual" [Undecided,New] | 21:06 |
| *** smatzek has joined #openstack-powervm | 21:06 | |
| edmondsw | esberglu a devstack change was merged yesterday, but the corresponding tempest change (which they don't seem to have realized is also necessary) hasn't merged yet | 21:06 |
| edmondsw | I pinged lbragstad on the keystone channel, but I think he's probably afk | 21:07 |
| esberglu | Alright. I'll just temporarily disable until that bug is fixed. Probably something I can do | 21:07 |
| edmondsw | I think the best solution for us is probably to disable security_compliance testing... KEYSTONE_SECURITY_COMPLIANCE_ENABLED=False in devstack local.conf | 21:08 |
| esberglu | We could just set security_compliance = False in tempest.conf for now | 21:09 |
| esberglu | Setting all the tempest config options using devstack will be part of the auto generated tempest config thing | 21:09 |
| esberglu | If we change that in our local.conf it won't actually do anything. Since we use our own tempest.conf not the one devstack makes | 21:10 |
| edmondsw | oh, interesting | 21:10 |
| esberglu | I'll post a change for it | 21:11 |
| edmondsw | cool | 21:11 |
| *** smatzek has quit IRC | 21:22 | |
| esberglu | efried: edmondsw: With the latest 5456 and the security_compliance change 5466 we should *fingers crossed* be passing | 21:22 |
| esberglu | thorst: ^ | 21:22 |
| esberglu | I'm working on testing 5466 right now | 21:23 |
| esberglu | edmondsw: efried: 5466 needs a tempest change. One of the tests doesn't have a security_compliance check | 21:31 |
| efried | esberglu You gonna propose it? | 21:32 |
| efried | And disable it in the meantime? | 21:32 |
| esberglu | efried: Yeah | 21:32 |
| efried | dig | 21:32 |
| esberglu | I have to take off and take care of a few things. But if you guys are cool with the remove neutron legacy change we can merge. And then I can check back in tonight on the status | 21:32 |
| esberglu | And get that tempest fix up | 21:32 |
| esberglu | efried: edmondsw: ^ | 21:33 |
| efried | esberglu You've got +2s from me. | 21:35 |
| *** burgerk_ has joined #openstack-powervm | 21:35 | |
| *** burgerk has quit IRC | 21:38 | |
| *** burgerk_ has quit IRC | 21:40 | |
| esberglu | efried: One more change to 5466 updating the skip lists | 21:42 |
| esberglu | Can you merge once you +2 I have to head out now | 21:42 |
| efried | esberglu Will do, but I don't see the patch up yet. | 21:43 |
| esberglu | Should be now. Thanks in advance | 21:43 |
| *** esberglu has quit IRC | 21:43 | |
| *** esberglu has joined #openstack-powervm | 21:44 | |
| efried | esberglu Do we have a bug for the tempest change? | 21:45 |
| efried | edmondsw I made another delta on 5466, can you please approve? | 21:48 |
| *** esberglu has quit IRC | 21:48 | |
| *** dwayne has quit IRC | 21:56 | |
| efried | effit, I gotta run too; I'm going to merge that sucker. | 21:57 |
| *** mdrabe has quit IRC | 22:00 | |
| edmondsw | efried sorry, had to step away. lgtm except I would have asked why we didn't use the uuid for the test in skip_tests | 22:05 |
| efried | Good point. | 22:05 |
| efried | Hopefully that skippage will be short-lived. | 22:05 |
| efried | I wouldn't mind putting up another rev with a bug number in it once that's opened against tempest; we can fix that then. | 22:05 |
| efried | But in the meantime, this should work, and I think esberglu wanted to get it in ASAP so we can start maybe passing. | 22:06 |
| edmondsw | efried I'm looking at that test to see why security_compliance would even matter there... | 22:07 |
| edmondsw | efried yeah, I don't think security_compliance should matter there... so I don't think we should have had to skip that test at all | 22:09 |
| efried | oh, okay, well then we just need it in the skip section that says, "This is failing and we don't know why, so until we figure it out, we're skipping it so our CI can pass." | 22:09 |
| edmondsw | efried I think it only failed because the other test ran first and locked the account. As long as the other test doesn't run, the account won't get locked, and this test should pass | 22:09 |
| efried | or that. | 22:09 |
| efried | Let's get some runs passing first; then we can nail that down. | 22:10 |
| edmondsw | yep, agreed | 22:10 |
| efried | I'm out. Hasta mañana. | 22:10 |
| edmondsw | we can clean this up tomorrow | 22:10 |
| edmondsw | bye | 22:10 |
| *** thorst has quit IRC | 22:14 | |
| *** kylek3h has quit IRC | 22:27 | |
| *** thorst has joined #openstack-powervm | 22:33 | |
| *** thorst has quit IRC | 22:34 | |
| *** esberglu has joined #openstack-powervm | 22:38 | |
| *** tjakobs has quit IRC | 22:39 | |
| openstackgerrit | Matthew Edmonds proposed openstack/nova-powervm master: Use CONF.host for powervm nodename https://review.openstack.org/474799 | 22:42 |
| *** esberglu has quit IRC | 22:43 | |
| *** esberglu has joined #openstack-powervm | 22:54 | |
| openstackgerrit | Eric Berglund proposed openstack/nova-powervm master: DNM: ci check https://review.openstack.org/328315 | 23:00 |
| openstackgerrit | Eric Berglund proposed openstack/nova-powervm master: DNM: CI Check2 https://review.openstack.org/328317 | 23:00 |
| openstackgerrit | Matthew Edmonds proposed openstack/nova-powervm master: Remove volue_group_vios_name https://review.openstack.org/468565 | 23:02 |
| *** edmondsw has quit IRC | 23:10 | |
| *** jwcroppe has quit IRC | 23:37 | |
| *** jwcroppe has joined #openstack-powervm | 23:38 | |
| *** jwcroppe has quit IRC | 23:42 | |
| *** thorst has joined #openstack-powervm | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!