| *** acormier has joined #openstack-nova | 00:01 | |
| *** penick has quit IRC | 00:02 | |
| *** tetsuro has quit IRC | 00:03 | |
| *** mriedem1 has joined #openstack-nova | 00:05 | |
| *** acormier has quit IRC | 00:05 | |
| *** mriedem has quit IRC | 00:05 | |
| *** mriedem1 is now known as mriedem | 00:08 | |
| *** thorst has joined #openstack-nova | 00:08 | |
| *** thorst has quit IRC | 00:09 | |
| *** yangyapeng has quit IRC | 00:11 | |
| *** yangyapeng has joined #openstack-nova | 00:12 | |
| mriedem | dansmith: interesting, listing without details, 500 error (cell0) and 500 active (cell1) is a lot faster than the 1000 active, | 00:13 |
|---|---|---|
| mriedem | i suppose because we don't have as much to join | 00:13 |
| dansmith | mriedem: with my patch or before? | 00:13 |
| mriedem | oh shit, nvm - copy paste error | 00:13 |
| mriedem | was using the compute endpoint url from my other devstack :) | 00:13 |
| mriedem | "wow this is fast!" | 00:14 |
| mriedem | hah, here we go, nice and slow | 00:15 |
| mriedem | fault loading mofos | 00:15 |
| *** yufei has joined #openstack-nova | 00:18 | |
| *** yufei has quit IRC | 00:18 | |
| mriedem | 4.495s with GET /servers, 1000 ACTIVE vms. 11.185s with 500 error, 500 active | 00:20 |
| *** moshele has quit IRC | 00:27 | |
| *** smatzek_ has quit IRC | 00:29 | |
| mriedem | 24.125s to list them with details | 00:30 |
| *** hemna_ has joined #openstack-nova | 00:34 | |
| *** nicolasbock has quit IRC | 00:35 | |
| *** yamahata has quit IRC | 00:36 | |
| *** Apoorva_ has joined #openstack-nova | 00:40 | |
| *** nicolasbock has joined #openstack-nova | 00:42 | |
| openstackgerrit | Michael Still proposed openstack/nova master: Move ploop commands to privsep. https://review.openstack.org/492325 | 00:42 |
| openstackgerrit | Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486 | 00:42 |
| openstackgerrit | Michael Still proposed openstack/nova master: Don't shell out to mkdir, use ensure_tree() https://review.openstack.org/492326 | 00:42 |
| openstackgerrit | Michael Still proposed openstack/nova master: Cleanup mount / umount and associated rmdir calls https://review.openstack.org/494423 | 00:42 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move lvm handling to privsep. https://review.openstack.org/495516 | 00:42 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move shred to privsep. https://review.openstack.org/495537 | 00:42 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xend existence probes to privsep. https://review.openstack.org/495538 | 00:42 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move the idmapshift binary into privsep. https://review.openstack.org/495541 | 00:42 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move loopback setup and removal to privsep. https://review.openstack.org/495664 | 00:42 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move nbd commands to privsep. https://review.openstack.org/500351 | 00:42 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move kpartx calls to privsep. https://review.openstack.org/500354 | 00:42 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move blkid calls to privsep. https://review.openstack.org/500398 | 00:42 |
| *** Apoorva has quit IRC | 00:43 | |
| mriedem | interesting, listing with details and microversion 2.53 is not much worse than with microversion 2.1 for the error/active mix case - it was nearly double between microversions when all were active | 00:44 |
| mriedem | dansmith: time for your change, do i need https://review.openstack.org/#/c/505456/ or just the one below it? | 00:47 |
| dansmith | mriedem: the one below it should orphan those so they're never called | 00:48 |
| dansmith | so you shouldn't notice any difference afaik | 00:48 |
| mriedem | ok | 00:48 |
| *** acormier has joined #openstack-nova | 00:49 | |
| *** acormier has quit IRC | 00:50 | |
| *** acormier has joined #openstack-nova | 00:51 | |
| *** jichen has joined #openstack-nova | 00:52 | |
| *** acormier has quit IRC | 00:53 | |
| *** acormier has joined #openstack-nova | 00:54 | |
| *** Apoorva_ has quit IRC | 00:54 | |
| *** acormier_ has joined #openstack-nova | 00:56 | |
| *** Shunli has joined #openstack-nova | 00:58 | |
| *** acormier has quit IRC | 00:58 | |
| *** esberglu has joined #openstack-nova | 00:59 | |
| *** yamamoto has quit IRC | 01:03 | |
| *** yamamoto_ has joined #openstack-nova | 01:03 | |
| *** phuongnh has joined #openstack-nova | 01:03 | |
| *** esberglu has quit IRC | 01:03 | |
| *** Sukhdev has quit IRC | 01:03 | |
| *** thorst has joined #openstack-nova | 01:06 | |
| *** thorst has quit IRC | 01:06 | |
| *** litao__ has joined #openstack-nova | 01:06 | |
| *** Tom_ has joined #openstack-nova | 01:13 | |
| *** yingjun has joined #openstack-nova | 01:24 | |
| *** Tom_ has quit IRC | 01:25 | |
| *** Tom_ has joined #openstack-nova | 01:25 | |
| *** Tom_ has quit IRC | 01:30 | |
| *** coreywright has quit IRC | 01:32 | |
| *** shaner has quit IRC | 01:33 | |
| *** ijw has quit IRC | 01:33 | |
| *** shaner has joined #openstack-nova | 01:34 | |
| *** yufei has joined #openstack-nova | 01:35 | |
| mriedem | dansmith: ok i have results in https://etherpad.openstack.org/p/nova-instance-list | 01:38 |
| mriedem | with your change | 01:38 |
| dansmith | is that faster/same except for details? | 01:39 |
| mriedem | compared to w/o your change, (1) GET /servers with microversion 2.1 is slightly faster | 01:39 |
| mriedem | GET /servers/detail with microversion is about the same, a bit faster | 01:39 |
| mriedem | 2.1 | 01:39 |
| mriedem | but, GET /server/details with microversion 2.53 is slower | 01:39 |
| mriedem | not a ton, but it's slower | 01:40 |
| mriedem | 25.78 compared to 30.10 | 01:40 |
| mriedem | but, it's not a huge different | 01:40 |
| dansmith | oh only detail with the later microversion | 01:40 |
| mriedem | right | 01:40 |
| mriedem | something about >2.1 always makes listing with details slower | 01:40 |
| dansmith | and there's some fault handling behavior difference? | 01:40 |
| mriedem | at least because of the joins on the (1) services table and (2) tags table | 01:40 |
| mriedem | i don't think there is any fault handling behavior differences with microversion >2.1 | 01:41 |
| *** mingyu has quit IRC | 01:41 | |
| mriedem | if there was, that might explain it | 01:41 |
| dansmith | okay I thought you were saying there was | 01:41 |
| dansmith | I dunno why because I'm pre-joining it when we were loading them separate | 01:41 |
| *** mingyu has joined #openstack-nova | 01:41 | |
| mriedem | the only other joins i can think of right now with microversion >2.1 is on the services table (2.16) and tags able (2.26) | 01:41 |
| mriedem | still, it's a difference of about 4 seconds, which isn't huge here | 01:42 |
| dansmith | so aside from fault, there's no difference in what I'm doing vs what we do currently, | 01:42 |
| dansmith | other than we're not serializing the queries | 01:42 |
| *** avolkov has quit IRC | 01:43 | |
| dansmith | without my change we issue the cell0 one and then the cell1 one, where now we're doing both at once | 01:43 |
| dansmith | is this a devstack vm on your laptop or something better? | 01:43 |
| mriedem | it's in a vexxhost vm | 01:43 |
| mriedem | the fault stuff is the only major difference i can think of, since we'll be joining on fault all the time, rather than just for instances in ERROR state | 01:44 |
| mriedem | maybe that is equaling things out somehow, idk, like if i had 1000 all in ERROR state before/after your change, that might be different in favor of yours | 01:45 |
| dansmith | hmm, yeah, I guess maybe that might be it | 01:45 |
| *** chyka has quit IRC | 01:46 | |
| mriedem | i do have the numbers from yesterday before your change with 1000 ACTIVE, | 01:46 |
| mriedem | so tomorrow i could run yours through with all active and see if there is a bigger difference because of the fault join | 01:46 |
| dansmith | well, I guess we could go back to the not automatic loading of fault | 01:46 |
| *** crushil has joined #openstack-nova | 01:46 | |
| mriedem | i'll run that all active scenario tomorrow to see if it could be the fault stuff, | 01:46 |
| mriedem | it's nearly 9pm so i'm not going to do it tonight | 01:46 |
| dansmith | there was something the API was doing that made it seem way better to do this than what it was doing | 01:47 |
| dansmith | but it's been a while now | 01:47 |
| dansmith | we could also plumb the logic of when to load the fault into the lower layers | 01:48 |
| mriedem | yup i was thinking that too | 01:49 |
| mriedem | another thing that might be causing the microversion bloat, is maybe the microversion to pull the embedded flavor out of the instance | 01:49 |
| mriedem | added in pike | 01:49 |
| dansmith | you could run through each microversion and see where the spike is | 01:50 |
| *** coreywright has joined #openstack-nova | 01:50 | |
| dansmith | the sorting layer on top of this really has nothing to do with what we're sorting though | 01:50 |
| dansmith | it doesn't make any more copies of things, nor iterate the list more times | 01:50 |
| mriedem | 2.47 | 01:50 |
| dansmith | so, the change right before the switchover should do the fault loading but not the sorting, so you could run against that and see if it's more like the earlier or more like the later | 01:51 |
| *** chyka has joined #openstack-nova | 01:52 | |
| mriedem | https://review.openstack.org/#/c/506774/ ? | 01:53 |
| mriedem | like, revert that on top of the change that uses the new code in the API | 01:53 |
| mriedem | ? | 01:54 |
| mriedem | oh, nvm, | 01:54 |
| dansmith | oh, I guess you were running on master already? | 01:54 |
| mriedem | yes, new devstack as of today | 01:54 |
| dansmith | yeah, okay | 01:54 |
| *** Apoorva has joined #openstack-nova | 01:54 | |
| mriedem | so 2.16 makes us join on services, 2.26 makes us join on tags, 2.47 returns instance.flavor, and your change always joins on faults | 01:55 |
| mriedem | 2.47 is suspicious | 01:55 |
| mriedem | since that's from instance_extra | 01:55 |
| dansmith | but again, it shouldn't be any different | 01:55 |
| mriedem | yeah, nvm, we also didn't start loading that in the api as of 2.47, we already pulled out instance.flavor to get the link stuff | 01:57 |
| *** hongbin has joined #openstack-nova | 01:58 | |
| mriedem | totally unrelated, but when we lazy-load instance.flavor, we're still joining on system_metadata now, we should be able to stop doing that | 01:59 |
| dansmith | yeah | 02:00 |
| *** takashin has quit IRC | 02:02 | |
| mriedem | ok, i'll run through with 1000 active instances tomorrow with your change and see if that makes a big difference, and if so, it could be the fault thing | 02:04 |
| dansmith | alright | 02:05 |
| *** thorst has joined #openstack-nova | 02:07 | |
| *** Tom_ has joined #openstack-nova | 02:24 | |
| *** Tom__ has joined #openstack-nova | 02:27 | |
| *** gbarros has quit IRC | 02:28 | |
| *** yushb has joined #openstack-nova | 02:28 | |
| *** Tom___ has joined #openstack-nova | 02:29 | |
| *** Tom_ has quit IRC | 02:29 | |
| yushb | JOIN #openstack-karbor | 02:29 |
| *** yushb has left #openstack-nova | 02:30 | |
| *** Tom__ has quit IRC | 02:31 | |
| *** vladikr has quit IRC | 02:31 | |
| *** vladikr has joined #openstack-nova | 02:32 | |
| *** yushb has joined #openstack-nova | 02:32 | |
| *** esberglu has joined #openstack-nova | 02:42 | |
| *** acormier has joined #openstack-nova | 02:46 | |
| *** esberglu has quit IRC | 02:47 | |
| *** erlon has quit IRC | 02:48 | |
| *** thorst has quit IRC | 02:48 | |
| *** mingyu has quit IRC | 02:48 | |
| *** acormier_ has quit IRC | 02:49 | |
| *** csuttles has quit IRC | 02:52 | |
| *** csuttles has joined #openstack-nova | 02:53 | |
| *** csuttles has quit IRC | 02:54 | |
| *** csuttles has joined #openstack-nova | 02:55 | |
| *** acormier has quit IRC | 02:56 | |
| dansmith | mriedem: I don't auto-join fault until: https://review.openstack.org/#/c/505456/10/nova/api/openstack/compute/servers.py | 03:01 |
| dansmith | so I don't think it's the fault | 03:01 |
| *** pino has joined #openstack-nova | 03:04 | |
| *** mingyu has joined #openstack-nova | 03:06 | |
| *** tojuvone has quit IRC | 03:08 | |
| *** itlinux has joined #openstack-nova | 03:12 | |
| *** ijw has joined #openstack-nova | 03:12 | |
| *** vladikr has quit IRC | 03:13 | |
| *** vladikr has joined #openstack-nova | 03:13 | |
| *** ijw has quit IRC | 03:17 | |
| *** nicolasbock has quit IRC | 03:21 | |
| *** TuanLA has joined #openstack-nova | 03:21 | |
| *** mingyu has quit IRC | 03:23 | |
| *** yufei has quit IRC | 03:27 | |
| *** tojuvone has joined #openstack-nova | 03:27 | |
| *** yushb has quit IRC | 03:29 | |
| *** yangyapeng has quit IRC | 03:32 | |
| *** yangyapeng has joined #openstack-nova | 03:32 | |
| *** gyee has quit IRC | 03:33 | |
| openstackgerrit | Michael Still proposed openstack/nova master: Move ploop commands to privsep. https://review.openstack.org/492325 | 03:33 |
| openstackgerrit | Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486 | 03:33 |
| openstackgerrit | Michael Still proposed openstack/nova master: Don't shell out to mkdir, use ensure_tree() https://review.openstack.org/492326 | 03:33 |
| openstackgerrit | Michael Still proposed openstack/nova master: Cleanup mount / umount and associated rmdir calls https://review.openstack.org/494423 | 03:33 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move lvm handling to privsep. https://review.openstack.org/495516 | 03:33 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move shred to privsep. https://review.openstack.org/495537 | 03:33 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xend existence probes to privsep. https://review.openstack.org/495538 | 03:33 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move the idmapshift binary into privsep. https://review.openstack.org/495541 | 03:33 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move loopback setup and removal to privsep. https://review.openstack.org/495664 | 03:33 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move nbd commands to privsep. https://review.openstack.org/500351 | 03:33 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move kpartx calls to privsep. https://review.openstack.org/500354 | 03:33 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move blkid calls to privsep. https://review.openstack.org/500398 | 03:33 |
| *** yangyapeng has quit IRC | 03:37 | |
| *** itlinux has quit IRC | 03:41 | |
| *** thorst has joined #openstack-nova | 03:45 | |
| *** Sukhdev has joined #openstack-nova | 03:46 | |
| *** Tom___ has quit IRC | 03:49 | |
| *** Tom has joined #openstack-nova | 03:50 | |
| *** Atom1234 has joined #openstack-nova | 03:51 | |
| *** vladikr has quit IRC | 03:52 | |
| *** owalsh_ has joined #openstack-nova | 03:52 | |
| *** yangzhenyu has quit IRC | 03:52 | |
| *** diga has joined #openstack-nova | 03:53 | |
| openstackgerrit | Merged openstack/nova master: Add slowest command to tox.ini https://review.openstack.org/507657 | 03:54 |
| *** Tom has quit IRC | 03:55 | |
| *** owalsh has quit IRC | 03:56 | |
| *** crushil has quit IRC | 03:57 | |
| *** vladikr has joined #openstack-nova | 04:00 | |
| *** krtaylor has quit IRC | 04:00 | |
| *** krtaylor has joined #openstack-nova | 04:02 | |
| *** ratailor has joined #openstack-nova | 04:08 | |
| *** krtaylor has quit IRC | 04:08 | |
| *** thorst has quit IRC | 04:16 | |
| *** jaosorior has joined #openstack-nova | 04:19 | |
| *** Apoorva_ has joined #openstack-nova | 04:21 | |
| *** hongbin has quit IRC | 04:21 | |
| *** Atom1234 has quit IRC | 04:21 | |
| *** kashyap` has joined #openstack-nova | 04:22 | |
| *** mhenkel_ has joined #openstack-nova | 04:23 | |
| *** spotz_ has joined #openstack-nova | 04:24 | |
| *** migi_ has joined #openstack-nova | 04:26 | |
| *** mdnadeem has joined #openstack-nova | 04:26 | |
| *** csuttles_ has joined #openstack-nova | 04:27 | |
| *** Sukhdev has quit IRC | 04:29 | |
| *** csuttles has quit IRC | 04:29 | |
| *** Apoorva has quit IRC | 04:29 | |
| *** hemna_ has quit IRC | 04:29 | |
| *** mriedem has quit IRC | 04:29 | |
| *** xinliang has quit IRC | 04:29 | |
| *** danpawlik has quit IRC | 04:29 | |
| *** cburgess has quit IRC | 04:29 | |
| *** spotz has quit IRC | 04:29 | |
| *** jcook has quit IRC | 04:29 | |
| *** sambetts|afk has quit IRC | 04:29 | |
| *** ujjain- has quit IRC | 04:29 | |
| *** mhenkel has quit IRC | 04:29 | |
| *** s1061123 has quit IRC | 04:29 | |
| *** ericyoung has quit IRC | 04:29 | |
| *** migi has quit IRC | 04:29 | |
| *** stephenfin has quit IRC | 04:29 | |
| *** obre has quit IRC | 04:29 | |
| *** kashyap has quit IRC | 04:29 | |
| *** gryf has quit IRC | 04:29 | |
| *** stephenfin has joined #openstack-nova | 04:30 | |
| *** s1061123 has joined #openstack-nova | 04:30 | |
| *** sambetts_ has joined #openstack-nova | 04:30 | |
| *** cburgess has joined #openstack-nova | 04:30 | |
| *** ericyoung has joined #openstack-nova | 04:30 | |
| *** claudiub has joined #openstack-nova | 04:30 | |
| *** ujjain has joined #openstack-nova | 04:31 | |
| *** ujjain has quit IRC | 04:31 | |
| *** ujjain has joined #openstack-nova | 04:31 | |
| *** jcook has joined #openstack-nova | 04:32 | |
| *** john5223_ has quit IRC | 04:32 | |
| *** slagle has quit IRC | 04:34 | |
| *** ansiwen has quit IRC | 04:34 | |
| *** mdbooth has quit IRC | 04:34 | |
| *** crushil has joined #openstack-nova | 04:34 | |
| *** udesale has joined #openstack-nova | 04:36 | |
| *** xinliang has joined #openstack-nova | 04:36 | |
| *** obre has joined #openstack-nova | 04:36 | |
| *** gryf has joined #openstack-nova | 04:37 | |
| *** hemna_ has joined #openstack-nova | 04:37 | |
| *** danpawlik has joined #openstack-nova | 04:37 | |
| *** manasm has joined #openstack-nova | 04:39 | |
| *** Apoorva_ has quit IRC | 04:40 | |
| *** sree has joined #openstack-nova | 04:40 | |
| *** slagle has joined #openstack-nova | 04:41 | |
| *** armax has joined #openstack-nova | 04:42 | |
| *** Shunli has quit IRC | 04:42 | |
| *** pino has quit IRC | 04:43 | |
| *** vladikr has quit IRC | 04:46 | |
| *** psachin has joined #openstack-nova | 04:47 | |
| *** mdbooth has joined #openstack-nova | 04:48 | |
| *** ansiwen has joined #openstack-nova | 04:48 | |
| *** vladikr has joined #openstack-nova | 04:50 | |
| *** vladikr has quit IRC | 04:56 | |
| *** acormier has joined #openstack-nova | 04:56 | |
| *** vladikr has joined #openstack-nova | 04:59 | |
| *** acormier has quit IRC | 05:00 | |
| *** chyka has quit IRC | 05:11 | |
| *** zhurong has joined #openstack-nova | 05:13 | |
| *** thorst has joined #openstack-nova | 05:13 | |
| *** vladikr has quit IRC | 05:14 | |
| *** vladikr has joined #openstack-nova | 05:17 | |
| *** phuongnh has quit IRC | 05:19 | |
| *** sridharg has joined #openstack-nova | 05:21 | |
| *** crushil has quit IRC | 05:21 | |
| *** vladikr has quit IRC | 05:22 | |
| *** lajoskatona has joined #openstack-nova | 05:23 | |
| *** esberglu has joined #openstack-nova | 05:25 | |
| *** Atom1234 has joined #openstack-nova | 05:26 | |
| *** Tom has joined #openstack-nova | 05:28 | |
| *** esberglu has quit IRC | 05:29 | |
| *** trinaths1 has joined #openstack-nova | 05:31 | |
| *** trinaths1 has left #openstack-nova | 05:31 | |
| *** Eran_Kuris has joined #openstack-nova | 05:38 | |
| *** elod has joined #openstack-nova | 05:43 | |
| *** avolkov has joined #openstack-nova | 05:46 | |
| *** thorst has quit IRC | 05:47 | |
| *** Tom has quit IRC | 05:49 | |
| *** ltomasbo has quit IRC | 05:49 | |
| *** markmc has quit IRC | 05:50 | |
| *** Atom1234 has quit IRC | 05:50 | |
| *** Tom has joined #openstack-nova | 05:51 | |
| *** Atom1234 has joined #openstack-nova | 05:51 | |
| *** jpena|off has quit IRC | 05:52 | |
| openstackgerrit | jichenjc proposed openstack/nova master: check query param for used_limits function https://review.openstack.org/499091 | 06:07 |
| *** Oku_OS-away is now known as Oku_OS | 06:10 | |
| *** avolkov has quit IRC | 06:13 | |
| *** avolkov has joined #openstack-nova | 06:14 | |
| *** liuyulong has joined #openstack-nova | 06:16 | |
| *** armax has quit IRC | 06:19 | |
| lennyb | Hi, I am working on devstack master, and my n-cond-cell1.service got stucked during stop #link http://paste.openstack.org/show/622005/. it's log is empty, no exceptions in other logs. Any tips/ideas will be appreciated. | 06:21 |
| *** ltomasbo has joined #openstack-nova | 06:21 | |
| *** Tom has quit IRC | 06:26 | |
| *** edand has joined #openstack-nova | 06:30 | |
| *** NehaAlhat has joined #openstack-nova | 06:34 | |
| *** markmc has joined #openstack-nova | 06:35 | |
| *** moshele has joined #openstack-nova | 06:36 | |
| *** andreas_s has joined #openstack-nova | 06:36 | |
| *** neha_alhat has quit IRC | 06:36 | |
| *** Atom1234 has quit IRC | 06:36 | |
| openstackgerrit | jichenjc proposed openstack/nova master: check query param for server groups function https://review.openstack.org/500347 | 06:37 |
| *** moshele has quit IRC | 06:38 | |
| openstackgerrit | Yikun Jiang proposed openstack/nova master: Update Instance action's updated_at when action event updated. https://review.openstack.org/507473 | 06:39 |
| *** rcernin has joined #openstack-nova | 06:39 | |
| *** armax has joined #openstack-nova | 06:41 | |
| *** yamamoto_ has quit IRC | 06:41 | |
| *** thorst has joined #openstack-nova | 06:45 | |
| *** yamamoto has joined #openstack-nova | 06:45 | |
| *** cfriesen_ has quit IRC | 06:46 | |
| *** Tom has joined #openstack-nova | 06:47 | |
| *** armax has quit IRC | 06:50 | |
| openstackgerrit | Lajos Katona proposed openstack/nova master: Moving more utils to ServerResourceAllocationTestBase https://review.openstack.org/499539 | 06:57 |
| openstackgerrit | Lajos Katona proposed openstack/nova master: factor out compute service start in ServerMovingTest https://review.openstack.org/503037 | 06:57 |
| openstackgerrit | Lajos Katona proposed openstack/nova master: Test resource allocation during soft delete https://review.openstack.org/495159 | 06:57 |
| openstackgerrit | Alex Xu proposed openstack/nova-specs master: Add trait support in the allocation candidates API https://review.openstack.org/497713 | 06:59 |
| *** sshwarts has joined #openstack-nova | 06:59 | |
| *** yamamoto_ has joined #openstack-nova | 07:04 | |
| *** yamamoto has quit IRC | 07:04 | |
| *** sree has quit IRC | 07:06 | |
| *** sree has joined #openstack-nova | 07:07 | |
| openstackgerrit | Yikun Jiang proposed openstack/nova-specs master: Add pagination and changes since filter support for os-instance-action API https://review.openstack.org/507762 | 07:10 |
| *** yamamoto_ has quit IRC | 07:10 | |
| *** sree has quit IRC | 07:11 | |
| *** bhagyashri_s has joined #openstack-nova | 07:11 | |
| *** ps_jadhav has joined #openstack-nova | 07:11 | |
| *** neha_alhat has joined #openstack-nova | 07:11 | |
| *** tssurya has joined #openstack-nova | 07:13 | |
| *** neha_alhat has quit IRC | 07:13 | |
| *** bhagyashri_s has quit IRC | 07:13 | |
| *** esberglu has joined #openstack-nova | 07:13 | |
| *** sree has joined #openstack-nova | 07:13 | |
| *** NehaAlhat has quit IRC | 07:13 | |
| *** bhagyashris has quit IRC | 07:14 | |
| *** pooja_jadhav has quit IRC | 07:14 | |
| *** ps_jadhav has quit IRC | 07:14 | |
| *** Dinesh_Bhor has quit IRC | 07:17 | |
| *** thorst has quit IRC | 07:17 | |
| *** esberglu has quit IRC | 07:17 | |
| *** phuongnh has joined #openstack-nova | 07:18 | |
| *** yamamoto has joined #openstack-nova | 07:18 | |
| openstackgerrit | Yikun Jiang proposed openstack/nova-specs master: Add pagination and changes since filter support for os-instance-action API https://review.openstack.org/507762 | 07:20 |
| *** neha_alhat has joined #openstack-nova | 07:20 | |
| *** bhagyashris has joined #openstack-nova | 07:20 | |
| *** pooja_jadhav has joined #openstack-nova | 07:21 | |
| *** Dinesh_Bhor has joined #openstack-nova | 07:21 | |
| *** tesseract has joined #openstack-nova | 07:21 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova stable/pike: Mention API behavior change when over quota limit https://review.openstack.org/507769 | 07:25 |
| *** jaosorior has quit IRC | 07:31 | |
| *** moshele has joined #openstack-nova | 07:37 | |
| *** ragiman has joined #openstack-nova | 07:39 | |
| *** manasm has quit IRC | 07:40 | |
| *** manasm has joined #openstack-nova | 07:41 | |
| *** pooja-jadhav has joined #openstack-nova | 07:46 | |
| *** jaosorior has joined #openstack-nova | 07:46 | |
| *** jpena has joined #openstack-nova | 07:47 | |
| *** ijw has joined #openstack-nova | 07:47 | |
| *** pooja_jadhav has quit IRC | 07:48 | |
| *** tetsuro has joined #openstack-nova | 07:52 | |
| *** alexchadin has joined #openstack-nova | 07:53 | |
| *** kashyap` is now known as kashyap | 07:57 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.openstack.org/507486 | 07:58 |
| *** ijw has quit IRC | 08:02 | |
| *** yamahata has joined #openstack-nova | 08:02 | |
| *** ralonsoh has joined #openstack-nova | 08:04 | |
| *** mingyu has joined #openstack-nova | 08:04 | |
| *** Tom has quit IRC | 08:06 | |
| *** efried has quit IRC | 08:08 | |
| *** zhurong has quit IRC | 08:11 | |
| *** thorst has joined #openstack-nova | 08:14 | |
| *** Tom has joined #openstack-nova | 08:16 | |
| *** efried has joined #openstack-nova | 08:19 | |
| *** Tom has quit IRC | 08:20 | |
| *** owalsh_ is now known as owalsh | 08:24 | |
| *** brault has quit IRC | 08:24 | |
| *** baoli has joined #openstack-nova | 08:28 | |
| *** yamahata has quit IRC | 08:29 | |
| *** Tom has joined #openstack-nova | 08:29 | |
| *** zhurong has joined #openstack-nova | 08:30 | |
| gibi | johnthetubaguy: hi! do you have any comments on http://lists.openstack.org/pipermail/openstack-dev/2017-September/122658.html ? mikal has already stated that rackspace private cloud is not using it http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2017-09-26.log.html#t2017-09-26T19:57:03 | 08:30 |
| *** baoli has quit IRC | 08:32 | |
| *** mingyu has quit IRC | 08:40 | |
| *** mingyu has joined #openstack-nova | 08:41 | |
| *** alexchadin has quit IRC | 08:41 | |
| *** alexchadin has joined #openstack-nova | 08:42 | |
| *** derekh has joined #openstack-nova | 08:47 | |
| *** alexchadin has quit IRC | 08:48 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove dead code of api.fault notification sending https://review.openstack.org/505164 | 08:49 |
| *** thorst has quit IRC | 08:50 | |
| *** hferenc has joined #openstack-nova | 08:54 | |
| ralonsoh | dansmith: hi, can I talk about https://review.openstack.org/#/c/449257/42/nova/objects/instance_pci_requests.py? | 08:56 |
| *** udesale has quit IRC | 08:57 | |
| *** udesale has joined #openstack-nova | 08:57 | |
| johnthetubaguy | gibi: I am very far removed from if that would be used now | 08:58 |
| johnthetubaguy | gibi: I believe the context around that was providing an SLA that involved knowing about the 500 errors in the API, and doing bug fixes to reduce them | 08:59 |
| johnthetubaguy | gibi: i.e. the SLA is related to the % of 5xx errors from the API, so errors in the API are really important to those folks | 08:59 |
| johnthetubaguy | gibi: I don't remember using those notifications myself, mostly got that data from ELK when I was last looking at that stuff | 09:00 |
| *** esberglu has joined #openstack-nova | 09:01 | |
| johnthetubaguy | gibi: tl;dr +1 on killing it | 09:01 |
| gibi | johnthetubaguy: thanks for the info | 09:02 |
| johnthetubaguy | gibi: np | 09:02 |
| stephenfin | gibi: Super easy docs patch needing another +2 here, if you're twiddling your thumbs at any point today :) https://review.openstack.org/#/c/502105/ | 09:04 |
| stephenfin | Well, kinda easy | 09:04 |
| gibi | stephenfin: I will look shortly | 09:04 |
| stephenfin | no rush | 09:04 |
| *** claudiub|2 has joined #openstack-nova | 09:05 | |
| *** esberglu has quit IRC | 09:06 | |
| *** claudiub|3 has joined #openstack-nova | 09:06 | |
| *** alexchadin has joined #openstack-nova | 09:07 | |
| *** alexchadin has quit IRC | 09:08 | |
| *** claudiub has quit IRC | 09:08 | |
| *** alexchadin has joined #openstack-nova | 09:08 | |
| *** claudiub|2 has quit IRC | 09:10 | |
| *** alexchadin has quit IRC | 09:11 | |
| *** hferenc has quit IRC | 09:13 | |
| *** yamamoto has quit IRC | 09:19 | |
| *** tssurya has quit IRC | 09:22 | |
| *** sambetts_ is now known as sambetts | 09:27 | |
| openstackgerrit | Moshe Levi proposed openstack/nova master: Don't overwrite binding-profile https://review.openstack.org/505613 | 09:29 |
| *** brault has joined #openstack-nova | 09:33 | |
| *** mvk has quit IRC | 09:34 | |
| *** josecastroleon has joined #openstack-nova | 09:41 | |
| gibi | stephenfin: left some comments in https://review.openstack.org/#/c/502105/ | 09:42 |
| stephenfin | gibi: On it. Thanks :) | 09:42 |
| gibi | stephenfin: sorry for being picky. I can accept if most of my comments are fixed in a followup | 09:42 |
| gibi | stephenfin: the only thing that I think is a must https://review.openstack.org/#/c/502105/1/doc/source/cli/nova-compute.rst@25 | 09:43 |
| *** thorst has joined #openstack-nova | 09:47 | |
| gibi | stephenfin: if you need something to review then I can suggest this test improvement series https://review.openstack.org/#/c/499539/ :) | 09:47 |
| *** afazekas is now known as afazekas|seek4fo | 09:48 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware https://review.openstack.org/507486 | 09:50 |
| *** sdague has joined #openstack-nova | 09:50 | |
| *** Atom1234 has joined #openstack-nova | 09:52 | |
| *** gszasz has joined #openstack-nova | 09:55 | |
| *** diga has quit IRC | 09:55 | |
| *** diga has joined #openstack-nova | 09:57 | |
| *** esberglu has joined #openstack-nova | 10:00 | |
| *** edand has quit IRC | 10:00 | |
| *** josecastroleon has quit IRC | 10:00 | |
| *** diga has quit IRC | 10:04 | |
| *** esberglu has quit IRC | 10:04 | |
| *** yingjun has quit IRC | 10:05 | |
| *** TuanLA has quit IRC | 10:06 | |
| *** pooja-jadhav is now known as pooja_jadhav | 10:11 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Rework man pages https://review.openstack.org/502105 | 10:12 |
| *** tetsuro has quit IRC | 10:16 | |
| *** thorst has quit IRC | 10:18 | |
| *** jichen has quit IRC | 10:19 | |
| *** Tom has quit IRC | 10:19 | |
| *** zhurong has quit IRC | 10:20 | |
| *** Tom has joined #openstack-nova | 10:20 | |
| *** yamamoto has joined #openstack-nova | 10:20 | |
| *** Tom has quit IRC | 10:21 | |
| *** rmart04 has joined #openstack-nova | 10:25 | |
| *** yamahata has joined #openstack-nova | 10:27 | |
| *** manasm has quit IRC | 10:30 | |
| *** manasm has joined #openstack-nova | 10:32 | |
| *** andreas_s_ has joined #openstack-nova | 10:37 | |
| *** andreas_s has quit IRC | 10:40 | |
| *** syqian has joined #openstack-nova | 10:44 | |
| *** andreas_s_ has quit IRC | 10:45 | |
| *** edand has joined #openstack-nova | 10:47 | |
| *** diga has joined #openstack-nova | 10:47 | |
| *** andreas_s has joined #openstack-nova | 10:47 | |
| *** mvk has joined #openstack-nova | 10:48 | |
| *** udesale has quit IRC | 10:52 | |
| openstackgerrit | Lee Yarwood proposed openstack/python-novaclient stable/newton: Fix aggregate_update name and availability_zone clash https://review.openstack.org/507816 | 10:52 |
| *** smatzek_ has joined #openstack-nova | 10:52 | |
| *** yassine has quit IRC | 10:55 | |
| *** Atom1234 has quit IRC | 10:57 | |
| *** syqian has quit IRC | 10:57 | |
| *** hemna_ has quit IRC | 10:57 | |
| *** yassine has joined #openstack-nova | 10:58 | |
| *** mingyu has quit IRC | 11:00 | |
| *** nicolasbock has joined #openstack-nova | 11:00 | |
| *** nicolasbock has quit IRC | 11:04 | |
| alex_xu | gmann: sorry, I can't make the api meeting today, have cold today, headache | 11:06 |
| *** jaosorior is now known as jaosorior_sick | 11:12 | |
| *** cdent has joined #openstack-nova | 11:13 | |
| *** thorst has joined #openstack-nova | 11:15 | |
| *** phuongnh has quit IRC | 11:18 | |
| *** diga has quit IRC | 11:23 | |
| *** acormier has joined #openstack-nova | 11:24 | |
| *** _pewp_ has quit IRC | 11:24 | |
| *** mingyu has joined #openstack-nova | 11:26 | |
| *** thorst has quit IRC | 11:27 | |
| cdent | efried: you on the scene yet? | 11:27 |
| *** pcaruana has joined #openstack-nova | 11:27 | |
| *** udesale has joined #openstack-nova | 11:28 | |
| cdent | efried: when you are, I’ve got a few questions about non-openstack managed workloads | 11:28 |
| *** liuyulong has quit IRC | 11:30 | |
| *** _pewp_ has joined #openstack-nova | 11:31 | |
| *** tylerderosagrund has joined #openstack-nova | 11:39 | |
| openstackgerrit | Lajos Katona proposed openstack/nova master: Change live_migrate tests to use fakedriver https://review.openstack.org/505202 | 11:46 |
| *** dave-mccowan has joined #openstack-nova | 11:50 | |
| *** acormier has quit IRC | 11:53 | |
| *** trinaths has joined #openstack-nova | 11:54 | |
| *** jpena is now known as jpena|lunch | 11:54 | |
| *** thorst has joined #openstack-nova | 11:55 | |
| manasm | bauzas: you may want to take a look at https://bugs.launchpad.net/nova/+bug/1719859 . | 11:57 |
| openstack | Launchpad bug 1719859 in OpenStack Compute (nova) "Resize failure due to instance group being None in request spec" [Undecided,New] | 11:57 |
| *** tylerderosagrund has quit IRC | 11:57 | |
| *** sahid has joined #openstack-nova | 12:01 | |
| *** dave-mccowan has quit IRC | 12:01 | |
| *** edand has quit IRC | 12:01 | |
| *** dave-mccowan has joined #openstack-nova | 12:04 | |
| *** tssurya has joined #openstack-nova | 12:04 | |
| *** litao__ has quit IRC | 12:04 | |
| *** chyka has joined #openstack-nova | 12:07 | |
| *** afazekas|seek4fo is now known as afazekas | 12:09 | |
| *** chyka has quit IRC | 12:11 | |
| *** acormier has joined #openstack-nova | 12:15 | |
| *** smatzek_ is now known as smatzek | 12:16 | |
| *** dtantsur|afk is now known as dtantsur | 12:16 | |
| *** alexchadin has joined #openstack-nova | 12:16 | |
| *** edmondsw has joined #openstack-nova | 12:17 | |
| *** pino has joined #openstack-nova | 12:18 | |
| *** vladikr has joined #openstack-nova | 12:23 | |
| *** acormier has quit IRC | 12:29 | |
| *** moshele has quit IRC | 12:29 | |
| *** gouthamr has joined #openstack-nova | 12:30 | |
| *** yangyapeng has joined #openstack-nova | 12:34 | |
| openstackgerrit | Yikun Jiang proposed openstack/nova master: Update Instance action's updated_at when action event updated. https://review.openstack.org/507473 | 12:35 |
| *** edand has joined #openstack-nova | 12:35 | |
| *** gouthamr_ has joined #openstack-nova | 12:36 | |
| *** pchavva has joined #openstack-nova | 12:37 | |
| *** yangyapeng has quit IRC | 12:38 | |
| *** gouthamr has quit IRC | 12:38 | |
| *** pino has quit IRC | 12:39 | |
| *** manasm has quit IRC | 12:40 | |
| gmann | alex_xu: oh, i also do not have much for this week. we can cancel the meeting | 12:42 |
| gmann | alex_xu: take rest and get well soon | 12:42 |
| *** trinaths has left #openstack-nova | 12:43 | |
| *** gcb has quit IRC | 12:45 | |
| *** jmlowe has quit IRC | 12:48 | |
| tssurya | Hi, I was just playing around with cell_v2 and came across the nova-manage cell_v2 map_instances command; so I went through the source code (manage.py) but was not able to find a connection as to which sql_connection string is used to query the 'instances' table in the nova DB. It should be taking this from the conf file ? But could someone point me to the section of code that does this ? | 12:53 |
| *** alexchadin has quit IRC | 12:55 | |
| *** mingyu has quit IRC | 12:55 | |
| *** alexchadin has joined #openstack-nova | 12:56 | |
| *** jpena|lunch is now known as jpena | 12:57 | |
| *** mingyu has joined #openstack-nova | 12:57 | |
| *** takashin has joined #openstack-nova | 12:58 | |
| *** esberglu has joined #openstack-nova | 12:58 | |
| sdague | tssurya: it's coming from the same nova.conf that your API server is expected to be using | 13:00 |
| *** ratailor has quit IRC | 13:01 | |
| sdague | tssurya: all nova-manage commands are assumed to be run from the shell on the API server (and have the same db access) | 13:01 |
| *** eharney has joined #openstack-nova | 13:01 | |
| openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep. https://review.openstack.org/507848 | 13:03 |
| *** liverpooler has joined #openstack-nova | 13:03 | |
| tssurya | sdague : thanks for the reply, so here is my situation : I have some instances lying in my nova_cell1 DB which need to be mapped to my new cell (cell1). however the db access URL for this is inside nova_cell1.conf ? so when I run the map instances command, they do not get mapped. | 13:05 |
| *** smatzek has quit IRC | 13:05 | |
| tssurya | sdague: I think its because the queries are sent to the nova_cell0 DB since nova.conf has that access url. | 13:06 |
| efried | cdent Yo | 13:07 |
| openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep https://review.openstack.org/507848 | 13:07 |
| cdent | efried: yo | 13:08 |
| *** mriedem has joined #openstack-nova | 13:09 | |
| cdent | efried: my question is basically: in powervm does the thing which is the actual hypervisor ever host workloads that are not managed by nova. The comments on this bug for context: https://bugs.launchpad.net/nova/+bug/1718212 | 13:09 |
| openstack | Launchpad bug 1718212 in OpenStack Compute (nova) "Compute resource tracker does not report correct information for drivers such as vSphere" [Medium,In progress] - Assigned to Radoslav Gerganov (rgerganov) | 13:09 |
| cdent | since placement is authoritative for dynamic workloads if nova is not doing all the managing, there may be need for additional things talking to placement | 13:10 |
| efried | cdent That's a multi-pronged question. | 13:10 |
| cdent | It’s a full on dinner fork | 13:10 |
| openstackgerrit | Sean Dague proposed openstack/nova master: Support qemu >= 2.10 https://review.openstack.org/505673 | 13:11 |
| efried | First, nothing is stopping the user from creating VMs outside the auspices of OpenStack. | 13:11 |
| *** jistr is now known as jistr|call | 13:12 | |
| efried | Second, there's a mode (the most common one, historically) where the I/O virtualization is done by one (or two, cause redundancy/HA) separate partitions, in which case those guys are consuming resources outside of Nova's purview. | 13:12 |
| efried | cdent Now I'll look at the bug... | 13:13 |
| bauzas | efried: creating VMs by the hypervisor directly is not supported by Nova | 13:13 |
| *** pino has joined #openstack-nova | 13:13 | |
| *** moshele has joined #openstack-nova | 13:13 | |
| *** krtaylor has joined #openstack-nova | 13:13 | |
| efried | bauzas Oh, Nova won't pick them up, for sure. But there's nothing stopping the user from doing it. | 13:14 |
| cdent | efried: So, short answer is “yes” so second question is “Do you yet have a plan on how to manage it” | 13:14 |
| *** lucasxu has joined #openstack-nova | 13:14 | |
| bauzas | efried: if the operator does that, then any bug related to that should be Wontfix | 13:14 |
| *** lyan has joined #openstack-nova | 13:14 | |
| efried | bauzas Well... | 13:14 |
| efried | I agree in principle. | 13:14 |
| bauzas | efried: lemme find where we say that we don't support direct hypervisor calls | 13:14 |
| efried | But that doesn't mean we won't try to accomodate out-of-band partitions when reporting inventory/usage. | 13:15 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Log consumer uuid when retrying claims in the scheduler https://review.openstack.org/507705 | 13:15 |
| efried | bauzas Sure, but I totally believe you. | 13:15 |
| openstackgerrit | Andrey Volkov proposed openstack/nova master: [WIP] List instances performace optimization https://review.openstack.org/507854 | 13:16 |
| *** MVenesio has joined #openstack-nova | 13:16 | |
| bauzas | efried: I didn't found any explanations either in https://docs.openstack.org/nova/latest/contributor/policies.html or https://docs.openstack.org/nova/latest/contributor/project-scope.html | 13:17 |
| bauzas | maybe we should say that | 13:17 |
| efried | cdent Okay, in general, we have very similar issues as described in comment #6 in that bug. | 13:17 |
| *** chyka has joined #openstack-nova | 13:18 | |
| efried | When I started looking into converting over to get_inventory, it was going to be a matter of reporting "reserved" amounts based on whatever's going on OOB. | 13:18 |
| johnthetubaguy | I thought we were heading down the not doing live updates of resource usage? | 13:18 |
| cdent | efried: do you have a convenient way of distinguishing between nova managed and not-nova managed stuff? | 13:18 |
| efried | ...and it's not trivial to figure out what's OOB. | 13:19 |
| efried | (I was just about to say :) | 13:19 |
| cdent | jinx! | 13:19 |
| efried | I can know off the bat to account for my Novalink partition (the node on which the compute service runs) and my Virtual I/O Servers. | 13:19 |
| cdent | johnthetubaguy: that seems to work okay for libvirt, but not so great otherwise | 13:19 |
| efried | But any plain ol' worker bees I have to figure out whether they came from Nova. | 13:20 |
| johnthetubaguy | cdent: I guess I am not seeing why, is that vmware bug the example? | 13:20 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Migrate from 'ip' commands to 'pyroute2' https://review.openstack.org/484386 | 13:20 |
| cdent | johnthetubaguy: at this stage efried, rgerganov and I are just sort of having a chat, not making any decisions | 13:20 |
| bhagyashris | johnthetubaguy, mriedem: Hi, | 13:20 |
| efried | Simplest would be, every time get_inventory/get_available_resource is called, I ask Nova for the full list of instances it knows about, ask my hypervisor for the instances IT knows about, and do a set-diff. | 13:21 |
| johnthetubaguy | I am just curious where the problem isn't nova driver expected resource usage | 13:21 |
| cdent | johnthetubaguy: that’s a specific case of a general issue of “things other than the nova-compute node using the stuff that nova-compute is also using” | 13:21 |
| efried | just so. | 13:21 |
| johnthetubaguy | I think I need a more concrete example | 13:21 |
| mriedem | cdent: i think the vcenter vms have some nova metadata associated with them, so you can tell which ones are nova-managed and which were created oob | 13:22 |
| efried | johnthetubaguy You try to spawn an instance and ask for 3 VCPU. Resource tracker reports you've got 3 VCPU, so the claim passes. | 13:22 |
| efried | johnthetubaguy But in fact, one VCPU is being consumed by a VM that was spawned out of band | 13:22 |
| rgerganov | mriedem, that may work for vcpu and memory but it won't work well for storage | 13:22 |
| *** chyka has quit IRC | 13:22 | |
| johnthetubaguy | its the out of band VMs, I thought we explicitly didn't support that | 13:22 |
| efried | johnthetubaguy Yeah, bauzas said the same, but didn't find where that's documented. | 13:23 |
| efried | Not saying that means it's supported, or that it should be :) | 13:23 |
| bauzas | so, Nova isn't a proxy layer for hypervisors | 13:23 |
| johnthetubaguy | yeah, I could have swarn it was in here, but I don't see it: https://docs.openstack.org/nova/latest/contributor/project-scope.html | 13:23 |
| bauzas | johnthetubaguy: yeah, I verified that | 13:24 |
| bauzas | lemme provide a change for it | 13:24 |
| johnthetubaguy | well, its a bit late I guess | 13:24 |
| cdent | So, even if the statement is “we don’t do that” the problem still holds for shared storage | 13:25 |
| efried | right | 13:25 |
| mriedem | nova doesn't import existing vms on the hypervisor, | 13:25 |
| mriedem | but that doesn't mean we don't try to adjust inventory based on things running on the hypervisor host | 13:25 |
| cdent | where “the problem” is the generic notion of mixed accounting | 13:25 |
| efried | mriedem Which actually makes it more problematic. | 13:25 |
| efried | right. | 13:25 |
| johnthetubaguy | cdent: yeah, that sounds like a real thing we have to support | 13:25 |
| mriedem | that's why we have reserved | 13:25 |
| efried | right | 13:25 |
| mriedem | even libvirt hosts have to account for things like ovs running on the same host | 13:26 |
| cdent | Right, so one of the underlying questions is: | 13:26 |
| cdent | Do we intend/expect that reserved will by dynamically adjuted, frequently | 13:26 |
| *** baoli has joined #openstack-nova | 13:26 | |
| cdent | Or in the cases where we want it to be dynamically adjusted we should instead make allocations, via some third party? | 13:26 |
| efried | Right, back to what johnthetubaguy mentioned earlier: "not doing live updates of resource usage" | 13:26 |
| efried | That is, will get_inventory() eventually be a thing that's run only once, rather than on a periodic? | 13:27 |
| cdent | exactly | 13:27 |
| bhagyashris | johnthetubaguy, mriedem: Could you please review patch: https://review.openstack.org/#/c/409644/ ? Addressed all review comments. Thank you :) | 13:27 |
| *** lbragstad has joined #openstack-nova | 13:27 | |
| mriedem | i don't think it will no | 13:27 |
| bauzas | I'm not seeing the reserved bit to be that dynamicv | 13:27 |
| mriedem | the operator can adjust reserved space dynamically | 13:27 |
| johnthetubaguy | cdent: I separate allocations are easier to update than a single reserved value | 13:27 |
| efried | I've heard tell (possibly from jay) that it's a no-no to update the *total* amounts. | 13:27 |
| mriedem | we talked about this in boston | 13:27 |
| *** sbezverk has joined #openstack-nova | 13:27 | |
| bauzas | but yeah, the operator can just set or raise the reserved bit when they want | 13:27 |
| mriedem | about how we don't account for overhead, and operators would have to handle that by toggling reserved values | 13:27 |
| cdent | mriedem: yes, but the new ingredient here is the dynamism | 13:28 |
| cdent | maybe | 13:28 |
| bauzas | mriedem: do you think we should clearly state that in https://docs.openstack.org/nova/latest/contributor/project-scope.html ? | 13:28 |
| efried | ...unless, like, they hot-plug a whole new disk, or jack up the disk size on the SAN or whatever. In that case, we should be able to bump the totals, yes? | 13:28 |
| bauzas | the fact that Nova doesn't support direct hypervisor VMs, but can leave room for them | 13:28 |
| cdent | efried: yes | 13:28 |
| efried | Same actually applies to CPUs | 13:28 |
| efried | Not sure if more primitive hypervisors have this, but there's a dynamic entitlement thing where you can unlock previously-unavailable CPUs on the fly. | 13:29 |
| *** mingyu has quit IRC | 13:29 | |
| johnthetubaguy | bhagyashris: there was a -1 review on that patch with no answer when I last looked, so I skipped looking any deeper, did you respond to their questions yet? | 13:29 |
| *** josecastroleon has joined #openstack-nova | 13:30 | |
| johnthetubaguy | mriedem: it totally feels like the toggling reserved values ticks the 80% case, with very few surpizes | 13:30 |
| *** takashin has quit IRC | 13:30 | |
| cdent | efried, mriedem, bauzas, johnthetubaguy, rgerganov: it feels like we probably have the tooling to make something workable for this, but as we figure it out it would be good to document some kind of (hate this phrase) best practice, | 13:31 |
| efried | Me, I'm concerned about the penalty of getting the full list of instances every time get_inventory is run. | 13:31 |
| johnthetubaguy | its feels like making the 80% case work really well should be step 1? And some rough best practices for the other bits for now, that could work. | 13:32 |
| cdent | efried: instances from the hypervisor’s point of view, nova’s point of view or placement’s point of view? or some combination? That is, which one worries you most? | 13:33 |
| johnthetubaguy | honestly doing both of those worry me | 13:33 |
| efried | Yeah, I have to do both. | 13:33 |
| johnthetubaguy | I like the system doing nothing when its idle, in an ideal world of course. | 13:33 |
| *** sridharg has quit IRC | 13:33 | |
| bauzas | we shouldn't really create a way where people would directly call their VMs if they wish, because that would mean we would silently say we support that | 13:33 |
| efried | Cause the alternative is maintaining some kind of cache and trying to keep it up to date every time a VM gets spawned or deleted (I can get notifications for the OOB ones, and update the in-band ones from spawn/destroy). | 13:34 |
| bhagyashris | johnthetubaguy: Actually I have not respond yet, but Andrey Volkov is given -1 and he suggested that we can fix this issue at schema level (same thing I have proposed in the patch set 3) but after some discussion we decided to fix this in logic and at schema level. | 13:34 |
| bauzas | if the operator doesn't know the dedicated amount their wanna reserve, then either they should disable some compute from the pool, or just set a high value | 13:34 |
| *** gbarros has joined #openstack-nova | 13:34 | |
| *** jmlowe has joined #openstack-nova | 13:34 | |
| johnthetubaguy | but isn't the easy case much easier here? deployer sets the reserved values, no period updates, worry about special cases later? | 13:35 |
| bauzas | providing a way to dynamically adjust resources in Nova would just mean we create things to support | 13:35 |
| cdent | johnthetubaguy, bauzas: perhaps sad, but true, there are in tree hypervisors where oob vms are fairly common. nova in itself may not need to support that, but the hypervisors will likely continue doing it | 13:35 |
| bhagyashris | s/and at schema level/and not at schema level | 13:35 |
| bauzas | johnthetubaguy: that's my point | 13:35 |
| dansmith | mriedem: have you seen the stuff I added to the etherpad last night? | 13:36 |
| bauzas | johnthetubaguy: if operators really want a space for out-of-band VMs, then they have the reserved bit they can set and try to modify when they feel necessary | 13:36 |
| bhagyashris | johnthetubaguy: i will respond to that question. | 13:36 |
| johnthetubaguy | cdent: but even those drivers, there are folks who run dedicated clusters just for OpenStack | 13:36 |
| bauzas | but trying to provide some mechanism for having a dynamic reserved bit seems to me something we shouldn't do | 13:36 |
| bauzas | and honestly, wouldn't it be better to just disable a host and play with it, if that's really needed ? | 13:37 |
| cdent | johnthetubaguy: yes, totally | 13:37 |
| johnthetubaguy | cdent: that was me arguing for making the sync optional, for when you know Nova owns the world | 13:37 |
| * cdent nods | 13:37 | |
| rgerganov | johnthetubaguy, most users runs dedicated clusters but also used shared storage | 13:37 |
| dansmith | johnthetubaguy: are you talking about the periodic that will try to delete unknown vms? | 13:38 |
| mriedem | dansmith: sure did | 13:38 |
| johnthetubaguy | bhagyashris: thanks, responding in the review will help wehn I get back to that | 13:38 |
| mriedem | dansmith: nope, talking about update_available_resource | 13:38 |
| mriedem | and using get_inventory to report reserved space for other things running on the hypervisor outside of nova's control | 13:38 |
| johnthetubaguy | dansmith: well, I was thinking about that early, with all the logging by default, but there I was meaning the resource tracker updater | 13:38 |
| dansmith | mriedem: eff that. there's a reserved count for that kind of stuff. | 13:39 |
| bhagyashris | johnthetubaguy: Thank you :) | 13:39 |
| mriedem | dansmith: meaning the config option? | 13:39 |
| *** tetsuro has joined #openstack-nova | 13:39 | |
| dansmith | meaning counting other things and subtracting it ourselves | 13:39 |
| mriedem | i think the config option is what's used to report reserved inventory in the RT today | 13:40 |
| dansmith | yes | 13:40 |
| dansmith | well, actually, I dunno that it is today, but it should be, and that should be your outlet to carve out space | 13:40 |
| *** takashin has joined #openstack-nova | 13:40 | |
| *** cleong has joined #openstack-nova | 13:41 | |
| dansmith | it is | 13:41 |
| dansmith | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L102-L125 | 13:41 |
| dansmith | make those hot-reloadable if you want, but that's the path, IMHO | 13:41 |
| *** mingyu has joined #openstack-nova | 13:42 | |
| efried | dansmith That guy still gets run on a periodic basis, right? I.e. dynamic tweaking of reserved amounts is supported (at least by the driver - not asking if those conf opts are dynamic) | 13:42 |
| johnthetubaguy | we should make SIG_HUP trigger a resource refresh? | 13:43 |
| dansmith | johnthetubaguy: HUP already triggers a config re-read | 13:43 |
| johnthetubaguy | I was meaning a resource tracker update | 13:43 |
| dansmith | johnthetubaguy: so if they were reloadable you'd get a new value the next time inventory runs | 13:44 |
| johnthetubaguy | so we don't have to keep the periodic thing, just do it on events | 13:44 |
| dansmith | I'm not sure there's any reason not to do it perioically | 13:44 |
| johnthetubaguy | system load, all that lists of instances, etc | 13:44 |
| dansmith | it doesn't need to anymore | 13:45 |
| dansmith | in pike, once there aren't any ocata computes around we're not doing healing for instances other than deleted ones | 13:46 |
| johnthetubaguy | I guess I am confused | 13:46 |
| * cdent gives johnthetubaguy a membership card | 13:46 | |
| *** lifeless has quit IRC | 13:46 | |
| johnthetubaguy | I thought we were left with no periodic updates in queens | 13:47 |
| johnthetubaguy | to inventory, etc | 13:47 |
| johnthetubaguy | although... ironic needs those | 13:47 |
| johnthetubaguy | dang it | 13:47 |
| dansmith | I'm saying, we can remove a bunch of the RT stuff for healing instances now I think, | 13:48 |
| *** awaugama has joined #openstack-nova | 13:48 | |
| johnthetubaguy | ah, OK, so maybe I am agreeing with you | 13:48 |
| dansmith | which will make the periodic mostly an inventory mechanism, which doesn't need to be listing instances and such | 13:48 |
| efried | The periodic will use driver.get_inventory() to update inventories? | 13:49 |
| efried | (still) | 13:49 |
| dansmith | or get_available_resource() if it doesn't have that | 13:50 |
| efried | Cool. | 13:50 |
| dansmith | or whatever that method is called | 13:50 |
| *** burt has joined #openstack-nova | 13:50 | |
| efried | So I think the point is that, in order to do dynamic adjustment of 'reserved' amounts properly, get_inventory itself may need to be listing instances and such. | 13:50 |
| dansmith | no | 13:50 |
| dansmith | inventory has nothing to do with instances | 13:51 |
| *** alexchadin has quit IRC | 13:51 | |
| rgerganov | dansmith, there could be no way to calculate reserve without listing instances | 13:51 |
| dansmith | reserved is a conf value | 13:52 |
| bauzas | that's my whole point | 13:52 |
| dansmith | you all were talking about other vms not owned by nova on a hypervisor right? and something external adding that value to reserved so nova doesn't try to use it, right? | 13:52 |
| bauzas | reserved is set by the operator | 13:52 |
| bauzas | so, if the operator wants to raise that value, fair enough | 13:53 |
| *** takashin has quit IRC | 13:53 | |
| dansmith | in that case, some other thing could be updating CONF.reserved_whatever and HUPing nova so that the next time the periodic runs, it just stabs the new scalar value into inventory.. no counting required. | 13:53 |
| bauzas | but I don't see why Nova should care of thaty | 13:53 |
| dansmith | bauzas: most definitely | 13:53 |
| bauzas | dansmith: +1 | 13:53 |
| *** crushil has joined #openstack-nova | 13:53 | |
| *** lifeless has joined #openstack-nova | 13:54 | |
| *** Tom_ has joined #openstack-nova | 13:54 | |
| *** dave-mccowan has quit IRC | 13:54 | |
| mriedem | bauzas: in case you didn't see this https://bugs.launchpad.net/nova/+bug/1719730 | 13:55 |
| openstack | Launchpad bug 1719730 in OpenStack Compute (nova) pike "Reschedule after the late affinity check fails with "'NoneType' object is not iterable"" [High,Confirmed] | 13:55 |
| *** udesale__ has joined #openstack-nova | 13:55 | |
| efried | As user experiences go, "Extend SAN disk, edit conf file, HUP n-cpu" ain't as friendly as "Extend SAN disk". | 13:55 |
| mriedem | bauzas: melwitt might be working on a fix, i'm not sure | 13:55 |
| mriedem | bauzas: but it's a regression in pike | 13:55 |
| bauzas | mriedem: yup, I just provided my thoughts litterally 5 mins ago :p | 13:55 |
| bauzas | mriedem: tl;dr I don't understand how we can get a ReqSpec group info that isn't accurate | 13:55 |
| rgerganov | efried +1 | 13:55 |
| dansmith | efried: nova wouldn't be reporting the inventory for a SAN disk, so no problem there :) | 13:56 |
| dansmith | efried: and anything that an agent is surveying would be updated automatically | 13:56 |
| *** udesale has quit IRC | 13:56 | |
| dansmith | efried: if you hotplug some memory into your compute node, then nova-compute would automatically report that on the next periodic | 13:56 |
| dansmith | efried: what you are talking about is updating the reserved value, which is static, operator-configured or operator-scripted | 13:57 |
| mriedem | bauzas: provided your thoughts where? don't see anything in the bug or the linked pike change | 13:57 |
| dansmith | efried: the HUP is so we notice the change to the config file.. certainly you don't want us to be live reloading the config file whenever we feel like it | 13:57 |
| bauzas | mriedem: oh snap, duplicate bug | 13:57 |
| *** Tom_ has quit IRC | 13:58 | |
| bauzas | mriedem: https://bugs.launchpad.net/nova/+bug/1719859 | 13:58 |
| openstack | Launchpad bug 1719859 in OpenStack Compute (nova) "Resize failure due to instance group being None in request spec" [Undecided,New] | 13:58 |
| *** hemna_ has joined #openstack-nova | 13:58 | |
| *** Tom_ has joined #openstack-nova | 13:58 | |
| efried | dansmith Right, extending SAN was a bad example. | 13:58 |
| bauzas | mriedem: mmm, nevermind, different issue, it seems | 13:58 |
| * efried is pretty good at coming up with bad examples that get us talking about the wrong part of a problem :( | 13:58 | |
| *** rtjure has quit IRC | 13:58 | |
| *** baoli has quit IRC | 13:58 | |
| efried | Not being super well-versed on how OVS comes into play for libvirt, can we apply the same principle to networking resources? | 13:59 |
| mriedem | bauzas: ok, mel's is pretty straight forward | 13:59 |
| *** baoli has joined #openstack-nova | 13:59 | |
| mriedem | we used to set group_members in the filter_properties, | 13:59 |
| mriedem | your change in pike stopped using filter_properties, | 13:59 |
| efried | I.e. similar UX statement: "Do OVS thingy, edit conf file, HUP n-cpu" not as friendly as "Do OVS thingy" | 13:59 |
| mriedem | but forgot to include the group_members stuff | 14:00 |
| mriedem | she already said that the 1 line change fixes the issue | 14:00 |
| mriedem | i just asked that a functional regression test get created | 14:00 |
| bauzas | mriedem: okay, I'll wait for melwitt's patch | 14:00 |
| dansmith | efried: no, nova-compute isn't reporting any network resources | 14:00 |
| *** ragiman has quit IRC | 14:00 | |
| bauzas | mriedem: thanks for the catch | 14:00 |
| mriedem | bauzas: which reminds me, did you cleanup that num_instances one? | 14:01 |
| bauzas | mriedem: I got some late pings yesterday | 14:01 |
| mriedem | melwitt found the problem, i just helped debug | 14:01 |
| mriedem | https://review.openstack.org/#/c/506093/ | 14:01 |
| dansmith | efried: I think the block you're stumbling over is that nova does not count things it does not count. Therefore, it does not dynamically update inventory for things it does not count. If you want to override the inventory for a thing it _does_ count to account for things it does not count, then you put that in config an HUP it to notice. | 14:01 |
| *** manasm has joined #openstack-nova | 14:01 | |
| bauzas | mriedem: yeah, just needs rebase | 14:01 |
| mriedem | i need to start making an etherpad of bug fixes we need in pike... | 14:01 |
| *** lajoskatona has left #openstack-nova | 14:01 | |
| bauzas | mriedem: I also have a couple of bugfixes that are on hold in my queue | 14:02 |
| bauzas | like the fact we don't destroy the Spec objects | 14:02 |
| bauzas | mriedem: an etherpad seems a good idea to me | 14:02 |
| bauzas | that would also help me focusing on rebasing such bugs | 14:02 |
| mriedem | https://etherpad.openstack.org/p/nova-pike-bug-fix-backports | 14:03 |
| *** rtjure has joined #openstack-nova | 14:03 | |
| bauzas | ack | 14:04 |
| *** coreywright has quit IRC | 14:04 | |
| tssurya | dansmith, mriedem, sdague : so basically I have a simple set up using devstack with one cell, then I added a new 'cell2' that has some unmapped instances in its DB. However when I run the map_instances command specifying the cell uuid, it does not work since like sdague said earlier, the db url is being taken from nova.conf (which is that of cell0) while the db url for the new cell is inside the cell_mappings table of nova_api db. So I | 14:04 |
| *** acormier has joined #openstack-nova | 14:04 | |
| dansmith | tssurya: create a small config with just the db url you need | 14:05 |
| efried | dansmith Thanks, that makes sense. | 14:05 |
| *** Drankis has joined #openstack-nova | 14:06 | |
| tssurya | dansmith : okay! so that means the current working is as expected ? | 14:06 |
| dansmith | tssurya: it's been a while since I looked at that command, but I would guess so | 14:07 |
| mriedem | tssurya: nova.conf in default devstack points at cell0 on purpose | 14:07 |
| mriedem | tssurya: nova-manage uses nova.conf if no other --config-file is specified | 14:08 |
| mriedem | on the command line | 14:08 |
| *** chyka has joined #openstack-nova | 14:08 | |
| mriedem | so if you want to do things to cell1, you would do something like: nova-manage --config-file /etc/nova/nova_cell1.conf db archive_deleted_instances --verbose --until-complete | 14:08 |
| mriedem | for archiving deleted instances in cell1 | 14:08 |
| stephenfin | efried: You didn't start work converting this to a seqdiag, did you? https://photos.google.com/share/AF1QipNpWVQKU8GK4_9wxVbiRJUqJnMzqPcBh6DvjVyBPIjjmi6ZU8r9TleQNo6pV1t9SA?key=NUl3OUFGYkRFTE8tMHhSX0lfc0Y1eEdoeHo4SUhn | 14:08 |
| openstackgerrit | Eric Berglund proposed openstack/nova master: WIP(5): PowerVM driver: ovs vif https://review.openstack.org/422512 | 14:08 |
| stephenfin | I think mriedem asked for it? | 14:09 |
| tssurya | mriedem, dansmith : okay thanks that makes sense | 14:09 |
| dansmith | tssurya: yeah look at the docstring on that method, it says it assumes the config points at the right database | 14:09 |
| mriedem | tssurya: related https://bugs.launchpad.net/nova/+bug/1719487 | 14:09 |
| openstack | Launchpad bug 1719487 in OpenStack Compute (nova) "nova-manage db archive_deleted_rows is not multi-cell aware" [Wishlist,Triaged] - Assigned to Zhenyu Zheng (zhengzhenyu) | 14:09 |
| dansmith | tssurya: you could make it smarter :) | 14:09 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146 | 14:09 |
| mriedem | stephenfin: that's already done | 14:09 |
| *** sree has quit IRC | 14:09 | |
| mriedem | stephenfin: https://docs.openstack.org/nova/latest/reference/live-migration.html | 14:09 |
| efried | stephenfin Yes, merged. | 14:10 |
| mriedem | there is one thing missing | 14:10 |
| mriedem | i noticed yesterday | 14:10 |
| *** yamamoto has quit IRC | 14:10 | |
| mriedem | on the failure path, it doesn't have a box saying that the source node is running the _rollback_live_migration method | 14:10 |
| mriedem | which does some stuff on the source node, and calls the dest node to cleanup | 14:10 |
| tssurya | dansmith , mriedem : ok, will have a look at it :) | 14:10 |
| *** felipemonteiro has joined #openstack-nova | 14:10 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187 | 14:11 |
| efried | mriedem That would be like a box to the left of that final 'call' arrow at the bottom? | 14:11 |
| *** sree has joined #openstack-nova | 14:11 | |
| mriedem | efried: i think more like how the post_live_migration box is in the success path on the source node | 14:11 |
| *** felipemonteiro_ has joined #openstack-nova | 14:12 | |
| *** smatzek has joined #openstack-nova | 14:13 | |
| * cdent goes to find some coffee to sit with sdague’s slides | 14:14 | |
| efried | sdague cdent Care to render an opinion/ruling on jichenjc's concerns here: https://review.openstack.org/#/c/488137/21/nova/conf/utils.py@85 ? | 14:14 |
| *** smatzek has quit IRC | 14:14 | |
| *** smatzek has joined #openstack-nova | 14:15 | |
| *** ragiman has joined #openstack-nova | 14:15 | |
| *** yamamoto has joined #openstack-nova | 14:15 | |
| *** felipemonteiro has quit IRC | 14:16 | |
| *** sree has quit IRC | 14:16 | |
| *** smatzek has quit IRC | 14:16 | |
| *** smatzek_ has joined #openstack-nova | 14:16 | |
| cdent | efried: remind me when I’m back in about 20 mins | 14:17 |
| *** cdent has quit IRC | 14:17 | |
| efried | ack, thx | 14:17 |
| *** coreywright has joined #openstack-nova | 14:17 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: _rollback_live_migration in live-migration seqdiag https://review.openstack.org/507871 | 14:17 |
| *** haobing has joined #openstack-nova | 14:18 | |
| efried | mriedem ^ | 14:18 |
| *** mdnadeem has quit IRC | 14:18 | |
| *** alex_xu has quit IRC | 14:18 | |
| mriedem | thanks | 14:19 |
| *** cdent has joined #openstack-nova | 14:19 | |
| mriedem | claudiub|3: hyperv ci seems to have gone crazy | 14:19 |
| mriedem | http://cloudbase-ci.com//nova/507871/1/console.log.gz | 14:19 |
| mriedem | not sure if that's a zuulv3 side effect or what | 14:20 |
| *** yamamoto has quit IRC | 14:20 | |
| *** cdent has quit IRC | 14:20 | |
| *** alex_xu has joined #openstack-nova | 14:21 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: xenapi: pass migrate_data to recover_method if live migrate fails https://review.openstack.org/507874 | 14:22 |
| *** smatzek_ is now known as smatzek | 14:22 | |
| *** haobing has quit IRC | 14:22 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146 | 14:23 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187 | 14:24 |
| *** hemna_ has quit IRC | 14:25 | |
| *** takashin has joined #openstack-nova | 14:26 | |
| *** dave-mccowan has joined #openstack-nova | 14:27 | |
| *** tidwellr has joined #openstack-nova | 14:27 | |
| *** hongbin has joined #openstack-nova | 14:27 | |
| mriedem | gibi: appears that test_live_migrate_delete has introduced a race http://logs.openstack.org/87/507687/2/check/gate-nova-tox-functional-ubuntu-xenial/90cc144/testr_results.html.gz | 14:27 |
| *** jistr|call is now known as jistr | 14:27 | |
| mriedem | i was noticing some stuff like this yesterday when writing another test, | 14:28 |
| mriedem | i noticed that we set migration and instance state before we're actually done cleaning up live migration things, | 14:28 |
| mriedem | so tests that rely on asserting the cleanups can get racy | 14:28 |
| *** takashin has quit IRC | 14:29 | |
| gibi | mriedem: I have to check if we can wait for notification or instance action to avoid the race | 14:30 |
| *** jmlowe has quit IRC | 14:31 | |
| gibi | but I guess we need a bug so I can go and file it if you haven't already filed it | 14:31 |
| mriedem | gibi: https://bugs.launchpad.net/nova/+bug/1719915 | 14:32 |
| openstack | Launchpad bug 1719915 in OpenStack Compute (nova) "test_live_migrate_delete race fail when checking allocations: MismatchError: 2 != 1" [Medium,Confirmed] | 14:32 |
| *** hemna_ has joined #openstack-nova | 14:34 | |
| gibi | mriedem: thanks, I go and dig for a solution | 14:34 |
| gibi | mriedem: btw, I will be mostly unavailable tomorrow and on Friday. Could you do the reporting about the notification subteam meeting on the weekly nova meeting? | 14:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146 | 14:36 |
| mriedem | gibi: sure | 14:37 |
| gibi | mriedem: thanks a lot | 14:37 |
| *** jmlowe has joined #openstack-nova | 14:37 | |
| johnthetubaguy | mriedem: I was looking at the requesting traits in flavors, did we ever talk about requesting the absence of a trait (like request no CPU_FLAG_X available) | 14:38 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187 | 14:38 |
| *** stvnoyes has quit IRC | 14:38 | |
| *** udesale__ has quit IRC | 14:38 | |
| *** dave-mccowan has quit IRC | 14:38 | |
| efried | johnthetubaguy That's a fun thought. Is there a real use case for it? | 14:39 |
| *** josecastroleon has quit IRC | 14:39 | |
| johnthetubaguy | efried: disable the VT CPU flag? | 14:39 |
| johnthetubaguy | or disable hyperthreading | 14:39 |
| dansmith | johnthetubaguy: not that I'm aware of, just required and preferred | 14:39 |
| *** Drankis has quit IRC | 14:40 | |
| efried | johnthetubaguy I have been chastised already to make the distinction between "Ask for a resource that *can* do this thing" and "Ask for a resource and then switch this thing on". | 14:40 |
| johnthetubaguy | I am happy to ignore all that for now | 14:41 |
| johnthetubaguy | need the basic thing first | 14:41 |
| efried | johnthetubaguy (or off). The latter thing definitely needs to be supported - just not via the placement/traits paths, I am led to understand. | 14:41 |
| efried | I can't remember whether it was a review or IRC where jaypipes and I had that talk. Looking... | 14:42 |
| efried | johnthetubaguy Whee: https://review.openstack.org/#/c/497713/4/specs/queens/approved/add-trait-support-in-allocation-candidates.rst@72 | 14:43 |
| *** yassine has quit IRC | 14:43 | |
| johnthetubaguy | cool, thanks, will have a read | 14:43 |
| *** yassine has joined #openstack-nova | 14:43 | |
| *** alex_xu has quit IRC | 14:44 | |
| claudiub|3 | mriedem: cool, ty for the heads up. should be fine now. | 14:44 |
| *** dave-mccowan has joined #openstack-nova | 14:44 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: PCI NUMA Policies https://review.openstack.org/361140 | 14:44 |
| tssurya | dansmith, mriedem : so like the bug here https://bugs.launchpad.net/nova/+bug/1719487 even for the map instances, instead of querying with respect to the config file, we could extract the info from the cell_mappings of API database ? | 14:44 |
| openstack | Launchpad bug 1719487 in OpenStack Compute (nova) "nova-manage db archive_deleted_rows is not multi-cell aware" [Wishlist,Triaged] - Assigned to Zhenyu Zheng (zhengzhenyu) | 14:44 |
| dansmith | tssurya: yeah | 14:45 |
| tssurya | dansmith : so to start working on this I would need a bug ? | 14:47 |
| dansmith | tssurya: it's not really a bug, just an enhancement, but mriedem loves paperwork, so probably easiest to have one | 14:47 |
| *** jmlowe has quit IRC | 14:48 | |
| *** jmlowe has joined #openstack-nova | 14:48 | |
| *** cdent has joined #openstack-nova | 14:48 | |
| tssurya | dansmith : yes :D will open one then | 14:48 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/ocata: Updated from global requirements https://review.openstack.org/490256 | 14:49 |
| *** jaypipes has joined #openstack-nova | 14:49 | |
| *** alex_xu has joined #openstack-nova | 14:49 | |
| *** dave-mcc_ has joined #openstack-nova | 14:49 | |
| *** mnestratov has joined #openstack-nova | 14:50 | |
| *** rmart04 has quit IRC | 14:50 | |
| mriedem | note that was marked as wishlist | 14:51 |
| *** dave-mccowan has quit IRC | 14:51 | |
| mriedem | because yeah it's not really a bug | 14:51 |
| mriedem | johnthetubaguy: idk, | 14:51 |
| mriedem | logic operators on required and preferred traits is not something i'm thinking about | 14:51 |
| *** edand has quit IRC | 14:51 | |
| cdent | not feels inevitable, but when the query parameters for allocation candidates becomes skynet, it’s not my fault | 14:55 |
| tssurya | mriedem : yes I understand its not exactly a bug ; | 14:56 |
| *** baoli has quit IRC | 14:56 | |
| *** baoli has joined #openstack-nova | 14:59 | |
| *** cfriesen_ has joined #openstack-nova | 15:00 | |
| mriedem | i wonder if anyone has actually tried benchmarking and comparing ocata to pike for claims in the scheduler, and with multiple scheduler processes, since we claim that's a safe thing to do now https://docs.openstack.org/releasenotes/nova/pike.html#id2 | 15:00 |
| mriedem | "The FilterScheduler driver now provides allocations to the Placement API, which helps concurrent schedulers to verify resource consumptions directly without waiting for compute services to ask for a reschedule in case of a race condition. That is an important performance improvement that includes allowing one to use more than one scheduler worker if there are capacity concerns. For more details, see the Pike Upgrade Notes fo | 15:01 |
| mriedem | acement." | 15:01 |
| dansmith | mriedem: so along those lines, I noticed that we are pegging the crap out of placement during scheduling 100 instances | 15:01 |
| mriedem | dansmith: red hat has a perf lab right? is ^ something they have talked about? | 15:01 |
| dansmith | even on my fast piece of hardware, placement pegs a CPU | 15:01 |
| dansmith | mriedem: not that I've heard | 15:01 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146 | 15:01 |
| cdent | yeah, we had “do some performance testing” in the weekly rp update for so long that I eventually took it out from apparent lack of interest | 15:02 |
| mriedem | ok. our public cloud guys have made tweaks to the scheduler for performance in mitaka, and lots of those tweaks i've said, "this thing in pike should resolve/replace that" but i don't have hard evidence | 15:02 |
| cdent | It would surprise me not one tiny bit that it is not as performant as expected, because the only testing I’m aware of was done using mostly just a database, and not any of the other bits | 15:02 |
| cdent | and since then the queries have modified quite a bit | 15:02 |
| cdent | and we’ve added more objects | 15:02 |
| *** jmlowe has quit IRC | 15:03 | |
| mriedem | cdent: so i don't think you were around yesterday when we were talking about this, | 15:03 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187 | 15:03 |
| mriedem | but i realized, after several hours, yesterday why i couldn't burst 500 (fake) guest vms on my single node devstack | 15:03 |
| mriedem | and they were all going novalidhost | 15:03 |
| cdent | I had an afternoon in an attorney’s office ... | 15:03 |
| dansmith | also keep in mind that a little slower scheduler performance compares very favorably to 10% retries in the background because we choose bad computes | 15:03 |
| cdent | mriedem: what was the cause? | 15:04 |
| mriedem | dansmith: that's why i'd want to compare ocata to pike | 15:04 |
| mriedem | cdent: the ultimate cause was a 409 response from placement when putting the allocations during scheduling | 15:04 |
| mriedem | we retry that 3 times, | 15:04 |
| dansmith | mriedem: yeah, I'm just saying you have to consider "time to all active" not just "time to building" or something | 15:04 |
| mriedem | but it wasn't enough apparently | 15:04 |
| mriedem | dansmith: agree | 15:04 |
| *** andreas_s_ has joined #openstack-nova | 15:04 | |
| cdent | the 409 was for generation mismatch? | 15:04 |
| *** jmlowe has joined #openstack-nova | 15:04 | |
| mriedem | dansmith: with all the spinning plates, i've been thinking about starting a perf scenario etherpad, polish that up and then send out to see if people can help | 15:04 |
| *** gyee has joined #openstack-nova | 15:05 | |
| dansmith | cdent: it's allocation, so it's the internal rp generation conflict I think | 15:05 |
| mriedem | cdent: this is the response, "There was a conflict when trying to complete your request.\n\n Inventory changed while attempting to allocate: Another thread concurrently updated the data. Please retry your update" | 15:05 |
| dansmith | cdent: and I'm not sure why placement isn't just retrying that for us | 15:05 |
| cdent | https://github.com/openstack/nova/blob/master/nova/objects/resource_provider.py#L1837-L1839 | 15:05 |
| mriedem | i didn't see any actual inventory updates from the virt driver, which shouldn't happen since the inventory in this case is static | 15:05 |
| dansmith | ah sweet | 15:05 |
| mriedem | so the message was a bit misleading | 15:05 |
| cdent | dansmith: yeah, that’s what I meant by generation mismatch | 15:05 |
| dansmith | cdent: yeah, I know where it's happening, but hadn't seen that TODO from jaypipes | 15:06 |
| dansmith | so that's cool | 15:06 |
| dansmith | cdent: I'm not sure why we're hitting it though, | 15:06 |
| dansmith | cdent: since it's a single thread of allocating for things, with no inventory updates coming from the compute | 15:06 |
| *** ragiman has quit IRC | 15:06 | |
| dansmith | so I'm not sure what is racing | 15:06 |
| cdent | thinking out loud: every time we write an allocation we update the generation | 15:07 |
| *** andreas_s has quit IRC | 15:07 | |
| dansmith | right | 15:07 |
| cdent | and we compare the generation with what the generation was before we entered the transaction | 15:07 |
| cdent | so we race to get the transaction | 15:07 |
| dansmith | and we conflict if something else changes the generation while we're trying to | 15:07 |
| gibi | mriedem: I don't think we saw a real race on master see my comment in the bughttps://bugs.launchpad.net/nova/+bug/1719915/comments/1 | 15:07 |
| openstack | Launchpad bug 1719915 in OpenStack Compute (nova) "test_live_migrate_delete race fail when checking allocations: MismatchError: 2 != 1" [Medium,Confirmed] - Assigned to Balazs Gibizer (balazs-gibizer) | 15:07 |
| cdent | we create an rp object for each allocation at the http layer | 15:08 |
| cdent | that’s the generation that’s being used | 15:08 |
| * cdent confirms that guess | 15:08 | |
| *** yangyapeng has joined #openstack-nova | 15:08 | |
| mriedem | gibi: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%5Bnova.api.openstack.requestlog%5D%20127.0.0.1%20%5C%5C%5C%22DELETE%20%2Fv2.1%2F%5C%22%20AND%20message%3A%5C%22%2Fmigrations%2F1%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | 15:09 |
| cdent | yeah | 15:09 |
| *** yamamoto has joined #openstack-nova | 15:10 | |
| cdent | most straightforward thing to do, presumably, is to do the TODO, and retry 10 times server side, so the client would be effectively retrying 30 times? | 15:10 |
| *** stvnoyes has joined #openstack-nova | 15:10 | |
| dansmith | cdent: sure, we should be retrying server side, | 15:10 |
| dansmith | cdent: my point is I don't know why we'd be hitting this need to retry with a single thread of allocations | 15:10 |
| cdent | (efried I haven’t got an opinion on that conf/utils.py issue) | 15:11 |
| mriedem | right, we process the instances in a for loop in the scheduler | 15:11 |
| efried | cdent Ack, thanks for looking. | 15:11 |
| mriedem | so we're put'ing the allocations to the same host, but in order | 15:11 |
| mriedem | and the compute shouldn't be changing any inventory since it's static | 15:11 |
| mriedem | i grep'ed the logs for PUT.*inventories and there was nothing | 15:11 |
| cdent | is there anything else putting allocations? | 15:12 |
| dansmith | cdent: no, single 100-instance boot, so one for loop | 15:12 |
| mriedem | would have to audit that, i didn't dig yet | 15:12 |
| mriedem | cdent: like the compute? | 15:12 |
| dansmith | I mean.. "shouldn't be" | 15:12 |
| mriedem | right, nothing else shoudl be | 15:12 |
| mriedem | since we're not doing any moves or anything | 15:12 |
| cdent | yeah, I’m wondering if we left something else somewhere that we forgot about? | 15:12 |
| cdent | I know it’s not supposed to be, but given everything... | 15:13 |
| dansmith | mriedem: remember I suggested to see if the compute was doing ocata fallback behavior for some reason | 15:13 |
| gibi | mriedem: OK, thats a different failure than the what originally was pasted to the bug report. I continue digging... | 15:13 |
| cdent | another possibility is that uwsgi is (somehow, who knows) letting things get out of order | 15:13 |
| mriedem | dansmith: do we log anything specific in that case? | 15:13 |
| mriedem | i see a buttload of the "we're on a pike compute with all pike computes, so not healing allocations" all the time | 15:13 |
| dansmith | mriedem: placement will log it | 15:13 |
| dansmith | mriedem: okay then that probably means it's not | 15:13 |
| mriedem | but ^ is from the periodic | 15:13 |
| mriedem | there are paths in the RT that go into that code w/o consciously passing the has_ocata_computes flag | 15:14 |
| mriedem | but i think it defaults to False anyway | 15:14 |
| *** Oku_OS is now known as Oku_OS-away | 15:14 | |
| cdent | mriedem: have you got a set of logs you can make available? | 15:15 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146 | 15:15 |
| cdent | this doesn’t feel like something it’s going to be easy to reason about without some files to grep | 15:16 |
| dansmith | cdent: it should be pretty easy to reproduce (or not) in a devstack and then you can instrument the code as needed | 15:16 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187 | 15:16 |
| mriedem | cdent: no, it's all local | 15:17 |
| mriedem | well, in this devstack vm which is not local | 15:17 |
| mriedem | but yeah i have the local.conf for the devstack if you want to reproduce | 15:17 |
| cdent | mriedem: sure, but you have tar and such? | 15:17 |
| mriedem | yeah | 15:17 |
| mriedem | is there a standard way to tar up the journald logs? | 15:18 |
| cdent | balls, I forgot about journald, meh | 15:18 |
| mriedem | it's tar'ed up in devstack-gate | 15:18 |
| mriedem | so i can just copy whatever we do in CI | 15:18 |
| * cdent nods | 15:18 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Change 'InstancePCIRequest' spec field https://review.openstack.org/449257 | 15:19 |
| cdent | I can’t really look with any real attention until about 3 hours from now, but if you get a chance to do it, that’s great it will useful, if not, just the local.conf will do | 15:19 |
| mriedem | https://github.com/openstack-infra/devstack-gate/blob/master/functions.sh#L698-L724 | 15:19 |
| *** manasm has quit IRC | 15:20 | |
| *** ragiman has joined #openstack-nova | 15:20 | |
| mriedem | you know, i could just do this with a devstack patch | 15:21 |
| mriedem | that's easier | 15:21 |
| mriedem | let the ci do the work | 15:21 |
| *** josecastroleon has joined #openstack-nova | 15:21 | |
| *** ragiman has quit IRC | 15:24 | |
| mriedem | needless to say, i'm doing a terrible job of reviewing code or specs, or writing specs | 15:25 |
| *** moshele has quit IRC | 15:25 | |
| mriedem | sdague: can you get this stable/pike novaclient bug fix backport? https://review.openstack.org/#/c/495901/ | 15:26 |
| mriedem | pretty nasty and we need to release it | 15:26 |
| sdague | mriedem: looking | 15:26 |
| *** gouthamr_ has quit IRC | 15:26 | |
| sdague | +A | 15:26 |
| *** lpetrut has joined #openstack-nova | 15:27 | |
| dansmith | mriedem: speaking of reviewing, I'm not sure what else to do on the base switchover patch since the difference is not measurable on my box. I'm poking at the fault thing in the later patch, but we can just strip that out and work on it in parallel | 15:27 |
| mriedem | i haven't reviewed the actual change yet | 15:27 |
| mriedem | was just getting 1000 active vms locally to test the fault thing we talked about last night | 15:28 |
| mriedem | i agree that what we found last night, for numbers, isn't worth holding things up | 15:28 |
| mriedem | sdague: thanks | 15:28 |
| dansmith | mriedem: okay, like I said in the etherpad, I don't think we're doing the fault thing in that patch yet | 15:28 |
| dansmith | and applying the one that does it definitely has an impact | 15:28 |
| mriedem | negative impact? | 15:29 |
| dansmith | yeah | 15:29 |
| *** manasm has joined #openstack-nova | 15:30 | |
| mriedem | because we've added a new unconditional join i suppose | 15:30 |
| *** Sukhdev has joined #openstack-nova | 15:30 | |
| dansmith | it's not a new join, but it's a new query yeah | 15:30 |
| mriedem | yeah, was just thinking that | 15:30 |
| dansmith | I'm messing with the later patch though to see if I can squash it out | 15:30 |
| mriedem | like you said, we could plumb that in the db api | 15:30 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/newton: Updated from global requirements https://review.openstack.org/373293 | 15:30 |
| dansmith | that's what I'm doing yeah | 15:31 |
| mriedem | if faults is in expected_attrs, get the instances in deleted/error and include the faults on those? | 15:31 |
| dansmith | yes | 15:32 |
| *** jmlowe has quit IRC | 15:32 | |
| *** dtantsur is now known as dtantsur|afk | 15:35 | |
| cdent | do dansmith and mriedem have an exciting new etherpad they are willing to share? | 15:36 |
| dansmith | cdent: https://etherpad.openstack.org/p/nova-instance-list | 15:37 |
| dansmith | cdent: but that's just incidental to him finding the problem | 15:37 |
| dansmith | it's not about the thing we were describing | 15:37 |
| * cdent nods | 15:37 | |
| cdent | you know I’m a total junkie for all the info | 15:37 |
| mriedem | started as getting a benchmark for testing before and after the big instance list change | 15:37 |
| mriedem | but have also added todos for weird things we've seen, like the concurrent update detected thing with 500 instances | 15:38 |
| *** andreas_s_ has quit IRC | 15:38 | |
| *** psachin has quit IRC | 15:40 | |
| mriedem | sdague: can i put [[post-config|$NOVA_CONF]] in stackrc? | 15:42 |
| cdent | mriedem, dansmith, (and sean mooney): If any of you get a chance to look at the discussion on https://review.openstack.org/#/c/504540/ (limiting GET /allocation_candidates ) it’s gotten to the point where we are trying to decide what it is that we are actually optimizing for, so could do with more input | 15:42 |
| sdague | mriedem: in local.conf | 15:43 |
| *** rcernin has quit IRC | 15:43 | |
| *** crushil_ has joined #openstack-nova | 15:43 | |
| mriedem | sdague: yeah, locally, but this is for running something through ci | 15:43 |
| *** josecastroleon has quit IRC | 15:44 | |
| sdague | there isn't really anyway to jam it into stackrc, you need to do project-config changes for stuff like that | 15:44 |
| sdague | if this is just for hacktastic stuff, just iniset whatever you want in lib/nova | 15:45 |
| mriedem | ok, just checking, i can hack this other ways | 15:45 |
| mriedem | yeah that's what i'm doing | 15:45 |
| *** yamahata has quit IRC | 15:45 | |
| *** mnestratov has quit IRC | 15:53 | |
| *** penick has joined #openstack-nova | 15:54 | |
| dansmith | mriedem: so with /servers/details on 2.53, we're pegging conductor real hard. With 2.1 there is no conductor interaction at all | 15:55 |
| dansmith | so we must be doing something stupid in the later microversion | 15:55 |
| dansmith | because we shouldn't be making rpc calls at all | 15:55 |
| dansmith | also, defeating all fault loading and all tag loading doesn't make anything faster | 15:55 |
| *** erlon has joined #openstack-nova | 15:55 | |
| *** xyang1 has joined #openstack-nova | 15:56 | |
| mriedem | how about services? | 15:56 |
| dansmith | services is included in the 2.1 columns so I didn't try | 15:57 |
| cdent | johnthetubaguy: you still want to hold your -2 on https://review.openstack.org/#/c/270116/ ? It’s got a blueprint now | 15:59 |
| mriedem | oh yeah | 16:00 |
| melwitt | mriedem: I'm working on the regression func test for the reschedule bug, FYI | 16:00 |
| dansmith | however, I can stop compute and conductor and still do the list with no failure (and no difference in speed) | 16:00 |
| dansmith | wtaf | 16:00 |
| mriedem | cdent: the spec on that isn't approved | 16:01 |
| *** bswartz has joined #openstack-nova | 16:01 | |
| mriedem | cdent: see my comment from may 26 on that patch | 16:01 |
| *** yamahata has joined #openstack-nova | 16:01 | |
| cdent | mriedem: okay | 16:02 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix race in delete allocation in ServerMovingTests https://review.openstack.org/507911 | 16:03 |
| gibi | mriedem: I've pushed a fix for https://bugs.launchpad.net/nova/+bug/1719915 ^^ | 16:03 |
| openstack | Launchpad bug 1719915 in OpenStack Compute (nova) "test_live_migrate_delete race fail when checking allocations: MismatchError: 2 != 1" [Medium,In progress] - Assigned to Balazs Gibizer (balazs-gibizer) | 16:03 |
| *** sbezverk has quit IRC | 16:03 | |
| mriedem | thanks | 16:04 |
| cdent | mriedem: I shall continue to remind the keepers of the CI | 16:04 |
| *** Apoorva has joined #openstack-nova | 16:04 | |
| *** crushil_ has quit IRC | 16:05 | |
| dansmith | mriedem: 2.26 adds ~3s to my runtime | 16:06 |
| dansmith | mriedem: 2.16 adds about 0.250s | 16:07 |
| cdent | sdague: you’ve comment on the bug report related to https://review.openstack.org/#/c/501359/ , can you comment on the fix when you get a chance. _might_ have backport potential. stephenfin you willing to upgrade your +1? | 16:08 |
| dansmith | mriedem: 2.26 was tags, btw | 16:08 |
| dansmith | mriedem: so this is 3s with tags short-circuited at the db layer, so I think the 3s is all api overhead for empty things | 16:08 |
| stephenfin | cdent: Yup, happy to +2 once someone sdague or dansmith has looked at it (I'm no expert in that area) | 16:09 |
| cdent | thanks stephenfin | 16:09 |
| johnthetubaguy | cdent: its normally dropped when the blueprint is approved, I don't remember how spec-less get approved now | 16:10 |
| cdent | johnthetubaguy: s’okay, matt’s cleared things up: until CI is super happy the blueprint won’t get approved | 16:10 |
| cdent | i hadn’t seen his comment in the middle of the stack | 16:10 |
| johnthetubaguy | cdent: ah, cool, I should read the scrollback better | 16:10 |
| cdent | and I should read the comments better :) | 16:11 |
| johnthetubaguy | oh yeah, I see now | 16:11 |
| *** edand has joined #openstack-nova | 16:11 | |
| *** smatzek_ has joined #openstack-nova | 16:11 | |
| *** r-daneel has joined #openstack-nova | 16:12 | |
| mriedem | dansmith: cdent: this is my super hack devstack patch to try and recreate the 500 instance burst failure https://review.openstack.org/507918 | 16:13 |
| *** smatzek has quit IRC | 16:15 | |
| *** hemna_ has quit IRC | 16:15 | |
| mriedem | dansmith: ok so we still don't know which microversion is making instance list go back through conductor | 16:16 |
| dansmith | mriedem: I have conductor stopped and nothing else is failing | 16:16 |
| dansmith | which I can't explain | 16:16 |
| mriedem | hmm, api going straight to db somewhere? | 16:17 |
| dansmith | api should be going straight to the db everywhere | 16:17 |
| dansmith | I'm not sure why conductor was doing anything during a list in the first place | 16:18 |
| dansmith | I tried turning it off to see what broke and nothing did | 16:18 |
| mriedem | maybe you hit a window where a periodic was hitting conductor at the same time as you were doing the instance list? | 16:18 |
| dansmith | could be, but I did it a few times | 16:18 |
| dansmith | either way, I'm going to go measure the 2.26 impact with just my change (not the short-circuiting i've done) and on master and see what the diff is | 16:19 |
| mriedem | i'm going to go preheat the oven because it's going to be pot pie time in about an hour | 16:20 |
| *** crushil has quit IRC | 16:23 | |
| *** tesseract has quit IRC | 16:24 | |
| *** shardy has joined #openstack-nova | 16:25 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix race in delete allocation in ServerMovingTests https://review.openstack.org/507911 | 16:26 |
| *** Tom_ has quit IRC | 16:28 | |
| *** lpetrut has quit IRC | 16:28 | |
| *** Tom_ has joined #openstack-nova | 16:29 | |
| *** yamahata has quit IRC | 16:30 | |
| openstackgerrit | Merged openstack/nova master: Fix IoOpsFilter test case class name. https://review.openstack.org/507205 | 16:33 |
| *** Tom_ has quit IRC | 16:34 | |
| openstackgerrit | Merged openstack/nova master: libvirt: bandwidth param should be set in guest migrate https://review.openstack.org/497455 | 16:34 |
| openstackgerrit | Merged openstack/nova stable/ocata: Provide hints when nova-manage db sync fails to sync cell0 https://review.openstack.org/501746 | 16:35 |
| openstackgerrit | Merged openstack/nova master: Ensure errors_out_migration errors out migration https://review.openstack.org/479802 | 16:35 |
| *** jmlowe has joined #openstack-nova | 16:35 | |
| efried | sdague got an opinion on https://review.openstack.org/#/c/488137/21/nova/conf/utils.py@85 ? | 16:35 |
| johnsom | I have an instance booted in nova (master) that nova/neutron shows two plugged ports, but the kernel is not seeing the second network interface. It was hot-plugged with attach. Any pointers for debugging this? | 16:36 |
| johnsom | the qemu process command line (ps -ef) only shows one interface, but I'm not sure if it should show a hot-plugged network interface or not. | 16:37 |
| johnsom | We have been seeing this in our gates off and on during Pike, but I just had it happen local so I can debug, etc. | 16:37 |
| openstackgerrit | Chris Dent proposed openstack/nova master: DNM: Don't monkey patch eventlet in functional https://review.openstack.org/506668 | 16:38 |
| openstackgerrit | Chris Dent proposed openstack/nova master: Do not monkey patch eventlet in unit tests https://review.openstack.org/507923 | 16:38 |
| openstackgerrit | Merged openstack/python-novaclient stable/pike: Allow boot server with multiple nics https://review.openstack.org/495901 | 16:39 |
| *** david-lyle has quit IRC | 16:40 | |
| *** manasm has quit IRC | 16:41 | |
| *** bhagyashris has quit IRC | 16:42 | |
| *** gouthamr has joined #openstack-nova | 16:43 | |
| *** bhagyashris has joined #openstack-nova | 16:43 | |
| *** mvk has quit IRC | 16:44 | |
| *** crushil has joined #openstack-nova | 16:44 | |
| *** hemna_ has joined #openstack-nova | 16:45 | |
| *** gouthamr_ has joined #openstack-nova | 16:45 | |
| johnthetubaguy | johnsom: were there any nova-compute logs about why the attach failed? | 16:46 |
| johnsom | I am looking through and collecting those, n-cpu? | 16:46 |
| johnthetubaguy | yeah, thats the ones I think | 16:47 |
| johnsom | nova show lists it as ACTIVE | 16:47 |
| *** cdent has quit IRC | 16:47 | |
| johnthetubaguy | it wouldn't got to an error state if it failed | 16:48 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Support qemu >= 2.10 https://review.openstack.org/505673 | 16:48 |
| johnthetubaguy | AFAIK | 16:48 |
| johnsom | Yeah, n-cpu doesn't have any ERROR level messages. I can see some vif lines related to the interface, but no ERROR | 16:48 |
| *** gouthamr has quit IRC | 16:49 | |
| johnthetubaguy | its probably a warn | 16:49 |
| johnthetubaguy | do you see the OVS bits wired up? | 16:49 |
| *** derekh has quit IRC | 16:49 | |
| johnsom | Sep 27 08:48:44 devstackpy27-2 nova-compute[21517]: WARNING nova.compute.manager [None req-48a86b9a-bf96-4f0e-bc60-00682c991e35 service nova] [instance: fb013f87-2e20-42d7-950d-bc9add853f2c] Received unexpected event network-vif-plugged-37ea16ee-b9bc-48c8-b23b-1221bece7c9a for instance with vm_state active and task_state None. | 16:49 |
| johnthetubaguy | how did you do the attach? | 16:50 |
| johnthetubaguy | via the Nova API? | 16:50 |
| johnsom | Yes | 16:50 |
| johnthetubaguy | its probably a case of tracing that through the code following the logs, seeing where it failed, I suspect in n-cpu but it could have been earlier | 16:51 |
| johnsom | Sep 27 08:48:44 devstackpy27-2 devstack@n-api.service[21452]: [pid: 21460|app: 0 | 16:52 |
| johnsom | |req: 28/58] 172.21.21.140 () {62 vars in 1337 bytes} [Wed Sep 27 08:48:38 2017] | 16:52 |
| johnsom | POST /compute/v2.1/servers/fb013f87-2e20-42d7-950d-bc9add853f2c/os-interface => | 16:52 |
| johnsom | generated 280 bytes in 5420 msecs (HTTP/1.1 200) 9 headers in 359 bytes (1 swit | 16:52 |
| johnsom | ches on core 0) | 16:52 |
| sdague | efried: the list_opts thing is fine | 16:52 |
| *** xyang1 has quit IRC | 16:52 | |
| *** sshwarts has quit IRC | 16:53 | |
| johnsom | Ok, well, I am going to attempt to collect world+dog logs and info to open a bug. Just wanted to ask if there were specific things I should look at while I have a "live" system. | 16:53 |
| *** penick has quit IRC | 16:53 | |
| johnthetubaguy | johnsom: you probably want to trace it to this code:https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5242 | 16:53 |
| johnthetubaguy | johnsom: in the n-cpu logs | 16:54 |
| johnthetubaguy | I was expecting to see this one I think: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5268 | 16:55 |
| johnthetubaguy | but sounds like you hit a different failure | 16:55 |
| *** yamamoto has quit IRC | 16:55 | |
| johnthetubaguy | probably in https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5253 | 16:55 |
| johnthetubaguy | looks like an exception in there would not get logged properly | 16:56 |
| *** penick has joined #openstack-nova | 16:57 | |
| johnthetubaguy | johnsom: sadly that means you need to look through where it got in here: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L849 | 16:57 |
| johnthetubaguy | johnsom: best of luck! | 16:57 |
| johnsom | Thanks! | 16:58 |
| *** gszasz has quit IRC | 16:58 | |
| *** yamamoto has joined #openstack-nova | 16:59 | |
| *** smatzek_ is now known as smatzek | 17:00 | |
| johnsom | Sep 27 08:48:39 devstackpy27-2 nova-compute[21517]: DEBUG nova.network.neutronv2.api [None req-cfaf7680-4a74-45a7-9dc8-fd793b93fde5 admin admin] [instance: fb013f87-2e20-42d7-950d-bc9add853f2c] Successfully updated port: 37ea16ee-b9bc-48c8-b23b-1221bece7c9a {{(pid=21517) _update_port /opt/stack/nova/nova/network/neutronv2/api.py:448}} | 17:00 |
| *** sree has joined #openstack-nova | 17:00 | |
| johnsom | Yeah, this is going to take some time. | 17:01 |
| *** penick_ has joined #openstack-nova | 17:01 | |
| *** Apoorva_ has joined #openstack-nova | 17:03 | |
| *** penick has quit IRC | 17:03 | |
| *** pino has quit IRC | 17:04 | |
| *** jpena is now known as jpena|off | 17:05 | |
| *** Apoorva has quit IRC | 17:06 | |
| *** sree has quit IRC | 17:06 | |
| *** tbachman has quit IRC | 17:06 | |
| *** dave-mcc_ has quit IRC | 17:06 | |
| *** cdent has joined #openstack-nova | 17:09 | |
| *** dave-mccowan has joined #openstack-nova | 17:11 | |
| johnthetubaguy | dansmith: traits for drivers that are not ironic, is it the driver that is meant to be reporting them upwards, or is that config, or both? | 17:12 |
| johnthetubaguy | I guess I was meaning that as a more general question really | 17:13 |
| *** david-lyle has joined #openstack-nova | 17:13 | |
| *** pcaruana has quit IRC | 17:14 | |
| *** itlinux has joined #openstack-nova | 17:17 | |
| dansmith | johnthetubaguy: at some point I think it'll be a little of both | 17:25 |
| mriedem | could be an external service | 17:25 |
| dansmith | johnthetubaguy: some things the compute manager probably adds to a list of virty things that the driver exposes | 17:25 |
| mriedem | this reminds me, i was going to put something in our "nova is not a metrics gatherer" policy doc about this | 17:25 |
| mriedem | because of the thing at the ptg where intel wanted nova-compute reporting some crazy cpu traits | 17:25 |
| johnthetubaguy | its just for ironic it feels like the virt driver pushes those up from iroinc | 17:25 |
| mriedem | i think ideally we don't want the ironic driver being a proxy to placement for this stuff | 17:26 |
| johnthetubaguy | the problem is when an admin deletes a trait in ironic, how do we know to delete it in placement | 17:26 |
| dansmith | johnthetubaguy: ironic could do it itself for sure | 17:26 |
| dansmith | johnthetubaguy: for libvirt it'd be the virt driver | 17:26 |
| johnthetubaguy | the problem is the nova creates the resource provider right now, using the compute node name, hashring details, etc | 17:26 |
| dansmith | johnthetubaguy: no, the rp uuid is the ironic node uuid | 17:26 |
| dansmith | johnthetubaguy: nova creates it if it's not there already, ironic could have done it | 17:27 |
| johnthetubaguy | hmm, I thought it had both for some reason, I need to trace that all properly so its clear in my head | 17:28 |
| johnthetubaguy | so I thought we said at the PTG the ironic virt driver would push this all up, but I am not totally against ironic doing that | 17:28 |
| *** Tengu has joined #openstack-nova | 17:29 | |
| Tengu | hello! | 17:29 |
| *** shardy has quit IRC | 17:29 | |
| Tengu | I'm having some issues setting up host aggregation and flavor matching (i.e. "flavor m1.medium shall start only on that aggregate" | 17:29 |
| *** gjayavelu has joined #openstack-nova | 17:30 | |
| *** yamamoto has quit IRC | 17:31 | |
| *** yamamoto has joined #openstack-nova | 17:31 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146 | 17:32 |
| *** markmc has quit IRC | 17:32 | |
| *** markmc has joined #openstack-nova | 17:33 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187 | 17:34 |
| openstackgerrit | melanie witt proposed openstack/nova master: Set group_members when converting to legacy request spec https://review.openstack.org/507938 | 17:34 |
| melwitt | mriedem: ^ I wrote that test by working from nova/tests/functional/regressions/test_bug_1671648.py and just now realized I guess I could have just added an instance group to the existing test to also test this. but maybe it's better to have the tests separated | 17:38 |
| *** vvargaszte has joined #openstack-nova | 17:39 | |
| melwitt | food for thought | 17:39 |
| mriedem | dansmith: L135 https://etherpad.openstack.org/p/nova-instance-list are my results for 1000 active instances with your change | 17:40 |
| mriedem | i'm pretty surprised at the improvements there | 17:40 |
| cdent | mriedem: which job results on https://review.openstack.org/#/c/507918/ are my best target for pokage? | 17:42 |
| dansmith | mriedem: hmm | 17:42 |
| dansmith | mriedem: if you roll back to the other patch does it go back to the perf you measured before? | 17:43 |
| dansmith | mriedem: with my patch we iterate the list fewer times | 17:43 |
| *** vvargaszte has quit IRC | 17:43 | |
| dansmith | I'd be surprised if it made that much difference, but it should make some | 17:44 |
| mriedem | can try that in a bit | 17:44 |
| dansmith | my microversion survey results are interesting | 17:44 |
| dansmith | and not good | 17:44 |
| dansmith | will be done in a few minutes | 17:44 |
| mriedem | cdent: i'd think just the normal tempest dsvm job http://logs.openstack.org/18/507918/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d0a5723/ | 17:45 |
| cdent | roger that | 17:45 |
| mriedem | lots of copied bash in here so i likely screwed something up | 17:45 |
| mriedem | hmm, didn't even get to my stuff | 17:46 |
| *** tbachman has joined #openstack-nova | 17:47 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback https://review.openstack.org/507687 | 17:48 |
| *** Swami has joined #openstack-nova | 17:48 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback https://review.openstack.org/507687 | 17:49 |
| dansmith | mriedem: check that out: https://imgur.com/a/2lmiw | 17:50 |
| dansmith | sdague: you too ^ | 17:50 |
| mriedem | jesus, graphs?! | 17:50 |
| mriedem | well, looking at 2.46 and 2.47, i think it's 2.47 https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id41 | 17:51 |
| dansmith | 2.26 was not free but 2.47 is killing us | 17:51 |
| dansmith | yeah | 17:51 |
| mriedem | this is 500 error/500 active right? | 17:51 |
| mriedem | dansmith: want to report a bug with details? | 17:51 |
| dansmith | mriedem: yes, 500/500 | 17:51 |
| *** itlinux has quit IRC | 17:51 | |
| *** sambetts is now known as sambetts|afk | 17:52 | |
| dansmith | so that was on my patch | 17:53 |
| dansmith | I'm going to run it again on master but starting at maybe 2.24 to make sure the knees are in the same place | 17:53 |
| mriedem | ok | 17:53 |
| sdague | dansmith: interesting.... any idea why the display on the embedded data is causing things to go nuts | 17:53 |
| mriedem | i did notice this w/o your change too though | 17:53 |
| mriedem | i wonder if we're lazy-loading the flavor extra specs? | 17:53 |
| dansmith | mriedem: yeah, I don't see differences in my numbers so I'm just doing it for completeness | 17:53 |
| dansmith | sdague: I haven't looked yet | 17:54 |
| sdague | mriedem: ah, right, that's probably it | 17:54 |
| dansmith | we shouldn't be lazy-loading extra specs | 17:54 |
| dansmith | they should be in the flavor in the instance | 17:54 |
| sdague | I mean, it is a bunch more data. I guess it could just be serialization cost of more data, though it seems weird. | 17:55 |
| mriedem | well, we were always getting the flavor | 17:56 |
| dansmith | mriedem: https://bugs.launchpad.net/nova/+bug/1719966 | 17:56 |
| openstack | Launchpad bug 1719966 in OpenStack Compute (nova) "Microversion 2.47 punches nova in its special place" [Undecided,New] | 17:56 |
| mriedem | even before 2.27 | 17:56 |
| mriedem | ha | 17:56 |
| dansmith | sdague: right, no difference in what we're pulling from the db across that boundary, just what we do with it in the api | 17:56 |
| dansmith | sdague: (I checked) | 17:57 |
| mriedem | https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L269 | 17:57 |
| *** cfriesen_ has quit IRC | 17:57 | |
| mriedem | so we were always pulling it https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L263 | 17:57 |
| mriedem | and we were always joining on it in the db https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L58 | 17:58 |
| mriedem | so why is this so much slower? https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L248 | 17:58 |
| mriedem | the policy check for each instance? | 17:58 |
| *** penick has joined #openstack-nova | 17:58 | |
| dansmith | the only thing we can lazy-load from flavor isprojects, BTW | 17:59 |
| dansmith | not extra_specs or anything else | 17:59 |
| dansmith | https://github.com/openstack/nova/blob/master/nova/objects/flavor.py#L318-L319 | 18:00 |
| mriedem | ok i thought that we always had extra_specs but didn't go back to check | 18:00 |
| sdague | mriedem: yeh, the policy check is going to be per instance | 18:00 |
| sdague | the policy check is a fs.stat as well | 18:00 |
| dansmith | we should check it once and pass it to the per-instance flavor method right? | 18:00 |
| mriedem | yes | 18:00 |
| *** vvargaszte has joined #openstack-nova | 18:00 | |
| sdague | because policy file is dynamically reread | 18:00 |
| mriedem | right | 18:00 |
| mriedem | ... | 18:00 |
| mriedem | jesus | 18:00 |
| dansmith | that might explain why mriedem sees a bigger hit | 18:00 |
| dansmith | you know what | 18:01 |
| mriedem | where is cfriesen when it's time to talk about performance degradation? | 18:01 |
| dansmith | I think we might want to backport this fix | 18:01 |
| mriedem | we for sure do | 18:01 |
| dansmith | I mean.. maybe | 18:01 |
| *** penick_ has quit IRC | 18:01 | |
| mriedem | the policy thing is backportable | 18:01 |
| mriedem | check once | 18:01 |
| dansmith | we could leave it and just further relegate pike to the trashcan of releases | 18:01 |
| mriedem | ha | 18:01 |
| mriedem | but, | 18:02 |
| mriedem | ocata is already in that trashcan | 18:02 |
| dansmith | haha | 18:02 |
| mriedem | i've literally been sending emails internally for weeks saying, "once you upgrade to pike, this should all be much better" | 18:02 |
| mriedem | should* | 18:02 |
| mriedem | *: not actual statement of fact backed up by any evidence | 18:02 |
| *** cfriesen_ has joined #openstack-nova | 18:03 | |
| melwitt | heh | 18:03 |
| sdague | it would be nice if a context only evaluated a particular policy rule once | 18:05 |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Support traits in the Ironic driver https://review.openstack.org/507052 | 18:05 |
| *** ralonsoh has quit IRC | 18:06 | |
| sdague | because it's going to be a little awkward to handle cases where you need to send down the preevaluted permission through a bunch of function calls | 18:06 |
| mriedem | this one shoudn't be too terrible | 18:07 |
| mriedem | dansmith: are you started on the fix? | 18:07 |
| dansmith | right this one should be easy | 18:07 |
| dansmith | mriedem: no but I can | 18:07 |
| dansmith | I'd rather fix this and you finish reviewing my patch | 18:07 |
| mriedem | which patch? the kahuna? | 18:08 |
| mriedem | i can't finish what i haven't started | 18:08 |
| dansmith | I'd rather fix this and you start reviewing my patch | 18:08 |
| Tengu | hello! anyone can point me a valid doc for pike and host aggregation + flavor pinning? I'm stuck right now trying to get all working, and I find contradictory docs :/ | 18:08 |
| mriedem | define "flavor pinning" | 18:09 |
| mriedem | you can associate a host aggregate with a specific flavor via metadata / extra specs | 18:09 |
| Tengu | mriedem: "m1.small must run on that aggregate, while m2.small must run on this aggregate" | 18:09 |
| mriedem | however, any other aggregate which is not tied to that flavor can still use it | 18:09 |
| mriedem | there is no exclusion | 18:09 |
| mriedem | Tengu: i think you're looking for this then https://review.openstack.org/#/c/381912/ | 18:10 |
| Tengu | what should I put in the metadata? | 18:10 |
| Tengu | ah, will check that | 18:10 |
| Tengu | 3 days ago? darn… pretty fresh | 18:11 |
| mriedem | that spec has been around quite awhile | 18:11 |
| mriedem | almost a year | 18:11 |
| Tengu | we saw something like that for Icehouse | 18:11 |
| mriedem | see L466 here https://etherpad.openstack.org/p/nova-ptg-queens | 18:11 |
| mriedem | apparently the stakeholders were not yet synergized as promised | 18:12 |
| Tengu | ah, that's for queens… we're running pike :/. don't tell me there isn't anything working right now? | 18:12 |
| mriedem | well, read the spec first and confirm if that's what you're asking for | 18:12 |
| Tengu | looks like what we want, yes. but that's strange, I found some doc, even at Redhat, saying "it works" but without proper example. | 18:13 |
| mriedem | to summarize, we talked about this at the pike ptg in february, we needed to have the various use cases documented in the spec to make sure the solution would cover them, and there were at least 2 stakeholders in the room saying, "we have an out of tree filter that does something like this" and we said, ok read this and tell us if it will replace your out of tree filter, and those people never replied to ack that it does | 18:14 |
| Tengu | erf | 18:14 |
| Tengu | may I explain what I did? | 18:14 |
| Tengu | and point to the doc I followed - maybe a solution might be found | 18:15 |
| *** pcaruana has joined #openstack-nova | 18:15 | |
| *** NostawRm has quit IRC | 18:15 | |
| *** NostawRm has joined #openstack-nova | 18:15 | |
| Tengu | mriedem: I followed https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/6/html/Administration_Guide/section-host-aggregates.html - I think that one has some equivalent in openstack "open" doc | 18:16 |
| *** Sukhdev has quit IRC | 18:17 | |
| Tengu | mriedem: I activated AggregateInstanceExtraSpecsFilter filter in nova.conf, and created two aggregate - all hosts are in those aggregates (in fact, for now, only two hosts - hence once per group). | 18:17 |
| Tengu | mriedem: the metadata is like "gen1=true" for first aggregate, "gen2=true" for the second. | 18:17 |
| melwitt | Tengu: I think if you tag your flavors with extra_specs and then use the AggregateInstanceExtraSpecsFilter you can do what you want | 18:17 |
| Tengu | after that, the flavor were created, and a metadata was added in the form "gen1=true" for m1.medium, and "gen2=true" for m2.medium | 18:18 |
| mriedem | the problem is, | 18:18 |
| mriedem | flavor1 is associated to agg1 and flavor2 is associated to agg2, | 18:18 |
| mriedem | but that doesn't exclude agg2 from using flavor1 | 18:18 |
| mriedem | and vice versa | 18:18 |
| mriedem | that's the strict isolation problme | 18:18 |
| Tengu | hmm ok. | 18:18 |
| mriedem | *problem | 18:18 |
| Tengu | not a really big issue - for now, we have "no host found" in fact | 18:18 |
| melwitt | I thought if the flavors were tagged it would require that key to pass? | 18:19 |
| mriedem | honestly i'd have to re-read https://review.openstack.org/#/c/381912/ | 18:19 |
| melwitt | I'm reading it again now | 18:19 |
| mriedem | i am definitely not an expert here on the existing capabilities and gaps | 18:19 |
| Tengu | melwitt: same for me - actually, for now, we're unable to start any instance because it doesn't find any host to run it | 18:19 |
| Tengu | mriedem: but maybe it's "just" the metadata format that fails me. is there any doc for that? | 18:20 |
| melwitt | Tengu: and you added gen1=true and gen2=true to your host aggregates? | 18:20 |
| Tengu | yup, as a metadata as well | 18:21 |
| mriedem | the now deleted ops guide might have had something specific for this | 18:21 |
| Tengu | :'( | 18:21 |
| cdent | it got moved to the wiki? | 18:22 |
| Tengu | I found a doc saying the metadata on the flavor should be in the form aggregate_instance_extra_specs:gen1='true' | 18:22 |
| Tengu | but that doesn't work either | 18:22 |
| cfriesen_ | mriedem: dansmith: just saw the mention of microversion 2.47...there was already a call to "instance.get_flavor()" previously, so I had assumed it would get the whole flavor. I suspect you're right that it's lazy-loading extra-specs. | 18:22 |
| mriedem | it would be in here if it existed https://docs.openstack.org/nova/latest/admin/index.html | 18:22 |
| dansmith | cfriesen_: no it's policy | 18:22 |
| dansmith | cfriesen_: the policy is checked per instance now, which is an fs call at least | 18:22 |
| cfriesen_ | dansmith: ah...I had a networking glitch, missed some irc. | 18:23 |
| *** yamahata has joined #openstack-nova | 18:23 | |
| cfriesen_ | dansmith: fix is what, cache the policy? | 18:23 |
| dansmith | cfriesen_: check once per list and not once per instance | 18:23 |
| dansmith | cfriesen_: I'm cooking it up now | 18:24 |
| dansmith | smells like bacon | 18:24 |
| cfriesen_ | dansmith: do we even need that? couldn't we check it the first time and cache it? | 18:24 |
| dansmith | cfriesen_: that's what I just said | 18:24 |
| cfriesen_ | I meant the first time on process startup | 18:24 |
| mriedem | Tengu: i've seen a better doc than that red hat one, sec | 18:24 |
| dansmith | cfriesen_: it depends per request | 18:25 |
| Tengu | mriedem: that would be nice :) | 18:25 |
| cfriesen_ | dansmith: ah, of course | 18:25 |
| melwitt | Tengu: I found this doc https://docs.openstack.org/ocata/config-reference/compute/schedulers.html#host-aggregates | 18:25 |
| dansmith | mriedem: confirmed the knee in the same place on master | 18:26 |
| Tengu | I've also followed https://blog.russellbryant.net/2013/05/21/availability-zones-and-host-aggregates-in-openstack-compute-nova/ - but failed. | 18:26 |
| Tengu | melwitt: ah, ocata, might work, pike is just one version ahead. will check that, thanks! | 18:26 |
| mriedem | melwitt: yeah https://docs.openstack.org/ocata/config-reference/compute/schedulers.html#example-specify-compute-hosts-with-ssds | 18:26 |
| mriedem | openstack flavor set --property aggregate_instance_extra_specs:ssd=true ssd.large | 18:26 |
| Tengu | duh… ok, I was also on that one -.-' | 18:27 |
| melwitt | Tengu: the main thing I saw ppl run into a snag is that you apparently have to use that prefix when you set the key on the flavor but NOT use it when you set the key on the aggregate | 18:27 |
| Tengu | melwitt: yup, I have done that | 18:27 |
| Tengu | but to no success until now. | 18:27 |
| mriedem | and that key prefix is only used with AggregateInstanceExtraSpecsFilter | 18:27 |
| cfriesen_ | was just going to mention the filter | 18:27 |
| mriedem | and you have to make sure you have that enabled | 18:27 |
| melwitt | Tengu: yeah, did you add that filter to your configured filters for the FilterScheduler? | 18:28 |
| melwitt | in nova.conf | 18:28 |
| Tengu | it's enabled. should it be in the first position? | 18:28 |
| mriedem | no | 18:28 |
| Tengu | melwitt: yep, it's present | 18:28 |
| mriedem | order only matters for performance | 18:28 |
| Tengu | and I rebooted the controllers in order to ensure all is running at the latest config version | 18:28 |
| melwitt | Tengu: no but you will want to check nova-scheduler logs to make sure some other filter isn't rejecting it | 18:28 |
| Tengu | mriedem: hmm ok. | 18:28 |
| melwitt | at DEBUG log level. it's possible something else is going wrong and not the key match for the metadata | 18:29 |
| Tengu | melwitt: yup, but I didn't see anything. the instance "directory" was created on the right node in /var/lib/nova/instances | 18:29 |
| melwitt | if you're getting NoValidHost you should see something | 18:29 |
| Tengu | but after a while, paff, directory is removed, and crash, "no host found"… although it actually HAD found a host | 18:29 |
| melwitt | unless a compute host rejected the request in which case you should see an error in the nova-compute logs or the nova-conductor logs | 18:30 |
| Tengu | hmmm. | 18:30 |
| Tengu | will check that one. | 18:30 |
| melwitt | the way it works is if scheduling filters all pass, it goes to nova-compute, if something fails while it builds it, it will tear it down, log stuff, and try to reschedule to another host if you have retries configured | 18:30 |
| Tengu | what would be the patter of a rejection in nova-compute.log ? | 18:31 |
| Tengu | hmm ok. | 18:31 |
| Tengu | I have the retryfilter | 18:31 |
| melwitt | should see something logged at ERROR level I think | 18:31 |
| melwitt | in nova-compute | 18:31 |
| Tengu | think this one will try to re-schedule | 18:31 |
| Tengu | duh | 18:31 |
| Tengu | corrupted image download o_O | 18:32 |
| *** pino has joined #openstack-nova | 18:32 | |
| Tengu | that might explain a bit. but that would point the glance storage | 18:32 |
| openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: config drive https://review.openstack.org/409404 | 18:33 |
| openstackgerrit | Eric Berglund proposed openstack/nova master: WIP(5): PowerVM driver: ovs vif https://review.openstack.org/422512 | 18:33 |
| Tengu | although… hmm. timestamp doesn't really match. will dig a bit more. | 18:33 |
| melwitt | Tengu: yeah, so you have other issues there. but as long as the instance is always landing on the host where the aggregate meta matches the flavor, you know at least the extra specs filtering is working correctly | 18:34 |
| melwitt | (for your original concern) | 18:34 |
| *** acormier has quit IRC | 18:34 | |
| Tengu | melwitt: right. | 18:35 |
| Tengu | so my debug steps weren't that wrong. I should have had a better look to the nova-compute.log file though. | 18:35 |
| mriedem | you can also trace the request id and/or instance id through the logs if you have your logs pumped to an ELK stack | 18:36 |
| mriedem | or journald like in devstack | 18:37 |
| Tengu | for now we don't have an ELK (it will run on the openstack… well, yes, that might cause some issues at some point ;)). | 18:38 |
| Tengu | but we want to do that, yep. | 18:38 |
| dansmith | mriedem: https://imgur.com/a/FY7Oq | 18:39 |
| dansmith | mriedem: over about 300 runs, my patch is consistently faster than master | 18:39 |
| mriedem | oh that's w/o the policy fix :) | 18:39 |
| mriedem | i was like, wtf | 18:39 |
| dansmith | yes | 18:39 |
| Tengu | but the image corruption is the best hint for now. Have to check why - the ceph cluster isn't a cluster for now and we have some failed disks on it, so it can explain a lot. it's not in prod for now, this also explain some issues | 18:39 |
| mriedem | dansmith: throw that in https://etherpad.openstack.org/p/nova-instance-list somewhere so we don't lose it | 18:41 |
| stvnoyes | hi mriedem, if you get a change to re-review https://review.openstack.org/#/c/463987/ it would be great. I am on vacation next week so there's still some time this week for me to turn the review around again if it's needed. thanks. | 18:42 |
| mriedem | ok | 18:43 |
| mriedem | dansmith: totally unrelated, but i'm think about throwing the ceph job in the experimental queue http://tinyurl.com/ydy3jek9 | 18:44 |
| stvnoyes | johnthetubaguy: pls take a look at https://review.openstack.org/#/c/506805/ when you get a chance. it's a pretty small change, and it's needed for the cinder v3 live migrate change. thanks. | 18:44 |
| dansmith | melwitt: ^ | 18:44 |
| melwitt | gdi | 18:44 |
| melwitt | it would take me awhile to unroll what is going on with that job | 18:46 |
| mriedem | this is compared to the normal dsvm tempest job http://tinyurl.com/ydemspkl | 18:46 |
| melwitt | yeah, hm. so it was tracking okay until around the 16th | 18:47 |
| *** lpetrut has joined #openstack-nova | 18:47 | |
| melwitt | I'll dig into it | 18:48 |
| mriedem | well, there were also spikes in the normal job then too, just not as bad | 18:48 |
| mriedem | 9/23 is where it goes nuts | 18:48 |
| melwitt | yeah. last we discussed it was at the last PTG and jbernard had some TODOs but I didn't know details about what they were. I thought the first thing was something to do with a job timeout being too short | 18:49 |
| mriedem | he was going to start restricting the tests | 18:49 |
| melwitt | restricting in what way? | 18:49 |
| mriedem | L138 https://etherpad.openstack.org/p/nova-ptg-pike | 18:49 |
| melwitt | okay. so that would be my starting point, aside from the latest craziness. which might just be more timeout and OOM stuff (have to dig) | 18:50 |
| *** lpetrut has quit IRC | 18:50 | |
| efried | mriedem Re-request tough love on https://review.openstack.org/#/c/488137/ please | 18:51 |
| *** lpetrut has joined #openstack-nova | 18:51 | |
| mriedem | melwitt: https://github.com/openstack/nova/commit/980d0fcd75c2b15ccb0af857a9848031919c6c7d merged on the 22nd | 18:51 |
| mriedem | cinder.tests.tempest.api.volume.test_volume_revert.VolumeRevertTests.test_volume_revert_to_snapshot_after_extended is what i see failing | 18:51 |
| mriedem | so my guess is, the ceph job doesn't care for live snapshots | 18:51 |
| melwitt | thanks for that info | 18:52 |
| *** penick has quit IRC | 18:53 | |
| mriedem | although the test that's failing is a cinder api test | 18:54 |
| *** sbezverk has joined #openstack-nova | 18:54 | |
| mriedem | and i don't see any related errors in n-cpu | 18:54 |
| *** moshele has joined #openstack-nova | 18:55 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Fix policy check performance in 2.47 https://review.openstack.org/507948 | 18:55 |
| *** cleong has quit IRC | 18:58 | |
| dansmith | oops, didn't finish the commit message on that one | 18:59 |
| dansmith | I'm just testing it in my devstack rig anyway | 18:59 |
| *** slaweq_ has joined #openstack-nova | 19:00 | |
| *** pcaruana has quit IRC | 19:00 | |
| *** sree has joined #openstack-nova | 19:01 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: make host aggregates examples more discoverable https://review.openstack.org/507950 | 19:02 |
| *** hemna_ has quit IRC | 19:02 | |
| mriedem | not even the bug link | 19:02 |
| *** r-daneel has quit IRC | 19:02 | |
| mriedem | you were so excited | 19:02 |
| mriedem | about getting punched in the naughty parts | 19:02 |
| mriedem | Tengu: melwitt: https://review.openstack.org/#/c/507950/ | 19:03 |
| mriedem | ^ should help a bit with doc discovery | 19:03 |
| *** r-daneel has joined #openstack-nova | 19:03 | |
| Tengu | mriedem: \o/ thanks ! | 19:03 |
| Tengu | for now I'm digging in glance, as apparently it's crashed. | 19:04 |
| *** sree has quit IRC | 19:06 | |
| dansmith | sdague: mriedem: cfriesen_: https://imgur.com/a/IQ0Vh | 19:07 |
| mriedem | dansmith: awesome | 19:07 |
| mriedem | also, | 19:08 |
| mriedem | i ran the 2.53 microversion, GET /servers/detail thing again w/o your patch, to see why i had just a big difference in numbers, and you're right, it's the vm | 19:08 |
| mriedem | so w/o your patch, it's still closer to with your patch, | 19:08 |
| mriedem | and over half of what it was the other day on the other vm | 19:08 |
| mriedem | so just need to chalk that up to public cloud | 19:08 |
| cdent | what happened at 2.47? | 19:09 |
| dansmith | mriedem: cool | 19:09 |
| mriedem | cdent: we started checking policy per instance when listing instances | 19:09 |
| mriedem | which adds up when you're listing 1000 instances | 19:09 |
| cdent | ouch | 19:09 |
| *** penick has joined #openstack-nova | 19:10 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Fix policy check performance in 2.47+ https://review.openstack.org/507948 | 19:10 |
| *** liangy has joined #openstack-nova | 19:11 | |
| cfriesen_ | cdent: my bad, I didn't realize policy check was expensive | 19:11 |
| *** lyan has quit IRC | 19:11 | |
| mriedem | i'm sure i approved the change so don't worry about it | 19:12 |
| cdent | cfriesen_: a reasonable thing to assume in a reasonable universe, but we probably left that one long ago | 19:12 |
| sdague | I kind of wonder if there are other places with embedded policy checks like that are expensive | 19:12 |
| sdague | cfriesen_: there is an implicit fstat because policy is live reread | 19:13 |
| cdent | speaking of, that’s a potential next microoptimization in the unit tests. that file gets read over and over and over over and over and over and ... | 19:13 |
| sdague | honestly, it might behoove us to change that behavior entirely, as we've got the hup handler now | 19:14 |
| bauzas | dansmith: you trampled me | 19:15 |
| *** sahid has quit IRC | 19:15 | |
| dansmith | bauzas: I did? | 19:15 |
| bauzas | dansmith: with Twitter | 19:15 |
| *** edmondsw has quit IRC | 19:15 | |
| bauzas | :p | 19:15 |
| bauzas | so, maybe you should be the next US president given you use Twitter for trampling folks :p | 19:16 |
| bauzas | mmm, maybe "trample" is not the right verb | 19:16 |
| dansmith | I'm not sure what trampling I did, but I definitely need not be president | 19:16 |
| penick | too late i'm writing you in | 19:16 |
| bauzas | I mean, I chilled :p | 19:16 |
| mriedem | you made sylvain spit out his coffee | 19:17 |
| mriedem | you "floored" him | 19:17 |
| bauzas | when I saw the tweet for 2.47 :p | 19:17 |
| bauzas | sorry for "trampling" | 19:17 |
| dansmith | bauzas: okay I replied to you about two seconds before you pinged me here so I thought you meant my reply was rude in some way | 19:18 |
| bauzas | emacron: maybe you should ask French folks to stop using French but rather English ? | 19:18 |
| *** edmondsw has joined #openstack-nova | 19:18 | |
| bauzas | dansmith: sorry, the verb wasn't good :) | 19:19 |
| dansmith | ack | 19:19 |
| bauzas | "chilling" is better | 19:19 |
| bauzas | dansmith: anyway, thanks for your tweet | 19:20 |
| *** liverpooler has quit IRC | 19:23 | |
| *** edmondsw has quit IRC | 19:23 | |
| *** liverpooler has joined #openstack-nova | 19:23 | |
| *** jmlowe has quit IRC | 19:25 | |
| cfriesen_ | dansmith: reviewing your patch. I assume the version check is a performance optimization to avoid the policy check if we can? | 19:25 |
| *** penick_ has joined #openstack-nova | 19:27 | |
| mriedem | cfriesen_: it's because we only ever care about showing flavor extra specs if you're requesting 2.47 or above | 19:28 |
| mriedem | so don't even make the policy check otherwise | 19:28 |
| *** liangy has quit IRC | 19:28 | |
| dansmith | cfriesen_: yeah | 19:28 |
| sdague | dansmith: so one thing to consider on that test, there is nothing in that test asserting the server list is > 1 right now | 19:28 |
| sdague | because it's all common setup | 19:28 |
| dansmith | sdague: true, but I did check that its 4 | 19:28 |
| dansmith | I can add another | 19:29 |
| sdague | I thought it was 5 | 19:29 |
| sdague | I was just running it | 19:29 |
| dansmith | it was 4 | 19:29 |
| mriedem | so just self.assertGreater(len(instances), 1) ? | 19:29 |
| dansmith | oh no, | 19:29 |
| dansmith | top index was 4 | 19:29 |
| dansmith | 0-4 | 19:29 |
| sdague | yeh, something like that | 19:30 |
| *** penick has quit IRC | 19:30 | |
| sdague | just so the test is more concisely valid | 19:31 |
| sdague | the reset seems fine to me | 19:31 |
| dansmith | done | 19:31 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Fix policy check performance in 2.47+ https://review.openstack.org/507948 | 19:34 |
| *** shaner has quit IRC | 19:36 | |
| *** liverpooler has quit IRC | 19:36 | |
| *** shaner has joined #openstack-nova | 19:36 | |
| cfriesen_ | just throwing this out there...could we use a global variable for show_extra_specs such that it's None for the first instance and then the calculated value is used for subsequent ones? That'd avoid the API changes, but globals are icky. | 19:36 |
| mriedem | this isn't an api change | 19:37 |
| cfriesen_ | picky picky...function signature changes | 19:37 |
| dansmith | cfriesen_: that doesn't work | 19:37 |
| dansmith | cfriesen_: because this code is running lots of lists for lots of people, some of which do and some of which don't have that permission | 19:37 |
| cfriesen_ | set it to None at the beginning of each call | 19:38 |
| *** gjayavelu has quit IRC | 19:38 | |
| mriedem | that's what this does... | 19:38 |
| mriedem | w/o a global | 19:38 |
| dansmith | cfriesen_: that also won't work | 19:38 |
| *** liverpooler has joined #openstack-nova | 19:38 | |
| dansmith | cfriesen_: because we don't necessarily complete a whole call through the stack before we go on to the next one | 19:38 |
| dansmith | cfriesen_: that would be known as a "CVE" | 19:39 |
| cfriesen_ | dansmith: due to eventlets I guess? | 19:39 |
| dansmith | threads in general | 19:39 |
| mriedem | i wonder if the fedex guy is required to jog from the truck to the house and back | 19:40 |
| mriedem | like, is there a camera watching him to make sure he jogs? | 19:40 |
| dansmith | there is at my house | 19:40 |
| dansmith | I call and complain any time he saunters instead of jogs | 19:40 |
| mriedem | what if he mosey's? | 19:40 |
| cfriesen_ | dansmith: I thought we were using processes for nova-api, not threads | 19:41 |
| dansmith | mosey is a saunter with slightly more vigor | 19:41 |
| mriedem | more tude | 19:41 |
| dansmith | cfriesen_: there are threads (greenthreads currently) in each process | 19:41 |
| cfriesen_ | ah, got it. | 19:41 |
| dansmith | cfriesen_: but seriously, setting a global for a permission flag and hoping it gets reset before the next call is like the worst idea ever :) | 19:41 |
| cfriesen_ | I'm pretty sure I've had worse. :) I just didn't like the fact that we were checking it in two different places depending on flow. | 19:42 |
| cfriesen_ | In C/C++ I'd just pass a pointer or pass it by reference. | 19:43 |
| mriedem | in fortran i'd goto that mothertrucker | 19:43 |
| * dansmith moves on | 19:43 | |
| cdent | dansmith: None and False meaning different things. ballsy. | 19:43 |
| cfriesen_ | I saw old fortran back in my engineering days that had goto with multiple targets....that was messed up. | 19:43 |
| dansmith | cdent: they are wholly different things :) | 19:44 |
| sdague | cfriesen_: if you wanted to handle it in a different way, the nova local caching model would be just to make context.can cache the parameter lists and answers | 19:44 |
| cdent | sure, but still | 19:44 |
| sdague | because contexts are constructed all the time | 19:44 |
| *** eharney has quit IRC | 19:44 | |
| sdague | and that would fastpath the check in tight loops like this | 19:44 |
| dansmith | sdague: so I was thinking about a way to make the fixture raise if a test checked the same exactly policy action/target in a single run | 19:45 |
| *** liverpooler has quit IRC | 19:45 | |
| cfriesen_ | sdague: yes, that'd be nice. but this isn't horrible | 19:45 |
| dansmith | sdague: to ferret out some of what you were saying might be there | 19:45 |
| dansmith | cfriesen_: sdague: this has to be minimal backport though | 19:45 |
| dansmith | because this _has_ to be backported | 19:45 |
| dansmith | else we lose our license to code | 19:45 |
| cfriesen_ | agreed | 19:45 |
| mriedem | retain our license to ill? | 19:45 |
| sdague | dansmith: yeh, I think context.can is the right place to shadow that | 19:46 |
| sdague | in tests | 19:46 |
| dansmith | sdague: yeah | 19:46 |
| sdague | I also really wonder how much better perf would get if we remove the policy reload entirely | 19:46 |
| sdague | because that's just kind of there because of the early rax operational model | 19:47 |
| dansmith | sdague: if you can give me a line to comment out I can run it on my rig while it's still built | 19:47 |
| mriedem | i think dims was looking at making the policy check call an external service...if that makes you worry at all | 19:47 |
| *** edmondsw has joined #openstack-nova | 19:47 | |
| sdague | mriedem: yeh, I think this small thing has pretty much shown you can only do that if it's a load on startup | 19:47 |
| *** lyan has joined #openstack-nova | 19:47 | |
| sdague | otherwise you are toast | 19:47 |
| sdague | dansmith: yeh, let me go look, it's been a minute since I've been in that code | 19:48 |
| dansmith | mriedem: jesus | 19:48 |
| *** edmondsw has quit IRC | 19:48 | |
| *** edmondsw has joined #openstack-nova | 19:48 | |
| *** markvoelker has quit IRC | 19:48 | |
| bauzas | dansmith: sdague: mriedem: any reason why https://review.openstack.org/#/c/507948/3 isn't yet +W'd ? | 19:49 |
| bauzas | can I pull the trigger? | 19:49 |
| mriedem | apparently there is a cache https://github.com/openstack/oslo.policy/blob/master/oslo_policy/policy.py#L630 | 19:49 |
| *** markvoelker has joined #openstack-nova | 19:49 | |
| *** shaner has quit IRC | 19:50 | |
| dansmith | mriedem: that means we're still doing the fstat() each time right? | 19:50 |
| dansmith | that's the painful bit I think, especially given policy files are empty now | 19:50 |
| *** shaner has joined #openstack-nova | 19:50 | |
| mriedem | true | 19:50 |
| dansmith | actually two calls | 19:50 |
| dansmith | exists and getmtime | 19:50 |
| mriedem | just return True from _is_directory_updated all the time? | 19:50 |
| dansmith | or only check the directory if it's been 30s since we last did it | 19:51 |
| dansmith | or only check it after you've sighup'd :) | 19:51 |
| cdent | inotify | 19:52 |
| dansmith | I guess you meant for this q&d check, yeah | 19:52 |
| mriedem | yeah just change that to return False all the time | 19:53 |
| mriedem | it would be False based on how it's used | 19:53 |
| *** pino has quit IRC | 19:53 | |
| mriedem | see if you shave a few seconds | 19:53 |
| *** pino has joined #openstack-nova | 19:53 | |
| sdague | mriedem: yeh, that's the stat calls | 19:54 |
| sdague | _is_directory_updated is different though | 19:54 |
| sdague | because that's the different terrible issue of a policy.d | 19:55 |
| sdague | which makes this *N worse | 19:55 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Fix policy check performance in 2.47+ https://review.openstack.org/507965 | 19:55 |
| dansmith | bauzas: pull the trigger | 19:55 |
| dansmith | sdague: so wait, is there more I need to kill? | 19:55 |
| bauzas | ack | 19:55 |
| openstackgerrit | melanie witt proposed openstack/nova master: Set group_members when converting to legacy request spec https://review.openstack.org/507938 | 19:55 |
| bauzas | honestly, I think fixing it now is better anyway | 19:55 |
| bauzas | hah, jinxed | 19:56 |
| *** moshele has quit IRC | 19:56 | |
| sdague | dansmith: I think you want to short circuit here - https://github.com/openstack/oslo.policy/blob/70ba1beb3e3c93fafc147633360df838155a82a9/oslo_policy/_cache_handler.py#L31 | 19:58 |
| *** vvargaszte has quit IRC | 19:58 | |
| sdague | and just return (True, "") | 19:58 |
| sdague | actually (False, "") | 19:58 |
| mriedem | "A tuple with a boolean specifying if the data is fresh or not" in the docstring doesn't seem to match the code | 19:59 |
| sdague | mriedem: yeh it does | 20:00 |
| mriedem | if the data "was refreshed" | 20:00 |
| sdague | https://github.com/openstack/oslo.policy/blob/70ba1beb3e3c93fafc147633360df838155a82a9/oslo_policy/policy.py#L676 | 20:00 |
| sdague | mriedem: ah, ok, english binary reversal | 20:00 |
| mriedem | right | 20:00 |
| sdague | honestly, I read it in the reversed context the first time | 20:00 |
| dansmith | I thought I should return (False, cache['filename']['data']) no? | 20:01 |
| mriedem | it's not not unfresh?! | 20:01 |
| sdague | dansmith: it could, but it could also just return garbage | 20:01 |
| dansmith | okay | 20:01 |
| sdague | because if the first param is False, nothing is done with it | 20:01 |
| dansmith | ah, okay | 20:01 |
| sdague | "" would be the production default state | 20:01 |
| dansmith | oh, right, | 20:02 |
| sdague | as policy.json is blank | 20:02 |
| *** chyka has quit IRC | 20:02 | |
| dansmith | was thinking I had to return the actual thing | 20:02 |
| *** chyka has joined #openstack-nova | 20:02 | |
| sdague | though, I think the impact is going to be minimal, given that the issue was 1000 of these calls, which added 3s in your env, so were looking at 3ms per policy check | 20:05 |
| dansmith | yeah | 20:05 |
| sdague | so, unless we have another nested thing, it's going to be hard to see the impact | 20:05 |
| dansmith | I see no impact as the first numbers are coming out | 20:05 |
| *** yamamoto has quit IRC | 20:08 | |
| mriedem | melwitt: couple comments on the test, but looks great otherwise | 20:09 |
| melwitt | thanks, looking | 20:13 |
| dansmith | mriedem: sdague: not likely any difference: https://imgur.com/a/gSIlq | 20:13 |
| *** edand has quit IRC | 20:14 | |
| *** r-daneel has quit IRC | 20:14 | |
| *** mnestratov has joined #openstack-nova | 20:15 | |
| *** r-daneel has joined #openstack-nova | 20:15 | |
| mriedem | sdague: i just git log -i --grep'ed for the first time i think | 20:18 |
| mriedem | after you've said it in channel once per week i think | 20:18 |
| mriedem | it might be the first thing you say when you wake up | 20:19 |
| mriedem | not sure | 20:19 |
| efried | oo. TIL. | 20:22 |
| sdague | dansmith: yeh, it's clear the current call is down to 2 policy checks, which is good | 20:26 |
| *** yamamoto has joined #openstack-nova | 20:27 | |
| openstackgerrit | Merged openstack/nova master: _rollback_live_migration in live-migration seqdiag https://review.openstack.org/507871 | 20:27 |
| *** smatzek has quit IRC | 20:29 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Make setenv consistent for functional and api_sample_tests https://review.openstack.org/507976 | 20:30 |
| *** yamamoto has quit IRC | 20:31 | |
| mriedem | cdent: i know what's going on in my devstack patch | 20:32 |
| *** gouthamr_ has quit IRC | 20:32 | |
| mriedem | oh do i ever | 20:32 |
| *** belmoreira has joined #openstack-nova | 20:32 | |
| mriedem | i've been mtreinish'd i think | 20:32 |
| cdent | oh? | 20:32 |
| dansmith | mriedem: unfortunately a unit test failure snuck past me in that fix patch | 20:32 |
| mriedem | where is my gd trombone | 20:33 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Fix policy check performance in 2.47+ https://review.openstack.org/507948 | 20:33 |
| *** jpena|off has quit IRC | 20:33 | |
| *** ltomasbo has quit IRC | 20:34 | |
| mriedem | cdent: this case statement used to have a wildcard https://github.com/openstack-dev/devstack/blob/master/stackrc#L682 | 20:34 |
| openstackgerrit | melanie witt proposed openstack/nova master: Set group_members when converting to legacy request spec https://review.openstack.org/507938 | 20:34 |
| *** gouthamr has joined #openstack-nova | 20:35 | |
| cdent | mriedem: whyreaka | 20:36 |
| mriedem | GAH | 20:36 |
| mriedem | f119121d21fa0446197b26378091677daac1606a | 20:36 |
| mriedem | sdague'ed | 20:36 |
| sdague | mriedem: ok, so what's the issue? | 20:37 |
| *** ltomasbo has joined #openstack-nova | 20:37 | |
| mriedem | VIRT_DRIVER=fake means you don't get any default image downloaded | 20:37 |
| *** jpena|off has joined #openstack-nova | 20:38 | |
| mriedem | so i assume you don't want the wildcard back | 20:38 |
| johnsom | FYI, after collecting a mountain of logs on this missing network interface issue I decided to force a PCI bus rescan inside the instance, boom, the interface appears as it should have. So, ubuntu/kernel/something issue and not nova/neutron | 20:38 |
| mriedem | so i'll just add a case for fake and make it the same as libvirt? | 20:38 |
| cdent | sounds right to me | 20:39 |
| johnsom | stock, current, ubuntu 16.04 cloud image | 20:39 |
| cdent | I’m dead, will check back in the morn | 20:39 |
| *** cdent has quit IRC | 20:39 | |
| sdague | mriedem: ok, because later everything assumes a real image? | 20:39 |
| *** gouthamr has quit IRC | 20:39 | |
| mriedem | sdague: well, tempest tries to get the image from glance to put in tempest.conf | 20:40 |
| mriedem | the image id i mean | 20:40 |
| mriedem | working a patch | 20:41 |
| *** gouthamr has joined #openstack-nova | 20:42 | |
| sdague | yeh, so, honestly we should probably figure out a better setup strategy for "throw away this image" | 20:43 |
| sdague | mriedem: also, the qemu 2.10 thing wasn't a race, it's actually python2.7 and 3.5 evaluating the mock sentinel differently in the comparison | 20:44 |
| sdague | for one of them it passes a >= check and the other it does not | 20:44 |
| *** belmoreira has quit IRC | 20:44 | |
| *** gouthamr has quit IRC | 20:45 | |
| openstackgerrit | Sean Dague proposed openstack/nova master: Support qemu >= 2.10 https://review.openstack.org/505673 | 20:45 |
| *** esberglu has quit IRC | 20:47 | |
| mriedem | oh | 20:47 |
| *** esberglu has joined #openstack-nova | 20:48 | |
| mriedem | melwitt: pep8 | 20:49 |
| mriedem | 507938 | 20:50 |
| mriedem | oops | 20:50 |
| mriedem | ./nova/tests/functional/regressions/test_bug_1719730.py:17:1: F401 'cast_as_call' imported but unused | 20:50 |
| melwitt | sigh, of course | 20:50 |
| melwitt | thanks | 20:50 |
| mriedem | i still have the trombone out if you want to borrow it | 20:50 |
| melwitt | haha yeah | 20:50 |
| melwitt | I thought, pep8 can't complain if I just deleted a line. yeah it could | 20:51 |
| *** lucasxu has quit IRC | 20:52 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Set group_members when converting to legacy request spec https://review.openstack.org/507938 | 20:52 |
| *** esberglu has quit IRC | 20:52 | |
| melwitt | gdi locally I'm failing a notifications func test because I'm missing a notification from conductor. but the code that's supposed to send that notification is running, so I don't know how it could be missing | 20:57 |
| *** LeoBud has joined #openstack-nova | 20:58 | |
| *** pchavva has quit IRC | 20:58 | |
| *** smatzek has joined #openstack-nova | 21:02 | |
| *** mvk has joined #openstack-nova | 21:02 | |
| *** sree has joined #openstack-nova | 21:02 | |
| *** penick_ has quit IRC | 21:03 | |
| *** eharney has joined #openstack-nova | 21:04 | |
| *** penick has joined #openstack-nova | 21:07 | |
| *** sree has quit IRC | 21:07 | |
| *** esberglu has joined #openstack-nova | 21:08 | |
| *** thorst has quit IRC | 21:09 | |
| *** crushil has quit IRC | 21:09 | |
| *** penick has quit IRC | 21:11 | |
| *** thorst has joined #openstack-nova | 21:11 | |
| *** MVenesio has quit IRC | 21:13 | |
| *** MVenesio has joined #openstack-nova | 21:13 | |
| *** bnemec has quit IRC | 21:14 | |
| *** thorst has quit IRC | 21:15 | |
| *** jpena|off has quit IRC | 21:16 | |
| *** ltomasbo has quit IRC | 21:16 | |
| *** MVenesio has quit IRC | 21:17 | |
| *** ltomasbo has joined #openstack-nova | 21:18 | |
| *** jpena|off has joined #openstack-nova | 21:18 | |
| *** LeoBud has quit IRC | 21:20 | |
| *** yamamoto has joined #openstack-nova | 21:28 | |
| *** thorst has joined #openstack-nova | 21:29 | |
| *** slaweq_ has quit IRC | 21:32 | |
| *** thorst has quit IRC | 21:34 | |
| *** yamamoto has quit IRC | 21:34 | |
| *** yamahata has quit IRC | 21:36 | |
| *** itlinux has joined #openstack-nova | 21:36 | |
| *** yamahata has joined #openstack-nova | 21:36 | |
| openstackgerrit | Merged openstack/nova master: Have one list of reboot task_states https://review.openstack.org/219981 | 21:38 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move ploop commands to privsep. https://review.openstack.org/492325 | 21:38 |
| openstackgerrit | Michael Still proposed openstack/nova master: Read from console ptys using privsep. https://review.openstack.org/489486 | 21:38 |
| openstackgerrit | Michael Still proposed openstack/nova master: Don't shell out to mkdir, use ensure_tree() https://review.openstack.org/492326 | 21:38 |
| openstackgerrit | Michael Still proposed openstack/nova master: Cleanup mount / umount and associated rmdir calls https://review.openstack.org/494423 | 21:38 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move lvm handling to privsep. https://review.openstack.org/495516 | 21:38 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move shred to privsep. https://review.openstack.org/495537 | 21:38 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xend existence probes to privsep. https://review.openstack.org/495538 | 21:38 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move the idmapshift binary into privsep. https://review.openstack.org/495541 | 21:38 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move loopback setup and removal to privsep. https://review.openstack.org/495664 | 21:38 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move nbd commands to privsep. https://review.openstack.org/500351 | 21:38 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move kpartx calls to privsep. https://review.openstack.org/500354 | 21:38 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move blkid calls to privsep. https://review.openstack.org/500398 | 21:38 |
| *** gjayavelu has joined #openstack-nova | 21:40 | |
| mriedem | sydney travel approved, hotel booked! | 21:42 |
| *** tetsuro has quit IRC | 21:42 | |
| *** awaugama has quit IRC | 21:43 | |
| melwitt | yay | 21:44 |
| *** gyee has quit IRC | 21:44 | |
| mriedem | time to go through the blistering visa process | 21:44 |
| *** dave-mccowan has quit IRC | 21:45 | |
| melwitt | it's easy to do on their site. there's no free option for US ppl, have to get the ETA visa and it costs $20 AUD | 21:45 |
| *** penick has joined #openstack-nova | 21:46 | |
| mriedem | yeah, i was joking | 21:46 |
| melwitt | oh | 21:46 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Fix policy check performance in 2.47+ https://review.openstack.org/507965 | 21:47 |
| mriedem | ooo the hotel has an infinity pool | 21:48 |
| mriedem | can we do the sessions from poolside?! | 21:48 |
| mriedem | with kangerbangers | 21:49 |
| melwitt | what? what kinda fancy hotel are you staying at | 21:49 |
| mriedem | the sofitel next to the conference center, for the non-refundable pay immediately and never get your money back rate | 21:49 |
| melwitt | nice | 21:49 |
| mtreinish | mriedem: the sofitel has an inifinity pool? I must have missed that on the website | 21:51 |
| mtreinish | not that I looked closely | 21:51 |
| openstackgerrit | Merged openstack/nova master: Implement query param schema for agent index https://review.openstack.org/506950 | 21:52 |
| mriedem | well, it will, it's not open yet | 21:53 |
| *** baoli has quit IRC | 21:54 | |
| melwitt | hm, it looks like the prices are lower than when I first looked several weeks ago | 21:54 |
| cfriesen_ | melwitt: yeah, I was surprised that Canadians don't get the free option. I figured what with commonwealth and all... | 21:54 |
| mriedem | the only thing you share, | 21:55 |
| *** penick has quit IRC | 21:55 | |
| mriedem | is your love of groveling for the royal family | 21:55 |
| mriedem | there are no other benefits | 21:55 |
| cfriesen_ | our love of beer, more like. | 21:55 |
| mriedem | germans don't love beer? | 21:55 |
| cfriesen_ | I didn't say it was exclusive | 21:55 |
| mriedem | but germans can't love the royalty the same way | 21:56 |
| cfriesen_ | half the british royalty was german anyway | 21:56 |
| mriedem | oh breeding | 21:56 |
| mriedem | i plan to marry my daughter to a fellow in the next county | 21:57 |
| mriedem | for the inter-county alliance | 21:57 |
| mriedem | against NE iowa | 21:57 |
| *** yamamoto has joined #openstack-nova | 21:57 | |
| *** mriedem is now known as mriedem_away | 21:57 | |
| *** yamamoto has quit IRC | 21:57 | |
| *** dave-mccowan has joined #openstack-nova | 21:57 | |
| *** thorst has joined #openstack-nova | 21:58 | |
| *** lpetrut has quit IRC | 21:59 | |
| *** lyan has quit IRC | 21:59 | |
| *** vladikr has quit IRC | 22:02 | |
| *** vladikr has joined #openstack-nova | 22:02 | |
| *** slaweq_ has joined #openstack-nova | 22:03 | |
| *** dave-mccowan has quit IRC | 22:03 | |
| *** takashin has joined #openstack-nova | 22:06 | |
| takashin | Spec cores, would you review https://review.openstack.org/#/c/489029/ ? It got one +2. | 22:09 |
| openstackgerrit | Merged openstack/nova-specs master: Move pike implemented specs https://review.openstack.org/500369 | 22:09 |
| *** smatzek has quit IRC | 22:15 | |
| *** gbarros has quit IRC | 22:22 | |
| *** edmondsw has quit IRC | 22:23 | |
| *** gbarros has joined #openstack-nova | 22:23 | |
| *** mnestratov has quit IRC | 22:28 | |
| *** tidwellr has quit IRC | 22:30 | |
| *** krtaylor has quit IRC | 22:30 | |
| *** gjayavelu has quit IRC | 22:31 | |
| *** slaweq_ has quit IRC | 22:36 | |
| *** slaweq_ has joined #openstack-nova | 22:39 | |
| *** krtaylor has joined #openstack-nova | 22:43 | |
| *** penick has joined #openstack-nova | 22:45 | |
| *** jaypipes has quit IRC | 22:48 | |
| *** gbarros has quit IRC | 22:50 | |
| *** penick has quit IRC | 22:52 | |
| *** yamamoto has joined #openstack-nova | 22:58 | |
| openstackgerrit | Merged openstack/nova master: Fix --max-count handling for nova-manage cell_v2 map_instances https://review.openstack.org/502236 | 22:59 |
| *** rtjure has quit IRC | 23:01 | |
| *** claudiub|3 has quit IRC | 23:03 | |
| cfriesen_ | is there a reason why nova uses named indexes in the DB rather than using "index=True" as part of the Column definition? | 23:03 |
| *** pino has quit IRC | 23:04 | |
| *** yamamoto has quit IRC | 23:05 | |
| *** slaweq_ has quit IRC | 23:13 | |
| *** baoli has joined #openstack-nova | 23:18 | |
| *** slaweq_ has joined #openstack-nova | 23:18 | |
| *** baoli_ has joined #openstack-nova | 23:22 | |
| *** baoli has quit IRC | 23:22 | |
| *** tidwellr has joined #openstack-nova | 23:22 | |
| *** hongbin has quit IRC | 23:25 | |
| *** gjayavelu has joined #openstack-nova | 23:26 | |
| *** felipemonteiro_ has quit IRC | 23:29 | |
| *** thorst has quit IRC | 23:30 | |
| mriedem_away | easier to change them later if needed | 23:31 |
| mriedem_away | cfriesen_: ^ | 23:31 |
| mriedem_away | since different backends name them if you do it automatically | 23:31 |
| *** mriedem_away is now known as mriedem | 23:31 | |
| *** Sukhdev has joined #openstack-nova | 23:32 | |
| *** jmlowe has joined #openstack-nova | 23:32 | |
| mriedem | although i think in recent years we just identify an index by the columns in it | 23:32 |
| *** krtaylor has quit IRC | 23:35 | |
| *** edmondsw has joined #openstack-nova | 23:40 | |
| *** ijw has joined #openstack-nova | 23:43 | |
| *** edmondsw has quit IRC | 23:44 | |
| *** moshele has joined #openstack-nova | 23:44 | |
| *** ijw has quit IRC | 23:45 | |
| *** ijw has joined #openstack-nova | 23:45 | |
| *** slaweq_ has quit IRC | 23:48 | |
| *** gjayavelu has quit IRC | 23:50 | |
| *** jmlowe has quit IRC | 23:52 | |
| *** slaweq_ has joined #openstack-nova | 23:52 | |
| *** bnemec has joined #openstack-nova | 23:52 | |
| *** jmlowe has joined #openstack-nova | 23:53 | |
| *** r-daneel has quit IRC | 23:58 | |
| *** yamahata has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!