Tuesday, 2019-10-22

*** xek has quit IRC00:02
*** mriedem has quit IRC00:02
*** markvoelker has joined #openstack-nova00:03
*** adriant has quit IRC00:04
*** macz has quit IRC00:05
*** TxGirlGeek has quit IRC00:05
*** adriant has joined #openstack-nova00:07
*** adriant has quit IRC00:09
*** david-lyle is now known as dklyle00:09
*** markvoelker has quit IRC00:13
*** adriant has joined #openstack-nova00:24
*** adriant has quit IRC00:25
*** adriant has joined #openstack-nova00:28
*** gyee has quit IRC00:28
*** ociuhandu has joined #openstack-nova00:31
*** adriant has quit IRC00:32
*** ociuhandu has quit IRC00:35
*** slaweq has joined #openstack-nova00:47
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Check target consumer before reverting allocations  https://review.opendev.org/68904900:51
*** slaweq has quit IRC00:51
*** spatel has joined #openstack-nova00:59
*** adriant has joined #openstack-nova01:08
*** brinzhang has joined #openstack-nova01:12
*** brinzhang_ has quit IRC01:15
*** brinzhang_ has joined #openstack-nova01:33
*** nanzha has joined #openstack-nova01:33
*** brinzhang has quit IRC01:36
*** Liang__ has joined #openstack-nova01:37
*** slaweq has joined #openstack-nova01:47
*** nanzha has quit IRC01:48
*** nanzha has joined #openstack-nova01:49
*** slaweq has quit IRC01:53
*** nanzha has quit IRC01:54
*** nanzha has joined #openstack-nova01:55
*** brinzhang has joined #openstack-nova02:03
*** brinzhang_ has quit IRC02:06
*** markvoelker has joined #openstack-nova02:14
*** nanzha has quit IRC02:18
*** markvoelker has quit IRC02:18
*** nanzha has joined #openstack-nova02:19
*** mkrai_ has joined #openstack-nova02:21
*** brinzhang_ has joined #openstack-nova02:23
*** brault has quit IRC02:24
*** brault has joined #openstack-nova02:25
*** brinzhang has quit IRC02:26
*** mkrai_ has quit IRC02:41
*** mkrai__ has joined #openstack-nova02:41
*** brinzhang has joined #openstack-nova02:42
*** brinzhang_ has quit IRC02:45
*** slaweq has joined #openstack-nova02:48
*** spsurya has joined #openstack-nova02:50
*** slaweq has quit IRC02:53
*** mkrai__ has quit IRC02:57
*** mkrai__ has joined #openstack-nova02:58
*** dave-mccowan has quit IRC03:05
*** mkrai__ has quit IRC03:32
*** mkrai_ has joined #openstack-nova03:32
*** brinzhang_ has joined #openstack-nova03:36
*** psachin has joined #openstack-nova03:36
*** brinzhang has quit IRC03:39
*** slaweq has joined #openstack-nova03:49
*** slaweq has quit IRC03:54
*** gbarros has quit IRC03:56
*** brault has quit IRC03:56
*** brault has joined #openstack-nova03:56
openstackgerritzhufl proposed openstack/nova master: [Trivial]Add missing ws between words  https://review.opendev.org/68992103:58
*** spatel has quit IRC04:02
*** spatel has joined #openstack-nova04:02
*** spatel has quit IRC04:03
*** udesale has joined #openstack-nova04:04
*** gbarros has joined #openstack-nova04:32
*** gbarros has quit IRC04:36
*** nanzha has quit IRC04:43
*** nanzha has joined #openstack-nova04:43
*** slaweq has joined #openstack-nova04:50
*** slaweq has quit IRC04:54
*** ileixe has joined #openstack-nova04:57
*** gbarros has joined #openstack-nova05:07
*** ratailor has joined #openstack-nova05:09
*** mkrai_ has quit IRC05:13
*** mkrai__ has joined #openstack-nova05:13
*** ralonsoh has joined #openstack-nova05:18
*** gbarros has quit IRC05:19
*** avolkov has joined #openstack-nova05:24
*** dakshina-ilangov has joined #openstack-nova05:24
*** slaweq has joined #openstack-nova05:29
*** janki has joined #openstack-nova05:30
*** brinzhang has joined #openstack-nova05:39
*** brinzhang_ has quit IRC05:42
*** slaweq has quit IRC05:59
*** brinzhang_ has joined #openstack-nova06:00
*** brinzhang has quit IRC06:04
*** nanzha has quit IRC06:15
*** nanzha has joined #openstack-nova06:15
*** rcernin has quit IRC06:16
*** markvoelker has joined #openstack-nova06:17
*** sapd1 has joined #openstack-nova06:21
*** markvoelker has quit IRC06:21
*** ttsiouts has joined #openstack-nova06:22
*** dpawlik has joined #openstack-nova06:27
*** igordc has joined #openstack-nova06:34
*** brinzhang has joined #openstack-nova06:34
*** ircuser-1 has quit IRC06:35
*** brinzhang_ has quit IRC06:38
*** ccamacho has joined #openstack-nova06:41
*** pcaruana has joined #openstack-nova06:47
*** brinzhang_ has joined #openstack-nova06:48
*** igordc has quit IRC06:48
*** mkrai__ has quit IRC06:48
*** brinzhang has quit IRC06:52
*** brinzhang_ has quit IRC06:53
*** trident has quit IRC06:53
*** damien_r has joined #openstack-nova06:53
*** damien_r has quit IRC06:53
*** damien_r has joined #openstack-nova06:54
*** trident has joined #openstack-nova06:58
*** dpawlik has quit IRC07:01
*** dpawlik has joined #openstack-nova07:02
*** slaweq has joined #openstack-nova07:04
*** udesale has quit IRC07:06
*** udesale has joined #openstack-nova07:06
*** larainema has joined #openstack-nova07:10
*** sridharg has joined #openstack-nova07:12
*** mjozefcz|lunch has joined #openstack-nova07:12
*** ttsiouts has quit IRC07:26
*** jawad_axd has joined #openstack-nova07:26
*** ttsiouts has joined #openstack-nova07:27
*** ttsiouts has quit IRC07:28
*** ttsiouts has joined #openstack-nova07:28
*** mjozefcz|lunch is now known as mjozefcz07:34
*** mjozefcz is now known as maciejjozefczyk07:34
*** jawad_axd is now known as gchristian07:35
*** jawad_axd has joined #openstack-nova07:36
*** trident has quit IRC07:40
*** udesale has quit IRC07:41
*** udesale has joined #openstack-nova07:42
*** priteau has joined #openstack-nova07:43
*** ratailor_ has joined #openstack-nova07:43
*** trident has joined #openstack-nova07:43
*** ivve has joined #openstack-nova07:43
*** janki has quit IRC07:46
*** ratailor has quit IRC07:46
*** mkrai_ has joined #openstack-nova07:51
*** jangutter has joined #openstack-nova07:52
bauzasgood morning Nova07:52
*** janki has joined #openstack-nova07:52
gibibauzas: good morning07:56
* bauzas just received his SIM card for a 4GB LTE coverage there in China07:57
gibibauzas: which provider did you use?07:57
bauzasgibi: just paid some "MySIM" vendor with amazon marketplace07:58
bauzasI'll see how it goes07:58
bauzas20EUR for 4GB and 10 days07:58
bauzasgibi: https://www.amazon.fr/MySim-pr%C3%A9pay%C3%A9e-donn%C3%A9es-Roaming-Mobile/dp/B07LC32XMB/ref=cm_cr_arp_d_product_top?ie=UTF8 (sorry, French)07:58
gibibauzas: thanks07:59
bauzasmy phone is dual-SIM so I don't bother with having text and phone calls08:01
gibibauzas: sounds like a good plan, my worldsim offers 2 GB for $128 in china08:02
bauzasI could pay some plan to my usual provider08:03
bauzasbut it's waaay more expensive08:03
gibiyeah08:03
bauzasthat said, if I was subscribing to another French provider called 'Free', then I'd get 25GB of data roaming by month, including China08:04
bauzasbut that's 3G (even not 3G+)08:04
bauzasI mean, HSDPA08:05
*** ttsiouts has quit IRC08:07
gibiI think my current local provider offest wordwide (inc China) data 10EUR for 2GB for 3 days which seems usable08:07
*** ttsiouts has joined #openstack-nova08:08
*** osmanlicilegi has left #openstack-nova08:09
bauzasthat's indeed reasonable, tho a bit expensive08:09
bauzasgibi: the fact is, my hotel is far from the conference venue plus I'll have the week-end there08:10
alex_xu$128 is crazy08:10
bauzasgibi: I can't just rely on Wifi connections08:10
gibialex_xu: indeed it is08:11
gibibauzas: yeah, I also have the weekend before the summit in china08:11
gibiarriving saturday morning08:11
*** ttsiouts has quit IRC08:12
bauzasgibi: heh, same here :)08:12
bauzasgibi: https://www.kayak.fr/trips/!B8vXZqcWMAbgwkrd08:12
gibithat saturday will be crazy to survive after the long flight08:12
*** ttsiouts has joined #openstack-nova08:13
bauzasI plan to sleep on plane08:15
bauzasI did it for going to Sydney as I made one stop in HK for the day08:15
bauzasit worked quite good08:15
gibibauzas: I arrive with OS75 to PVG, landing at 06:50 in China08:15
openstackgerritAdrian Chiris proposed openstack/nova master: Revert "vif: Resolve a TODO and update another"  https://review.opendev.org/67577608:15
bauzasdidn't know OpenStack was running as airline :)08:16
gibiAustrian Airlines08:16
gibi:)08:16
gibibut the ticker fits well :)08:16
bauzas:p08:16
bauzasI arrive a bit later, around 9:10am08:16
bauzasso I mostly expect some visits during the afternoon hopefully08:17
*** tesseract has joined #openstack-nova08:21
*** tesseract has quit IRC08:21
openstackgerritdo3meli proposed openstack/nova master: [Trivial] Add missing ws between words  https://review.opendev.org/68992108:22
bauzasalex_xu: could you please make me a favor and tell me whether https://www.sosh.fr works in China ? I just wanna make sure I can reach my phone provider just in case08:23
alex_xubauzas: just thinking how to ensure that, asking the local telco provider?08:25
*** mkrai_ has quit IRC08:26
bauzasalex_xu: yeah, for example08:27
bauzasI'm pretty sure it works08:27
bauzasbut I'm not sure08:27
alex_xubauzas: hah, let me try to call china mobile client service08:28
*** jawad_axd has quit IRC08:28
*** andyzon has joined #openstack-nova08:29
openstackgerritMerged openstack/nova master: Func: bug 1849165: mig race with _populate_assigned_resources  https://review.opendev.org/68986608:31
openstackbug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried)08:31
*** dpawlik has quit IRC08:31
alex_xubauzas: china mobile tell me you should ask SOSH :)08:32
*** sapd1 has quit IRC08:33
bauzasalex_xu: heh, no worries08:34
bauzasthanks for the call08:34
bauzasI just meant accessing the website08:34
alex_xuhah08:34
bauzassorry for the confusion08:34
alex_xuno worries08:34
bauzasbut I definitely appreciate you called your service provider !08:35
bauzasappreciated*08:35
bauzaslooks we both misunderstood :)08:35
alex_xuno, I understand you :)08:35
bauzasalex_xu: FWIW, I can reach the customer service by some internal chat app, that's why I just wanted to make sure it could work08:36
*** derekh has joined #openstack-nova08:37
alex_xuoh, so you need network first to reach the service08:37
alex_xubauzas: but remember don't depend on google maps for travel, that may not works in china ;)08:38
bauzasalex_xu: yup, I know, hence my wonders about whether Sosh.fr works there08:41
bauzasanyway08:41
bauzasalex_xu: which application do you use for maps and directions in Shanghai, including routes and public transportation ?08:41
bauzasdo you s/use/recommend08:42
bauzasmy bad08:42
bauzasapparently this time, the Foundation doesn't provide some tips&tricks for navigating around Shanghai08:42
alex_xuI use local internet commpany apps, like baidu, alibaba, I'm not sure they have multiple language, but the Apple maps and bing maps should works.08:43
alex_xubauzas: but you can try Baidu Maps, see if they works for franch or english, if not, bing maps or apple maps(if you use iphone) is safe choice I think08:44
*** ricolin_ is now known as ricolin08:49
bauzasalex_xu: I have an Android-based phone (Huawei)08:50
bauzasbut I'll figure that out then08:50
bauzasmy point is, I won't be able to use the Play Store while in China for downloading the apps that I want :D08:51
bauzashence some bits of preparation08:51
alex_xuyea, good to download the apps first08:52
alex_xuwhen I travel to US, the app store is connecting back to china server, that is crazy slow...08:52
*** tssurya has joined #openstack-nova08:53
bauzasanyway, a good opportunity for de-googling myself :)08:54
*** dakshina-ilangov has quit IRC08:54
*** dtantsur|afk is now known as dtantsur08:56
*** xek has joined #openstack-nova09:00
*** dpawlik has joined #openstack-nova09:01
openstackgerritkangyufei proposed openstack/python-novaclient master: Switch to Ussuri jobs  https://review.opendev.org/69001309:05
*** ricolin_ has joined #openstack-nova09:15
*** ricolin has quit IRC09:17
*** ricolin_ is now known as ricolin09:20
*** Liang__ has quit IRC09:38
*** dpawlik has quit IRC09:48
*** nanzha has quit IRC09:51
*** mkrai_ has joined #openstack-nova09:52
*** nanzha has joined #openstack-nova09:53
*** dpawlik has joined #openstack-nova10:02
*** janki has quit IRC10:07
*** ociuhandu has joined #openstack-nova10:10
*** ttsiouts has quit IRC10:11
*** derekh has quit IRC10:16
*** pcaruana has quit IRC10:16
*** markvoelker has joined #openstack-nova10:19
*** pcaruana has joined #openstack-nova10:21
*** mkrai_ has quit IRC10:24
*** markvoelker has quit IRC10:24
*** ttsiouts has joined #openstack-nova10:40
*** SonPham has joined #openstack-nova10:40
*** SonPham has quit IRC10:42
*** SonPham has joined #openstack-nova10:44
SonPhamHi. I want response data from nova (server.py) to nova-pythonclient after nova-pythonclient request to nova10:44
SonPhamHow can i do it10:44
*** tbachman has quit IRC10:44
SonPhamHi. I want response data from nova (server.py) to novaclient after novaclient request to nova10:44
*** SonPham has quit IRC10:45
*** ttsiouts has quit IRC10:45
sean-k-mooneybauzas: hi can you review this quickly, i tried to get this in to the train RCs  but i think if fell through the cracks so it would be nice to land and backport it quickly  https://review.opendev.org/#/c/675776/710:47
*** SonPham has joined #openstack-nova10:48
*** derekh has joined #openstack-nova10:53
*** ttsiouts has joined #openstack-nova11:05
*** damien_r has quit IRC11:09
*** udesale has quit IRC11:18
*** bbowen has quit IRC11:30
*** tonyb has joined #openstack-nova11:31
*** dpawlik has quit IRC11:31
*** xek has quit IRC11:37
*** xek has joined #openstack-nova11:37
*** dpawlik has joined #openstack-nova11:43
*** ratailor__ has joined #openstack-nova11:43
*** ttsiouts has quit IRC11:44
*** ttsiouts has joined #openstack-nova11:44
*** dviroel has joined #openstack-nova11:46
*** ratailor_ has quit IRC11:46
*** SonPham has quit IRC11:50
*** ratailor__ has quit IRC11:53
*** dpawlik has quit IRC11:55
*** markvoelker has joined #openstack-nova12:01
*** dpawlik has joined #openstack-nova12:01
*** belmoreira has joined #openstack-nova12:03
efriedgibi: can I get a quick +A on https://review.opendev.org/#/c/688979/ please?12:05
gibiefried: looking...12:05
efriedyou were +2 before the reno12:07
efriedOh, I guess the predecessor still needs reviews12:07
*** maciejjozefczyk has quit IRC12:07
gibiI'm +A on the trait applying patch12:08
* gibi is on a meeting in parallel12:08
efriedgibi: +2 on the ItemsMatcher fup. If you want to +2 the bottom one, maybe we can convince bauzas to +A both and all three of these will merge.12:09
*** spatel has joined #openstack-nova12:09
*** damien_r has joined #openstack-nova12:11
*** spatel has quit IRC12:15
gibiefried: I can add some unit tests for that fup. But as you are happy with the fup I'm happy with your original patch.12:16
*** tbachman has joined #openstack-nova12:17
efriedgibi: like I said, I think we can live without test-for-test-util; when we use it in tests we'll uncover any bugs and fix them then. And it won't impact prod.12:18
efriedbut if you're losing sleep over it, I'll happily review tests :)12:19
*** dpawlik has quit IRC12:20
*** maciejjozefczyk has joined #openstack-nova12:22
gibiefried: sure, I got your comment. I would like to add test cases12:22
gibiefried: if the patch merges before I can add test cases then I will add those in a separate patch12:23
efriednow we need to get that spy stuff merged12:23
gibiefried: that will be harder I think12:23
efriedbeen at least three or four cases that could have made use of it.12:23
*** derekh has quit IRC12:23
efriedsince it was proposed12:23
*** eharney has quit IRC12:25
gibiefried: I think I saw two of those cases. It think not everybody sees the same value we thing the spy adds12:26
efriedgibi: I'm almost convinced it should be added to the mock library itself. If we did that, we could probably iron out those last pesky wrinkles that are annoying us...12:27
*** ttsiouts has quit IRC12:27
*** ttsiouts has joined #openstack-nova12:28
*** ttsiouts has quit IRC12:29
*** ttsiouts has joined #openstack-nova12:29
*** nanzha has quit IRC12:30
*** larainema has quit IRC12:32
*** nanzha has joined #openstack-nova12:34
*** dpawlik has joined #openstack-nova12:36
*** mdbooth has quit IRC12:37
*** mdbooth has joined #openstack-nova12:39
*** bbowen has joined #openstack-nova12:40
openstackgerritAlex Xu proposed openstack/nova master: Add test for the race between post live migration and update available resource  https://review.opendev.org/69006612:40
alex_xuefried: ^ I feel the race is case due to the post live migration, not the pre live migration12:41
efriedalex_xu: ack, I was about to get around to trying to grok your comment :)12:41
efriedDid you see the repro though? That triggers the race by running the periodic before the dest claim.12:42
*** bbowen_ has joined #openstack-nova12:42
alex_xuefried: yes, but I think that is due to the no numa instance without migration context, I feel it isn't race problem12:42
alex_xuefried: we assign migration.dest and instance.migration_context under the same lock, so it shouldn't happen race12:43
efriedOnly on cold migration12:43
efriedon live migration the former happens on the conductor, the latter on the compute.12:44
alex_xuoh, I only see LM, didn't look at cold one12:44
*** bbowen has quit IRC12:44
alex_xubut it should be same, since both cold migration and LM using same move claim12:44
alex_xuefried: here is https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L306 and https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L37012:46
*** dtantsur is now known as dtantsur|brb12:47
efriedalex_xu: cold migration uses _claim_existing_migration but live migration uses _create_migration -- or the other way around, now I can't remember :(12:49
alex_xureally...i will hate that12:50
*** spatel has joined #openstack-nova12:50
alex_xuefried: quick grep the code, both cold and live using _claim_existing_migration12:51
alex_xuthey create the migration obj in conductor12:51
efriedalex_xu: If you can, probably best to wait for mriedem -- he understands these paths way better than I do.12:52
alex_xuefried: yea, no hurry, whatever the fix is right12:52
efriedI put together these patches to further my understanding, but he was leading the way12:52
*** nanzha has quit IRC12:53
efriedLike, mission accomplished, I understand things a little bit better than I did before, but it's still like |...me..................mriedem..|12:53
*** derekh has joined #openstack-nova12:54
*** spatel has quit IRC12:55
efriedbtw, I ran into the same COMPUTE_RESOURCE_SEMAPHORE deadlock while I was experimenting yesterday, and considered working around it by manually calling _populate_assigned_resources -- but I didn't think that was a realistic test.12:55
alex_xuyea...that deadlock spend hours for me, totally not worth...12:56
efriedoriginally I was using a cold migration and injecting update_available_resource before _prep_resize, but that didn't trigger the problem. So I tried to inject it lower down, after _claim_existing_migration, and ran into the deadlock.12:59
efriedThat's when mriedem suggested I instead use a live migration and inject before pre_live_migration instead.13:00
efriedThat reproduced the race13:00
efriedor... *a* race, anyway :)13:00
*** nanzha has joined #openstack-nova13:01
*** nweinber__ has joined #openstack-nova13:02
*** xek has quit IRC13:03
*** spatel has joined #openstack-nova13:03
*** xek has joined #openstack-nova13:05
*** mriedem has joined #openstack-nova13:12
bauzassean-k-mooney: ah sorry, was at the gym, will review it ASAP13:13
*** eharney has joined #openstack-nova13:14
*** derekh has quit IRC13:14
openstackgerritGhanshyam Mann proposed openstack/nova master: Fix policy doc for host_status and extended servers attribute  https://review.opendev.org/68983313:16
openstackgerritGhanshyam Mann proposed openstack/nova master: Fix policy doc for host_status and extended servers attribute  https://review.opendev.org/68983313:16
mriedemhear ye hear ye we need stable/stein reviews https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/stein13:16
bauzasmriedem: ack13:17
bauzassean-k-mooney: urgent revert, I guess ?13:17
bauzassean-k-mooney: I can fast-approve it as per our policy13:17
sean-k-mooneybauzas: its not a pure revert13:21
sean-k-mooneye.g. we updated the comments to reflect our understanding of why we cant make the chagne13:21
sean-k-mooneybauzas: i also want to see the meloxn ci run an pass on it13:22
bauzassean-k-mooney: ergh ok13:22
bauzasI'd have preferred a classic revert so we could spin another change13:22
bauzasanyway, it's done13:22
sean-k-mooneywell we could just do a strait revert13:23
sean-k-mooneyand then apply this after on master to fix the comments13:23
sean-k-mooneywithout the patch we get a traceback in nova while a vm is running with a macvtap interface13:24
sean-k-mooneyit does not break the vm13:24
sean-k-mooneyone of the periodic task that track resouce usage is unhappy13:24
bauzascool13:27
*** sridharg has quit IRC13:29
*** belmoreira has quit IRC13:29
*** udesale has joined #openstack-nova13:31
*** bbowen_ has quit IRC13:32
*** belmoreira has joined #openstack-nova13:37
*** xek has quit IRC13:40
*** xek has joined #openstack-nova13:40
*** damien_r has left #openstack-nova13:42
mriedemgmann: in a multinode job, was is the "peers" group for hosts? and why would that need to be configured over just the "subnode" group vars to get configuration into the subnode compute?13:44
gmannmriedem: i think that was old name i remember, later we did 'peers' -> 'subnode'13:45
*** jmlowe has quit IRC13:46
*** ociuhandu has quit IRC13:48
mriedemok i was wondering why we have both subnode and peers configured in some multinode jobs and it sounds like we could just move all the 'peers' config to 'subnode' vars config13:48
mriedeme.g. https://github.com/openstack/nova/blob/master/.zuul.yaml#L340-L35213:49
*** trident has quit IRC13:50
*** ociuhandu has joined #openstack-nova13:50
gmannlet me check if any other reason to keep that.13:50
openstackgerritMatt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3  https://review.opendev.org/68398813:51
*** mlavalle has joined #openstack-nova13:51
*** trident has joined #openstack-nova13:54
mriedemefried: want to do the backports for https://review.opendev.org/#/c/689842/ and the func test patch below it?13:57
*** ociuhandu has quit IRC13:58
mriedemsean-k-mooney: i'm assuming you'll be backporting this revert to train yeah? https://review.opendev.org/#/c/675776/13:59
*** jmlowe has joined #openstack-nova13:59
mriedemnot sure why that wasn't flagged for train-rc-potential14:00
*** belmoreira has quit IRC14:00
mriedemso we have 2 train regressions identified and fixed already, we should probably do a patch release soon14:03
*** nweinber_ has joined #openstack-nova14:04
mdboothThis is weird. utils.synchronized(instance.uuid) seems to be ignored in my functional test14:04
sean-k-mooneymriedem: yes i can do that if adrianc does not14:04
mdboothi.e. I'm hitting breakpoints in critical sections locked by the same key concurrently14:05
sean-k-mooneymdbooth: we might be mocking the lock14:05
sean-k-mooneyor the file acess14:05
mdboothsean-k-mooney: Yeah, I was hoping somebody would shout if that was the case :) However, that's definitely where I'm heading next.14:06
mdboothBut why would we do that?14:06
mriedemhttps://github.com/openstack/nova/blob/master/nova/test.py#L23914:06
sean-k-mooneymdbooth: we make most things serial in the fucntional tests14:06
mdboothmriedem: Thanks! Why would you ever turn that off, btw?14:06
sean-k-mooneyso i guess locking is not always needed14:07
*** belmoreira has joined #openstack-nova14:07
mdboothExcept if you didn't want it to work14:07
mriedemsee the note above the condition14:07
*** nweinber__ has quit IRC14:07
mdboothmriedem: Guessing that was written in the days before functional/ ?14:08
sean-k-mooneyoh i guess it would also cause issues because we use the same instance uuids14:08
sean-k-mooneye.g. for parallel test executions on the same host14:08
mdboothBecause non-test code uses locks, and we're not going to 'fix' that code to not use locks14:08
mdboothJust so we can execute it from a test14:08
mdboothmriedem: For reference, the locks which are being ignored are the locks in _do_build_and_run_instance, and terminate_instance in ComputeManager14:10
mriedemmdbooth: it's not before functional tests existed14:11
mdboothMy test assumes that the locks works as expected. It would be weird to expect otherwise.14:11
sean-k-mooneymdbooth: most fucntional tests assume that there are not cuncuretn action taking place14:12
mdboothIt's kinda hard to write a test for a delete race when locks have different semantics in tests.14:12
sean-k-mooneysince all the service runn in the same interperter instnace14:12
mdboothRight. You can still have locks, though.14:12
mriedemi would prefer that we don't have tests that have to rely on locks14:12
sean-k-mooneyyes but they would not race with anything14:12
mdboothWell the locks are critical to the semantics of the code14:13
sean-k-mooneyright but in any code path we should not have a singel action reneter the same critical section multiple times right14:13
mdboothSpecifically, in this case they prevent the delete from completing while it is still building, which is the actual behaviour.14:13
sean-k-mooneyso the lock would always be aqired and never block14:13
mdboothIf we disable locks, I have to write a race test which assumes that an instance can be completely deleted while it is still being built.14:14
mdboothThat's obviously possible, but it's wrong.14:14
mdboothi.e. possible to write.14:14
mdboothBut I'd be testing something which can't happen on a real system.14:14
sean-k-mooneymdbooth: well the instnace can at least be partly deleted while its building14:14
mdboothsean-k-mooney: Absolutely, yes.14:14
mdboothsean-k-mooney: That's the race I'm writing a test for.14:14
sean-k-mooneyyep this is the thing stephen was working on14:15
sean-k-mooneydid you see the message i left14:15
mdboothsean-k-mooney: Did you leave it since this morning?14:15
sean-k-mooneyno14:15
mdboothsean-k-mooney: Then yeah. I've already written the test, though ;)14:15
mdboothI've been badgering you to review it for a couple of days :P14:16
* mdbooth is updating it following a suggestion from mriedem14:16
sean-k-mooneyyes i know :) and i will look at it now14:17
mdboothIn general it's better, but not having functioning locks would be super weird.14:17
mdboothIt would basically invalidate the test.14:17
mdboothsean-k-mooney: Not worth it right now. I'm basically rewriting it.14:17
*** ociuhandu has joined #openstack-nova14:17
sean-k-mooneyok so you tried mocking _update_instance_after_spawn to do the api delete and now your rewriting it based on mriedem suggestion14:18
sean-k-mooneyhavent read all the comments since then but is see there is discussion on the patch14:19
*** artom has quit IRC14:19
mdboothsean-k-mooney: My previous test worked fine. mriedem wants to deprecate IntegratedTestBase, thouogh, so requested a rewrite which doesn't use it.14:19
*** andyzon has quit IRC14:19
mriedemi just left more comments14:19
sean-k-mooneyah ok14:19
mriedemyou're making this unnecessarily libvirt-specific when the fix isn't14:20
mriedemand relying on a refactor which is going to muddy backports14:20
mdboothmriedem: Yeah, my rewrite also ditches libvirt14:20
mriedemiow you're making this harder than it needs to be imo14:20
mriedemok good14:20
*** andyzon has joined #openstack-nova14:20
sean-k-mooneymriedem: so on that should i consume mdbooth refactor or avoid it for https://review.opendev.org/#/c/687957/314:20
sean-k-mooneygive i do want to backport it would you be ok with backporting https://review.opendev.org/#/c/689186/214:21
mriedemit's common sense to avoid depending on as many things as possible if you intend on backporting a patch14:21
gmannmriedem: you can merge the peers and subnode groups setting. it was merged in base job also. Initially it was separate job for single and multinode jobs but during rocky PTG we discussed to merge both and have a single base job doing same network setting via subnode var14:21
mriedemsean-k-mooney: your fix is also not libvirt specific14:22
mriedemso i'm not sure why you need to depend on the libvirt functional test base stuff14:22
mriedemyou should be able to write isolated functional tests for the rebuild + numa stuff14:22
mdboothsean-k-mooney: I've pulled the non-placement bits out of the test mriedem pointed to. So it's still a small refactor, but I wanted to ability to not pull in a ton of placement stuff I don't need.14:22
sean-k-mooneythat is a good point. i only depended on it because that is the only numa fucntional test we have14:22
sean-k-mooneybut you are right i could proably  remove all the libvirt stuff14:22
sean-k-mooneyi should think about that14:23
*** igordc has joined #openstack-nova14:23
* mdbooth needs still needs the locks in computemanager to work, though :/14:24
sean-k-mooneymdbooth: well we cant ignore placement on rebuild in general14:24
sean-k-mooneybut yes its not relevent to what you are fixing/testing14:24
mriedemif a test needs to create a server then you are going to require the placement fixture somewhere in your functional test14:24
mdboothYeah. The resulting placement stuff could be a mixin, tbh.14:24
mriedemotherwise you can't get through the scheduler14:24
sean-k-mooneyits also not technicall directly relevent to what im testing but if the traits chagned on rebuilt that in principal would need to be validated with placement14:25
mdboothmriedem: Right, it's all the placement helper stuff which isn't needed in general.14:25
*** andyzon has quit IRC14:25
sean-k-mooneymriedem: yes14:25
mdboothI'm still starting placement.14:25
mriedemmdbooth: if you need locks in the test you just add REQUIRES_LOCKING = True right?14:25
mdboothmriedem: Possibly, but I also didn't expect this lock to be filesystem based. I'm going to investigate thoroughly before assuming it's all ok.14:26
sean-k-mooneymdbooth: the syconise decorators are always filesystem based14:27
mdboothsean-k-mooney: I thought that was only for external=True14:27
* mdbooth is reading the code, anyway14:28
mdboothre-reading14:28
sean-k-mooneyoh am maybe14:28
gibifyi tomorrow is a national holiday here in Hungary so I will be off tomorrow14:29
*** gibi is now known as gibi_off14:29
mriedemdansmith: i've abandoned my attempt at fixing https://bugs.launchpad.net/nova/+bug/1815082 but you might be interested in that14:29
openstackLaunchpad bug 1815082 in OpenStack Compute (nova) rocky ""DBNonExistentTable: (sqlite3.OperationalError) no such table: services" when starting nova-metadata under uwsgi" [Medium,Triaged]14:29
sean-k-mooneymdbooth: i guess it default to false https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L28414:29
mriedemdansmith: tl;dr trying to get the indirection api working in n-api-meta in certain cases if we don't have direct access to the db14:30
openstackgerritGhanshyam Mann proposed openstack/nova master: Fix policy doc for host_status and extended servers attribute  https://review.opendev.org/68983314:30
mriedemer i guess we use the indirection api for n-api-meta when running with evetlet but not wsgi14:30
*** dpawlik has quit IRC14:30
dansmithmriedem: hmm, what you have seems like it should work, although your comment about it hitting superconductor isn't right, unless nova-cpu.conf has api db creds14:32
dansmithmriedem: I could pick up on that, but unless someone is beating down the door for it and/or going to test it prior to merge, I'd probably set it aside too14:33
*** andyzon has joined #openstack-nova14:33
efriedmriedem: I can do those backports, yeah.14:33
*** slaweq has quit IRC14:33
sean-k-mooneymriedem: before i go refactor my own work you would prefer i use ProviderUsageBaseTestCase instaead of _IntegratedTestBase and make it non libvirt specific14:33
mriedemjroll: i have given up on https://review.opendev.org/#/c/637217/ - someone else that cares about completing the ironic flavor data migration added in pike is going to have to take over14:33
*** maciejjozefczyk has quit IRC14:34
mriedemdansmith: i had a tempest patch that depended on the nova fix to test it14:34
mriedemhttps://review.opendev.org/#/q/status:abandoned+topic:bug/181508214:34
mriedemdansmith: looking at latest comments the issue was n-api-meta was starting before conductor which blew up14:35
*** jaosorior has joined #openstack-nova14:35
mriedemfrickler: ^ i've abandoned those changes for your bug fyi14:36
mriedemsean-k-mooney: yes don't use _IntegratedTestBase for new tests14:36
sean-k-mooneyok thanks14:36
mriedemand yes it'd be cool if things that aren't libvirt specific didn't build on libvirt specific functional test base classes14:37
sean-k-mooneyya i think i can create a small numa file that is not in the libvirt folder to test this14:37
sean-k-mooneyalthough there are few non libvirt numa things that will likely be added to it14:38
*** maciejjozefczyk has joined #openstack-nova14:38
jrollmriedem: thanks for the heads up14:39
mriedemsean-k-mooney: just my opinion, don't bend over backward based on something i'm not actively reviewing14:41
*** dtantsur|brb is now known as dtantsur14:42
*** belmoreira has quit IRC14:43
sean-k-mooneymriedem: ok be even so i tend to give you opipion weight even if you are not activly reviewing it since others often agree14:46
*** belmoreira has joined #openstack-nova14:47
*** belmoreira has quit IRC14:47
*** cdent has joined #openstack-nova14:48
mdboothWhoa. It appears from tracing in oslo.concurrency that something is allowing a threading.Semaphore to be taken twice14:49
mdboothI wonder... is threading.Semaphore re-entrant by default?14:49
cdentthose have to be some most mdboothy statements I've ever seen14:50
cdent"hey, everything might be wrong!"14:50
cdentwhich I, of course, love14:51
mdboothcdent: I usually hope to follow them up by being wrong, but I like to think them first :)14:51
mdboothAnd unfortunately I'm not always wrong.14:51
*** xek_ has joined #openstack-nova14:52
bnemecIt seems highly unlikely that a concurrency class would be non-reentrant.14:53
mdboothbnemec: s/non-//14:53
mdboothbnemec: I would *not* expect Semaphore to be re-entrant, and certainly in a regular environment it isn't:14:54
*** xek has quit IRC14:54
bnemecAre you sure the tracing is in the right place? We spent a bunch of time tracking down a double-locking problem at one point that was caused by logging in the wrong place.14:54
mdbooth>>> import threading                                                                                                                                                   >>> s = threading.Semaphore()14:54
mdbooth>>> with s:14:54
mdbooth...   with s:                                                                                                                                                          ...     print("Hello world")14:54
mdbooth...14:54
mdboothEurgh14:54
mdboothSorry14:54
mdboothAnyway, takes same semaphore twice, hangs14:55
bnemecOh, well no that wouldn't work. But that's not what reentrancy is.14:55
mdboothbnemec: Wouldn't rule it out, but I don't think so. It's in a functional test, though, so my prime suspect is environmental.14:55
bnemecReentrancy has to do with concurrent execution, which you can't get in a single thread.14:55
mdboothbnemec: You might want to look that up ;)14:56
bnemecI did. Did you? https://en.m.wikipedia.org/wiki/Reentrancy_%28computing%2914:56
bnemec"In computing, a computer program or subroutine is called reentrant if multiple invocations can safely run concurrently."14:56
mdboothhttps://docs.python.org/3/library/threading.html#semaphore-objects14:56
mdboothhttps://en.wikipedia.org/wiki/Reentrant_mutex14:57
bnemecAh, we're talking about different things.14:57
*** pcaruana has quit IRC14:58
mdboothSo, Semaphore doesn't appear to be mocked, and yet I see 2 enters followed by 2 exits14:59
* mdbooth strokes chin15:00
openstackgerritEric Fried proposed openstack/nova stable/train: Func: bug 1849165: mig race with _populate_assigned_resources  https://review.opendev.org/69009915:00
openstackbug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried)15:00
mdboothbnemec: To be clear, I'm *not* expecting to find a bug in oslo.concurrency. I'm expecting to find a fixture somewhere doing something I haven't thought of yet.15:01
*** ratailor has joined #openstack-nova15:03
mriedemsmells like a bro down in here https://www.youtube.com/watch?v=CHnbC1N_G6w15:03
*** ratailor has quit IRC15:03
mdboothOh, great, it's a heisenbug >:(15:05
*** slaweq has joined #openstack-nova15:06
adriancsean-k-mooney, i can do the backport for https://review.opendev.org/#/c/675776/, however it will take me some time to get a setup up to test it15:07
bnemecmdbooth: By any chance are you running tests in parallel? The external lock fixture from oslo.concurrency would make the locks independent.15:08
*** mmethot has quit IRC15:09
openstackgerritDan Smith proposed openstack/nova master: Add image precaching docs for aggregates  https://review.opendev.org/68734815:09
*** mmethot has joined #openstack-nova15:09
openstackgerritEric Fried proposed openstack/nova stable/train: Don't populate resources for not-yet-migrated inst  https://review.opendev.org/69010015:11
efriedthere must be an easier way to cherry pick a series15:11
*** ivve has quit IRC15:13
*** otsukahy has joined #openstack-nova15:13
*** otsukahy has quit IRC15:13
*** trident has quit IRC15:13
*** maciejjozefczyk has quit IRC15:13
*** otsukahy has joined #openstack-nova15:14
*** ttsiouts has quit IRC15:14
sean-k-mooneyefried: cerry picking a serise is baically a rebase15:15
sean-k-mooneydo you mean for backporting?15:15
efriedyes15:15
sean-k-mooneyah15:15
sean-k-mooneyi was refering to a feature branch before15:16
sean-k-mooneythere is proably a way to do it in like 1 comamnd in git but i dont know what it is15:16
sean-k-mooneyoh git cherry-pick takes a list of commit15:17
sean-k-mooneyso if you do a git log betwen the base of the serise and top and just get the commits you shuld be able to pipe the revers of that to reverse and xargs to invoke the cherry pick15:18
efriedwill that pile them up in a series?15:21
efriedthat's probably the ticket, then. I'll try that next time.15:21
sean-k-mooneyi think so also looking at the exmaple in the git man pages15:21
sean-k-mooneyyou might be able to do somting like this15:21
sean-k-mooneygit cherry-pick start..end15:21
dansmithcan't you do a range?15:21
sean-k-mooneywhere start and end are two comites15:21
dansmithyeah that15:21
dansmithworks for any other git command that takes multiple commits15:22
dansmithif you have them all in a branch, then something like "git cherry-pick -x branch~4.." probably works15:22
sean-k-mooneythey have an example fo "git cherry-pick ..master"15:22
sean-k-mooneythat cherrypick anything on master but not locally15:22
sean-k-mooneyso i assuem you can close the range15:22
openstackgerritMatt Riedemann proposed openstack/nova master: Require Migration object arg to migrate_instance_finish method  https://review.opendev.org/69010615:23
umbSublimeI have a specific workload where I need to enable invTSC, for this to work with migrations I also need to set the tsc frequency. How can I configure the tsc frequency in nova.conf on my compute hosts (is it possible with cpu_model_extra_flags)? in qemu it looks like this: "<clock><timer name='tsc' frequency='3000000000'/></clock>"15:24
sean-k-mooneyumbSublime: im pretty sure you cant15:25
*** nsherry4 has joined #openstack-nova15:25
sean-k-mooneyyou could enabel a specific cpu flag but i dont think you can spcify the frequency15:26
*** nsherry4 has left #openstack-nova15:26
*** trident has joined #openstack-nova15:26
*** slaweq has quit IRC15:28
umbSublimeWould the solution in this case be to create a custom CPU model and use that in nova.conf ?15:29
umbSublimeor is there an alternative workaround I didn't think about15:29
sean-k-mooneywell this is really a feature request so i would start with the uscase15:29
sean-k-mooneyyou are trying to fake a specifc cpu frequece and provide a "stable" tsc to the guest correct?15:30
umbSublimeI'm only the operator for this cloud. But the use case is a software that requires that flag. Only issue is with only the flag enbaled and the frequency unset, migrations are not possible15:30
umbSublimedue to the nature of the flag15:30
sean-k-mooneyso that if you migrate betwwen hosts with differen cpu fequecne it does not chagne15:30
*** TxGirlGeek has joined #openstack-nova15:31
umbSublimeexactly there are some extensive discussions about that and how it was implemented in qemu/libvirt15:31
umbSublimewhen you set a specific frequency, there is a calculation made so that migration is possible and the value of the register is mirrored over15:31
sean-k-mooneyya so nova dose not currently suport hat15:32
sean-k-mooney*that15:32
umbSublimeAll I know is the software they run on the guest requires that invTSC is presented to the guest. From my operation pov I also need the VM to be migratable15:32
sean-k-mooneywe could ad support if we had a defied usecase15:33
sean-k-mooneydo they need invTSC set or do the want the TSC to be marked as reliable in the vm15:33
umbSublimeIt's a game engine software so using other counters isn't really possible15:33
umbSublimethey need invTSC to be set15:33
*** gyee has joined #openstack-nova15:33
sean-k-mooneyok so setting tsc=reliable on the kernel commandline wont be viable15:34
*** udesale has quit IRC15:34
*** psachin has quit IRC15:34
sean-k-mooneyim basicaly wonder if https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/kernel-parameters.txt#L4832-L483815:35
umbSublimeWould the custom cpu_model be a working temporary solution  ?15:35
sean-k-mooneywas another alternitive15:35
umbSublimeOhh I didn't mention it's a windows guest :/15:35
sean-k-mooneyoh :)15:35
sean-k-mooneyso that would be a know to linux kernel args then15:35
sean-k-mooneyin which case we shoudl be settin ghte hyperv features15:35
sean-k-mooneyif you modified the libvirt xml defition and created a new cpu model it might work15:37
sean-k-mooneybut i doubt it15:37
umbSublimeIf I virsh edit a running vm and add the frequency, then reboot the VM it is migrated just fine by openstack15:37
sean-k-mooneyumbSublime: if you virsh edit it then it will be lost if nova regenerate the xml15:38
sean-k-mooneye.g. via hard reboot or any move opertaiton15:38
umbSublimeYes of course ~!15:38
sean-k-mooneye.g. i would not expect it to survie a migraton15:38
sean-k-mooneyoh15:38
sean-k-mooneyyou mean15:38
umbSublimeIt was just to confirm that migration would work if forcing the freq. The next step was to configure in nova :)15:38
sean-k-mooneybefore you live migrate you remove the invTSC flag15:39
umbSublimenope both the invTSC and freq migrated with the VM15:39
umbSublimeI was actually surprised by that15:39
sean-k-mooneywell we dont update all the xml on live migration15:40
sean-k-mooneywe just do a diff15:40
sean-k-mooneyand modify what needs to be chagned15:40
umbSublimeOk that explains it !15:40
sean-k-mooneybut a hard reboot of the vm would loose any manual change you make15:40
umbSublimeyes absolutely15:40
sean-k-mooneydumb question but do you neeed live migration. or would cold migration be enough15:40
umbSublimethis was just for testing purposes15:40
umbSublimelive-migration would really be the ideal goal (we need sometimes to update compute hosts without impacting guests)15:41
umbSublimeIf this really isn't possible at the moment, I'm sure that aspect could be discussed internally15:42
sean-k-mooneyi think unfotuetly the best path forwad would be to add this as a new feature15:42
sean-k-mooneywhat release of openstack are you running15:42
*** otsukahy has quit IRC15:43
umbSublimeCurrently that stack is on Queens15:43
sean-k-mooneythis is the only reference to invtsc in the nova code base15:44
sean-k-mooneyhttps://github.com/openstack/nova/blob/1ec87c227c4a506ddc3355108a95390cdaa57d62/nova/conf/libvirt.py#L608-L61315:44
*** otsukahy has joined #openstack-nova15:44
umbSublimehehe yes, that's the only one I found also. Most of the work to make exposing that flag to the guest possible was made in qemu/libvirt projects. There many discussions in the MLs about this15:45
sean-k-mooneyumbSublime: can i suggest you file a bluepirnt for this. or maybe a bug fix.15:46
sean-k-mooneyi would like to call it a bug so it could be backproted but its really a feautre since invtsc frequesce configuration was never previously supproted so it shoudl be  a specless blueprint not a bug15:47
*** otsukahy has quit IRC15:48
umbSublimesean-k-mooney: I'll add it to my todo list :)  (I've never done any of those, but would gladly do it)15:49
*** andyzon has quit IRC15:50
umbSublimeThanks a lot helping me out :) really appreciated15:50
sean-k-mooneyno worries. sorry i could not point you at an exisitng knob to tweak15:51
*** davee__ has quit IRC15:51
openstackgerritMatt Riedemann proposed openstack/nova master: Stop converting Migration objects to dicts for migrate_instance_start  https://review.opendev.org/69011515:51
*** davee__ has joined #openstack-nova15:51
*** ircuser-1 has joined #openstack-nova15:54
mriedemdansmith: this is the requirements bump you'll need for your osc change https://review.opendev.org/#/c/690097/15:59
mriedembut the latest novaclient is breaking horizon...15:59
dansmithmriedem: woot, I was just looking for it15:59
*** artom has joined #openstack-nova16:00
dansmithguh16:00
mriedemhttps://bugs.launchpad.net/horizon/+bug/184935116:02
openstackLaunchpad bug 1849351 in OpenStack Dashboard (Horizon) "horizon doesn't work with python-novaclient 16.0.0" [Undecided,New]16:02
*** otsukahy has joined #openstack-nova16:03
*** otsukahy has quit IRC16:03
*** ricolin_ has joined #openstack-nova16:04
cdentI think I know that bug. That's the "horizon needs more maintainers" bug.16:04
dansmithmriedem: do you know what the deal is? brin's list of changes does't include anything that pops out at me16:04
mriedemhttps://docs.openstack.org/releasenotes/python-novaclient/unreleased.html#upgrade-notes16:04
mriedemwe removed the list_extensions stuff16:04
mriedemwhich horizon uses here https://github.com/openstack/horizon/blob/master/openstack_dashboard/api/nova.py#L101816:05
mriedemi'm guessing to enable/disable features based on extensions but since i think liberty everything is enabled and non-configurable16:05
mriedemunless you have out of tree hacks16:05
mriedemanyway, i'm cloning horizon to hack on it16:05
dansmithoh this is more of a bump for them just the latest novaclient release?16:05
*** ricolin__ has joined #openstack-nova16:05
mriedemhorizon just has to remove usage of deprecated novaclient code that is now gone16:06
*** jmlowe has quit IRC16:06
*** ricolin has quit IRC16:07
dansmithnovaclient hash 364cad41912e2c0f99a30f78b2835f3480a18d6e is what I was looking for16:07
dansmithI guess that's why we went from 15 to 1616:07
mriedemefried: ooo hot damn this is working now https://review.opendev.org/#/c/683988/16:07
mriedemdansmith: that and the cells v1 things that were removed also yeah16:07
dansmithack16:08
efried\o/16:08
*** otsukahy has joined #openstack-nova16:09
*** ricolin_ has quit IRC16:09
*** macz has joined #openstack-nova16:12
mriedemsince stephen is MIA until after the summit, i wonder if we should move py27 jobs to experimental so we can run them on-demand but not automatically on every change?16:13
mriedemthat would at least be py27 and functional16:13
mriedemmaybe that's not worth the trouble right now16:13
dansmithshouldn't we change functional-py3whatever to functional and functional to functional-py27 ?16:14
cdentwhy bother?16:14
cdentthat was in response to mriedem not dansmith16:14
dansmithjust to make it default for people running local?16:14
dansmithcdent: I know :)16:14
mriedemdansmith: stephen's patch might do that...i'd have to check. but at some point i'm guessing we'll have functional-py37 too16:14
mriedemi think that's already in experimental16:14
mriedemyup https://github.com/openstack/nova/blob/master/.zuul.yaml#L9716:15
dansmithmriedem: I've been typing py27 for unit for ten years almost, it's annoying to have the target name change every time we bump py3 versions16:15
*** dpawlik has joined #openstack-nova16:15
dansmithso I'd love to have a -efunctional and -eunit so I don't have to track16:15
sean-k-mooneydansmith: we did that in os-vif16:15
mriedemi've been typing -e functional-py36 for years now b/c that's the one i always broke :)16:15
sean-k-mooneyhttps://github.com/openstack/os-vif/blob/master/tox.ini#L32-L4316:16
mriedemanyway, just a thought, ignore me16:16
sean-k-mooneyya i have that in muscel memory too16:16
sean-k-mooneydansmith: oh we did it that way in os-vif because we only had py27 fucntional tests at the time16:17
*** rpittau is now known as rpittau|afk16:17
sean-k-mooneyso when we added python3 func tests i decided to make functional py3 by defualt16:18
*** igordc has quit IRC16:20
*** cdent has left #openstack-nova16:20
sean-k-mooneydansmith: i would like to see a "-e unit" too by the way as i woudl liek a way to have just "tox" work on any system regardless of what version of python you have installed16:22
dansmithyeah16:22
mriedemthis horizon fix is pretty simple, should have it up soon16:23
sean-k-mooneythat said 90% of the time i run it by typeing "CTRL-r tox" to grab the command form my bash history so it does not really bother me.16:24
openstackgerritAdrian Chiris proposed openstack/nova stable/train: Revert "vif: Resolve a TODO and update another"  https://review.opendev.org/69012616:27
openstackgerritMerged openstack/nova master: Don't populate resources for not-yet-migrated inst  https://review.opendev.org/68984216:27
openstackgerritMerged openstack/nova stable/queens: Stop sending bad values from libosinfo to libvirt  https://review.opendev.org/68806916:27
openstackgerritMerged openstack/nova stable/stein: Ignore sqla-migrate inspect.getargspec deprecation warnings on py36  https://review.opendev.org/68507216:27
*** ricolin__ is now known as ricolin16:30
*** nanzha has quit IRC16:38
*** eharney has quit IRC16:39
mriedemgah, this horizon thing is actually kind of a mess,16:39
mriedemthey have their own config option for blacklisting compute API extensions,16:39
mriedemso i essentially have to remove that without a deprecation period in horizon16:39
mriedemand the novaclient change https://review.opendev.org/#/c/686516/ definitely jumped the gun on removal of the deprecated list_extensions stuff16:40
mriedemdidn't even wait a full server side release16:40
dansmithblargh16:40
mriedemso, not sure what to do. i could plow ahead with the horizon thing, release note it and hope the horizon people are lax about their compat rules (makes me feel dirty)16:41
*** pcaruana has joined #openstack-nova16:41
mriedemor we revert the novaclient thing, blacklist 16.0.0, release a patch and then deprecate the option in horizon16:41
mriedemmaybe it's time for the ML16:41
dansmithor release a 15.2.0 and keep everything on 15.x for a cycle?16:42
*** dpawlik has quit IRC16:42
mriedemalways fun cleaning up the fallout from cores with itchy trigger fingers to delete shit16:42
mriedemdansmith: that doesn't solve the upper-constraints issue16:42
mriedemyou need something for osc, and horizon needs to not be broken16:42
dansmithmriedem: can't we set u-c to 15.9 or something to keep everything from getting 16?16:43
mriedemdansmith: the novaclient change you need is only in 16.0.0 though16:43
mriedemiow you're tied to the sinking ship unless we revert this other novaclient patch16:43
dansmithright, that's why I said release 15.2, essentially backporting what we need to 15.x for a cycle16:43
mriedemyou mean backport your aggregate-image-cache feature to stable/train?16:44
mriedemin novaclient i mean16:44
dansmithwell,16:44
dansmithdon't we have branches in the clients for number versions16:44
mriedemthat's what we'd have to do i think, and that's not really kosher16:44
dansmith?16:44
mriedemwe have branches and stable/train is the 15.x series yeah,16:45
mriedembut still it would mean backporting the feature16:45
dansmithyeah so your concern is train would pull in 15.2?16:45
mriedemright16:45
dansmithbut yeah, it's a backport16:45
mriedemsimpler just to revert this rushed removal patch to me16:45
mriedembut i'll send to the ML to see what the horizon people say16:45
dansmithokay so what does that mean for versions? 16.1 becomes the next version with the stuff back in it?16:46
dansmithso we effectively bumped to 16 for no reason?16:46
mriedem16.0.0 would be blacklisted, we'd revert https://review.opendev.org/#/c/686516/ and release as 16.0.1 i thnk16:46
mriedem*think16:46
mriedemnot sure it needs to be a minor16:46
dansmiththat's what I mean yeah16:46
mriedemu-c would get 16.0.116:47
mriedemyou'd get your thing for osc,16:47
mriedemand horizon would have a release to deprecate it's option16:47
mriedemand we revert the revert in X16:47
mriedemwhich is what should have happened anyway according to our novaclient deprecation removal guidelines https://docs.openstack.org/python-novaclient/latest/contributor/deprecation-policy.html16:47
dansmithidk, I agree that plan would be the by-the-book way,16:48
dansmithbut being a client it seems like it shouldn't be that big of a deal, but I'm clearly not the expert on the politics here16:48
mriedemit's not really politics, it's just that we screwed horizon without time to react,16:49
mriedemso they can find an alternative solution without novaclient (i don't know how to write that patch), they can drop their thing without a deprecation period, or we can try to make it right16:49
mriedemand option 5 is i just stop caring and let someone else sort out this mess :)16:50
mriedemlike stephenfin!16:50
dansmithI mean the "getting everyone to agree on the solution" where some want the by-the-book and some might go a more pragmatic route16:50
dansmithanyhoo16:50
*** pcaruana has quit IRC16:52
*** markvoelker has quit IRC16:54
dansmithif we leave it to stephenfin then we're not going to get anything resolved until well after ptg16:56
mriedemi know, i was joking16:56
dansmithso hopefully that isn't the strategy16:56
mriedemcrap, forgot i was preheating an oven, oops16:57
*** dtantsur is now known as dtantsur|afk17:02
*** xek__ has joined #openstack-nova17:07
*** xek_ has quit IRC17:10
*** markvoelker has joined #openstack-nova17:12
*** spatel has quit IRC17:15
*** mkrai has joined #openstack-nova17:17
mriedemwill see if this does the trick https://review.opendev.org/69013417:18
*** tbachman has quit IRC17:22
*** priteau has quit IRC17:22
*** otsukahy has quit IRC17:24
*** eharney has joined #openstack-nova17:25
*** tbachman has joined #openstack-nova17:28
*** ozzzo has quit IRC17:32
*** tssurya has quit IRC17:32
*** ociuhandu_ has joined #openstack-nova17:32
*** mmethot_ has joined #openstack-nova17:35
*** mmethot has quit IRC17:35
*** ociuhandu has quit IRC17:36
*** ociuhandu_ has quit IRC17:39
*** ociuhandu has joined #openstack-nova17:43
*** andyzon has joined #openstack-nova17:46
*** ociuhandu has quit IRC17:47
dansmithmriedem: were you going to send an email about this or just hope horizon peeps are okay with that?18:00
*** ricolin has quit IRC18:01
*** jaosorior has quit IRC18:01
openstackgerritMatt Riedemann proposed openstack/nova master: Adds view builders for keypairs controller  https://review.opendev.org/34728918:02
mriedemdansmith: i started working up the email but then if that change is OK i don't think it's worth the email18:03
mriedemor revert discussion18:03
*** otsukahy has joined #openstack-nova18:05
*** tbachman has quit IRC18:07
*** ttsiouts has joined #openstack-nova18:07
*** otsukahy has quit IRC18:09
*** mkrai has quit IRC18:13
*** igordc has joined #openstack-nova18:14
*** ivve has joined #openstack-nova18:15
*** tbachman has joined #openstack-nova18:18
*** ralonsoh has quit IRC18:22
*** CeeMac has joined #openstack-nova18:26
*** spsurya has quit IRC18:30
*** gbarros has joined #openstack-nova18:37
*** gbarros has quit IRC18:41
openstackgerritMatt Riedemann proposed openstack/nova master: FakeDriver: adding and removing instances on live migration.  https://review.opendev.org/24361318:42
*** ttsiouts has quit IRC18:44
*** ttsiouts has joined #openstack-nova18:44
*** gbarros has joined #openstack-nova18:45
*** igordc has quit IRC18:47
*** ttsiouts has quit IRC18:49
*** igordc has joined #openstack-nova18:56
*** dpawlik has joined #openstack-nova18:57
*** andyzon has quit IRC19:00
*** gbarros has quit IRC19:01
*** tbachman has quit IRC19:03
*** pcaruana has joined #openstack-nova19:04
*** jmlowe has joined #openstack-nova19:09
*** dpawlik has quit IRC19:09
*** gbarros has joined #openstack-nova19:12
*** gbarros has quit IRC19:19
*** pcaruana has quit IRC19:19
*** gbarros has joined #openstack-nova19:20
*** slaweq has joined #openstack-nova19:20
openstackgerritMatt Riedemann proposed openstack/nova master: api-ref: mark device response param as optional for list/show vol attachments  https://review.opendev.org/69038319:24
*** henriqueof has joined #openstack-nova19:27
*** gbarros has quit IRC19:27
*** otsukahy has joined #openstack-nova19:32
*** gbarros has joined #openstack-nova19:35
mriedemthis is pretty funny https://review.opendev.org/#/c/323250/19:36
mriedem"Happy 3rd birthday patch!! Soon you will go to school!"19:36
dansmithlol19:37
*** gbarros has quit IRC19:37
*** maciejjozefczyk has joined #openstack-nova19:40
*** gbarros has joined #openstack-nova19:40
*** slaweq has quit IRC19:46
*** dpawlik has joined #openstack-nova19:48
mriedemunfortunately i still have to -1 them19:50
*** slaweq has joined #openstack-nova19:51
mriedemis there some way we could have an instance that is both deleted (instance.deleted != 0) and SOFT_DELETED? (instance.vm_state == 'soft-delete')?19:54
*** nweinber_ has quit IRC20:00
*** maciejjozefczyk has quit IRC20:04
*** otsukahy has quit IRC20:05
*** otsukahy has joined #openstack-nova20:05
openstackgerritMatthew Booth proposed openstack/nova master: Add new simple test base for functional tests  https://review.opendev.org/68918620:18
openstackgerritMatthew Booth proposed openstack/nova master: Functional test for UnexpectedDeletingTaskStateError  https://review.opendev.org/68927820:18
openstackgerritMatthew Booth proposed openstack/nova master: Unplug VIFs as part of cleanup of networks  https://review.opendev.org/66338220:18
openstackgerritMatthew Booth proposed openstack/nova master: Fix incorrect vm and task state after build failure race  https://review.opendev.org/68938820:18
openstackgerritMatthew Booth proposed openstack/nova master: Make image fixtures simpler to reference  https://review.opendev.org/69040320:18
mdboothmriedem: https://review.opendev.org/#/c/689186/3/nova/tests/functional/integrated_helpers.py is what I did to ProviderUsageBaseTestCase20:19
mdboothtl;dr renamed it to SimpleTestBase and put all the provider test specific support code in a ProviderUsageBaseTestCase subclass20:20
mdboothThen I subclassed SimpleTestBase, which is clean without the provider stuff20:21
mriedemi don't want to backport any of those refactors20:22
mriedemi thought i was pretty clear about that20:23
mdboothI can't explain the locking thing I hit, btw, but it made no sense, went away when I looked harder, and never came back. I assume it was PEBKAC and I probably had an unsaved editor window open or something.20:23
mdboothmriedem: Sure, if you like.20:23
mdboothmriedem: For the backport we could just use ProviderUsageBaseTestCase20:23
mriedemyes, please20:23
mdboothAnd for the image fixture thing we could just use a hardcoded image20:23
mriedemright now you have 2 refactor changes before the new regression test20:24
mdboothBut lets do that in the backport20:24
*** tbachman has joined #openstack-nova20:24
melwittmriedem: without looking at the code first, I'd have thought "yes" because the instance would begin as SOFT_DELETED and then when the reclaim interval comes around and it's time to permadelete it, it would then become deleted != 0 as well. it would surprise me if the vm_state is cleared when that happens, but who knows20:26
mriedemmelwitt: the periodic queries the db for instances which are SOFT_DELETED and not undergoing some task_state transition on the given host,20:27
mriedemthen deletes them which changes the vm_state to DELETED and destroys the instance20:28
mriedemso if we updated the vm_state to DELETED it shouldn't be in the db query results in the periodic20:28
*** gbarros has quit IRC20:28
mriedemand if we didn't update that, like the update failed, then the instance won't be deleted either (deleted == 0)20:28
mriedemthe db api query will return deleted instances by default since the filter being used doesn't include deleted=False20:29
mriedembut still, not sure how to get into deleted=id and vm_state=soft-delete at the same time20:29
mriedemi was thinking maybe some local delete in the api wonkiness but didn't see anything obvoius20:29
mriedemmelwitt: also while i've got you, it looks like NOVA_CONSOLE_PROXY_COMPUTE_TLS in our nova-next job isn't getting honored anymore, i.e. we don't have the vencrypt config in that job20:30
melwittok, it sounds like by "could have" you mean can the periodic return an instance like that. I answered it as though it were, "can we have any records in that state?" and I thought yes. so... nevermind me20:30
mriedembecause NOVA_CONSOLE_PROXY_COMPUTE_TLS=true in localrc but https://github.com/openstack/devstack/blob/1a46c898db9c16173013d95e2bc954992121077c/lib/nova#L17520:30
melwittmriedem: argh again huh... ok, I'll look. I've had to fix that before in the past20:31
mriedemi'm trying to figure out why https://review.opendev.org/#/c/683988/ is intermittenly failing novnc tests,20:31
melwittI'll look up what I did last time and compare to see what has changed20:31
mriedemwhich you recently fixed for multinode jobs here https://review.opendev.org/#/c/675721/20:31
mriedemi think it's just the conversion of the nova-next job from d-g to zuulv320:32
mriedemhttps://review.opendev.org/#/c/670196/20:32
mriedemd-g would pass the value through in the localrc but zuulv3 is yaml'ing it and converting True to true20:33
mriedemyaml-ama-ding-dong20:33
melwittthis is what I had to do last time to fix enablement of tls between nova-compute and the proxy https://review.opendev.org/64304520:33
mriedemwe aren't even getting that far because https://review.opendev.org/#/c/643045/9/lib/nova@65920:34
mriedem [[ ${NOVA_CONSOLE_PROXY_COMPUTE_TLS} = "True" ]]20:34
mriedemNOVA_CONSOLE_PROXY_COMPUTE_TLS: true20:34
melwittoh, you're saying that flag is not getting reflected as true? O.o20:34
KeithMnemonicis this a valid command option, using --deleted and --limit -1. it seems to cause an infinite hang on Rocky. i did not check newwer releases yet20:34
mriedemmelwitt: hard to say because local.conf here looks OK https://zuul.opendev.org/t/openstack/build/9726aebf48804b83bbebe290bd8187ac/log/controller/logs/local_conf.txt.gz#5420:35
melwitthmm20:35
*** dpawlik has quit IRC20:35
melwittKeithMnemonic: I didn't think that limit -1 was a thing. unlimited happens by omitting --limit, I thought. but regardless of that, there shouldn't be an indefinite hang, that I would call a bug20:36
mriedemlimit=-1 is a novaclient thing to page until there are no more results20:36
melwittoh :\ I thought it did that on its own in the absence of --limit20:37
KeithMnemonic"  --limit <num-servers>20:37
KeithMnemonic                        Maximum number of servers to display. If limit equals20:37
KeithMnemonic                        -1, all servers will be displayed. If limit is greater20:37
KeithMnemonic                        than 'osapi_max_limit' option of Nova API,20:37
KeithMnemonic                        'osapi_max_limit' will be used instead.20:37
KeithMnemonic"20:37
melwittnevermind, sorry20:37
KeithMnemonicfrom the cli help20:37
KeithMnemonicok i can log a bug. i will see if this happens on something newer20:37
mriedemmelwitt: nope, by default limit is 1000 results in the api20:37
KeithMnemonici did not see anything open around this currently20:37
melwittmriedem: that I know, but I thought the client would auto-page by itself without need to tell it20:37
melwittand I was wrong20:37
KeithMnemonicin the env i tested, i only have maybe 20 deleted at most20:38
melwittyeah, something is whack there, I think20:38
KeithMnemoniclet me see what debug i can get to add to the bug to help isolate it20:39
mdboothmriedem: Responded. I still think that's good to go. I'd prefer to make the requested changes in the backports. They're trivial enough.20:39
melwittKeithMnemonic: yeah run with --debug to see where it gets stuck20:40
*** otsukahy has quit IRC20:46
*** cgoncalves has quit IRC20:49
mriedemmelwitt: oh here is the vencrypt stuff, i was looking in the wrong config file (there are only 20 nova configs in a devstack job)20:50
mriedemhttps://zuul.opendev.org/t/openstack/build/9726aebf48804b83bbebe290bd8187ac/log/controller/logs/etc/nova/nova_cell1_conf.txt.gz#3220:50
*** colby_ has quit IRC20:50
mriedemso i wonder if NOVA_CONSOLE_PROXY_COMPUTE_TLS=True just doesn't work for the subnode since we've never had a multinode job use that yet20:51
melwittmriedem: ah yeah. awhile back I split things at stephenfin request (and I agree was a good idea) to only config which settings were required on the proxy and compute host, respectively. so they don't just all appear in all configs like they used to back in the day20:52
melwittmriedem: the subnode is nova-compute only, right? if so, then it would make sense to *not* run the proxy service there and it might just be that we have a wrong assumption in the conditional that the service will necessarily be running if we're doing tls between the computes and the proxy?20:54
mriedemyeah subnode is nova-compute only20:54
mriedemno n-novnc service running on the subnode20:54
melwittand maybe the intermittent is because depending on whether the instance being tested for vnc lands on the subnode or not?20:55
mriedemyup i think so20:55
*** cgoncalves has joined #openstack-nova20:55
melwittyeah, so it sounds like we need to use some other condition for whether to do the stuff, not the presence of the tls proxy service thing20:56
*** gbarros has joined #openstack-nova20:57
mriedemok so maybe for now i'll rev this patch, comment out the usage of NOVA_CONSOLE_PROXY_COMPUTE_TLS since it doesn't work for multinode jobs, and see if that at least gets this patch moving forward20:57
melwittok. and I can work on a change in devstack to make it work with multinode, and then stack a change on top of yours that Depends-On it, to see if it clears up the problem21:00
melwittor sorry, I'll just do the devstack change and then you can uncomment and Depends-On to test21:00
mriedemsure that works, thanks21:01
openstackgerritMatt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3  https://review.opendev.org/68398821:01
*** slaweq has quit IRC21:08
KeithMnemonicso what i can tell is first it calls /server/detail?deleted=True and this returns a good list, then the next call is that same but with &marker=<id of last server> and it loops here21:11
KeithMnemonicso something about the marker. if i try and call it using curl i get a 401 but my guess is you that is not a valid test21:12
openstackgerritMatt Riedemann proposed openstack/nova master: Revert "Log CellTimeout traceback in scatter_gather_cells"  https://review.opendev.org/69041721:12
mriedemKeithMnemonic: you're getting a 401 with curl b/c you're not providing a token21:13
KeithMnemonicno i am21:13
KeithMnemonicit only 401 with &marker21:13
KeithMnemonicwith just ?delete=True it is fine21:13
mriedem401 is an auth issue so that doesn't make much sense21:13
KeithMnemonichttp://paste.openstack.org/show/785488/21:15
KeithMnemonicyou can see something with adding &marker, but it could be my test is wrong21:15
KeithMnemonici was just trying to curl what is was looping on21:15
KeithMnemonicthe --debug show it loops once it adds on &marker=<last deleted UUID>21:16
KeithMnemonichttps://bugs.launchpad.net/nova/+bug/177394521:19
openstackLaunchpad bug 1773945 in OpenStack Compute (nova) "nova client servers.list crashes with bad marker" [Medium,Incomplete]21:19
*** gbarros has quit IRC21:19
mriedemi think i see the problem21:20
KeithMnemonicnot quite my issue , but something with markers21:20
mriedemhttps://review.opendev.org/#/c/527564/1/nova/compute/api.py21:20
mriedemobjects.BuildRequestList.get_by_filters returns an empty list if filtering on deleted=True,21:21
mriedemthe code assumes that if there was a marker and we got a build request list (not MarkerNotFound), we found our marker and can set marker=None since we won't find it in a cell,21:21
mriedembut that's not the case here21:21
mriedemthat goes beyond rocky https://review.opendev.org/#/q/I1aa3ca6cc70cef65d24dec1e7db9491c9b73f7ab21:22
mriedemhttps://review.opendev.org/#/c/527564/1/nova/objects/build_request.py@37621:22
mriedemKeithMnemonic: in your reproduce do you have any deleted servers in the db?21:24
KeithMnemonicespecially since i only have  18 deleted, it seems it should not need the marker right?21:24
mriedemyou must otherwise you wouldn't get a marker21:24
*** gbarros has joined #openstack-nova21:24
melwittmaybe the deleted instances are located across separate cells?21:26
mriedemi meant i see the reason for the infinite loop21:26
mriedemKeithMnemonic: did you open a bug?21:26
KeithMnemonicnot yet, was trying to get more debug to make the bug report useful21:27
KeithMnemonicthat is when i found it looped on marker21:27
melwittI was talking about why a marker when only 18 deleted?21:27
melwittI dunno21:27
mriedembecause limit is in the request i think21:27
mriedemlimit == paging21:28
KeithMnemoniccorrect --limit -121:28
melwittoh ok21:28
KeithMnemonicmriedem, ok i can open the bug, do you need anything specific in it or you have a handle on the issue?21:29
mriedemthe paste is useful and recreate steps21:29
KeithMnemonicok let me add all of that21:29
mriedemnice to know that regressions i introduced in queens are coming back to find me now21:31
KeithMnemoniclol21:32
efriedmriedem: serves you right for not giving your features enough soak time.21:34
mriedemheh21:40
mriedemdoesn't really fit here, but i can see that's been smoldering in your pocket for awhile21:40
mriedemi can never remember if you specify marker if the marker instance comes back as the first item in the list21:42
mriedemor if it's everything after the marker21:42
mriedemaha, ":param marker: the last item of the previous page; we returns the next21:43
mriedem                    results after this value."21:43
*** otsukahy has joined #openstack-nova21:44
mriedemKeithMnemonic: ok i've got a recreate functional test ready to push up whenever you have this bug filed21:52
openstackgerritMatt Riedemann proposed openstack/nova master: Add functional regression test for bug xxxxxx  https://review.opendev.org/69042822:00
openstackgerritMatt Riedemann proposed openstack/nova master: Fix listing deleted servers with a marker  https://review.opendev.org/69042922:00
*** otsukahy has quit IRC22:01
*** otsukahy has joined #openstack-nova22:04
mriedemok well there is the reproduce and fix ^, will update once i have a bug id22:06
*** otsukahy has quit IRC22:07
*** otsukahy has joined #openstack-nova22:07
*** otsukahy has quit IRC22:14
*** otsukahy has joined #openstack-nova22:15
*** TxGirlGeek has quit IRC22:18
*** mriedem has quit IRC22:23
*** otsukahy has quit IRC22:29
*** xek__ has quit IRC22:37
*** otsukahy has joined #openstack-nova22:40
*** otsukahy has quit IRC22:44
*** ivve has quit IRC22:45
*** ociuhandu has joined #openstack-nova22:46
*** ociuhandu has quit IRC22:51
*** tkajinam has joined #openstack-nova22:53
*** avolkov has quit IRC23:13
*** macz has quit IRC23:43
*** igordc has quit IRC23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!