Monday, 2020-04-06

*** ociuhandu has joined #openstack-nova01:15
*** ociuhandu has quit IRC01:19
*** tetsuro has joined #openstack-nova01:26
*** tetsuro has quit IRC02:14
openstackgerritGhanshyam Mann proposed openstack/nova master: Introduce scope_types in os-flavor-manage  https://review.opendev.org/71481802:37
openstackgerritGhanshyam Mann proposed openstack/nova master: Add new default roles in os-flavor_manage policies  https://review.opendev.org/71481902:37
openstackgerritGhanshyam Mann proposed openstack/nova master: Pass the actual target in os-flavor-manage policy  https://review.opendev.org/71482202:38
*** mkrai has joined #openstack-nova02:56
*** ralonsoh has joined #openstack-nova03:04
*** ociuhandu has joined #openstack-nova03:13
openstackgerritGhanshyam Mann proposed openstack/nova master: Add new default roles in server topology policies  https://review.opendev.org/71758503:14
*** mvkr has quit IRC03:15
*** psachin has joined #openstack-nova03:19
*** ociuhandu has quit IRC03:22
*** ociuhandu has joined #openstack-nova03:24
*** mvkr has joined #openstack-nova03:28
*** tetsuro has joined #openstack-nova03:29
*** ociuhandu has quit IRC03:29
*** tetsuro has quit IRC03:58
*** tetsuro has joined #openstack-nova04:02
*** prometheanfire has left #openstack-nova04:09
*** evrardjp has quit IRC04:36
*** evrardjp has joined #openstack-nova04:37
*** udesale has joined #openstack-nova04:51
*** udesale has quit IRC04:52
*** udesale has joined #openstack-nova04:52
*** udesale has quit IRC05:03
*** udesale has joined #openstack-nova05:03
*** ratailor has joined #openstack-nova05:04
openstackgerritmelanie witt proposed openstack/nova master: Reset the cell cache for database access in Service  https://review.opendev.org/71766205:39
*** tetsuro has quit IRC06:05
*** tetsuro has joined #openstack-nova06:21
*** ociuhandu has joined #openstack-nova06:28
*** dpawlik has joined #openstack-nova06:28
*** ociuhandu has quit IRC06:33
*** ccamacho has joined #openstack-nova06:34
*** rpittau|afk is now known as rpittau06:35
*** ccamacho has quit IRC06:46
*** ccamacho has joined #openstack-nova06:46
*** ttsiouts has joined #openstack-nova06:47
*** nightmare_unreal has joined #openstack-nova06:53
*** slaweq has joined #openstack-nova07:01
*** zhanglong has joined #openstack-nova07:03
*** dklyle has quit IRC07:04
*** maciejjozefczyk has joined #openstack-nova07:08
*** ociuhandu has joined #openstack-nova07:13
*** ociuhandu has quit IRC07:18
*** tesseract has joined #openstack-nova07:29
*** dtantsur|afk is now known as dtantsur07:41
*** xek__ has joined #openstack-nova07:46
*** happyhemant has joined #openstack-nova07:46
bauzasgibi: stephenfin: good morning, I worked on finishing the implementation for the vGPU multiple types https://review.opendev.org/#/c/715490/07:47
bauzasI will provide a new change for having a functional test07:47
*** mkrai has quit IRC07:51
gibibauzas: ack, I will try to look at it today07:54
gibiand good morning07:54
gibi:)07:54
*** tosky has joined #openstack-nova07:58
*** ralonsoh has quit IRC07:58
*** spsurya_ has joined #openstack-nova07:59
*** links has joined #openstack-nova08:02
*** alex_xu has joined #openstack-nova08:08
* alex_xu go to super-market today, and found there is machine for tracking people face and body temperature08:09
*** mkrai has joined #openstack-nova08:09
bauzasalex_xu: gtk08:11
*** ralonsoh has joined #openstack-nova08:11
bauzasthe only problem is that some of us are not having temperature issues if they have COVID-1908:11
alex_xubauzas: yea08:11
alex_xubauzas: another fun is the phone tracked here, if you are out of the city, you will be notice to ioslate for 14days08:12
bauzasalex_xu: you're living in Peking, right?08:13
alex_xubauzas: the joke is if you use vpn, then you are being notice for isolation :)08:13
alex_xubauzas: yes08:13
bauzasack okay08:13
alex_xuso Matthew is right in shanghai summit, we are totally tracked :)08:15
*** mkrai has quit IRC08:29
*** dpawlik has quit IRC08:29
bauzasalex_xu: well, we have some kind of discussion in France about being tracked or not08:31
bauzasthe problem is that a lot of folks around me are just walking outside while they shouldn't08:31
bauzasand we're now locked down since 3 weeks, and for maybe 3 or 4 weeks again08:31
*** priteau has joined #openstack-nova08:32
alex_xubauzas: yea, i'm pretty sure you will feel safe on the stree. but yes...the big brother is watching you. but anyway, we discuss the same issue privately :)08:32
alex_xubauzas: pretty sure you need another 3 weeks...from the experience we have...08:33
alex_xubauzas: ^ the work above the above one, I'm talking about the another side of tracking :)08:34
alex_xus/work/word/08:35
bauzasyeah In understand you08:35
alex_xubauzas: we have people on the street now, since people already lock in the room since Feb... just shopping mall is pretty empty, but the park is full of people...08:36
*** kashyap has joined #openstack-nova08:37
*** dpawlik has joined #openstack-nova08:38
*** martinkennelly has joined #openstack-nova08:40
*** ociuhandu has joined #openstack-nova08:41
*** ociuhandu has quit IRC08:45
*** zhanglong has quit IRC08:46
*** avolkov has joined #openstack-nova08:48
*** luyao has joined #openstack-nova08:49
*** mkrai has joined #openstack-nova08:50
*** martinkennelly has quit IRC08:51
*** tetsuro has quit IRC08:54
*** rcernin has quit IRC09:03
*** ociuhandu has joined #openstack-nova09:13
openstackgerritMarcin Juszkiewicz proposed openstack/nova master: libvirt: check for AMD SEV only on x86-64  https://review.opendev.org/71442509:19
*** vishalmanchanda has joined #openstack-nova09:24
bauzasstephenfin: FWIW, /me starts to review https://review.opendev.org/#/c/704643/09:30
stephenfinTa. vGPU is on my list for today :)09:30
johnthetubaguyah, I was just looking at that too, maybe bad timing09:43
bauzasstephenfin: johnthetubaguy: I'll provide comments and a -109:46
bauzasjust downloading the patch for verifying somethin09:46
johnthetubaguystephenfin: is it worth being more specific on what happens when you don't set an extra spec? or describing dependencies, like things that imply numa aware placed VMs, etc09:51
gibicores, here is a quick functional test stabilization patch needing a second core https://review.opendev.org/#/c/71707009:52
stephenfinYou mean in the descriptions for the extra specs themselves?09:52
bauzasgibi: on it, I'm mostly done09:52
johnthetubaguystephenfin: somewhere like that09:52
gibibauzas: thanks09:52
johnthetubaguystephenfin: comparing it to configuration, basically09:53
stephenfinYeah, sure. I probably need to better flesh out the descriptions for much of them, including things like what virt driver supports what extra spec09:53
stephenfinIn a follow-up though, preferably? I'd like to focus more on making sure every possible extra spec is listed there first09:53
johnthetubaguystephenfin: I thought you did fairly well on the virt support, I guess there are a bunch of libvirt only in there that are not spelled out09:53
johnthetubaguystephenfin: yeah, I am OK with that, just wondering about your plans really09:54
johnthetubaguystephenfin: there are things like PCI alias, when you check the format and not the content, which is typical of json schema checks, just wondering about your thoughts there09:54
stephenfinYeah, I plan to massively improve the documentation and do things like add cross-referencing09:54
*** mkrai has quit IRC09:55
johnthetubaguyto be clear, I would be fine just checking the keys and not the value, as a step forward, so this is a step ahead of that09:55
stephenfinAh, so for more detailed things like that, I was planning to leave it to the virt driver for now09:55
stephenfinditto for things like "you need to specify 'hw:cpu_policy' to use 'hw:cpu_thread_policy'"09:56
stephenfinI wanted to do that via the validator initially but it was way too much /o\09:56
johnthetubaguyI guess I see this as our global API abstraction, that has been implemented by limited drivers... but yeah, its totally a next step09:57
*** mkrai has joined #openstack-nova09:59
bauzasstephenfin: I don't see anything in the spec about API interop consistency with updates on, say, https://review.opendev.org/#/c/704643/21/nova/api/validation/extra_specs/capabilities.py10:05
bauzaslike, I want 'baz' to be accepted in Victoria10:05
bauzasbut calling a Ussuri API will tell you 'sorry but no'10:05
bauzasstephenfin: do you plan to address this later on ?10:05
johnthetubaguyits not turned on yet right, haven't set the microversion where it starts10:06
*** mkrai has quit IRC10:06
*** mkrai_ has joined #openstack-nova10:06
*** mkrai_ has quit IRC10:07
*** mkrai has joined #openstack-nova10:08
bauzasjohnthetubaguy: indeed, see my comments https://review.opendev.org/#/c/704643/2110:08
stephenfinbauzas: There's no easy answer for that. I think that's the main reason we had to make the policy configurable. We didn't want to require a microversion to add a new extra spec so people needed a escape lever10:08
bauzasstephenfin: okay, I just feel my biggest concern is that if we avoid to whitelist some key for some filter, then people will see a behavioural change10:09
bauzasright?10:09
stephenfinbauzas: tbh though, it's the exact same issue we see with image metadata today and configuring flavor extra specs, unlike configuring image metadata, is admin-only10:09
bauzasalas. people can use 2.85 microversion if they want to disable it10:10
stephenfinbauzas: Correct, but we have signalled that change with a microversion and we've provided a way to "escape" validation10:10
bauzasstephenfin: yeah I understood it10:10
stephenfinor pass '?validation=permissive', iirc10:10
bauzasstephenfin: I'm just afraid that we could merge something that would opt-in10:10
bauzasclose to RC110:10
bauzasstephenfin: but we can go like it is now10:11
bauzasat least if we have ways to address issues easilty10:11
bauzasbecause reviewing all the key and value regexes for every filter isn't an easy thing10:11
bauzasbut I don't want to hold this one10:11
bauzasso, my take is : if we mess things up, we all assume that we're about to provide bugfixes that will touch the API behaviour and should be backportable10:12
bauzasif we all agree on this, I'm thumbs up10:12
bauzasgibi: stephenfin: johnthetubaguy: ^10:12
bauzasor we go permissive *by default*10:12
johnthetubaguyhang on... I am missing something10:13
bauzas(which is not the default case IIUC)10:13
johnthetubaguydon't we currently do zero validation by default right now?10:13
gibibauzas: fixing bugs in the extra_spec validation will not need a new microversion in my view.10:13
johnthetubaguyincluding after the proposed validation10:13
bauzasgibi: me too, but I'm just saying that we will change the API behaviour silently10:13
johnthetubaguyhang about10:14
bauzasso we all need to accept it10:14
stephenfinjohnthetubaguy: no, after https://review.opendev.org/#/c/704643/ it's enabled by default10:14
johnthetubaguythis totally requires a microversion10:14
gibibauzas: we fix bugs in the API behavior. Which was allowed before as well10:14
bauzasjohnthetubaguy: this is enabled later in another change10:14
stephenfinwhoops10:14
stephenfinhttps://review.opendev.org/#/c/708436/10:14
stephenfinjohnthetubaguy: got that ^10:14
bauzas2.8610:14
stephenfinjohnthetubaguy: It's a noop with the first patch. The microversion change to turn it on is a separate change10:14
bauzastbc, I'm okay with approving https://review.opendev.org/#/c/704643/ provided stephenfin (or me) just changes the commit msg10:15
johnthetubaguystephenfin: yeah, i.e. its added in a microversion10:15
bauzasbut I'm not okay with https://review.opendev.org/#/c/708436/ until we basically agree on the stuff I said10:15
bauzasbut as gibi said, bugfixes are accepted, so maybe I'm overthinking10:15
johnthetubaguyso... if you do an old microversion, there is no change10:15
bauzascorrect10:16
gibicorrect10:16
johnthetubaguynope, bug fixes than change the API would be really bad right?10:16
johnthetubaguyI am really confused here10:16
johnthetubaguycode looks OK to me, but not agreed with the discussion here10:16
gibijohnthetubaguy: if we broke the API we have to fix them with a bugfix. I don't see how that is a bad thing.10:16
johnthetubaguygibi: nope, we just keep a broken microversion, we did exactly that in the past, see the limits APIs10:17
johnthetubaguywell, it depends on the bug...10:17
stephenfinwe're saying that additions to and bug fixes for the _registry_ of extra specs are okay and wouldn't need a microversion10:17
gibiI'm sure that in other cases we said "client should not have to opt in to get a fix"10:17
stephenfinnot the API itself10:17
bauzasokay, see, that's why I wanted some discussion about this10:17
johnthetubaguyhmm...10:17
bauzas... or we set it to permissive by default10:17
johnthetubaguypermissive by default seems pointless to me, needs to get on by default in a new microversion10:18
bauzasand then we have a later change for turning it by default that would require approvals separately10:18
stephenfinthat has to be okay, because operators are currently allowed to specify their own extra specs for use in e.g. custom scheduler filters10:18
gibiI think the differentiator was always like "Was the bug in the API was clearly a bad behavior?"10:18
bauzasjohnthetubaguy: I agree, the microversion bump would have to be on the 'turn on by default'10:18
stephenfinif we were to say new extra spec == new microversion, we'd have to take that away from operators10:18
johnthetubaguyit was, "does it impact security" I think10:18
johnthetubaguystephenfin: we would have to give them a namespace like CUSTOM:10:19
johnthetubaguyso to use the new API microversion, they need to update their flavors10:19
stephenfinif we were doing this from scratch, yes10:19
stephenfinbut there are custom extra specs in the wild now, so that ship has sailed10:19
johnthetubaguywell, not quite10:20
johnthetubaguythey would still work with older microversions right?10:20
johnthetubaguyto move forward, they need a migration path10:20
bauzasI don't want to rediscuss the spec https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/flavor-extra-spec-validators.html10:20
johnthetubaguywell... I guess we should be comparing this to image properties10:21
bauzasI'm just okay with the plan, but from an implementation point of view, I feel that has necessarly to be discussed10:21
stephenfinyeah, they could use the older microversions10:21
stephenfincompared to image metadata properties, I think we're mostly in the same place now10:22
stephenfinyou can add a new image metadata property or modify an existing one (add a new value to an enum)10:23
johnthetubaguyso I am stuggling to understand everyone's position in text form10:23
stephenfinfor example, setting the 'hw_pci_numa_affinity_policy' property would be rejected by a Train cloud but not an Ussuri cloud, regardless of microversion10:25
stephenfin(since it was only added this cycle)10:25
johnthetubaguybut to be rejected, they would need to be calling the new microversion right?10:26
bauzasjohnthetubaguy: stephenfin: the consensus I think is that https://review.opendev.org/#/c/704643/ is a nobrainer10:26
gibimy position: I assume that the defined key-values are good (reviewed a good chunk of it), if we break something then we will fix it in a bugfix, I don't see other ways to prevent a break. custom extra_specs can be used with old microversion, and custom validator can be added to use the new microversion10:27
bauzasbut https://review.opendev.org/#/c/708436/ is debatable until we reach an agreement10:27
johnthetubaguyI think we agree with the patch, but are unsure on future changes?10:27
stephenfinjohnthetubaguy: my position is that tying extra spec registry modifications to microversions is a lot of work and gives us almost nothing in return w.r.t. API interop10:28
bauzasjohnthetubaguy: I'm okay with https://review.opendev.org/#/c/704643/ because it's just a non-enabled framework10:28
bauzasbut we need to draw a line10:29
bauzaswe said in the spec that a microvesion would signal the fact that we enforce now10:29
bauzasand I'm still OK with this10:29
stephenfinbecause A) many extra specs are virt driver specific and therefore do different things on different clouds, B) flavor extra specs are admin-only by default, C) we've provided an escape lever for people that don't want this, and D) this is how image metadata props also work10:29
johnthetubaguybauzas: because that is what the current patches do right?10:30
johnthetubaguystephenfin: no, they mean the same thing on every cloud, just not all clouds have them available, at least that is the design10:30
bauzasjohnthetubaguy: correct, my only concern is the potential issues we could raise or any potential new keys we would want to provide10:31
johnthetubaguyplus the wild west of custom scheduler stuff, which we need to keep10:31
bauzasand that wasn't addressed in the spec AFAIK10:31
johnthetubaguybauzas: yeah, agreed with that worry10:31
bauzasthere are actually 2 different things in my mind10:32
bauzasA/ a forgotten key10:32
bauzasB/ a new key in a future release10:33
bauzasA/ sounds a bug, and I personnally agree with gibi on the fact we should just accept to backport a bugfix aiming to fix such things10:33
bauzasB/ is still undesigned in my mind10:33
*** sean-k-mooney has joined #openstack-nova10:34
bauzasbecause 'Cloud OVH' could unsupport 'myfancynewkey' while 'Cloud Vexxhost' would10:34
gibibauzas: we might now better what to do with B as when we have the situation to add an new in tree key10:34
stephenfinkick that decision to Victoria? :)10:35
bauzascompared to now where both support it (well, actually, just accepting it silently)10:35
gibibauzas: those OVH and Vexxhost keys won't be in tree keys10:35
gibiso they need to implement custom validators10:35
gibiI guess10:35
bauzasgibi: if this key is written to be used in a Victoria change, and if OVH lags with a Ussuri cloud compared to Vexxhost, then you'll see a change10:36
stephenfinonly the cloud operators themselves will see it though10:36
gibibauzas: do you mean a situation, when I have a script that creates flavors both in OVH and Veexhost?10:36
stephenfinand they'd see the same thing for image metadata10:36
bauzasthat's exactly why 3 years ago, we left the custom filters to be wildcards10:36
bauzasgibi: for a flavor, this requires an admin by default policy10:37
bauzasso i wouldn't worry too much10:37
gibibauzas: then in what situation you worry?10:37
*** rcernin has joined #openstack-nova10:38
bauzasimage properties that can be user-defined10:38
bauzasdo we expose the flavor extra specs to the users ? I think so too10:38
gibistephenfin: does the same validator code runs for the image properties?10:38
*** mensis has joined #openstack-nova10:39
sean-k-mooneybauzas: image propertiese cant be userdeined10:39
stephenfingibi: no, image properties are already validated because they're mapped to o.vos10:39
sean-k-mooneybauzas: we made them ovo years ago10:39
bauzashem you're right10:39
sean-k-mooneybauzas: yes its configurable by policy but extra specs are shown10:40
gibibauzas: so the image properties are a different story10:40
bauzasokay, I just to reconsider whether we have a problem or not10:40
stephenfindifferent but also the same, in as far as they're not tied to microversions10:40
sean-k-mooneygibi: they use to be a blank sting until like extra specs but peopel were exploiting them to pass virt driver specific stuff10:40
gibiadding an extra spec is also admin only by default, like creating flavor10:40
bauzasI said (and gibi too) that A/ (a bugfix for adding a forgotten key) isn't a problem. johnthetubaguy, you okay with this ?10:40
gibisean-k-mooney, stephenfin: ack, thanks10:41
sean-k-mooneyin the current spec we have the query arg to disable validation right10:42
* gibi asked these questions to tease out the problematic scenario10:42
johnthetubaguysorry in a meeting10:42
sean-k-mooneyif we really really want too we can have that vailadte=false flag be valdiate=2020-01 or some other validation version10:43
sean-k-mooneyso kind of like the microverions if you want the ussuri behavior then you just set teh right validation version10:43
sean-k-mooneyi think thats proably overkill untill people ask for it or clould have issue with it but its a simple enough solution to implement10:44
*** derekh has joined #openstack-nova10:45
sean-k-mooneyalso morning all o/10:46
bauzasfwiw, I just left a comment on https://review.opendev.org/#/c/708436/ to summarize our thoughts10:46
alex_xustephenfin: I just go through them all https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/use-pcpu-and-vcpu-in-one-instance10:46
alex_xustephenfin: a question for https://review.opendev.org/714658, I think we need to add data migration script, right? it isn't hard I think10:47
mensisHello, we are currently using OpenStack Pike version. i wanted to ask a question about VM Resize operation. After i resize a virtual machine, when i log in to it via SSH, i get 'kernel:NMI watchdog: BUG: soft lockup - CPU#12 stuck for 29s! [python:9846]' message. And VM is really slow right now. When i check CPU utilization, sometimes i see %1000. Any suggestions, please?10:47
bauzasstephenfin: could you just respin a revision of https://review.opendev.org/#/c/704643/ by just amending the commit msg and explaining why you add into the ignore list H328 ?10:47
bauzasstephenfin: do this and I +210:48
bauzas(+2/+W) actually10:48
stephenfinalex_xu: yeah, we could (and probably should) add an online migration for that, yes. It shouldn't be difficult. You saw my patch to update the ComputeNode.numa_topology and Instance.numa_topology fields10:49
johnthetubaguystephenfin: I think I am ok with the trade off, because its an admin API, I think if we changed the meaning of an existing extra spec (in a big way, like a rename) we would want a microversion (basically we shouldn't do that)10:49
stephenfinbauzas: cool. gimme 5. Finishing up doc rework10:49
alex_xustephenfin: the huaqiang's one, about adding pcpuset field10:49
bauzasjohnthetubaguy: stephenfin: actually, that's a good call10:50
alex_xustephenfin: oops, misunderstand your word. right, I saw your patch10:50
stephenfinalex_xu: cool :)10:50
alex_xucool10:50
bauzasjohnthetubaguy: stephenfin: we could just say that we won't support new filters with fancy new keys upstream until we somehow come up with a plan10:50
bauzas\o/10:50
stephenfinjohnthetubaguy: Yeah, sticking in TODOs is as close as I've gotten to reworking the extra specs yet10:50
bauzasproblem solved.10:50
bauzasjohnthetubaguy: stephenfin: that would solidly refrain the need for pushing new filters in-tree and would at least require a spec, which I think is always good10:51
bauzasand since people can provide their own out-of-tree filters and do what they want, they wouldn't be hit10:52
bauzasI like that plan actually10:52
gibibauzas: replied in https://review.opendev.org/#/c/708436/10:54
bauzasgibi: cool, we're on the same page10:55
*** ierdem has joined #openstack-nova10:55
bauzasI'd appreciate johnthetubaguy to agree on https://review.opendev.org/#/c/708436/16//COMMIT_MSG@1210:56
bauzasbut if he's ok, I'll change my vote to +2 (at least once I'm fully done with reviewing)10:56
openstackgerritStephen Finucane proposed openstack/nova master: api: Add framework for extra spec validation  https://review.opendev.org/70464310:57
stephenfinbauzas: ^10:58
*** tkajinam has quit IRC10:59
bauzasstephenfin: +W11:01
bauzasstephenfin: but you need to rebase the whole tree now11:01
bauzas(sorry, if it was other thing but a commit msg, I would have proposed a FUP)11:01
stephenfinI'm not sure if I do or if Gerrit will do it for me, but I'm reworking the doc patch so can do so shortly if needed11:02
stephenfinI'll wait to see if there are review comments first11:02
*** ociuhandu has quit IRC11:05
*** ociuhandu has joined #openstack-nova11:05
*** ociuhandu has quit IRC11:10
ierdemHello guys, i have a question about resizing a running VM. I tried to increase vCPU counts of a running VM and after that i connected succesfully but it is too slow to run anything. When i see process list via "top" command, i saw CPU usage is approximately 1000 percent and it  decrease sometimes but increase again. Any suggestions please?11:11
* bauzas goes out for lunch11:12
stephenfinbauzas: out out? :O11:12
stephenfin:P11:12
bauzasout from my room :p11:12
bauzasWho Let the Dogs out out ?11:13
bauzashttps://www.youtube.com/watch?v=Qkuu0Lwb5EM11:13
sean-k-mooneywell regardign fancy new keys. i would rather just define a custom: name spaces that should be used for user defined extra_specs and declare all other namespaced keys as owned by nova11:15
sean-k-mooneyand if  a new intree filter wants to add keys it gets its own namespace11:16
*** ociuhandu has joined #openstack-nova11:16
sean-k-mooneyif our of tree filter need to define keys the do it via custom:11:16
sean-k-mooneyideally with a prefix e.g. custom:myfilter_mykey11:17
*** ociuhandu has quit IRC11:23
*** zhanglong has joined #openstack-nova11:25
*** ttsiouts has quit IRC11:26
*** ttsiouts has joined #openstack-nova11:27
sean-k-mooneyierdem: i assume you have oversubsrtion enabled on your cloud?11:30
sean-k-mooneyierdem: is the host over subsibed11:30
sean-k-mooneyierdem: it should like you either have someing in the vm that is broken/maliusly consumeing cpu, the vm workload need more vcpus or the host is over subsribed11:31
sean-k-mooneyierdem: simply seeing high cpu usage in the guest is not an indication that something is wrong form a nova point of view11:32
ierdemhmm, how can i check if the host is oversubscribed?11:33
ierdemby the way, before resizing vCPU count, it was working fine11:34
kplanta resize can move the instance to another hypervisor11:35
*** JamesBenson has joined #openstack-nova11:36
kplantif you just do a 'nova hypervisor-show <uuid>'11:36
kplantyou can check vcpus vs vcpus_used11:36
kplantif vcpus_used > vcpus; you're over subscribed11:36
ierdemok, thanks i will try and return to you11:37
kplantalso vcpus_used == vcpus is a bad idea, unless nova knows about vcpus you're reserving11:38
sean-k-mooneyyes by default resize to same host is disabled so it normally does a move and will only stay on the same host if the weigher consider it to be the best host11:38
sean-k-mooneyif you enable same host resize in the config11:39
sean-k-mooneykplant: if you use vcpu_pin_set or (cpu_shared_set and cpu_dedicated_set) then yes vcpu_used == vcpu shared is fine11:40
kplantyeah, if not ideal :-)11:40
sean-k-mooneyideally you should sue the *_sets for doing host reservation instead fo the reserved_host_cpus11:41
sean-k-mooney*use11:41
sean-k-mooneyif you use the set you can choose which cpus to reserve and then you can use systemd to run the host process only on thos cores11:42
kplantyou can go further and use the bootloader11:42
sean-k-mooneyusing isolcpus? is so you should really avoid that11:42
kplantmy only complaint with *_sets is you can't have open ended ranges11:43
kplantlike "3-"11:43
kplantmakes it easier to deal with hosts with, not vastly, different configs11:43
sean-k-mooneyyou can do negations11:43
sean-k-mooney"^0-2"11:43
kplantooo11:43
sean-k-mooneyi think negated ranges works11:44
sean-k-mooneyyou can negate indivcual cores11:44
sean-k-mooneywe prably could add open ended ranges too you could always file a bug/blueprint11:45
kplanti think negated ranges provide the same result11:45
kplantnice11:45
sean-k-mooneyill triple check the code to make sure that works11:46
sean-k-mooneywe support it for cpu_realtime_mask11:46
sean-k-mooneybut i think its supported in teh *_sets too11:46
sean-k-mooneykplant: this is the code and it inclde the negation example for 1 core11:47
openstackgerritBalazs Gibizer proposed openstack/nova-specs master: template: consider openstack client besides novaclient  https://review.opendev.org/71772211:48
*** tbachman has joined #openstack-nova11:48
sean-k-mooneykplant: yep negated ranges should work https://github.com/openstack/nova/blob/31aa4a6d7f0b8c301b093ad176ee2b44b5d3cec8/nova/virt/hardware.py#L134-L14011:48
sean-k-mooneysorry miss read that11:50
sean-k-mooneythe negation only works with 1 cpu11:50
*** tkajinam has joined #openstack-nova11:51
kplantlooking at 113 -> 11611:51
kplanti think you might have been right originally11:51
kplantit tests for ^str == '^'11:51
kplantthen proceeds as normal11:51
kplantthe next try/catch is for a range11:51
kplantoof, that's ignoring what you highlighted though11:52
sean-k-mooneyhttps://github.com/openstack/nova/blob/824bc358c25564ed6603d8f94587abd8902fe5af/nova/tests/unit/virt/test_hardware.py#L132-L13311:53
openstackgerritBalazs Gibizer proposed openstack/nova master: doc: require openstack client change for every new API microversion  https://review.opendev.org/71772711:53
sean-k-mooneyunit tests are awsome :)11:53
kplantthat's still not a show stopper11:53
sean-k-mooneyalthough docs would also help11:53
*** nweinber has joined #openstack-nova11:54
kplantas long as the _set string is _only_ negations, every other vcpu should implicitly be available for scheduling, no?11:54
sean-k-mooneythat is the behavior i would want as an enduser11:54
sean-k-mooney but no unfortunetly that is not what the behvior is today11:56
kplanti probably just over thought it and gave myself a reason to be lazy and use reserved_host_cpus instead of cpu_dedicated_set11:56
kplantoh, really?11:56
openstackgerritMerged openstack/nova master: Add info about affinity requests to the troubleshooting doc  https://review.opendev.org/71509211:57
sean-k-mooneyit starts with an empty set  cpuset_ids = set()11:57
openstackgerritMerged openstack/nova master: Stabilize functional tests  https://review.opendev.org/71707011:57
kplanteek11:57
openstackgerritMerged openstack/nova master: Introduce scope_types in security groups policy  https://review.opendev.org/71678611:57
sean-k-mooneyand after we loop over every thing   cpuset_ids -= cpuset_reject_ids11:57
sean-k-mooneyit would not be hard to add that behavior.11:57
kplanti would expect assuming all cpus to actually be faster code11:58
kplantthat's more likely in line with what the result will be11:58
sean-k-mooneykplant: that code does not know how many cpus you have11:58
kplantfair, it would come from sql11:59
sean-k-mooneynot quite where this is used is on the comptue node before we store the info in the db but anyway if you do have a propoasl for improving this feel free to write it up12:00
kplantis this the better behavior? the current behavior forces the user to be explicit12:02
kplanti guess i can always submit it and get input that way12:02
*** rcernin has quit IRC12:04
*** rcernin has joined #openstack-nova12:05
sean-k-mooneykplant: if you dont set the config options all cores are assumed to be usable12:06
sean-k-mooneykplant: so the idea was of you opt in you should say what you want12:06
kplanti think that mindset still applies, just change the behavior from cpu_*_set to a merge behavior instead of replace12:08
kplanti think that's reasonable12:08
sean-k-mooneykplant: that said i have wanted to remove the reserved_host_cpus options since we first added vcpu_pin_set so makeing it nicer to use the reserved_host_cpus will help with that goal12:08
kplantjust want to make sure before i waste time with a bp12:08
kplantwaste other people's time*12:10
sean-k-mooneykplant: well we would need to keep backwards compatbliy so we could not make it merge by defaul but we could change the behavior so that if you only specify negation then we woud assume all cpus were valid and apply the negation12:10
*** ratailor has quit IRC12:11
kplantvery fair point12:11
kplantthat would make cpu_dedicated_set = "4" == all cpus12:11
sean-k-mooneykplant: ill file a bug12:12
kplantappreciate that12:12
*** nweinber has quit IRC12:13
*** nweinber has joined #openstack-nova12:13
sean-k-mooneykplant: the only issue really is that now that we have two ranges cpu_share_set and cpu_dedicated_set12:16
sean-k-mooneyit become less uesful but its still useful12:16
kplanti guess the winner between the two would be the more explicit option?12:17
kplantshare_set: "1-5"12:17
kplantdedicate_set: "3"12:17
sean-k-mooneyno you get an error if you do that12:17
sean-k-mooneyand i dont think we want that much magic in the config option parsing12:17
*** tkajinam has quit IRC12:18
kplantthat works12:18
kplantso i guess here's a difficult question12:19
kplantif you do mix shared and dedicated on the same host, and leave N cpus unspecified12:19
kplantare they shared? are they dedicated?12:19
kplantwith the current implementation they're neither12:19
sean-k-mooneyhttps://bugs.launchpad.net/nova/+bug/187109612:21
openstackLaunchpad bug 1871096 in OpenStack Compute (nova) "when only a negation is specified for cpu_*_sets we should assume all cpus are vaild and subtract the negated cpus" [Wishlist,Triaged]12:21
sean-k-mooneykplant: if you dont set the config options all cpus will be reported as VCPU resouce class which is used for shared cpus12:22
sean-k-mooneyand then we have fallback logic in the scheulder currently to allow pinned guest to land there12:23
sean-k-mooneyif you use cpu pinning however we woudl stonly prefer if you used cpu_dedicated_set12:23
sean-k-mooneyeventually we might remove the fallback and required it12:24
johnthetubaguystephenfin: did you update the api-ref for those extra params?12:25
*** udesale_ has joined #openstack-nova12:28
*** udesale has quit IRC12:31
kplantsean-k-mooney: thanks!12:31
*** Liang__ has joined #openstack-nova12:32
*** priteau has quit IRC12:35
*** ociuhandu has joined #openstack-nova12:42
stephenfinjohnthetubaguy: Oh, probably not. Will respin now12:43
johnthetubaguystephenfin: I am struggling with this validation mode stuff... should have been a discussion on the spec I know, but didn't see this one go by12:43
johnthetubaguynot going to block it or anything, just can't see how it works for users12:44
stephenfinThe discussion for that was mostly done on IRC, unfortunately :( Earlier versions of the spec didn't have the concept. It was all or nothing12:47
johnthetubaguystephenfin: that "disabled" mode I think is what worries me, I think the "permissive" isn't so bad, although I would prefer some namespacing of keys, as its easier to understand12:47
stephenfinwdym namespacing of keys?12:48
johnthetubaguywell placement traits is the example12:48
johnthetubaguyif as a user you list the traits12:48
johnthetubaguyyou can see which ones are standard, and which ones your deployer has probably made up12:49
johnthetubaguyi.e. can you go read the openstack docs to find out what it is, or otherwise12:49
johnthetubaguy... I had a lot of pushback in Rackspace on not allowing vendor extensions, which is what extra specs is right. Namespacing the crazy seems the least we could do for our users12:50
stephenfinSo insist on a namespace for any custom extra specs?12:51
johnthetubaguythink of the user listing extra specs on the flavors12:51
johnthetubaguythey want to workout what they mean, where do they look12:51
johnthetubaguyyou could just see its missing in openstack docs, but that doesn't mean too much12:52
johnthetubaguythen you grep the source code... nothing12:52
*** rcernin has quit IRC12:52
johnthetubaguythen you ask the group running your cloud, and they forgot, the person who added the flavor left last week12:52
johnthetubaguy... I got carried away there, but really its that problem I am thinking of fixing12:53
johnthetubaguysay we used a namesapce, anything invalid keys could get translated to CUSTOM_<old_name> in the new API microversion12:53
johnthetubaguythere could well be a better fix12:54
stephenfinah, see I'd been more focused on catching typos or extra specs that don't do anything (like 'hw:mem_policy', which a lot of TripleO roles were setting for years, despite it never being implemented)12:55
*** nweinber_ has joined #openstack-nova12:56
johnthetubaguy... an API that lists all supported extra_spec keys in a given release, isn't a bad way forward, I would allow adding new keys without a microversion12:56
stephenfinso for this, I was under the impression that things were pretty freeform and outside of the extra specs we control (in-tree ones), they had to stay that way12:57
johnthetubaguyyeah, agreed the typos are important to fix, and you have done that (with or without the disabled and permissive flags)12:57
johnthetubaguyI don't think we want that, its just we never got around to fixing it12:57
*** nweinber_ has quit IRC12:58
johnthetubaguysimilar conversions around scheduler hints12:58
*** nweinber has quit IRC12:58
*** nweinber_ has joined #openstack-nova12:58
johnthetubaguyalthough, that might be just what is in my head12:58
*** Luzi has joined #openstack-nova13:01
sean-k-mooneyjohnthetubaguy: the reason disabled exists is so we can in the future have a new micoroverions that modify flavor creattion without forceing validation13:01
*** nweinber_ has quit IRC13:01
*** nweinber has joined #openstack-nova13:01
sean-k-mooneyjohnthetubaguy: flavor specs can be seen by users and we allow third party filters so i dont think traslating them to custom: is valid13:02
johnthetubaguysean-k-mooney: my worry is it means someone has decided to change how some existing key works, which sounds dangerous to me13:02
sean-k-mooneyjohnthetubaguy: that is not what htis is for13:02
johnthetubaguysean-k-mooney: it would only be in a new microversion, for all unsupported keys13:02
johnthetubaguyin a new microversion, the API can basically do what we want (ish), i.e. requires client changes to uses it13:03
sean-k-mooneyjohnthetubaguy: its so that if you are using non standard flavor extra specs for filters or out of try drivers you can trun off the validation13:03
sean-k-mooneyjohnthetubaguy: the flavor creation api can but the rest of openstack has to work the same13:03
sean-k-mooneyso the filters wont be mircoverion dependant13:03
sean-k-mooneyjohnthetubaguy: i would not expect the translation to happen by default13:05
sean-k-mooneywhich is what would happen with nova client13:05
sean-k-mooneywhich is guess is another reason to prefer osc13:05
sean-k-mooneyjohnthetubaguy: as i said before i think a good way forward would be to deprecated non namespaces extra specs13:07
sean-k-mooneycreate a custom: namespcae for those that need it13:07
sean-k-mooneyand reserve all other namespaces for nova to use13:07
*** eharney has joined #openstack-nova13:07
sean-k-mooneywe have already broken the behvairo or extra specs but we have so far never removed them but i would ok tighening the contract around extra specs and how they can be modifed13:09
*** artom has joined #openstack-nova13:09
sean-k-mooneyi aggree however with the current approch in the flavor validation work.13:10
*** ociuhandu has quit IRC13:11
*** ociuhandu has joined #openstack-nova13:12
sean-k-mooneyalso extra_specs are not for vender extentions. they are missues for that but they are for feature requests13:12
openstackgerritMerged openstack/nova master: Add test coverage of existing server external events policies  https://review.opendev.org/71715513:13
openstackgerritMerged openstack/nova master: Introduce scope_types in server external events  https://review.opendev.org/71716713:13
*** mkrai has quit IRC13:13
*** lbragstad_ is now known as lbragstad13:16
*** ociuhandu has quit IRC13:17
*** dtantsur is now known as dtantsur|brb13:21
*** ccamacho has quit IRC13:25
*** mriedem has joined #openstack-nova13:26
*** amodi has joined #openstack-nova13:32
*** ccamacho has joined #openstack-nova13:35
*** mdbooth has joined #openstack-nova13:39
openstackgerritStephen Finucane proposed openstack/nova master: api: Add support for new cyborg extra specs  https://review.opendev.org/71622213:41
*** spatel has joined #openstack-nova13:44
*** mkrai has joined #openstack-nova13:44
spatelsean-k-mooney: morning! hope you guys are safe.13:44
spatelI want to talk about - https://bugs.launchpad.net/os-vif/+bug/183725213:45
openstackLaunchpad bug 1837252 in os-vif stein "[OSSA-2019-004] Ageing time of 0 disables linuxbridge MAC learning (CVE-2019-15753)" [High,Fix committed] - Assigned to sean mooney (sean-k-mooney)13:45
spatelI have two cloud running queens and stein and queens has ageing 300 but stein has 013:46
spatelits kind of security issue at this point13:46
sean-k-mooneyspatel: that has already been fixed in stien13:46
spatelhmm! why i am still seeing flooding on stein ?  i check aging and its 0 ( disabled)13:47
sean-k-mooneyspatel: you are using linux bridge or ovs?13:47
spatelLinuxbridge13:47
sean-k-mooneythen you dont have the correct version of os-vif13:48
sean-k-mooneylet me check if we have don a release13:49
spatelhmm! how do i verify and lets say i don't have then any hand fix ?13:49
sean-k-mooneyhttps://github.com/openstack/os-vif/commit/ec9d5430300c908ea9a1c64151eee7af522a44e713:49
sean-k-mooneywe merged it onto the stable branch in july13:49
sean-k-mooneyspatel: it was fixed in 1.15.213:50
sean-k-mooneyhttps://github.com/openstack/releases/commit/f23ff65cbf52ea1c5f7b2985cab56ce23a5264b713:50
spatelsean-k-mooney: how do i verify os-vif version?13:51
sean-k-mooneyhow did you install openstack13:51
spatelopenstack-ansible13:51
sean-k-mooneypackage install or source install13:51
spatelsources13:51
spatel  /openstack/venvs/neutron-19.0.0.0rc3.dev613:52
sean-k-mooneythe source install uses pip in a viruatl env so you would enter the venv for nova and do "pip freeze"13:52
*** mkrai_ has joined #openstack-nova13:52
sean-k-mooneyso "source /openstack/nova.../bin activate" followed by "pip freeze | grep vif" then "deactivate"13:53
spatelnova or neutron?13:54
sean-k-mooneyso "source /openstack/nova.../bin/activate" followed by "pip freeze | grep vif" then "deactivate"13:54
spatellet me try13:54
sean-k-mooneynova13:54
sean-k-mooneywell both should have the same version but neutorn only started using os-vif in stine or train13:54
*** zhanglong has quit IRC13:54
sean-k-mooneynova has used it for much longer13:54
*** mkrai has quit IRC13:55
spatelsean-k-mooney: i am running os-vif==1.15.113:55
stephenfinbauzas, gibi: Are you okay with me dropping the 'disabled' policy from https://review.opendev.org/#/c/708436/16/nova/api/openstack/compute/schemas/flavors_extraspecs.py@39 ?13:55
spatellook like buggy version13:56
sean-k-mooneyspatel: yep so you need to follow osa insturction to upgrade13:56
stephenfinThe idea being that no one can override our definitions of extra specs13:56
spatelis the a way to fix by hand ?13:56
sean-k-mooneyyou can fix it by hand but no i would not expect that to be the correct way13:56
*** damien_r has joined #openstack-nova13:56
stephenfinas a reminder, "strict" = key and value validation, "permissive" = value validation only, "disabled" = no validation13:57
sean-k-mooneyspatel: you should proably just do a minor update to the latest version fo stable stine13:57
spatelsean-k-mooney: i want to try by hand first and then will do upgrade to make sure it works13:57
spatelis this compute side fix or neutron server?13:57
spatelsorry controller i meant13:57
stephenfinso with this, one could configure e.g. 'myfancyextraspec=foo', but never 'hw:cpu_policy=garbage'13:57
sean-k-mooneystephenfin: i dont think we should drop disabled13:57
stephenfinhow come?13:58
sean-k-mooneyfor the resaons i suggested it was required in the first place13:58
sean-k-mooneyi want to ensure if we modify the flavor create command in the futre we can still disable validation andu use whatever was intoduced in the new microverion13:58
johnthetubaguysean-k-mooney: I am not understanding those yet, could you give a concrete example please?13:59
*** mkrai_ has quit IRC13:59
bauzasstephenfin: on a call14:00
sean-k-mooneyif we modfiy flavor create to allow setting the flavor accesss as part of a singel atoimc command in 2.xy14:00
sean-k-mooneybut still want to disable validation we cant do both without disabled14:00
johnthetubaguysean-k-mooney: sorry, I don't understand why14:00
sean-k-mooneyim assuming 2.xy is after the micoverion that intoduces validation14:01
sean-k-mooneyand since its on by default that woudl be an issue right14:01
johnthetubaguyin some future microversion, we can change what is allowed, to whatever we need it to be14:01
*** damien_r has quit IRC14:01
sean-k-mooneypermissive still spams the logs with the warning14:02
sean-k-mooneyand not everyone wants that14:02
*** mkrai has joined #openstack-nova14:02
johnthetubaguyso like we shouldn't spam the logs, but I don't understand the API request you are trying to describe and why its a problem?14:03
sean-k-mooneyits only a problm if i have a vendor exteion14:03
openstackgerritjayaditya gupta proposed openstack/nova master: Support for --overwrite flag for nova-manage placement heal_allocations command Closes-Bug:#1868997  https://review.opendev.org/71539514:03
sean-k-mooneyor out of tree driver14:04
johnthetubaguysean-k-mooney: but that is what permissive is there for14:04
stephenfinthose are permitted by permissive14:04
*** lennyb has quit IRC14:04
sean-k-mooneypermissve still does value valdiation14:04
sean-k-mooneyand there is nothing stopping out of tree driver or vendor extions form extenting the allowed values14:04
johnthetubaguycorrect, your out of tree thing is BAD if it changes an existing extra_spec, and needs fixing14:04
sean-k-mooneyproably but it proably been bad for years14:05
johnthetubaguyagreed14:05
sean-k-mooneyi dont really understand why having a way to turn off validation is an issue14:05
sean-k-mooneywe have to have that code path for older microverions anyway14:06
sean-k-mooneyso its just if disable do old code path14:06
sean-k-mooneye.g. do nothing14:06
johnthetubaguybecause it encourages really bad, basically unsupported, behaviour14:06
johnthetubaguyhttps://docs.openstack.org/nova/latest/contributor/policies.html#out-of-tree-support14:06
sean-k-mooneynot really14:06
sean-k-mooneyit would encurage it if it was the default14:07
sean-k-mooneybut its not14:07
sean-k-mooneyanyway i dont want to hold up the validation work as i want to see that land14:07
johnthetubaguyTo be clear, our dev policy says we should not support any of these out of tree things14:07
sean-k-mooneyso if it must be removed so be it but i dont think that is the correct chose14:08
*** mgariepy has joined #openstack-nova14:08
sean-k-mooneysure but this is not jsut for out of tree things14:08
bauzasjohnthetubaguy: stephenfin: I'm back14:08
*** efried has quit IRC14:08
sean-k-mooneysome people will  just not want to pay the cost of validation14:08
*** efried has joined #openstack-nova14:09
johnthetubaguysean-k-mooney: they can use the older microversion, till sometime after the the sun becomes a red giant and we up the minimum supported microversion14:10
sean-k-mooneyjohnthetubaguy: they cant if they want to use a feature in a later microverion at the same time14:10
sean-k-mooneyjohnthetubaguy: this is modifying flaovr update and create14:10
sean-k-mooneyif we modify either after we add valdiation we either always get validation or you have to do two api calls14:11
sean-k-mooneythat was my orginal argment for disabled14:11
johnthetubaguyto be clear, this is our policy on these matters: https://docs.openstack.org/nova/latest/contributor/policies.html#out-of-tree-support14:11
sean-k-mooneyjohnthetubaguy: to be clear my orginal argument was nothing to do with "out of tree support" :)14:12
johnthetubaguynow I think allowing out of tree keys in extra specs has been allowed for so long, we have to support it somehow, which technically violates the policy14:12
johnthetubaguythe question I have is how best to do that14:12
sean-k-mooneyjohnthetubaguy: we dont require microverion bumps for extra specs14:13
johnthetubaguythe disabled flag basically allows interop problems in the API14:13
sean-k-mooneyso there is also no way to determin if an extra spec is supproted form the api14:13
johnthetubaguysean-k-mooney: we will after this change14:13
sean-k-mooneyjohnthetubaguy: that was not part of the proposal14:13
* bauzas tried to look at the convo but can someone tl;dr: me ?14:13
bauzasstephenfin: ^14:13
bauzas?14:13
johnthetubaguyI was just trying to do that really14:13
johnthetubaguynot very well mind :/14:14
johnthetubaguyI think we shouldn't have the disabled flag14:14
johnthetubaguy... actually I don't like the permissive flag, but I agree we need to support the use case it allows14:14
sean-k-mooneyjohnthetubaguy: where did you think we were agreeign to do a microverion bump on extra spec changes?14:14
sean-k-mooneybecasue that was not discussed as part of the spec or planned14:15
johnthetubaguysean-k-mooney: because any time we change API validation, its a microversion bump14:15
sean-k-mooneynot with what was propsoed14:15
bauzassean-k-mooney: I think we all agree on this14:15
*** Luzi has quit IRC14:15
johnthetubaguyI am more stating the current API rules, which we haven't proposed a change to14:15
bauzas(about providing a microversion for a new extraspec key)14:15
sean-k-mooneybauzas: if we do then we can never do feature backports downstream14:16
bauzasbut we can discuss on a spec modification if you want14:16
sean-k-mooneyno api microverion bumps in backports remember14:16
bauzassean-k-mooney: indeed, but it's the same with the existing14:16
sean-k-mooneywe allow flavor extra specs to change in backports14:16
sean-k-mooneyas they are not consierd a api change downstream14:16
sean-k-mooneyflavor extra specs are considerd unversioned14:17
johnthetubaguywell, the spec was approved to make them part of the API14:17
bauzasI mean, if we were discussing about backporting a feature adding a new API modification (even for a filter key), I would have said "sorry but no"14:17
sean-k-mooneyjohnthetubaguy: i dont recall tat being in the spec14:17
johnthetubaguythese implications where not gone through, that is true14:18
sean-k-mooneyjohnthetubaguy: its not stated in the spec and i would have objected to that change without makeing flavor extraspecs ovo14:19
sean-k-mooneyhttps://github.com/openstack/nova-specs/blob/master/specs/ussuri/approved/flavor-extra-spec-validators.rst14:19
johnthetubaguysean-k-mooney: ovo would have been a good approach, that is why I suggested custom namespacing instead of the flag14:20
sean-k-mooneyspecicilly if we want to make extraspecs versioned then i wouls have propsed seperating this into two fileds14:20
bauzasor rather https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/flavor-extra-spec-validators.html ;)14:20
sean-k-mooneyone that is an ovo and the other that is a bag for random stings14:20
johnthetubaguynot sure why we need two apis, but it would work14:21
sean-k-mooneyjohnthetubaguy: backwards compat14:21
johnthetubaguyI mean instead of key namespace like traits14:21
johnthetubaguyanyways, it seems a bit late to reopen all that14:21
sean-k-mooneywell we could but it might be hard to detangel14:21
sean-k-mooneyperhaps a bit :)14:22
sean-k-mooneyso the current approch is predicated on extra_sepcs not beign microverion bumps14:22
sean-k-mooneythat is the assumtion that spec is making14:22
sean-k-mooneyif we want to be stricter we could but i kind of feel like we shoudl do that as a followup in ussuri14:23
sean-k-mooney*victoria14:23
johnthetubaguysean-k-mooney: is that stated in the spec though, I think it is just the assumption some people made, and others made the opposite14:23
johnthetubaguy... but if we add anything in the API we have to support it *for ever*14:24
sean-k-mooneyhttps://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/flavor-extra-spec-validators.html#rest-api-impact14:24
sean-k-mooneythat is all that is stated14:24
johnthetubaguyso I would rather we land the most restrictive thing, and make it more open in the future14:24
sean-k-mooneyif we do that it will never happen14:25
johnthetubaguynot if people don't need the extra things, which is great, we get a better smaller API14:25
sean-k-mooneyi would like dansmith to weigh in on this before we make any discision14:26
johnthetubaguyme too14:26
sean-k-mooneyi understand where you are comming form and i was supportive of this because i wanted the api to be stricter14:27
dansmithshould I just read the scrollback?14:27
sean-k-mooneydansmith: we were talking about the flavor extra specs validatiors, specifcally the query arg to contol validation14:28
sean-k-mooneyit would appear that some assumed after this spec all extra_spec changes would involve a micro version bump14:28
sean-k-mooneywhich raise the question why have the policy. i made the opisite assumtion that we would continue to not bump the microver when altering extra_specs14:29
dansmithis the question not the microversion for the validation param, but whether or not we bump the microversion for future validators?14:29
*** gary_perkins has quit IRC14:30
sean-k-mooneythere are two questions. do we bump the mircoverion for evey extra_spec change going forward14:30
sean-k-mooneyand what are the implciation of the validation parm in either case14:31
*** ociuhandu has joined #openstack-nova14:31
dansmithso, I thought the plan was to make validation optional through the flag, isn't that right?14:31
bauzasdansmith: see the open discussion in https://review.opendev.org/#/c/708436/16//COMMIT_MSG@1214:31
*** gary_perkins has joined #openstack-nova14:32
sean-k-mooneydansmith: yes that was the plan in the spec14:32
dansmithbecause if so, I would expect that we don't treat the extra_specs *themselves* as versioned and schema-controlled, which means no microversion for each new one,14:32
bauzasdansmith: the main concern we were discussing is whether there was an interop issue14:32
dansmithand rather the only thing we're versioning is the *behavior* of optionally validating them14:32
openstackgerritLee Yarwood proposed openstack/nova master: virt: Provide block_device_info during rescue  https://review.opendev.org/70081114:34
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Add support for stable device rescue  https://review.opendev.org/70081214:34
openstackgerritLee Yarwood proposed openstack/nova master: compute: Report COMPUTE_RESCUE_BFV and check during rescue  https://review.opendev.org/70142914:34
openstackgerritLee Yarwood proposed openstack/nova master: api: Introduce microverion 2.86 allowing boot from volume rescue  https://review.opendev.org/70143014:34
openstackgerritLee Yarwood proposed openstack/nova master: compute: Extract _get_bdm_image_metadata into nova.utils  https://review.opendev.org/70521214:34
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Support boot from volume stable device instance rescue  https://review.opendev.org/70143114:34
johnthetubaguyhmm, I guess I worry *what* validation is actually being done, i.e. what is the supported list of that given Nova endpoint14:34
dansmithjohnthetubaguy: meaning you say validation=true and you don't know if the endpoint is new enough to validate numa_nodes or something?14:34
bauzasjohnthetubaguy: dansmith: stephenfin: sean-k-mooney: gibi: honestly, can we just enable the feature by being permissive now (and not having a microversion now) and discuss about those concerns in a later change ?14:35
dansmithbecause if so, the easy way is to make validation=(yes|no|strict), and if you ask for strict validation while passing a key it doesn't support, then it fails and tells you14:35
dansmithbauzas: not sure how we would do that14:36
stephenfinbauzas: It's not permissive at the moment though, it's a no-op14:36
johnthetubaguyits strict by default in the new microversion right?14:36
stephenfinyes14:36
bauzasdansmith: I personnally feel we only need a microversion once we default to be strict14:36
dansmithbauzas: we need a microversion as soon as we add a parameter, AFAIK14:36
bauzasmy counter-proposal is to enable this feature but be opt-in14:36
johnthetubaguydansmith: hmm, I guess, although I thought we were against that approach before. I am more worried about the user listing the extra specs and trying to understand them14:37
bauzasdansmith: yeah, if you mean a extraspec key, I don't disagree14:37
dansmithjohnthetubaguy: to be honest, the importance of this feature is pretty low to me14:37
johnthetubaguythe strict/permissive/disabled thing sure needs a microversion to be added14:37
dansmithwe have no param right now, so we need a microversion regardless right?14:38
johnthetubaguydansmith: +114:38
dansmithand why as the yes/no/strict thing rejected before?14:38
dansmith*was14:38
bauzasokay, nevermind my counter-proposal, you're right14:38
bauzasa microversion has to be added anyways14:38
stephenfinit's not been rejected: that's what we have at the moment14:38
johnthetubaguyit might have been silently ignored actually...14:38
bauzasbut I just feel we need to be permissive as default until we come up with a solid consensus on what we agree14:39
sean-k-mooneyjohnthetubaguy: that would be an implemation detail of the route lib we are using if it was ignored14:39
stephenfinjohnthetubaguy: yeah, no query arg validation on that end point at the moment14:39
sean-k-mooneyits still an invalid query14:39
johnthetubaguy(just because that API didn't have any query params before)14:39
bauzasI tried to capture the problems and the proposals in https://review.opendev.org/#/c/708436/16//COMMIT_MSG14:39
dansmithbauzas: I would prefer we default to non-strict validation if that's what you mean14:39
bauzascorrect14:39
bauzas2.86 would enable the feature14:40
bauzasby being permissive14:40
johnthetubaguydansmith: OK, yeah, that would consistent with not needing a microversion for new extra specs, but lets admins opt into avoid a typo14:40
johnthetubaguyhonestly, my preference is to make these a real part of the API, and controlled in microversions, as I find the whole thing a mess to use right now14:41
sean-k-mooneyjohnthetubaguy: extra specs are generaly backend speicic too so are not portabl in genreal14:41
dansmithjohnthetubaguy: but extra_specs are open-ended anyway, so it seems odd to me to have some set of them, not even namespaced, be hard-version-controlled14:41
bauzassean-k-mooney: and I hate this14:41
sean-k-mooneye.g. they need knoladge of the way the cloud was configured (filters, virtdriver, versions)14:41
bauzassean-k-mooney: in particular the libvirt knobs we introduced14:42
dansmithsean-k-mooney: that's why I don't really want this validation in the first place14:42
bauzasthose are utterly specifics14:42
sean-k-mooneybauzas: sure but without normallising extra_spcs wich is a differnet topic we cant fix that14:42
bauzashence being permissive14:42
sean-k-mooneybauzas: no the libvirt ones are no worse the the powervm or vmware ones14:42
openstackgerritGhanshyam Mann proposed openstack/nova master: Add new default roles in tenant tenant usage policies  https://review.opendev.org/71758714:42
sean-k-mooneybauzas: many of the other virt diriver stated using the libvirt ones after we standarised them14:43
*** psachin has quit IRC14:43
johnthetubaguydansmith: I guess I like what we did with traits, and wanted that for scheduler hints and extra specs, granted its hard to do retrospectively14:43
bauzas(12:17:50) bauzas: ... or we set it to permissive by default14:43
bauzastbc, there are two alternatives14:43
bauzasoption A : let the operators opt-in14:44
sean-k-mooneyjohnthetubaguy: schduler hints are seperate14:44
bauzasoption B : microversion everything14:44
johnthetubaguysean-k-mooney: that was because we asked them to, so our API is an abstraction that is useful, rather than a bag of virt driver flags14:44
dansmithjohnthetubaguy: yeah, doesn't seem like we can do that now without a large migration of everyone's custom specs, or our system ones14:44
sean-k-mooneyjohnthetubaguy: yep and i would like if we could add back abstrations14:44
sean-k-mooneyjohnthetubaguy: i noted as much in early verion of the validation patch14:44
sean-k-mooneyjohnthetubaguy: its why we have we should move x and other comment in the code14:45
johnthetubaguydansmith: yeah, I would sadly agree that doing that is likely a waste of our valuable time at this point, FWIW, I find the documentation and structure useful on its own, it helped with image props.14:45
sean-k-mooneyya the migration of the instance embeded flavor would be a pain although we could do it if neded14:45
dansmithjohnthetubaguy: agree, it's not worth it at this point14:46
sean-k-mooneyjohnthetubaguy: well part of stephens work is to imporve the docs14:46
johnthetubaguyso my proposal earlier was to alias in the new microversion, so you add a prefix for all non-system ones, but its a pain14:46
dansmithso, crazy idea14:46
dansmithwhat if we add a new namespace prefix that itself implies that the spec is validated.. something like system:numa_nodes, which would require validation, but otherwise be the same as numa_nodes, at least for the foreseeable future?14:47
sean-k-mooneydansmith: i would prefer to define custom:14:48
dansmiththat would give us something we could eventually warn people about, deprecate the non-system numa_nodes in the future and then eventually land on anything not in system: becomes the "wild west of custom stuff"14:48
sean-k-mooneyto opt out14:48
sean-k-mooneyrather then opt in14:48
sean-k-mooneyand just validate all other namespaced extra specs14:48
bauzassorry, need to drop, urgent kitchen issue14:48
dansmithsean-k-mooney: well, the problem is right now the entire non-namespaced space is custom14:48
bauzas(kids at home, lovely)14:48
sean-k-mooneydansmith: actully not quite14:49
sean-k-mooneywe messed up one key14:49
sean-k-mooneybut more or less14:49
fricklerI'm seeing nova-api lockup when running under apache2 mod-wsgi-py3 on stein with >1 thread enabled, is this a known issue? traceback looks like this and the server stops responding after answering some few requests correctly http://paste.openstack.org/show/PhzU6ZXQnkfrDjyAAAoe/14:49
dansmithsean-k-mooney: like bauzas says, I think that throwing the switch to opt-out right now is not great14:49
sean-k-mooneyya whcih is why the valdiation parma was added14:49
sean-k-mooneyor old micorverions14:50
dansmithright,14:50
dansmithbut as johnthetubaguy has said, it's hard for people to know what is being checked or not14:50
sean-k-mooneyyes that is true14:50
johnthetubaguya list of validated namespaces is a nice way forward14:51
sean-k-mooneywe also will have different behavior between osc and nova client14:51
*** Liang__ has quit IRC14:51
johnthetubaguyideally the ones we already use would all fit into that list14:51
sean-k-mooneyjohnthetubaguy: currently that would be everyting nova knows about in tree14:51
stephenfinI can do permissive for now, then add an API to list all recognized extra specs in a future version with a new microversion. At that point, we could also switch to strict14:51
stephenfinWould that do the trick?14:51
johnthetubaguythe others are just not validated... we land at permissive for everything14:51
dansmithsean-k-mooney: actually, it'd be everything we know how to validate14:51
johnthetubaguystephenfin: what dansmith is suggesting is a bit nicer, and the reverse of my CUSTOM namespace proposal14:52
sean-k-mooneydansmith: stephenfin added validation for everyting i think14:52
sean-k-mooneydansmith: but yes you are technically correct14:52
*** dklyle has joined #openstack-nova14:52
johnthetubaguybasically be strict for all the in tree namesapces we use today, permissive for everything else14:52
stephenfinjohnthetubaguy: right, but who's going to do that work?14:52
sean-k-mooneyi had envisioned that after this if you adde an extra specs you would add a validator or extend the existing ones14:53
dansmithsean-k-mooney: it would be easier to enforce that in code with a namespace14:53
sean-k-mooneyyes i just dont want operator to have to updated there exiting instance14:54
dansmithwell, they won't for the time being14:54
dansmithwe still honor the non-namespaced version, unvalidated for the foreseeable future14:55
stephenfinjohnthetubaguy: ah, wait, you mean be strict for e.g. 'hw:' namespaced extra specs, but not 'foo:' ?14:55
johnthetubaguyso possible way forward, maybe... we drop the strict and disabled mode, we merge with permissive as the default, but we redefine the API as what dansmith said (i.e. only a microversion to add a new supported namesapce)14:55
sean-k-mooneythe other downside to a new name spaces if if we only have one the we have to put everyhtin into it14:55
johnthetubaguystephenfin: yeah14:55
*** cmorpheus is now known as cmurphy14:55
johnthetubaguywhile its not what your code currently does... its actually an API definition / docs change.14:55
dansmithjohnthetubaguy: what does permissive mean? just log errors?14:55
sean-k-mooneydansmith: yep14:55
stephenfinnope14:56
johnthetubaguydansmith: it means only validate known keys,14:56
dansmithsean-k-mooney: I was expecting we'd namespace everything, so things would become system:hw:thing14:56
stephenfinpermissive means 'hw:numa_nodes=asdassdf' is rejected, but 'sdfsdfsd:sdfsdf' isn't14:56
sean-k-mooneystephenfin: it logs unknone keys14:56
sean-k-mooneyor did you not do that in the end14:56
stephenfinand rejects them with a 4xx error14:56
stephenfinoh,sorry14:56
johnthetubaguyI didn't see a log in the code, thankfully14:56
dansmiththat's not permissive14:56
stephenfinlogs the unknown keys, yes14:56
stephenfinor does it14:56
stephenfinI'm not sure if I bothered included a log14:57
stephenfin*including14:57
bauzasyou didn't14:57
stephenfinETOOMUCHNOISE14:57
sean-k-mooneydansmith: strict rejects unkown keys with an error14:57
sean-k-mooneypermisive allows them and validates the values of keys it know about14:57
bauzashonestly, that's why I want to move on14:57
sean-k-mooneyand disabled is a noop14:57
dansmithsean-k-mooney: unknown keys in namespaces like hw: I assume?14:57
bauzaswe need to enable something and then discuss on a separate change14:57
stephenfinany unknown keys14:57
stephenfine.g. hw:cpu_policyyy14:58
stephenfinwhich is why I wanted strict14:58
*** dtantsur|brb is now known as dtantsur14:58
dansmithbut people can use their own custom keys for scheduler behavior yeah? so we can't reject *everything*14:58
bauzasstephenfin: it can still be a recommended choice14:58
johnthetubaguyyeah, I think if we are strict for all known namespaces, it would be much better14:58
bauzasstephenfin: but this would only be a doc thing14:58
bauzasjohnthetubaguy: we had this discussion 3 years ago14:59
johnthetubaguybauzas: we probably did, I might have been wrong then too14:59
stephenfindansmith: yes, and I've documented how they can add their own validators for whatever extra specs they want14:59
bauzasjohnthetubaguy: and we said we had to be accepting *any* key because we can't assume whether it's a typo or a custom key14:59
dansmithwait,14:59
stephenfinassuming they don't want to simply opt of validation14:59
sean-k-mooneydansmith: right hence the idea of having "custom:" namespace where you can add stuff we woudl never validate14:59
dansmithso in strict mode they have to write code in order to be able to use things like the json filter?14:59
bauzasthe 'custom:' namespace is good but this requires some proper signaling (and at least one cycle of deprecation)15:00
sean-k-mooneydansmith: json filter uses shcduler hint15:00
bauzasie. we say "okay, look, custom filters will continue to work without any change in U and V"15:00
dansmiththere's some filter we can hit custom fields on the flavor isn't there?15:00
bauzas"but starting from W, you will need to change your flavors to use a custom: prefix"15:00
sean-k-mooneydansmith: yes compute capablity filter and the aggartioninstnaceType filter15:01
stephenfinyes, 'CapabilitiesFilter' and 'AggregateInstanceExtraSpecsFilter' filters15:01
sean-k-mooneydansmith: both are supproted15:01
stephenfinand I've enumerated the known values for both15:01
sean-k-mooneyit jsut allows any key15:01
sean-k-mooneyif you use there namespaces15:01
stephenfin*known keys15:01
stephenfinthe values are basically wildcards because of how flexible thoseare15:01
openstackgerritLee Yarwood proposed openstack/nova master: workarounds: Add option to disable native LUKSv1 decryption by QEMU  https://review.opendev.org/70803015:01
openstackgerritLee Yarwood proposed openstack/nova master: workarounds: Add option to locally attach RBD volumes  https://review.opendev.org/70802915:01
stephenfin*those are15:01
melwittfrickler: yes https://docs.openstack.org/releasenotes/nova/stein.html#known-issues15:01
sean-k-mooneystephenfin: you allow any key in there namespace right15:02
*** beekneemech is now known as bnemec15:02
sean-k-mooneybecause those filters allow arbitray key names as long as they are in the same namespaces15:02
bauzassean-k-mooney: dansmith: stephenfin: honestly, I think I'm +2 on a deprecation policy for those filters15:02
bauzaslike, no longer supporting those wildcards at W15:03
bauzasand the same goes for custom filters15:03
sean-k-mooneybauzas: that a sperate topic15:03
bauzassean-k-mooney: no, that's tied15:03
stephenfinsean-k-mooney: not the Capabilities filter anyway - those have to map to a field on the HostState object or ComputeNode o.v.o15:03
fricklermelwitt: ha, thx for pointing me to the obvious docs15:03
bauzaswe can't be strict on enforcing keys until we fixed those messed keys15:03
sean-k-mooneystephenfin: ya i was thinking of the other one15:03
bauzas(internal team meeting FTW)15:03
sean-k-mooneybauzas: we can they have there own namespace15:04
dansmithokay I see the a.i.e.s filter already has a namespace, I didn't realize15:04
stephenfinsean-k-mooney: yes, that's wildcarded15:04
sean-k-mooneybut ya i guess i should join intrenal call15:04
stephenfinyup, since Grizzly (thanks, Intel)15:04
bauzasthat bears me (joke)15:04
sean-k-mooneydansmith: yep i made sure those fileter would still work in the spec review15:05
sean-k-mooneysicne they used to be used for dpdk/numa stuff alot15:05
openstackgerritLee Yarwood proposed openstack/nova master: DNM - Test stable device rescue tests with BFV instances  https://review.opendev.org/71005015:05
bauzashonestly, that's now 4 hours we're discussing over this and I'm out of steam now15:06
dansmithbauzas: wait until people start asking about how to do this in osc in a year :)15:07
bauzasso, again, either we make this permissive now or we just punt this for now15:07
bauzas...15:08
stephenfinjohnthetubaguy: so in this permissive for unknown namepaces only model, would we reject e.g. every unrecognized 'hw:' extra spec?15:08
bauzasdansmith: I just feel we made too many gifts in the past with custom and in-tree filters15:08
johnthetubaguystephenfin: I think that is a yes, to prevent the most obvious typos15:08
bauzaswe should ask for some return15:08
johnthetubaguystephenfin: or rather, for the user to know things are validated if its in a known namespace15:08
stephenfinjohnthetubaguy: okay, in that case do we need to continue to provide a way to disable validation in case there are people there with e.g. 'hw:something_custom' right now?15:09
johnthetubaguystephenfin: bonus, we would only need a microversion to add a new namespace15:09
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_MULTIATTACH  https://review.opendev.org/71023815:10
johnthetubaguystephenfin: I think we would just drop that flag for now, and see how people take it15:10
johnthetubaguyI like the simpler API of there being a list of validated namespaces15:10
johnthetubaguystops all the mess of migrating people towards "custom" something15:10
stephenfinokay, so unless our hand was forced we'd be essentially saying "you need to change this if you ever want to use this microversion"15:11
johnthetubaguyI think so, yes15:12
johnthetubaguywell, you can use the new API version to add your new key, and delete the old bad key15:12
stephenfinCool. Last one. Do we still need strict in this model (or that parameter in general), seeing as we are effectively strict for all recognized namespaces15:12
johnthetubaguyI don't think so15:12
johnthetubaguyat least not to start with15:13
stephenfinOkay, so I can drop that parameter15:13
johnthetubaguyyou could typo the namespace... but, hey, at least we are helping you more now15:13
stephenfinThat all works for me. gibi, bauzas, sean-k-mooney, dansmith: any significant concerns with that before I do the needful ^ ?15:13
stephenfin(last 15 lines or so of scrollback)15:14
openstackgerritLee Yarwood proposed openstack/python-novaclient master: Microversion 2.86 - Stable device boot from volume rescue  https://review.opendev.org/71495615:14
johnthetubaguystephenfin: for completeness, the old microversions are still unchanged, no validation at all there15:15
stephenfinYeah, agreed15:15
johnthetubaguysorry that all took so long, but I think what we have at the end is better15:16
johnthetubaguyand easier to document and use :)15:16
sean-k-mooneyill read back in a few minutes.15:16
lyarwoodstephenfin: ah, both of us are going for 2.86 again, want me to move to 2.87?15:17
stephenfinlyarwood: Yup :) Your call but it might make sense15:17
lyarwoodstephenfin: yup no issues, I wasn't paying attention15:18
bauzasstephenfin: pardon my French but I may have misunderstood the agreement in between you and johnthetubaguy15:20
bauzaswhat's the outcome when it's said "drop this flag" ? drop the microversion ?15:21
stephenfinkeep the microversion, but drop the '?validate' argument15:21
bauzasand the default being ?15:21
bauzaspermissive as I can understand crom 'I don't think so' from johnthetubaguy ?15:22
bauzasfrom*15:22
stephenfinstrict for all known namespaces (hw:, os:, vmware:, ...)15:22
stephenfindon't care for anything outside that set15:22
bauzasI'm cool with this15:24
bauzasstephenfin: ^15:24
bauzasno upgrade impact, pretty clear15:24
bauzasstull leaves nothing unchanged except for already-defined namespaces15:24
bauzasthat works for me15:24
*** vishalmanchanda has quit IRC15:33
*** mkrai has quit IRC15:36
*** ociuhandu has quit IRC15:36
*** ierdem has quit IRC15:37
*** ociuhandu has joined #openstack-nova15:37
*** gyee has joined #openstack-nova15:38
*** ociuhandu has quit IRC15:42
*** alex_xu has quit IRC15:57
gibistephenfin: and if you have out of tree hw:xxx key then add a custom validator?15:59
gibior use the old microversion15:59
stephenfinyes15:59
gibiso custom validators can add to in-tree name spaces?16:00
stephenfinthey shouldn't but they could, yes16:00
gibiOK16:00
stephenfinwe just prevent overriding specific extra specs e.g. 'hw:cpu_policy'16:01
gibiOK, cannot override in-tree defined validators16:01
melwittgmann: we have a minor emergency regarding the policy warning logging (see the -infra channel). it's melting the logstrash indexing16:01
gibiwhat if I have namespaced out of tree keys today? like ericsson:awesome_feature_flag ?16:02
stephenfinnothing happens until you add a validator16:02
stephenfinwe ignore all unknown namespaces16:02
gibiOK, make sense16:02
openstackgerritJohn Garbutt proposed openstack/nova master: Update quota_class APIs for db and api limits  https://review.opendev.org/71214316:02
stephenfin(and those without namespaces, fwiw)16:02
openstackgerritJohn Garbutt proposed openstack/nova master: Add stub unified limits driver  https://review.opendev.org/71213716:03
openstackgerritJohn Garbutt proposed openstack/nova master: Assert quota related API behavior when noop  https://review.opendev.org/71214016:03
gibistephenfin, johnthetubaguy: seems like a good idea.16:03
openstackgerritJohn Garbutt proposed openstack/nova master: Make unified limits APIs return reserved of 0  https://review.opendev.org/71214116:03
openstackgerritJohn Garbutt proposed openstack/nova master: Add logic to enforce local api and db limits  https://review.opendev.org/71213916:03
openstackgerritJohn Garbutt proposed openstack/nova master: Enforce api and db limits  https://review.opendev.org/71214216:03
openstackgerritJohn Garbutt proposed openstack/nova master: Update quota_class APIs for db and api limits  https://review.opendev.org/71214316:04
johnthetubaguymelwitt: oops :/16:05
johnthetubaguyprobably means the same for our users16:05
melwittyeah :( we were just talking about this the other day but I didn't realize how intense it is16:05
johnthetubaguydoes it mean almost every API call?16:06
johnthetubaguyor just a massive storm on restart?16:06
melwittwe need to get it removed asap. we were already talking to gmann about it and he was cool with removing it and doing a reno-only announcement re: policy changes16:06
johnthetubaguymelwitt: is there a patch up?16:06
melwittlooks like every API call16:06
johnthetubaguyyeah, that isn't right :/16:07
melwittno not that I know of, no patch yet16:07
*** rpittau is now known as rpittau|afk16:08
*** xek__ is now known as xek16:09
*** dtantsur is now known as dtantsur|afk16:11
openstackgerritAlexandre arents proposed openstack/nova master: Calculate disk_over_committed for raw instances  https://review.opendev.org/71703716:14
johnthetubaguyits probably all system only API calls... as our tests probably don't have a system admin user yet16:14
openstackgerritStephen Finucane proposed openstack/nova master: api: Add microversion for extra spec validation  https://review.opendev.org/70843616:16
openstackgerritStephen Finucane proposed openstack/nova master: docs: Add documentation for flavor extra specs  https://review.opendev.org/71003716:16
openstackgerritStephen Finucane proposed openstack/nova master: Drop concept of '?validation' parameter  https://review.opendev.org/71778916:16
openstackgerritStephen Finucane proposed openstack/nova master: docs: Move description of groups to document itself  https://review.opendev.org/71779016:16
openstackgerritStephen Finucane proposed openstack/nova master: docs: Add 'nova' domain and include extra specs in it  https://review.opendev.org/71779116:16
stephenfinjohnthetubaguy: I think that matches up with what you were expecting ^16:16
stephenfingibi: Also, I retooled the docs. It's _much_ nicer now (we can cross-reference!)16:17
gibistephenfin: will look but sounds awesome!16:17
openstackgerritJohn Garbutt proposed openstack/nova master: Update limit APIs  https://review.opendev.org/71270716:19
*** tesseract has quit IRC16:21
*** mensis has quit IRC16:21
* bauzas calls it a day, see you folks16:22
gibibauzas: o/ sorry I could not get to the vgpu series of yours16:23
gibimaybe tomorrow16:24
bauzasgibi: thanks and no worries16:24
bauzasgibi: in the meantime, I'll write to provide a func change for seeing the behaviour16:24
openstackgerritJohn Garbutt proposed openstack/nova master: Update quota sets APIs  https://review.opendev.org/71274916:24
gibibauzas: thanks16:24
openstackgerritJohn Garbutt proposed openstack/nova master: Tell oslo.limit how to count nova resources  https://review.opendev.org/71330116:29
*** udesale_ has quit IRC16:30
openstackgerritJohn Garbutt proposed openstack/nova master: Enforce resource limits using oslo.limit  https://review.opendev.org/61518016:33
*** evrardjp has quit IRC16:36
*** ircuser-1 has joined #openstack-nova16:37
*** evrardjp has joined #openstack-nova16:37
melwittjohnthetubaguy: I wonder if we should revert https://review.opendev.org/701624 for now? or is there some other way to approach this cc gmann16:40
gibido somebody know the irc nick of the owner of https://review.opendev.org/#/c/713089/ ? we would need to release the novaclient this week and patches are blocked on this16:41
melwittgibi: looks like it might be alisterle https://launchpad.net/~alistarle16:43
gibimelwitt: thenk I will try to ping him when he is up16:43
johnthetubaguymelwitt: are we sure we know what is causing the logs, I think it might be the scope chagnes16:43
openstackgerritJohn Garbutt proposed openstack/nova master: Add legacy limits and usage to unified limits  https://review.opendev.org/71349816:44
openstackgerritJohn Garbutt proposed openstack/nova master: Update quota apis with keystone limits and usage  https://review.opendev.org/71349916:44
openstackgerritJohn Garbutt proposed openstack/nova master: Add reno for unified limits  https://review.opendev.org/71527116:44
melwittjohnthetubaguy: I don't have deep enough knowledge to know that but I was thinking if we unmarked the things as deprecated it would stop the deprecated messages?16:45
melwittI guess I would propose the revert as -W to see for sure16:45
johnthetubaguywe have about 20 or more patches like that though16:45
johnthetubaguydo you have the log message causing the issue to hand?16:45
melwittoh really? it's not just the one that marks deprecated16:45
johnthetubaguylogs I am looking at at failing to load now... which could be related16:45
johnthetubaguywe have deprecated loads though16:46
melwittok I see16:46
johnthetubaguywe might have to change the logging config for oslo.policy16:46
johnthetubaguyassuming that is possible somehow16:46
gmannjohnthetubaguy: melwitt let me propose it to disable it temp and then find a good way16:47
johnthetubaguygmann: thanks, sounds like a plan16:47
johnthetubaguygmann: which log message is it?16:47
melwittthis is an example, https://6d82362f2cdc504b27f1-9f757b11a1d2b00e739d31e1ecad199a.ssl.cf5.rackcdn.com/717662/1/check/tempest-integrated-compute/b3260ce/controller/logs/screen-n-api.txt16:48
gmannjohnthetubaguy: it is from base rule16:48
melwittgmann: yay you're here, thank you. once we disable it I need to tell clarkb so he can dump the current logs from indexing. he said it will never catch up and we need to start fresh16:49
openstackgerritGhanshyam Mann proposed openstack/nova master: Disable the policy warning temporary  https://review.opendev.org/71780216:55
gmannmelwitt: johnthetubaguy ^^16:56
*** ociuhandu has joined #openstack-nova16:57
johnthetubaguygmann: assuming that works, lets do it.16:58
gmannjohnthetubaguy: yeah, let's wait for gate result also16:59
melwittgibi: fyi ^ patch to temporarily disable policy deprecation warnings currently making nova-api logs huge with log spam (example is linked few messages back in backscroll)17:01
*** ttsiouts has quit IRC17:03
johnthetubaguyoh my, its very chatty :/17:04
melwittyeah it's ... a lot more logging than I had realized17:05
*** ociuhandu has quit IRC17:08
dansmithonly 15k in that one log17:09
melwitt"only"17:09
johnthetubaguyah, right, that isn't so bad then (ducks)17:09
dansmithmelwitt: yes, extreme sarcasm around the only17:10
melwittI know, I thought it was funny17:10
gmannjohnthetubaguy: melwitt dansmith : to sync up on some approach on policy warning and  adopting new behaviour (  lbragstad and I discussed last week i think)17:11
lbragstado/17:12
gmannWarning: 1. do not log warning for policy changing defaults (but still not enabled as we support old defaults also) 2. do log warning where policy names are changed (granular cases in our case)17:12
*** nightmare_unreal has quit IRC17:12
gmannNew behaviour: 1. one way was to make enforce_policy as all-new-together a flag to disable the deprecation rules also. means if this flag is true then support only new defaults_scope17:13
lbragstadi was thinking about this a little more, did we want to provide a way for people to opt into the new defaults without writing to the policy file (again)?17:14
gmannyeah, that one. with enforce_scope flag right ?17:15
melwittI thought there was a way, by setting enforce_scope = True? or is that not something a user can do17:15
gmannyeah there is but that only control scope_type checks not the deprecated old rules17:16
lbragstadkind of - but we didn't expect to use that option to adjust deprecation behavior17:16
melwittoh I see17:16
gmannor we can do with new flag which we can keep it for future usual policy changes also17:17
lbragstadbut i can understand the usecase where deployers want to opt into the new policy system without having to write "new" defaults back into the policy file to get around noisy logs and the logical OR in oslo.policy17:17
gmannso when enforce_scope if true by default or we remvoe that in future we can keep new flag for deprecation things always17:18
lbragstadi guess that's the part i'd like to walk through, does it make sense to reuse that option or do we need something new?17:18
bnemecI think they're separate things. enforce_scope is a temporary thing while everyone gets their policies scope-ready, this new deprecation flag is something that we would keep indefinitely because it will have use any time a policy is deprecated for any reason.17:20
gmannIMO, something new make sense for considering the future cases17:20
*** kaisers has quit IRC17:21
bnemecAlso, I should note that we are past all of the freeze dates that apply to oslo.policy, so whatever we do it needs to be ASAP so we can request an FFE.17:21
gmannbnemec: yeah that is what i was thinking yesterday and about to ask if oslo.policy is already released ?17:22
sean-k-mooneystephenfin: ill review the new version you pushed. i think i can live with that compromise but im not entirely sure its an improvemnt17:24
bnemecYeah, that ship has sailed. Our last planned feature release (which was an FFE itself) happened Friday.17:25
gmannok17:25
bnemecI think this is worth an FFE, but it's no longer up to me alone.17:25
gmannI (or if lbragstad want to do) can propose that if all agree on that ?17:26
*** kaisers has joined #openstack-nova17:27
lbragstadgmann i'm happy to review if you push something up17:28
gmannlbragstad: ok. I will try to push that today.17:30
*** kaisers has quit IRC17:34
*** kaisers has joined #openstack-nova17:38
*** ttsiouts has joined #openstack-nova17:40
*** ttsiouts has quit IRC17:45
openstackgerritMerged openstack/nova stable/rocky: Unplug VIFs as part of cleanup of networks  https://review.opendev.org/71540417:50
*** dpawlik has quit IRC17:51
*** ttsiouts has joined #openstack-nova17:52
*** ralonsoh has quit IRC18:03
*** tonyb[m]_ has joined #openstack-nova18:06
*** lseki_ has joined #openstack-nova18:07
*** tonyb[m] has quit IRC18:08
*** lseki has quit IRC18:08
*** tonyb[m]_ is now known as tonyb[m]18:08
*** lseki_ is now known as lseki18:08
*** links has quit IRC18:08
openstackgerritSasha Andonov proposed openstack/nova master: rbd_utils: increase _destroy_volume timeout  https://review.opendev.org/70576418:11
*** d34dh0r53 has quit IRC18:13
*** d34dh0r53 has joined #openstack-nova18:14
*** ttsiouts has quit IRC18:14
*** mlavalle has joined #openstack-nova18:18
openstackgerritGhanshyam Mann proposed openstack/nova master: Fix new context comparison workaround in base tests class  https://review.opendev.org/71782518:22
*** derekh has quit IRC18:24
openstackgerritGhanshyam Mann proposed openstack/nova master: Fix new context comparison workaround in base tests class  https://review.opendev.org/71782518:26
*** ociuhandu has joined #openstack-nova18:32
*** ociuhandu has quit IRC18:52
*** ociuhandu has joined #openstack-nova18:53
*** kukacz has quit IRC18:55
*** kukacz has joined #openstack-nova18:57
*** ociuhandu has quit IRC18:59
openstackgerritGhanshyam Mann proposed openstack/nova master: Fix misc comments on policy work  https://review.opendev.org/71783518:59
*** dking_desktop has quit IRC19:20
*** ccamacho has quit IRC20:15
*** nweinber has quit IRC20:17
*** haleyb has joined #openstack-nova20:18
*** xek has quit IRC20:37
openstackgerritLee Yarwood proposed openstack/nova master: workarounds: Add option to disable native LUKSv1 decryption by QEMU  https://review.opendev.org/70803021:01
openstackgerritLee Yarwood proposed openstack/nova master: workarounds: Add option to locally attach RBD volumes on compute hosts  https://review.opendev.org/70802921:01
*** ociuhandu has joined #openstack-nova21:02
*** ociuhandu has quit IRC21:06
openstackgerritMerged openstack/nova master: Disable the policy warning temporary  https://review.opendev.org/71780221:13
*** efried has quit IRC21:34
*** efried has joined #openstack-nova21:36
sean-k-mooneymelwitt: regarding the oslo.policy cahnge. if we cant deliver that via a FFE is the plan to leave the deprecation warning disabled or do something slighly hacky like monky patch oslo.policy to do what we want for ussuri21:40
sean-k-mooneymelwitt: i know we have monkeypatched other libs in the past but i would feel weired doing it to oslo so i assume it would be left disabled21:41
melwittsean-k-mooney: yeah, I'm thinking about the same thing and I am not sure. I would think leave it disabled in the worst case scenario of not being able to solve it in oslo.policy via FFE. but I know that leaves us in a bind too wrt to any policy name changes that are also occurring21:44
melwittgmann: did you have any thoughts on this yet, what do we do if we can't get the oslo.policy stuff figured out? ^21:44
*** slaweq has quit IRC21:47
gmannmelwitt: I will push both things on oslo side today if those cannot be merged due to any reason, then i think we left with no option than keep it disabled.21:49
*** spatel has quit IRC21:49
gmannwe are saying to support the old defaults by 2 cycles at least so existing deployement are not going to break immediately which mean no-warning things also not so bad21:50
melwittgmann: ack thanks21:58
melwittthat's true we have some time to sort it out from that perspective21:59
*** JamesBenson has quit IRC22:00
*** m1p has joined #openstack-nova22:24
*** m1p has left #openstack-nova22:26
*** rcernin has joined #openstack-nova22:30
*** maciejjozefczyk has quit IRC22:33
*** maciejjozefczyk has joined #openstack-nova22:36
*** tkajinam has joined #openstack-nova22:42
*** mriedem has left #openstack-nova22:43
*** Liang__ has joined #openstack-nova23:07
sean-k-mooneygmann: the oslo team are currently asking for an FFE for some libs so if we want to get this in we should ask them to include it in that FFE23:14
gmannsean-k-mooney: yeah, I am working on changes and will ask FFE23:16
*** tosky has quit IRC23:28
*** Liang__ has quit IRC23:39
openstackgerritLuigi Toscano proposed openstack/nova master: zuul: Switch to the Zuulv3 grenade job  https://review.opendev.org/70436423:41
*** brinzhang has joined #openstack-nova23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!