14:00:03 <abhishekk> #startmeeting glance
14:00:03 <opendevmeet> Meeting started Thu Jun 24 14:00:03 2021 UTC and is due to finish in 60 minutes.  The chair is abhishekk. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:03 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:03 <opendevmeet> The meeting name has been set to 'glance'
14:00:06 <abhishekk> #topic roll call
14:00:12 <jokke_> o/
14:00:12 <abhishekk> #link https://etherpad.openstack.org/p/glance-team-meeting-agenda
14:00:16 <abhishekk> o/
14:00:30 <abhishekk> lets wait couple of minutes more for others
14:00:35 <Luzi> o/
14:00:38 <rosmaita> o/
14:00:44 <dansmith> o/
14:00:53 <abhishekk> cool lets start
14:00:57 <abhishekk> #topic release/periodic jobs update
14:01:07 <abhishekk> M2 3 weeks from now
14:01:19 <Steap_> o/
14:01:24 <abhishekk> Which also means our spec freeze date is also approaching
14:01:51 <abhishekk> I would like all cores and members to go through policy refctoring spec as soon as possible
14:02:14 <abhishekk> I would like to get it approved before next meeting
14:02:31 <abhishekk> Periodic jobs, all green again
14:02:51 <abhishekk> Moving ahead,
14:02:52 <rosmaita> \o/
14:03:04 <abhishekk> #topic M2 Target progress check
14:03:25 <abhishekk> For quotas implementation, documentation, tempest work is complete for long now
14:03:38 <abhishekk> We need REVIEWS on top priority
14:03:57 <abhishekk> To achieve its due date and get it done before M2
14:04:31 <abhishekk> So once again requesting voluntary core members and all members to spend some time on reviews as well
14:04:43 <abhishekk> Cache-API
14:05:05 <abhishekk> For cache API there is one suggestion for Delete all call
14:05:30 <abhishekk> at the moment we proposed to delete entire cache + queued images for delete all call
14:06:08 <abhishekk> So we would like to provide one flag which will tell to delete cached images only and keep queued list as it is
14:06:43 <abhishekk> In legacy tool we had separate calls for these two operations which we are merging into one in this proposal
14:06:56 <abhishekk> So any objection over this suggestion?
14:07:05 <dansmith> sounds better to be able to evict or expire one thing and not the whole cache, especially given the size
14:07:20 <jokke_> Or more like a scope where the requestor would be able to clear the current cache or the queue separately instead of just hitting them both
14:07:44 <jokke_> dansmith: so individual control exists in the current proposal already
14:07:50 <dansmith> oh I just re-read,
14:08:01 <dansmith> the separation is between the queue and currently-cached, I got it
14:08:03 <jokke_> on image basis, but the clear all whacks both
14:08:08 <dansmith> yep, sorry
14:08:37 <jokke_> so if oyur automation went crazy and queued 100 images, you could clear that queue without affecting current cache
14:08:54 <dansmith> for sure
14:09:17 <abhishekk> cool, as our proposal is already merged, how we can go for this
14:09:21 <jokke_> or wise versa, you could just whack the current cache if you know they are not in use anymore to speed up the queue processing
14:09:33 <abhishekk> 1. submit different spec (lite) for this change
14:09:39 <abhishekk> 2. amend the original spec
14:09:46 <abhishekk> 3. Cover it in documentation
14:10:15 <jokke_> As this is pretty simple adding a header to the delete call, do you guys feel we need to amend the spec or is this something we can consider implementation detail and handle on the code review?
14:11:19 <abhishekk> Ok, I will decide on approach later, lets move ahead
14:11:21 <dansmith> just amend the spec after you get the implementation done, IMHO
14:11:37 <abhishekk> ack
14:11:47 <jokke_> I'm fine with that
14:12:04 <abhishekk> Hopefully by next week we will have patches up for reviews
14:12:23 <abhishekk> #topic Image Encryption spec lite
14:12:45 <abhishekk> #link https://review.opendev.org/c/openstack/glance-specs/+/792134
14:12:55 <abhishekk> Is Luzi around?
14:13:16 <Luzi> yeah
14:14:02 <abhishekk> do you have any specific questions?
14:14:05 <Luzi> i had a question specific to dansmith - in the spec-lite you mentioned this might be worth a full spec
14:14:51 <Luzi> i want to understand why? the WIP-patch builds up on a full spec with secret Consumers
14:14:54 <dansmith> yeah, IMHO this has a lot of API impact and user behavior stuff in it, which I think is worth defining in more detail
14:15:06 <Luzi> https://review.opendev.org/c/openstack/glance-specs/+/609667/13/specs/victoria/approved/glance/image-encryption.rst
14:15:16 <dansmith> yeah, the original spec just says "meh, we'll figure this out later" for this part right?
14:15:57 <dansmith> specifically around the consumer bits
14:16:21 <dansmith> the original spec is for image encryption in general, and has lots of detail, but it just assumes the consumer stuff will be done
14:16:53 <Luzi> exactly - because the secret consumer API is still under work.
14:17:03 <dansmith> right, but the impact to glance's API is pretty substantial,
14:17:11 <jokke_> dansmith: how?
14:17:23 <rosmaita> yeah, i don't see that either
14:17:25 <dansmith> because you're talking about promising something to the users which you can't provide today, but  may in the future.. that'll be a behavioral change that you can't insulate them from
14:17:51 <rosmaita> no, all we're promising is that if they have keys available, we will use them
14:17:57 <rosmaita> but key management is up to them
14:18:03 <dansmith> because the implication, according to the original spec, is that when you've created an image based on the key, that you've grabbed a refcount on that key
14:18:13 <Luzi> so glance will only deal with two things: registering a secret consumer when creating an image and unregistering when it is deleted.
14:18:24 <dansmith> "Glance register as a consumer of that
14:18:24 <dansmith> key (secret in Barbican [1]) when the corresponding encrypted image is
14:18:24 <dansmith> uploaded and unregister as a consumer when the image is deleted in Glance."
14:18:26 <dansmith> from the spec ^
14:18:49 <rosmaita> i don't know that the consumer API is keeping an actual refcount
14:18:49 <dansmith> but what the lite spec is describing is just making that silently fail
14:19:18 <dansmith> rosmaita: it's keeping a list of consumers, right? I'm just shortening that to refcount because that's the goal of it, IMHO
14:19:20 <jokke_> dansmith: so what comes to the Glance side, is that we store the information and have dedicated property keys for them. Yes we do have in the full spec stating that we expect the consumer ap being present for this so we can register as consumer, and drop our registration, but glance is not doing the encryption nor the key management for those encrypted images, so I fail to see how this affects
14:19:26 <jokke_> our API
14:20:15 <rosmaita> dansmith: yeah, whether its an actual refcount is not relevant, i was just wondering
14:20:34 <dansmith> jokke_: because the spec says that we will have registered as a consumer of the key, which is the right behavior.. if we don't do that, and just return the image anyway, the user thinks they've got a hold of the key when they don't
14:20:45 <Luzi> dansmith, it is all up to the user. The secret consumers are just a help for users so a "secret delete" will not succeed as long as there are consumers (you should be able to force it though)
14:21:18 <dansmith> jokke_: so my suggestion was to provide an intent flag during creation of an image with a key, saying "I really care that you secured a reference as a consumer on the key" or "I don't really care"
14:21:19 <abhishekk> dansmith, are you worrying about accidental deletion of key by user?
14:21:23 <Luzi> wait - lets start with the workflow here
14:21:27 <dansmith> today we can honor the latter, but can't honor the former
14:21:41 <dansmith> but after the consumer api is done, we can enable the former, and keep the same api semantics
14:21:51 <jokke_> That said, like with lot of other our new features, I think would make lot of sense clearly mark this feature as EXPERIMENTAL until the secret consumer side is worked out and not claim it stable without
14:21:56 <dansmith> otherwise people have to know what version of glance is on the backend in order to know if it should work ornot
14:22:47 <dansmith> Luzi: yes, as I've said, I understand that it's just informational to the user, but it's a thing they are expecting to have to avoid losing data by deleting a key that is still in use
14:23:12 <dansmith> jokke_: well, leaving it experimental is certainly better than nothing, I'd just rather make it very explicit
14:24:20 <jokke_> dansmith: while I see your point and somewhat agree with it, As I understood the point with this lite-spec was to get the implementation parts moving while waiting for barbican (that we have done for over year now)
14:24:38 <abhishekk> I guess our intention behind allowing this place holding to get things moving
14:24:41 <Luzi> dansmith, i would really like to have the secret consumers ready now. but the barbican team is drowning in other work as it seems and they are at a point where i cannot help them.
14:24:47 <dansmith> jokke_: yep, and all I'm saying is that if we make the API intent more explicit, we're totally good to do that :)
14:24:53 <jokke_> To have the bits in place to actually iron the rest out meanwhile. And experimental would be the approach for that
14:25:56 <dansmith> POST /v2/images?consumer=required  and POST /v2/images?consumer=optional
14:26:18 <dansmith> the former would always be 409 today, the latter would be 200, and we can go straight to supported for this API
14:26:37 <jokke_> dansmith: with the intent flag on the API call, the problem is that we're stuck with it in the API ... ok, if it's experimental we in theory could drop it before making the API stable, but I'd rather avoid doing backwards incompatible changes to the API
14:26:40 <dansmith> the users today have to declare "I understand that I may not have a refcount on my key"
14:27:07 <dansmith> jokke_: well, it always makes sense to say "I don't care about the accounting" so it's not really something we're stuck with, it's just something we get
14:27:31 <dansmith> jokke_: in fact, you could s/optional/ignore/ and then we could avoid the trip to barbican anyway, even once it's supported, if the user doesn't care
14:28:00 <dansmith> but see, all I said was "I think we might want to discuss this in a real spec" ... for this reason :D
14:28:39 <jokke_> IMO that ^^ is the worst option around and will confuse shit out of everyone :D
14:29:09 <Luzi> i don't think we should support a mix between being able to add a secret consumer and just not do it...
14:29:11 <dansmith> which? the spec or ignore?
14:29:18 <jokke_> dansmith: ignore
14:29:23 <dansmith> ack
14:29:35 <jokke_> sorry, I was fixing my typos while you posted the spec comment
14:30:05 <Luzi> still this is confusing to me - as i am the one writing things since years now...
14:30:24 <Luzi> we have an original spec with the secret consumers
14:30:45 <rosmaita> the other thing is, isn't it possible for a user to upload the image data, and then add the metadata later? or are we disallowing that somehow?
14:31:05 <abhishekk> I don't think we allow it
14:31:11 <dansmith> they have to be write-once
14:31:21 <dansmith> otherwise people will try to re-encrypt their images with a new key right? :)
14:31:21 <Luzi> if you want to upload an encrypted image some sort of metadata is needed
14:31:40 <Luzi> re-encrypting would be a whole new issue
14:32:12 <Luzi> well but whats different to the current situation in cinder for example
14:32:26 <jokke_> I agree with the write once and the key metadata should be present when the container type is encrypted before the image goes active
14:33:00 <Luzi> you can go to barbican and wildly delete random keys... or is there any way to prevent this?
14:33:21 <dansmith> the whole point of the consumer thing, is to at least *inform* the user that deleting a key will orphan real data
14:33:39 <dansmith> they should still be able to do it, of course, but the accounting is seriously important
14:34:06 <dansmith> also for auditing.. knowing which images or volumes or whatever are all encrypted with the same key is what keeps auditors up at night
14:34:17 <jokke_> Luzi: the point was that we should protect the glance metadata. If you go and delete those keys from barbican, the image is useless after, just delete it and re-create with up-to-date metadata
14:34:23 <dansmith> and making it easy to create a thousand images where all the accounting silently failed is problematic
14:35:12 <jokke_> Luzi: but we do have mechanism for that already, so it's one of those implementation details that are pretty easy to enforce
14:35:38 <jokke_> and imo not relevant for the intent of this light spec to just get the work moving while we wait for barbican
14:35:44 <Luzi> jokke_, with "you" I mean a user or project administrator
14:36:07 <Luzi> not glance
14:36:22 <jokke_> Luzi: correct, glance will never delete those keys
14:36:35 <abhishekk> So coming back to square 1, how much time it will take barbican to complete consumer work?
14:36:41 <rosmaita> ok, i guess i agree with dan (assuming this is his point) ... we need to amend the original spec assuming that there is no secret consumer api and say precisely what the workflows are
14:36:45 <jokke_> only once the consumer api is done letting brbican know that we refer to them
14:37:20 <dansmith> rosmaita: that's not my point, but that would be better than nothing :)
14:37:33 <Luzi> abhishekk, i really have no idea - could be weeks or months... :/
14:37:44 <abhishekk> Luzi ack
14:38:22 <jokke_> Based on the fact that we have been waiting this for over a year and the last update I got was "maybe in this cycle" I'd not hold my breath with it
14:38:35 <Luzi> so instead of this lite spec you want me to edit the original spec - once for now without the secret consumers and once when we add them?
14:39:07 <abhishekk> Luzi I think you can convert this lite spec to spec
14:39:12 <dansmith> is everyone else opposed to an intent flag? because that lets us out of this now and later when we get the consumer api
14:39:21 <dansmith> the default can be "optional" and nothing really has to change today
14:39:39 <abhishekk> and mention the parent/former spec will be the feature work on top of this
14:40:32 <jokke_> I'd prefer not. If we change the spec to state we are not using the consumer api that gives the expectation that we won't ... I'd rather really just start merging the bits Luzi need to keep the momentum going without expecting the secret consumer part is the first thing we depend on and not claim the work done before that's in place
14:41:03 <jokke_> I thought that was the point of this lite-spec
14:41:04 <dansmith> okay, and when the consumer api comes, we do what? convert to always using it *and* failing hard if we can't register?
14:41:19 <jokke_> dansmith: correct
14:41:23 <jokke_> as intended
14:41:25 <Luzi> dansmith, that would be the way
14:41:40 <dansmith> okay, as long as we don't squelch the failures (which is what the lite spec proposes) then I'm fine with that
14:42:15 <abhishekk> ack
14:43:20 <abhishekk> Luzy hopefully your concern is addressed now
14:43:48 <Luzi> it's a bit more clear to me, what the problem is
14:44:04 <abhishekk> Limited time, lets move ahead and circle back to it in open discussion if we have time left
14:44:19 <abhishekk> #topic Policy refactoring
14:44:26 <abhishekk> Master spec: https://review.opendev.org/c/openstack/glance-specs/+/796753
14:44:27 <abhishekk> Tests refactoring lite spec: https://review.opendev.org/c/openstack/glance-specs/+/797593/1
14:44:50 <abhishekk> As I said we are approaching spec freeze and this time I want to honor it
14:44:59 <abhishekk> Kindly provide some time to review the spec
14:45:07 <abhishekk> I have one +2 on the spec
14:45:22 <abhishekk> rosmaita, jokke_ Steap_ smcginnis ^^^
14:45:41 <rosmaita> ack
14:45:47 <abhishekk> Also I would like to have one session about this work sometime in next two weeks
14:46:05 <abhishekk> so if anyone has any concerns then we can discuss it there only
14:46:31 <dansmith> abhishekk: one session? meaning a voice call?
14:46:44 <abhishekk> yeah
14:46:48 <dansmith> roger
14:47:35 <abhishekk> I will shoot one mail for the time slot and we could finalize the one
14:48:04 <jokke_> ++
14:48:20 <abhishekk> If we fail to get it done this time then I guess it will never going to happen in future
14:48:50 <abhishekk> that's it from me for today
14:49:07 <abhishekk> Kindly put some time in reviews as well
14:49:16 <abhishekk> #topic Open discussion
14:49:51 <abhishekk> Nothing from me
14:50:15 <dansmith> please review quota patches!
14:50:32 <abhishekk> yes
14:50:57 * Steap_ is on his way to the reviewing room
14:51:00 <abhishekk> I guess if that's it then we can wrap up and utilize remaining time in review
14:51:22 <abhishekk> Thank you all
14:51:52 <abhishekk> Have a nice weekend ahead
14:52:01 <abhishekk> #endmeeting