21:00:08 <notmyname> #startmeeting swift
21:00:09 <openstack> Meeting started Wed Jan 20 21:00:08 2016 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:12 <openstack> The meeting name has been set to 'swift'
21:00:13 <notmyname> hello, everyone
21:00:19 <notmyname> who's here for the swift meeting?
21:00:24 <Zyric_> Hello
21:00:24 * onovy 
21:00:27 <ho_away> o/
21:00:29 <minwoob> o/
21:00:29 <tdasilva> hi
21:00:29 <cutforth> o/
21:00:30 <cschwede> o/
21:00:32 <kota_> hello
21:00:32 <cbartz> Hi
21:00:33 <peterlisak> hi
21:00:33 <blmartin> o/
21:00:38 <mattoliverau> o/
21:00:42 <jlhinson> o/
21:00:44 <eranrom> hi
21:00:45 <acoles> here
21:00:50 <jrichli> hello
21:00:51 <torgomatic> .
21:00:59 <bill_az> Hi
21:01:02 <sgundur> hi
21:01:24 <notmyname> good group. welcome!
21:01:31 <notmyname> agenda is
21:01:32 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:01:40 <notmyname> several things to start with
21:01:44 <notmyname> #topic swift release
21:01:55 <notmyname> a few things here on the topic of a swift release
21:02:20 <notmyname> first, pip did a major version upgrade yesterday, and this sortof broke the whole openstack world
21:02:34 <notmyname> so the gate seems to be at more than 28 hours now
21:02:36 <joeljwright> not just the openstack world, grumble grumble
21:02:43 <notmyname> heh
21:02:45 <clarkb> and greater python world. ya
21:02:54 <notmyname> we've got 8 patches that are already in the gate that will land
21:03:18 <notmyname> including the CVE fixes disclosed this morning
21:03:23 <wbhuber_> o/
21:03:51 <notmyname> #link https://bugs.launchpad.net/swift/+bug/1493303
21:03:52 <openstack> Launchpad bug 1493303 in OpenStack Object Storage (swift) "[OSSA 2016-004] Swift proxy memory leak on unfinished read (CVE-2016-0738)" [Critical,Confirmed]
21:04:02 <notmyname> in case you didn't see the security bug and patch, there it is ^
21:04:31 <notmyname> summary is that if a client starts to download a large object and then stops, the server will leak a socket file descriptor and a small amount of memory
21:04:35 <notmyname> you should upgrade
21:04:57 <notmyname> lots of other good stuff in this release too
21:05:18 <notmyname> you can see the proposed changelog in patch 269911
21:05:18 <patchbot> notmyname: https://review.openstack.org/#/c/269911/ - authors and changelog updates for 2.6.0
21:05:30 <notmyname> hasn't landed yet, but I'll approve it to land shortly
21:05:39 <notmyname> which gets to the next thing
21:05:54 <notmyname> there are 2 patches not approved that need to land in this release
21:06:09 <notmyname> first is the authors changelog one. that's just boilerplate and shouldn't worry anyone
21:06:26 <notmyname> the other is patch 269811
21:06:26 <patchbot> notmyname: https://review.openstack.org/#/c/269811/ - Bump eventlet min version to 0.17.4
21:06:40 <notmyname> it was suggested to bring up this one in the meeting
21:07:11 <notmyname> the summary is that we've got some IPv6 support in swift (and have for a while, actually), but unless you are running at least eventlet 0.17, it doesn't work
21:07:23 <notmyname> our current min eventlet version says 0.16.1
21:07:35 <cschwede> i’ll check this for recent rhel/centos versions
21:07:39 <notmyname> this patch makes it match what's in global requirements. 0.17.4
21:07:42 <notmyname> cschwede: thanks
21:07:46 <onovy> debian is fine
21:07:51 <notmyname> onovy: thanks
21:08:08 <onovy> ad changelog: what about that one commit i wrote in comment?
21:08:10 <notmyname> looking at ubuntu, even 0.16.1 is later than what's in either precise or trusty
21:08:26 <notmyname> so no new challenge there
21:08:34 <acoles> notmyname: do you know what the # MIT means after the eventlet version?
21:08:40 <notmyname> acoles: the license
21:08:50 <acoles> notmyname: oic. thanks
21:09:05 <notmyname> basically, if we change *anything* in the requirements file, it must be an exact string match for the line in global requirements
21:09:22 <notmyname> which also means that our choice is to either keep 0.16.1 or to go to 0.17.4. there is no other option
21:09:34 <notmyname> and if we stay with 16.1, IPv6 does not work
21:09:37 <onovy> +1 for upgrade
21:09:43 <notmyname> IMO, we absolutely should upgrade
21:09:58 <cschwede> +1
21:10:05 <Zyric_> +1
21:10:20 <torgomatic> agreed; otherwise we'll just end up introducing workarounds for bugs already fixed in eventlet
21:10:22 <onovy> we are using 0.17.4 eventlet in production with swift 2.3.0
21:10:31 <notmyname> torgomatic: exactly!
21:10:31 <onovy> and everything working fine
21:10:50 <notmyname> cschwede: since you're one of our RH representatives, can you please add that +1 comment to gerrit?
21:10:55 <notmyname> and I'd be fine if you also +A it
21:11:00 <notmyname> or someone else can
21:11:16 <cschwede> notmyname: yep, just checked RDO, now checking OSP; will add my +2/A afterwards
21:11:20 <notmyname> cschwede: thanks
21:11:21 <notmyname> ok
21:12:35 <notmyname> onovy: FWIW, I didn't mention that, not because it isn't a cool thing, but because it looked to me to be a very minor change. it's great, but it was a value judgement when I wrote it up. nothing against the work at all. there's a lot of stuff in each release bundled under "other minor changes and bug fixes" :-)
21:12:46 <onovy> mkey
21:13:25 <notmyname> so after cschwede puts +A on the eventlet patch, I'll land the changelog one, and whenever that lands, I'll do the release tag stuff.
21:13:33 <notmyname> so based ont he gate, that might be friday? who knows.
21:13:51 <notmyname> any other questions or comments on this release?
21:14:21 <notmyname> ok, next topic
21:14:22 <eranrom> nonameentername,: IBMs gratitude for the last minute cs review
21:14:30 <notmyname> eranrom: you're welcome
21:14:34 <notmyname> #topic hackathon
21:14:46 <notmyname> the hackathon in bristol is coming up!
21:15:02 <notmyname> acoles got the hotel block extended, so if you haven't booked a room yet, DO IT NOW!
21:15:42 <notmyname> in the next few weeks, be putting together your ideas of what to discuss there
21:15:57 <notmyname> I think it will be a very densely packed week. lots of stuff to go over
21:16:33 <notmyname> acoles: anything else to add? any logistics we need to know about or plan for?
21:16:53 <mattoliverau> Keep good notes for those of us who can't make it
21:17:12 <jrichli> mattoliverau: we will miss having you there!
21:17:39 * notmyname is going to visit mattoliverau next week!
21:17:43 <acoles> only that i'm working on plans for the social evening - if anyone is bringing partners let me know so I can account for them in headcount
21:18:10 <notmyname> acoles: thanks for working on it
21:18:27 <acoles> we'll keep an empty seat in honour of mattoliverau
21:18:33 <joeljwright> :)
21:18:47 <notmyname> a seat and a half for little mattoliverau-ette
21:18:58 <kota_> lol
21:19:02 <notmyname> next topic
21:19:07 <notmyname> #topic community status
21:19:24 <notmyname> so last week I went on a little bit about what's going on in the community
21:19:49 <notmyname> and I asked everyone some questions: what are you working on, what's important, what are you happy about, and what are you worried about
21:19:58 <notmyname> you've all given me great responses!
21:20:25 <notmyname> I'm putting them together, and I'll share that, but here's some current community highlights
21:20:35 <notmyname> first, active contributors continues to grow
21:20:39 <notmyname> #link http://d.not.mn/swift_active_contribs.png
21:20:55 <notmyname> and nearly everyone had very positive things to say about the community
21:21:47 <notmyname> one of my favorite comments was (paraphrased): "before I got involved in Swift, I thought it wouldn't work well because of all the different time zones. But it's absolutely awesome."
21:21:58 <notmyname> (that last part was a direct quote)
21:22:09 <notmyname> there were many comments along those lines
21:22:14 <notmyname> so I'm really happy to hear that
21:22:47 <notmyname> there is some room for improvement for helping newer community members get involved
21:22:55 <notmyname> so keep being friendly and awesome :-)
21:23:05 <kota_> +1
21:23:17 <notmyname> as far as what's important, there was a lot of overlap
21:23:20 <mattoliverau> notmyname and I will just have to have our own midcycle in Geelong next week (thanks all, turns out going fast on the highway means irc comes in bits and pieces)
21:24:05 <notmyname> the top 3 things that stuck out as far as the "what's important" question goes are: container sharding, encryption, and fast-post
21:24:17 <notmyname> which I don't think surprises anyone
21:24:40 <acoles> wow someone said fast-post, it wasn't me!
21:25:00 <notmyname> it was the 3rd most mentioned item!
21:25:11 <acoles> trending up since Austin then :)
21:25:19 <cschwede> now i’m curious about the 4th-most mentioned…
21:25:28 <mattoliverau> acoles: \o/ fast post
21:25:32 <acoles> cschwede: lol
21:25:32 <notmyname> however, note that the total list of "what's important" is long. turns out when you ask someone what they are working on and what's important, they tell you that what they are working on is important :-)
21:25:52 <notmyname> cschwede: container sync, swift3, and RBAC
21:26:35 <cschwede> notmyname: thx! i’m wondering if no one (except me?) mentioned partition power increase…
21:26:40 <notmyname> oh! just looked at my notes. another thing multiple people mentioned was "seeing more >10PB clusters in production"
21:27:07 <notmyname> (mentioned as something exciting that's happening)
21:27:15 <mattoliverau> cschwede: I mentioned it too
21:27:32 <mattoliverau> Somewhere
21:27:44 <notmyname> mattoliverau: right here, right now ;-)
21:27:55 <notmyname> ok, so on to some of the "what's worrying you" answers
21:27:59 <mattoliverau> Lol
21:28:06 <cschwede> anyways, can’t wait to see notmyname’s summary - thx for working on this!
21:28:11 <notmyname> there are 2 thing that stand out, again to no surprise to anyone
21:28:18 <notmyname> first is "slow review times"
21:28:33 <ho_away> cschwede: +100
21:28:34 <notmyname> and general slow speed of integrating changes
21:29:03 <notmyname> this is not new (nor is any attempt to fix it)
21:29:19 <notmyname> so it's something to keep working on. but it's a really hard problem, it seems
21:29:39 <notmyname> I have an idea here, but I want to flesh it out more before proposing it to the community
21:30:08 <notmyname> the second most common thing said (by a large margin) is "increased complexity with associated costs"
21:30:29 <notmyname> basically, people feel like swift is getting pretty complicated, and it's hard to reason about what's going on and keep it all inyour head
21:30:39 <notmyname> so if you've been feeling that way, you're not alone!
21:31:26 <notmyname> I think this one will take some very specific and deliberate action on all of our parts to overcome
21:31:47 <notmyname> and it's a topic that should be addressed at the hackathon (if not specifically, then at least in general)
21:31:53 <notmyname> so that's what I've got!
21:32:19 <notmyname> well, I've got a lot more raw data, and some paragraphs summarizing it, but that's the highlights of the raw data
21:32:30 <notmyname> thank you for giving me your feedback
21:32:45 <joeljwright> thanks for putting it all together!
21:32:48 <notmyname> any questions or comments? anything you want me to specifically look for?
21:32:50 <eranrom> notmyname,: I think this is a great initiative!
21:33:14 <jrichli> +1
21:33:20 <kota_> +1
21:33:23 <ho_away> +1
21:33:35 <minwoob> +1
21:33:57 <notmyname> patches topic time
21:34:10 <notmyname> #topic patches -- ionice
21:34:15 <notmyname> patch 238799
21:34:16 <patchbot> notmyname: https://review.openstack.org/#/c/238799/ - Change schedule priority of daemon/server in config
21:34:32 <notmyname> onovy and peterlisak shared some results of testing with this patch
21:34:39 <notmyname> which is very helpful!
21:34:51 <notmyname> in the past we've raised 2 questions about this patch
21:34:59 <notmyname> (1) does this actually do anything good?
21:35:12 <onovy> i think first question is answered: yes
21:35:27 <notmyname> to which we now know the answer (at least for auditors) is unequivocally yes
21:35:39 <notmyname> and IMO "just" the auditors is good enough
21:36:03 <onovy> yep. we know i works for auditor and doesn't work for reaper
21:36:07 <onovy> it
21:36:12 <onovy> (now)
21:36:26 <notmyname> (2) should we do this in swift instead of leaving it as an ops/distro thing?
21:36:39 <onovy> simple answer would be: do it in init script
21:36:46 <onovy> BUT i say: in swift
21:37:11 <notmyname> that's what I'd like to get some feedback on today
21:37:16 <notmyname> what do people think?
21:37:28 <onovy> 1. swift-init vs. init scripts confusion
21:37:44 <onovy> 2. future work, where reaper can work too
21:37:50 <notmyname> should swift be setting ionice via config? or should that be set in distro init scripts (by either the distro/packager or the operator)?
21:38:25 <notmyname> what do you think?
21:38:47 <torgomatic> the auditors are sort of special; they do a ton of IO, and all their work is local. most swift daemons send messages on other cluster nodes and induce work there, so setting ionice and such on one end doesn't really buy us much
21:39:01 <onovy> torgomatic: yep, my point "2"
21:39:04 <torgomatic> but it's certainly good to have for the auditors
21:39:10 <onovy> this is just first patch
21:39:35 <onovy> i can imagine we can set some kind of header for this type of prioritization
21:39:41 <onovy> and other servers should respect it
21:39:44 <notmyname> note that in past meetings, ops people have told is it's fine to do ionice in swift. so it's a question of the "right" place for it. eg we could say "not in the code, but let's document it"
21:39:49 <onovy> then it will work for reaper too
21:40:25 <torgomatic> sticking to the question of nice/ionice applied in configs vs. init scripts, I'm fine with either one
21:41:50 <notmyname> any other comments there?
21:42:18 <onovy> peterlisak: ?
21:42:26 <notmyname> ...not everyone all at once
21:42:50 <peterlisak> onovy, no comments, you described it very well ;)
21:43:39 <notmyname> ok, IMO it's fine to add for swift
21:43:48 <onovy> perfect
21:43:50 <acoles> by default the patch leaves things as they are correct - has to be actively added to config?
21:43:55 <notmyname> right
21:43:57 <onovy> acoles: yep
21:44:01 <acoles> k
21:44:08 <jrichli> acoles: i was just now typing something like that up :-)
21:44:23 <notmyname> jrichli: I'd imagine that this would be very interesting for softlayer
21:44:29 <peterlisak> acoles,  it has not to be added
21:44:45 <notmyname> onovy: peterlisak: I'll add a comment to the patch
21:44:47 <notmyname> peterlisak: ?
21:45:22 <onovy> default is: do nothing / don't change ionice/nice
21:45:31 <mattoliverau> I'll ask the Rackspace Private Cloud ops. Openstack Ansible swift is currently using init scripts, but maybe that's because they can't set it in config.. I'll ask the question.
21:45:52 <acoles> peterlisak: i mean, unless an admin actively adds this to their config, there will be no change in their daemon operation?
21:46:14 <mattoliverau> But happy to go either I guess. Is there an Ops meetup this cycle ;)
21:46:17 <mattoliverau> ?
21:46:19 <onovy> mattoliverau: and if they will not be happy with this inside swift, they can use it in init scripts
21:46:48 <onovy> but big plus for inside swift is you can do: swift-init all restart
21:46:55 <onovy> and ionice/nice is set correctly
21:47:21 <notmyname> ok, I think we're good on this topic
21:47:26 <onovy> thanks all
21:47:35 <notmyname> #topic auditor watcher patches
21:47:49 <notmyname> patch 212824 and patch 268830
21:47:50 <patchbot> notmyname: https://review.openstack.org/#/c/212824/ - Let developers/operators add watchers to object audit
21:47:51 <patchbot> notmyname: https://review.openstack.org/#/c/268830/ - Let developers/operators add watchers to account a...
21:47:54 <mattoliverau> onovy, peterlisak: thanks for all your work on it :)
21:48:00 <notmyname> these 2 patches are very closely related
21:48:13 <onovy> mattoliverau: thanks for review :)
21:48:16 <Zyric_> The account audit watcher has diverged slightly in that it uses inheritance to get a logger while the original object auditor watcher has it passed in. I would like to know if I'm going in the right direction with this, and whether they should be kept apart or integrated.
21:48:38 <Zyric_> The idea behind audit watchers is to allow operators to run custom code on each object/account in a cluster without adding additional I/O by using data derived from Auditors.
21:48:39 <peterlisak> acoles, yes, default is no change
21:49:13 <acoles> peterlisak: thanks
21:49:19 <notmyname> as an example of where the auditor watchers is useful, think of the ops meeting we had in tokyo. one request was a common way to get a list of accounts in the cluster
21:49:25 <notmyname> and Zyric_ is working on that now, actually
21:49:38 <notmyname> implemented as an auditor watcher hook
21:49:48 <onovy> shouldn't be account list supported natively?
21:50:10 <notmyname> onovy: never has been before. that's what Zyric_ is working on :-)
21:50:11 <onovy> i like idea behind watchers, but that acount list... i don't know
21:50:20 <onovy> notmyname: Zyric_ is working on watchers
21:50:33 <onovy> and 1 example watcher which appends account name to file
21:50:46 <torgomatic> another use case: I've got people in my company working on search, and they want a way to crawl existing objects and make sure they get indexed once a cluster turns on search stuff
21:50:56 <onovy> i mean something like native sqlite db, sharded across cluster
21:51:13 <onovy> torgomatic: yep, that's correct usecase for watcher (for me)
21:51:15 <torgomatic> and a watcher lets me do it (a) without more IO load, and (b) without having to fork the auditor, and (c) without bugging you guys :)
21:51:29 <mattoliverau> Zyric_: can you get account also have the option of passing in a logger (to override the inherited one) we may never need to, but it'll give both a similar API. (Noting I haven't looked at the code yet)
21:51:37 <notmyname> onovy: Zyric_'s patch will be changing to not be in the current "examples" directory
21:52:01 <eranrom> cough...storlets...cough :-)
21:52:13 <mattoliverau> Zyric_: I'll have at least a fist parse of them today.
21:52:21 <notmyname> so yeah. tons of uses to do "something" when an auditor comes across a thing in swift
21:52:23 <onovy> notmyname: so it will append account name to /tmp file by default?
21:52:48 <notmyname> onovy: pay more attention to the concept of the account auditors that the specific patch as it is
21:53:00 <Zyric_> mattoliverau: thanks, I'll add the logger back.
21:53:02 <notmyname> onovy: I'm working with Zyric_ as part of her outreachy internship on this project
21:53:04 <onovy> as i sad. i like watchers
21:53:14 <onovy> but don't like this correct use-case :)
21:53:21 <onovy> /correct/exact/
21:53:24 <cschwede> torgomatic: that, +1!
21:53:25 <Zyric_> onovy: Latest patch saves to recon-cache :)
21:53:37 <onovy> Zyric_: and still appends-only?
21:53:49 <acoles> Zyric_: mattoliverau +1 for making interfaces as similar as possible. (I also have not looked at code yet!)
21:54:27 <mattoliverau> k, I need to head in to the doctors appointment. So I gotta go. bbl
21:54:36 <notmyname> thanks mattoliverau
21:54:37 <Zyric_> onovy: Overwrites now too.
21:54:59 <Zyric_> I know this isn't high-priority as far as Swift is concerned, but it is important for my internship so all feedback is much appriciated.
21:55:00 <onovy> Zyric_: i will take a look
21:55:13 <notmyname> patch 212824 needs eyes first (the object one by torgomatic)
21:55:14 <patchbot> notmyname: https://review.openstack.org/#/c/212824/ - Let developers/operators add watchers to object audit
21:55:31 <eranrom> I lie the idea will have a look
21:55:34 <eranrom> like
21:55:35 <notmyname> then Zyric_'s patch can build on that. but it's in relatively good shape now and could use feedback too
21:55:40 <notmyname> eranrom: onovy: thanks
21:56:06 <notmyname> Zyric_: it's probably a good idea to get whatever updated code you have locally pushed asap to get eyes on that and not be distracted by stuff you've already fixed
21:56:15 <notmyname> Zyric_: will you have a chance to do that today?
21:56:23 <onovy> +1 on this. release early :)
21:56:24 <acoles> Zyric_: torgomatic notmyname fwiw I like the idea, will try to review
21:56:36 <notmyname> appreciate it
21:56:52 <notmyname> oh my, look at the time
21:57:05 <notmyname> we're not going to get to the full agenda today :-(
21:57:16 <notmyname> #topic other things (summary/wrap-up
21:57:30 <notmyname> patch 202411 needs reviews
21:57:30 <patchbot> notmyname: https://review.openstack.org/#/c/202411/ - Add functional test for access control (RBAC) with...
21:57:41 <notmyname> IMO, aside from release-blocking bugs, this one should be reviewed first
21:57:56 <kota_> \o/
21:58:04 <ho_away> yay!
21:58:20 <notmyname> even if you hate it technically, we definitely owe it to ho to do the reviews on it
21:58:36 <notmyname> (and you probably don't hate it anyway ;-)
21:58:42 <acoles> yup
21:58:46 <notmyname> point is, go review that one
21:59:16 <notmyname> ho_away: you had a few more things listed on the agenda related to
21:59:18 <notmyname> #link patch 212824 needs eyes
21:59:18 <patchbot> notmyname: https://review.openstack.org/#/c/212824/ - Let developers/operators add watchers to object audit
21:59:23 <notmyname> #undo
21:59:24 <openstack> Removing item from minutes: <ircmeeting.items.Link object at 0x9805350>
21:59:32 <notmyname> #link http://paste.openstack.org/show/484089/
21:59:35 <notmyname> that one
21:59:41 <ho_away> we don't have a time now so please check the url and the patches.
21:59:50 <notmyname> what to do with even-nbumber of replicas
22:00:03 <cschwede> yep, and especially patch 266199 from ho. pls see ho’s and mine discussion there
22:00:04 <patchbot> cschwede: https://review.openstack.org/#/c/266199/ - Fix deleting containers behavior when half of cont...
22:00:13 <notmyname> and cbartz, I'm sorry we don't have time to get to your tempurl with cotainer-level scope
22:00:31 <cbartz> ok, please just look over the review again
22:00:34 <ho_away> cschwede: thanks!
22:00:38 <cbartz> just wanted to retrieve some more feedback
22:00:41 <notmyname> I'll leave it on the agenda for next week, but let's not wait until then to think about it :-)
22:00:47 <notmyname> time's up
22:00:57 <notmyname> thanks for coming! great work on swift, everyone
22:00:59 <notmyname> #endmeeting