21:00:27 <notmyname> #startmeeting swift
21:00:28 <openstack> Meeting started Wed Jun 10 21:00:27 2015 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:31 <openstack> The meeting name has been set to 'swift'
21:00:34 <notmyname> who's here for the swift meeting?
21:00:37 <torgomatic> ✔️
21:00:41 <jrichli> hello
21:00:42 <mattoliverau> o/
21:00:43 <hurricanerix_> o/
21:00:43 <gvernik> hello
21:00:44 <ho> o/
21:00:46 <dmorita> hi
21:00:47 <tdasilva> hi
21:00:48 <minwoob_> o/
21:00:49 <kota_> o/
21:00:55 <cschwede> o/
21:01:05 <clayg> \o/
21:01:18 <cutforth> o/
21:01:23 <acoles> hi
21:01:26 <aerwin> o/
21:01:40 <notmyname> welcome!
21:01:42 <notmyname> so as I said^Wthreatened in vancouver, I've been spending some time on figuring out what's going on
21:01:54 <notmyname> which means we have a lot of stuff on the agenda today
21:01:59 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:13 <notmyname> so let's get started and see how far we can make it
21:02:27 <notmyname> #topic python 3
21:02:30 <torgomatic> I bet we get all the way to the end of the meeting by the time we're done
21:02:39 <notmyname> first up, python 3
21:02:52 <mattoliverau> torgomatic: wise words
21:02:53 <notmyname> this is an ongoing discussion in the larger openstack community
21:03:02 <notmyname> but for us specifically, what are we doing?
21:03:24 <notmyname> specifically, if we start now with supporting py3, we'll be adding a new dependency
21:03:28 <clayg> I see ss-python-six already in my builds so I'm down with adding the depends
21:03:35 <notmyname> #link https://review.openstack.org/#/c/189495/
21:03:37 <haypo> python3 for the win!
21:03:48 <notmyname> yeah, it's already a dependency for swiftclient
21:03:52 <notmyname> and other openstack things
21:03:58 <notmyname> but not yet for swift server-side
21:04:24 <haypo> notmyname, six is the defacto choice to add py3 support to a python project
21:04:27 <clayg> notmyname: what about python-future - it's like six - but with support upstream?
21:04:39 <notmyname> clayg: I don't know about it
21:04:43 <notmyname> tell me more :-)
21:04:44 <torgomatic> seems like we need it by 2016-04 since that's when the new Ubuntu LTS will show up with py3 by default
21:04:56 <clayg> notmyname: maybe i should just ask on the ML -> http://python-future.org/
21:05:10 <haypo> six is now used in all oslo projects, a lot of dependencies and more and more openstack applications
21:05:17 <notmyname> seems that six is how the rest of openstack is doing it
21:05:20 <clayg> better -> http://python-future.org/faq.html#what-is-the-relationship-between-future-and-six
21:05:51 <haypo> clayg, what do you mean by "support upstream" for python-future?
21:06:23 <notmyname> clayg: do you think we should hold off on six because of questions around python-future that should be answered first?
21:06:30 <haypo> #link https://wiki.openstack.org/wiki/Python3#Port_Python_2_code_to_Python_3
21:06:42 <haypo> yes, six is the choice to add python 3 support in openstack
21:07:11 <clayg> haypo: I need to dig up the link - there was a pep somewhere
21:07:49 <haypo> clayg, i'm a python core developer and i'm not aware of this pep :-/
21:08:02 <clayg> lol - perhaps I'm mistaken then!
21:08:26 <notmyname> haypo: hang on, I want to hear what clayg says about holding off on six if there are other questions he has :-)
21:08:41 <haypo> clayg, all PEPs are listed here: https://www.python.org/dev/peps/
21:09:35 <haypo> do you know applications using python-future? i'm not aware of any project using python-future :-/
21:10:37 <tdasilva> haypo: i don't thik clayg is saying we have to use python-future, he just raised a question of something that maybe we should look at
21:10:38 <zaitcev> let's move on, come on
21:10:52 <clayg> yeah don't wait on me
21:10:55 * notmyname is waiting for clayg
21:11:00 <clayg> notmyname: stop it :P
21:11:04 <notmyname> heh
21:11:12 <clayg> I'm fine with six - like I said I already had it packaged
21:11:19 <notmyname> clayg: do you think there are question about python-future that need to be answered ?
21:11:49 <notmyname> ie before using six?
21:11:53 <haypo> clayg, (python 3 has a builtin concurrent.futures modules, this module was backported as "futures" for python 2 on PyPI. the module has a PEP. is it the "futures" PEP?)
21:12:05 <clayg> awhile back portante said if we wait long enough on py3 the industry will standardize on the tools to run a 2/3 compatible code base and we just follow that instead of trying to cut the path - i wasn't sure if the winners have already been picked
21:12:24 <clayg> I do like futures.install_aliases() instead of form six import name
21:12:38 <notmyname> ok
21:13:08 <notmyname> but, as you said earlier, you're ok with six
21:13:22 <haypo> clayg, "the industry will standardize" you didn't answer. do you know if python-future is used? i'm not aware of any project using it. does openstack have this dependency in global requirements for example?
21:14:00 <haypo> i cannot find python-future on PyPI!?
21:14:03 <clayg> haypo: sorry man, but I think you mis-understood my question - i'm not advocating
21:14:09 <notmyname> haypo: hold on. nobody is attacking using six
21:14:32 <ho> btw which version of py3 should we support? latest?
21:14:45 <notmyname> the question is only about adding the dependency of six to swift today. is that something that should be done
21:15:00 <haypo> ho, in openstack, it was decided to focus on 3.4
21:15:09 <ho> haypo: thanks!
21:15:10 <haypo> ho, oslo already dropped 3.3 support
21:15:16 <clayg> haypo: https://pypi.python.org/pypi/future
21:15:36 <haypo> clayg, oh thanks :)
21:15:36 <notmyname> the fact that openstack projects are converging on six is strong support for six
21:15:37 <torgomatic> if we want to deal with python 3 now, then we should add six (or whatever) now... the more interesting question to me is whether we want to deal with it now or push it off until later
21:15:45 <notmyname> torgomatic: yes, that
21:15:48 <redbo> I don't know, every time someone says "X is the openstack choice", it makes me think we should do something completely different.
21:15:54 <notmyname> redbo: :-)
21:15:55 <clayg> redbo: lol
21:15:58 <haypo> so no, "future" is not in openstack global requirements
21:15:59 <acoles> :)
21:16:05 <notmyname> redbo: which is why I like talking about other stuff that's out there :-)
21:16:11 <torgomatic> redbo wins this meeting
21:16:15 <notmyname> lol
21:16:48 <notmyname> ok, the question is if anyone things we should NOT do six/py3 compat today (ie start the patches now instead of later)
21:17:00 <haypo> torgomatic, i started to send py3 patches to swift because almost all dependencies are now python3 compatible
21:17:10 <clayg> notmyname: so can you make a "chore" task to try and ask on the ML if anyone knows of any projects (not openstack) that are choosing python-future instead of six for the single code base that runs on python 2 or 3 strategy
21:17:14 <torgomatic> so, my opinion: I don't want to deal with python 3 right now. There's enough stuff to do with fixing all the damage EC did when it went in, among other things, that it's not a priority for me. HOWEVER
21:17:21 <clayg> notmyname: oh i mean - chore for me
21:17:29 <clayg> notmyname: well anyway - point is - I'll do the email
21:17:41 <haypo> only pyeclib doesn't support py3 yet, and i sent a patch for that
21:17:45 <torgomatic> if other people want to take on the effort, I won't object. I'm just not going to make py3 stuff a priority right now.
21:17:47 <haypo> https://bitbucket.org/kmgreen2/pyeclib/pull-request/24/fix-python-3-issues/diff
21:18:00 <clayg> notmyname: but back to torgomatic's question I'd be keen on "not standing in the way of someone who wants to work on swift python3"
21:18:21 <clayg> notmyname: as in I can run unitests against python3 just fine - so if someone makes that work i'll make sure I don't regress it
21:18:33 <notmyname> that is the impression I get from others: not a priority but would support someone else doing it
21:18:47 <notmyname> ho: redbo: acoles: do you agree?
21:18:54 <notmyname> tdasilva:
21:18:59 <notmyname> zaitcev:
21:19:08 <redbo> sure
21:19:14 <ho> notmyname: i agree
21:19:35 <haypo> clayg, my plan for swift is that same than other openstack projects (i'm porting nova, ceilometer, glance & swift to py3): add a py34 gate as soon as possible, and make it voting
21:19:38 <notmyname> (unfortunately, in swift, we are the people who get to actually support it. we can't just punt to someone else)
21:20:15 <clayg> notmyname: you know with dropping py2.6 I already sorta feel like we got a whole new python
21:20:20 <notmyname> :-)
21:20:41 <notmyname> ok, so here's a resolution:
21:20:47 <acoles> notmyname: agree but would like a gate job to avoid regressions
21:21:12 <notmyname> let's add the dependency for six today, and allow the py3 patches to continue. propose and review as other patches
21:21:16 <clayg> haypo: oh no wonder you're all up in my grill about something different - you'd be the one dealing with the different :P
21:21:45 <mattoliverau> haypo sounds like he's keen to make it work, and we already use six in swiftclient so lets add it as a dependancy
21:22:03 <notmyname> anyone opposed to starting the py3 patches in swift? ie saying that we will review and maintain py3 code in swift?
21:22:36 <cschwede> some time in the future we have to do it, and if there are volunteers right now that’s fine IMO
21:22:41 <tdasilva> sounds good to me. should the py34 gate be done before any patches go in?
21:22:45 <haypo> notmyname, the more important side effect is that new code will have to be compatible with py3, developers will have to be aware of python3
21:22:56 <notmyname> ho: which brings up the testing issue
21:22:59 <notmyname> haypo: ^
21:23:20 <haypo> notmyname, which issue?
21:23:25 <notmyname> haypo: can you propose a -infra test and the necessary swift tox.ini changes to add a py34 test?
21:23:30 <cschwede> we could add a py3 tox env?
21:23:33 <notmyname> haypo: the fact that we need it :-)
21:23:41 <haypo> notmyname, sure, that's part of my plan ;)
21:23:42 <notmyname> cschwede: yeah, that's what I'm asking haypo to do
21:24:00 <notmyname> haypo: great. then I'd like to see that ASAP. sooner the better
21:24:03 <notmyname> ok
21:24:04 <cschwede> yes, posted at the same time :)
21:24:12 <haypo> notmyname, today, it doesn't make sense because there are too many failures. we need most of the py3 patches that i submited to run a subset of tests on py 3.5
21:24:15 <haypo> py 3.4*
21:24:25 <haypo> notmyname, i agree (ASAP)
21:24:31 <notmyname> #agreed allow for the new six dependency and py3 changes in swift
21:24:44 <notmyname> haypo: I want to see it added as non-voting asap, even if it fails today
21:25:08 <notmyname> #action haypo to add a py3 check job (non-voting) to swift
21:25:09 <haypo> notmyname, ok, i can do that
21:25:12 <notmyname> haypo: thanks
21:25:20 <notmyname> ok, moving on :-)
21:25:25 <haypo> coolness
21:25:29 <acoles> notmyname: do we have a "grace" period while the py34 tests is non-voting - or do any pending reviews that fail have to be fixed by their authors?
21:25:33 <notmyname> haypo: thank you for your work on this
21:25:35 <zaitcev> anything that helps knowing if we re-introduce 2.x code
21:25:49 <haypo> notmyname, you're welcome. you can help by reviewing patches :-D
21:25:52 <notmyname> acoles: I think that's ok. I'd say use your discression as a reviewer
21:26:10 <acoles> notmyname: ok but that implies the gate job is non-voting
21:26:22 <notmyname> acoles: it will have to be initially
21:26:26 <acoles> cool
21:26:30 <notmyname> #topic priority reviews
21:26:36 <notmyname> ok, first general question
21:26:53 <notmyname> at the summit we got a great list of stuff to improve from ops
21:27:07 <notmyname> however, I'm not sure really how to highlight those feature requests
21:27:17 <tdasilva> is there an etherpad?
21:27:28 <notmyname> ...somewhere... ;-)
21:27:45 <notmyname> tdasilva: summary list on https://wiki.openstack.org/wiki/Swift/PriorityReviews
21:28:02 <tdasilva> perfect, thanks
21:28:04 <notmyname> question is, do we keep trackign them there on a wiki page, or do we need something else?
21:28:14 <mattoliverau> #link https://etherpad.openstack.org/p/liberty-swift-ops-feedback
21:28:19 <notmyname> these don't have patches yet, so gerrit doesn't help
21:28:21 <notmyname> mattoliverau: thanks
21:28:31 <tdasilva> mattoliverau: thanks
21:28:33 <notmyname> here's something I'm thinking, and I'd love your feedback
21:28:59 <notmyname> start using the priority reviews wiki page as a place to track what's going on and stuff that should be prioritized for new stuff (like ops requests)
21:29:17 <notmyname> and use only the gerrit dashboards to prioritize patches to review
21:29:32 <notmyname> (the change is that we used to list patches to review on the priority reviews wiki page)
21:29:35 <notmyname> what do you think?
21:29:45 <mattoliverau> +1
21:30:01 <mattoliverau> that sounds sensible, and there's no point listing patches in review twice
21:30:11 <tdasilva> notmyname: how about using launchpad bugs with Importance as wishlist?
21:30:19 <mattoliverau> The name prioroty reviews might need to change tho
21:30:52 <notmyname> tdasilva: I think that's a good idea, but LP bugs are rather crude for tracking stuff. IMO they're only marginally good for tracking bugs. and pretty terrible for "todo tasks"
21:31:07 <notmyname> (I'll get to bugs later in the meeting)
21:32:10 <notmyname> eg one reason LP bugs are bad for todo tasks is becausee the status are new, confirmed, fix released, invalid, etc. not stuff that maps well to new features
21:32:29 <acoles> notmyname: imho its great to have the wiki as a place that summarise whats going on, it was good having the reviews listed there but we can experiment with managing that through gerrit and see how it works
21:32:30 <tdasilva> notmyname: it's just that you can add comments to them, start building a history on the discussion and what not
21:33:06 <notmyname> tdasilva: hmm..I'll have to think more on that. it's a good point
21:33:09 <kota_> acoles: +1
21:33:40 <ho> acoles: +1
21:33:51 <tdasilva> notmyname: you started doing such a great job with LP bugs grooming! :D
21:33:58 <notmyname> ok, I'll keep the wiki as an overview of stuff going on and the dashboard as a way to track priority reviews
21:34:05 <notmyname> tdasilva: "started". lots more to do ;-)
21:34:14 <notmyname> ok, now for some specific stuff
21:34:15 <cschwede> what about „empty“ commits in gerrit? and the one who starts working on it does a „git commit —amend —reset-author“.
21:34:34 <notmyname> cschwede: that's interesting
21:35:04 <notmyname> mattoliverau: what's the status of container sharding? can you give us an update?
21:35:13 <mattoliverau> sure
21:35:41 <mattoliverau> Firstly, blmartin, a coworker of jrichli has started helping, he’s helping out with unit tests until he feels more confident to tackle something more meaty
21:36:00 <notmyname> great
21:36:11 <mattoliverau> The root container now contains copies of all distributed nodes, so can “short circuit” (as discussed at summit).
21:36:21 <mattoliverau> clayg: ^^
21:36:22 <acoles> nice
21:36:30 <notmyname> mattoliverau: have you been able to get some large containers to test and experiment on?
21:36:50 <mattoliverau> only ones I have made myself using word lists
21:36:52 <notmyname> I was hoping hurricanerix_ or redbo could help with that ;-)
21:37:01 <mattoliverau> :)
21:37:07 <mattoliverau> I have a new algorithm for finding the best candidate sub tries for sharding, the new method in testing has been quicker and far far more memory efficient. It uses what I call a counting trie, that has counters at each intermediate node which increment when adding an object.
21:37:14 <notmyname> (since they would be able to do that in the same company with less privacy issues)
21:37:24 <mattoliverau> good point :)
21:37:36 <mattoliverau> As soon as a node counter has hit the magic max_group count a method is fired, but the trie will only store the deepest candidate node.
21:37:57 <mattoliverau> To keep it memory efficient, as soon as you create a new child at any node, all other children in that node are recursively deleted. Meaning it deletes nodes as quickly as they are added. This means objects need to be added in order, but we do that cause database.
21:38:12 <notmyname> redbo: hurricanerix_: could you work with mattoliverau to get some large container DBs to help with the container sharding effort?
21:38:20 <mattoliverau> These both have been added and in testing are working well.
21:38:23 <hurricanerix_> notmyname: i can take that on
21:38:26 <notmyname> hurricanerix_: thanks
21:38:40 <mattoliverau> I’m working on benchmarks ATM, and was hoping to have something today, but I don’t. I need to benchmark different max_group_counts to find out whats best, that may take a while seeing as I will be dealing with large containers each time.
21:38:55 <notmyname> #action hurricanerix_ to get large container dbs for mattoliverau to test container sharding on
21:39:01 <mattoliverau> Further, shrinking will be hard.. but I have some ideas and will get to that post initial benchmarking.
21:39:16 <mattoliverau> </brain dump>
21:39:17 <notmyname> ok
21:39:19 <mattoliverau> I will be writing this up as a new patch to the spec.
21:39:20 <notmyname> thanks for the update
21:39:25 <notmyname> ah, perfect
21:39:48 <notmyname> acoles: jrichli: could you give an update on encryption work?
21:39:58 <jrichli> sure, i could do that
21:40:10 <acoles> jrichli: go for it
21:40:19 <jrichli> The updates to the spec landed!
21:40:26 <acoles> thanks mattoliverau torgomatic
21:40:31 <kota_> nice
21:40:41 <jrichli> I have updated the trello recently with status of issues: https://trello.com/b/63l5zQhq/swift-encryption
21:40:55 <jrichli> Also, I try to keep the list of "TODO's" up-to-date on the patch 157907
21:40:56 <patchbot> jrichli: https://review.openstack.org/#/c/157907/
21:41:14 <jrichli> I am currently working on the functest errors and adding more unit tests
21:41:35 <notmyname> jrichli: what do you need right now from the rest of us?
21:42:01 <jrichli> Please start to supply feedback on the review.  I know there is a lot, and its a WIP,
21:42:08 <jrichli> but some feedback to start on would be good
21:42:14 <notmyname> ok
21:42:34 <notmyname> I've starred it
21:42:38 <notmyname> thanks for the update
21:42:40 <jrichli> cool, thx
21:42:48 <notmyname> swifterdarrell isn't here to talk about his patch
21:42:54 <notmyname> #link https://review.openstack.org/#/c/184189/
21:43:03 <notmyname> "Allow 1+ object-servers-per-disk deployment"
21:43:24 <mattoliverau> That's a shame cause it's an exciting one :)
21:43:36 <zaitcev> I think I manhandled him into relocating the most interesting blob from bin/
21:43:39 <notmyname> summary is that it allows for a lot of isolation of the disk IO and shows some very impressive improvements in performance when there is a drive issue
21:43:46 <notmyname> zaitcev: good!
21:44:14 <zaitcev> so we could review the patch and imagine it's already back in swift/
21:44:41 <notmyname> it's definitely a priority review. the analysis of performance linked in the commit message are quite extensive
21:44:50 <torgomatic> yeah, that one's pretty exciting
21:44:54 <notmyname> please go review that patch
21:45:01 <torgomatic> it's all the IO isolation of the threadpools without all the slowdown of the threadpools
21:45:16 <zaitcev> I'm thinking he may need some help imagining unit tests for that thing. Like: we fork this and this, and this process dies, then what happens?
21:45:32 <notmyname> hurricanerix_: redbo: I'd especially love to see anything you can do on it. if you can put it in your lab that would be great
21:45:32 <zaitcev> He already does probe tests which is pretty impressive.
21:45:59 <notmyname> hurricanerix_: redbo: you've got some of the densest object servers I know, and you've talked about stuggling with this issue
21:46:39 <notmyname> (issue == inconsistent latency on responses)
21:47:24 <notmyname> hurricanerix_: redbo: also, I'd like to ask you to work on similar sorts of comparative testing for python/golang. like what's in darrell's patch
21:47:25 <hurricanerix_> notmyname: redbo had to step out, but I will pass it on to the team to see what we can do.
21:47:31 <notmyname> hurricanerix_: great
21:48:06 <notmyname> hurricanerix_: basically, the sorts of comparisons that are in that patch i want to see for golang vs python. I think that would help make a very data-driven discussion
21:48:22 <notmyname> whew, look at the time...
21:48:38 <notmyname> mattoliverau: you mentioned on the agenda about landing specs
21:48:43 <mattoliverau> yeah
21:48:55 <notmyname> #link https://review.openstack.org/#/q/status:open+project:openstack/swift-specs,n,z
21:49:01 <notmyname> we do have a lot of not-landed specs
21:49:17 <mattoliverau> I wanted to see if we could just land specs that we all verbally agreed upon on at summit
21:49:21 <notmyname> mattoliverau: which ones in particular did you want to call attention to?
21:50:03 <mattoliverau> teiring, metadata, event notification, large containers.
21:50:17 <mattoliverau> stuff we have agreeed we are working on
21:50:33 <mattoliverau> we should get them so they appear on specs.o.o
21:51:20 <dmorita> I think changing policies is also agreed to move forward :)
21:51:29 <mattoliverau> yeah, that too
21:51:51 <notmyname> ok, thanks
21:52:13 <mattoliverau> I'm happy to go +A them to get them in, but wanted to ask first :)
21:52:22 <notmyname> I have starred those. I'll call people out over the next week to look at them if nothing isn't happening
21:52:31 <mattoliverau> thanks
21:52:42 <notmyname> #topic bugs
21:52:52 <notmyname> ok, I want to be better about LP bugs
21:53:01 <notmyname> as youve seen from your email, I've gone through some
21:53:16 <notmyname> my plan is to first get everything out of the "new" state
21:53:40 <notmyname> so if there is a bug in "new" that you see, feel free to comment, etc.
21:53:53 <notmyname> after I get through all the "new" bugs, I'll start with some sort of prioritization
21:54:17 <notmyname> a _lot_ of the bugs are very old and need to be reverified or just see what's going on there
21:54:29 <notmyname> so for a few of them to mention this week
21:54:48 <notmyname> #link https://bugs.launchpad.net/swift/+bug/1263776
21:54:49 <openstack> Launchpad bug 1263776 in OpenStack Object Storage (swift) "timestamp comparisons" [Medium,Incomplete]
21:54:53 <notmyname> needs to be reverified
21:55:01 <notmyname> as dones
21:55:03 <notmyname> #link https://bugs.launchpad.net/swift/+bug/1019712
21:55:03 <openstack> Launchpad bug 1019712 in OpenStack Object Storage (swift) "Database replicator will rsync too often" [Low,Incomplete]
21:55:05 <notmyname> *does
21:55:16 <notmyname> can I get a volunteer to look at each of these?
21:55:40 <cschwede> i take the first one
21:55:45 <notmyname> cschwede: thanks
21:56:26 <notmyname> cschwede: I assigned it to you. after you've verified one way or another, feel free to removed yourself
21:56:35 <notmyname> anyone for the second one?
21:56:44 <notmyname> taking a look at DB replication
21:56:45 <notmyname> ?
21:57:12 <mattoliverau> I'll take it
21:57:16 <notmyname> mattoliverau: thanks
21:57:35 <notmyname> mattoliverau: same as cschwede above
21:58:04 <notmyname> eranrom: we've kinda run out of time. can we talk container sync next week?
21:58:35 <eranrom> sure, I wonder if I should just open a bug and dump my proposed solution there
21:58:41 <notmyname> eranrom: that would be great
21:58:47 <eranrom> will do
21:59:16 <notmyname> thanks everyone for coming this week. I want to try to do more like this in the next few meetings: get some status updates and highlight some patches and bugs
21:59:29 <notmyname> please update https://wiki.openstack.org/wiki/Meetings/Swift as you have stuff for the meeting
21:59:40 <notmyname> thanks for coming! and thanks for working on swift!
21:59:44 <notmyname> #endmeeting