21:00:13 <notmyname> #startmeeting swift
21:00:13 <openstack> Meeting started Wed Dec 20 21:00:13 2017 UTC and is due to finish in 60 minutes.  The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:15 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:17 <openstack> The meeting name has been set to 'swift'
21:00:20 <notmyname> who's here for the swift team meeting?
21:00:36 <m_kazuhiro> o/
21:00:50 <clayg> meeting meeting meeting
21:00:54 <tdasilva> hi
21:00:57 <clayg> tdasilva: !!
21:01:06 <tdasilva> clayg!
21:01:20 <kota_> hello
21:01:20 <timburke> o/
21:01:41 <acoles> hello
21:02:24 <rledisez> hi o/
21:02:45 <torgomatic> hi
21:02:49 <joeljwright> hello :)
21:03:04 <clayg> rledisez: acoles: torgomatic: joeljwright: !!! everyone is here !!!
21:03:08 * clayg steps out for a bit
21:03:08 <notmyname> welcome, everyone
21:03:11 <notmyname> lol
21:03:19 <notmyname> agenda at
21:03:22 <notmyname> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:03:41 <notmyname> a few things to review and a new topic from m_kazuhiro
21:03:58 <notmyname> first up, some FYI stuff...
21:04:01 <notmyname> #topic general stuff
21:04:23 <notmyname> patch 528582 is making its way through the gate
21:04:24 <patchbot> https://review.openstack.org/#/c/528582/ - swift - Native Zuul v3 tox jobs
21:04:29 <notmyname> and in fact it changes the gate
21:04:42 <notmyname> it's the in-repo definition of check and gate jobs for swift
21:04:50 <notmyname> using the new zuul v3 stuff
21:05:35 <notmyname> it's a great improvement because it allows us to define test jobs in our own repo (AIUI, even with ansible scripts to set up an environment, if needed)
21:06:02 <notmyname> this first patch is the last step int he old centralized job definition to the new zuul v3 way
21:06:11 <clayg> tdasilva: ^^^ pretty cool huh!?
21:06:24 <clayg> all hail zuul v3
21:06:25 <tdasilva> very! was thinking we could add libec, probe tests
21:06:30 <notmyname> yup
21:06:41 <clayg> ~ a whole new world ~
21:06:43 <kota_> +1
21:06:43 <joeljwright> :)
21:07:17 <notmyname> speaking of probe tests, we all probably noticed by now that the swiftstack community qa cluster has been throwing errors. I've finally burned through enough on my todo list to get to that item
21:07:45 <clayg> burn it down!
21:07:47 <notmyname> (1) I hope to fix it, of find someone who can (2) some of it may not be needed any more with the zull v3 definitions
21:08:08 <notmyname> any other general stuff to bring up?
21:08:11 <clayg> oh... i mean fix it... but you COULD burn it down - that would *also* be fun...
21:08:20 <notmyname> I thought I had something else, but of course I didn't write it down...
21:08:22 <timburke> i was actually thinking of that wrt to the func tests...
21:08:35 <kota_> i think, libec and pyeclib are also in progress for zuul v3
21:08:35 <clayg> timburke: yeah burn 'em!
21:08:42 <notmyname> I heard something about some conference talks being accepted, acoles cschwede, tdasilva
21:08:42 <timburke> we should totally get some cross-policy testing happening in the gate though!
21:08:47 <acoles> notmyname:  IIRC the community cluster functional test was setup to cover an EC policy which we now have an in process test for
21:08:55 <notmyname> acoles: ya
21:09:06 <clayg> cschwede: !!!
21:09:06 <kota_> e.g. https://review.openstack.org/#/c/528546
21:09:07 <patchbot> patch 528546 - pyeclib - Convert tox job to native v3 Zuul
21:09:53 <tdasilva> notmyname: https://fosdem.org/2018/schedule/track/software_defined_storage/
21:10:18 <tdasilva> there are two swift talks missing from there right now, but i expect they will be there soon
21:10:26 <notmyname> nice
21:10:44 <acoles> tdasilva: and cschwede assure me they will be there too, otherwise I'll have to drink belgian beer on my own :/
21:10:55 <tdasilva> and give 3 talks
21:11:28 <timburke> kota_: oh man! that makes me so happy that i made a liberasurecode-git tox env for pyeclib...
21:11:43 <acoles> tdasilva: t's swift, same talk 3 times over :)
21:11:56 <notmyname> acoles: tdasilva: cschwede: that's great :-)
21:12:24 <notmyname> ok, let's move on to the agenda topics
21:12:28 <notmyname> #topic symlink status
21:12:35 <notmyname> patch 232162
21:12:36 <patchbot> https://review.openstack.org/#/c/232162/ - swift - Symlink implementation. (MERGED)
21:12:37 <timburke> whoooo!!!
21:12:39 <notmyname> it's merged!
21:12:42 <kota_> yey
21:12:52 <notmyname> (what follow-ons are still open?)
21:13:06 <joeljwright> awesome!
21:13:14 <m_kazuhiro> great!
21:13:33 <kota_> https://review.openstack.org/#/c/527583/
21:13:33 <patchbot> patch 527583 - swift - functest for symlink + versioned writes
21:14:02 <kota_> it looks like that it got a merge conflict
21:14:03 <acoles> notmyname: https://review.openstack.org/#/c/527583 - clayg or i need to rebase it
21:14:04 <patchbot> patch 527583 - swift - functest for symlink + versioned writes
21:14:29 <acoles> the conflict is all my fault, think I fixed a teardown in two patches
21:14:34 <notmyname> other than the rebase, how's it look?
21:15:11 <clayg> booo `liberasurecode-git tox env`
21:15:30 <clayg> er... booo `The change could not be rebased due to a conflict during merge` even
21:15:35 <clayg> copy/paste error
21:15:46 <clayg> merge all the things!
21:16:04 <notmyname> anything else needed for symlinks follow-on?
21:16:13 <acoles> clayg: if you dont get there before me i will fix the conflict tomorrow
21:16:15 <notmyname> clayg: didn't you have a pile of them? did they all get squashed or landed?
21:16:42 <clayg> what symlink follow-on?  idk... i'd have to work on a list
21:16:43 <kota_> there ware 5 follow up patches but 4 of them landed already AFAIK
21:16:44 <clayg> a bunch got landed
21:16:56 <clayg> notmyname: see kota_  is on it - you don't have to worry - i'm not worried
21:16:57 <notmyname> ok
21:17:06 <clayg> 😎
21:17:22 <notmyname> sounds good!
21:17:28 <notmyname> #topic SLO data segments
21:17:35 <notmyname> patch 365371
21:17:35 <patchbot> https://review.openstack.org/#/c/365371/ - swift - Add support for data segments to SLO and Segmented...
21:17:38 <timburke> oh man, so many. it was great. p 527595 p 528073 p 528072 p 527477 p 527475
21:17:39 <patchbot> https://review.openstack.org/#/c/527595/ - swift - Add more assertions for Symlink + Copy unit tests (MERGED)
21:17:40 <patchbot> https://review.openstack.org/#/c/528073/ - swift - add symlink to probetest for reconciler (MERGED)
21:17:42 <patchbot> https://review.openstack.org/#/c/528072/ - swift - add symlink to container sync default and sample c... (MERGED)
21:17:43 <notmyname> timburke: looks like you just pushed a patch set
21:17:44 <patchbot> https://review.openstack.org/#/c/527477/ - swift - Assert X-Newest and X-Backend headers are propagat... (MERGED)
21:17:46 <patchbot> https://review.openstack.org/#/c/527475/ - swift - Symlink doc clean up (MERGED)
21:18:04 <patchbot> timburke: don't abuse the patchbot
21:18:13 <joeljwright> :D
21:18:23 <kota_> oh, 6 total and 5 got mreged.
21:18:28 <kota_> *merged
21:18:33 <notmyname> nice
21:18:34 <timburke> notmyname: yup. had to fix it up after the name change in 528081
21:18:43 <timburke> p 528081
21:18:43 <patchbot> https://review.openstack.org/#/c/528081/ - swift - rename utils function less like stdlib (MERGED)
21:18:54 <notmyname> timburke: thanks
21:19:14 <notmyname> joeljwright: wanna give it a quick look to see if it still resembles what you remember? :-)
21:19:27 <joeljwright> I've been keeping an eye on it
21:19:27 <timburke> gate looks happy -- just waiting on the extra long test
21:19:42 <joeljwright> but I'll give it another look for patchset 40
21:19:52 <notmyname> timburke: joeljwright: did either of you test the nested SLOs with a recent patch set? when I looked, it looked good, but that was an outstanding question
21:19:59 <joeljwright> it's evolved into something really cool thanks to timburke
21:20:14 <notmyname> yeah, it's a pretty cool feature
21:20:24 <notmyname> I suspect it will become a commonly-used feature
21:20:53 <timburke> notmyname: yeah, there's a func test for that now. look for 'nested-data-manifest'
21:20:59 <notmyname> nice!
21:21:27 <notmyname> sounds good
21:21:28 <notmyname> #topic s3api work status
21:21:29 <joeljwright> I haven't tested since patch 38 or 39, happy to take another look tomorrow
21:21:29 <patchbot> https://review.openstack.org/#/c/38/ - openstack-infra/system-config - Added entry for building manuals. (MERGED)
21:21:35 <notmyname> kota_: tdasilva: what's going on here?
21:21:36 <timburke> ttfb on SLOs got you down? inline the first meg! ;-)
21:21:36 <joeljwright> sorry
21:21:53 <notmyname> timburke: first 7k ;-)
21:22:10 <tdasilva> TBH i haven't had a chance to take a look at s3api for a while, definetely need to back to it :(
21:22:12 <notmyname> I havne't looked at the s3api feature branch in a bit
21:22:25 <notmyname> same for you timburke?
21:22:28 <tdasilva> last i was working on merging tests
21:22:48 <notmyname> kota_: have you spent any time on it lately?
21:22:51 <kota_> oh, sorry I was left for a bit
21:22:54 <kota_> yup
21:22:59 <tdasilva> merge as in merging trees, not landing new tests
21:23:10 <notmyname> got it
21:23:12 <kota_> I was starting to work on it yesterday
21:23:16 <notmyname> nice!
21:23:20 <kota_> wait a bit
21:23:27 <kota_> paste the trillo link...
21:23:44 <kota_> https://trello.com/b/ZloaZ23t/s3api
21:24:07 <kota_> i took 2 of tasks from backlog
21:24:31 <kota_> https://review.openstack.org/#/c/529238/
21:24:31 <patchbot> patch 529238 - swift (feature/s3api) - Add s3api section into docs
21:24:32 <kota_> docs
21:24:46 <kota_> https://review.openstack.org/#/c/529268/
21:24:46 <patchbot> patch 529268 - swift (feature/s3api) - Avoid global LOGGER instance
21:24:47 <kota_> logger
21:25:05 <kota_> and some minor things
21:25:06 <tdasilva> nice!
21:25:17 <notmyname> that's great. thanks for keeping work going on it
21:25:33 <clayg> kota_: is on a roll1
21:25:38 <clayg> roll!
21:25:52 <kota_> hopefully, i expect timburke helps to land. in particular LOGGER one starts to touch a bunch of places
21:26:37 <kota_> tdasilva: if you have a patch for functests even if its intermediate, i may help you to cleanup.
21:26:59 <tdasilva> kota_: that sounds great, i'll try to propose as a WIP
21:27:09 <timburke> yeah, i'll be sure to help review when i can
21:27:11 <notmyname> yay team swift :-)
21:27:29 <kota_> that's all from me for s3api
21:27:29 <notmyname> #topic general task queue
21:27:39 <notmyname> m_kazuhiro: this is what you've been working on
21:27:51 <notmyname> #link https://wiki.openstack.org/wiki/Swift/ideas/task-execution
21:27:58 <notmyname> #link  https://docs.google.com/document/d/11sBbB6pBvLYNeM9wjTdvvsJIu8Dl8i13UH2NfRVNOqg/edit#heading=h.u5kq7utbivxa
21:27:59 <clayg> general task queue!!!
21:28:04 <notmyname> https://review.openstack.org/#/c/517389
21:28:04 <patchbot> patch 517389 - swift - WIP: Update object expirer to use general task que...
21:28:24 <clayg> whoa
21:28:33 <m_kazuhiro> This is updating work for expirer's task queue and this is next work for auto-tiering.
21:28:35 <notmyname> m_kazuhiro: and you've been working with rledisez and mattoliverau(?) on it?
21:28:42 <m_kazuhiro> notmyname: yes.
21:28:46 <notmyname> great
21:29:00 <notmyname> how's work going?
21:29:31 <clayg> rledisez: is all about it
21:29:57 <rledisez> totally :)
21:29:59 <m_kazuhiro> I implemented main code for that. But I want to discuss some topics about it.
21:30:09 <notmyname> ok
21:30:15 <clayg> m_kazuhiro: do you have your tickets booked for PTG?
21:30:45 <m_kazuhiro> clayg: I'm going to get tickets now.
21:30:47 * clayg is just asking - he hasn't booked his travel either - but I think other people have started doing so
21:31:01 <clayg> whoot!
21:31:28 <tdasilva> m_kazuhiro: great good doc
21:31:39 <tdasilva> google
21:31:59 <m_kazuhiro> the great doc is made by mattoliverau !!
21:32:16 * tdasilva high fives mattoliverau
21:32:55 <rledisez> +1 very good summary of discussion we had in sidney
21:33:08 <notmyname> is there a particular question that we need to discuss in this meeting?
21:33:32 <notmyname> looks like it will be a very good topic in dublin
21:34:06 <tdasilva> Romain Expirer Algorithm
21:34:16 <tdasilva> yay, swift finally has named an algorithm
21:34:17 <clayg> neat idea to use partitions to break up work...
21:34:34 <notmyname> heh
21:35:33 <m_kazuhiro> Today's main discussion topic is in end of the google doc.
21:35:55 <m_kazuhiro> "Known problems"
21:36:46 <clayg> m_kazuhiro: can you restate that problem - i don't understand it in that context... I read something about a migration?  maybe taking the old containers and backfilling into the new per-partition containers?
21:37:14 <rledisez> basically, if every objects server fetch a different account for object-expiring tasks, what about the legacy tasks. all objects servers will fetch the same account. if you have a lot of object server, it's a DDoS of the account server
21:37:35 <rledisez> migration could be difficult path if you have hundreds of millions of tasks
21:38:13 <clayg> rledisez: i agree - i'd rather now have to translate the rows
21:38:21 <torgomatic> shard the migration work with the new queue system
21:38:27 <m_kazuhiro> rledisez: Thanks
21:38:29 <torgomatic> (half serious :) )
21:39:11 <clayg> rledisez: how many object-expirier's do you currently run?  You're worried about a bunch of account GETs ... i... I'm not sure I'm scared... once it's in the page cache... I'm not sure
21:39:25 <clayg> like... what... 10K requests?  meh..
21:39:43 <timburke> could we have some config option that says "continue pulling work from the old location, but write new work down in two places" then later flip it to "pull work from new place (and only write down new work in one place)"?
21:39:44 <rledisez> 10K requests, every minutes ?
21:39:56 <rledisez> that's a lot, I already killed a cluster that way
21:40:47 <tdasilva> timburke: i was about to propose something like object-expirer2
21:40:57 <rledisez> i was thinking of something based on the device id and current day. a process would fetch task if day % device_id == 0 not that great, but it would limit the concurrency on legacy account
21:42:02 <rledisez> well, it's actually a very bad ideai, forget it :)
21:43:08 <m_kazuhiro> I got an idea about that from kota_... and I think it is great. The idea is that expirer has "check_legacy_queue" boolean configure. We can controll expirer count which checks legacy queue with the value.
21:44:16 <notmyname> m_kazuhiro: does that answer help you move forward for now?
21:44:38 <rledisez> i'd rather have something the operator does not need to think about. if the operator forget that new param, the feature is broken. doesn't feel nice. but as we require that the operator run object-expirer on object-server, it might be OK I guess
21:46:21 <m_kazuhiro> notmyname: Yes. I can move forward with the answer. But I want to get opinions about the idea.
21:46:36 <clayg> rledisez: so you HAVE deployed enough object-expirers that the container server's holding the .expiring_objects account were not happy with the current amount of listings?  I don't think i've ever had that problem.
21:47:09 <clayg> er... account servers or account/container servers
21:47:33 <rledisez> clayg: it was not object-expirer, but an other internal process working pretty much the same way AFAIR
21:47:38 <clayg> I've seen specific expiring_object/<timeslice> *containers* get busy (but that's because they're eating row updates)
21:48:03 <clayg> and that part at least has been better since the RAX guys did the thing to make 100 containers for every timeslice or however dfg wrote it
21:48:09 <rledisez> maybe i should do an other test on our biggest cluster to see how it behave now
21:48:37 <timburke> i feel like i must be missing something... probably need to read about it more. but the migration path for inherently ephemeral data seems like it shouldn't be a stumbling block. is it that we're changing the cardinality of workers?
21:49:46 <clayg> does the new design have a bunch of accounts?  I thought it was account-per-task so I don't quite follow how the load at the account layer is reduced ... and I'm not sure how many GETs on account would need to happen before I'd be concerned...
21:50:13 <rledisez> clayg: account per $task-$ring-$partition_number
21:50:22 <clayg> I mean... i think we can write these processes to be pretty nice to swift - and we'll only have as many clients as nodes.. and there's probably more external clients than internal nodes...
21:50:35 <clayg> ORLY!?  ok then...
21:51:47 <clayg> ok, yeah I minunderstood the diagram - the indentation is account -> container -> object - gotcha
21:51:54 <clayg> ok cool!
21:51:57 <torgomatic> as long as we don't wind up with some Kafka-esque rules for how our queue system works, things should be fine
21:52:43 <notmyname> m_kazuhiro: thanks for bringing this topic up. it's a good question
21:53:17 <notmyname> to everyone, where's the best place to keep discussing it? in -swift? or something else like the google doc?
21:53:24 <notmyname> m_kazuhiro: what woudl you prefer?
21:54:56 <clayg> I think we should ignore it
21:54:58 <m_kazuhiro> I think remainning topic can be discussed on IRC (not this meeting). So we can finish acout task queue for now.
21:55:00 <clayg> this problem at least
21:55:13 <notmyname> ok
21:55:26 <clayg> We could come up with all kinds of arbitrary ways to limit the number of deployed nodes which handle the legacy data
21:55:32 <notmyname> that's brought us pretty close to full time
21:55:37 <notmyname> #topic open discussion
21:55:39 <clayg> "also_do_legcy_work = false"
21:55:47 <notmyname> I remembered the other thing I wanted to bring up at the beginning
21:55:49 <notmyname> next meeting time
21:55:55 <kota_> yup
21:56:01 <clayg> "legacy_processes = 4" "legacy_process = 1"
21:56:14 <notmyname> I suggest we skip the next 2 weeks
21:56:21 <notmyname> and next meeting is january 10
21:56:37 <clayg> notmyname: can I still get into openstack-swift and tell everyone what santa brings me?
21:56:41 <kota_> sounds good
21:56:42 <clayg> on the 27th
21:56:46 <notmyname> any objections? (objecting is the same as volunteering to run a meeting)
21:56:53 <clayg> no objections
21:56:56 <notmyname> :-)
21:56:56 <acoles> notmyname: no objections
21:57:03 * joeljwright stays silent
21:57:04 <acoles> ugh, I lost
21:57:09 <rledisez> ok for me, i'll be off starting friday
21:57:10 <notmyname> ok, next meeting is january 10
21:57:17 <clayg> rledisez: have a good holiday!
21:57:28 <rledisez> thx clayg :)
21:57:35 <kota_> rledisez: enjoy!
21:57:36 <notmyname> if you are taking a holiday, I hope you have a good time with friends and family
21:57:40 <acoles> rledisez: bon vacances
21:58:01 <rledisez> thx all, i'm sure i'll enjoy it!
21:58:15 <notmyname> thanks, everyone, for all your work on swift :-)
21:58:24 <notmyname> thanks for a good swift-in-2017
21:58:30 <notmyname> #endmeeting