Friday, 2014-12-19

*** annegent_ has joined #openstack-swift00:04
*** Masahiro has joined #openstack-swift00:25
*** Masahiro has quit IRC00:29
openstackgerritSamuel Merritt proposed openstack/swift: Add notion of overload to swift-ring-builder  https://review.openstack.org/14145200:32
openstackgerritSamuel Merritt proposed openstack/swift: Tiny speedup to ring builder  https://review.openstack.org/14294500:32
*** annegent_ has quit IRC00:33
*** dmorita has joined #openstack-swift00:39
notmynamecschwede: zaitcev: I left comments on https://review.openstack.org/#/c/99824/. I'd appreciate confirmation or correction on my understanding00:55
notmynamedmorita: oh! you might help there too! (and let kota know. don't see him online right now) ^00:55
zaitcevI'm looking00:58
*** addnull has joined #openstack-swift00:59
*** Masahiro has joined #openstack-swift01:01
notmynamethanks01:03
notmynameI'm not going to be able to look any more today. I'll check later though01:04
*** nellysmitt has joined #openstack-swift01:11
*** annegent_ has joined #openstack-swift01:12
*** nellysmitt has quit IRC01:16
*** oomichi has joined #openstack-swift01:39
*** masonhsiung has joined #openstack-swift01:42
mattoliverauBack in Melbourne (arrived home from Christmas party elsewhere in Oz). Now you all have to put up with me again :p01:42
*** annegent_ has quit IRC01:44
openstackgerritSamuel Merritt proposed openstack/swift: Add notion of overload to swift-ring-builder  https://review.openstack.org/14145201:44
*** oomichi has quit IRC01:55
*** haomaiwang has joined #openstack-swift01:57
*** Nadeem has quit IRC01:58
*** gyee has quit IRC02:03
*** Shihta has joined #openstack-swift02:17
*** nosnos has joined #openstack-swift02:18
*** Shihta_ has joined #openstack-swift02:26
*** Shihta has quit IRC02:30
notmynametonight I'll be updating launchpad and sending ttx the go-ahead for the final 2.2.1 release02:36
mattoliverauCool :)02:47
*** Shihta_ has quit IRC02:55
*** mahatic has quit IRC03:01
*** nosnos has quit IRC03:10
*** nosnos has joined #openstack-swift03:11
*** nellysmitt has joined #openstack-swift03:12
*** nosnos has quit IRC03:16
*** nellysmitt has quit IRC03:17
*** nosnos has joined #openstack-swift04:02
*** ppai has joined #openstack-swift04:04
*** addnull has quit IRC04:26
*** addnull has joined #openstack-swift04:26
*** lcurtis has quit IRC04:26
notmynamedinner eaten. kids down. drinks made. time for swift04:49
openstackgerritJohn Dickinson proposed openstack/swift: Added Nicolas, who was erantly left out of AUTHORS  https://review.openstack.org/14298305:03
notmynameeikke: sorry about that. fix ^^05:03
*** nellysmitt has joined #openstack-swift05:13
*** nellysmitt has quit IRC05:18
*** miurahr has joined #openstack-swift05:44
*** miurahr has quit IRC05:54
*** miurahr has joined #openstack-swift06:01
*** miurahr has quit IRC06:09
*** ahonda has quit IRC06:28
*** ahonda has joined #openstack-swift06:28
*** NM has joined #openstack-swift06:29
*** k4n0 has joined #openstack-swift06:30
*** pcaruana has joined #openstack-swift06:49
notmynameok, all written up. I'll send it out tomorrow06:54
notmynamegood night06:54
*** NM has quit IRC06:55
*** ttrumm has joined #openstack-swift07:05
openstackgerritYuan Zhou proposed openstack/swift: Fix probe tests with EC policy  https://review.openstack.org/14300207:07
*** ttrumm has quit IRC07:10
*** zaitcev has quit IRC07:11
*** nellysmitt has joined #openstack-swift07:14
*** nellysmitt has quit IRC07:19
*** SkyRocknRoll has joined #openstack-swift07:36
*** miurahr has joined #openstack-swift08:02
*** ttrumm has joined #openstack-swift08:05
*** miurahr has quit IRC08:10
*** nosnos has quit IRC08:16
*** nosnos has joined #openstack-swift08:17
*** geaaru has joined #openstack-swift08:21
*** nosnos has quit IRC08:22
*** rledisez has joined #openstack-swift08:39
*** silor has joined #openstack-swift08:42
*** nellysmitt has joined #openstack-swift08:52
*** addnull has quit IRC09:06
*** addnull has joined #openstack-swift09:10
*** addnull has quit IRC09:11
*** jordanP has joined #openstack-swift09:17
*** dmorita has quit IRC09:17
*** silor has quit IRC09:28
*** nellysmitt has quit IRC09:31
*** david-ly_ has quit IRC09:32
*** aix has joined #openstack-swift09:33
*** david-lyle has joined #openstack-swift09:33
*** david-lyle has quit IRC09:38
*** nellysmitt has joined #openstack-swift09:38
*** ppai has quit IRC10:08
openstackgerritDonagh McCabe proposed openstack/swift: Add multiple reseller prefixes and composite tokens  https://review.openstack.org/13708610:19
*** ppai has joined #openstack-swift10:23
*** masonhsi_ has joined #openstack-swift10:29
*** masonhsiung has quit IRC10:32
*** masonhsi_ has quit IRC10:34
openstackgerritNicolas Trangez proposed openstack/swift: Add test coverage for `splice` and `tee` failure scenarios  https://review.openstack.org/14303110:34
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Merge "fix dlo manifest file getting versioned"  https://review.openstack.org/14303310:41
eikkenotmyname: heh, np, I only later realized my patch landed too late anyway :-)10:52
*** Masahiro has quit IRC10:59
*** aix has quit IRC11:00
*** nellysmitt has quit IRC11:13
*** aix has joined #openstack-swift11:21
*** kaliya has quit IRC11:22
*** nellysmitt has joined #openstack-swift11:38
*** mahatic has joined #openstack-swift11:39
*** haomaiwang has quit IRC11:40
*** nellysmitt has quit IRC11:50
*** Masahiro has joined #openstack-swift11:59
*** Masahiro has quit IRC12:04
*** nellysmitt has joined #openstack-swift12:13
*** jasondotstar is now known as jasondotstar|afk12:16
*** k4n0 has quit IRC12:23
*** nellysmitt has quit IRC12:35
*** jordanP has quit IRC12:35
*** jordanP has joined #openstack-swift12:35
*** nellysmitt has joined #openstack-swift12:40
*** NM has joined #openstack-swift12:43
*** swifterdarrell has quit IRC12:44
*** amandap has quit IRC12:45
*** chrisnelson has quit IRC12:45
*** ahonda has quit IRC12:46
*** swifterdarrell has joined #openstack-swift12:46
*** ChanServ sets mode: +v swifterdarrell12:46
*** tdasilva has quit IRC12:46
*** ahonda has joined #openstack-swift12:46
*** tdasilva has joined #openstack-swift12:46
*** SkyRocknRoll has quit IRC12:46
*** chrisnelson has joined #openstack-swift12:47
*** amandap has joined #openstack-swift12:48
*** geaaru has quit IRC12:58
*** Masahiro has joined #openstack-swift13:01
*** nellysmitt has quit IRC13:02
*** Masahiro has quit IRC13:06
*** nellysmitt has joined #openstack-swift13:08
*** exploreshaifali has joined #openstack-swift13:21
*** ppai has quit IRC13:27
remix_tjis possible to change in a second moment the replication address used in the ring configuration? my network admins didn't defined the networks yet -_-13:36
*** miurahr has joined #openstack-swift13:58
*** fandi has joined #openstack-swift13:59
*** miurahr has quit IRC14:11
*** lpabon has joined #openstack-swift14:32
*** exploreshaifali has quit IRC14:39
*** ttrumm has quit IRC14:43
*** tellesnobrega_ has joined #openstack-swift14:49
*** NM has quit IRC14:50
*** Masahiro has joined #openstack-swift14:50
*** tellesnobrega_ has quit IRC14:51
*** lcurtis has joined #openstack-swift14:54
*** Masahiro has quit IRC14:55
*** NM has joined #openstack-swift14:57
*** tellesnobrega_ has joined #openstack-swift15:10
*** tellesnobrega_ has quit IRC15:11
*** tellesnobrega_ has joined #openstack-swift15:14
*** esmute has quit IRC15:14
*** ferahgo has quit IRC15:15
*** jasondotstar|afk has quit IRC15:16
*** esmute has joined #openstack-swift15:18
*** tellesnobrega_ has quit IRC15:22
*** jasondotstar|afk has joined #openstack-swift15:24
*** bill_az has joined #openstack-swift15:39
*** bill_az has quit IRC15:39
*** bill_az has joined #openstack-swift15:39
*** jasondotstar|afk has quit IRC15:41
*** geaaru has joined #openstack-swift15:41
*** pberis has quit IRC15:49
*** pcaruana has quit IRC15:59
*** pberis has joined #openstack-swift15:59
*** jasondotstar has joined #openstack-swift16:00
*** rdaly2 has joined #openstack-swift16:06
notmynamegood morning16:09
tdasilvagood morning!16:16
*** Nadeem has joined #openstack-swift16:19
notmynamehttp://lists.openstack.org/pipermail/openstack-dev/2014-December/053351.html   <-- 2.2.1 released16:22
*** ChanServ changes topic to "Swift 2.2.1 | Priority Reviews: https://wiki.openstack.org/wiki/Swift/PriorityReviews | Ideas: https://wiki.openstack.org/wiki/Swift/ideas | Review Dashboard: http://goo.gl/r2mxbe"16:22
*** themadcanudist has joined #openstack-swift16:22
*** delatte has quit IRC16:22
themadcanudisthey guys, why would there be like 80-90 forks/s from container-server on my swift cluster?16:23
themadcanudistis that normal behaviour?16:23
notmynamelots of workers + lots of per-drive threads + lots of drives?16:23
* notmyname is being loose with the definition of "forks"16:23
themadcanudistoh, i mean OS clones/forks16:24
themadcanudistPER second16:24
themadcanudistmy swift cluster has always been very high load, and I notice that each node spends a lot of time forkbombing itself. Trying to figure out if this is normal. the process that execs() and exits immediately is <container-server>16:24
themadcanudist90 times a second16:24
notmynamemaybe your server room is too cold and skynet thinks it needs to keep it warm with cpu cylces? ;-)16:25
themadcanudisthehe16:25
themadcanudistthis is not normal, correct?16:25
notmynamethemadcanudist: seriously, though. that doesn't sound normal, nor does anything immediately spring to mind like "oh that's something that happens when...."16:26
themadcanudisthrm, kk16:26
*** exploreshaifali has joined #openstack-swift16:26
notmynamethemadcanudist: you've got more than one container server in your cluster right?16:27
themadcanudistyeah16:27
* notmyname goes to check an option...16:27
notmynamethemadcanudist: good thing I checked. my guess was way off16:29
notmynamethemadcanudist: ok, so if you want to play "mad scientist" and vivisect a live cluster, here's something to try. maybe it will give some more info about what's happening16:29
themadcanudistI think it's container-updater16:30
notmynamethemadcanudist: on one container server, turn it off (should have little impact on your cluster). then, from the prompt run `swift-init container no-daemon`. That will start it in interactive mode (not fork to background and dump all the logs to the tty16:30
themadcanudisti narrowed it down and stopped the proc temporarily16:31
notmynameah16:31
themadcanudistload dropped as well as clones/s16:31
mahaticgood morning!16:31
notmynamethemadcanudist: the "no-daemon" will work for any of the swift processes started with swift-init16:32
themadcanudistyeah, checking16:32
themadcanudistcontainer-updated can be down for a bit without impact, eh?16:32
notmynameya16:32
themadcanudistit's just a consistency check after the fact, correct?16:32
notmyname"just" ;-)16:33
notmynameit keeps the account metadata up to date. which in a lot of clusters (not sure about yours) directly affects billing/chargeback info. but yeah, it's designed to expect failures16:33
themadcanudist=)16:34
*** miurahr has joined #openstack-swift16:37
*** Masahiro has joined #openstack-swift16:39
notmynamethemadcanudist: ok, so maybe the container updater process doing a a fork bomb sounds vaguely familiar. looking...16:42
themadcanudistinteresting16:43
*** Masahiro has quit IRC16:43
*** NM has quit IRC16:44
notmynamethemadcanudist: this is what I was thinking about https://review.openstack.org/#/c/125197/16:45
notmynamethemadcanudist: that was for the auditor, but perhaps there's something similar in the container updater?16:45
themadcanudisthrm16:45
*** nellysmitt has quit IRC16:45
*** NM has joined #openstack-swift16:47
themadcanudistok, thanks for the info!16:47
notmynamehmmm...looks like it might in fact be the same sort of issue16:49
themadcanudistreally?16:50
notmynamethemadcanudist: what are the chances you'd be able to test it out and submit a patch?16:50
themadcanudisthrm, i'm not using the latest16:50
*** miurahr has quit IRC16:51
notmynamethemadcanudist: I'm not 100% sure. I've only looked at the code and not run it with this in mind. but it does look similar.16:51
themadcanudistshoudln't there be an error log of sorts?16:51
notmynamethemadcanudist: but, if it is the same, then it's the result of a mis config (eg the permissions described in the commit message of the other patch). but even with a mis config, then swift shouldn't fork bomb your system16:52
themadcanudistyeah, the perms are good16:53
themadcanudistso, i'm not sure what would cause it?16:53
notmynamethemadcanudist: not sure. but that's why I was curious about running it in no-daemon mode. you might see a traceback there16:54
themadcanudistyeah16:54
themadcanudisttrying to reproduce. I really appreciate the help, just going to log my terminal and i need to step away16:55
themadcanudistbut I'll be around!16:55
*** gyee has joined #openstack-swift16:58
*** masonhsiung has joined #openstack-swift17:02
*** NM has quit IRC17:04
*** nellysmitt has joined #openstack-swift17:05
*** masonhsiung has quit IRC17:07
*** nellysmitt has quit IRC17:08
*** masonhsiung has joined #openstack-swift17:09
*** rdaly2 has quit IRC17:10
*** NM has joined #openstack-swift17:11
*** NM has quit IRC17:12
*** miurahr has joined #openstack-swift17:14
themadcanudistnotmyname: quick update. it's not a fork bomb17:15
themadcanudistit's simply how aggressive it is17:15
themadcanudisti used slowdown 0.05 and concurrency 1 on the updater17:15
themadcanudistand now it does it's job with some pacing17:15
notmynameyay17:15
themadcanudist;)17:15
*** lpabon has quit IRC17:15
notmynamethemadcanudist: I believe you misspelled "it's simply how awesome swift is"17:15
themadcanudistlol17:15
themadcanudist;)17:15
*** NM has joined #openstack-swift17:17
*** jordanP has quit IRC17:18
*** geaaru has quit IRC17:23
*** aix has quit IRC17:30
*** masonhsiung has quit IRC17:37
*** exploreshaifali has quit IRC17:45
*** lcurtis has quit IRC18:01
*** rledisez has quit IRC18:25
mahaticnotmyname, I'll have  something to work on if you take a look at this https://review.openstack.org/#/c/140103/ ;)18:26
*** Masahiro has joined #openstack-swift18:27
notmynamemahatic: I'm going to be unavailable for the next few hours. a quick glance looks like you're on the right track. a few minor things to work out (calling the parent class, only setting _allowed_mothods = None in the parent class)18:29
mahaticnotmyname, okay. I thought about that too. Leave your further comments/suggestions when you find time. thanks!18:31
mahaticnotmyname, thought about the _allowed_methods thing. Not sure about the "calling parent class though)18:32
*** Masahiro has quit IRC18:32
notmynamemahatic: google for "python super()"18:33
mahaticnotmyname, oh, that's what you mean. I read what you wrote as two diff things. sorry18:35
mahaticgot it now. will correct18:35
*** masonhsiung has joined #openstack-swift18:48
*** masonhsiung has quit IRC18:53
*** rdaly2 has joined #openstack-swift18:59
*** openstack has joined #openstack-swift19:09
*** openstackstatus has joined #openstack-swift19:10
*** ChanServ sets mode: +v openstackstatus19:10
*** tdasilva has quit IRC19:30
*** lpabon has joined #openstack-swift19:34
*** ondergetekende_ is now known as ondergetekende19:40
*** bgmccollum has left #openstack-swift19:41
*** panbalag has quit IRC19:41
*** mahatic has quit IRC19:44
*** tdasilva has joined #openstack-swift19:45
*** dmsimard_away is now known as dmsimard19:47
*** dmsimard is now known as dmsimard_away19:47
*** mahatic has joined #openstack-swift19:59
*** Masahiro has joined #openstack-swift20:16
*** Masahiro has quit IRC20:20
*** exploreshaifali has joined #openstack-swift20:27
notmyname /back20:34
*** NM has quit IRC20:36
*** gyee has quit IRC20:39
*** zaitcev has joined #openstack-swift20:41
*** ChanServ sets mode: +v zaitcev20:41
*** gyee has joined #openstack-swift20:55
*** tongli has joined #openstack-swift20:56
*** nellysmitt has quit IRC20:58
*** nellysmitt has joined #openstack-swift21:00
tdasilvanotmyname: hi, what is the purpose of this patch? https://review.openstack.org/#/c/143033/21:16
notmynametdasilva: ah, interesting. hadn't seen it yet. there's a story there21:17
notmynametdasilva: after every release, the bot will automatically merge the tag back into master so that HEAD always has the latest release as an ancestor in the git history21:18
notmynametdasilva: this is really important when there are multiple RCs and stuff that lands for the release after the RC is cut21:18
notmynametdasilva: in this case (with 2.2.1), it's a no-op21:19
*** exploreshaifali has quit IRC21:19
notmynametdasilva: thanks. I abandoned it21:20
tdasilvanotmyname: cool, i just saw it by chance...21:25
*** nellysmitt has quit IRC21:34
openstackgerritMahati proposed openstack/swift: Implement OPTIONS verb for storage nodes.  https://review.openstack.org/14010321:53
*** Masahiro has joined #openstack-swift22:05
*** pberis has quit IRC22:08
*** NM has joined #openstack-swift22:09
*** NM has quit IRC22:09
*** Masahiro has quit IRC22:09
*** lpabon has quit IRC22:12
*** sandywalsh has quit IRC22:40
*** sandywalsh has joined #openstack-swift22:43
*** IRTermite has quit IRC22:45
openstackgerritSamuel Merritt proposed openstack/swift: Add notion of overload to swift-ring-builder  https://review.openstack.org/14145222:52
openstackgerritSamuel Merritt proposed openstack/swift: Only move too-close-together replicas when they can spread out.  https://review.openstack.org/14087922:52
torgomaticclayg: ping22:56
claygtorgomatic: pong22:59
claygtorgomatic: if you're just going to tell me I should review your ring patches - I already know I *should*22:59
*** jordanP has joined #openstack-swift22:59
clayg... but obvioulsy haven't yet :\22:59
torgomaticclayg: I was just excited that I got the overload one working22:59
torgomaticit was lots trickier than it sounded at first glance23:00
claygheh23:02
claygsweet, I'll curl up with it and a hot coco over the weekend23:02
claygit's very rainy over here23:02
torgomaticput some whiskey in the cocoa; you may need it23:02
claygheh23:02
pelusemmmm... whiskey23:22
notmynamepeluse: you mis-spelled tequila23:25
peluseman, I'm going to need to crack a beer open here in a minute, having a reaction to all this talk :)23:26
chrisnelsontorgomatic: peluse: notmyname: There are lots of IRC messages that fly across my screen in a day and get ignored, then there are those which cannot help but get my attention...  :)23:26
peluseand I take back everything I say about the recontruction part of the reconstructor being the easiest part...23:26
notmynamecold and rainy on my side of town too. I'd be fine with some whiskey23:26
notmynamechrisnelson: :-)23:26
notmynamepeluse: lol23:27
peluseclayg I'm going to plumb in a quick and dirty dumb EC get function for me to use in recontruction and will wrangle in your stuff when ready23:27
notmynamepeluse: the only easy parts are the ones that have already been written and verified to work23:27
peluseamen23:27
pelusechrisnelson, all work and no play, something something something....23:28
chrisnelsonpeluse: ha23:28
notmynamechrisnelson: I tried this reposado a while back (from the texas legend himself) http://www.maninblacktequila.com/Kinkyabout.html23:31
*** fandi has quit IRC23:32
chrisnelsonnotmyname: ah, I've had it.  Nice!  On a related topic, I picked up some Del Maguey Vida mezcal the other day and served some Mexican Last Words at a holiday party.  Well received.23:33
*** nellysmitt has joined #openstack-swift23:34
notmynamevery nice23:36
notmynamechrisnelson: the vida is good, but little harsh for my taste. Mezcal Vago Espadin is very good and not too much more. and Del Maguey has some crazy expensive stuff too that, of course, is tastey.23:37
chrisnelsonnotmyname: Good to know.  That was my first taste of mezcal in many years, so I'll have to try some more.23:38
*** nellysmitt has quit IRC23:39
*** Masahiro has joined #openstack-swift23:54
*** bill_az has quit IRC23:58
*** Masahiro has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!