Thursday, 2014-08-21

*** gyee_ has quit IRC00:21
*** dmorita has joined #openstack-swift00:32
openstackgerritSamuel Merritt proposed a change to openstack/swift: Refactor object PUT to make EC easier to add  https://review.openstack.org/11408400:36
*** Midnightmyth has quit IRC01:00
*** tgohad has joined #openstack-swift01:09
*** tsg has quit IRC01:10
*** bill_az_ has quit IRC01:14
*** sungju has joined #openstack-swift01:15
*** shakamunyi has joined #openstack-swift01:17
*** nosnos has joined #openstack-swift01:44
openstackgerritZhang Hua proposed a change to openstack/swift: Add distributed tracing capablities in logging.  https://review.openstack.org/9367701:46
*** haomaiw__ has quit IRC02:12
*** tongli has quit IRC02:12
*** haomaiwang has joined #openstack-swift02:13
*** addnull has joined #openstack-swift02:15
*** tgohad has quit IRC02:16
*** addnull_ has joined #openstack-swift02:16
*** addnull has quit IRC02:19
*** addnull_ has quit IRC02:19
*** addnull has joined #openstack-swift02:20
*** atan8 has quit IRC02:20
*** addnull has quit IRC02:20
*** addnull has joined #openstack-swift02:20
*** addnull has joined #openstack-swift02:21
*** addnull has quit IRC02:21
*** addnull has joined #openstack-swift02:21
*** addnull has quit IRC02:22
*** addnull has joined #openstack-swift02:22
*** shakayumi has joined #openstack-swift02:24
thurloatgood day / evening to sirs / ladies02:26
thurloatour swift server appears to be running out of ephemeral ports for replication tasks02:27
thurloatgetting: object-replicator: Error syncing with node: .... EADDRNOTAVAIL02:27
*** shakamunyi has quit IRC02:28
thurloatnetstat -napt | wc -l is outputting ~28k, mostly in TIME_WAIT02:29
thurloatwhen they finally clear up, it looks like it starts ploughing through again until it runs out of ports and starts spewing the same errors02:29
*** alexiz has joined #openstack-swift02:30
thurloattwo questions out of this: is it okay for these replication requests to get delayed like this?, and if I were to start tuning, would it be swift config to throttle the replicators or try massaging the system to time out requests quicker / increase the ephemeral ports count02:31
*** shakayumi has quit IRC02:34
*** tsg has joined #openstack-swift02:41
*** addnull has quit IRC02:52
*** liuxf has joined #openstack-swift02:55
*** liuxf has quit IRC02:55
openstackgerritTakashi Kajinami proposed a change to openstack/swift: Ensure update of the container by object-updater  https://review.openstack.org/9959802:58
*** haomai___ has joined #openstack-swift03:03
*** haomaiwang has quit IRC03:06
*** chandankumar has joined #openstack-swift03:27
*** chandankumar has quit IRC03:44
*** gvernik has joined #openstack-swift03:49
*** stevemac has joined #openstack-swift03:50
*** tgohad has joined #openstack-swift03:51
*** tsg has quit IRC03:54
*** shakamunyi has joined #openstack-swift03:54
*** stevemac has quit IRC03:55
*** alexiz has quit IRC03:57
*** gvernik has quit IRC03:58
*** elambert has joined #openstack-swift04:03
openstackgerritTakashi Kajinami proposed a change to openstack/swift: Ensure update of the container by object-updater  https://review.openstack.org/9959804:14
*** nexusz99 has joined #openstack-swift04:14
*** ZBhatti has quit IRC04:26
*** atan8 has joined #openstack-swift04:28
*** nexusz99 has quit IRC04:32
*** nexusz99 has joined #openstack-swift04:32
*** nexusz99 has quit IRC04:34
*** nexusz99 has joined #openstack-swift04:35
*** nexusz99 has quit IRC04:41
*** nexusz99 has joined #openstack-swift04:41
*** nexusz99_ has joined #openstack-swift04:43
*** nexusz99 has quit IRC04:43
*** tgohad has quit IRC04:43
*** nexusz99_ has quit IRC04:48
*** nexusz99 has joined #openstack-swift04:49
*** nexusz99 has quit IRC04:53
*** tsg has joined #openstack-swift05:09
*** joeljwright has joined #openstack-swift05:21
openstackgerritOpenStack Proposal Bot proposed a change to openstack/swift: Updated from global requirements  https://review.openstack.org/8873605:22
*** infotection has joined #openstack-swift05:22
*** chandankumar has joined #openstack-swift05:25
*** nshaikh has joined #openstack-swift05:26
*** addnull has joined #openstack-swift05:27
*** shakamunyi has quit IRC05:37
*** elambert has quit IRC05:57
*** gvernik has joined #openstack-swift06:17
*** gvernik has quit IRC06:17
*** k4n0 has joined #openstack-swift06:25
*** nshaikh has quit IRC06:28
*** ppai has joined #openstack-swift06:30
*** atan8 has quit IRC06:55
*** shakamunyi has joined #openstack-swift07:03
*** nshaikh has joined #openstack-swift07:03
*** shakamunyi has quit IRC07:08
*** thurloat has quit IRC07:14
*** thurloat has joined #openstack-swift07:15
*** sungju has quit IRC07:32
*** infotection has quit IRC07:41
*** infotection has joined #openstack-swift07:41
*** aix has joined #openstack-swift07:53
*** shakamunyi has joined #openstack-swift08:04
*** shakamunyi has quit IRC08:09
*** geaaru has joined #openstack-swift08:21
*** kragniz has quit IRC08:27
*** kragniz has joined #openstack-swift08:29
*** aix has quit IRC08:46
*** ByteSore has joined #openstack-swift08:47
ByteSoremorning all08:47
*** nexusz99 has joined #openstack-swift08:49
ByteSoreI just installed openstack on Rhel7 and swift on another Rhel7 machine. following the installation manual on access.redhat.com. It said i need to run swift list to see if there is a connection to the proxy but all i get is bash: swift: command not found. is this changed?08:50
*** nexusz99 has quit IRC08:53
*** mrmoje has joined #openstack-swift08:55
*** shakamunyi has joined #openstack-swift09:05
*** sungju has joined #openstack-swift09:06
*** homegrown has joined #openstack-swift09:07
*** shakamunyi has quit IRC09:09
*** bkopilov has quit IRC09:17
*** aix has joined #openstack-swift09:24
*** tsg has quit IRC09:34
*** sungju has quit IRC09:39
*** sungju has joined #openstack-swift09:47
*** mrmoje_ has joined #openstack-swift09:54
*** dmorita has quit IRC09:55
*** mrmoje has quit IRC09:56
*** Midnightmyth has joined #openstack-swift10:02
*** shakamunyi has joined #openstack-swift10:06
*** shakamunyi has quit IRC10:10
*** aswadr has quit IRC10:12
*** mkollaro has joined #openstack-swift10:19
*** bkopilov has joined #openstack-swift10:28
*** addnull has quit IRC10:41
*** haomai___ has quit IRC10:50
*** shakamunyi has joined #openstack-swift11:07
*** shakamunyi has quit IRC11:11
*** nexusz99 has joined #openstack-swift11:26
*** ppai has quit IRC11:29
*** mkerrin1 has quit IRC11:38
*** zul has joined #openstack-swift11:57
*** shakamunyi has joined #openstack-swift12:07
*** sungju has quit IRC12:08
*** shakamunyi has quit IRC12:11
*** nosnos has quit IRC12:13
*** Midnightmyth has quit IRC12:19
*** traz_ has quit IRC12:30
*** zul has quit IRC12:44
*** zul has joined #openstack-swift12:50
*** bnelson has joined #openstack-swift12:54
*** aix has quit IRC12:57
*** bill_az_ has joined #openstack-swift12:57
*** shakamunyi has joined #openstack-swift13:03
*** xianghuihui has joined #openstack-swift13:06
*** xianghui has quit IRC13:09
*** pberis has quit IRC13:11
*** pberis has joined #openstack-swift13:12
*** xianghuihui has quit IRC13:13
*** shakamunyi has quit IRC13:19
*** openstackgerrit has quit IRC13:21
*** sas has joined #openstack-swift13:33
*** shakamunyi has joined #openstack-swift13:36
*** zul has quit IRC13:39
*** Midnightmyth has joined #openstack-swift13:42
*** nexusz99 has quit IRC13:42
*** zul has joined #openstack-swift13:49
*** k4n0 has quit IRC13:55
*** openstackgerrit has joined #openstack-swift14:00
*** tongli has joined #openstack-swift14:11
*** dmsimard_away is now known as dmsimard14:11
*** mrmoje_ has quit IRC14:17
*** mrmoje has joined #openstack-swift14:19
*** aix has joined #openstack-swift14:20
*** tdasilva has joined #openstack-swift14:22
ByteSorethat one is fixed..14:25
ByteSoredoes someone know how to fix this? swift: ERROR Insufficient Storage 192.168.0.102:6002/accounts14:25
ByteSorei get it when i run swift list14:25
*** mrmoje has quit IRC14:29
*** tsg has joined #openstack-swift14:30
openstackgerritThiago da Silva proposed a change to openstack/swift: moving object validation checks to top of PUT method  https://review.openstack.org/11599514:35
openstackgerritThiago da Silva proposed a change to openstack/swift: code shuffle post expired headers refactor  https://review.openstack.org/11280414:45
openstackgerritThiago da Silva proposed a change to openstack/swift: moving object validation checks to top of PUT method  https://review.openstack.org/11599514:45
*** mwstorer has joined #openstack-swift14:50
*** nshaikh has quit IRC14:55
*** goodes has quit IRC14:56
*** goodes has joined #openstack-swift14:57
*** astellwag has quit IRC14:58
*** mjseger has quit IRC15:02
*** astellwag has joined #openstack-swift15:05
*** zul has quit IRC15:16
*** mkollaro has quit IRC15:19
notmynameByteSore: I'm assuming that you're getting that message when you actually think you have plenty of space available?15:30
*** occupant has quit IRC15:31
notmynamethurloat: look at setting tcp_tw_recycle and/or tcp_tw_reuse15:33
notmynamethurloat: eg http://www.fromdual.com/huge-amount-of-time-wait-connections15:33
notmynamethurloat: (dated, but still generally good info) http://docs.openstack.org/developer/swift/deployment_guide.html#general-system-tuning15:33
*** infotection has quit IRC15:34
openstackgerritThiago da Silva proposed a change to openstack/swift: Storage Policy defined constraints middleware  https://review.openstack.org/11332515:34
*** cebruns has quit IRC15:34
thurloatthanks notmyname, I see that it's an RTFM that we missed setting it up.15:35
thurloatwill apply asap15:36
*** chandankumar has quit IRC15:36
*** infotection has joined #openstack-swift15:37
notmynamethurloat: one other thing to consider is that it may be rsync chewing through the ports too. check what's goin on there, and see if you may need to limit how many rsync connections are allowed at once15:37
thurloatThat change dropped the connection count to a much more reasonable ~8k15:38
thurloatcurrently not seeing many rsync connections open, will keep my eye out for it.15:39
notmynamecool15:39
*** annegentle has joined #openstack-swift15:40
thurloatnotmyname: would there have been any replication failures because of those errors? or does it pick up where it leaves off on failure?15:40
notmynamethurloat: it picks up where it leaves off (sortof. the answer to the spirit of your question is "yes")15:40
thurloati.e. some objects not 3x replicated15:40
notmynamethurloat: that being said, you can run the replicator in a "once" mode and even give it a specific drive or partition to replicate.15:41
*** zul has joined #openstack-swift15:42
notmynamethurloat: swift-dispersion-report is good for letting you know what's going on there (if you need to worry about some partition). also, you can use swift-ring-builder to find common partitions eg if you have multiple failed devices in the cluster15:42
*** mwstorer has quit IRC15:44
notmynameok, I'm out for a bit. time to get ready to go in to the office and do the meetings15:46
*** zslot has joined #openstack-swift15:51
thurloatthanks again for the guidance notmyname!15:51
*** DisneyRicky has quit IRC15:55
*** infotection has quit IRC15:56
*** gyee_ has joined #openstack-swift15:59
*** DisneyRicky has joined #openstack-swift15:59
*** aix has quit IRC16:00
openstackgerritSamuel Merritt proposed a change to openstack/swift: Let admins add a region without melting their cluster  https://review.openstack.org/11544116:00
*** infotection has joined #openstack-swift16:01
*** DisneyRicky has quit IRC16:03
*** mwstorer has joined #openstack-swift16:04
*** aix has joined #openstack-swift16:13
*** homegrown has quit IRC16:13
*** DisneyRicky has joined #openstack-swift16:18
*** mwstorer has quit IRC16:20
*** elambert has joined #openstack-swift16:21
*** aix has quit IRC16:21
*** infotection has quit IRC16:30
*** mwstorer has joined #openstack-swift16:33
*** zul has quit IRC16:34
bnelsonI had a question about how to prevent unauthenticated connections to the swift services.  The proxy server has authentication checking in the pipeline, but the object, container and account servers don't.16:35
*** infotection has joined #openstack-swift16:35
bnelsonIs there a way to enforce authenticated connections to those servers?  If I add auth checking to their pipelines, will that cause problems when the swift services themselves (reaper, replicator, etc) need to contact the object, container and account servers?16:36
pelusebnelson: that's because, by design, those should be on a protected network (they don't communicate with clients directly)16:36
pelusebnelson:  see http://docs.openstack.org/security-guide/content/object-storage.html16:37
pelusebnelson:  wrt your question on securing the back-end servers, will let someone else field that one, never thought about it before :)16:38
bnelsonThanks.  I'll look into that link.  I think I should be able to structure my setup so that the backend servers only listen on private networks.  But I would be interested in any approaches which could secure those backend servers when they are on the external network.16:39
*** mwstorer has quit IRC16:44
*** zul has joined #openstack-swift17:00
*** mwstorer has joined #openstack-swift17:07
notmynamebnelson: peluse just gave you excellent advice. some other things you could do would be to keep the IP manifest of the other swift servers (you've actually already got this in the ring) and then set up firewall rules so only those IPs can connect17:12
notmynamebnelson: we've talked about adding tls to the proxy->storage connections. it's certainly good in some deployments, but it's also pretty expensive (cpu/throughput)17:13
notmynamebnelson: there is another option. /me goes to look for link17:13
notmynamebnelson: peluse: see https://blueprints.launchpad.net/swift/+spec/secure-internal-network-requests17:13
notmynameold idea, but nobody has actually prioritized it enough to work on it17:14
notmynamebasically, hmac sign the requests and reject bad sigs17:14
notmynamedoesn't encrypt data, but would prevent malicious messages17:14
bnelsonnotmyname: thanks. that's a good idea, too.  So it looks like the securing the backend services is accomplished by limiting network access. Thanks for clarifying that17:15
notmynameyup17:15
notmynameand I'd love to see that blueprint implemented17:15
werI just generate an  rsync config every few minutes from the ring.  Cause sad or migrating hardware with incorrect ring info can be problematic.17:15
werI think it happened to me first time I removed a node and built a new cluster.....17:16
werIt's lame but it works.17:16
notmynamewer: "problematic" == data moved needlessly or otherwise to the "wrong" place?17:16
openstackgerritpaul luse proposed a change to openstack/swift: Make some sqlite UPDATE calls only to records that have changed  https://review.openstack.org/11577517:17
werproblematic mean.... it died... or someone did something.17:17
notmynamewer: hmm...that does sound "problematic" ;-)17:17
werIf a machine comes up with old ring info for example.17:17
werOr if you forget to removed from a cluster.... or rings are not fully distributed yet.17:18
notmynamewer: but we expect different versions of the ring to be in the cluster at the same time. ie you can't atomically deploy a new ring to 100+ servers at the exact same time. so if you found an issue, I'd like to work through it to see i there are things we can do to make your life easier17:18
werSo, I just deleted a node in a cluster.  And was standing up a new cluster..... The new node was on the same ip.  The cluster started dumping into it.... the new cluster (which had other nodes) began to replicate it :)17:19
*** cebruns has joined #openstack-swift17:21
*** tong_ has joined #openstack-swift17:23
notmynamewer: yes, I could imagine that would cause problems. essentially you're telling your 2 independent swift clusters that they overlap. and that will, in almost all cases, probably not turn out well17:24
*** annegentle has quit IRC17:24
werMy solution was to have each node generate an rsync config from it's point of view every few minutes and that stopped that behavior....   However I am using a pretty simple ring.  All 3 rings are the same.17:24
weryeah  it was fun.  I had made the assumption that the ring hashes prevented that....  But no, it's just rsync... and I still don't know how I would clean up a scenario like that.  But I can tell you it totally happens.  And in a large system, it could go unnoticed.17:25
*** tongli has quit IRC17:26
werso why wouldn't you use the swift_hash_path_suffix/prefix for this?  I guess I need to go remember what those are supposed to do again.  But I agree tls would be too expensive.  Another auth would be a pain.  But with a more complex ring structure one or both might need to be used.17:29
werderp.  placement is what they are used for  :)  yeah, cleanup could be a pain.17:31
werwith an overlap like that.17:31
*** joeljwright has quit IRC17:31
notmynameya, just hash salts17:33
*** chandankumar has joined #openstack-swift17:34
*** zul has quit IRC17:34
wermaybe you oculd use them with rsync.secrets or something.  And call it done. /etc/rsyncd.secrets  prefix:suffix17:35
*** chandankumar has quit IRC17:37
*** cebruns has quit IRC17:41
*** cebruns has joined #openstack-swift17:46
*** sas has quit IRC17:48
*** tongli has joined #openstack-swift17:48
*** cebruns has quit IRC17:51
*** tong_ has quit IRC17:52
*** cebruns has joined #openstack-swift17:53
*** geaaru has quit IRC17:53
*** pberis has quit IRC17:57
*** Midnightmyth has quit IRC17:59
*** annegentle has joined #openstack-swift18:00
*** annegentle has quit IRC18:06
*** annegentle has joined #openstack-swift18:11
*** morganfainberg is now known as morganfainberg_Z18:13
*** mrmoje has joined #openstack-swift18:26
*** shakamunyi has quit IRC18:27
*** occupant has joined #openstack-swift18:30
*** zul has joined #openstack-swift18:35
*** gyee_ has quit IRC18:36
*** zul has quit IRC18:37
*** zul has joined #openstack-swift18:46
*** morganfainberg_Z is now known as morganfainberg18:47
openstackgerritJesse J. Cook proposed a change to openstack/python-swiftclient: Python sessions, streaming, and increased pool  https://review.openstack.org/11606518:48
*** Midnightmyth has joined #openstack-swift18:50
*** tsg has quit IRC19:01
*** zul has quit IRC19:03
tdasilvaclayg: thanks for the review on 11599519:07
*** tongli has quit IRC19:10
claygtdasilva: I think it's a great cleanup; I'm going try and figure out what it is I think I should know about the content-length/transfer-encoding thing19:11
tdasilvaok...i was trying to maintain the same functionality19:12
claygtdasilva: acctually; any isnight you have might be helpful19:12
tdasilvain that case, the check_object_creation had a check to make sure content-length or transfer-encoding was provided19:12
claygtdasilva: well it seemed like the KeyError was trying to drive at a subtle distinction between some variation in how the client presents the Content-Length header?19:12
claygtdasilva: yeah I thought maybe somehow the existing checks in the controller were already making that distinction; or at least enough of one to decide if the request was deemed "valid"19:13
*** mwstorer has quit IRC19:13
tdasilvaright..so i removed the check in check_object_creation19:13
claygI see no harm in moving that logic into the check function - but I wasn't sure if the result was exactly the same?  maybe it worked out the same in practice19:13
tdasilvaand added to swob19:14
claygoh; is that all that happened?19:14
tdasilvait's what i tried to do :-)19:14
tdasilvabut i'm glad you are double-checking19:14
tdasilvaclayg: and the ml not being none was a copy-paste from the PUT19:15
tdasilvain controllers/obj.py19:16
*** zul has joined #openstack-swift19:16
tdasilvaso the issue is that we had a call in controllers/obj.py to message_length which was ok if there was no content-length or transfer-encoding19:17
claygbah; i can't see it :'(19:17
tdasilvait was ok, because later on in the PUT function we added content-length right before calling check_object_creation19:17
claygI'm going to just test it; if it works it's fine :D19:18
tdasilvahehehe...sorry if i'm confusing19:18
claygno no that's not it - I think the code was confusing to begin with because we were checking basically the same stuff in different places different ways19:19
claygwhich is why it's so good to get it consolidated - we'll acctually have a chance at understand what all that crap is trying validate and what our rules acctually ARE19:19
tdasilvaclayg: https://github.com/openstack/swift/blob/master/swift/proxy/controllers/obj.py, look at lines 495, 502, 543, 55019:19
tdasilvaexactly...that's what i was trying to do19:20
tdasilvajust to simplify that code19:20
*** mwstorer has joined #openstack-swift19:21
*** zul has quit IRC19:21
tdasilvaclayg: what I'm going to try to do is write up a gist of the code flow before and after, and maybe that will help in finding any issues19:23
*** morganfainberg has quit IRC19:25
*** morganfainberg has joined #openstack-swift19:27
claygdid anyone else have any issues with the new futures depends on the SwiftServices change for python-swiftclient?  https://review.openstack.org/#/c/85453/19:33
*** zul has joined #openstack-swift19:36
*** bobby2_ is now known as bobby219:39
*** joeljwright has joined #openstack-swift19:56
joeljwrightclayg: thanks for taking a look at python-swiftclient patch 8545319:59
joeljwrightclayg: are you still having issues importing/using the futures?20:00
joeljwrightclayg: I'm completely baffled by the behaviour you're seeing20:00
*** mrmoje has quit IRC20:03
*** Midnightmyth has quit IRC20:08
*** marcusvrn has quit IRC20:11
*** morganfainberg is now known as morganfainberg_Z20:31
openstackgerritA change was merged to openstack/python-swiftclient: fixed unit tests when env vars are set  https://review.openstack.org/11513120:34
*** bsdkurt has quit IRC20:45
*** morganfainberg_Z is now known as morganfainberg20:47
*** zul has quit IRC21:00
*** joeljwright has left #openstack-swift21:02
claygwhat version of setuptools is everyone on these days?  pip freeze | grep -i setuptools is not showing me anything (!?)21:06
claygI had python-setuptools installed on my precise which never gave me issue before (the system packaged setuptools with lucid was a disaster of course) - but when trying to get the futures depends installed for the swiftclient change I saw my old friend TypeError dist must be a Distribution21:07
goodeshas anyone had luck using the code 'Red Hat Hackathon' to book the room at the group rate?21:13
openstackgerritA change was merged to openstack/swift: Fix sporadic false failure in xprofile unit test code (master)  https://review.openstack.org/11529221:20
*** tsg has joined #openstack-swift21:32
*** pberis has joined #openstack-swift21:36
pelusegoodes:  I picked a Marriott that's somewhere close by I think (I hope)21:37
goodespeluse: did not work on the website, so ended up calling them, that worked21:39
*** annegentle has quit IRC22:04
*** bill_az_ has quit IRC22:08
clayghas anyone tried to build python-swiftclient docs like... ever?22:16
claygI *think* they're a thing?  http://docs.openstack.org/developer/python-swiftclient/22:16
claygthere's a tox command for "docs"?22:16
*** erlon has quit IRC22:20
*** dmsimard is now known as dmsimard_away22:29
*** annegentle has joined #openstack-swift22:34
mattoliverauMorning all22:35
*** mrmoje has joined #openstack-swift22:37
*** shakamunyi has joined #openstack-swift22:37
claygjoelwright: where'd you go!?22:38
*** sungju has joined #openstack-swift22:38
*** annegentle has quit IRC22:39
openstackgerritClay Gerrard proposed a change to openstack/python-swiftclient: add some autodoc on swiftclient.service  https://review.openstack.org/11612022:40
*** shakamunyi has quit IRC22:41
claygnotmyname: do you or anyone you care about have a process around alerting for new depends (e.g. swiftclient post services patch will depend on futures 2.1.3, which may not have system packages on some distros e.g. precise has 2.1.2)22:44
mattoliverauShould we rely on pip rather than system packages then?22:46
mattoliverauCause isn't that what requirements.text is all about22:46
mattoliverau*Txt (damn auto complete)22:47
*** shakamunyi has joined #openstack-swift22:51
*** annegentle has joined #openstack-swift22:52
*** zul has joined #openstack-swift22:55
*** pberis has quit IRC22:59
*** annegentle has quit IRC23:05
claygmattoliverau: where "we" is people who develop on these code bases or people managing deployments of these services?23:05
*** annegentle has joined #openstack-swift23:05
*** mrmoje has quit IRC23:05
*** pberis has joined #openstack-swift23:08
*** zul has quit IRC23:17
*** annegentle has quit IRC23:22
mattoliverauclayg: good point, Dev, testing and in ci testing where I guess can get away with with pip. I work with one of the PyPi core Dev/maintainers (Richard Jones) so I'm probably just biased :)23:27
*** acoles has quit IRC23:33
*** tsg has quit IRC23:51
*** annegentle has joined #openstack-swift23:52
*** annegentle has quit IRC23:57
*** tsg has joined #openstack-swift23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!