Monday, 2014-06-16

*** dmorita has joined #openstack-swift00:33
*** matsuhashi has joined #openstack-swift00:35
*** matsuhashi has quit IRC00:35
*** matsuhashi has joined #openstack-swift00:35
*** mlipchuk has quit IRC00:41
openstackgerritMatthew Oliver proposed a change to openstack/swift: When a filesystem does't support xattr return a 507  https://review.openstack.org/9988300:47
*** nsquare has joined #openstack-swift00:47
*** haomaiwang has quit IRC00:51
*** haomaiwang has joined #openstack-swift00:54
*** nosnos has joined #openstack-swift00:55
*** zhiyan_ is now known as zhiyan02:00
*** yuan has quit IRC02:06
*** zhiyan is now known as zhiyan_02:18
openstackgerritMatthew Oliver proposed a change to openstack/swift: When a filesystem does't support xattr return a 507  https://review.openstack.org/9988302:22
*** zhiyan_ is now known as zhiyan02:27
*** fifieldt__ is now known as fifieldt02:30
*** haomai___ has joined #openstack-swift02:33
*** haomaiwang has quit IRC02:34
*** bkopilov has quit IRC02:47
*** nosnos has quit IRC03:36
*** nsquare has quit IRC03:50
*** nosnos has joined #openstack-swift04:15
*** bkopilov has joined #openstack-swift04:22
*** bkopilov_ has joined #openstack-swift04:28
*** bkopilov_ has joined #openstack-swift04:29
*** bkopilov has quit IRC04:31
*** kashyapk has joined #openstack-swift04:39
*** haomai___ has quit IRC04:44
*** haomaiwa_ has joined #openstack-swift04:45
*** kashyapk has quit IRC04:52
*** kashyapk has joined #openstack-swift04:52
*** kashyapk_ has joined #openstack-swift04:57
*** kashyapk has quit IRC04:58
openstackgerritA change was merged to openstack/swift: replaced e.message with a bare "e" object ref  https://review.openstack.org/9357004:58
*** haomai___ has joined #openstack-swift05:00
*** haomaiwa_ has quit IRC05:03
*** ppai has joined #openstack-swift05:05
*** psharma has joined #openstack-swift05:16
*** nshaikh has joined #openstack-swift05:17
*** kashyapk_ has quit IRC05:21
*** kashyapk has joined #openstack-swift05:22
*** kashyapk has quit IRC05:30
*** kashyapk has joined #openstack-swift05:30
*** asdfsf has quit IRC05:34
*** nsquare has joined #openstack-swift05:42
*** zhiyan is now known as zhiyan_05:55
*** zhiyan_ is now known as zhiyan05:56
*** matsuhashi has quit IRC06:00
*** matsuhashi has joined #openstack-swift06:01
*** matsuhas_ has joined #openstack-swift06:38
*** kashyapk has quit IRC06:38
*** matsuhashi has quit IRC06:38
*** kashyapk_ has joined #openstack-swift06:39
*** itarchitectkev has joined #openstack-swift06:48
*** haomai___ has quit IRC06:53
*** haomaiwang has joined #openstack-swift06:53
*** jamiehannaford has joined #openstack-swift06:56
*** matsuhas_ has quit IRC07:01
*** kashyapk_ has quit IRC07:03
*** matsuhashi has joined #openstack-swift07:04
*** chandan_kumar has quit IRC07:05
*** haomaiw__ has joined #openstack-swift07:08
*** haomaiwang has quit IRC07:10
*** haomaiw__ has quit IRC07:14
*** nsquare has quit IRC07:30
*** mmcardle has joined #openstack-swift07:30
*** kashyapk has joined #openstack-swift07:35
*** kashyapk has quit IRC07:38
*** kashyapk has joined #openstack-swift07:39
*** scohen has joined #openstack-swift07:42
mattoliverauWell I'm calling it a day, night all! And for those of you on the other side of the world, enjoy your Monday, mine is over :)07:42
*** kashyapk has quit IRC07:45
*** kashyapk has joined #openstack-swift07:45
*** kashyapk has quit IRC07:47
*** kashyapk has joined #openstack-swift07:47
*** rieglflo has joined #openstack-swift07:48
*** nacim has joined #openstack-swift07:54
rieglflohi everyone. is there any way to restrict container/object access to only one user/account while prohibiting that user from changing the acl to give everyone access?07:55
rieglfloa hint would be really useful, thanks07:56
*** ppai has quit IRC07:57
*** foexle has joined #openstack-swift08:03
hugokuorieglflo: well.... Swift has only container and account ACL.08:07
hugokuorieglflo: no Object level ACL tho.08:08
hugokuorieglflo: Would you like information about container or account level ACLs ?08:08
hugokuorieglflo: For container or account ACLs, only account Admin is able to set the ACL.08:09
*** kashyapk_ has joined #openstack-swift08:11
*** Midnightmyth has joined #openstack-swift08:11
*** kashyapk has quit IRC08:11
rieglflohugokuo: so setting X-Container-Read: .r:*,.rlistings is only possible via an admin role?08:14
hugokuorieglflo: yup...08:15
rieglflohugokuo: great. is there any other way for an authenticated user to make any objects publicly available expect the temp url plugin?08:17
rieglflohugokuo: except ... via the temp url plugin?08:18
*** fbo_away is now known as fbo08:20
hugokuograpsus_: no object level ACL implementation. So tempurl prolly the best way to make publicity on "single object"08:20
hugokuorieglflo: no object level ACL implementation. So tempurl prolly the best way to make publicity on "single object"08:20
*** ppai has joined #openstack-swift08:27
rieglflothank you very much!08:28
*** nshaikh has quit IRC08:37
*** circ-user-eYrGs has joined #openstack-swift08:45
*** help has joined #openstack-swift08:48
*** help is now known as Guest1763908:48
*** circ-user-eYrGs has quit IRC08:50
*** mkollaro has joined #openstack-swift08:53
*** nacim has quit IRC08:54
*** zhiyan is now known as zhiyan_08:56
*** Guest17639 has quit IRC08:58
openstackgerritEamonn O'Toole proposed a change to openstack/swift: Fix swift-recon --auditor bug  https://review.openstack.org/10019209:01
*** nacim has joined #openstack-swift09:06
*** charz has joined #openstack-swift09:09
*** nacim has quit IRC09:10
openstackgerritZhang Hua proposed a change to openstack/swift: Add distributed tracing capablities in logging.  https://review.openstack.org/9367709:12
openstackgerritEamonn O'Toole proposed a change to openstack/swift: Fix swift-recon --auditor bug  https://review.openstack.org/10019209:12
*** JelleB has quit IRC09:16
*** AbyssOne has quit IRC09:16
*** nacim has joined #openstack-swift09:23
*** kashyapk_ has quit IRC09:25
*** kashyapk has joined #openstack-swift09:26
hugokuocharz:09:26
*** itarchitectkev_ has joined #openstack-swift09:26
*** itarchitectkev has quit IRC09:29
*** kashyapk has quit IRC09:30
*** kashyapk has joined #openstack-swift09:30
*** davidhadas_ has quit IRC09:32
*** ppai has quit IRC09:34
*** Midnightmyth has quit IRC09:46
*** luisbg has quit IRC09:50
openstackgerritOpenStack Proposal Bot proposed a change to openstack/python-swiftclient: Updated from global requirements  https://review.openstack.org/8925009:52
openstackgerritOpenStack Proposal Bot proposed a change to openstack/swift: Updated from global requirements  https://review.openstack.org/8873609:53
*** a1|away has joined #openstack-swift09:54
*** a1|away is now known as AbyssOne09:54
*** itarchitectkev_ has quit IRC09:56
*** itarchitectkev_ has joined #openstack-swift09:58
*** madhuri has joined #openstack-swift09:58
madhuriHi, can anyone tell me where does the actual file content is read in case of object GET?09:59
madhuriMainly, the code part09:59
*** AbyssOne_ has joined #openstack-swift09:59
*** ondergetekende has quit IRC10:05
*** openstackgerrit has quit IRC10:06
*** ppai has joined #openstack-swift10:06
*** openstackgerrit has joined #openstack-swift10:06
*** davidhadas has joined #openstack-swift10:09
*** Midnightmyth has joined #openstack-swift10:09
*** haomaiwa_ has joined #openstack-swift10:13
*** kashyapk has quit IRC10:14
acolesmadhuri: DiskFileReader class - https://github.com/openstack/swift/blob/master/swift/obj/diskfile.py#L74710:15
madhuriacoles: thanks.But where does this __iter__ gets called?10:17
*** charz has quit IRC10:21
*** haomaiwa_ has quit IRC10:23
*** haomaiwang has joined #openstack-swift10:23
*** matsuhashi has quit IRC10:24
*** matsuhashi has joined #openstack-swift10:24
portantemadhuri: it is invoked by the wsgi protocol itself10:24
acolesmadhuri: it gets set in the response here https://github.com/openstack/swift/blob/master/swift/obj/server.py#L498. the iter will then be called by the wsgi server.10:24
*** ondergetekende has joined #openstack-swift10:25
portantethe iterator is invoked when the wsgi code (in eventlet) starts to construct the body10:25
*** haomaiw__ has joined #openstack-swift10:25
*** kashyapk has joined #openstack-swift10:26
*** haomaiwang has quit IRC10:28
madhuriportante: Thanks a lot10:30
madhuriacoles: Thanks a lot. Got it10:30
*** scohen has quit IRC10:31
*** matsuhashi has quit IRC10:33
*** matsuhashi has joined #openstack-swift10:34
*** openstackgerrit has quit IRC10:35
*** openstackgerrit has joined #openstack-swift10:35
*** uksysadmin has joined #openstack-swift10:39
*** matsuhashi has quit IRC10:41
*** matsuhashi has joined #openstack-swift10:41
*** itarchitectkev_ has quit IRC10:42
*** tdasilva has joined #openstack-swift10:57
*** nshaikh has joined #openstack-swift10:58
*** mmcardle has quit IRC11:00
*** davidhadas___ has joined #openstack-swift11:03
*** ppai has quit IRC11:03
*** scohen has joined #openstack-swift11:03
*** mmcardle has joined #openstack-swift11:04
*** davidhadas has quit IRC11:06
*** kashyapk is now known as kashyap_11:08
*** kashyap_ is now known as kashyapk11:08
*** nosnos has quit IRC11:15
*** ppai has joined #openstack-swift11:23
*** kashyapk is now known as kopparam11:25
*** kopparam is now known as kashyapk11:26
*** kashyapk is now known as kopparam11:27
*** matsuhashi has quit IRC11:29
*** matsuhashi has joined #openstack-swift11:29
*** tdasilva has left #openstack-swift11:29
*** matsuhashi has quit IRC11:33
*** mmcardle has quit IRC11:45
*** dmorita has quit IRC11:46
*** Trixboxer has joined #openstack-swift11:54
*** mmcardle has joined #openstack-swift12:10
*** tdasilva has joined #openstack-swift12:14
*** wer has joined #openstack-swift12:15
*** davidhadas___ has quit IRC12:24
*** ppai has quit IRC12:24
*** scohen has quit IRC12:25
*** miqui has joined #openstack-swift12:33
openstackgerritA change was merged to openstack/swift: Remove unused variable  https://review.openstack.org/9728712:42
*** scohen has joined #openstack-swift12:42
*** zhiyan_ is now known as zhiyan12:44
*** kopparam has quit IRC12:44
*** mmcardle has quit IRC12:57
*** haomaiw__ has quit IRC12:58
*** nshaikh has quit IRC13:00
*** sandywalsh has quit IRC13:01
*** ujjain has joined #openstack-swift13:02
*** ujjain has quit IRC13:04
*** ujjain has joined #openstack-swift13:04
*** sandywalsh has joined #openstack-swift13:04
*** bkopilov_ has quit IRC13:06
*** dmsimard_away is now known as dmsimard13:27
acoleschmouel: if you have a moment could you look at this one again https://review.openstack.org/#/c/91788/ , now swiftclient tests are passing again :) thx13:28
chmouelacoles: will do thanks, unfortunately prob not have time before wed tho13:30
acoleschmouel: ok. thanks.13:30
openstackgerritEamonn O'Toole proposed a change to openstack/swift: Parallel object auditor  https://review.openstack.org/5977813:30
*** kopparam has joined #openstack-swift13:40
*** psharma has quit IRC13:48
*** kopparam has quit IRC13:49
*** kopparam has joined #openstack-swift13:50
*** uksysadmin has quit IRC13:55
*** uksysadmin has joined #openstack-swift13:56
*** nsquare has joined #openstack-swift14:18
*** asdfsf has joined #openstack-swift14:23
*** itarchitectkev_ has joined #openstack-swift14:23
*** uksysadmin has quit IRC14:26
*** asdfsf has quit IRC14:31
*** mmcardle has joined #openstack-swift14:33
*** sts has joined #openstack-swift14:33
stshello folks. Is it possible to prevent a user from creating new containers in SWIFT?14:34
*** chandan_kumar has joined #openstack-swift14:36
*** Kr4zy has joined #openstack-swift14:37
Kr4zyWhat is the recommended SSL terminator for performance and scalability? I heard Nginx, but have also heard of issues. Pound sounds like a good alternative, but heard that it doesn't scale.14:38
*** tdasilva has quit IRC14:38
*** tdasilva has joined #openstack-swift14:44
*** mmcardle has quit IRC14:44
otoolee@cschwede I've posted comments on your swift-recon fix, and I've also posted an alternative fix which retains the generator (I need the generator for the parallel object auditor patch) - see https://review.openstack.org/#/c/100192/214:44
*** lpabon has joined #openstack-swift14:45
*** bkopilov_ has joined #openstack-swift14:45
portanteKr4zy: haproxy is another, stud (https://github.com/bumptech/stud) is yet another14:46
portanteI believe RAX uses haproxy14:46
*** mmcardle has joined #openstack-swift14:46
notmynamegood morning world14:47
portantenotmyname: sts asked about container creation prevention for a given user above14:49
notmynamesounds like a non swift owner user14:49
*** lpabon has quit IRC14:51
*** foexle has quit IRC14:55
otooleecschwede: (repeat of above comment) I've posted comments on your swift-recon fix, and I've also posted an alternative fix which retains the generator (I need the generator for the parallel object auditor patch) - see https://review.openstack.org/#/c/100192/214:55
*** judd7 has joined #openstack-swift14:57
portantests, I looked at the code, and I don't immediately see a way to prevent a user from creating a container, but I could be missing something obvious14:59
Kr4zyportante: I am using HAProxy for Load balancing only on a dedicated server. I need SSL termination on the proxy servers. Is HAProxy or Stud the better of the bunch for SSL termination in terms of performance and scalability?15:00
creihtKr4zy: haproxy15:01
ahaleyeah we found newer dev releases of haproxy with ssl were better than stable + stud15:01
portantecreiht: do you run haproxy locally for each proxy server or front a few haproxy servers in front of the swift proxy servers?15:01
ahalewe have a bunch infront15:02
creihthowdy ahale :)15:02
portanteahale, pardon my ignorance, are you RAX?15:02
creihtportante: yeah he will know a lot more about that stuff than I will :)15:02
* portante guesses yes15:02
ahaleyeah15:02
ahaleops side rather than dev15:03
portanteahale: where the cool kids play15:03
notmynameFWIW we run pound locally on each proxy box behind the LB tier. but we have a task in the queue to move away from pound to haproxy15:03
creihtportante: if you come to the openstack paris summit, then you will likely get a chance to meet the amazing ahale :)15:03
creihtand his sidekick omame15:04
ahalehehe15:04
creiht:)15:04
portantecreiht: cool, trying to swing that, hopefully15:04
ahalewhen is that summit?15:04
portanteNov beg15:04
creihtthough, I'm not really sure which is the sidekick :)15:04
notmyname"The next OpenStack Summit will be held in Paris, France, November 3-7, 2014. "15:05
creihtahale: I guess I assumed you were going to go, will you and omame be able to make it?15:06
omamecreiht: he took off, be back in ~15'. as for myself, I'm leaning towards buying a train ticket and head there15:10
Kr4zycreiht: how does haproxy compare to nginx and pound in terms of performance and scalability? haproxy ssl was just introduced in the latest version (1.5); is it ready for production?15:11
creihtKr4zy: we've been running the dev versions for quite a while, and like ahale said, they have been more stable then the previous release15:12
creihtomame: cool15:12
creihtKr4zy: and haproxy continues to beat the others in performance15:13
creihtI also would generally not reccomend nginx for swift, as it spools all requests15:14
Kr4zycreiht: thanks for the info :)15:15
*** Midnightmyth has quit IRC15:16
*** gholt has joined #openstack-swift15:19
*** ChanServ sets mode: +v gholt15:19
*** zul has quit IRC15:22
*** zul has joined #openstack-swift15:23
notmynameok, IRC buffers read. email handled. coffee in mug. time to dive in for the day.15:24
notmynamefirst up is figuring out where we are with the SP patch chain15:25
notmynamelooks like the non-SP patches that were reviewed have landed15:26
notmynamewhich leaves just the SP patches15:26
notmynamehow do we feel about them?15:26
portantepretty good, more reviews coming on my part, gonna add test environment where in-memory object server is setup as a another storage policy by default15:32
* portante heads out for a ride, back after 2 PM EDT15:33
notmynameI'd like to see the SP patches approved today so that we can get the mechanics working for merging it (new feature branch, force push the chain, gate the merge commit).15:36
serverascodeHi All, another question...we have a small two region swift cluster, when we upload larger objects they fail to replicate due to a rsync timeout of 30 seconds...anyone have any suggestions as to how to choose a timeout value?15:37
serverascodeie. the timeout for object replicator, I think15:38
*** gholt has quit IRC15:38
*** kopparam has quit IRC15:41
*** zhiyan is now known as zhiyan_15:52
creihtheh15:56
creihthttp://blog.zhaw.ch/icclab/evaluating-the-performance-of-ceph-and-swift-for-object-storage-on-small-clusters/15:56
creihtwhile noting some of the things that I reported, he didn't even mention the biggest thing which was the fact that his performance numbers didn't add up for swift :/15:57
ahalecreiht: yeah i think we plan to go to paris - I've been kinda grumpy that I haven't been allowed since boston16:01
*** nacim has quit IRC16:01
*** scohen has quit IRC16:04
*** scohen has joined #openstack-swift16:05
*** Midnightmyth has joined #openstack-swift16:08
*** mwstorer has joined #openstack-swift16:09
*** scohen has quit IRC16:09
*** jergerber has joined #openstack-swift16:12
*** kevinc_ has joined #openstack-swift16:14
*** gyee has joined #openstack-swift16:26
*** Kr4zy has quit IRC16:29
goodesgood morning all16:30
*** Ju_ has quit IRC16:31
notmynamegoodes: good morning16:32
*** nsquare has quit IRC16:33
*** itarchitectkev_ has quit IRC16:35
goodesnoob question - when you see > name: hello there   - is that somene typing ‘name: hello’ or is that an IRC command that you use to reference the person?16:36
notmynamegoodes: that's what I'm typing to specifically address you. many IRC clients have some sort of alert associated with your current user name. this means that eg I can pick out stuff addressed specifically to me in a busy channel16:37
notmynamegoodes: all IRC commends start with a "/" and aren't generally displayed here in the normal chat16:38
notmynameexcept for /me. for example:16:38
* notmyname uses /me to show goodes16:38
goodesbut do you just type my name then semicolon or is that being done my an irc command ie /cmd handle hello there?16:38
notmynamegoodes: well, my IRC client has a "tab-to-autocomplete" your nick, so I type "g" and hit tab and it expands to "goodes: "16:39
goodesnotmyname: thanks, obvious now that you said it16:39
*** kopparam has joined #openstack-swift16:43
notmynamegoodes: I think you were in here this weekend asking some questions. this channel is more busy during the week (most of the active people are in north america, although we have a few in the EU and Asia/Australia)16:43
notmynamegoodes: feel free to ask questions, and someone will probably answer or at least point you to some docs16:43
*** itarchitectkev_ has joined #openstack-swift16:44
*** scohen has joined #openstack-swift16:44
notmynameacoles: chmouel: cschwede: may already be too late for you guys, but I'd love to have a go/no-go from you on the SP patches before your end-of-day today, please16:47
*** kopparam has quit IRC16:50
*** mmcardle has quit IRC16:55
*** scohen has quit IRC16:58
*** scohen has joined #openstack-swift16:59
*** lpabon has joined #openstack-swift17:13
acolesclayg: where are at with the timestamp offset format - current patch is still 8 digits, are you going to bump this up to 12?17:14
acolesnotmyname: working on it...17:14
*** nsquare has joined #openstack-swift17:23
*** jamiehannaford has quit IRC17:26
*** mkollaro has quit IRC17:29
*** mkollaro has joined #openstack-swift17:31
*** nsquare has quit IRC17:32
*** nsquare has joined #openstack-swift17:36
*** ChanServ changes topic to "Storage Policies have been proposed to master. Soft freeze in effect until these patches land | First SP patch: https://review.openstack.org/#/c/96026/ | Last SP patch: https://review.openstack.org/#/c/96049/"17:40
*** zaitcev has joined #openstack-swift17:43
*** ChanServ sets mode: +v zaitcev17:43
openstackgerritSamuel Merritt proposed a change to openstack/swift: Fix SLO test with old simplejson  https://review.openstack.org/9980517:44
*** annegentle has joined #openstack-swift17:46
*** Kr4zy has joined #openstack-swift17:48
Kr4zyanyone know why proxy-buffering on nginx is not recommended for Swift?17:50
notmynameKr4zy: you don't want 10 concurrent users uploading a 5GB all at the same time17:51
notmynameor 100017:51
creihtKr4zy: it works fairly well for small objects, but as your objects get larger, it starts to fall apart17:51
creihtand what notmyname said :)17:51
Kr4zyOkay17:51
Kr4zythanks17:51
Kr4zySince HAProxy is still in dev, is it ready for production? Anyone using it in production?17:52
Kr4zyThe latest 1.5 release17:52
creihtKr4zy: yeah we (rackspace) have been using it in production for quite a while17:52
creihtalmost a year I think17:52
creihtKr4zy: of course, we are just some random dudes on the internet :)17:54
Kr4zylol ;)17:54
torgomaticKr4zy: also horrible timeouts; your client sends the last byte of a 5GB object, then nginx starts to send stuff to the proxy... this takes some time, during which your client decides the server is dead and times out the connection17:55
creihtthe above was more in reference to the nginx comments17:55
torgomaticalso also instead of spreading the IO load around all your big, fat, disk-stuffed object servers, it concentrates things on your poor /tmp fs on your proxies17:55
creihtyup17:55
torgomaticbasically nothing good happens with nginx in front of swift17:56
*** lpabon_ has joined #openstack-swift17:56
*** lpabon_ has quit IRC17:56
*** lpabon has quit IRC17:57
creihtotherwise, nginx is really awesome :)17:57
zaitcevHAProxy does not terminate SSL, however, so it's only useful for large installations where you are already in hw termination zone. For small deployments I think Pound is still appropriate.17:58
creihtzaitcev: it does now17:59
*** lpabon has joined #openstack-swift17:59
*** lpabon has quit IRC17:59
notmynamezaitcev: and pound prevents the use of expect: 100-continue17:59
*** haomaiwang has joined #openstack-swift18:02
*** lpabon has joined #openstack-swift18:02
openstackgerritJoel Wright proposed a change to openstack/python-swiftclient: Add importable SwiftService incorporating shell.py logic  https://review.openstack.org/8545318:09
*** joeljwright has joined #openstack-swift18:16
*** Kbee has joined #openstack-swift18:38
*** zaitcev is now known as zaitcev_tooth18:38
KbeeHello, while doing a PUT, i'm getting 503 error - with the following logged in syslog in storage node18:38
KbeeERROR __call__ error with PUT /sdb1/15/AUTH_2a5b67092df94b19a9e33d6dcd2a3ad7/cont : #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/dist-packages/swift/account/server.py", line 284, in __call__#012    res = method(req)#012  File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 2217, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 837, in _timing_stat18:38
Kbees#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.7/dist-packages/swift/account/server.py", line 125, in PUT#012    broker.put_container(container, req.headers['x-put-timestamp'],#012  File "/usr/lib/python2.7/dist-packages/swift/common/swob.py", line 219, in __getitem__#012    return self.environ[self._normalize(key)]#012KeyError: 'HTTP_X_PUT_TIMESTAMP' (txn: tx3975efcea54346fb9669b-00539f38c9)18:38
Kbeecan someone help in rootcausing this one ?18:39
Kr4zycreiht: what about pound? What's wrong with it?18:39
notmynameKr4zy: pound is ok, but it limits some of the HTTP functionality. there's really no reason to use pound now that HAProxy supports ssl termination and pipelined requests18:40
creihtKr4zy: it is reasonbly capable for a lot of things, but you will have a hard time saturating 10Gb ethernet with ssl18:40
*** kopparam has joined #openstack-swift18:40
*** nsquare has quit IRC18:41
notmynameKr4zy: and both pound and haproxy use openssl, so the engine is the same and you could expect similar performance from both as to ssl-termination18:41
*** scohen has quit IRC18:41
notmynamecreiht: right? ^18:41
creihtnotmyname: that isn't clear to me18:41
notmynameah18:41
Kbeesorry pasted content instead of gist- here is error message - http://paste.openstack.org/show/84200/18:42
creihtI know our guys had a hard time getting pound to perform reasonably a while back18:42
KbeeWhile doing a PUT, i'm getting 503 error - with the following logged in syslog in storage node - error -> http://paste.openstack.org/show/84200/18:42
notmynameKbee: grep your whole cluster's logs for tx3975efcea54346fb9669b-00539f38c9 (the transaction id for that request) and pastbin that18:42
*** scohen has joined #openstack-swift18:43
creihtKr4zy: but realistically the best thing to do would be to set up a cluster with each of the load balancers you would like to evaluate and benchmark them18:44
*** scohen has quit IRC18:44
*** scohen has joined #openstack-swift18:44
Kbeenotmyname,  pasted..18:44
Kr4zynotmyname: and creiht, if I want to use HAProxy as dedicated LBs and it as SSL terminators on the proxy servers, would I do a SSL passthrough on the LBs?18:44
creihtKr4zy: in theory that sounds right, though we do the ssl termination at the LB18:46
*** joeljwright has quit IRC18:46
Kr4zycreiht: I would too, but security requirement.18:46
creihtahh18:46
*** kopparam has quit IRC18:47
Kr4zycreiht: Since there's a layer in front of the proxy service, how would I handle health checks on the HAProxy?18:48
notmynameKr4zy: note that swift assumes a secure, private network within the cluster (ie proxy->storage communication or replication traffic isn't encrypted)18:48
notmynameKr4zy: you could do simple IP:port connect checks or you could still use the /healthcheck endpoint for each proxy18:48
creihtKr4zy: also, I'm just a dev that is parroting most of what I've heard our ops guys say over the years :)18:50
Kbeenotmyname: http://paste.openstack.org/show/84201/18:50
notmynamecreiht: lol. ditto :-)18:51
Kr4zynotmyname: right, our setup does put the Swift cluster in a private network. We encrypt only the communication between the proxies and the Swift clients. proxy -> storage is not encrypted. Security wants user credentials to be encrypted. For us, it is the Keystone token. We also configured the proxy services to listen on localhost and have only the SSL terminators publicly accessible. So we can't really run a health checks to t18:52
Kr4zynotmyname: at least I don't know how.18:52
Kr4zynotmyname: I am new to HAProxy btw.18:52
notmynameKr4zy: I too am new to HAProxy configs. but looking at the online docs it seems pretty straightforward. honestly all I can do is point you to docs someone else has written18:53
*** lpabon has quit IRC18:54
creihtKr4zy: also I think in a setup with ssl passthrough you lose the client source ip18:55
*** nsquare has joined #openstack-swift18:55
creihtKbee: sorry, we still need more context to try to help18:56
notmynamecreiht: but can't you tell haproxy to add the x-forwarded-for header with the client ip?18:56
Kr4zynotmyname: I currently have health checks to the proxy ssl terminators which works if I turn off the ssl terminator. The client will failover to the other HAProxy LB. But if I don't turn off the SSL terminator, but turns off the swift proxy service, HAProxy will still thinks the service is online. Clients connecting to it will hang and eventually fail.18:56
creihtnotmyname: ahh true18:57
Kbeecreiht: what other information is needed ?18:58
creihtKbee: it looks like that is just the logs for the one server18:58
creihtcan you grep the logs on all of your servers (proxy, account, container, object)18:59
*** mlipchuk has joined #openstack-swift18:59
creihtfor that txid, that helps us to understand the request that was going through18:59
notmynameKr4zy: ya, I think the setup you describe loses the backpressure feature you get with healthcheck endpoints. that is, you've got the ssl terminator in the middle of the data path, so if you can't check the proxy service directly then you can't make intelligent decisions at the LB layer18:59
* notmyname grabs the popcorn and watches the mailing list thread about the openstack-ci gates throwing too many errors and running too many tests19:01
portanteclayg, acoles, notmyname, torgomatic: so now the end of the chain for SP is 96026, and the beginning is 96027?19:01
torgomaticportante: sounds about right19:01
notmynameportante: current end point is in the topic19:01
Kr4zynotmyname: yeah, any idea on going around this? btw, what are you guys using for identify service19:01
notmynameportante: wait. that doesn't sound right19:01
portantenotmyname: does not look right19:01
notmynamelooks like I fixed the topic with an old version of it19:02
* notmyname goes hunting19:02
acolesportante: notmyname: it is that way19:02
portanteacoles, torgomatic: thanks for confirming19:02
*** nsquare has quit IRC19:02
notmynameah yes, https://review.openstack.org/#/c/96026/ is now the very end (the docs)19:03
*** ChanServ changes topic to "Storage Policies have been proposed to master. Soft freeze in effect until these patches land | First SP patch: https://review.openstack.org/#/c/96027/ | Last SP patch: https://review.openstack.org/#/c/96026/"19:03
notmynamethere19:03
Kbeecreiht, notmyname  - updated with all logs - http://paste.openstack.org/show/84203/19:04
creihtnotmyname: but I thought nothing was broken, it is the way things *must* be done :)19:04
Kr4zynotmyname: also, there aren't any concern about having the user credentials being pass in plain text between the LBs and the proxy servers in a private network? Does this comply with the Openstack security guide?19:04
* notmyname invites creiht to share his popcorn19:04
creihtnotmyname: heh19:05
creihtI haven't read that thread since last week19:05
acolesportante: also, 96031 has morphed to 99979, but the content is pretty similar19:06
creihtKbee: hrm, do you know what version of swift you are running?19:06
*** nsquare has joined #openstack-swift19:06
portanteacoles: thanks19:07
*** mmcardle has joined #openstack-swift19:07
notmynameKbee: creiht: that is a really weird error. why would the timestamp not be in the environ?19:07
torgomaticnotmyname: well, if race conditions are slipping in due to random chance, then clearly what we need is a deterministic race-condition detector19:07
creihtnotmyname: yeah that's what I'm wondering19:07
notmynametorgomatic: does it have to also be correct? I can deterministically say "yes, there is a race" for every question ;-)19:08
torgomaticnotmyname: might help if it were correct ;)19:08
Kbeecreiht, notmyname swift 2.0.3, icehouse bits ubuntu19:08
notmynameKbee: that's the client version. not the server version19:09
notmynameKbee: I guess "ubuntu icehouse" is the server-side19:09
notmynameI guess they packaged 1.13.1 correctly?19:09
Kbeenotmyname, correct..19:09
notmynameKbee: can you do `curl -i <swift domain>/info`? that should have the swift version number in it19:10
Kbeenotmyname,  whats <swift domain> ??19:12
notmynameKbee: whatever hostname/IP you're using to talk to your swift cluster. looks like you are using "192.168.33.20:8080". I didn't put that in earlier because I was getting a temporary error from the pastbin19:13
* notmyname goes to eat lunch19:14
*** scohen has quit IRC19:14
*** scohen has joined #openstack-swift19:14
Kbeenotmyname, "version": "1.13.1"19:15
*** mmcardle has quit IRC19:18
*** mmcardle has joined #openstack-swift19:18
*** joeljwright has joined #openstack-swift19:23
*** kevinc_ has quit IRC19:24
*** leews has quit IRC19:25
*** miqui has quit IRC19:25
*** miqui has joined #openstack-swift19:26
*** joeljwright has quit IRC19:26
*** joeljwright has joined #openstack-swift19:29
*** Kbee has quit IRC19:36
*** Kbee has joined #openstack-swift19:36
Kbeenotmyname:  "version": "1.13.1"19:45
*** kopparam has joined #openstack-swift19:45
*** itarchitectkev_ has quit IRC19:46
*** annegentle has quit IRC19:48
*** kopparam has quit IRC19:50
*** haomaiwang has quit IRC19:52
notmyname /mack19:56
notmynameor "back", as the case may be19:57
notmynamemy 4 year-old just invented the word "frizzed" as in the past-tense of throwing something like a frisbee. eg "yesterday, I frizzed the plate across the room"19:59
*** kevinc_ has joined #openstack-swift20:01
*** mkollaro has quit IRC20:01
portantecool20:02
Kbeenotmyname:  "version": "1.13.1"20:03
*** jamiehannaford has joined #openstack-swift20:13
portanteclayg, peluse_, others: a gist to bring coverage of storage policies module to 100%: https://gist.github.com/portante/6a7aecc635731f1a905b20:13
portanteI can submit that after this lands, if you want.20:13
notmynamecool20:14
*** scohen_ has joined #openstack-swift20:14
*** scohen has quit IRC20:14
*** Midnightmyth has quit IRC20:14
Kbeenotmyname, any leads ?20:14
*** annegentle_ has joined #openstack-swift20:16
*** mmcardle has quit IRC20:17
*** mlipchuk has quit IRC20:20
peluse_portante:  awesome, thanks!20:23
*** zaitcev_tooth is now known as zaitcev20:23
*** Kbee has quit IRC20:26
*** CaioBrentano has joined #openstack-swift20:27
*** mmcardle has joined #openstack-swift20:34
notmynamettx: I'd like to move our 1:1 earlier this week. we're normally scheduled at 8:30 (pacific). I'd like to do it sometime between 6am and 8am20:37
*** scohen_ has quit IRC20:37
*** foexle has joined #openstack-swift20:43
*** annegentle_ has quit IRC20:47
*** Midnightmyth has joined #openstack-swift20:49
notmynameand I just realized that was supposed to go into the IRC window just above this one.20:52
portante;)20:53
peluse_i figured it was the drugs still20:54
notmynameI wonder how long I'll be able to blame stuff on that ;-)20:55
peluse_milk it for as long as you can :)20:58
*** mmcardle1 has joined #openstack-swift21:08
*** joeljwright has quit IRC21:08
*** joeljwright has joined #openstack-swift21:08
*** mmcardle has quit IRC21:10
*** foexle has quit IRC21:10
*** Trixboxer has quit IRC21:13
*** jamiehannaford has quit IRC21:16
notmynamelooks like there's been lots of reviews on the current SP patches. but none yet really from RAX. creiht, dfg, glange, redbo, [gholt]21:26
notmynameaside from a few small things, the parts that have been reviewed are trending towards "this looks good", from what I can tell21:26
*** Midnightmyth has quit IRC21:35
*** kevinc_ has quit IRC21:40
*** kevinc_ has joined #openstack-swift21:43
*** dmsimard is now known as dmsimard_away21:48
*** kopparam has joined #openstack-swift21:49
scotticusnotmyname: really?21:49
scotticusdidn'21:49
scotticusdidn't redbo and creiht go to the hackathon?21:49
scotticusand gholt has been here thoroughly going over the patches21:49
notmynamescotticus: yes. I only mean on the reviews in gerrit. they were both there and provided valuable feedback and insight21:49
scotticusdfg is on vacation...21:50
scotticusthey aren't done yet.21:50
notmynamescotticus: I'm not saying y'all aren't doing anything. just that the current gerrit patches don't have any rackspace comments on them.21:50
scotticusgholt is still reviewing, afaik everyone else is too.  we're doing it :)21:51
notmynameawesome! :-)21:51
*** kopparam has quit IRC21:54
*** joeljwright has quit IRC21:59
*** foexle has joined #openstack-swift22:01
creihtnotmyname: yeah I'm mostly done... I know gholt was still poking at it22:28
notmynamecreiht: cool, thanks22:29
creihtthough I'm mostly done due to attrition22:35
*** jergerber has quit IRC22:35
notmynamecreiht: you're not the first person I've heard that from ;-)22:36
creihthehe22:36
*** dmsimard_away is now known as dmsimard22:49
*** Kr4zy has quit IRC22:53
mattoliverauMorning all!22:56
notmynamemattoliverau: hi!22:56
mattoliveraunotmyname: how you going? recovering well?22:57
notmynamemattoliverau: I'm alive. recovering pretty well I think :-)22:57
*** mkollaro has joined #openstack-swift23:10
*** dmsimard is now known as dmsimard_away23:11
goodesHow critical is it to use one email address of all of openstack work? Does it screw things up if you start changing emails (i.e. jobs)? Would it be best just to use your personal standard github email?23:12
torgomaticgoodes: that's what .mailmap is for23:12
goodestorgomatic: thanks23:14
mattoliveraunotmyname: being alive is a good start :P Glad to hear recovery is going well, if there anything you need then don't hesitate to ask.. my distance might be a hindrence, tho. But if it's Australian Tim Tams in the mail then consider it done ;) You might not know this, but in Atlanta when you got a cold/flu pack to Mikal, they were passed on to me later in the week when I was hit with the flu.. so23:18
mattoliverauthank you!!23:18
notmynamemattoliverau: nice!23:18
*** tdasilva has left #openstack-swift23:19
*** acoles has quit IRC23:20
*** mkollaro has quit IRC23:24
*** acoles has joined #openstack-swift23:44
*** ChanServ sets mode: +v acoles23:44
*** mwstorer has quit IRC23:48

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!