Thursday, 2015-06-18

*** pberis has joined #openstack-swift00:00
*** haomaiw__ has quit IRC00:04
*** wer_ has joined #openstack-swift00:04
*** haomaiwang has joined #openstack-swift00:05
*** thurloat_isgone is now known as thurloat00:08
*** pberis has quit IRC00:11
*** delatte has quit IRC00:15
*** delattec has quit IRC00:15
*** doxavore has quit IRC00:16
torgomaticnotmyname: so it looks like oslo.policy has a hard dependency on oslo.config; not so much for configuration, but just in terms of import lines00:17
torgomatichowever, based on a quick reading of the code, it seems like there's just some missing plumbing for the policy path in Enforcer.__init__00:18
*** gyee has quit IRC00:23
*** kota_ has joined #openstack-swift00:26
*** ChanServ sets mode: +v kota_00:26
*** haomaiwang has quit IRC00:29
*** MooingLemur has quit IRC00:30
*** zhill has quit IRC00:31
*** thurloat is now known as thurloat_isgone00:31
*** dmorita_ has joined #openstack-swift00:34
*** jamielennox|away is now known as jamielennox00:35
*** jasondotstar has quit IRC00:39
*** cdelatte has joined #openstack-swift00:39
*** delattec has joined #openstack-swift00:39
tdasilvaredbo: re current expiring object container getting hammered. doesn't that patch to shard containers help with that?00:46
*** cdelatte has quit IRC00:49
*** delattec has quit IRC00:50
*** cdelatte has joined #openstack-swift00:57
*** delattec has joined #openstack-swift00:57
*** jamielennox is now known as jamielennox|away01:00
*** lastops has joined #openstack-swift01:05
*** jamielennox|away is now known as jamielennox01:08
*** haomaiw__ has joined #openstack-swift01:09
*** lastops has quit IRC01:10
*** pgbridge has joined #openstack-swift01:13
*** haomaiw__ has quit IRC01:15
*** janonymous__ has quit IRC01:18
*** proteusguy has joined #openstack-swift01:21
*** Kennan has quit IRC01:22
*** Kennan has joined #openstack-swift01:23
*** jrichli has joined #openstack-swift01:54
redboI guess you could try to balance your expiring objects container interval against what size containers are sharded at and how fast a container can process DELETEs, but we could probably just redesign it to be less goofy and centralized.01:56
redbolike if a container can process a million deletes an hour and they het sharded at 5 million you have to set the interval to 20 minutes02:10
redbos/het/get02:11
*** kcaj has joined #openstack-swift02:13
*** asettle is now known as asettle-afk02:17
*** haomaiwang has joined #openstack-swift02:17
zaitcevprobably02:19
*** jamielennox is now known as jamielennox|away02:20
*** zaitcev has quit IRC02:20
*** bkopilov has quit IRC02:25
*** jamielennox|away is now known as jamielennox03:03
*** pgbridge has quit IRC03:13
*** silor has joined #openstack-swift03:14
*** jamielennox is now known as jamielennox|away03:15
*** jamielennox|away is now known as jamielennox03:25
*** asettle-afk is now known as asettle03:30
jamielennoxdoes swiftclient support keystone v3 authentication?03:32
jamielennoxand i mean using the Connection object - not the CLI03:32
*** delattec has quit IRC03:37
*** cdelatte has quit IRC03:37
claygwho was working on lp bug #145255303:37
openstackLaunchpad bug 1452553 in OpenStack Object Storage (swift) "don't rebuild existing fragments" [Undecided,New] https://launchpad.net/bugs/1452553 - Assigned to Minwoo Bae (minwoob)03:37
claygminwoo - what's his handle?03:37
claygpeluse: you're gunna love this - i was telling him he should look at the patch I attached to the lp bug - and guess what - the patch had a bug in it03:39
claygi'm like 0/203:40
claygjamielennox: i think it does - but idk what all you have to fill into the os_options dict to make it work03:43
jrichliclayg: minwoob03:43
claygjrichli: thanks - but he's not in chan :\03:43
clayganyway - if anyone seems minwoob - can you tell him that patch I gave him was crap and I updated the bug :P03:44
jrichliI can tell him tomorrow morning :-)03:44
jrichlihe works with me03:44
jamielennoxclayg: it seems (and i say seems because i'm still trying to figure out the path from shell -> connection) like passing an os_options dict might work, but that whole pattern would indicate it's really only used by the CLI03:44
claygjamielennox: well the tricky bit from shell -> connection goes through service03:46
jamielennoxclayg: yep, is that expected to be something that other services would use?03:46
jamielennoxor a CLI only thing?03:46
jamielennoxfor context i need to look at how the swift backend for glance can be made to use v3 authentication03:48
jamielennoxand at the moment i'm tempted to just use requests directly :)03:48
claygjamielennox: but i'm not sure that's relevant unless you want a swiftservice instance - if you just want a connection it seems like os_options and auth_version should do the trick03:48
clayghey using requests directly is a great way to do it!03:49
claygthe SwiftServices thing is great if you can deal with it's opionated mannor - it's going to spin up some worker threads and manaage a bunch of stuff like outputting on it's own03:49
jamielennoxclayg: so i'm not on the swift or glance teams - is that a "swift official position" ?03:49
*** bkopilov has joined #openstack-swift03:49
jamielennoxlike i can do all this with requests and tell glance that this was recommended?03:50
claygyou can probably *override* the bits you want - but most of the time I think folks need something less opinonated - either the Connection class - or just roll your own over your favorite http library03:50
jamielennoxyea, i don't think i want to be dealing with the multithreaded aspects of the SwiftService from another service03:51
claygyeah so just use a connection - i mean it'll buy you the retry handling and stuff at least03:51
claygbut the keystone v3 stuff should be a non-issue - the connection class will just proxy your kwargs to keystoneclient03:52
jamielennoxok, just figure out the mix of where things need to go between regular params and os_options03:52
claygand while using a simple lowlevel http lib to talk *to swift* is probably reasonable - I doubt you attempt something like that for keystone03:52
jamielennoxclayg: well keystoneclient sessions buy me everything i need to talk to keystone already03:53
claygseems to me it's all in swiftclient.client03:53
jamielennoxwhich would leave me with pretty much an authenticated requests.Session i could use for swift03:53
jamielennoxincl. some retry handling03:53
claygjamielennox: so can you just use preauth token and preauth url?03:53
claygor subclass connection and swap out get_auth03:54
jamielennoxi could, it would mean i'd have to handle all the cacerts and stuff twice but that's not the end of the world03:55
jamielennoxsubclassing is not a bad idea, i'll need to look at that one03:55
openstackgerritKuo-tung Kao proposed openstack/swift: Use testtools for base test class  https://review.openstack.org/19296003:56
jamielennoxclayg: ok, that's a few leads for me to go on, thanks03:56
*** jrichli has quit IRC03:57
clayggl03:57
claygacoles_away: the 'project' thing is crazy - it's acctually an *attribute* of CONF objects - which also happen to override __getattr__ for missing attributes04:03
claygacoles_away: so if you don't initalize the conf object the attribute is missing and it will fall through to picking it up from the conf file - so crazy - total hack04:04
*** ppai has joined #openstack-swift04:35
*** nadeem has joined #openstack-swift04:36
*** nadeem has quit IRC04:37
*** nadeem has joined #openstack-swift04:37
*** SkyRocknRoll has joined #openstack-swift04:55
*** SkyRocknRoll has quit IRC05:00
*** SkyRocknRoll has joined #openstack-swift05:01
*** Kennan has quit IRC05:09
*** Kennan has joined #openstack-swift05:10
*** kota_ has quit IRC05:23
*** jamielennox is now known as jamielennox|away05:23
*** jamielennox|away is now known as jamielennox05:26
*** mmcardle has joined #openstack-swift05:31
*** mmcardle has quit IRC05:37
*** chlong has quit IRC05:38
*** ianbrown has quit IRC05:39
openstackgerritKuo-tung Kao proposed openstack/swift: Use testtools for base test class  https://review.openstack.org/19296005:44
*** SkyRocknRoll has quit IRC05:46
*** SkyRocknRoll has joined #openstack-swift05:48
*** ronenkat has quit IRC05:49
*** chlong has joined #openstack-swift05:53
*** Kennan has quit IRC05:58
*** Kennan has joined #openstack-swift06:01
*** Kennan has quit IRC06:06
*** Kennan has joined #openstack-swift06:06
*** pgbridge has joined #openstack-swift06:08
*** nadeem has quit IRC06:39
*** ianbrown has joined #openstack-swift06:46
*** krykowski has joined #openstack-swift06:59
openstackgerritKuo-tung Kao proposed openstack/swift: Use testtools for base test class  https://review.openstack.org/19296007:04
*** ianbrown has quit IRC07:06
*** ppai has quit IRC07:09
*** ppai has joined #openstack-swift07:14
*** ianbrown has joined #openstack-swift07:15
*** slavisa has joined #openstack-swift07:18
*** chlong has quit IRC07:18
*** jordanP has joined #openstack-swift07:34
*** proteusguy has quit IRC07:44
*** acoles_away is now known as acoles07:50
acolesgood morning07:53
*** joeljwright has joined #openstack-swift07:55
*** ChanServ sets mode: +v joeljwright07:55
*** ianbrown has quit IRC07:55
*** joeljwright has quit IRC07:57
*** geaaru has joined #openstack-swift07:58
acolesclayg: huh. so sounds like project could be whatever you put in conf file if you fail to init the object. if i understand you.07:59
*** joeljwright has joined #openstack-swift08:02
*** ChanServ sets mode: +v joeljwright08:02
*** joeljwright has quit IRC08:02
*** proteusguy has joined #openstack-swift08:06
hoacoles: morning!08:25
*** joeljwright has joined #openstack-swift08:37
*** ChanServ sets mode: +v joeljwright08:37
*** mmcardle has joined #openstack-swift08:39
acolesho: hello08:40
cschwedeGood Morning!08:41
acolesjoeljwright: i have a feeling that figuring out how to support keystone sessions is going to be significant for swiftclient going forwards http://lists.openstack.org/pipermail/openstack-dev/2015-June/067381.html08:41
*** mmcardle has quit IRC08:42
hocschwede: morning!08:49
*** mmcardle has joined #openstack-swift08:49
cschwedeho: hello!08:50
*** mmcardle has quit IRC09:12
*** mmcardle has joined #openstack-swift09:23
*** haomaiwang has quit IRC09:29
*** joeljwright has quit IRC09:34
*** slavisa has quit IRC09:34
*** aix has joined #openstack-swift09:37
*** joeljwright has joined #openstack-swift09:38
*** ChanServ sets mode: +v joeljwright09:38
*** jasondotstar has joined #openstack-swift09:52
*** mmcardle has quit IRC09:54
*** dmorita_ has quit IRC09:55
*** mmcardle has joined #openstack-swift10:00
joeljwrightacoles: Yeah, keystone sessions are on our list of things to looks at10:01
joeljwrightacoles: thanks for the link, will take a read of that later today10:02
acolesjoeljwright: i need to get up to speed on them. the historical impediment with swiftclient has been needing to support v1 auth and not wanting a hard dependency on keystoneclient10:03
acolesi had this hand-waving idea that maybe v1 auth could be encapsulated in a 'fake' session-like class so client then becomes session compatible10:04
* acoles stops waving hands10:04
joeljwrightacoles: I spent a bit of time talking with Tim in Vancouver about something like a fake session object (it might have been triggered by a comment made by you)10:05
acolesjoeljwright: did you trash the idea: :P10:05
joeljwrightI'm crazy busy with a software release until the end of June10:05
acoles?10:05
joeljwrightbut I want to look at that asap afterwards10:05
acolesme too.  :) back to the day job...10:06
joeljwrightyeah me too...10:06
*** SkyRocknRoll has quit IRC10:08
*** aix has quit IRC10:21
openstackgerritHisashi Osanai proposed openstack/swift-specs: Oslo config support in Swift  https://review.openstack.org/19209410:25
*** haomaiwa_ has joined #openstack-swift10:30
*** mmcardle has quit IRC10:33
*** joeljwright has quit IRC10:33
*** mmcardle has joined #openstack-swift10:37
*** ho has quit IRC10:38
*** joeljwright has joined #openstack-swift10:39
*** ChanServ sets mode: +v joeljwright10:39
*** CR7 has joined #openstack-swift10:43
*** SkyRocknRoll has joined #openstack-swift10:57
*** SkyRocknRoll has joined #openstack-swift10:57
*** silor has quit IRC10:59
*** mmcardle has quit IRC11:06
*** joeljwright has quit IRC11:06
*** ronenkat has joined #openstack-swift11:17
*** aix has joined #openstack-swift11:32
*** bkopilov has quit IRC11:34
*** jasondotstar has quit IRC11:46
*** SkyRocknRoll has quit IRC11:53
*** mmcardle has joined #openstack-swift12:00
*** joeljwright has joined #openstack-swift12:00
*** ChanServ sets mode: +v joeljwright12:00
*** csmart has joined #openstack-swift12:01
*** SkyRocknRoll has joined #openstack-swift12:08
*** jasondotstar has joined #openstack-swift12:10
*** kei_yama has quit IRC12:13
*** km has quit IRC12:13
*** jordan__ has joined #openstack-swift12:18
*** silor has joined #openstack-swift12:19
*** thurloat_isgone is now known as thurloat12:21
*** jordanP has quit IRC12:22
*** SkyRocknRoll has quit IRC12:26
*** jordan__ has quit IRC12:31
*** jordanP has joined #openstack-swift12:31
*** jasondotstar has quit IRC12:42
*** SkyRocknRoll has joined #openstack-swift12:42
*** jasondotstar has joined #openstack-swift12:55
*** petertr7_away is now known as petertr712:58
*** mordred has joined #openstack-swift13:03
*** bkopilov has joined #openstack-swift13:07
*** ppai has quit IRC13:07
*** CR7 has quit IRC13:09
*** amoturi has joined #openstack-swift13:09
*** cdelatte has joined #openstack-swift13:16
*** jasondotstar has quit IRC13:18
*** delattec has joined #openstack-swift13:26
*** cdelatte has quit IRC13:29
*** fifieldt_ is now known as fifieldt13:36
*** SkyRocknRoll has quit IRC13:36
*** jkugel has joined #openstack-swift13:37
*** NM has joined #openstack-swift13:44
*** eranrom has joined #openstack-swift13:57
*** jlhinson has joined #openstack-swift13:57
*** pgbridge has quit IRC13:58
eranromGot a question on user Vs. system metadata: Take for example x-container-read, is that a sysmeta in nature or user metadata in nature. On one hand the documnetation states that ACLs are examples of core swift features that should have been sysmeta and on the other they are set by users13:59
eranromand are not filtered out in responses13:59
*** blmartin has joined #openstack-swift14:02
*** blmartin has quit IRC14:09
*** acampbel11 has joined #openstack-swift14:11
*** barker has joined #openstack-swift14:12
*** jrichli has joined #openstack-swift14:16
*** wbhuber has joined #openstack-swift14:18
*** SkyRocknRoll has joined #openstack-swift14:19
*** SkyRocknRoll has joined #openstack-swift14:19
*** pgbridge has joined #openstack-swift14:20
kragnizjoeljwright: yeah, I spent a while looking at that gate-tempest-dsvm-neutron-src-python-swiftclient failure a couple of days ago14:26
kragnizjoeljwright: it's pretty odd14:26
joeljwrightthere's surely no reason for it14:28
joeljwrightall the patch does is rename some non-kwargs14:28
joeljwrightso it must be an infra problem14:29
joeljwrightbut all the other project dsvm neutron tests are ok14:29
*** jasondot_ has joined #openstack-swift14:32
*** blmartin has joined #openstack-swift14:34
blmartinGood morning all14:34
*** MooingLemur has joined #openstack-swift14:34
kragnizjoeljwright: yeah, it would be worrying if that actually broke things14:36
*** minwoob has joined #openstack-swift14:46
*** NM has quit IRC14:48
*** NM has joined #openstack-swift14:49
jrichliblmartin: good morning14:50
*** NM has quit IRC14:50
*** jasondot_ has quit IRC14:51
*** jasondotstar has joined #openstack-swift14:51
*** NM has joined #openstack-swift14:52
*** wbhuber has quit IRC14:52
*** zhill has joined #openstack-swift14:56
*** mmcardle1 has joined #openstack-swift14:56
*** mmcardle has quit IRC14:57
*** barker has quit IRC15:01
*** blmartin has quit IRC15:04
*** blmartin has joined #openstack-swift15:05
portanteso the swift guys are just working on a patch, if it has not been submitted already, https://review.openstack.org/184189, which up the number of workers per-disk to get higher levels of concurrenty15:07
portanteoh, hey swift guys15:07
portantewas err ... ah ... um ... just talking about you guys15:08
* portante goes and finds the right channel now ...15:08
*** haomaiw__ has joined #openstack-swift15:08
swifterdarrellportante: lol15:08
*** haomaiwa_ has quit IRC15:09
*** acampbel11 has quit IRC15:14
*** barker has joined #openstack-swift15:16
*** wbhuber has joined #openstack-swift15:23
*** janonymous_ has joined #openstack-swift15:27
janonymous_Hi , this might seem lame but could anybody please give me an idea about : pile = GreenAsyncPile(len(conns)); for response in pile: ...15:29
janonymous_This is a snippet of obj.py15:29
janonymous_how does it work when no response is in the pile ?15:30
*** NM has quit IRC15:31
*** gyee has joined #openstack-swift15:32
janonymous_say out of 5 response only 2 came , how  does the loop goes ? could somebody help on this15:33
*** Fin1te has joined #openstack-swift15:46
notmynamegood morning, world15:46
notmynameeranrom: as a first pass, sysmeta is x-object-sysmeta-*. and user meta is x-object-meta-* :-)15:47
*** haomaiwang has joined #openstack-swift15:48
notmynamejoeljwright: any progress on the swiftclient gate issue from yesterday?15:48
joeljwrightnot really15:49
joeljwrighton #openstack-qa discussing it now15:49
*** haomaiw__ has quit IRC15:49
pgbridgegood morning15:50
joeljwrightnotmyname: potentially related to bug 1463200 but "that one's still lacking in any significant analysis. as i said there could be more than one cause"15:50
openstackbug 1463200 in OpenStack Compute (nova) "check-tempest-dsvm-multinode-full fails due to "Failed to compute_task_migrate_server: No valid host was found"" [Undecided,New] https://launchpad.net/bugs/146320015:50
*** SpamapS has joined #openstack-swift15:52
SpamapSHI! I'm looking at a possible bug in swiftclient.service where some methods take options['header']and some take optoins['headers'] and it seems the actual implementation is a) untested and b) inconsistent. Before I get too deep, is there anybody here paying attention that might be familiar with this and able to shed light on why the distinction may be important? Thanks.15:53
SpamapSs/optoins/options/15:54
notmynameSpamapS: if joeljwright isn't available, I can help. (I just got online this morning and am going through emails, etc)15:55
joeljwrightSpamapS: I'm here15:55
joeljwrightSpamapS: If you point me at the locations I can have a look and tell you if there's anything important there (or a bug)15:56
SpamapSjoeljwright: https://git.openstack.org/cgit/openstack/python-swiftclient/tree/swiftclient/service.py#n168015:57
SpamapSjoeljwright: I believe that should be 'headers'15:57
SpamapSjoeljwright: according to the docs for upload15:57
SpamapSjoeljwright: but download takes 'header'15:57
SpamapSjoeljwright: and we did just now find that 'headers' does not work on upload.15:58
SpamapSI'm working on a test that will break before I get too deep15:58
joeljwrightI can't find anywhere that it calls options['headers']15:59
joeljwrightbut I think I know why it's option['header']15:59
*** acampbell has joined #openstack-swift15:59
eranromnotmyname: yeah I got that. thanks. Just wondering if we are not missing CDMI's equivalent of 'data system metadata' which is a system metadata that can be manipulated by the user instructing the system how it should work (e.g. ACLs) the current definition of sysmeta is more of a metadta that cannot be manipulated by the user, but rather for internal usage. Anyway, its not very important15:59
joeljwrightthe argparse option in the CLI is '—header' which produces a list of key:value pairs16:00
SpamapSjoeljwright: https://git.openstack.org/cgit/openstack/python-swiftclient/tree/swiftclient/service.py#n120516:00
SpamapSjoeljwright: docs say 'headers'16:00
*** amoturi has quit IRC16:00
joeljwrightah, in that case it's a doc bug16:00
SpamapSjoeljwright: let me try 'header'16:01
joeljwrightprobably because I wrote that and subliminally agree with you :)16:01
*** jasondotstar is now known as jasondotstar|afk16:01
joeljwrightall the code usages are options['header'], so it's certainly a bug in the docstring16:01
notmynameeranrom: yeah, I'd say there's almost 3 classes of metadata: sysmeta that's hidden and set by the system (eg encryption or EC stuff); user meta where the user can do whatever (x-object-meta-*); and then "other" like content-type, etag, ACLs, etc that are more like first-class HTTP headers (or otherwise don't follow another naming pattern)16:02
notmynameeranrom: but the "other" category can sometimes be set by the user directly and sometimes indirectly. eg content-type is directly set. timestamp and etag are indirectly set16:02
eranromnoymyname: Agree, I guess this makes it 4 classes :-)16:03
SpamapSjoeljwright: cool, I'm testing using that, but I wanted to make sure I wasn't crazy. Thanks for confirming.16:03
SpamapSjoeljwright: should I push up a patch?16:03
joeljwrightSpamapS: yes please, a patch would be great16:04
*** jordanP has quit IRC16:04
notmynameeranrom: your 4th class is stuff liek the acls?16:04
openstackgerritClint 'SpamapS' Byrum proposed openstack/python-swiftclient: Fix docstring typo for SwiftService.upload  https://review.openstack.org/19318716:05
eranromnotmyname: exactly. 3rd is like etag, 4th ACLs16:05
joeljwrightSpamapS: thanks16:05
notmynameeranrom: with your container sync stuff, you're building a whitelist of headers to sync, right?16:06
*** haomaiw__ has joined #openstack-swift16:06
notmynameeranrom: is that set per container or in a config somewhere?16:06
SpamapSjoeljwright: Néni zač. :)16:07
*** haomaiwang has quit IRC16:07
notmynameSpamapS: joeljwright: approved16:07
eranromnotmyname: I started off thinking it should be in a config, but now leaning towards a per container definition, which brings me to that 4th class...16:07
notmynameeranrom: in order to filter the configured headers?16:08
eranromnotmyname: yep16:08
notmynameeranrom: I'd definitely want to see starting small for now and leaving out ACLs as something to sync16:09
eranromnotmyname: that is: the metadata that holds "whitlist" per container16:09
notmynameeranrom: right. if you let users set what to sync, then you have to filter that based on what is allowed to sync16:09
joeljwrightnotmyname: fastest swiftclient patch ever!16:09
notmynameeranrom: ie set(allowed headers) | set(requested headers) => set(headers that are synced)16:10
notmynameeranrom: or is that &? point is, the intersection of those sets :-)16:10
*** petertr7 is now known as petertr7_away16:10
notmynameintersection is | and union is & right?16:11
joeljwrightnotmyname: also a good chance to see if this gate-tempest-dsvm-neutron-src-python-swiftclient bug strikes again16:11
*** haomaiwa_ has joined #openstack-swift16:11
*** haomaiw__ has quit IRC16:11
notmynamejoeljwright: was there a patch that landed, or is it random?16:11
joeljwrightstill trying to work that out16:11
joeljwrightnotmyname: there are a no other swiftclient patches in a -verified state16:12
joeljwrightnotmyname: but I need to dig through them to see if people have been rechecking endlessly16:12
eranromnotmyname: yeah, |,& can be interpreted that way. Anyway, I was thinking more of a flag saying user metadata yes/no and a white list of additional headers, but am open to suggestions.16:12
notmynamejoeljwright: I didn't see many (any?) rechecks in my gerrit emails this morning16:12
notmynameeranrom: yeah, that might work, but I'd be worried about the categorization/naming. ie naming is hard and making that clear (what is "user metadata" and is that clear to a user/operator)?16:14
eranromnotmyname: good point. an alternative would be just a white list.16:14
notmynameeranrom: my first guess (and thus can likely be improved) is to have a whitelist with x-cotnaienr-meta-* pattern and a list of explicit headers16:14
notmynameeranrom: yeah. a whitelist with a * pattern for stuff we know is always syncable (x-container-meta-*)16:15
eranromnotmyname: agree.16:15
notmynameI've seen this pattern somewhere else in swift. with objects16:15
eranromwill look for this16:15
notmynameeranrom: I think it's the object server allowed_headers16:15
eranromok, thanks16:15
notmynameeranrom: eg you can send any header you want for an object PUT. but swift will only remember the x-object-meta-* headers and the headers listed in allowed_headers on the object server config16:16
eranromis there a container equivalent?16:16
eranromI can look it up...16:16
openstackgerritRomain LE DISEZ proposed openstack/swift: Close connections from SegmentedIterable on client disconnection  https://review.openstack.org/19319216:17
notmynameeranrom: IIRC, no16:17
eranromnotmyname: ok, perhaps its high time we have one :-)16:17
notmynameeranrom: seems like a thing independent of container sync syncing metadata :-)16:18
eranromnotmyname: agree.16:18
notmynameeranrom: personally, I wouldn't prioritize arbitrary container header setting as a thing. there's a lot of other stuff that I've heard a lot of end-user and operator requests for :-)16:19
*** aix has quit IRC16:20
eranromnotmyname: so you take back: "whitelist with x-cotnaienr-meta-* pattern and a list of explicit headers"16:20
notmynameeranrom: as stuff to sync or as stuff to set?16:21
acolesnotmyname: eranrom iirc there is some filtering of headers passed to container/account servers in the proxy controllers16:21
acolesnotmyname: eranrom take a look around swift/proxy/controllers/base.py transfer_headers()16:21
eranromnotmyname: I thought you were referring to a stuff to set, but I guess you meant stuff to sync16:22
eranromacoles: sure will do16:22
*** panbalag has joined #openstack-swift16:22
acolesits been a while since i looked at it, so i hope i'm not confusing matters16:22
*** openstackgerrit has quit IRC16:22
*** openstackgerrit has joined #openstack-swift16:23
acolesbut IIRC for object requests most headers get passed through to object server backend, where allowed_headers is applied, but for container and account the proxy controller filters more16:23
*** panbalag has quit IRC16:23
notmynametorgomatic: can you take a look at https://review.openstack.org/#/c/193192/ please?16:25
eranromacoles: Any chance you refer to pass_through_headers in controllers/container.py?16:26
acoleseranrom: yes, so there you see the container/account specific 'allowed' list16:27
eranromacoles: ok, getting back to the question of what container metadata to replicate as part of container sync. How about having a boolean flag at the container level saying replicate metadata yes/no if set to yes then the replcated metadata would be x-container-meta-* plus operators whilelist in container-sync.conf which can be a subset of the container 'allowed' list?16:29
*** gsilvis has quit IRC16:30
eranroms/replcated/replicated16:30
*** nadeem has joined #openstack-swift16:32
*** nadeem has quit IRC16:33
*** jasondotstar|afk is now known as jasondotstar16:33
*** nadeem has joined #openstack-swift16:33
acoleseranrom: maybe. you would need to have the client send that new container level flag but somewhere in swift you'll need to translate the client generated header to a x-container-sysmeta-* key so it gets persisted. Then container sync can query it when required.16:37
*** gsilvis has joined #openstack-swift16:37
acoleseranrom: i guess it could be a yes/no or a client generated whitelist. depends on use cases.16:37
*** NM has joined #openstack-swift16:40
eranromacoles: right. If I got notmyname, then he would like to start with a limited client configuration, e.g. no client generated whitlelist.16:40
*** ni638629 has joined #openstack-swift16:40
acoleseranrom: k, i will defer to whatever notmyname is saying, i was just being nosey and dropping a hint about the transfer_headers filter really :)16:41
eranromacoles: ok, thanks. Question: If I translate the client's header to x-container-sysmeta-* key, how can the client tell the state? My understanding is that sysmeta is not visible but perhaps I am wrong16:42
eranromaclose: state = whether this is a yes or a no to replicate container metadata16:42
acoleseranrom: you are correct. you would also need to translate the sysmeta value back to client header key in responses16:42
eranromacoles: ok gotcha, np16:43
*** ni638629 has left #openstack-swift16:43
acolessysmeta cannot be sent to/from client16:43
acolesbut it does get persisted for you!16:43
*** jasondotstar is now known as jasondotstar|afk16:44
eranromacoles: ok, is there a special naming convention for user defined metadata that is persisted as sysmeta (such as ACLs)16:45
*** jasondotstar|afk is now known as jasondotstar16:46
*** jasondotstar is now known as jasondotstar|afk16:46
eranromacoles: or as ACLs should have been...16:46
acoleseranrom: heh, yeah there is stuff that predates sysmeta that gets plumbed all the way through, sysmeta just saves you some of that plumbing.16:48
acoleseranrom: i would suggest you use a header with key stemmed X-Container-Sync-* since it relates to X-Container-Sync-To etc16:49
eranromacoles: good idea! will do.16:49
eranromaclose: notmyname: I gotta go, this was very helpful, thanks. I will update the bug and code accordingly.16:50
*** jasondotstar|afk is now known as jasondotstar16:53
*** jasondotstar is now known as jasondotstar|afk16:53
*** eranrom has left #openstack-swift16:55
*** NM has quit IRC17:01
*** geaaru has quit IRC17:02
*** krykowski has quit IRC17:03
*** jasondotstar|afk has quit IRC17:03
*** barker has quit IRC17:05
*** barker has joined #openstack-swift17:07
*** blmartin has quit IRC17:09
*** wbhuber has quit IRC17:11
*** NM has joined #openstack-swift17:14
*** petertr7_away is now known as petertr717:14
*** wbhuber has joined #openstack-swift17:16
*** jasondotstar has joined #openstack-swift17:20
*** jasondotstar is now known as jasondotstar|afk17:20
*** jasondotstar|afk is now known as jasondotstar17:20
*** rledisez has joined #openstack-swift17:21
*** acampbell has quit IRC17:21
*** barker has quit IRC17:22
openstackgerritpaul luse proposed openstack/swift: EC Ssync:  Update parms to include node and frag indices  https://review.openstack.org/19152117:25
rledisezswifterdarrell: hello darrell17:26
*** jasondotstar is now known as jasondotstar|afk17:26
rledisezswifterdarrell: i think i might need some help about writing unit test for https://review.openstack.org/#/c/193192/17:26
rledisezswifterdarrell: i don’t see how to start, because it would involve launching a wsgi server, and inside an app establishing a connection from SegmentedIterable17:27
*** barker has joined #openstack-swift17:27
rledisezswifterdarrell: is it doable inside a unit test? seems a bit « too much ». what do you think?17:27
*** mmcardle1 has quit IRC17:28
*** joeljwright has quit IRC17:28
*** jasondotstar|afk is now known as jasondotstar17:29
*** jlhinson has quit IRC17:29
*** barker has quit IRC17:31
*** barker has joined #openstack-swift17:34
notmynamerledisez: if you look at the existing unit tests for FakeApp (there are several), you'll see the way we've tackled that sort of thing in the past17:38
*** barker has quit IRC17:38
rlediseznotmyname: ok, i’ll check that. thx17:39
notmynamehmm... it is unfortunate that the string SegmentedIterable doesn't appear anywhere in the test/unit directory17:40
*** breitz has quit IRC17:42
*** breitz has joined #openstack-swift17:42
rledisezyeah, i grepped that and it’s not tested at all for now17:42
notmynamerledisez: it's indirectly tested by the dlo and slo range read tests17:44
*** jlhinson has joined #openstack-swift17:50
*** openstackgerrit has quit IRC17:50
*** openstackgerrit has joined #openstack-swift17:51
*** marzif has joined #openstack-swift17:52
*** acampbell has joined #openstack-swift17:53
*** pgbridge has quit IRC17:54
*** fthiagogv has joined #openstack-swift17:56
*** pgbridge has joined #openstack-swift17:56
torgomatichm, looks like there's no unit tests at all for the closing of the underlying app iters17:57
torgomaticI did this http://paste.openstack.org/show/AG0em7UM9ZHSmp3T5KjU/ and nothing failed17:57
notmynamehowever, it reports 99% coverage :-/18:02
torgomaticthose lines are the 1%18:06
*** jasondotstar is now known as jasondotstar|afk18:07
openstackgerritMerged openstack/swift: Use just IP, not port, when determining partition placement  https://review.openstack.org/19197018:10
*** acoles is now known as acoles_away18:11
*** jasondotstar|afk is now known as jasondotstar18:15
*** jasondotstar is now known as jasondotstar|afk18:15
*** ronenkat has quit IRC18:16
openstackgerritMerged openstack/python-swiftclient: Fix docstring typo for SwiftService.upload  https://review.openstack.org/19318718:19
*** petertr7 is now known as petertr7_away18:26
*** NM has quit IRC18:27
*** jasondotstar|afk has quit IRC18:27
*** NM has joined #openstack-swift18:31
openstackgerritMichael Barton proposed openstack/swift: go: document replication  https://review.openstack.org/19324418:33
notmynameredbo: thanks!18:36
*** marzif has quit IRC18:37
*** mariusv has joined #openstack-swift18:38
*** mariusv has quit IRC18:38
*** mariusv has joined #openstack-swift18:38
*** marzif has joined #openstack-swift18:40
*** lcurtis has joined #openstack-swift18:41
*** mariusv has quit IRC18:42
*** petertr7_away is now known as petertr718:42
*** mariusv has joined #openstack-swift18:43
*** mariusv has quit IRC18:43
*** mariusv has joined #openstack-swift18:43
*** mariusv has quit IRC18:43
*** NM has quit IRC18:47
*** marzif has quit IRC18:47
*** jasondot_ has joined #openstack-swift19:01
*** jasondot_ is now known as jasondotstar|afk19:07
*** wer_ has quit IRC19:07
*** acampbell has quit IRC19:08
*** blmartin has joined #openstack-swift19:10
*** uschreiber_ has joined #openstack-swift19:23
*** uschreiber_ has quit IRC19:25
*** jasondotstar|afk is now known as jasondot_19:27
*** jasondot_ is now known as jasondotstar|afk19:27
*** jasondotstar|afk has quit IRC19:27
*** acampbell has joined #openstack-swift19:31
*** zhill has quit IRC19:32
*** NM has joined #openstack-swift19:33
torgomaticoh man, this closing stuff is a total mess19:36
*** jasondotstar has joined #openstack-swift19:36
torgomaticthere's just tons of places where we don't close WSGI iterables19:36
rledisezactually, i’m looking at the server where i applyed my patch19:36
rledisezthe patch does not fix everything there is still leakage19:36
torgomaticheh, yeah, no kidding :|19:37
notmynameremember that one time we though "yeah, we handle sockets well now. no more socket leaks. yay"19:39
torgomaticsoftware is a cross between gardening and nailing jello to a tree19:43
openstackgerritDarrell Bishop proposed openstack/swift: Allow 1+ object-servers-per-disk deployment  https://review.openstack.org/18418919:44
*** jasondotstar is now known as jasondotstar|afk19:47
swifterdarrellnotmyname: clayg: torgomatic: tdasilva: mattoliverau: portante: kota: cschwede: rebased and ready to go ^^^^^^^^^^^^19:49
*** jasondotstar|afk is now known as jasondotstar19:52
openstackgerritMinwoo Bae proposed openstack/swift: EC Reconstructor: Do not reconstruct existing fragments.  https://review.openstack.org/19327919:53
*** delatte has joined #openstack-swift19:59
*** jlhinson has quit IRC20:01
*** zaitcev has joined #openstack-swift20:02
*** ChanServ sets mode: +v zaitcev20:02
claygi love this go community!20:02
claygI'm trying to figure out why I can't get hummingbird to show up when I 'godoc -http=:6060'20:02
claygfind this https://code.google.com/p/go/issues/detail?id=154020:02
*** delattec has quit IRC20:02
claygand the comment by russ "working-as-intended"20:03
clayg"if you top letting yourself get pushed around by FHS you will be happier"20:03
claygit's like "all those guys trying to make distros are idiots - just put the files where my tool wants them and it will be fine"20:03
*** jasondotstar is now known as jasondotstar|afk20:06
*** jasondotstar|afk is now known as jasondotstar20:06
claygi'm not sure i follow the subsequent discussion - but it *looks* like the debian package just carries the "make godoc work with symlinks" as a patch!?20:06
*** jasondotstar is now known as jasondotstar|afk20:06
claygthat's awesome20:06
openstackgerritMichael Barton proposed openstack/swift: go: restructure cmd/hummingbird.go  https://review.openstack.org/18893920:07
*** jasondotstar|afk is now known as jasondotstar20:08
swifterdarrellclayg: It's not their fault the rest of the world is doing it wrong...20:10
*** wbhuber_ has joined #openstack-swift20:10
*** wbhuber has quit IRC20:10
*** SkyRocknRoll has quit IRC20:12
zaitcevclayg: It is sometimes necessary to carry a fork. Usually it's avoided by distros because it's labor intensive.20:14
*** jlhinson has joined #openstack-swift20:22
*** delattec has joined #openstack-swift20:25
*** delatte has quit IRC20:28
*** aix has joined #openstack-swift20:33
openstackgerritSamuel Merritt proposed openstack/swift: Get better at closing WSGI iterables.  https://review.openstack.org/19330320:34
torgomaticrledisez: I've just proposed a patch that should plug more of the holes. If you have time, could you try it out and see if it works (and comment on the patch with your findings)?20:35
*** Fin1te has quit IRC20:35
*** silor has quit IRC20:37
rlediseztorgomatic: sure, i’ll give it a try20:37
*** marzif has joined #openstack-swift20:41
*** fthiagogv has quit IRC20:46
claygminwoob: hey did you run tests on patch 193279 - i think we're not doing a great job mocking response headers20:52
patchbotclayg: https://review.openstack.org/#/c/193279/20:52
minwoobclayg: Yeah, I was trying to get them to work.20:55
minwoobclayg: I'll keep digging at it.20:55
claygyeah with your change the mocked responses need to set the fragment index in the response - or otherwise they'll all be "None" and thye'll get skipped20:56
claygyou could make your test verify the header is not-None before it throws it away (letting the tests get off w/o having to set the header) - but that seems like cheating20:56
minwoobRight.20:58
*** Fin1te has joined #openstack-swift21:00
*** nadeem has quit IRC21:00
*** jamielennox is now known as jamielennox|away21:01
*** nadeem has joined #openstack-swift21:02
*** marzif has quit IRC21:03
*** acampbell has quit IRC21:07
claygminwoob: so this test might get you started -> https://gist.github.com/clayg/4aebb0a21589d485785e21:09
*** jamielennox|away is now known as jamielennox21:09
* notmyname is currently running benchmark tests for the async container updates patch21:09
claygit doesn't cover the case of duplicates - you need a seperate test for that - and it doesn't blow up on master because the jerasure backend doesn't have the same problem as isa_l21:09
claygnotmyname: doesn't the go server already do that?21:10
notmynameperhaps. I think so21:10
notmynamehttps://review.openstack.org/#/c/189080/21:11
*** petertr7 is now known as petertr7_away21:11
*** Fin1te has quit IRC21:18
minwoobclayg: Got it. Thank you!21:20
openstackgerritMerged openstack/swift: go: document replication  https://review.openstack.org/19324421:28
*** ianbrown has joined #openstack-swift21:29
*** jlhinson has quit IRC21:32
*** jlhinson has joined #openstack-swift21:33
*** rledisez has left #openstack-swift21:34
*** jrichli has quit IRC21:34
*** jlhinson has quit IRC21:42
*** blmartin has quit IRC21:45
notmynamebaseline run finished22:04
*** peluse has quit IRC22:15
*** minwoob has quit IRC22:18
*** jkugel has quit IRC22:20
mattoliverauMorning22:22
notmynameclayg: greenthread.GreenThread().wait() doesn't take any arguments :-(22:22
notmynamehi mattoliverau22:22
notmynameclayg: but if wait() blocks the reactor (at least for the current thread), then I can't wrap it in a Timeout either or I'll just get hub starvation, right?22:25
claygwat?22:27
notmynameI could do sawn_n and then eventlet.sleep(100), but that sounds pretty horrible22:27
claygwith Timeout(): gt.wait()22:27
notmynamethe timeout won't have a chance to fire until after wait() finishes22:27
claygif the gt blocks spawn or spawn_n won't make a difference22:27
notmyname(there's a little bit of a question there)22:27
claygnotmyname: heh - one of us is confused :D22:28
notmynameI vote for me :-)22:28
* clayg goes to hack22:28
notmynamemy first test shows I'm the confused one22:31
notmynamehttps://gist.github.com/notmyname/0004945577d4b3cd2855  finishes in about 2 seconds22:31
notmynameie showing that .wait() doesn't block the reactor until the method finishes22:32
claygthis works sort of like I expect - https://gist.github.com/clayg/1313818bbf96688d042722:33
claygnotmyname: lol22:33
claygnotmyname: only so many ways to spell it I guess22:33
notmyname:-)22:33
claygyou beat me tho :\22:33
notmynameI didn't worry with pesky usage strings ;-)22:33
clayganyway - I think it would acctually make the diff a little smaller because less test churn22:34
claygthey fact that we won't have to revert it after we release it is a bonus22:34
notmynameyeah, the thing that convinces me that you are right is the user expectation that normally the listings are right22:35
clayg*less test churn22:35
claygfucking users22:35
claygstorage would be easy if a) we didn't have to give the data back or b) no one used it22:35
notmynameI liked it whenyou said somethign earlier, but I had already started down this road and wanted to see what it looked like (so I wasn't ignoring you earlier)22:35
clayg:P22:35
claygI SHALL NOT BE IGNORED!!!!22:35
notmynamebut, FWIW, I'm currently runnign the patch that's in gerrit in my test. should be done in ~45 minutes22:36
claygsure np, you can tell me to stow it too22:36
notmynamebasically, 3600 seconds of zero-byte PUTs to 1 container (2 proxies, 2 storage servers, all spinning disks)22:36
*** wbhuber_ has quit IRC22:38
notmynamecurrent master was able to do, well... it's taking a while to report what it did22:39
notmyname95% latency of .441 with max latency of 5.56122:39
*** delattec has quit IRC22:45
zigoswifterdarrell: Any update on fixing the 1.0.7m release?22:45
swifterdarrellzigo: I heard that the upstream dudes were making a 1.0.8 that will be awesome22:46
*** jasondotstar has quit IRC22:46
zigoswifterdarrell: I don't need awesome, I just need something that works ! :/22:46
swifterdarrellzigo: best bet would be to get in contact with paul, tushar, or kevin directly22:47
zigoswifterdarrell: I only have the email address of Tushar, can you give me the others?22:48
swifterdarrellzigo: probalby not, but clayg surely can!22:48
swifterdarrellclayg: (nevermind, send emails via PM)22:50
*** ianbrown has quit IRC22:54
*** NM has quit IRC22:55
*** openstackgerrit has quit IRC22:55
*** tellesnobrega has quit IRC22:55
*** klrmn has quit IRC22:55
*** EmilienM has quit IRC22:55
*** dmsimard has quit IRC22:55
claygho: holy crap man - thanks for the responses in the review on patch 149930 - I don't think i understood quite a badly the olso.policy's register check pattern was fighting us :\22:56
patchbotclayg: https://review.openstack.org/#/c/149930/22:56
*** lcurtis has quit IRC22:57
*** km has joined #openstack-swift22:59
*** NM has joined #openstack-swift23:06
*** openstackgerrit has joined #openstack-swift23:06
*** tellesnobrega has joined #openstack-swift23:06
*** klrmn has joined #openstack-swift23:06
*** EmilienM has joined #openstack-swift23:06
*** dmsimard has joined #openstack-swift23:06
*** ianbrown has joined #openstack-swift23:06
*** marzif has joined #openstack-swift23:10
openstackgerritSamuel Merritt proposed openstack/swift: Get better at closing WSGI iterables.  https://review.openstack.org/19330323:12
*** nadeem has quit IRC23:18
*** kei_yama has joined #openstack-swift23:19
*** NM has quit IRC23:42
*** thurloat is now known as thurloat_isgone23:45
*** chlong has joined #openstack-swift23:47
*** ho has joined #openstack-swift23:47
hogood morning!23:53
torgomatichi23:55
hotorgomatic: hello!23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!