Saturday, 2016-10-08

*** amit213 has joined #openstack-swift00:30
*** janonymous has joined #openstack-swift00:32
*** tongli has quit IRC00:55
*** klrmn has quit IRC00:56
*** tqtran has quit IRC01:25
*** jaakkos has quit IRC01:43
*** jaakkos has joined #openstack-swift01:43
*** links has joined #openstack-swift01:56
*** tuan_luong has joined #openstack-swift02:10
*** Jeffrey4l has joined #openstack-swift02:39
*** Jeffrey4l_ has joined #openstack-swift02:43
*** tuan_luong has quit IRC02:45
*** Jeffrey4l_ has quit IRC03:45
*** links has quit IRC04:05
*** dmorita has quit IRC04:07
*** dmorita has joined #openstack-swift04:14
*** tqtran has joined #openstack-swift04:24
*** dmorita_ has joined #openstack-swift04:25
*** dmorita__ has joined #openstack-swift04:27
*** dmorita has quit IRC04:29
*** dmorita_ has quit IRC04:29
*** tqtran has quit IRC04:29
*** dmorita__ has quit IRC04:36
*** dmorita has joined #openstack-swift04:38
*** dmorita has quit IRC04:41
*** dmorita has joined #openstack-swift04:41
*** dmorita has quit IRC04:43
*** links has joined #openstack-swift05:14
*** ChubYann has quit IRC05:18
*** dmorita has joined #openstack-swift05:19
*** dmorita has quit IRC05:20
*** dmorita has joined #openstack-swift05:24
*** ChubYann has joined #openstack-swift05:32
*** dmorita has quit IRC05:32
*** dmorita has joined #openstack-swift05:36
*** dmorita_ has joined #openstack-swift05:39
*** dmorita__ has joined #openstack-swift05:41
*** dmorita has quit IRC05:41
*** dmorita has joined #openstack-swift05:44
*** dmorita_ has quit IRC05:45
*** dmorita has quit IRC05:46
*** dmorita has joined #openstack-swift05:46
*** dmorita__ has quit IRC05:47
*** dmorita has quit IRC05:48
*** dmorita_ has joined #openstack-swift05:48
*** mingyu has joined #openstack-swift05:52
*** klrmn has joined #openstack-swift05:53
*** dmorita_ has quit IRC05:54
*** dmorita has joined #openstack-swift05:56
*** klrmn has quit IRC05:56
*** psachin has joined #openstack-swift05:57
*** dmorita has quit IRC05:59
*** tqtran has joined #openstack-swift06:17
*** dmorita has joined #openstack-swift06:24
*** ukaynar has joined #openstack-swift06:25
*** dmorita has quit IRC06:29
*** tqtran has quit IRC06:30
*** ukaynar has quit IRC06:49
*** tesseract has joined #openstack-swift07:00
*** tesseract is now known as Guest3718407:01
*** tesseract- has joined #openstack-swift07:01
*** Guest37184 has quit IRC07:05
*** silor has joined #openstack-swift08:10
*** silor has quit IRC08:13
*** silor has joined #openstack-swift08:16
*** silor has quit IRC09:07
*** Trixboxer has quit IRC09:34
*** Trixboxer has joined #openstack-swift09:35
*** links has quit IRC10:13
*** dmorita has joined #openstack-swift10:21
*** dmorita has quit IRC10:25
*** ChubYann has quit IRC11:13
*** ChubYann has joined #openstack-swift11:14
*** silor has joined #openstack-swift12:10
*** mingyu has quit IRC12:13
*** dmorita has joined #openstack-swift12:37
*** dmorita has quit IRC12:41
*** silor has quit IRC13:13
*** mingyu has joined #openstack-swift13:47
*** mingyu has quit IRC14:02
*** mingyu has joined #openstack-swift14:03
*** mingyu has quit IRC14:22
*** SkyRocknRoll has joined #openstack-swift14:43
*** mingyu has joined #openstack-swift14:44
*** mingyu has quit IRC14:55
*** McMurlock1 has joined #openstack-swift15:24
*** tesseract- has quit IRC15:50
cnfhow can I see _every_ bucket that is in swift?16:22
cnfI made some before, but the users / project that created them are gone16:22
cnfnow I can't see (nor delete) them16:22
*** klrmn has joined #openstack-swift16:29
*** McMurlock1 has quit IRC16:31
*** dmorita has joined #openstack-swift16:38
*** dmorita has quit IRC16:42
*** silor has joined #openstack-swift16:48
*** silor has quit IRC16:52
*** psachin has quit IRC16:54
*** SkyRocknRoll has quit IRC16:58
*** mingyu has joined #openstack-swift17:25
*** mingyu has quit IRC17:29
cnfm0017:41
*** mingyu has joined #openstack-swift17:55
*** mingyu has quit IRC18:00
cnfso can anyone tell me how to see orphaned buckets?18:08
*** ukaynar has joined #openstack-swift18:20
*** ukaynar has quit IRC18:45
*** ukaynar has joined #openstack-swift18:48
cnfno one here, huh?18:52
notmynamecnf: nope18:52
cnfhmm18:52
cnfcan't figure out how to claim orphaned containers18:53
notmynameside effect of most people who work on openstack being paid to do so means less IRC coverage on the weekends18:53
cnfand I can't get swift3 working18:53
cnfyeah, or during the week at times when it is useful to me :/18:53
notmynameheh, we've got several people in the EU. should be some coverage then18:53
notmynamebut in answer to your question (then I need to run errands) is that the way to find the containers is to send a GET to the enclosing account18:54
notmynameyou'd have to do that for every account to get a list of every container18:54
cnfwell, they don't have an account associated with them18:54
cnfthat is why they are orphaned18:54
notmynameah18:54
notmynamehow'd you end up with that?18:55
cnfaccidentally wiped my keystone database18:55
notmynameoops18:55
cnfit's not a disaster, it's all testing still, but now I do want to figure out how to recover from it18:55
cnfbut I just can't find a way to do it18:56
cnfI have data on disk, and no way to access it18:56
cnfor even delete is18:56
notmynamethe answer is relatively straightforward, but definitely more in the "advanced" category18:57
notmynamethe simple answer is "walk the drives and see what's there". doing that efficiently and dealing with replicas is what's slightly more complicated18:57
notmynamebut there's some good news :-)18:57
cnfbut everything on there is hashed?18:58
notmynameif you have a utilization trackign system that is based on logs and walking drives (IMO the right way to do it) instead of polling based on eg keystone data, then you can find what you need18:59
cnfuhm18:59
notmynameand i know of such a system that already exists :-)18:59
notmynamehttps://github.com/notmyname/slogging18:59
notmynameit used to live in swift, then it was extracted and now it lives in my github repo18:59
cnfwhy was it extracted?19:00
cnfalso, I don't really understand how that is related, tbh19:00
notmynamebecause a utilization system isn't in scope for a storage engine, and we wanted to be able to release each independently as needed. not sure if we'd make the same decision today, but that's the call we made several years ago19:01
notmynameslogging will help because it crawls all the drives in the cluster and reports what it finds. that's the only way to find orphaned stuff19:01
cnfok19:01
notmynameit can find accounts and containers19:02
cnfhmm, that kinda sucks19:02
cnfany way to reclaim it? or can you only delete it?19:02
notmynamesure. as long as it's associated with some token int eh auth system, the data is perfectly ok19:02
cnfand how do you associate it?19:03
notmynameTBH, I don't know. it would be different for each auth system. I'd imagine you'd "just" add that account name to some keystone user19:03
notmynamecnf: unfortunately, I need to step away now19:03
cnfah, sure19:03
notmynamegood luck19:04
cnfthanks19:04
cnfi might try my swift3 luck some more19:04
cnfor give up for the day19:04
notmyname(of course, the alternative is delete everythign and start over, but that's not necessary. one of the good things about swift is being able to inspect the on-disk data and do something with it19:04
cnfyeah, I don't know how the "do something with it" bit works :P19:05
cnfright, I deleted everything19:38
cnfi got a bit of an idea looking at the sqlite databases...19:38
cnfI guess I could script some python to find hanging ones, and reattach them19:38
cnfbut that is an excersise for later19:39
cnfexersise19:39
cnfthingie19:39
*** mingyu has joined #openstack-swift19:41
*** mingyu has quit IRC19:45
*** ndk has quit IRC19:59
*** ndk_ has joined #openstack-swift20:00
ahaleyeah the "do something with it" bit can get terrifying sometimes, sometimes the craziest ideas you end up with are the best. but like notmyname implies, swift at least gives you that chance where you might otherwise be lost or dependant on vendors20:11
cnfuhu20:13
cnfi have put it off for a "learn later"20:13
cnfi need to get more comfortable with the rest of it, first20:13
cnfbut I did see it's all just sqlite, so that can't be too hard to script against20:13
cnfahale:  you don't have a working swift3 setup by any chance? :P20:14
cnftimburke: was helping me last time, but he needed to go20:14
ahaleafraid not, im rackspace so i only know the old stuff that we support20:14
*** nikivi has joined #openstack-swift20:14
cnfah, ok20:15
cnfit seems keystone needs some stuff enabled for it to work20:15
cnfbut I can not find any references to it20:15
ahaleyeah those are both things i now nothing at all about20:16
cnf^^;20:16
ahalethe trawling, inspecting on disk stuff is something im trying to work on at the moment for a problem i have in one of my clusters, not so much orphaned data, but some bad in another similar way20:21
cnfuhu20:21
ahaleits hard because it can feel overwhelming, but good because its normally kind of simple to script and deal with stuff programaticallly, and get things to a state where swift can get itself going again20:22
*** nikivi has quit IRC20:22
cnfuhu20:22
cnfopenstack in generally is rather overwhelming20:22
cnfbeen trying to get the hang of it for 3 weeks now20:22
cnfand I still barely know what I am doing :P20:22
ahalehehe yeah i have been doing it for 6 years and still have no idea outside the pure-swift world really20:23
cnfhaha20:23
*** ukaynar has quit IRC20:24
cnfeh, $currentemployer is willing to pay me my day rate to let me play with openstack20:24
cnfi'm not complaining :P20:24
*** nikivi has joined #openstack-swift20:24
ahalenice20:24
ahalei kind of wish i had had more exposure to the other parts, but being in a largish cloud, we have a lot of teams with very limited scope20:26
cnfuhu20:27
cnfwell, I think Swift is the one component that works stand-alone20:27
ahaleabsolutely20:27
cnfthough it could use a better auth backend apart from keystone for that20:27
ahaleyah, we run swauth and a custom kinda keystone thing, but thats somebodyelsesproblem20:28
cnfhaha20:29
cnfswauth is just a tad basic20:29
cnfthough not too bad20:30
cnfdoes need moar docs, though :P20:30
ahaleyeah , i mean we probably benefited from having the guy who wrote it on our team at the time ;) and only use it for internal stuff, storing logs, processing slogging type stuff20:31
cnfuhu20:32
ahalefrom being mostly swift - i kinda feel that some of the other openstack projects like keystone are sometimes a little over complicated- lots of moving parts20:32
cnfoh, you are not kidding...20:33
cnfand such a pain to debug, never any useful debug info20:33
cnfthough swift has that problem a bit, as well20:33
ahalei've heard that a couple of times about swift from people but havent really felt it myself20:34
ahalehave you got any examples of situations that were hard? cos whenever ive had chance to dev input stuff i normally think its ok20:34
cnfprobably because you already know it well20:35
cnfbut python stack traces don't really help when you don't know how the project works that well20:35
ahaledo you think its lack of meaningful error messages, or lack or docs on architecture basics and stuff20:36
cnfwell, both really20:36
cnfwell, no, lack of docs, is not entirely right20:36
cnfthere are SO many docs, they are all over the place20:37
ahalemeaningful docs for operators20:37
ahaleahh20:37
cnfand yes, it lacks meaningful error messages20:37
cnfI would like more docs that explain how things work20:37
cnfmind, this really isn't just swift, most other OS projects are much worse20:38
ahalethings like how a request flows throuh the different layers and wsgi apps, how things can go wrong?20:38
cnfyeah, that's a good example20:38
cnfwhen things go wrong, there is little help20:38
cnfand it doesn't have to be holding hands type of docs, but a "this is how the flow is, and this bit you can read up on here, and this here"20:39
ahaleyeah , holding your hand through everything would be impossible - but there does feel a lack of things to equip people with knowing where to start, how to break down a problem20:40
cnflike, for example20:40
cnfi have been looking at swift3 for DAYS now20:40
ahale(i have a few newish team members who are starting to take over looking at our prod stuff so this is really interestng to me)20:41
cnfi _think_ the problem might be that swift3 calls v2 of the api20:41
cnfwhich is depricated, and as such disabled20:41
cnfno idea if this is true, or where to "fix" this, or anything20:41
cnfbut that should not take days to figure out ^^;20:41
cnfwell, maybe i'm just slow, but still20:42
ahale:) yeah no that shouldnt take days - something in the stack should be emitting a clear error somewhere20:42
ahaleshould as in ideally should, not should as in "you did configs wrong"20:43
cnfyeah20:43
cnfso yeah, I find the documentation needed for debugging to rather slim20:43
ahaleits interesting that you mentioned both swift3 and also swauth as lacking on the docs, as they both used to be core swift code20:44
cnfhmm, so yeah20:44
cnfi'm wrong20:44
cnfit's not the api version20:44
cnfi think?20:44
cnfno occurance of s3token in the keystone code20:44
cnfbut it needs to respond to a call to /v2.0/s3token20:45
cnfo,O20:45
cnfi don't understand this20:45
ahaleoO that sounds super weird20:45
cnfyeah :/20:45
ahalelike keystone shouldnt need to know anything about anything s320:45
cnfwell, swift3 is calling keystone on /v2.0/s3token20:46
cnfi dno why20:46
cnfand I found an OLD post that say something about it20:46
ahalei'd have figured the swift3 middlware would handle all that and make keystone reqs back - and that you were probably on the right track assuming it was a version thing - that keystone would move faster than the out-of-tree s3 code20:46
ahalebut like i said i know nothing about that stuff :)20:46
cnfdamn, I lost the link20:46
*** ukaynar has joined #openstack-swift20:49
timburkecnf: on for a bit -- the keystone side is in https://github.com/openstack/keystone/blob/master/keystone/contrib/s3/core.py which gets exposed as a paste filter_factory named "s3_extension"20:50
cnfhey timburke20:50
cnfand that exposes /v2.0/s3token ?20:50
cnfand I have20:50
cnfsec20:50
timburkeso somewhere there should be a paste pipeline definition similar to what's going on in swift's proxy-server.conf20:50
cnfyeah, I do20:51
cnftimburke:  http://paste.openstack.org/show/585000/20:51
timburkeyeah; it can actually be exposed as either /v2.0/s3tokens or /v3/s3tokens, but swift3 hardcodes the v2.0 :(20:51
cnfit's not in my domain config files, though, is that needed?20:52
*** nikivi has quit IRC20:52
timburkenot sure; i'm not familiar enough with setting up keystone20:52
cnfhmm20:53
cnfI really don't understand this20:53
cnfkeystone just gives me a 404 when I hit /v2.0/s3token (or v3/s3token)20:53
timburke"s3token" or "s3tokens"?20:53
cnfneither work20:54
timburkeyou'll also want to make sure you're doing POST requests; from experimenting, GETs will 40420:54
cnfah20:54
cnfok, a POST gives me a 40020:55
timburkesomething like `curl -v http://keystone-server/v2.0/s3tokens -X POST` should respond with a 400 instead20:55
timburkeyeah, nice20:55
cnfINFO eventlet.wsgi.server [req-0cf736ea-9537-49f3-8037-64e5cbe81af7 - - - - -] 172.17.0.1,172.17.0.1 - - [08/Oct/2016 20:56:29] "POST /v2.0/s3tokens HTTP/1.1" 404 318 0.02301620:55
cnfis my keystone log20:55
cnfwt...20:56
cnfnow I am really confusef20:56
cnfd20:57
timburkethat's...strange... hmm20:59
cnfand it's also not noted in the API docs20:59
timburkedid it at least mention your 400?20:59
cnf>,<20:59
cnfyeah, the 400 gives me a nice stack trace20:59
cnfERROR keystone.common.wsgi TypeError: argument of type 'NoneType' is not iterable21:00
cnfahale:  example of "not useful debug info :P21:00
patchbotError: No closing quotation21:00
ahaleugh21:00
cnfhow the hell can one request give a 404, and the other a 400 on the same bloody path? o,O21:01
cnfpardon me, but i'm a bit frustrated21:01
timburkeso, funny story: apparently something like `curl -v http://keystone/v2.0/s3tokens -X POST -H content-type:application/json -d '{"credentials": {"signature": "", "access": "", "token": ""}}'` will 404! with a message like "Could not find credential: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"21:03
timburke*no idea* where that came from, but w/e21:03
cnfo,O21:03
cnfwhat...21:03
cnfthe...21:03
cnfactual...21:03
cnfahale:  another example of _really_ bad debug info? :P21:04
cnftimburke: ok, so i'm just not using the right credentials?21:04
timburkesending a legit credential (instead of blank) gets me a 40121:04
cnfok, then I have no idea how to construct the right credentials for s321:04
timburkei think maybe? what's your access_key? is it the keystone user name, or the crazy hex "Access" thing from keystone?21:05
*** geaaru has joined #openstack-swift21:05
ahalewow that 404 is kind of awesome tim21:06
cnfI have no idea...21:06
cnftimburke:  i am using the "access" thing from `openstack ec2 credentials show name`21:06
cnfuser_id also doesn't work21:07
cnfhmm21:08
timburkeyeah, keystone generates some separate id instead of the user_id. i guess so you could have multiple credentials per user?21:08
cnfI think so?21:08
cnfbut I can't get any working21:08
cnfI don't know which one I am supposed to use21:09
timburkei'd expect the access to work :-/ mind trying a curl like what i sent above, but pasting in your access id in the appropriate place?21:12
cnfwhat do I use for signture?21:13
timburkeleave it blank; we'll see if it 401s or 404s21:14
cnf40421:15
cnfwith signature blank21:15
cnfuhm, no21:16
cnf40121:16
timburkeyay! i like 401s!21:16
cnfHTTP/1.1 401 Unauthorized21:16
cnfhmm, weird21:18
cnfso 1) why is is unauthorized, and 2) why is what swift3 is sending not found?21:18
timburkeif you've got debug logging on in the proxy, i think we'll see the access credential that's sent to keystone21:20
timburkethe unauthorized is because we provided a blank signature, which can't match21:21
timburkeif you wanted to generate a signature to send, you could do something like `python -c "import hmac, hashlib; print hmac.new('<your secret>', '', hashlib.sha1).digest().encode('base64')"`21:22
timburkewhich should get you a 200 with a blank token21:23
cnfthen I have to sign my secret with it, no?21:26
cnfok, can I just say "wtf" now?21:27
cnftimburke:  so when I do a call against swift with an s3 client, I get a log entry saying Connecting to Keystone sending this JSON:21:28
cnfand then the json with all the values21:28
cnfif I send that EXACT json, I get a 20021:28
cnfswift gets a 40421:28
cnfo,O21:28
timburkewha-?21:28
cnfuuuuuuhu...21:29
timburkemaybe double check the ports? dns resolution? try curling from the proxy?21:29
cnfcurl from the proxy gets me a 20021:30
cnfoh, hmm21:30
cnfwt?21:32
cnfok, so21:32
cnfif I curl21:32
cnfI see a INFO keystone.common.wsgi [req-37d4d40a-d702-4d50-a7a8-95a8c2eeb7c0 - - - - -] POST http://<dns.entry>:35357/v2.0/s3tokens21:33
cnfand then a NFO eventlet.wsgi.server [req-37d4d40a-d702-4d50-a7a8-95a8c2eeb7c0 - - - - -] 172.17.0.1 - - [08/Oct/2016 21:33:28] "POST /v2.0/s3tokens HTTP/1.1" 200 2111 0.23043721:33
cnfthe one from swift does NOT show the keystone.common.wsgi entry21:33
timburkebbiab, gotta move laundry21:36
cnfok, and I have found something21:37
*** ukaynar has quit IRC21:38
cnfyeah... that is stupid...21:39
*** ukaynar has joined #openstack-swift21:43
*** geaaru has quit IRC21:56
*** tries_ has quit IRC22:04
*** tries_ has joined #openstack-swift22:16
cnftimburke:  got it working22:43
cnfok, now how do container permissions work...22:44
*** Raymii has quit IRC23:01
*** Raymii has joined #openstack-swift23:05
*** Nype has joined #openstack-swift23:11
NypeHello, Is it possible to have access logs (with IP etc) files hosted with swift?23:12
cnf Nype  you want access logs OF swift, or store access logs of something else IN swift?23:12
cnfNype: ?23:18
NypeI want access logs of swift23:35
Nypesorry for time answer :s23:35
cnfuse the swift-proxy access logs? don't you run it on apache or something?23:40
NypeI use swift throught OVH, should I contact them to know if i can have an acces to these logs ?23:42
cnfoh, it's not your own swift install?23:43
Nypenope, it's like amazon s3 based on openstack swift23:44
cnfand you want access logs from that?23:44
Nypei pay per GB/month and GB out of trafic23:45
cnfyou should ask them, I guess23:45
notmynameNype: yeah. if you aren't running the cluster yourself, you'll have to ask your provider for them. eg rackspace has that feature. not sure if ovh does.23:45
*** ukaynar has quit IRC23:45
cnfohai notmyname23:45
cnfi got swift3 working23:45
* notmyname isn't here ;-)23:45
Nypei want theses logs to have a lot of stats mainly23:45
Nypeok thank you guys, i'll contact them23:45

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!