Tuesday, 2015-02-17

notmynamegreat!00:00
notmynamemahatic: now I've got to ask the important question: what in the world are you doing awake right now?!?! ;-)00:00
notmynameisn't it like 4am for you?00:00
notmyname(or 5:30)00:00
mahaticnotmyname, lol. I really thought I'd get done with this by 2 am and now it's 5.30!00:01
*** dmsimard is now known as dmsimard_away00:01
mahaticgit ate up more than an hour!00:02
mahaticnotmyname, about the patch: I don't have a test for scout_server_type yet. I plan to get this reviewed and then a diff patch for that test.00:03
mahatic(because I think it will be easier to review, than all code at once. Correct me if I'm wrong)00:04
*** ho has joined #openstack-swift00:16
hogood morning!00:18
*** nellysmitt has joined #openstack-swift00:38
*** dmorita has joined #openstack-swift00:40
mattoliverauho: morning00:41
*** nellysmitt has quit IRC00:43
klrmnok, probe tests care if the servers are stopped vs killed…who knew?00:45
*** jrichli has quit IRC01:16
*** jamielennox is now known as jamielennox|away01:16
*** jrichli has joined #openstack-swift01:16
*** jrichli has quit IRC01:16
*** jrichli has joined #openstack-swift01:18
notmynameI've been working a little today on using the openstack wiki swift page for project tracking stuff. like we talked about last week01:18
notmynamefor reference, I move the current page to https://wiki.openstack.org/wiki/OldSwift01:18
notmynameand I've started on https://wiki.openstack.org/wiki/Swift01:18
notmynamethe "project links" section is where I want to spend most of my time01:19
notmynameI want to make that more visible and more full-featured. some of that I'll have to lear how to do on the wiki01:19
notmynameand the "related pages" section at the bottom is pretty cool, i think01:19
notmynamenext up, I'm going to work on the review dashboard and the priority reviews page to get those better01:20
notmynameI'll be back for more tomorrow.01:21
notmynamegood night01:21
jrichligood night!01:22
*** mahatic has quit IRC01:24
*** jamielennox|away is now known as jamielennox01:31
*** jamielennox is now known as jamielennox|away01:41
homattoliverau: I would like to have your re-review for #138342 (recon). Can I have it?02:06
*** jamielennox|away is now known as jamielennox02:09
*** dmsimard_away is now known as dmsimard02:25
mattoliverauho: sure, I'll take a look this arvo, sorry things just keep getting in my way of fun swift stuff today :( I guess that happens when you've been out of the office for a week. But think I'm all caught up now02:35
*** nellysmitt has joined #openstack-swift02:39
*** nellysmitt has quit IRC02:44
homattoliverau: thanks! I'm sorry, I know you mast be busy...02:48
mattoliverauho: nah its fine, its my first day back is all, yours is at the top of my list :)02:58
homattoliverau: great!03:02
*** doxavore has joined #openstack-swift03:16
mattoliveraudmorita: notmyname wanted to let you know yesterday that slogging is updated to work with current swift, and clayg wanted you to know that he's got a patch up for muliple reconciler (regarding your SP migration research) https://review.openstack.org/#/c/10377903:53
*** reed has quit IRC03:56
dmoritamattoliverau: thanks for letting me know that. I noticed slogging update this morning and I would like to give notmyname a word of thank you. I will check clayg's patch soon deeply but it must be great patch :)04:03
dmoritanotmyname: I am very happy that slogging now supports json format and different timezone output. Thanks for merging my pull requests! In addition, I am very sorry for bothering you in hack-a-thon to modify unit tests. I should fix tests before sending pull requests.04:09
doxavoreany idea why tempauth would succeed for an /auth/v1.0 request and return an AUTH_tk, then a HEAD /d0/231526/AUTH_test returns a 400?04:13
doxavorei've tried turning up debug logging, but the output is exactly the same04:14
doxavoreall of my nodes are completely fresh, only an empty "objects" dir in their mounted locations04:16
mattoliveraudoxavore: have you got memcache running?04:29
doxavoreyes, i did a flush_all on all of them and verified they were empty04:30
mattoliveraudoxavore: tempauth stores the token in memcache when you authenticate, usually memcache on the proxy. Is /d0/231526/AUTH_test the storage url returned then you authenticate?04:32
mattoliverau*when04:32
doxavoreit's returning http://127.0.0.1:8080/v1/AUTH_services as the X-Storage-Url - the swift client 2.3.1 is calling the /d0 location when I issue a `swift stat`04:36
doxavoreeven when I attempt to use a testing user that I've never used before, the stat is trying to load /d0/####/AUTH_account and getting a 400 - on the file system /srv/node/d0/objects is empty04:37
*** nellysmitt has joined #openstack-swift04:40
mattoliveraudoxavore: well if you want to issue a HEAD to the proxy while authenticated, you are suppose to use  http://127.0.0.1:8080/v1/AUTH_services as the url though the proxy to access your account. I guess the question is what are you trying to do? A normal user wont have access to HEAD a storage node directly, which is when the drive path may come into play.04:42
*** nellysmitt has quit IRC04:45
doxavoremattoliverau: so I have ST_AUTH, ST_USER, and ST_KEY all set in my env. When I run `swift stat` I get an "account not found error". here's what I'm trying to do and the log output associated with it: https://gist.github.com/doxavore/c93ca0792629c5082cd704:47
doxavorethat's where I'm getting the /d0 from04:50
doxavorethat is, i'm not trying to hit it directly, i'm just trying to use the cli client04:51
mattoliveraudoxavore: thats a call from the object server, so you can't use that.. if you use test:tester does it work?04:53
doxavoreno, i get the same errors04:55
doxavoreif i'm only using tempauth, at what point does it create the account DB? it almost looks like it's trying to access one that just isn't on the filesystem (because, nothing is on the file system...)04:56
mattoliveraudoxavore: in your proxy-server.conf, is it the default? (i.e unchanged) Cause the tester3 user, cause if that is the case the tester3 user isn't member of the .admin or .resller_admin groups, so they only have access if someone in the .admin group gives them access to a container via ACLs.05:02
mattoliveraudoxavore: https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L208-25705:03
*** dmsimard is now known as dmsimard_away05:03
mattoliveraudoxavore: actually https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L239-257 is more specific05:04
doxavoremattoliverau: yes it's the default tempauth config. when i use the admin:admin/admin login which is still a .reseller_admin I get the same exact error. It's just not creating any accounts, with allow_account_management and account_autocreate both enabled05:04
mattoliveraudoxavore: you will have the account_autocreate = True in proxy-server.conf, so an account will be auto created when you start using it.05:06
doxavoreunder app:proxy-server I have both "allow_account_management = true" and "account_autocreate = true"05:07
mattoliveraudoxavore: https://gist.github.com/doxavore/c93ca0792629c5082cd705:08
mattoliverauonce you export the admin details, what happens when you swift stat?05:09
doxavorethe same thing - account not found, syslog shows it giving a 400 on a different /d0 address05:10
doxavorefor the tester, tester3, and admin users in their respective accounts05:10
*** jrichli has quit IRC05:11
mattoliveraudoxavore: is this on a SAIO?05:13
doxavoreit's running all the services, but installed 2.2.2 via .deb packages and our same puppet config that was working on 2.2.005:14
*** devlaps has quit IRC05:16
doxavoreso, it's AIO but I'm not sure if "SAIO" is specifically the dev build. I don't mean to be vague :)05:16
mattoliveraudoxavore: lol, fair enough, what's the output of the curl command (comment in your gist)? is it authenticating?05:17
mattoliveraudoxavore: if you want to show the output then paste.openstack.org or gist it, don't dump in channel (which you've been doing, thanks)05:18
doxavoremattoliverau: certainly. added to existing gist05:19
mattoliveraudoxavore: lol, sorry it's support to be curl not url (copy and paste error)05:19
mattoliveraudoxavore: cool so it authenticates, next run the following curl command.. see gist once I've actaully pasted it :P05:20
*** ppai has joined #openstack-swift05:22
mattoliveraudoxavore: pasted, it should be one line and you should recieve a 20405:23
doxavoremattoliverau: updated, gives me a 400 without a body05:23
mattoliverauNOTE: I used the authtoken you recieved05:23
mattoliveraudoxavore: hmm, I would have expected to get a 401 if we were just loosing the token (ie. a memcache issue). What does your logs say this time? Same thing?05:26
doxavoremattoliverau: also just dropped in the log entry. So the proxy seems to be translating it from what we requested to HEAD /d0/231526/AUTH_test - which I imagine is trying to find the account DB somewhere on the file system to give container stats, and that doesn't work...05:26
doxavoreon the filesystem, there's a /srv/node/d0/objects directory but that's _it_05:27
notmynamedmorita: it was no bother for you to ask me about slogging. the unit tests were more for my comfort. and actually I think they're better overall now05:32
notmynamedmorita: and yes. json output is much nicer :-)05:32
mattoliveraudoxavore: the /d0/<part>/<account> is normal.. this is a request to the account server which will be translated to disk. The question is why in your gist is it pointing to the object server? or is your "object" server your account server?05:34
mattoliveraudoxavore: because otherwise this may be why your receiving a 400.05:35
doxavoremattoliverau: they're on the same node...05:35
doxavoreahhhhhhh05:36
mattoliveraudoxavore: but it looks like it is trying to issue a command to the object server not the account server part05:36
doxavorethat's it. I was already sleep-deprived when I created these rings, and I threw everything to the same port05:36
mattoliveraudoxavore: lol, that would do it :)05:37
doxavorefortunately, now I know what that looks like... :-/05:37
doxavoremattoliverau: thank you much for your help, that just wasn't clicking for me.05:38
mattoliveraudoxavore: anytime, I'm glad we got there in the end :), I should've just noticed the object-server in the logs you posted.. now I'll know what to look at next time also ;)05:39
*** pcaruana has quit IRC05:47
*** SkyRocknRoll has joined #openstack-swift05:49
*** doxavore has quit IRC05:59
*** dmsimard_away has quit IRC06:09
*** dmsimard_away has joined #openstack-swift06:09
*** dmsimard_away is now known as dmsimard06:09
notmynameand that's what mahatic's patch will solve (or at least detect) with one simple cli command06:12
mattoliveraunice, looking forward to it :)06:18
mattoliveraunotmyname: now shouldn't you be thinking about sleeping?06:18
notmynamemattoliverau: I was just up talking about swift tshirt designs :-)06:18
mattoliveraunotmyname: oh, in that case, no sleep for you until its done :P06:19
notmyname:-)06:19
*** nshaikh has joined #openstack-swift06:37
*** nellysmitt has joined #openstack-swift06:41
*** nellysmitt has quit IRC06:45
*** silor has joined #openstack-swift06:49
*** SkyRocknRoll has quit IRC07:04
*** SkyRocknRoll has joined #openstack-swift07:21
*** panbalag has quit IRC07:27
*** panbalag has joined #openstack-swift07:46
*** mkerrin has joined #openstack-swift07:54
*** kajinamit has joined #openstack-swift08:02
*** rledisez has joined #openstack-swift08:06
*** fifieldt has joined #openstack-swift08:09
*** chlong has quit IRC08:11
*** SkyRocknRoll has quit IRC08:21
*** nshaikh has quit IRC08:27
*** geaaru has joined #openstack-swift08:30
*** mmcardle has joined #openstack-swift08:36
*** SkyRocknRoll has joined #openstack-swift08:39
*** nellysmitt has joined #openstack-swift08:42
*** jordanP has joined #openstack-swift08:45
*** nellysmitt has quit IRC08:46
*** nellysmitt has joined #openstack-swift08:47
*** joeljwright has joined #openstack-swift08:49
*** SkyRocknRoll has quit IRC08:50
*** jistr has joined #openstack-swift08:51
*** mkerrin has quit IRC08:53
*** fifieldt has quit IRC08:56
*** SkyRocknRoll has joined #openstack-swift09:02
*** donagh_ has joined #openstack-swift09:03
*** kajinamit has quit IRC09:08
*** SkyRocknRoll has quit IRC09:15
*** aix has quit IRC09:21
*** SkyRocknRoll has joined #openstack-swift09:32
*** acoles_away is now known as acoles09:40
*** aix has joined #openstack-swift09:48
*** nshaikh has joined #openstack-swift10:00
*** aix has quit IRC10:05
*** aix has joined #openstack-swift10:17
*** geaaru has quit IRC10:20
*** leopoldj has joined #openstack-swift10:32
*** geaaru has joined #openstack-swift10:34
*** dmorita has quit IRC10:38
*** leopoldj has quit IRC10:38
openstackgerritAlistair Coles proposed openstack/swift: Update guest VM OS recommendation in SAIO doc  https://review.openstack.org/15653910:49
*** foexle has joined #openstack-swift10:57
*** SkyRocknRoll has quit IRC10:58
openstackgerritDonagh McCabe proposed openstack/swift: Add multiple reseller prefixes and composite tokens  https://review.openstack.org/13708611:06
acolescschwede: hi, you here?11:15
*** SkyRocknRoll has joined #openstack-swift11:15
*** dmsimard has quit IRC11:22
*** SkyRocknRoll has quit IRC11:26
*** dmsimard_away has joined #openstack-swift11:30
*** dmsimard_away is now known as dmsimard11:30
*** khivin has quit IRC11:33
*** mwilliams_ct has joined #openstack-swift11:34
*** mahatic has joined #openstack-swift11:42
*** SkyRocknRoll has joined #openstack-swift11:44
*** SkyRocknRoll has quit IRC11:55
*** ho has quit IRC11:58
*** SkyRocknRoll has joined #openstack-swift12:12
*** SkyRocknRoll has quit IRC12:30
*** arnaud_o has quit IRC12:44
*** arnaud_o has joined #openstack-swift12:45
*** SkyRocknRoll has joined #openstack-swift12:47
*** ppai has quit IRC13:19
*** HenryG has quit IRC13:49
*** SkyRocknRoll has quit IRC13:51
*** HenryG has joined #openstack-swift13:52
*** tdasilva has joined #openstack-swift14:02
*** mwilliams_ct has left #openstack-swift14:02
*** Guest90435 has joined #openstack-swift14:02
*** Guest90435 is now known as annegentle14:02
*** jkugel has joined #openstack-swift14:06
*** nshaikh has quit IRC14:06
*** donagh_ has quit IRC14:33
*** donagh_ has joined #openstack-swift14:33
*** doxavore has joined #openstack-swift14:33
openstackgerritJoel Wright proposed openstack/python-swiftclient: Fix crash when stat'ing objects with non-ascii names  https://review.openstack.org/14784614:48
*** badgas has joined #openstack-swift14:49
openstackgerritJoel Wright proposed openstack/python-swiftclient: Fix crash when stat'ing objects with non-ascii names  https://review.openstack.org/14784614:52
*** SkyRocknRoll has joined #openstack-swift15:06
*** geaaru has quit IRC15:23
*** geaaru has joined #openstack-swift15:36
*** daddyjoseph97 has joined #openstack-swift15:39
*** SkyRocknRoll has quit IRC15:44
*** doxavore has quit IRC15:45
*** daddyjoseph97 has quit IRC15:45
*** jrichli has joined #openstack-swift15:49
*** khivin has joined #openstack-swift15:50
*** dmsimard is now known as dmsimard_away15:54
openstackgerritDonagh McCabe proposed openstack/swift: Support HTTP_X_SERVICE_IDENTITY_STATUS in keystoneauth  https://review.openstack.org/15663416:00
cschwedeacoles: yes, i’m here :)16:05
*** doxavore has joined #openstack-swift16:06
*** reed has joined #openstack-swift16:11
*** sgotliv has joined #openstack-swift16:14
doxavorewhen write_affinity is enabled, how does it select which devices it writes to? the write_affinity_node_count suggests it has nothing to do with the ring-specified handoff node, so how would the replicator find it to sync/remove?16:20
acolescschwede: hi. hope you had a good journey home. i had some more thoughts about undelete...16:21
cschwedeacoles: thanks, yes, trip was quite relaxed. and you? had some ideas about undelete on the way home?16:22
cschwedei’m curious :)16:22
acolesyes, good flight thanks. so this is based on the discussion in sfo around using a manifest like object to point to versions of an object16:23
acolescouple of things: (a) it *might* be that the service token deature is useful, or can be adapted. I guess we would like to 'hide' the container with the object versions so the user only sees a single container listed (the one with the manifests).16:24
acoleswe could hide the version container in another reseller prefix namespace16:24
cschwedeyep, maybe even use a container in a different account16:24
cschwedeor a different prefix, yes16:24
acoleswhich service tokens does. and maybe use service token auth to manage access to the version container/namespace16:25
cschwedeand manifests could be very similar to SLO manifests, or not?16:25
acolese.g. an undelete middleware (was that the idea?) would have a service token to be able to auth operation son the versions, which the user alone could not do16:26
cschwedemy first idea was a middleware, but would probably require a change in the object server too16:26
acoles(would also mean that an admin/third party could create/delete versions via API if they ha daccess to a service token and the user token)16:27
serverascodehi, hw question: is anyone using the supermicro 72 bay chassis? good idea bad idea? ( eg. http://www.supermicro.com/products/system/4U/6047/SSG-6047R-E1R72L.cfm )16:28
cschwedeacoles: well, at the end there is always someone who could delete stuff - but maybe only a server admin16:28
*** abhirc has quit IRC16:28
cschwedeacoles: but if we could provide a solution where a user with a „normal“ account+token can store data without the possibility to delete it, that would be a big improvement16:29
acolescschwede: yes, the point being that with auth+service token an admin can perform manual deletes via API and not need a backdoor to cleanup. re-using service token might require some tweaks but there seemed to be some overlap in terms of restricting user operations and hiding object in another prefix16:30
cschwedeone question remains if we use manifests: if someone deletes the manifest (meaning the object should be deleted later on), the „real“ object needs an meta-data update too. to mark it for later deletion16:30
cschwedeacoles: i will have a closer look to the proposed service tokens and we could reuse them, maybe we can combine our efforts16:30
acolescschwede: yes, i think we said we might need fast-post to be able to update the real object x-delete-at ??16:31
acoles(well, post as copy would work too)16:31
cschwedeyes, that would simplifiy this a lot, because then we could avoid i/o-heavy rewrites16:31
acolescschwede: so, (b) second thought, re the manifests...16:31
acolescschwede: i think we said that a user delete would be transformed to a PUT manifest with a 'magic' content-type that allowed those 'deleted' objects t be filtered out from the container listing.16:33
acoless/t/to/16:33
cschwedeif we keep that manifest, yes16:33
acolescschwede: i *think* that was so that we never lost track of the real object - a privileged container GET *would* include the 'deleted' manifests16:33
acolescschwede: ^^ is that correct ??16:34
cschwedeyes, maybe using a query parameter like GET http://saio/v1/AUTH_test/container/?show_deleted or something similar16:34
acolesok, so the problem i see is that the manifest timestamp must always move forwards in time, otherwise the ocntainer server will ignore an update, so its hard to perform an undelete (i.e. PUT manifest with re-instated pointer) without the manifest timestamp being newer than the last delete, and therefore the listing timestamp being newer than the 'undeleted' real object16:36
acolescschwede: but i wondered if we can make use of two vector timestamps to help16:36
cschwedefrom fast-post?16:37
acolesit is perhaps similar to the container reconciler scenario, where an object needs to be PUT/DELETED/PUT without changing 'user-visible' time16:37
acolescschwede: no, the two-vector from reconciler/policy feature16:38
*** sgotliv has quit IRC16:38
acolesfast-post has a triple-timestamp ;)16:38
*** Nadeem has joined #openstack-swift16:38
cschwedeah yes, one more :)16:39
acolescschwede: i don't know for sure, but the thought was that when 'deleting' the manifest, we increment its timestamp offset part (but not the 'normal' timestamp) which is enough for container to create a new row16:39
notmynamegood morning, world16:39
* acoles also notes this has to work for container sync too16:39
acolesnotmyname: morning, were you there for the undelete discussion?16:40
notmynameserverascode: nice box. I'll ask around16:40
notmynameacoles: in sfo or in scrollback just now?16:40
acolesnotmyname: sfo, but both16:40
serverascodenotmyname: cool thanks any info would be helpful :)16:40
notmynamejust checking scrollback now16:40
notmynameserverascode: yes, in general, it should work fine. I've heard of other deployments that have similar (or greater!) density. you'll have to pay attention to capacity adjustments and CPU load with that many drives16:42
acolescschwede: ok, that was a bit of a brain dump from me. meant to be helpful though :)16:42
cschwedeacoles: thanks a lot, i will include your ideas into the next spec patchset :)16:42
acolescschwede: cool - so are you planning to update the spec?16:43
cschwedeacoles: yes16:43
acolesexcellent. well, i can always add comments there16:43
serverascodenotmyname: thanks!16:43
notmynameserverascode: generally you only want boxes that big if you have a big cluster. so that each box isn't a huge percentage of the total capacity16:44
serverascodeyeah for sure16:45
*** nellysmitt has quit IRC16:47
acolescschwede: the final comment, to which i don't have an answer, is how to allow a container to appear deleted, and yet retain all its manifests.16:49
acolesand then be able to undelete the container16:49
acoles(in the case when all the manifests are in 'deleted' state so the container is apparently empty)16:50
btorchserverascode: what will be your head unit ?16:50
cschwedeacoles: the container with the „real“ objects could use the same container name but with a suffix, thus it can be undeleted and listed with the service token16:51
serverascodebtorch: head unit? proxy?16:51
btorchserverascode: the node attached to the supermicro jbod unit16:51
btorchserverascode: the server type/hardware that will have the supermicro attached16:52
serverascodeit's a server itself with 72 bays16:52
serverascodeunless I linked to the wrong page16:52
notmynamebtorch: looks like that box includes some CPU in the box16:52
btorchserverascode: sorry I miss read it :)16:52
acolescschwede: yes. or if the real object container is in a different prefix it can just have the same name.16:53
btorchserverascode: not sure how many storage nodes you will use but I would consider that first16:55
notmynamebtorch: what's up? haven't seen you around in a while!16:55
acolescschwede: in fact, i need to be reminded why we need to keep a 'deleted' manifest? was it to avoid dark data? seems like we could always find any 'orphaned' real objects by listing the real object container (assuming its name is a transform of the original)16:56
notmynameacoles: cschwede: re undelete. looks like you've got it well in hand :-)16:56
btorchserverascode: if it's several nodes spread around 5+ zones I would go with something like that but it's just a few nodes I would think twice, obviously you don't have to fill all the bays and start it slow and then backfill16:56
acolesnotmyname: yeah, right ;)16:56
btorchnotmyname: not much, I'm around just don't talk much in here16:56
notmynamebtorch: are you still doing swift things? last I heard you were doing rax private clouds or something16:57
btorchyeah swift16:57
notmynamecool16:57
serverascodebtorch: yeah will keep that in mind, they keep talking about 2PB usable16:57
cschwedeacoles: yes, that makes sense, we only need to take care of the name length constraint16:58
*** cca has joined #openstack-swift17:00
*** mahatic has quit IRC17:04
notmynameacoles: cschwede: can you comment on https://bugs.launchpad.net/swift/+bug/1422237 before you leave today?17:05
openstacknotmyname: Error: malone bug 1422237 not found17:05
notmynameopenstack: thanks. you aren't allowed allowed to see that17:05
acolesnotmyname: ok17:05
*** daddyjoseph97 has joined #openstack-swift17:10
notmynameanyone have any experience with mechanical keyboards? specifically cherry mx switches (brown vs clear). I'm looking at http://www.wasdkeyboards.com/index.php/products/code-keyboard/code-87-key-mechanical-keyboard.html17:10
notmynamebut I've never used one17:11
btorchserverascode: ah one thing another swift ops just pointed out, check with supermicro if they provide 10g instead of the 1g nics17:12
serverascodebtorch: yeah for sure good point17:13
*** dmsimard_away is now known as dmsimard17:19
*** daddyjoseph97 has quit IRC17:22
*** jistr has quit IRC17:24
cschwedenotmyname: will comment on that later, in about 2 hours17:27
notmynamecschwede: ok, thanks17:27
*** annegent_ has joined #openstack-swift17:29
*** rledisez has quit IRC17:30
openstackgerritDaniel Wakefield proposed openstack/python-swiftclient: Add flag to disable automatic checksum validation.  https://review.openstack.org/15667717:36
*** miqui has quit IRC17:37
hurricanerix_Here is Hisashi Osanai's photo from dinner the first night of the hackathon for anybody that wants a copy (I did a little color correction)17:38
hurricanerix_http://blog.hurricanerix.me/images/_DSC4967.png17:38
hurricanerix_notmyname: I am taking time off till the 25th, so if anybody is really anxious about getting https://review.openstack.org/#/c/155985/ merged in, feel free to let them take it over.  Otherwise I will work on it next week when I am back.17:39
notmynamehurricanerix_: thanks17:40
hurricanerix_np17:40
notmynamehurricanerix_: can you leave a comment in gerrit about that?17:40
hurricanerix_notmyname: oh sure.17:40
notmynamethanks17:41
hurricanerix_done17:42
tdasilvahurricanerix_: enjoy your time off :)17:44
hurricanerix_tdasilva: Thanks, I will. =)17:45
jrichlihurricanerix_: thanks for sharing the photo!17:46
hurricanerix_jrichli: np17:46
notmynameFYI, the _infra team has just proposed a change to the swift functional tests job to use "tox -e func" instead of ".functests" (ie nose). the main change is that tox sets up a venv with our requirements file and nose just uses whatever is on the system17:47
notmynameIf you have comments or concerns, please leave them on https://review.openstack.org/#/c/156676/17:47
openstackgerritAlistair Coles proposed openstack/swift-specs: Updates to encryption spec  https://review.openstack.org/15431817:50
acolesjrichli: torgomatic ^^ still work in progress but comments/corrections welcome17:51
jrichlithanks, I will take a look17:51
openstackgerritAlistair Coles proposed openstack/swift-specs: Updates to encryption spec  https://review.openstack.org/15431817:52
*** gyee has joined #openstack-swift17:52
*** doxavore has quit IRC17:53
*** acorwin_ is now known as acorwin17:58
*** abhirc has joined #openstack-swift18:05
*** nellysmitt has joined #openstack-swift18:06
*** doxavore has joined #openstack-swift18:11
*** mmcardle has quit IRC18:11
doxavorenotmyname: i use a CODE keyboard with cherry MX clear switches and it's beautiful. I'm not sure my office mates care for it as much :>18:13
notmynamedoxavore: I'm initially buying it for home. :-)18:13
notmynamedoxavore: but yeah, it looks pretty good. but I haven't actually used those switches before, so it's hard to say if I like it. I'd like to try it out before dropping 150 on it :-)18:14
*** annegent_ has quit IRC18:14
notmynamedoxavore: I'm glad to hear you like it though18:14
notmynamefrom the specs, it seems the clear switches have the same activation pressure as the keyboard I'm currently using at work (an apple chicklet wireless--don't laugh, I like it). so I think it should feel fine. but it's a lot different18:15
*** jbonjean has quit IRC18:16
*** jbonjean has joined #openstack-swift18:17
cschwedenotmyname: i used mechanical keyboards in the past, nowadays i prefer the keyboards from apple… you might look at http://www.daskeyboard.com/ as well if you’re looking into buying a mechanical keyboard18:19
notmynamecschwede: ya, I've been comparing das keyboard (the das keyboard 4C) against http://www.wasdkeyboards.com. The CODE keyboard from WASD looks pretty interesting. if for nothing else, I like the backlighting :-)18:19
notmynamehttp://www.wasdkeyboards.com/index.php/products/code-keyboard/code-87-key-mechanical-keyboard.html18:20
notmynameoops. I already linked that :-)18:20
doxavoreI switch between it and the macbook's built-in keyboard pretty seamlessly without any grief. I just feel more accurate with that tactile click. I've had other WASD keyboards and can't recommend them enough.18:20
*** jbonjean has quit IRC18:21
notmynameI wish there were a place where I could go try out a few. maybe there is but I just don't even know what to search for18:21
*** sgotliv has joined #openstack-swift18:21
cschwedei remember typing on an IBM M keyboard - they were actually built like a tank. that was some tactile feedback!18:22
notmynameya18:22
notmynamejeblair says he has a stack of model Ms at his house.18:23
notmynameso let's see what the magic of twitter can do ;-) https://twitter.com/notmyname/status/56775103278913945618:23
cschwedenotmyname: hmm, i would imagine that there is a market in SF for a store selling „extraordinary“ IT stuff?18:23
notmynameno kidding18:23
notmynameand staffed by people with waxed moustaches.18:24
notmyname;-)18:24
doxavoreis anyone here familiar with how devices are selected for the write_affinity_node_count? i'm having trouble tracking it down in any documentation and just want a slightly better understanding of how we should expect it to behave under failure conditions...18:26
notmynameyes18:26
*** jbonjean has joined #openstack-swift18:27
notmynamedoxavore: let's start with the way the default works (there's 3 ways affinity works)18:27
notmynamethe default is "shuffle"18:27
notmynameso when swift makes a call to ring.get_nodes, it gets back the primary locations18:27
notmyname(thinking of the GET path first)18:27
notmynameso with shuffle, it calles random.shuffle() on the returned primary node list18:28
notmynameso it's a simple load balancer (with no feedback)18:28
notmynamethe next affinity method is "timing"18:28
notmynamewith that mode, every time the proxy connects to a node, it tracks how long it took to connect (ie create the socket)18:29
notmynamethen it chooses the fastest one out of the primary node list18:29
notmynameso that balances, but as a node gets busy it will get slower18:29
notmynameand "farther away" nodes will not be selected18:29
notmynamethe last affinity mode is explicit. in that way the proxy is configured to know what its own region and zone is18:30
notmynameand it sorts the returned primary nodes by finding one that better matches. eg same zone matches before same region18:30
notmynameand you can explicitly configure a proxy to prefer a specific region or zone18:31
notmynameso that's the read path. make saense?18:31
notmynamesense even18:31
doxavoreyes18:32
notmynameok good :-)18:32
notmynameso moving on to the write path18:32
notmynamesince a write has more than one node, it's similar but slightly different18:32
notmynameeg shuffle affinity wouldn't do anything since it would still be sent to all primary nodes18:33
notmynameso let's move to the write affinity18:33
notmynameerr, the explicit write affinity18:33
notmynameyou can set it to explicitly prefer a given region18:33
notmynameso eg you'd configure proxyA in region1 to prefer region 1. and proxyB in region2 to prefer region 218:34
notmynamewhat happens is that it will attempt to find nodes in that region to write to before it chooses nodes in a different region18:34
notmynameimportantly, THESE ARE NOT THE PRIMARY NODES18:34
*** cebruns_ is now known as cebruns18:34
notmynamebecause the primary nodes will be spread throughout the cluster18:35
notmynameso one is probably the primary. but maybe not18:35
notmynameand so the write_affinity_node_count is how many nodes it checks in the identified region before it puts something into a different region18:35
*** jordanP has quit IRC18:35
notmynamethe default is 2*replica count (ie 6 for 3 replicas)18:35
klrmnjust in case anybody is wondering, today i am investigating why my tests think they have set up a swift node / gateway share but the cluster deploy page thinks it hasn't got them. i suppose there is an outside chance that this is related to changes made for the large cluster template scenario18:36
notmynameso if you have proxyA, it will try to write to 6 drives in region 1 before it even attempts to find nodes in region2. regardless of what the primaries are.18:36
notmynameso one of the region 1 nodes is probably one of the primaries18:36
notmynamebut the other two will be written to handoff nodes (handoffs == anything not a primary)18:37
klrmnit helps if i do not send my messages to the wrong channel tho18:37
notmynameand replication will eventually move it over to the right primary location18:37
notmynameklrmn: :-)18:37
notmynamedoxavore: make sense?18:37
doxavoreyes, i follow that.18:37
*** mmcardle has joined #openstack-swift18:38
doxavorei'm familiar with the single handoff node in the ring, but how are those handoffS selected?18:38
notmynamedoxavore: the biggest thing to consider is that write affinity is only possible if you have bursty writes and/or you have a region-to-region network that's faster than your data ingest18:38
*** devlaps has joined #openstack-swift18:40
notmynamedoxavore: handoffs are selected deterministically by walking through the ring.18:42
*** sgotliv has quit IRC18:42
notmynamehang on. I'll find the link18:42
doxavorei guess it's the replicator that handles moving the files from the handoffs to the primary, and it makes sense when writing to the single handoff node listed in the ring, but when we're talking some number of handoffs not listed in the ring...18:42
notmynamedoxavore: https://github.com/openstack/swift/blob/master/swift/common/ring/ring.py#L31018:43
notmynameno,no. everything is listed in the right18:43
notmyname*ring18:43
notmynameit's the difference in get_nodes() and get_more_nodes()18:44
notmynamedoxavore: so with https://github.com/openstack/swift/blob/master/swift/common/ring/ring.py#L337 it walks through the ring looking for other regions. in big rings, it will walk through every 65536 partitions. for small rings it will look at every other partition18:45
*** jasondotstar has joined #openstack-swift18:45
notmynamedoxavore: look at the 2 line comments in the for loops below that link18:45
doxavoreok, so that's quite clear18:46
notmynamedoxavore: to summarize in english: walk through the ring and find stuff that's widely spaced (ish) until you run out of places to look18:47
*** geaaru has quit IRC18:49
doxavoreand since we tie the proxy to a region, it will try to put all of those copies in the same region... is there a way to monitor how many of these files aren't in their primaries yet? something like the dispersion report, but ... for this. :)18:49
*** devlaps has quit IRC18:49
*** acoles is now known as acoles_away18:50
*** pjnaik1990_ has joined #openstack-swift18:51
notmynamedoxavore: what you'd monitor is the replication cycle time18:51
pjnaik1990_Hello!!I am new to Openstack. I am interested to participate in OPW program to contribute to swift..I am looking for mentors who can help me and provide me with a projectidea.18:53
notmynamepjnaik1990_: did you send me an email this morning?18:54
*** mmcardle has quit IRC18:54
doxavorenotmyname: that was all very helpful. thanks for taking the time. i have to same i'm pretty impressed with how Swift is put together so far.18:54
notmynamedoxavore: cool. glad it helped. and thanks18:55
pjnaik1990_@notmyname:Hi!! yes18:55
notmynamepjnaik1990_: I haven't had a chance to respond yet. I'll try to get to that later today18:56
notmynamepjnaik1990_: but in general, yes. I'd be happy to give you some info.18:56
pjnaik1990_Thankyou!!18:56
notmynamepjnaik1990_: but until i get the chance for email, you should find mahatic when she's online. she's currently doing and OPW for a project in swift and might be able to answer any questions you have18:56
notmynamebut for now, I've got to go to a meeting.....18:57
pjnaik1990_okay I will talk to her.Thanks18:57
*** mmcardle has joined #openstack-swift19:14
*** annegent_ has joined #openstack-swift19:15
*** abhirc has quit IRC19:17
*** openstackgerrit has quit IRC19:20
*** silor1 has joined #openstack-swift19:20
*** openstackgerrit has joined #openstack-swift19:20
*** silor has quit IRC19:20
*** annegent_ has quit IRC19:20
*** abhirc has joined #openstack-swift19:21
openstackgerritSean Dague proposed openstack/python-swiftclient: add functional tox target  https://review.openstack.org/15671919:25
*** silor1 has quit IRC19:29
*** lcurtis has joined #openstack-swift19:30
*** mahatic_ has joined #openstack-swift19:33
mahatic_notmyname, hello! I logged back in to remind you if you could give any inputs for this  -> https://review.openstack.org/#/c/153617/19:35
mahatic_tata, off to sleep, (on time tonight :P)19:35
*** pjnaik1990_ has quit IRC19:35
*** mahatic_ has quit IRC19:36
*** pjnaik1990 has joined #openstack-swift19:39
*** annegent_ has joined #openstack-swift19:40
lcurtisHello all...wondering if there is any way of replacing sqlit db in swift with something else?19:42
lcurtissqlite19:42
*** mmcardle has quit IRC19:42
lcurtisi am concerned about performance overall if we hit 100million+ objects and more19:43
lcurtisi have seen some solutions describe using ssd's19:43
lcurtisto counteract19:43
*** zhill has joined #openstack-swift19:45
tdasilvalcurtis: hi, container performance is a current concern. Last week there were some discussion on this topic and there was a spec written about a proposed solution: https://review.openstack.org/#/c/139921/19:47
lcurtisthanks tdasilva19:48
lcurtisis spreading objects over multiple containers most viable current solution?19:49
clayglol @ torgomatic's comment on https://review.openstack.org/#/c/156095/19:54
claygtorgomatic: times, they are a changin'19:55
torgomatichehe19:55
tdasilvalcurtis: the recommendations I have seen proposed is to use SSDs and to try to limit the size of containers (by spreading objects over multiples containers)20:00
lcurtistdasilva thanks much20:01
claygtdasilva: did you ever dust off the put/close-connection patch?20:03
tdasilvayes, I have it here, was finishing the obj. ver. middleware first and will post both20:04
tdasilvaclayg: just a reminder that unit tests are not passing yet20:05
claygtdasilva: nice20:06
*** nellysmitt has quit IRC20:09
*** nellysmitt has joined #openstack-swift20:12
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements  https://review.openstack.org/8873620:17
*** annegent_ has quit IRC20:18
*** occupant has quit IRC20:18
*** annegent_ has joined #openstack-swift20:41
claygacoles_away: does anyknow else in channel have some experience with devstack lately?  stack.sh is bailing out on me with an error from keystone creating the test swift users20:43
*** annegent_ has quit IRC20:53
*** rdaly2 has joined #openstack-swift21:01
*** annegent_ has joined #openstack-swift21:17
*** aix has quit IRC21:20
mattoliverauMorning21:22
jrichlimorning21:24
openstackgerritJohn Dickinson proposed openstack/swift: update the getting started doc  https://review.openstack.org/15638821:24
*** nellysmitt has quit IRC21:25
mattoliveraujrichli: morning :)21:25
*** doxavore has quit IRC21:27
*** annegent_ has quit IRC21:29
notmynameinteresting mailing list post just now. "On Object placement"21:29
*** doxavore has joined #openstack-swift21:33
mattoliveraunotmyname: hmm, yeah, I think he might have come into channel a few weeks ago to talk about it. Answered his questions on how the ring knew where to place and SP might be a solution in the way swift currently works.21:33
notmynamewow. https://www.openstack.org/vote-vancouver/Presentation/faith-the-secret-ingredient-of-a-successful-system-integration21:44
torgomaticnotmyname: that's... yeah, "wow" is probably the best way to say it21:46
clayghttps://www.youtube.com/watch?v=lu3VTngm1F021:47
doxavorethis is old, but has the state of backup not changed? https://answers.launchpad.net/swift/+question/149943 I'm not sure a container-to-container sync would be appropriate if you don't ever want the receiving end to delete? (and for us, objects are immutable, so not too concerned with an update)21:47
openstackgerritThiago da Silva proposed openstack/swift: versioned writes middleware  https://review.openstack.org/13434721:53
doxavorei know that's likely an ugly topic, but I have to imagine the bigger Swift players aren't relying only on online backups of connected regions :>21:54
*** gyee has quit IRC21:55
notmynamedoxavore: it depends on what you mean22:08
notmynamedoxavore: at swiftstack we've got many customers using multiple regions to keep offsite copies for DR22:08
doxavorenotmyname: to be more specific, what happens if someone issues a DELETE on an object in the main region?22:09
notmynamedeleted everywhere. it's more of DR (hot failover) rather than backups22:09
doxavoresay, they have a legal requirement that all the data in that account not be deleted (so long as some setting is set, etc)22:09
torgomaticthat is a very hard problem to solve22:10
torgomaticboth DELETE and PUT can destroy data, the latter by overwriting22:10
torgomaticso one might think "block PUT if the object exists"22:10
torgomaticbut in a distributed, eventually-consistent system, you can't answer "does the object exist?"22:11
torgomaticat best, you get a semipredicate: you can find it and decide "yes, it exists", or you can fail to find it and decide "well, maybe"22:11
torgomaticfor example, a copy might exist on some random node in the system that's not a primary node for that partition, or on any of the first 100 handoffs, because it hit a handoff and then a ring rebalance happened22:12
torgomaticor a copy might be on a disk on a node that's down22:12
notmynamedoxavore: interesting enough, we're actually working with a customer right now who needs the "prevent overwrite or delete" feature. but they already have a sclable service with a good api that exists that tracks the stuff that is "locked". so it's kinda easy there because there's already an existing, consistent locking service22:13
torgomaticso if you allow the PUT when you don't find the object, you run the risk of flattening the object anyway22:13
torgomaticnotmyname: yeah, it's easy when you have an external CP-type system to track this stuff in :) , but for those of us in AP-land, it's basically impossible22:14
notmynamethe -infra team needs us to land https://review.openstack.org/#/c/156719/1 asap. it's blocking some of their changes. one of which allows our tests to be run against our own requirements file (not the global requirements file)22:15
notmynametorgomatic: +100022:15
*** rdaly2 has quit IRC22:17
doxavorenotmyname: torgomatic: is there an easy way to block DELETE on an account, or would one look to just use middleware? in our case, the PUT of a schrodinger's file might be able to qualify as an acceptable risk :>22:18
torgomaticdoxavore: you'd have to write your own middleware to do it22:18
torgomaticblock DELETE, plus PUT of existing object22:19
torgomaticso, given that you said "easy", probably not ;)22:19
doxavoredo all the replicators, anything else that could delete files, work through the object-server API to hit that middleware, or would we only be protecting against external threats, and not so much misconfiguration (which could ultimately still exist if middleware exists, but..)22:20
torgomaticdoxavore: you'd do it in the proxy, not object servers22:20
torgomaticif your clients can hit object servers directly, you have no security against misbehavior at all22:21
eikkelol @ 'faith' talk link22:26
torgomaticeikke: I can't decide whether I should vote for it or not. It'd be like voting for a train wreck: amazing to watch, but better to avoid it in the first place.22:29
*** panbalag has quit IRC22:30
eikketorgomatic: I'd be in the "Let's not bring such things to our events"-camp22:31
*** badgas has quit IRC22:31
*** gyee has joined #openstack-swift22:49
*** jkugel has quit IRC22:56
*** foexle has quit IRC22:58
openstackgerritClay Gerrard proposed openstack/python-swiftclient: add functional tox target  https://review.openstack.org/15671923:00
openstackgerritThiago da Silva proposed openstack/swift: WIP: initial pass at refactoring the PUT method  https://review.openstack.org/15682523:05
tdasilvaclayg: ^^^ my first attempt at adding dependencies in gerrit, messy!23:07
*** chlong has joined #openstack-swift23:08
*** joeljwright has quit IRC23:10
*** doxavore has quit IRC23:25
*** dmsimard is now known as dmsimard_away23:30
*** Nadeem has quit IRC23:35
*** jrichli has quit IRC23:38
*** jrichli has joined #openstack-swift23:38
*** madhuri_ has joined #openstack-swift23:43
*** jrichli has quit IRC23:50
tdasilvanotmyname: ever been to Zazie in SFO?23:51
notmynametdasilva: nope23:51
notmynamewhat is it?23:52
tdasilvarestaurant, it was on the news today23:52
tdasilvathey pay full benefits to their employees23:52
tdasilvaand I think their website just went down :P23:53
notmynameyup. seems that way :-)23:53
*** abhirc has quit IRC23:53

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!