| *** annegent_ has joined #openstack-swift | 00:03 | |
| *** jdag__ has quit IRC | 00:03 | |
| *** jdag___ has joined #openstack-swift | 00:06 | |
| *** annegent_ has quit IRC | 00:08 | |
| *** dmsimard_away is now known as dmsimard | 00:09 | |
| *** dmsimard is now known as dmsimard_away | 00:10 | |
| taras_ | redbo: are you there? | 00:11 |
|---|---|---|
| taras_ | i did some more testing | 00:11 |
| *** ekarlso- has joined #openstack-swift | 00:11 | |
| *** swat30_ has joined #openstack-swift | 00:11 | |
| taras_ | on smaller dbs index doesnt really help | 00:11 |
| taras_ | on big ones it helps a lot | 00:11 |
| taras_ | but it uses up 30% of the db here | 00:11 |
| *** stenchlarge has quit IRC | 00:12 | |
| *** swat30 has quit IRC | 00:12 | |
| *** ekarlso has quit IRC | 00:12 | |
| *** thurloat has quit IRC | 00:12 | |
| *** esmute has quit IRC | 00:12 | |
| *** esmute has joined #openstack-swift | 00:12 | |
| *** swat30_ is now known as swat30 | 00:12 | |
| *** annegent_ has joined #openstack-swift | 00:12 | |
| *** annegent_ has quit IRC | 00:12 | |
| *** thurloat has joined #openstack-swift | 00:13 | |
| taras_ | redbo: however one could simulate a hash index with a trigger a custom hash func and another table | 00:14 |
| tab_ | how is the thing with multi-site data replication within Swift storage policy. If I put 2 replicas of data in US-region datacetner and 2 replicas of the same data into EU-region data center. Are both geo regions writable or is the second ("backup") region in EU just for read purposes? | 00:19 |
| taras_ | though that only cuts ~10% | 00:19 |
| mattoliverau | tab_: both regions are writable | 00:22 |
| taras_ | i think the proper thing to do is to implement hash indexes in sqlite and get it over with :( | 00:22 |
| taras_ | redbo: the other thing to look into is to have a bimodal access pattern | 00:25 |
| taras_ | not use the index until some threshold is crossed by db | 00:25 |
| taras_ | fadvise for dbs under a few mb | 00:25 |
| taras_ | and use indexes for bigger ones | 00:25 |
| *** dmorita has joined #openstack-swift | 00:27 | |
| mattoliverau | tab_: If have a 2 region cluster (us and eu) swift will place replicas on both regions, as it will try and keep the repicas as seperated as it can, first by regon then zone, then drive, etc. You can tweak these by giving wieghts to different regions etc. | 00:28 |
| taras_ | err nm | 00:28 |
| taras_ | now i cant reproduce index-less scans being slower | 00:29 |
| taras_ | nm, was using wrong file | 00:30 |
| torgomatic | tab_: regions, zones, servers... it's all sort of the same thing, just getting higher latency as you go | 00:31 |
| openstackgerrit | Billy Olsen proposed a change to openstack/swift: Fix getaddrinfo if dnspython is installed. https://review.openstack.org/116618 | 00:31 |
| redbo | eh.. on SSDs, using the index is way faster than not using the index. | 00:35 |
| redbo | http://paste.openstack.org/show/105499/ | 00:37 |
| tab_ | torgomatic, mattoliverau: thx for answers. | 00:37 |
| openstackgerrit | Billy Olsen proposed a change to openstack/swift: Fix getaddrinfo if dnspython is installed. https://review.openstack.org/116618 | 00:39 |
| taras_ | redbo: depends | 00:46 |
| taras_ | it's faster for large db sizes | 00:46 |
| taras_ | it's identical for db sizes of a few mb | 00:46 |
| taras_ | while being 30% smaller and more likely to fit into ram | 00:46 |
| *** esmute has quit IRC | 00:50 | |
| *** esmute has joined #openstack-swift | 00:50 | |
| tab_ | let's say I have 10 disks in the system, and only 6 of them in use (formated xfs and ready to go). In case i suddenly loss 2 or 3 disks, wil the rest of 4 disks out of then (that are not in formated and in use) become automatically available for further swift replication proces - or could this be solved only manual or by some shell script? | 00:57 |
| clayg | tab_: if they're not in the rings they're not getting used | 00:58 |
| clayg | tab_: if you have unformatted drives in the rings they'll get error limited pretty fast :\ | 00:59 |
| *** tgohad has joined #openstack-swift | 01:00 | |
| clayg | tab_: I think the short answer is that swift doesn't do anything like that; you'd format and add capacity in the rings as it comes online and swift will start replicating and evenly distributing data to use the new capacity fairly | 01:01 |
| *** tgohad__ has joined #openstack-swift | 01:03 | |
| *** tgohad has quit IRC | 01:03 | |
| *** tsg has quit IRC | 01:03 | |
| tab_ | clayg: thx. maybe also this question for you. Is Swift write speed mostly dependant on the way sqlite works? -> http://www.sqlite.org/wal.html, it says "Write transactions are very fast since they only involve writing the content once (versus twice for rollback-journal transactions) and because the writes are all sequential. " | 01:05 |
| tab_ | But there is problem on read, when this sqlite databse file is getting biger: "On the other hand, read performance deteriorates as the WAL file grows in size " , how is this solved in Swift? | 01:07 |
| redbo | We don't use WAL journaling. We should look at it, though. The answer to that problem is to force wal checkpoints before the wal file gets too big. | 01:08 |
| *** addnull has joined #openstack-swift | 01:09 | |
| tab_ | What is than in use in Swift? | 01:09 |
| tab_ | default automatic commit? | 01:10 |
| redbo | yeah, it just uses delete journaling, which I think is the default. I don't think we've really benchmarked any others very well. | 01:11 |
| clayg | well, there no easy answer for what is the "write bottleneck" in swift - it depends totally on your deployment and work load - did I miss some context where we're specifically looking at how fast a container server can eat object PUT's? | 01:12 |
| *** bill_az_ has quit IRC | 01:12 | |
| redbo | We tried WAL out a couple of years ago when it was really new, and we did have problems with the WAL file getting big under heavily concurrent writes. We couldn't seem to force it to checkpoint. Not sure if that's been changed or not. | 01:12 |
| *** tgohad__ has quit IRC | 01:13 | |
| tab_ | clayg: No, you are not missing the context. I would just like to understand the how is Swift in advantage over strongly consistency SDS systems. Reading the Joe Arnolds book, it says: "To focus on availability over consistency, Swift has no transaction or locking delays." , meaning that Swift some how should be faster ..... | 01:19 |
| *** tsg has joined #openstack-swift | 01:32 | |
| *** tsg has quit IRC | 01:41 | |
| *** tsg has joined #openstack-swift | 01:41 | |
| *** pitterfumped has joined #openstack-swift | 01:44 | |
| *** pitterfumped has quit IRC | 01:45 | |
| clayg | tab_: well, idk, you might have to ask joearnold - i don't want to put words in his mouth. But if you're talking about the tradeoffs of leaning on the A; it's almost always in the context of a failure scenario or some kind. | 01:47 |
| clayg | here it seems like we're talking about how fast a single container db can store object updates - and that's not nessecarily represenative of the aggregate scaling story for the storage system as a whole | 01:48 |
| *** tab_ has quit IRC | 01:50 | |
| *** HenryG has quit IRC | 01:51 | |
| *** nosnos has joined #openstack-swift | 01:54 | |
| *** dmsimard_away is now known as dmsimard | 01:57 | |
| *** addnull has quit IRC | 02:07 | |
| *** dmsimard is now known as dmsimard_away | 02:09 | |
| *** theanalyst has quit IRC | 02:13 | |
| *** theanalyst has joined #openstack-swift | 02:15 | |
| openstackgerrit | Matthew Oliver proposed a change to openstack/swift: Add concurrent reads option to proxy https://review.openstack.org/117710 | 02:33 |
| *** HenryG has joined #openstack-swift | 02:40 | |
| *** addnull has joined #openstack-swift | 02:42 | |
| zaitcev | https://ask.openstack.org/en/questions/scope:all/sort:activity-desc/tags:swift/page:1/ - this is all kinds of horrible | 02:44 |
| clayg | zaitcev: hey you've been active! i was doing that for awhile... | 02:46 |
| *** addnull has quit IRC | 02:52 | |
| *** addnull has joined #openstack-swift | 02:53 | |
| clayg | zaitcev: wait... what part is all kinds of horrible? | 02:54 |
| zaitcev | clayg: the desperation of people who look for information | 02:55 |
| *** addnull has quit IRC | 02:58 | |
| *** tsg has quit IRC | 02:58 | |
| *** addnull has joined #openstack-swift | 03:04 | |
| *** tongli has quit IRC | 03:05 | |
| *** Sanchit has joined #openstack-swift | 04:12 | |
| Sanchit | Hi. I am trying to put a container in an account. Just wanted to get some understanding about the X-timestamp (created_at) field and put_timestamp field in account_stat info db file. | 04:15 |
| Sanchit | Which timestamp remains intact and which timestamp changes when performing operations on an account? | 04:15 |
| *** addnull has quit IRC | 04:16 | |
| *** tkay has joined #openstack-swift | 04:16 | |
| clayg | zaitcev: sounds mean when you say it like that | 04:20 |
| clayg | Sanchit: you could poke around at it with swift-[account|container]-info; but as a general rule if it's not exposed in the API it's subject to change (and in fact recently did, at least for containers, wrt storage policies) | 04:23 |
| clayg | Sanchit: that is to say - why do you ask :D | 04:23 |
| *** morganfainberg_Z is now known as morganfainberg | 04:29 | |
| clayg | Sanchit: maybe you were just asking who fills in the X-Timestamp header on a container GET/HEAD and that would be a top of container/server.py - it comes from the created_at field in the db | 04:29 |
| Sanchit | clayg: I am very new to this development environment. I am accessing account_stat, container_stat using sqlite3 from the db file. While all the other fileds have updated, the Put-Timestamp hasn't changed even after adding containers and objects within. | 04:29 |
| clayg | portante: k, i got my lwn article all set up | 04:30 |
| clayg | Sanchit: ok | 04:30 |
| Sanchit | clayg: So I just wanted to know what does these timestamp actually depict? | 04:30 |
| clayg | Sanchit: when the account was PUT probably | 04:30 |
| clayg | at least I wouldn't guess it to get updated with every update from the container-updater - but who knows | 04:32 |
| Sanchit | clayg: Although I have figured out that X-Timestamp is the timestamp when the account was actually created, I am still unclear about the put_timestamp field. | 04:33 |
| clayg | well... that's not exposed in the API | 04:35 |
| clayg | but it happens to get updated when a PUT request for an account hits the account server FWIW | 04:35 |
| clayg | Sanchit: I'm about to head out; I might be more helpful if I understand what you were acctually trying to get at - rather than chipping around at the edges | 04:37 |
| clayg | Sanchit: but undirected learning is good too! just saying I'm about to have to bolt | 04:37 |
| clayg | ;) | 04:37 |
| Sanchit | clayg: I am myself unclear at the moment but still Thanks for replying and helping in the best way possible | 04:38 |
| clayg | heh - good luck! feel free to keep posting in channel here while you look around at it - maybe someone will be on later, or some non US folks come online, or we'll see it and responding in the morning | 04:39 |
| clayg | g'night | 04:39 |
| Sanchit | clayg: Sure thing. Have a good Sleep! | 04:39 |
| *** tkay has quit IRC | 04:53 | |
| *** echevemaster has quit IRC | 05:02 | |
| *** ppai has joined #openstack-swift | 05:05 | |
| *** zaitcev has quit IRC | 05:22 | |
| *** addnull has joined #openstack-swift | 05:28 | |
| *** nshaikh has joined #openstack-swift | 05:31 | |
| *** ppai has quit IRC | 05:42 | |
| *** ppai has joined #openstack-swift | 05:57 | |
| *** ttrumm has joined #openstack-swift | 06:08 | |
| *** ppai has quit IRC | 06:08 | |
| *** ttrumm_ has joined #openstack-swift | 06:16 | |
| *** ttrumm has quit IRC | 06:18 | |
| *** nshaikh has quit IRC | 06:19 | |
| *** ppai has joined #openstack-swift | 06:21 | |
| *** bvandenh has joined #openstack-swift | 06:33 | |
| *** kopparam has joined #openstack-swift | 06:34 | |
| *** ttrumm has joined #openstack-swift | 06:46 | |
| *** ttrumm_ has quit IRC | 06:50 | |
| *** nshaikh has joined #openstack-swift | 06:54 | |
| *** joeljwright has joined #openstack-swift | 06:58 | |
| openstackgerrit | Matthew Oliver proposed a change to openstack/swift: Add concurrent reads option to proxy https://review.openstack.org/117710 | 07:04 |
| *** ttrumm_ has joined #openstack-swift | 07:13 | |
| *** ttrumm has quit IRC | 07:16 | |
| *** addnull has quit IRC | 07:23 | |
| *** homegrown has joined #openstack-swift | 07:32 | |
| *** geaaru has joined #openstack-swift | 07:48 | |
| *** Sanchit has quit IRC | 08:25 | |
| *** addnull has joined #openstack-swift | 08:34 | |
| *** addnull has quit IRC | 08:38 | |
| *** addnull has joined #openstack-swift | 08:57 | |
| *** foexle has joined #openstack-swift | 08:58 | |
| *** mkollaro has joined #openstack-swift | 09:10 | |
| mkollaro | does anyone know how to debug the S3 tests in tempest? it's just giving me the cryptic message "S3ResponseError: 412 Precondition Failed\nNone" in create_bucket | 09:14 |
| mkollaro | and I have no clue as to what to do with it | 09:14 |
| *** jdag___ has quit IRC | 09:51 | |
| *** jdag___ has joined #openstack-swift | 09:53 | |
| clayg | mkollaro: can you read the body of the response? | 09:55 |
| mkollaro | clayg: this is all I'm getting http://pastebin.com/zAyNE0HG | 09:57 |
| *** dmsimard_away is now known as dmsimard | 10:03 | |
| *** addnull has quit IRC | 10:05 | |
| *** dmsimard is now known as dmsimard_away | 10:06 | |
| *** nosnos has quit IRC | 10:14 | |
| *** nosnos has joined #openstack-swift | 10:15 | |
| *** nosnos_ has joined #openstack-swift | 10:18 | |
| *** nosnos has quit IRC | 10:19 | |
| openstackgerrit | A change was merged to openstack/swift: code shuffle post expired headers refactor https://review.openstack.org/112804 | 10:20 |
| *** k4n0_ has quit IRC | 10:33 | |
| *** k4n0 has joined #openstack-swift | 10:35 | |
| *** dmsimard_away is now known as dmsimard | 10:36 | |
| *** aix has joined #openstack-swift | 10:42 | |
| *** erlon has joined #openstack-swift | 10:46 | |
| *** dmorita has quit IRC | 10:58 | |
| *** ttrumm has joined #openstack-swift | 11:09 | |
| *** bkopilov has quit IRC | 11:12 | |
| *** ttrumm_ has quit IRC | 11:13 | |
| *** ppai has quit IRC | 11:22 | |
| *** bkopilov has joined #openstack-swift | 11:29 | |
| *** ppai has joined #openstack-swift | 11:38 | |
| *** pandemicsyn2 has joined #openstack-swift | 11:49 | |
| *** acorwin_ has joined #openstack-swift | 11:50 | |
| *** jroll|dupe has joined #openstack-swift | 11:50 | |
| *** rpedde_ has joined #openstack-swift | 11:50 | |
| *** jroll has quit IRC | 11:50 | |
| *** acorwin has quit IRC | 11:51 | |
| *** rpedde has quit IRC | 11:51 | |
| *** pandemicsyn has quit IRC | 11:51 | |
| *** rpedde_ is now known as rpedde | 11:51 | |
| *** jroll|dupe is now known as jroll | 11:51 | |
| *** ttrumm_ has joined #openstack-swift | 11:52 | |
| *** ttrumm has quit IRC | 11:55 | |
| *** tongli has joined #openstack-swift | 11:55 | |
| *** xrandr has joined #openstack-swift | 12:26 | |
| xrandr | Hi, I was wondering if OpenStack Swift can also connect to Google Drive, Drop Box, and BOX.NET the way that ownCloud can? | 12:27 |
| *** mrsnivvel has joined #openstack-swift | 12:31 | |
| *** dosaboy_ is now known as dosaboy | 12:38 | |
| *** tab_ has joined #openstack-swift | 12:44 | |
| *** vr2 has joined #openstack-swift | 12:49 | |
| *** kopparam has quit IRC | 12:55 | |
| *** miqui has joined #openstack-swift | 12:56 | |
| *** kenhui has joined #openstack-swift | 12:58 | |
| *** tsg has joined #openstack-swift | 12:59 | |
| *** foexle_ has joined #openstack-swift | 13:00 | |
| *** foexle has quit IRC | 13:04 | |
| *** tsg has quit IRC | 13:10 | |
| *** ppai has quit IRC | 13:10 | |
| *** k4n0 has quit IRC | 13:12 | |
| btorch | xrandr: what what I see on https://owncloud.org/ that seems to be some sync and share software that has the ability, it seems, to connect to other storage like S3 | 13:18 |
| btorch | xrandr: so should the question not be if ownCloud can connect to openstack-swift ? I don't see it in the list of External Storage support | 13:19 |
| *** bvandenh has quit IRC | 13:23 | |
| *** bvandenh has joined #openstack-swift | 13:23 | |
| *** tdasilva has joined #openstack-swift | 13:28 | |
| tab_ | btorch: you can use Swift as a backend for owncloud: http://blog.adityapatawari.com/2014/01/using-openstack-swift-as-owncloud.html | 13:45 |
| *** tsg has joined #openstack-swift | 13:46 | |
| btorch | tab_: cool | 13:47 |
| tab_ | btorch: also this: http://doc.owncloud.org/server/6.0/admin_manual/configuration/custom_mount_config.html | 13:48 |
| *** nosnos_ has quit IRC | 13:50 | |
| *** nosnos has joined #openstack-swift | 13:51 | |
| *** portante has quit IRC | 13:52 | |
| *** nosnos has quit IRC | 13:55 | |
| *** ttrumm_ has quit IRC | 14:05 | |
| *** themadcanudist has joined #openstack-swift | 14:07 | |
| *** peluse_ has joined #openstack-swift | 14:08 | |
| *** annegent_ has joined #openstack-swift | 14:09 | |
| themadcanudist | hey guys… does the openstack proxy server issue multiple calls to the storage backend and log them independently? | 14:13 |
| themadcanudist | that's what it appears in my proxy-server.log | 14:13 |
| *** cschwede_ has joined #openstack-swift | 14:13 | |
| vr2 | themadcanudist: what do you mean by "log them independently" ? | 14:16 |
| *** mkollaro has quit IRC | 14:17 | |
| *** ekarlso- has quit IRC | 14:17 | |
| *** peluse has quit IRC | 14:17 | |
| *** cschwede has quit IRC | 14:17 | |
| *** lpabon has quit IRC | 14:17 | |
| *** annegentle has quit IRC | 14:17 | |
| *** pconstantine_ has quit IRC | 14:17 | |
| themadcanudist | two requests (with slightly different time-to-completion fields) | 14:18 |
| themadcanudist | 1 external request generates two log lines | 14:19 |
| themadcanudist | like the proxy is going to two storage servers? | 14:19 |
| vr2 | possible if you have 2 copies | 14:19 |
| themadcanudist | That's odd | 14:20 |
| themadcanudist | sec | 14:20 |
| themadcanudist | only one call to object server | 14:20 |
| themadcanudist | or one log line in object log | 14:20 |
| themadcanudist | but two lines with diff times in proxy-log | 14:20 |
| vr2 | can you print the 2 lines ? | 14:20 |
| themadcanudist | hrm, it's all sensitive info | 14:21 |
| themadcanudist | anything in particular you're looking for? | 14:21 |
| *** pconstantine_ has joined #openstack-swift | 14:22 | |
| *** ekarlso- has joined #openstack-swift | 14:22 | |
| vr2 | PUT, GET DELETE ? | 14:23 |
| themadcanudist | HEAD | 14:23 |
| *** portante has joined #openstack-swift | 14:23 | |
| *** ChanServ sets mode: +v portante | 14:24 | |
| *** mkollaro has joined #openstack-swift | 14:25 | |
| themadcanudist | it's so strange. all the calls get double logged | 14:25 |
| themadcanudist | same tx id's | 14:25 |
| themadcanudist | only one entry in object-server.log though | 14:25 |
| tdasilva | themadcanudist: I might be missing something, but the request would go to two different object-servers | 14:26 |
| vr2 | and with a GET ? | 14:26 |
| themadcanudist | all calls | 14:26 |
| themadcanudist | GET/HEAD | 14:26 |
| ahale | proxy head would only go to one object server, as long as it replied quick and wasnt an x-newest ? | 14:26 |
| ahale | even then it would only log once right? and have a error log entry for failing to hit object | 14:27 |
| themadcanudist | would proxy-logging in two spots of my pipeline:main be the issue? | 14:27 |
| tdasilva | ahale: sorry, didn't see themadcanudist it was a HEAD request | 14:27 |
| tdasilva | *mention it was a HEAD request | 14:28 |
| vr2 | possible it is a log configuration issue | 14:28 |
| themadcanudist | vr2 ^ what I posted about having proxy-logging in two spots in my main:pipeline? | 14:29 |
| themadcanudist | that could be it, no? | 14:29 |
| *** kenhui has quit IRC | 14:29 | |
| *** kenhui has joined #openstack-swift | 14:30 | |
| ahale | probably not, its meant to be there twice | 14:30 |
| themadcanudist | hrm | 14:32 |
| *** 17SAA4BEV has joined #openstack-swift | 14:33 | |
| *** peluse has joined #openstack-swift | 14:33 | |
| *** cschwede has joined #openstack-swift | 14:33 | |
| *** annegentle has joined #openstack-swift | 14:33 | |
| *** sendak.freenode.net sets mode: +v peluse | 14:33 | |
| *** 17SAA4BEV has quit IRC | 14:33 | |
| *** annegentle_ has joined #openstack-swift | 14:33 | |
| *** peluse has quit IRC | 14:34 | |
| *** cschwede has quit IRC | 14:34 | |
| *** annegentle has quit IRC | 14:34 | |
| *** annegentle_ is now known as Guest18621 | 14:34 | |
| *** bkopilov has quit IRC | 14:34 | |
| *** lpabon has joined #openstack-swift | 14:34 | |
| *** annegent_ has quit IRC | 14:37 | |
| *** annegentle has joined #openstack-swift | 14:38 | |
| vr2 | I'm running the functional tests and it seems there is some kind of state (the setup tries to erase old files). Is it possible to force the start from a clean state ? | 14:38 |
| peluse_ | vr2, run resetswift before you start | 14:40 |
| vr2 | where ? | 14:40 |
| peluse_ | its a script provided as part of the SAIO setup, I assume you are using Swift All In One? | 14:41 |
| vr2 | no I'm using devstack | 14:41 |
| peluse_ | vr2, OK, I've not tried that.... | 14:41 |
| vr2 | it seems dangerous since it wipes the storage | 14:41 |
| peluse_ | well, its for a development env | 14:41 |
| vr2 | I'm just interested to know if there is a state in the functional test framework | 14:42 |
| vr2 | because it is trying to erase files which were created in previous runs | 14:42 |
| vr2 | (it seems) | 14:42 |
| peluse_ | read towards the bottom of http://docs.openstack.org/developer/swift/development_saio.html | 14:42 |
| vr2 | it seems functests delete everything in the configured accounts | 14:43 |
| peluse_ | yes | 14:43 |
| vr2 | how to avoid that ? | 14:43 |
| *** peluse_ is now known as peluse | 14:44 | |
| vr2 | without wiping the store | 14:44 |
| *** themadcanudist has left #openstack-swift | 14:44 | |
| vr2 | I assume it is not possible | 14:44 |
| peluse | its not how the func tests are used, they're meant for testing in a dev env | 14:44 |
| *** ChanServ sets mode: +v peluse | 14:44 | |
| vr2 | but there is an option --failed in functests (nosetest) | 14:46 |
| vr2 | which replays failed tests | 14:46 |
| vr2 | so there is a "memory" somewhere | 14:47 |
| *** kopparam has joined #openstack-swift | 14:48 | |
| peluse | vr2, DK off the top of my head, log file perhaps somewhere there off of where you run it | 14:49 |
| *** kopparam has quit IRC | 14:51 | |
| *** bkopilov has joined #openstack-swift | 14:51 | |
| *** Dafna has quit IRC | 14:52 | |
| *** kopparam has joined #openstack-swift | 14:54 | |
| notmyname | good morning | 14:56 |
| notmyname | vr2: that "memory" is either in .tox or .testrepository | 14:56 |
| vr2 | ok thanks | 14:56 |
| *** tsg has quit IRC | 14:56 | |
| vr2 | I applied a "more radical" approach | 14:56 |
| *** mahatic has joined #openstack-swift | 14:56 | |
| *** acorwin_ is now known as acorwin | 14:56 | |
| *** sandywalsh has quit IRC | 14:57 | |
| *** sandywalsh has joined #openstack-swift | 14:57 | |
| *** nshaikh has quit IRC | 14:59 | |
| peluse | notmyname, good morning | 15:00 |
| *** kenhui has quit IRC | 15:04 | |
| *** kenhui has joined #openstack-swift | 15:04 | |
| *** dencaval has joined #openstack-swift | 15:08 | |
| *** kenhui has quit IRC | 15:09 | |
| *** kenhui has joined #openstack-swift | 15:10 | |
| acoles | notmyname: re gvernik's migration patch https://review.openstack.org/#/c/64430/ ... | 15:10 |
| notmyname | acoles: what about it? | 15:12 |
| acoles | notmyname: as it stands if one drops the middleware in pipeline with conf in proxy-server.conf-sample, then any user can copy files off proxy server filesystem | 15:12 |
| acoles | notmyname: which some might say is undesirable ! | 15:12 |
| notmyname | ouch | 15:12 |
| acoles | notmyname: so wondering how we document/present default config so folks are aware before they deploy? | 15:12 |
| acoles | notmyname: is it sufficient to have the filesystem driver commented out in sample config? so you need to explicitly configure it to be active? | 15:14 |
| * acoles likes to avoid accidents | 15:14 | |
| tdasilva | acoles: in addition to commenting out, there should also be some kind of warning message above that line, what do you think? | 15:18 |
| acoles | tdasilva: agree | 15:20 |
| notmyname | acoles: is there no configured root that all fs migrations have to ba anchored at? eg you can migrate any file on the local filesystem as long as its path starts with /foo/bar/baz | 15:22 |
| notmyname | and strip out '..' too | 15:22 |
| acoles | notmyname: not currently - i had same thought | 15:23 |
| acoles | notmyname: yeah, strip .. ;) | 15:23 |
| acoles | but i keep all my passwords under /foo/bar/baz ;) | 15:23 |
| dencaval | hey guys, would it be a possible scenario to configure one-to-one user/container with read/write permission? For example, if I have 100 containers, I would have 100 users and each of them would have access only to its container? | 15:23 |
| *** kenhui has quit IRC | 15:24 | |
| *** kenhui has joined #openstack-swift | 15:25 | |
| tdasilva | acoles, notmyname: I think gvernik's original idea was to have a root dir, but it was later removed | 15:27 |
| tdasilva | still, it was set as a header and not part of the config file | 15:27 |
| acoles | tdasilva: yes, the 'path' - but as you say that was set in header by user so no restriction | 15:29 |
| tdasilva | exactly, it would cause same issue... | 15:29 |
| acoles | i'm wondering what a suitable default anchor dir would be, i figure there'd need to be a default value in the conf | 15:30 |
| notmyname | acoles: I'm ok with making that anchor directory required and not putting in a default | 15:31 |
| acoles | notmyname: yup, so an explicit step needed to open up access | 15:32 |
| notmyname | right | 15:32 |
| acoles | notmyname: tdasilva: so require anchor dir AND comment out fsystem driver, or just require anchor dir to be configured? | 15:33 |
| tdasilva | acoles: i would comment out both lines and add a little msg above, otherwise it will cause an error if they don't define the anchor dir | 15:35 |
| * tdasilva is checking other swift config files for similarities | 15:36 | |
| notmyname | acoles: you should be required to set the drivers and the if using the fsdriver, you should set the anchor path | 15:36 |
| acoles | dencaval: are you using keystone auth? | 15:37 |
| acoles | notmyname: tdasilva: cool, consensus - i'll feed that back to gvernik on gerrit | 15:38 |
| tdasilva | acoles: another topic on migration middleware... | 15:38 |
| *** joeljwright has quit IRC | 15:38 | |
| acoles | tdasilva: sure | 15:39 |
| tdasilva | I was thinking a little bit further on that idea of a future driver for same cluster container migration, and I think it would make sense to make use of COPY request for that | 15:40 |
| tdasilva | the issue, is that the current driver API requires a get_object to return the data and the migration middleware handles the putting the object in the new container | 15:41 |
| tdasilva | so I was wondering if it would make sense to move the upload_object to the DataMigrationDriver class | 15:42 |
| tdasilva | so that all data migration is handled by the driver | 15:42 |
| tdasilva | and the middleware is more of a controller | 15:42 |
| *** tsg has joined #openstack-swift | 15:44 | |
| tdasilva | acoles: does that make any sense? it was just an idea.... | 15:44 |
| *** themadcanudist has joined #openstack-swift | 15:45 | |
| acoles | tdasilva: yes it makes sense, same had occurred to me but not thought it through yet | 15:46 |
| themadcanudist | hrm, i'm having some really odd behaviour here, where when tracing down these swift transactions that take really long, 10 sec+, and return a 499, the object-server log shows that it found the object REALLY fast… What's the best way I can trace down why the proxy-server may be taking too long, when the object servers are super-fast? | 15:46 |
| *** geaaru has quit IRC | 15:46 | |
| acoles | tdasilva: do you think it can be refactored after this patch lands? | 15:47 |
| notmyname | themadcanudist: 499 is the catch-all for client disconnects, so if the object server is fast, the things to look at are the proxy cpu or netowrk | 15:47 |
| * themadcanudist nods | 15:48 | |
| themadcanudist | notmyname: is there some technique/tool so that I may be able to track it down more effectively and possibly definitively? | 15:50 |
| themadcanudist | network and cpu seem fine | 15:51 |
| themadcanudist | it's buys, but nowhere near capacity | 15:51 |
| tdasilva | acoles: I'm just trying to think of how "big problem" it is right now...I keep trying to come up with other examples of driver that would have the same issues. Basically if we know we definetely need to do this, then I'd say let's get it right the first time | 15:54 |
| tdasilva | if we can come up with other possible solutions or if this is a corner case, then it can and probably should wait for a future patch | 15:54 |
| notmyname | themadcanudist: can you reproduce the issue on-demand? | 15:56 |
| themadcanudist | i can't. I see it happeneing to external clients because there is so much traffic, but when I try to retrieve the same assets, they're very fast for me =( | 15:56 |
| *** geaaru has joined #openstack-swift | 16:00 | |
| acoles | tdasilva: looking at it, maybe its a pretty simple refactor now we have the base class | 16:00 |
| acoles | tdasilva: will think more about other use cases | 16:00 |
| *** mahatic has quit IRC | 16:01 | |
| themadcanudist | notmyname: I suppose I'm just looking for a way to identify if the proxy is stuck at keystone, object-get/head/put via object-server, some other operation, or network transfer? | 16:03 |
| notmyname | themadcanudist: I was just looking back through some email. something you said reminded me of something clayg said in a review recently | 16:03 |
| themadcanudist | ah | 16:03 |
| themadcanudist | thanks! | 16:03 |
| *** mahatic has joined #openstack-swift | 16:03 | |
| notmyname | themadcanudist: something along the lines of sending a 404 back and then the client not reading the body and it gets logged as a 499. I don't remember the exact details | 16:04 |
| themadcanudist | hrm… interesting | 16:04 |
| notmyname | themadcanudist: so maybe, just maybe, the client is intentionally dropping the connection after getting a certain status code, and that leads to a timeout and 499 on the server side | 16:06 |
| themadcanudist | ah, gotcha | 16:06 |
| themadcanudist | that is a possibility | 16:06 |
| notmyname | themadcanudist: that answer has the convenient ideas of both being reasonable and calling it the client problem ;-) | 16:08 |
| notmyname | themadcanudist: which is kinda scary, because it may or may not be the answer | 16:08 |
| themadcanudist | lol | 16:09 |
| themadcanudist | ok, how about this one | 16:09 |
| mahatic | hello, is there any reason the file system suggested in installing swift is XFS? | 16:09 |
| themadcanudist | my swift proxy-server is logging every transaction twice, the only difference being the transaction time field at the end, they're different. the object-server logs the request once. | 16:09 |
| themadcanudist | i'm concerned, because sometimes the xaction time is significantly different.. ie. sub seconds for one, many seconds for the second one… it happens for all HTTP requests | 16:10 |
| notmyname | mahatic: yes. because xfs is the least-bad when it comes to lots and lots of inodes as the drive gets more full | 16:10 |
| notmyname | themadcanudist: can you paste your proxy pipeline? | 16:10 |
| themadcanudist | pipeline = catch_errors healthcheck proxy-logging cache swift3 tempurl s3token authtoken keystone proxy-logging proxy-server | 16:10 |
| mahatic | notmyname, okay | 16:11 |
| *** Snivilis has joined #openstack-swift | 16:11 | |
| notmyname | themadcanudist: what version of swift? | 16:12 |
| themadcanudist | folsom | 16:13 |
| themadcanudist | 1.7.4 | 16:13 |
| themadcanudist | sorry, 1.7.5 | 16:13 |
| *** Snivilis is now known as m|k|e | 16:13 | |
| notmyname | was proxy-logging supposed to be in the pipeline twice that long ago? maybe that was only done more recently | 16:14 |
| themadcanudist | hrm, possibly | 16:14 |
| themadcanudist | i may have mixed new docs with old install | 16:14 |
| notmyname | themadcanudist: yup, looks like it. proxy-logging twice was added in 1.8.0 | 16:15 |
| themadcanudist | keep the earlier or later one? | 16:15 |
| notmyname | themadcanudist: I don't remember. let me check | 16:16 |
| themadcanudist | kk, that did work. I kept the second last one, seems to have fixed the issue | 16:16 |
| notmyname | themadcanudist: the last one | 16:16 |
| themadcanudist | hrm, there is utility to this. I can see how long the operations in the pipeline took =) | 16:17 |
| notmyname | themadcanudist: also, because this is an open-source project and we're chatting on IRC, don't the "rules" say I have to tell you to upgrade? :-) | 16:18 |
| *** mwstorer has joined #openstack-swift | 16:20 | |
| acoles | clayg: re 103777 ... aaah, thanks for spelling it out! just tried against cluster. 201 is crazy too though ;) | 16:23 |
| *** kenhui has quit IRC | 16:24 | |
| *** gyee has joined #openstack-swift | 16:25 | |
| *** tkay has joined #openstack-swift | 16:26 | |
| *** tab__ has joined #openstack-swift | 16:27 | |
| *** vr2 has quit IRC | 16:32 | |
| *** jergerber has joined #openstack-swift | 16:32 | |
| *** bsdkurt has joined #openstack-swift | 16:37 | |
| *** annegent_ has joined #openstack-swift | 16:37 | |
| *** annegentle has quit IRC | 16:38 | |
| *** annegent_ has quit IRC | 16:39 | |
| notmyname | peluse: yesterday you said you were going to work on the trello board. have you had a chance yet? | 16:50 |
| peluse | notmyname, will check it out right now... | 16:51 |
| peluse | notmyname, OK, its up to date. There are a few cards getting close to moving to the right but not there yet... | 16:56 |
| notmyname | peluse: thanks | 16:56 |
| peluse | tsg, when you get a chance pelase scrub the PyECLib cards and update as needed. thanks | 16:56 |
| *** kopparam has quit IRC | 16:56 | |
| *** kopparam has joined #openstack-swift | 16:57 | |
| *** mahatic has quit IRC | 16:57 | |
| tsg | peluse: i have a couple of updates, will get those posted shortly | 16:59 |
| *** mahatic has joined #openstack-swift | 17:00 | |
| *** chuck_ has joined #openstack-swift | 17:05 | |
| *** shri has joined #openstack-swift | 17:07 | |
| *** tsg has quit IRC | 17:07 | |
| *** tsg has joined #openstack-swift | 17:09 | |
| *** kopparam has quit IRC | 17:10 | |
| *** bvandenh has quit IRC | 17:10 | |
| *** kopparam has joined #openstack-swift | 17:11 | |
| *** kopparam has quit IRC | 17:15 | |
| tdasilva | acoles: sounds good. I've been thinking about writing another driver just for testing purposes, just haven't gotten around to it yet | 17:17 |
| *** chuck_ has quit IRC | 17:19 | |
| *** aix has quit IRC | 17:19 | |
| *** annegentle has joined #openstack-swift | 17:23 | |
| *** zaitcev has joined #openstack-swift | 17:26 | |
| *** ChanServ sets mode: +v zaitcev | 17:26 | |
| themadcanudist | notmyname: LOL, re: opensource and telling me to upgrade | 17:29 |
| notmyname | themadcanudist: :-) | 17:29 |
| themadcanudist | you would believe it man… I get that in ALL OSS channels I go to. | 17:29 |
| themadcanudist | =D | 17:29 |
| themadcanudist | it's hilarious | 17:29 |
| notmyname | themadcanudist: "hilarious" | 17:29 |
| notmyname | themadcanudist: my favorite is the "your use case is invalid and you should do it differently" | 17:30 |
| themadcanudist | LOL | 17:30 |
| themadcanudist | i feel you are the only one on the internet who understands. | 17:30 |
| themadcanudist | based on those two comments | 17:30 |
| notmyname | swift: the friendly open-source project ;-) | 17:31 |
| themadcanudist | swift: we're self reflective too ;) | 17:31 |
| *** homegrown1 has joined #openstack-swift | 17:31 | |
| notmyname | themadcanudist: actually, speaking of upgrades, is there anything on my side that I can do to help you do upgrades? of course I prefer people to use later versions. sometimes people don't upgrade because of things in the upstream code. other times it's for internal reasons. I can help with the first, I hope | 17:33 |
| homegrown1 | Can anyone point me to the ordering of the middleware in swift? Currently have pipeline = pipeline = catch_errors healthcheck cache ratelimit swift3 s3token authtoken keystone cname_lookup domain_remap staticweb proxy-server | 17:35 |
| themadcanudist | well, what's the fear factor of upgrading from 1.7.5 to latest… in terms of object/container/account data consistency and explosion?! also, keystone 2012.4-> latest | 17:35 |
| notmyname | themadcanudist: I understand. of course, it gets scarier the longer you wait :-) | 17:35 |
| themadcanudist | no kidding | 17:35 |
| notmyname | homegrown1: what version of swift are you using? | 17:35 |
| themadcanudist | basically has anyone lost all their data ;) | 17:35 |
| themadcanudist | i can always upgrade in steps anyway, right. | 17:36 |
| notmyname | themadcanudist: you should read https://swiftstack.com/blog/2013/12/20/upgrade-openstack-swift-no-downtime/ | 17:36 |
| * themadcanudist nods | 17:36 | |
| themadcanudist | i have | 17:36 |
| themadcanudist | thanks! | 17:36 |
| notmyname | :-) | 17:36 |
| notmyname | themadcanudist: no, I don't know of anyone who has lost data stored in swift | 17:36 |
| notmyname | homegrown1: the sample proxy-server config file has a good example: https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L79 | 17:37 |
| themadcanudist | sweet, that's awesome! | 17:37 |
| homegrown1 | 2.03 | 17:37 |
| notmyname | homegrown1: hmm...that doesn't look like a version number that swift or swiftclient has released. Swift is currently at 2.1.0 | 17:39 |
| themadcanudist | notmyname: last question, is there anything in the upgrade path from 1.7.5 -> latest that would change the data/metadata in a way that downgrading would be impossible? | 17:39 |
| homegrown1 | notmyname: thanks, having a look. specifically interested in cname_lookup & domain_remap which seems to break things for me at the moment | 17:40 |
| notmyname | themadcanudist: I don't think so. of course, check the changelog file. the biggest change (and the reason we bumped the major version to 2) is because of storage policies. if you start using multiple storage policies and then downgrade, you can't get the data back. but aside from that, no | 17:40 |
| notmyname | themadcanudist: I'll go look at it too, just to check | 17:40 |
| themadcanudist | gotcha | 17:40 |
| themadcanudist | thanks dude! much appreciated | 17:40 |
| homegrown1 | notmyname: it was built using mirantis fuel (icehouse on ubuntu) | 17:41 |
| *** annegentle has quit IRC | 17:42 | |
| *** annegentle has joined #openstack-swift | 17:43 | |
| notmyname | themadcanudist: no, nothing that affects the data on disk. https://github.com/openstack/swift/blob/master/CHANGELOG | 17:44 |
| notmyname | themadcanudist: but there have been some config file changes. | 17:44 |
| themadcanudist | perfect | 17:44 |
| themadcanudist | sorry, one last question. just curious if there are any noteable efficiency/speed increases in the replicator/updater daemons? =) | 17:45 |
| notmyname | themadcanudist: wow. you don't have global clusters, static large objects, bulk requests, per-disk threadpools, /info | 17:45 |
| *** annegentle has quit IRC | 17:47 | |
| notmyname | themadcanudist: maybe | 17:47 |
| * themadcanudist nods | 17:48 | |
| themadcanudist | sweet! | 17:48 |
| notmyname | themadcanudist: eg there is now support for using a separate replication network so you can separately QoS it | 17:49 |
| notmyname | themadcanudist: and there is now support for running replication to move misplaced partitions first | 17:50 |
| themadcanudist | ah, cool | 17:50 |
| clayg | wow this default port thing is getting all chicken and the egg on us | 17:50 |
| clayg | notmyname: no wonder we ignored it for so long? | 17:50 |
| notmyname | themadcanudist: there are also some disk io improvements like using threadpools. this helps the whole server keep up performance when a drive is down | 17:51 |
| notmyname | clayg: what do you mean? | 17:51 |
| themadcanudist | yeah, i saw that | 17:51 |
| themadcanudist | i like | 17:51 |
| notmyname | clayg: did I miss something? | 17:51 |
| clayg | notmyname: i was just looking at the triplo comments on the patch about making bind_port required | 17:51 |
| notmyname | oh, I did miss that. /me goes to look | 17:52 |
| zaitcev | Tripple-O is possibly the only place that objects. They could've easily done the right thing months ago. There's a patch in their Gerrit for crissakes. We just need notmyname confront that l-something dude. | 17:52 |
| themadcanudist | notmyname: do you recommend I go through a few upgrades from 1.7.5->latest or just jump? | 17:52 |
| notmyname | themadcanudist: I was wondering that myself. if you're on 1.7.x, you might want to try 1.9.0 then 1.13.1 then 2.1.0. while I think going straight to current would probably work just fine, I think the important part is to have a checkpoint at 1.13.1 | 17:54 |
| zaitcev | Actually lifeless' comment was: "Making this a parameter to the overall deployment feels very wrong given that this is all discoverable via keystone." IOW he didn't like the extra knob. | 17:54 |
| zaitcev | And then rwsu folded up and didn't improve the patch as his PTL suggested. | 17:54 |
| themadcanudist | notmyname: good call | 17:54 |
| zaitcev | Or we could've had Tripple-O write bind_port long ago. | 17:54 |
| zaitcev | I looked at their code, but it's like Prolog to me. I cannot propose that patch in Richard's stead. | 17:55 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 17:55 |
| *** geaaru has quit IRC | 17:55 | |
| *** annegentle has joined #openstack-swift | 17:56 | |
| notmyname | clayg: zaitcev: ya, I don't really get those objections either | 17:57 |
| *** homegrown1 has quit IRC | 17:57 | |
| notmyname | clayg: about catching errors in the config file, I didn't because we don't do that for other required config variables | 17:59 |
| notmyname | clayg: wow, I just changed the max_clients setting in the proxy config to abc and that resulted in a whole lot of hurt | 17:59 |
| zaitcev | themadcanudist: I cannot remember, but could you find out if the format of memcached was changed away from pickle after 1.7.5 or in it or before? That change requires a special procedure: update packages, then flip the config, then restart once again (one by one). | 18:01 |
| clayg | well, this is case where we known we're taking a known working config file and making it blow up - i think it's worth the extra effort regardless of how bad validation is for other config options. | 18:01 |
| notmyname | zaitcev: themadcanudist: oh yeah. good call on remembering that change | 18:01 |
| clayg | notmyname: IMHO of course | 18:01 |
| notmyname | clayg: ya, I'm not disagreeing | 18:01 |
| zaitcev | themadcanudist: If you can reboot the whole cluster, then it it's a non-problem, but otherwise you need to follow that proceedure to upgrade live. | 18:01 |
| notmyname | zaitcev: themadcanudist: that was in 1.7.0! | 18:02 |
| notmyname | so he's already there | 18:02 |
| zaitcev | safe | 18:02 |
| mahatic | to edit the howto_intallmultinode doc, will i have to clone this https://github.com/openstack/openstack-manuals and edit? | 18:04 |
| mahatic | Or, i find the doc in here https://github.com/openstack/swift - or should i edit it here? | 18:05 |
| notmyname | mahatic: no. the doc you are referencing is in the swift source tree | 18:05 |
| notmyname | mahatic: yes, the second thing you said | 18:05 |
| mahatic | okay | 18:05 |
| mahatic | notmyname, any recommended tool should i be using? | 18:06 |
| notmyname | mahatic: https://github.com/openstack/swift/blob/master/doc/source/howto_installmultinode.rst | 18:06 |
| notmyname | mahatic: it's restructured text, so just a text editor | 18:06 |
| *** annegentle has quit IRC | 18:06 | |
| themadcanudist | zaitcev: Thanks! | 18:06 |
| mahatic | notmyname, okay. thank you. | 18:06 |
| notmyname | mahatic: you'll have to jump through the "getting started to contribute to openstack" steps | 18:06 |
| themadcanudist | so it doesn't affect me | 18:07 |
| themadcanudist | 1.7.5 | 18:07 |
| notmyname | mahatic: clone it, install git-review, set up git-review, then do the edits, then push to gerrit | 18:07 |
| *** wolsen has joined #openstack-swift | 18:07 | |
| notmyname | oh, sign the CLA. and make sure you have a launchpad account. *sigh* | 18:07 |
| notmyname | themadcanudist: correcty | 18:07 |
| notmyname | s/y// | 18:08 |
| mahatic | notmyname, heh, yup, have created the accounts, ssh etc. I had earlier tried to push a sample file. | 18:09 |
| mahatic | notmyname, When i was trying to learn git | 18:11 |
| *** annegentle has joined #openstack-swift | 18:18 | |
| *** mahatic has quit IRC | 18:19 | |
| *** mahatic has joined #openstack-swift | 18:24 | |
| *** mkollaro has quit IRC | 18:36 | |
| notmyname | did you see the email about a new openstack project? "Introducing Poppy - An Open API for CDN Provisioning" | 18:43 |
| torgomatic | who can keep track any more? | 18:43 |
| *** elambert has joined #openstack-swift | 18:44 | |
| zaitcev | At least Stacatto died. I think. | 18:45 |
| notmyname | what was that? | 18:45 |
| *** jdag___ is now known as jdaggett | 18:45 | |
| zaitcev | A service that copied images Glance-Glance. | 18:45 |
| zaitcev | The guy who pushed it quit and went to work for Dell, and we did not hear about it since. | 18:46 |
| zaitcev | https://github.com/stackforge/staccato/commit/5876e8664d593ae31d8de59d255af15de13b7328 - last commit almost exacltly 1 year ago | 18:47 |
| openstackgerrit | A change was merged to openstack/swift: Refactor object PUT to make EC easier to add https://review.openstack.org/114084 | 18:47 |
| zaitcev | https://github.com/stackforge - wow, what a zoo | 18:48 |
| zaitcev | 11 pages | 18:49 |
| dfg | hurricanerix_: https://github.com/dpgoetz/tcod/blob/master/txtime | 18:49 |
| dfg | oops | 18:49 |
| *** mikehn_ is now known as mikehn | 18:57 | |
| zaitcev | Hi Pete, Are you familiar with OpenStack Swift but would like to learn more? Now's your chance! | 19:01 |
| zaitcev | Absolutely | 19:01 |
| wolsen | hey torgomatic, thanks for the review and feedback on https://review.openstack.org/#/c/116618/ - have a minute to talk about it? | 19:04 |
| *** physcx has joined #openstack-swift | 19:12 | |
| *** themadcanudist has left #openstack-swift | 19:18 | |
| *** jasondotstar is now known as jasondotstar|afk | 19:24 | |
| *** StevenK has quit IRC | 19:38 | |
| *** IRTermite has quit IRC | 19:40 | |
| *** StevenK has joined #openstack-swift | 19:43 | |
| *** IRTermite has joined #openstack-swift | 19:52 | |
| *** tdasilva has quit IRC | 20:02 | |
| *** astellwag has quit IRC | 20:08 | |
| mahatic | notmyname, i checked out the branch source, edited the howto_installmultinode.rst and did a commit | 20:15 |
| notmyname | great | 20:15 |
| notmyname | mahatic: it's in one commit right? not several | 20:15 |
| mahatic | notmyname, http://paste.openstack.org/show/106011/ | 20:17 |
| notmyname | mahatic: great. now you need to do `git review` to send it to gerrit :-) | 20:17 |
| openstackgerrit | Mahati proposed a change to openstack/swift: Added instructions for mounting with a label. https://review.openstack.org/119193 | 20:20 |
| mahatic | notmyname, done! | 20:20 |
| *** tsg has quit IRC | 20:22 | |
| openstackgerrit | A change was merged to openstack/swift: fix my name in AUTHORS https://review.openstack.org/117081 | 20:26 |
| *** tsg has joined #openstack-swift | 20:29 | |
| *** astellwag has joined #openstack-swift | 20:30 | |
| *** elambert has quit IRC | 20:31 | |
| *** pberis has joined #openstack-swift | 20:40 | |
| *** tgohad has joined #openstack-swift | 20:41 | |
| *** tsg has quit IRC | 20:43 | |
| openstackgerrit | A change was merged to openstack/swift: warn against sorting requirements https://review.openstack.org/118694 | 20:44 |
| *** mkollaro has joined #openstack-swift | 20:54 | |
| *** jasondotstar|afk is now known as jasondotstar | 20:56 | |
| torgomatic | wolsen: I'm here now if you still are | 21:00 |
| wolsen | torgomatic, I am :) | 21:00 |
| *** mkollaro has quit IRC | 21:00 | |
| tab__ | how does per-disk thread pool work? does that mean that with configuration of this parameter someone specify how manny I/O it allows to a disk? Does configuration of this parameter applays to all disk or it could be per-disk configuration? | 21:00 |
| wolsen | torgomatic, so again thanks for your review - and I think you make some valid points about the blocking nature of the call | 21:00 |
| torgomatic | tab__: it's pending-IO/disk/worker | 21:00 |
| torgomatic | tab__: and it applies to each disk; if you want it per-disk, have one obj server per disk with a different config | 21:01 |
| torgomatic | wolsen: I hope it's not too harsh criticism... the good news is swift has a threadpool in it already that you could grab | 21:02 |
| wolsen | torgomatic, one thing about this change is that its not limited to the dns resolution in the cname_lookup middleware... unfortunately, when dnspython is in the mix and eventlet does the monkey patching, it patches the socket.getaddrinfo for an ipv4 only implementation, which affects things such as socket connections | 21:03 |
| wolsen | torgomatic, oh no not at all - its why we have the reviews and discussions :) | 21:03 |
| torgomatic | wolsen: right, but if rings have IP addresses in them and not hostnames, then I can't think of much else besides cname_lookup that requires connections to things... | 21:04 |
| torgomatic | I mean there's memcache servers, but typically one uses IPs for those as well | 21:04 |
| torgomatic | do you have some more things that break? I'm sure I can't think of all of it | 21:04 |
| *** annegentle has quit IRC | 21:05 | |
| wolsen | hmm, I'm not fully aware of everything that would break - I'll need to do some more investigation on that - my first thought was going to the socket.create_connection | 21:05 |
| wolsen | the ideal solution would be to get eventlet to support ipv6 :) | 21:07 |
| torgomatic | wolsen: sure, anything that feeds in a hostname there will have troubles; typically, though, cluster operators deploy configs with IPs everywhere possible so that DNS failures don't impact the cluster | 21:07 |
| torgomatic | wolsen: well, yes... go bang on that pull request some more ;) | 21:07 |
| torgomatic | FWIW, Swift will soon(ish) be requiring the latest eventlet instead of 0.9.whatever because of some new features in 0.15.2 | 21:07 |
| torgomatic | well, one new feature | 21:07 |
| *** IRTermite has quit IRC | 21:08 | |
| wolsen | ah interesting, well that will be good to be closer - I think the eventlet is just as interested in getting ipv6 in, but they want to make sure its baked properly | 21:08 |
| *** IRTermite has joined #openstack-swift | 21:08 | |
| torgomatic | wolsen: right, but notably that makes the dependency greater than what ships with Trusty *or* RHEL 7, so there'll be less resistance to bumping it even further | 21:09 |
| torgomatic | Swift typically tries to stay with versions that distros ship for ease of deployment | 21:09 |
| torgomatic | (library versions) | 21:09 |
| wolsen | torgomatic, yeah that makes sense - easier for everyone :) | 21:10 |
| torgomatic | it's like static friction vs. dynamic friction... it takes a lot to break free of a distro version, but then it's easier to keep it rolling | 21:10 |
| wolsen | torgomatic, so assuming I find nothing else that breaks, a threadpool wrapper around the dns resolution in the cname middleware would be preferred | 21:11 |
| wolsen | over what is proposed at least | 21:11 |
| torgomatic | wolsen: well, in order of preference... (1) fix eventlet (2) workaround in cname_middleware (inf) config for blocking DNS | 21:12 |
| peluse | torgomatic, quick sanity check if you could. It is the case that if I have one HDD die and it had N paritions on it (a) replcaitors would work to push copies to handoffs and (b) there could potentially be as many unique handoff nodes as paritions. Is there anything incorrect about either of those statements? | 21:12 |
| torgomatic | wolsen: also some load testing with the blocking DNS would be required to make sure nothing else breaks... that can be done in parallel with the cname_lookup changes by removing cname_lookup from the test system's pipeline | 21:12 |
| peluse | s/It is/In | 21:13 |
| torgomatic | peluse: I'm not certain about (a), but (b) is correct | 21:13 |
| torgomatic | and that's not a veiled "I think you're wrong", but real uncertainty :) | 21:13 |
| peluse | torgomatic, OK, thanks. That's where I was least certain, I'll go dig into the code for (a). Thanks! | 21:13 |
| wolsen | torgomatic, yeah that's my understanding of the preferred ordering as well | 21:13 |
| wolsen | torgomatic, thanks for the chat and review - I'll investigate this a bit more and come back as needed | 21:15 |
| torgomatic | wolsen: hope it helped | 21:15 |
| wolsen | torgomatic, it did until I have more questions :) | 21:16 |
| torgomatic | wolsen: I'm typically here. :) | 21:16 |
| wolsen | torgomatic, awesome | 21:16 |
| torgomatic | also, I (like most Swift cores) have an IRC bouncer, so ask anytime and I'll respond when I get back to my desk | 21:17 |
| wolsen | ack | 21:17 |
| peluse | torgomatic, so FYI it looks like (a) is a correct statement. Assuming the remote diskfile does a mount_check (def yes) and os.lstat errors if the drive is totally dead, the the remote server REPLICATE handler will return diskFul and the local replicator will skip it and move to the next in the node list so will indeed replicate to a handoff | 21:30 |
| torgomatic | peluse: good to know, thanks | 21:30 |
| peluse | torgomatic, relevant for the new section I'm putting in the reconstructor section in the EC spec based on your review of the reconstructor framework.... thx | 21:31 |
| *** mahatic has quit IRC | 21:34 | |
| wolsen | torgomatic, fyi it appears that you cannot bind to an ipv6 address with eventlet in the mix without the bypassing the greendns implementation in evently | 21:37 |
| wolsen | s/evently/eventlet/ | 21:37 |
| torgomatic | wolsen: hm... well, that's unfortunate | 21:37 |
| wolsen | torgomatic, indeed | 21:37 |
| wolsen | torgomatic, so it seems that another work around would need to be provided or wait for upstream | 21:38 |
| torgomatic | wolsen: does the proposed patch fix that, or is it another issue? | 21:38 |
| *** annegentle has joined #openstack-swift | 21:38 | |
| wolsen | torgomatic: the proposed patch does fix it - but at the cost that you mentioned - what the proposed patch does is cause the socket.getaddrinfo and such to not be patched and use the default (blocking) impl | 21:39 |
| *** annegentle has quit IRC | 21:40 | |
| wolsen | torgomatic, I'm getting a sense that this is quite a difficult problem beyond setting the env variable since it seems other openstack projects have gone the env variable route for resolving the ipv6 issue | 21:40 |
| torgomatic | wolsen: yeah, it gets tricky since Swift typically services lots more requests (concurrency and total) than other OpenStack projects do | 21:41 |
| torgomatic | you don't call the Nova API once the VM is up, you know? | 21:42 |
| wolsen | torgomatic, oh I absolutely acknowledge that | 21:42 |
| wolsen | right | 21:42 |
| wolsen | yeah swift has a much different use case | 21:42 |
| wolsen | with >>> volume of calls | 21:42 |
| torgomatic | wolsen: if tests with that patch and without cname_lookup don't show any other blocking-DNS slowdowns, then taking that patch (at least until eventlet gets sorted out) seems (to me) like a reasonable thing to do | 21:43 |
| torgomatic | well, plus the cname_lookup changes, but whatever | 21:43 |
| wolsen | torgomatic, yeah and I think its probably best to config option it as well | 21:44 |
| torgomatic | wolsen: yes | 21:44 |
| wolsen | I suppose the question would be is it only proxy server that needs it | 21:45 |
| openstackgerrit | paul luse proposed a change to openstack/swift-specs: Updates to EC Design Spec https://review.openstack.org/116486 | 21:45 |
| torgomatic | wolsen: the unpatched DNS, or the threadpool stuff? | 21:46 |
| wolsen | torgomatic, was thinking the unpatched dns... threadpool around cname middleware would be proxy server I believe... thinking the unpatched dns goes for all | 21:47 |
| torgomatic | wolsen: yep... putting the patch in utils like that should get everyone, though | 21:47 |
| wolsen | torgomatic, yeah but not sure where/which config file to pull it from :) | 21:48 |
| wolsen | as it gets patched in prior to utils intentionally | 21:48 |
| wolsen | anyways, thanks for your help torgomatic - really appreciate it | 21:51 |
| *** cutforth has quit IRC | 21:52 | |
| peluse | torgomatic, awesome replies to my questions on your GET splice patch - no followups if you can believe it :) | 21:55 |
| torgomatic | peluse: :) | 21:55 |
| peluse | torgomatic, re-running probtests now on the latest - my SAIO VM was a little off so I'm on a new one now, almost done w/no failures so far | 21:56 |
| *** m|k|e has quit IRC | 21:58 | |
| mattoliverau | Morning all | 21:58 |
| *** tgohad has quit IRC | 21:58 | |
| *** tongli has quit IRC | 21:59 | |
| peluse | howdy | 21:59 |
| mattoliverau | peluse: I have a heat template that builds an saio in the rackspace cloud if that helps. | 21:59 |
| peluse | mattoliverau, that's kinda cool but I like old school ground and pound or I forget how all this shit is stitched together :) | 22:00 |
| mattoliverau | Lol, fair enough. :) | 22:01 |
| torgomatic | The head bone's connected to the neck bone | 22:02 |
| torgomatic | The neck bone's connected to the shoulder bone | 22:02 |
| torgomatic | The shoulder bone's connected to... mysql? Who thought this was a good idea? | 22:02 |
| peluse | heh, sometimes I'm not sure what's connected to my neck bone - not always my brain though! | 22:03 |
| peluse | mattoliverau, so we were chatting about LCA a long time ago - my talk was accepted however I'm burning my travel budget for Q4 and Q1 on Paris so I can't go :( | 22:04 |
| peluse | mattoliverau, we're asking them if they'd be cool with notmyname covering it though which would still be way cool (but I really wanted to check out NZ) | 22:05 |
| mattoliverau | Damn! That's sux, wouldn't been awesome to have you down this way :( | 22:05 |
| mattoliverau | Would've | 22:05 |
| mattoliverau | Damn auto correct | 22:06 |
| mattoliverau | peluse: but at least you get to come to Paris. I guess the downside is the summit closest to LCA each year is fairly close to the not american ones. | 22:08 |
| mattoliverau | Were you speaking in the OpenStack mini conference or the main one? I work with both chairs of the paper committee and one of the organisors of the OpenStack miniconf. So can push anything you need. | 22:10 |
| *** erlon has quit IRC | 22:10 | |
| *** tsg has joined #openstack-swift | 22:12 | |
| *** occupant has quit IRC | 22:15 | |
| *** foexle_ has quit IRC | 22:26 | |
| openstackgerrit | paul luse proposed a change to openstack/swift-specs: Updates to EC Design Spec https://review.openstack.org/116486 | 22:27 |
| peluse | ^ slightly updated picture in the concurrency section for the reconstructor | 22:28 |
| peluse | mattoliverau, was that last question for me? | 22:29 |
| *** stonemessenger has joined #openstack-swift | 22:38 | |
| mattoliverau | Yup, sorry | 22:39 |
| *** stonemessenger has quit IRC | 22:40 | |
| *** tsg has quit IRC | 22:42 | |
| tab__ | is there any swift middleware that would move older data to slower storage and save all new data or hot data (that are read a lot) to faster storage, in means of cache-tiering? | 22:50 |
| *** marcusvrn has quit IRC | 22:51 | |
| notmyname | mattoliverau: peluse's talk was for the main conference. I don't think miniconfs have opened their CFP yet | 22:53 |
| notmyname | tab__: no. but.... | 22:53 |
| notmyname | tab__: so we have storage policies now. and you can have a storage policy for each tier of your storage. and I'd expect that people will write daemons that do move data to different tiers based on some condition | 22:54 |
| tab__ | notmyname: ok thx. So that is still work to do :) | 22:55 |
| *** antelopebyzantiu has joined #openstack-swift | 22:55 | |
| notmyname | tab__: yes. but it's not work that I expect to be in upstream swift | 22:55 |
| *** antelopebyzantiu has quit IRC | 22:56 | |
| *** dmsimard is now known as dmsimard_away | 22:56 | |
| *** geaaru has joined #openstack-swift | 22:57 | |
| tab__ | notmyname: yes, understood. | 22:57 |
| MooingLemur | storage policies are only granular to the container level though, right? | 22:58 |
| MooingLemur | so it's not as if you can transparently shuffle old objects to slower disks | 23:00 |
| notmyname | MooingLemur: correct. they are set at the container level | 23:00 |
| notmyname | MooingLemur: yes, but you could create a static manifest in the original tier to point to the slower one. the manifest is small, and the url wouldn't change that way | 23:01 |
| MooingLemur | aha, right | 23:01 |
| torgomatic | also note that middleware is the worst place to shuffle data between tiers; if you do it in a daemon, then clients don't have to wait for it | 23:01 |
| notmyname | torgomatic: peluse: are you ok with landing the living EC spec changes? | 23:27 |
| notmyname | torgomatic: peluse: https://review.openstack.org/#/c/116486/ | 23:27 |
| torgomatic | notmyname: sure | 23:27 |
| notmyname | torgomatic: ok. I'll do it | 23:27 |
| openstackgerrit | A change was merged to openstack/swift-specs: Updates to EC Design Spec https://review.openstack.org/116486 | 23:29 |
| notmyname | oh that was fast | 23:32 |
| torgomatic | remember when code review was slow because of the humans? | 23:35 |
| openstackgerrit | John Dickinson proposed a change to openstack/swift: make the bind_port config setting required https://review.openstack.org/118200 | 23:49 |
| notmyname | clayg: I think that's what you were looking for? ^^ | 23:49 |
| clayg | notmyname: maybe, i'm pretty hard to make happy | 23:49 |
| notmyname | lol | 23:50 |
| clayg | notmyname: heh, is there supposed to be a sys.exit in there after that logger.error - or does it like fall back to the old defaults? | 23:52 |
| notmyname | doh! | 23:53 |
| notmyname | see I knew there was something to be unhappy with :-) | 23:53 |
| notmyname | clayg: actually, looks like it should be "return 1" | 23:54 |
| openstackgerrit | John Dickinson proposed a change to openstack/swift: make the bind_port config setting required https://review.openstack.org/118200 | 23:55 |
| notmyname | there | 23:55 |
| clayg | HEH | 23:55 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!