| *** haomaiwang has joined #openstack-swift | 00:02 | |
| *** Trixboxer has quit IRC | 00:07 | |
| *** haomaiwang has quit IRC | 00:16 | |
| *** haomaiwang has joined #openstack-swift | 00:23 | |
| *** dmorita has joined #openstack-swift | 00:26 | |
| *** tsg has joined #openstack-swift | 00:26 | |
| *** haomaiwang has quit IRC | 00:30 | |
| *** haomaiwa_ has joined #openstack-swift | 00:30 | |
| *** tsg has quit IRC | 00:33 | |
| *** swat30 has quit IRC | 00:36 | |
| *** thurloat has quit IRC | 00:36 | |
| *** thurloat has joined #openstack-swift | 00:39 | |
| *** swat30 has joined #openstack-swift | 00:39 | |
| *** tsg has joined #openstack-swift | 00:40 | |
| *** haomaiwa_ has quit IRC | 00:41 | |
| openstackgerrit | Yuan Zhou proposed a change to openstack/swift: Use mkstemp() instead of deprecated mktemp() https://review.openstack.org/118084 | 00:42 |
|---|---|---|
| *** haomaiwa_ has joined #openstack-swift | 00:43 | |
| *** swills has left #openstack-swift | 01:05 | |
| *** addnull has joined #openstack-swift | 01:31 | |
| *** haomaiw__ has joined #openstack-swift | 01:39 | |
| *** haomaiwa_ has quit IRC | 01:40 | |
| *** nosnos has joined #openstack-swift | 01:48 | |
| *** developer has joined #openstack-swift | 01:49 | |
| *** haomaiw__ has quit IRC | 01:50 | |
| developer | Just testing connection. | 01:50 |
| *** haomaiwang has joined #openstack-swift | 01:56 | |
| *** miqui has quit IRC | 01:57 | |
| openstackgerrit | Yuan Zhou proposed a change to openstack/swift: Use mkstemp() instead of deprecated mktemp() https://review.openstack.org/118084 | 01:58 |
| *** haomaiwang has quit IRC | 02:01 | |
| *** haomaiwang has joined #openstack-swift | 02:02 | |
| *** zul has quit IRC | 02:08 | |
| *** miqui has joined #openstack-swift | 02:19 | |
| *** addnull has quit IRC | 02:31 | |
| *** addnull has joined #openstack-swift | 02:37 | |
| *** haomaiwang has quit IRC | 02:42 | |
| *** addnull has quit IRC | 02:47 | |
| *** haomaiwang has joined #openstack-swift | 02:48 | |
| *** addnull has joined #openstack-swift | 03:03 | |
| *** shakamunyi has joined #openstack-swift | 03:15 | |
| *** haomaiw__ has joined #openstack-swift | 03:15 | |
| *** haomaiwang has quit IRC | 03:19 | |
| *** shakamunyi has quit IRC | 03:20 | |
| *** haomaiw__ has quit IRC | 03:27 | |
| *** haomaiwang has joined #openstack-swift | 03:28 | |
| *** addnull has quit IRC | 03:38 | |
| *** xianghuihui has joined #openstack-swift | 03:39 | |
| *** xianghui has quit IRC | 03:42 | |
| *** nosnos has quit IRC | 03:44 | |
| *** nosnos has joined #openstack-swift | 03:45 | |
| *** nosnos has quit IRC | 03:49 | |
| *** haomaiwang has quit IRC | 03:52 | |
| *** haomai___ has joined #openstack-swift | 03:52 | |
| *** tsg has quit IRC | 03:53 | |
| *** addnull has joined #openstack-swift | 03:56 | |
| *** tsg has joined #openstack-swift | 04:04 | |
| openstackgerrit | Tushar Gohad proposed a change to openstack/swift: EC: Make quorum_size() specific to storage policy https://review.openstack.org/111067 | 04:05 |
| *** theanalyst has joined #openstack-swift | 04:05 | |
| *** developer has quit IRC | 04:07 | |
| openstackgerrit | Tushar Gohad proposed a change to openstack/swift: EC: Make quorum_size() specific to storage policy https://review.openstack.org/111067 | 04:09 |
| *** chandankumar has joined #openstack-swift | 04:11 | |
| *** addnull has quit IRC | 04:15 | |
| *** addnull has joined #openstack-swift | 04:15 | |
| *** shakamunyi has joined #openstack-swift | 04:16 | |
| *** nosnos has joined #openstack-swift | 04:17 | |
| *** xianghuihui has quit IRC | 04:19 | |
| *** addnull has quit IRC | 04:20 | |
| *** shakamunyi has quit IRC | 04:21 | |
| *** zaitcev has quit IRC | 04:21 | |
| *** chandankumar has quit IRC | 04:38 | |
| *** chandankumar has joined #openstack-swift | 04:51 | |
| *** kota_ has joined #openstack-swift | 04:54 | |
| *** addnull has joined #openstack-swift | 05:08 | |
| *** kopparam has joined #openstack-swift | 05:13 | |
| *** shakamunyi has joined #openstack-swift | 05:16 | |
| *** haomai___ has quit IRC | 05:17 | |
| *** shakamunyi has quit IRC | 05:21 | |
| *** haomaiwa_ has joined #openstack-swift | 05:24 | |
| *** nshaikh has joined #openstack-swift | 05:27 | |
| *** haomaiwa_ has quit IRC | 05:28 | |
| *** haomaiwang has joined #openstack-swift | 05:28 | |
| *** haomaiwang has quit IRC | 05:37 | |
| *** haomaiwa_ has joined #openstack-swift | 05:43 | |
| *** haomaiwa_ has quit IRC | 05:51 | |
| *** k4n0 has joined #openstack-swift | 06:06 | |
| *** shakamunyi has joined #openstack-swift | 06:17 | |
| *** shakamunyi has quit IRC | 06:22 | |
| openstackgerrit | Kota Tsuyuzaki proposed a change to openstack/swift: Bump pyeclib version in requirements https://review.openstack.org/118264 | 06:25 |
| *** bvandenh has joined #openstack-swift | 06:50 | |
| *** kopparam has quit IRC | 07:00 | |
| *** kopparam has joined #openstack-swift | 07:00 | |
| *** chandankumar has quit IRC | 07:01 | |
| *** openstackgerrit has quit IRC | 07:02 | |
| *** kopparam has quit IRC | 07:05 | |
| *** echevemaster has quit IRC | 07:16 | |
| *** addnull has quit IRC | 07:18 | |
| *** shakamunyi has joined #openstack-swift | 07:18 | |
| *** addnull has joined #openstack-swift | 07:20 | |
| *** shakamunyi has quit IRC | 07:22 | |
| *** joeljwright has joined #openstack-swift | 07:23 | |
| *** tsg has quit IRC | 07:24 | |
| *** bkopilov has joined #openstack-swift | 07:44 | |
| *** haomaiwang has joined #openstack-swift | 07:45 | |
| *** bkopilov has quit IRC | 07:49 | |
| *** kopparam has joined #openstack-swift | 07:49 | |
| *** acoles_away is now known as acoles | 08:01 | |
| *** mkollaro has joined #openstack-swift | 08:11 | |
| *** morganfainberg is now known as morganfainberg_Z | 08:12 | |
| *** geaaru has joined #openstack-swift | 08:14 | |
| *** shakamunyi has joined #openstack-swift | 08:19 | |
| *** foexle has joined #openstack-swift | 08:20 | |
| *** bkopilov has joined #openstack-swift | 08:22 | |
| mattoliverau | Well I'm calling it a night, night all. | 08:22 |
| acoles | mattoliverau: good morning ;) | 08:23 |
| *** shakamunyi has quit IRC | 08:23 | |
| mattoliverau | acoles: lol, have a great day :) | 08:23 |
| *** ppai has joined #openstack-swift | 08:28 | |
| *** infotection has quit IRC | 08:41 | |
| *** infotection has joined #openstack-swift | 08:46 | |
| *** zul has joined #openstack-swift | 09:02 | |
| *** shakamunyi has joined #openstack-swift | 09:20 | |
| *** shakamunyi has quit IRC | 09:24 | |
| *** addnull has quit IRC | 09:27 | |
| *** addnull has joined #openstack-swift | 09:27 | |
| *** aix has joined #openstack-swift | 09:29 | |
| *** addnull has quit IRC | 09:32 | |
| *** addnull has joined #openstack-swift | 09:38 | |
| *** kota_ has quit IRC | 09:42 | |
| *** shakamunyi has joined #openstack-swift | 10:20 | |
| *** shakamunyi has quit IRC | 10:25 | |
| *** IAm_thor has joined #openstack-swift | 10:27 | |
| *** IAm_thor has left #openstack-swift | 10:27 | |
| *** addnull has quit IRC | 10:47 | |
| *** Anju has joined #openstack-swift | 10:51 | |
| *** shakamunyi has joined #openstack-swift | 11:21 | |
| *** shakamunyi has quit IRC | 11:25 | |
| *** addnull has joined #openstack-swift | 11:31 | |
| *** tab___ has quit IRC | 11:39 | |
| *** k4n0 has quit IRC | 11:46 | |
| *** dmorita has quit IRC | 12:12 | |
| *** jasondotstar has quit IRC | 12:19 | |
| *** shakamunyi has joined #openstack-swift | 12:22 | |
| *** shakamunyi has quit IRC | 12:27 | |
| *** kopparam has quit IRC | 12:34 | |
| *** kopparam has joined #openstack-swift | 12:35 | |
| *** kopparam has quit IRC | 12:40 | |
| *** ppai has quit IRC | 12:50 | |
| *** Trixboxer has joined #openstack-swift | 12:51 | |
| *** tongli has joined #openstack-swift | 13:00 | |
| *** kenhui has joined #openstack-swift | 13:06 | |
| *** bsdkurt has joined #openstack-swift | 13:06 | |
| *** tdasilva has joined #openstack-swift | 13:17 | |
| *** shakamunyi has joined #openstack-swift | 13:23 | |
| *** annegent_ has joined #openstack-swift | 13:26 | |
| *** shakamunyi has quit IRC | 13:27 | |
| *** Anju has quit IRC | 13:42 | |
| *** lnxnut has joined #openstack-swift | 13:51 | |
| *** tdasilva has quit IRC | 13:58 | |
| *** sandywalsh has joined #openstack-swift | 13:59 | |
| *** bsdkurt has quit IRC | 14:02 | |
| *** openstackgerrit has joined #openstack-swift | 14:07 | |
| *** tsg has joined #openstack-swift | 14:08 | |
| *** shakamunyi has joined #openstack-swift | 14:10 | |
| *** shakamunyi has quit IRC | 14:11 | |
| *** judd7 has joined #openstack-swift | 14:13 | |
| *** tdasilva has joined #openstack-swift | 14:13 | |
| *** bkopilov has quit IRC | 14:16 | |
| *** mahatic has joined #openstack-swift | 14:17 | |
| *** bvandenh has quit IRC | 14:25 | |
| *** tsg has quit IRC | 14:28 | |
| *** bkopilov has joined #openstack-swift | 14:29 | |
| *** sandywalsh has quit IRC | 14:42 | |
| *** addnull has quit IRC | 14:47 | |
| *** shakamunyi has joined #openstack-swift | 14:47 | |
| *** nshaikh has quit IRC | 14:52 | |
| *** tsg has joined #openstack-swift | 14:55 | |
| *** bkopilov has quit IRC | 14:59 | |
| *** judd7 has quit IRC | 15:03 | |
| *** nosnos has quit IRC | 15:06 | |
| *** nosnos has joined #openstack-swift | 15:07 | |
| *** shakamunyi has quit IRC | 15:07 | |
| *** foexle has quit IRC | 15:10 | |
| *** shakamunyi has joined #openstack-swift | 15:11 | |
| *** nosnos has quit IRC | 15:11 | |
| *** pberis has quit IRC | 15:13 | |
| *** aerwin has joined #openstack-swift | 15:22 | |
| *** annegentle has quit IRC | 15:32 | |
| *** otoolee has quit IRC | 15:39 | |
| *** mahatic has quit IRC | 15:40 | |
| *** mwstorer has joined #openstack-swift | 15:44 | |
| *** gyee has joined #openstack-swift | 15:45 | |
| *** shakamunyi has quit IRC | 15:46 | |
| *** acoles has quit IRC | 15:51 | |
| *** acoles has joined #openstack-swift | 16:01 | |
| *** ChanServ sets mode: +v acoles | 16:01 | |
| *** shakamunyi has joined #openstack-swift | 16:04 | |
| *** tdasilva has quit IRC | 16:04 | |
| *** aerwin has quit IRC | 16:07 | |
| *** otoolee- has joined #openstack-swift | 16:10 | |
| *** morganfainberg_Z is now known as morganfainberg | 16:11 | |
| *** annegent_ has quit IRC | 16:13 | |
| *** annegent_ has joined #openstack-swift | 16:14 | |
| *** Alex_Gaynor has quit IRC | 16:17 | |
| *** Alex_Gaynor has joined #openstack-swift | 16:18 | |
| *** shakamunyi has quit IRC | 16:20 | |
| *** mahatic has joined #openstack-swift | 16:30 | |
| *** joeljwright has quit IRC | 16:31 | |
| samwebster | Hi everyone. I had a question about ring rebalancing. When I change the devices in the ring around, which of the swift processes is responsible for migrating the existing data to the new location? | 16:39 |
| openstackgerrit | Alistair Coles proposed a change to openstack/swift: Check for change before container replicator updates db https://review.openstack.org/118401 | 16:40 |
| *** tdasilva has joined #openstack-swift | 16:40 | |
| notmyname | good morning | 16:41 |
| *** bill_az_ has joined #openstack-swift | 16:41 | |
| notmyname | acoles: you're back! | 16:42 |
| peluse | good morning and, yeah, welcome back acoles | 16:42 |
| notmyname | samwebster: replication | 16:42 |
| notmyname | samwebster: {account,container,object}-replicator | 16:42 |
| acoles | notmyname: indeed, although in spirit i am still sailing on lake garda ;) | 16:42 |
| notmyname | acoles: :-) | 16:42 |
| * tdasilva google images lake garda | 16:43 | |
| tdasilva | wow! | 16:43 |
| samwebster | notmyname: thanks! How does the replicator know what needs to be moved? Does it compare the ring in memory to the ring on disk? If the replicator was down when I changed the ring file, would it still be able to move the old data to the new location? | 16:44 |
| *** mahatic has quit IRC | 16:46 | |
| acoles | tdasilva: yeah, its pretty spectacular | 16:47 |
| notmyname | samwebster: well, that's not a one-line answer :-) | 16:47 |
| tdasilva | acoles: nice, sounds like you had a great time! | 16:47 |
| acoles | tdasilva: btw hope to review patch soon, catching up... | 16:47 |
| notmyname | samwebster: but, yes it automatically reloads the ring files from disk. and replication can be stopped and restarted and still work | 16:48 |
| tdasilva | acoles: the constraints middleware? | 16:48 |
| notmyname | acoles: I thought tdasilva was just making a big deal out of it. then I saw the pictures. wow | 16:48 |
| samwebster | notmyname: thanks for the help. | 16:49 |
| acoles | tdasilva: https://review.openstack.org/115995 , but middleware too i guess ;) | 16:49 |
| tdasilva | acoles: oh yes, it would be nice to get that landed | 16:49 |
| tdasilva | acoles: the middleware one I'm still working on it, I think it will change quite a bit. I had a nice talk with torgomatic last week and we kinda of see a new direction for it | 16:50 |
| acoles | tdasilva: ok, i'll wait for that then | 16:51 |
| notmyname | samwebster: please ask if you have more questions. there's quite a few things online that I can point you to if i know what the questions are | 16:51 |
| acoles | notmyname: google 'image ferrari museum' to complete the picture | 16:52 |
| peluse | double wow on the lake garda thing, man o man | 16:53 |
| samwebster | notmyname: My core question is that I'm wondering if all the swift services are down when I update the ring, how will the replicator know what to move? If it doesn't have the old ring file to compare to, how does it know what needs to move? | 16:54 |
| notmyname | samwebster: ah, ok. data placement is deterministic. so every object gets put into a "partition" of the ring keyspace, and the ring determines what partitions are on what drives. replication simply makes sure the partitions are in the right place based on the current version of the ring | 16:56 |
| notmyname | peluse: acoles: tdasilva: I put a new link in the channel topic yesterday. https://wiki.openstack.org/wiki/Swift/ideas Just a few simpler ideas that IMO would be nice to have in Swift. a "I'm looking for something to do list" | 16:58 |
| notmyname | if you've got other stuff to add, that would be great | 16:58 |
| notmyname | /meeting train starting | 16:59 |
| acoles | notmyname: ack | 17:00 |
| samwebster | notmyname: I'm guess I'm not clear on how it knows about the data in the old location. If I initially had my container ring with the device "z1doriginal" and then change the device name to "z1dnew", how does it know to move the existing container db files in the z1doriginal device to the z1dnew device? | 17:03 |
| *** judd7 has joined #openstack-swift | 17:04 | |
| *** dencaval has joined #openstack-swift | 17:04 | |
| tdasilva | notmyname: nice idea | 17:05 |
| ahale | samwebster: the replicator looks at what partitions it has, what the ring says it should have, and if theres data that shouldnt be there (new ring / handoff writes / whatever) then it will try to put it where it thinks it should be, based on the ring it has | 17:11 |
| samwebster | ahale: So it looks at the disk and finds the old database z1doriginal/..../1234.db. It realizes that doesn't fit into the ring, so it sees where it should go and moves it there. I think I see now. | 17:14 |
| ahale | yep pretty much - at a basic level | 17:15 |
| portante_ | hey folks, ric wheeler here at Red Hat suggested I take a look at this article at how to properly sync data to a local file system, thought I'd share it: http://lwn.net/Articles/457667/ | 17:18 |
| *** portante_ is now known as portante | 17:18 | |
| *** ChanServ sets mode: +v portante | 17:18 | |
| portante | it points out that after creating the temp file in the object server, we perform the rename, but I don't think we perform an fsync of the directory to make that rename persistent | 17:19 |
| portante | it has been a while since I looked at the code, so if I am wrong, then we're good | 17:19 |
| portante | otherwise, we might want to consider the impact/trade-off of fsync the directory itself after the rename | 17:19 |
| * portante goes back to his little hole in the ground ... | 17:20 | |
| *** tdasilva has quit IRC | 17:20 | |
| *** tkay has joined #openstack-swift | 17:21 | |
| openstackgerrit | Lorcan Browne proposed a change to openstack/swift: Add "--no-overlap" option to swift-dispersion populate https://review.openstack.org/118411 | 17:26 |
| *** tdasilva has joined #openstack-swift | 17:28 | |
| *** bnelson has joined #openstack-swift | 17:35 | |
| *** tsg has quit IRC | 17:35 | |
| *** aix has quit IRC | 17:35 | |
| *** geaaru has quit IRC | 17:44 | |
| *** bkopilov has joined #openstack-swift | 17:49 | |
| *** miqui has quit IRC | 18:00 | |
| *** occup4nt has joined #openstack-swift | 18:23 | |
| *** occup4nt is now known as occupant | 18:23 | |
| *** shri1 has joined #openstack-swift | 18:26 | |
| *** mkollaro has quit IRC | 18:32 | |
| *** charz has quit IRC | 18:38 | |
| *** charz has joined #openstack-swift | 18:39 | |
| openstackgerrit | David Goetz proposed a change to openstack/swift: Delete expired objects in slightly smarter way. https://review.openstack.org/118434 | 18:47 |
| openstackgerrit | David Goetz proposed a change to openstack/swift: Delete expired objects in slightly smarter way. https://review.openstack.org/118434 | 18:48 |
| *** bnelson has quit IRC | 18:49 | |
| *** wer has joined #openstack-swift | 18:58 | |
| *** annegent_ has quit IRC | 19:22 | |
| peluse | portante, hey welcome back man... | 19:26 |
| peluse | portante, if you have a dusty old 'add policy support to in process func tests' somewhere I'd be glad to pick it up. Would like to get it in there, almost forgot about it... | 19:27 |
| *** mahatic has joined #openstack-swift | 19:28 | |
| *** erlon has joined #openstack-swift | 19:29 | |
| *** tsg has joined #openstack-swift | 19:30 | |
| *** miqui has joined #openstack-swift | 19:40 | |
| *** samwebster has quit IRC | 19:44 | |
| *** bnelson has joined #openstack-swift | 19:45 | |
| peluse | tsg, you there? | 19:46 |
| *** mahatic has quit IRC | 19:46 | |
| tsg | peluse: yes | 19:46 |
| peluse | ok, cool. just checking my client | 19:46 |
| dfg | hey anybody have time to check out https://review.openstack.org/#/c/117842/ ? ( i know it hasn't been there very long and i'm horrible about reviews but.. anyway. i'd like to move that one along if possible. it plus redbo's already approved change should really help container perf and I want to try it out) | 19:48 |
| *** annegent_ has joined #openstack-swift | 19:52 | |
| *** morganfainberg is now known as needs | 19:57 | |
| *** needs is now known as needscoffee | 19:57 | |
| *** zacksh has quit IRC | 20:32 | |
| *** zacksh has joined #openstack-swift | 20:39 | |
| *** tdasilva has quit IRC | 20:45 | |
| *** needscoffee is now known as morganfainberg | 20:56 | |
| *** tab_ has joined #openstack-swift | 20:59 | |
| *** tsg has quit IRC | 21:00 | |
| *** tsg has joined #openstack-swift | 21:01 | |
| *** zaitcev has joined #openstack-swift | 21:07 | |
| *** ChanServ sets mode: +v zaitcev | 21:07 | |
| *** annegent_ has quit IRC | 21:14 | |
| *** tsg has quit IRC | 21:14 | |
| *** morganfainberg is now known as steve_notmorgan | 21:32 | |
| *** steve_notmorgan is now known as morganfainberg | 21:32 | |
| *** lnxnut has quit IRC | 21:46 | |
| dfg | looks like I found the best way to stop all communication in the openstack-swift channel. ask for a review :) | 21:50 |
| notmyname | heh | 21:50 |
| *** annegent_ has joined #openstack-swift | 21:53 | |
| *** taras_ has joined #openstack-swift | 21:57 | |
| taras_ | hi, i've been poking at swift implementation because I have prior experience with fixing sqlite footguns in mozilla. Was curious if swift trips over similar ones | 21:59 |
| taras_ | came up with some observations in http://piratepad.net/D5eb87Rgrp | 21:59 |
| *** annegent_ has quit IRC | 21:59 | |
| taras_ | is irc a good place to try to understand swift design? | 21:59 |
| taras_ | or should i do mailing list? | 22:00 |
| peluse | dfg, I'll set aside some time tomorrow or Thu at the latest if its still in need of a second... booked the rest of today though | 22:00 |
| taras_ | i'm curious why swift decided to use sqlite for keyvalue store? | 22:01 |
| taras_ | which properties of sqlite are appealing for swift? | 22:01 |
| *** annegent_ has joined #openstack-swift | 22:02 | |
| *** Trixboxer has quit IRC | 22:03 | |
| dfg | peluse: ok thx. no crazy rush or anytrhing | 22:03 |
| peluse | dfg, cool | 22:04 |
| peluse | taras_, looks like some good questions, hopefully someone with some history will jump in here... | 22:04 |
| peluse | taras_, if not then formulating your observations into more specific questions and using the mailing list might work better | 22:05 |
| taras_ | ok, happy to do that | 22:05 |
| peluse | taras_, wrt general swift design/arch hjave you reviewd the available dev and arch docs? | 22:06 |
| peluse | taras_, here: http://docs.openstack.org/developer/swift/overview_architecture.html | 22:06 |
| taras_ | yes | 22:06 |
| peluse | cool... if you have other questions not so sql-specific I might be able to help now... fire away. my sql knowledge is limited to the changes I've had to make for features I've worked on (not part of original design) | 22:07 |
| taras_ | yeah | 22:08 |
| taras_ | though it's an extension of the sqlite question | 22:08 |
| taras_ | do you know what the replication logic is | 22:08 |
| taras_ | as i understand things are kept in sqlite | 22:08 |
| taras_ | so one can copy that file over | 22:08 |
| peluse | reasonably well I suppse, what's the question? | 22:08 |
| taras_ | and keep it in object server | 22:08 |
| taras_ | but then how do you know which sqlite file to modify, when to fetch it from obj store and when to use local copy, etc | 22:09 |
| taras_ | eg keeping listing in sqlite, allows to reuse the object store replication | 22:09 |
| mattoliverau | Morning all | 22:09 |
| taras_ | but i'm not clear on what that actually looks like | 22:09 |
| peluse | so you're asking specifically about database replication as opposed to replciation of the objects themselves? | 22:10 |
| taras_ | yes | 22:10 |
| peluse | OK, I haven't spent much time in that area either but do know the obj replication pretty well, let me scan the code real quick and at least give you some pointers of what to go look at | 22:11 |
| * peluse looking at swift/container/replciator.py | 22:12 | |
| taras_ | btw, i love you guys for not overdoing it on indexes and not using an objrelational mapping | 22:12 |
| zaitcev | But I can already tell that terminology needs to be adjusted. The "object server" in Swift only refers to the service that stores objects. It contains no sqlite of any kind. | 22:12 |
| zaitcev | The sqlite underpins implementation of account and container servers. | 22:12 |
| peluse | ahh, there's someone that can help! | 22:13 |
| zaitcev | oops | 22:13 |
| peluse | hehe | 22:13 |
| taras_ | :) | 22:13 |
| *** annegent_ has quit IRC | 22:15 | |
| zaitcev | In a sense, however, a and c servers are similar to o server, only instead of object bodies, they store sqlite DBs. The replication mechanisms are identical. We just rsync the xxxx.db over as needed. Well, with some complications. | 22:15 |
| *** annegentle has joined #openstack-swift | 22:15 | |
| zaitcev | EXCEPT | 22:15 |
| zaitcev | peluse is working on an entirely new implementation for object server | 22:15 |
| *** dmsimard is now known as dmsimard_away | 22:16 | |
| taras_ | zaitcev: so will new object server use a new way to list entries? | 22:16 |
| taras_ | or will it only change details of sqlite/rsync replication | 22:17 |
| zaitcev | taras_: no. Object server does not list entries even now. Only container server can do that. | 22:17 |
| peluse | zaitcev, what exactly am I working on? | 22:17 |
| zaitcev | peluse: EC | 22:17 |
| peluse | ahh, OK :) | 22:17 |
| taras_ | zaitcev: so when you say "new object server", you assume "old container" server? | 22:17 |
| peluse | so to be clear thats not a totally new implementation for the object server, its a new stoage policy and will change some code paths in the proxy and obj servers both (conditionally) but not as much as you might think | 22:18 |
| zaitcev | taras_: well... as far as replication is concerned, there's no chance. It adds a new columns and uses a view to look like the old one. | 22:18 |
| zaitcev | s/chace/change/ | 22:18 |
| taras_ | that's maily to support erasure coding? | 22:18 |
| zaitcev | Yes | 22:18 |
| zaitcev | Well, policies. | 22:18 |
| taras_ | ok | 22:18 |
| taras_ | so my sqlite poking wont be time wasted | 22:19 |
| zaitcev | Policy index attaches to container, so... | 22:19 |
| zaitcev | You're not the first, I think almost every operator has tools to peek into .db. It's not officially encouraged, but people like it. | 22:19 |
| * taras_ is not an op | 22:20 | |
| taras_ | i just read some blog posts re slow PUTs | 22:20 |
| taras_ | so setup devstack to see how much blame can be placed on sqlite there | 22:20 |
| taras_ | given my prior sqlite footgun-fixing experience on mozilla | 22:20 |
| zaitcev | ooooh, nice | 22:20 |
| *** tsg has joined #openstack-swift | 22:21 | |
| taras_ | so my data is super-sketchy, cos devstack :) | 22:21 |
| *** kenhui has quit IRC | 22:21 | |
| zaitcev | I remember how Awesome Bar came out in something like FF 23 or so. Man, it was shit. | 22:21 |
| zaitcev | Like, really hanging everything | 22:21 |
| taras_ | ff4 yeah | 22:21 |
| zaitcev | And now I use it all the time | 22:21 |
| taras_ | things got less bad somewhere between 14 and 20 | 22:22 |
| zaitcev | That far back, huh. Time flies. | 22:22 |
| zaitcev | Anyway, good luck. | 22:22 |
| taras_ | pain lingers :) | 22:22 |
| zaitcev | So, do you have any URLs for those blog posts? | 22:23 |
| peluse | taras_, so if you want to look more at the implementation of the db replciator check out swift/common/db_replicator.py | 22:24 |
| *** echevemaster has joined #openstack-swift | 22:24 | |
| taras_ | peluse: thank you | 22:24 |
| zaitcev | oh, wait, I see links in piratepad | 22:24 |
| zaitcev | taras_: you need to engage torgomatic and clayg to answer some of the questions in the pad | 22:28 |
| taras_ | hope they notice on mailing list | 22:29 |
| peluse | they both watch ing | 22:30 |
| peluse | ing=it | 22:30 |
| torgomatic | I'm sure there's suboptimal sqlite usage going on, I'm just glad taras_ is able to identify some of it | 22:41 |
| taras_ | it's always a question of whether it actually matters | 22:43 |
| zaitcev | taras_: what was the subject | 22:43 |
| taras_ | didnt sent it yet | 22:43 |
| taras_ | figured i'd poke irc first | 22:43 |
| zaitcev | aww | 22:43 |
| zaitcev | okay. | 22:43 |
| torgomatic | taras_: a change just landed to do INSERT batching, I think | 22:44 |
| torgomatic | like in the last couple of days | 22:44 |
| taras_ | hmm didnt see it when is scanned code last week | 22:46 |
| zaitcev | go go gadget git pull | 22:46 |
| torgomatic | commit 1fcd6096 | 22:46 |
| taras_ | i don't see that in https://github.com/openstack/swift/commits/master | 22:48 |
| taras_ | is there a devbranch? | 22:49 |
| peluse | https://github.com/openstack/swift/commit/85362fdf4e7e70765ba08cee288437a763ea5475 | 22:50 |
| taras_ | ok | 22:50 |
| taras_ | thanks | 22:51 |
| taras_ | now i get it | 22:52 |
| taras_ | ok | 22:52 |
| taras_ | torgomatic: is not fsyncing the pending file an option? | 22:55 |
| taras_ | eg S3 is ok with losing some PUTs | 22:55 |
| taras_ | seems like if you have a powercut, it's reasonable to miss a few entries in a container | 22:55 |
| torgomatic | taras_: the problem is that, if all 3 container replicas hit by the same powercut at the same time, there's nothing that would eventually end up populating the listing | 22:56 |
| taras_ | err nm, you dont fsync on flush | 22:57 |
| taras_ | torgomatic: that's an orphan in s3 lingo | 22:57 |
| torgomatic | taras_: well, TIL | 22:57 |
| taras_ | which is ok | 22:57 |
| taras_ | anyway, thanks for pointing this out | 22:57 |
| taras_ | i missed this | 22:58 |
| *** judd7 has quit IRC | 23:04 | |
| *** tgohad has joined #openstack-swift | 23:08 | |
| *** tsg has quit IRC | 23:09 | |
| *** tongli has quit IRC | 23:16 | |
| *** tongli has joined #openstack-swift | 23:16 | |
| taras_ | torgomatic: ha | 23:53 |
| taras_ | thanks | 23:53 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!