| *** gyee has quit IRC | 00:04 | |
| *** aselius has joined #openstack-swift | 00:08 | |
| *** mingyu has quit IRC | 00:11 | |
| *** mingyu has joined #openstack-swift | 00:17 | |
| notmyname | acoles: I didn't get to the sharding docs today. :-( | 01:12 |
|---|---|---|
| notmyname | but I've blocked off time on my calendar to do it tomorrow | 01:12 |
| *** ianychoi_ has quit IRC | 01:14 | |
| openstackgerrit | Vu Cong Tuan proposed openstack/slogging master: Update "auth_url" and "auth_uri" https://review.openstack.org/567708 | 01:42 |
| *** rcernin has joined #openstack-swift | 02:14 | |
| *** psachin` has joined #openstack-swift | 02:24 | |
| *** ianychoi has joined #openstack-swift | 02:29 | |
| *** gkadam has joined #openstack-swift | 02:47 | |
| *** klamath has quit IRC | 03:01 | |
| *** musalbas356 has joined #openstack-swift | 03:33 | |
| *** musalbas356 has quit IRC | 03:36 | |
| *** QN0BE0Skaag has joined #openstack-swift | 03:39 | |
| *** mingyu has quit IRC | 03:40 | |
| *** QN0BE0Skaag has quit IRC | 03:40 | |
| *** mingyu has joined #openstack-swift | 03:40 | |
| *** supamatt has quit IRC | 03:42 | |
| *** supamatt has joined #openstack-swift | 03:44 | |
| *** mingyu has quit IRC | 03:45 | |
| *** cshastri has joined #openstack-swift | 03:49 | |
| *** kei_yama has quit IRC | 03:52 | |
| *** kei_yama has joined #openstack-swift | 04:02 | |
| *** germs has quit IRC | 04:04 | |
| *** gkadam has quit IRC | 04:20 | |
| *** gkadam has joined #openstack-swift | 04:20 | |
| *** gyan_ has joined #openstack-swift | 04:25 | |
| *** gyankum has joined #openstack-swift | 04:26 | |
| *** aselius has quit IRC | 04:27 | |
| *** klamath has joined #openstack-swift | 04:34 | |
| *** klamath has quit IRC | 04:44 | |
| *** mingyu has joined #openstack-swift | 05:52 | |
| *** torgomatic has quit IRC | 06:08 | |
| *** MrDetoniawm has joined #openstack-swift | 06:10 | |
| *** MrDetoniawm has quit IRC | 06:12 | |
| *** hseipp has joined #openstack-swift | 06:17 | |
| *** karel_440 has joined #openstack-swift | 06:19 | |
| *** karel_440 has quit IRC | 06:22 | |
| *** links has joined #openstack-swift | 06:31 | |
| *** pcaruana has joined #openstack-swift | 06:32 | |
| *** gkadam has quit IRC | 06:39 | |
| *** klamath has joined #openstack-swift | 06:41 | |
| *** cshastri has quit IRC | 06:43 | |
| *** klamath has quit IRC | 06:46 | |
| *** armaan has joined #openstack-swift | 06:47 | |
| *** cshastri has joined #openstack-swift | 06:51 | |
| *** joeljwright has joined #openstack-swift | 06:57 | |
| *** ChanServ sets mode: +v joeljwright | 06:57 | |
| *** ccamacho has joined #openstack-swift | 07:00 | |
| *** geaaru has joined #openstack-swift | 07:02 | |
| *** klamath has joined #openstack-swift | 07:07 | |
| *** klamath has quit IRC | 07:12 | |
| *** tesseract has joined #openstack-swift | 07:20 | |
| acoles | good morning | 07:46 |
| *** mikecmpbll has joined #openstack-swift | 07:52 | |
| *** mingyu has quit IRC | 07:56 | |
| *** mingyu has joined #openstack-swift | 07:56 | |
| *** yuxin has joined #openstack-swift | 07:56 | |
| *** wangyuxin has joined #openstack-swift | 07:57 | |
| *** rcernin has quit IRC | 07:58 | |
| *** mingyu has quit IRC | 07:59 | |
| *** wangyuxin is now known as yuxin_ | 08:01 | |
| *** yuxin_ has quit IRC | 08:23 | |
| *** gkadam has joined #openstack-swift | 08:30 | |
| *** yuxin_ has joined #openstack-swift | 08:33 | |
| *** yuxin_ has quit IRC | 08:39 | |
| *** yuxin_ has joined #openstack-swift | 08:39 | |
| *** yuxin_ has quit IRC | 08:47 | |
| *** yuxin has quit IRC | 08:47 | |
| *** wangyuxin has joined #openstack-swift | 08:56 | |
| *** klamath has joined #openstack-swift | 09:08 | |
| *** wangyuxin has quit IRC | 09:08 | |
| *** openstack has joined #openstack-swift | 09:27 | |
| *** ChanServ sets mode: +o openstack | 09:27 | |
| *** yuxin has quit IRC | 09:28 | |
| *** yuxin_ has joined #openstack-swift | 09:29 | |
| *** yuxin_ has quit IRC | 09:36 | |
| *** yuxin_ has joined #openstack-swift | 09:42 | |
| *** armaan has quit IRC | 10:07 | |
| *** armaan has joined #openstack-swift | 10:08 | |
| *** armaan has quit IRC | 10:14 | |
| *** armaan has joined #openstack-swift | 10:14 | |
| *** cshastri_ has joined #openstack-swift | 10:32 | |
| *** cshastri has quit IRC | 10:35 | |
| *** klamath has joined #openstack-swift | 11:11 | |
| *** klamath has quit IRC | 11:16 | |
| *** gkadam_ has joined #openstack-swift | 12:03 | |
| *** gkadam has quit IRC | 12:04 | |
| *** armaan has quit IRC | 12:14 | |
| *** armaan has joined #openstack-swift | 12:15 | |
| *** armaan has quit IRC | 12:20 | |
| *** vinsh has quit IRC | 12:21 | |
| *** vinsh has joined #openstack-swift | 12:21 | |
| *** armaan has joined #openstack-swift | 12:27 | |
| *** cshastri_ has quit IRC | 12:39 | |
| *** germs has joined #openstack-swift | 12:50 | |
| *** germs has quit IRC | 12:50 | |
| *** germs has joined #openstack-swift | 12:50 | |
| *** vinsh has quit IRC | 12:54 | |
| *** vinsh has joined #openstack-swift | 12:54 | |
| *** klamath has joined #openstack-swift | 13:12 | |
| *** openstackstatus has joined #openstack-swift | 13:12 | |
| *** ChanServ sets mode: +v openstackstatus | 13:12 | |
| *** kei_yama has quit IRC | 13:15 | |
| -openstackstatus- NOTICE: Due to a Zuul outage, patches uploaded to Gerrit between 09:00UTC and 12:50UTC, were not properly added to Zuul. Please recheck any patches during this window and apologies for the inconvenience. | 13:16 | |
| *** klamath has quit IRC | 13:16 | |
| *** germs has quit IRC | 13:20 | |
| *** germs has joined #openstack-swift | 13:20 | |
| *** germs has quit IRC | 13:24 | |
| *** joeljwright has quit IRC | 13:26 | |
| *** armaan has quit IRC | 13:32 | |
| *** klamath has joined #openstack-swift | 13:32 | |
| *** armaan has joined #openstack-swift | 13:33 | |
| *** armaan has quit IRC | 13:37 | |
| *** gyan_ has quit IRC | 13:45 | |
| *** gyankum has quit IRC | 13:46 | |
| *** psachin` has quit IRC | 13:50 | |
| *** mvenesio has joined #openstack-swift | 13:52 | |
| *** armaan has joined #openstack-swift | 13:53 | |
| *** mvenesio has quit IRC | 13:57 | |
| *** germs has joined #openstack-swift | 13:59 | |
| *** germs has quit IRC | 13:59 | |
| *** germs has joined #openstack-swift | 13:59 | |
| *** armaan has quit IRC | 14:15 | |
| *** armaan has joined #openstack-swift | 14:16 | |
| *** germs has quit IRC | 14:18 | |
| *** armaan has quit IRC | 14:20 | |
| acoles | timburke: thanks for all your diffs, on gerrit they seem to get indented, any chance you could link to future ones in paste or gisthub? | 14:37 |
| acoles | kota_: FYI I am going to revert a change I made earlier this week based on your comments (see https://review.openstack.org/#/c/565742/4/swift/common/db.py@649 ) | 14:40 |
| patchbot | patch 565742 - swift (feature/deep-review) - Refactoring, test infrastructure changes and cleanup | 14:40 |
| acoles | zaitcev: timburke I'm going to accumulate unresolved review issues on trello so that they don't get lost when new patchsets are pushed to gerrit e.g. https://trello.com/c/VgEtBloj | 14:43 |
| acoles | timburke: clayg likewise https://trello.com/c/z2g5nBYD | 14:44 |
| *** germs has joined #openstack-swift | 14:47 | |
| *** germs has quit IRC | 14:47 | |
| *** germs has joined #openstack-swift | 14:47 | |
| *** germs has quit IRC | 14:53 | |
| clayg | 🤙 | 14:56 |
| *** pcaruana has quit IRC | 15:07 | |
| *** links has quit IRC | 15:24 | |
| *** armaan has joined #openstack-swift | 15:28 | |
| *** gyee has joined #openstack-swift | 15:41 | |
| notmyname | good morning | 15:47 |
| *** klamath has quit IRC | 15:49 | |
| *** klamath has joined #openstack-swift | 15:50 | |
| *** mikecmpbll has quit IRC | 16:05 | |
| *** germs has joined #openstack-swift | 16:16 | |
| *** germs has quit IRC | 16:16 | |
| *** germs has joined #openstack-swift | 16:16 | |
| *** germs has quit IRC | 16:16 | |
| *** germs has joined #openstack-swift | 16:17 | |
| *** geaaru has quit IRC | 16:36 | |
| timburke | acoles: yeah, i think because you need the leading space to ensure it's fixed-width. i'll do pastes in the future | 16:39 |
| acoles | timburke: thanks | 16:40 |
| *** armaan_ has joined #openstack-swift | 16:46 | |
| *** armaan has quit IRC | 16:47 | |
| *** armaan has joined #openstack-swift | 16:47 | |
| *** armaan_ has quit IRC | 16:48 | |
| *** tesseract has quit IRC | 16:59 | |
| *** openstackstatus has quit IRC | 17:00 | |
| *** openstack has joined #openstack-swift | 17:03 | |
| *** ChanServ sets mode: +o openstack | 17:03 | |
| *** openstackstatus has joined #openstack-swift | 17:03 | |
| *** ChanServ sets mode: +v openstackstatus | 17:03 | |
| *** armaan has quit IRC | 17:11 | |
| *** armaan has joined #openstack-swift | 17:12 | |
| *** armaan has quit IRC | 17:20 | |
| *** armaan has joined #openstack-swift | 17:20 | |
| *** armaan has quit IRC | 17:21 | |
| *** armaan has joined #openstack-swift | 17:21 | |
| *** hseipp has quit IRC | 17:23 | |
| *** germs has quit IRC | 17:32 | |
| *** armaan has quit IRC | 17:33 | |
| *** armaan_ has joined #openstack-swift | 17:33 | |
| *** armaan has joined #openstack-swift | 17:34 | |
| *** armaan_ has quit IRC | 17:34 | |
| *** germs has joined #openstack-swift | 17:41 | |
| *** germs has joined #openstack-swift | 17:41 | |
| *** germs has quit IRC | 17:43 | |
| *** germs has joined #openstack-swift | 17:43 | |
| *** germs has quit IRC | 17:43 | |
| *** germs has joined #openstack-swift | 17:43 | |
| acoles | notmyname: I'm about to push a new set of patches. in the docs, there's some todo's 'above the line' to move diagrams from 'below the line' - mattoliverau has done some work on the diags but I'm not sure which. Then there is clean up 'below the line' todo. | 18:05 |
| notmyname | ok | 18:05 |
| notmyname | I've got my afteroon blocked off for docs review :-) | 18:06 |
| acoles | notmyname: comments like 'I need more info on this topic' will be useful | 18:07 |
| acoles | maybe just embed those as TODO's ^^ | 18:08 |
| *** openstackgerrit has joined #openstack-swift | 18:10 | |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep-review: Refactoring, test infrastructure changes and cleanup https://review.openstack.org/565742 | 18:10 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep-review: Add ShardRange class https://review.openstack.org/565743 | 18:10 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep-review: Add support for sharding in ContainerBroker https://review.openstack.org/565744 | 18:10 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep-review: Add shard range support to container server https://review.openstack.org/565745 | 18:10 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep-review: Enable proxy to build listings from shards https://review.openstack.org/565746 | 18:10 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep-review: Redirect object updates to shard containers https://review.openstack.org/565747 | 18:10 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep-review: Add sharder daemon, manage_shard_ranges tool and probe tests https://review.openstack.org/565748 | 18:10 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep-review: Add container sharding documentation https://review.openstack.org/565749 | 18:10 |
| openstackgerrit | Alistair Coles proposed openstack/swift feature/deep-review: SQUASH: Use separate replication verb for sending shard ranges https://review.openstack.org/567920 | 18:10 |
| acoles | timburke: clayg been meaning to do this for a while, but think it's worth quick review before I squash it into the chain https://review.openstack.org/567920 | 18:11 |
| patchbot | patch 567920 - swift (feature/deep-review) - SQUASH: Use separate replication verb for sending ... | 18:11 |
| *** brimestone has joined #openstack-swift | 18:11 | |
| *** brimestone has left #openstack-swift | 18:12 | |
| *** armaan has quit IRC | 18:12 | |
| clayg | NEWS: we have sharded a 145M row container | 18:13 |
| *** armaan has joined #openstack-swift | 18:13 | |
| notmyname | woohoo! | 18:13 |
| *** nicodemus has joined #openstack-swift | 18:18 | |
| nicodemus | hello | 18:18 |
| nicodemus | I have a 4 node swift cluster that is nearly full. We're in the process of provisioning another node, but in the meantime we're seeing that PUTs to the cluster are quite slow | 18:20 |
| nicodemus | If an object should be written in a disk that is currently full, what happens to that object? Is it saved on two locations instead of three? Or is it written in a fourth disk until the third has available space? | 18:21 |
| notmyname | internally, the object server may return a 507 to the proxy server. the proxy server will then attempt to put it on a different server, and replication will keep trying to move it back to the first one | 18:22 |
| notmyname | without going into a lot of detail, the object will still be on three drives until the full drive has available space | 18:24 |
| nicodemus | notmyname: great, thanks! | 18:25 |
| notmyname | nicodemus: note that having full drives in a swift cluster is something you cannot ignore and you need to take care of that as soon as possible | 18:26 |
| notmyname | because of the way data placement works in swift, it's likely you don't have just one full drive. most drives will fill up at around the same time | 18:26 |
| nicodemus | One could assume that if half of the disks are full, a PUT could be noticeably slower until the proxy finds a server that does have available space | 18:27 |
| notmyname | and because even deletes do writes (they write down a marker saying that the object has been deleted -- this is important to handle hardware failures), a very very full drive can even prevent you from deleting data! | 18:27 |
| DHE | I thought that worst case scenario swift will settle for only 2 copies (ceil(50.0001% of 3))... | 18:27 |
| DHE | of course, at that point you're literally full.. | 18:27 |
| nicodemus | notmyname: of course, several disks are nearly full. However we do have burocratic issues that will only be solved when the service is compromised :( | 18:28 |
| notmyname | DHE: correct. that during the write (eg if the third fails during the write, two successful fsyncs are good enough). but replication will move it to a third location, even if that's a handoff | 18:28 |
| *** mvenesio has joined #openstack-swift | 18:28 | |
| notmyname | nicodemus: you should definitely consider it compromised now | 18:28 |
| DHE | so if the proxy server streams without buffering, it can't "gracefully" recover from an out-of-space issue and would have to give up one (or more!?) of its 3 write target attempts | 18:29 |
| notmyname | the thing about swift is that it's so good at working in degraded scenarios that when really bad stuff happens the user just keeps seeing it working. and working. and working. until is doesn't and *nothing* works. then you're in a world of hurt as an admin | 18:29 |
| notmyname | DHE: exactly right | 18:30 |
| nicodemus | thanks a lot for the info! And yes, I'm insisting on the urgency of the issue | 18:30 |
| DHE | so I'd expect random total upload failures of large-ish items at this point onwards | 18:30 |
| DHE | yay I'm learning :) | 18:30 |
| notmyname | nicodemus: imagine a not-really-full cluster that has 6 nodes. 1 fails, everything is ok. 2 fail, everything is still ok. 3 fail, and you're still good. 4 fail, and IT STILL IS WORKING! 5 fail, and suddenly nothing works at all | 18:31 |
| notmyname | nicodemus: the "urgent issue, fix is NOW" time was way before the 5th server died. :-) | 18:31 |
| nicodemus | We've been in the "urgent issue" state for a week sadly | 18:32 |
| nicodemus | notmyname: let me ask you another question. I've been looking for a way to migrate swift's data, and was wondering if the swift-s3-sync script is ready to be used | 18:36 |
| notmyname | nicodemus: you mean https://github.com/swiftstack/swift-s3-sync ? | 18:37 |
| nicodemus | correct | 18:37 |
| notmyname | yes, absolutely. it's being used in production currently with several very large clusters and many smaller ones | 18:38 |
| notmyname | what are migrating to/from? | 18:38 |
| nicodemus | from Swift to CEPH (RadosGW) | 18:39 |
| notmyname | :-( | 18:40 |
| mvenesio | notmyname: hahah | 18:40 |
| nicodemus | I believe there were a keystone issue regarding the swift-s3-sync | 18:41 |
| notmyname | nicodemus: I'd guess it would work. but that's certainly not a tested migration target | 18:41 |
| notmyname | nicodemus: yeah, there's a couple of pull requests for getting keystone support into that project. I expect it to be there soon | 18:42 |
| nicodemus | so, it wouldn't be recommended for a production environment (at least not without the proper testing) | 18:42 |
| *** armaan has quit IRC | 18:43 | |
| *** armaan has joined #openstack-swift | 18:43 | |
| *** ukaynar_ has joined #openstack-swift | 19:10 | |
| *** mvenesio has quit IRC | 19:18 | |
| *** mvenesio has joined #openstack-swift | 19:18 | |
| *** mvenesio has quit IRC | 19:23 | |
| *** gkadam_ has quit IRC | 19:28 | |
| *** nicodemus has quit IRC | 19:55 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: DNM: Use assertRaises more https://review.openstack.org/567933 | 20:00 |
| openstackgerrit | Clay Gerrard proposed openstack/swift feature/deep-review: SQUASH: use a tpool for merge_items https://review.openstack.org/567934 | 20:02 |
| openstackgerrit | Tim Burke proposed openstack/swift feature/deep-review: SQUASH: fix-up signal handling https://review.openstack.org/567938 | 20:19 |
| openstackgerrit | Tim Burke proposed openstack/swift feature/deep-review: SQUASH: Handle SIGINT more like we do SIGHUP and SIGTERM https://review.openstack.org/567939 | 20:19 |
| *** mikecmpbll has joined #openstack-swift | 20:20 | |
| *** armaan has quit IRC | 20:22 | |
| *** armaan has joined #openstack-swift | 20:22 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: DNM: Make the decision between primary/handoff sets more obvious https://review.openstack.org/567940 | 20:25 |
| *** torgomatic has joined #openstack-swift | 20:53 | |
| *** ChanServ sets mode: +v torgomatic | 20:53 | |
| *** armaan has quit IRC | 21:06 | |
| *** armaan has joined #openstack-swift | 21:06 | |
| *** geaaru has joined #openstack-swift | 21:09 | |
| *** mvenesio has joined #openstack-swift | 21:13 | |
| *** mvenesio has quit IRC | 21:15 | |
| *** ukaynar_ has quit IRC | 21:31 | |
| timburke | hmm.... http://logs.openstack.org/34/567934/1/check/swift-probetests-centos-7/c9325c1/ara-report/result/91bb39b1-9b9e-4933-b1a8-a7bf9979fd31/ | 21:35 |
| timburke | http://mirror.centos.org/$contentdir/7/virt/x86_64/kvm-common/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found | 21:36 |
| *** ukaynar_ has joined #openstack-swift | 21:36 | |
| *** mvenesio has joined #openstack-swift | 22:18 | |
| *** itlinux has joined #openstack-swift | 22:57 | |
| *** itlinux has quit IRC | 23:00 | |
| *** itlinux has joined #openstack-swift | 23:00 | |
| *** ukaynar_ has quit IRC | 23:05 | |
| *** mtreinish_ has joined #openstack-swift | 23:19 | |
| *** mtreinish has quit IRC | 23:19 | |
| *** mtreinish_ is now known as mtreinish | 23:20 | |
| openstackgerrit | Tim Burke proposed openstack/swift feature/deep-review: SQUASH: only use textual db_states https://review.openstack.org/567961 | 23:32 |
| openstackgerrit | John Dickinson proposed openstack/swift feature/deep-review: Add container sharding documentation https://review.openstack.org/565749 | 23:34 |
| notmyname | mattoliverau: acoles: docs updated. still a few TODOs. most of the stuff below the line was removed | 23:35 |
| *** itlinux has quit IRC | 23:37 | |
| *** itlinux has joined #openstack-swift | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!