| *** mrmoje has joined #openstack-swift | 00:00 | |
| *** haomaiwa_ has quit IRC | 00:01 | |
| *** mrmoje has quit IRC | 00:01 | |
| *** lcurtis has quit IRC | 00:01 | |
| *** haomaiwang has joined #openstack-swift | 00:01 | |
| *** baojg has joined #openstack-swift | 00:06 | |
| *** mrmoje has joined #openstack-swift | 00:06 | |
| *** baojg has quit IRC | 00:16 | |
| *** then3rd has joined #openstack-swift | 00:17 | |
| *** baojg has joined #openstack-swift | 00:18 | |
| *** mingdang1 has joined #openstack-swift | 00:18 | |
| then3rd | Dumb question.. I can't seem to remember or locate the swift api url that reports the version number. Anyone know what I'm thinking of? | 00:19 |
|---|---|---|
| *** PsionTheory has quit IRC | 00:19 | |
| timburke | then3rd: /info | 00:19 |
| *** nadeem has quit IRC | 00:19 | |
| then3rd | perfect, thank you. I was looking right at it. | 00:20 |
| *** mingdang1 has quit IRC | 00:25 | |
| *** mrmoje has quit IRC | 00:27 | |
| *** mrmoje has joined #openstack-swift | 00:29 | |
| *** bill_az has quit IRC | 00:31 | |
| *** asettle has joined #openstack-swift | 00:39 | |
| *** chlong has joined #openstack-swift | 00:40 | |
| *** mingdang1 has joined #openstack-swift | 00:43 | |
| *** takashi has joined #openstack-swift | 00:45 | |
| *** daemontool has joined #openstack-swift | 00:47 | |
| *** daemontool_ has quit IRC | 00:48 | |
| *** mingdang1 has quit IRC | 00:54 | |
| *** zhill has quit IRC | 00:57 | |
| *** haomaiwang has quit IRC | 01:01 | |
| *** jamielennox is now known as jamielennox|away | 01:01 | |
| *** haomaiwang has joined #openstack-swift | 01:01 | |
| rickyrem | I'm running the kilo release of Swift. I have 2 proxies, but only one of them is connecting to rabbitmq | 01:11 |
| rickyrem | Do any of your have any pointer on what the issue could be? | 01:12 |
| zaitcev | Swift connecting to Rabbit... what? | 01:12 |
| zaitcev | What middleware do we have that does it? Not recon, surely. | 01:12 |
| rickyrem | Oh, ceilometer | 01:13 |
| zaitcev | Oh, right. Thanks | 01:13 |
| zaitcev | Hmm... | 01:13 |
| rickyrem | I diff'ed the configs and they look the same | 01:13 |
| rickyrem | both servers had their services installed via rpm | 01:14 |
| zaitcev | Unfortunately, I'm fairly clueless about Ceilometer. I don't even know if it has any agent processes or if middleware talks to it directly, on the context of Swift proxy. | 01:15 |
| zaitcev | I suppose asking at #rdo may help, but I'm not sure | 01:16 |
| clayg | is the client facing traffic hitting both of them - what are you doing for load balancing | 01:16 |
| rickyrem | haproxy as the balancer | 01:17 |
| rickyrem | and yes, the logs show both servers are getting hit | 01:17 |
| rickyrem | zaitcev: ill keep in mind that I could go there | 01:18 |
| clayg | and how are you deciding that only one of them is connected to rabbit - some sort rabbitmqctl command to list connections? | 01:18 |
| rickyrem | proxy1 is the only one that shows connection when running `ss -plant | grep 5672`. So then I went into the rabbitmq admin panel and saw that proxy2 wasn't listed as a connection either. | 01:20 |
| *** Jeffrey4l has joined #openstack-swift | 01:22 | |
| *** asettle has quit IRC | 01:24 | |
| zaitcev | I'd look at firewalls first. What if firewall-cmd screwed up... | 01:25 |
| rickyrem | Firewalld isn't running and iptables has no rules. | 01:27 |
| rickyrem | Well i mean the only rules are accept everything. | 01:27 |
| *** garthb has quit IRC | 01:27 | |
| zaitcev | self._notifier = oslo_messaging.Notifier(oslo_messaging.get_transport(cfg.CONF, url=conf.get('url')), publisher_id='ceilometermiddleware', .....) | 01:33 |
| zaitcev | my, my | 01:34 |
| zaitcev | So it is fairly simple after all. All the connection-making stuff is buried inside Oslo. | 01:34 |
| rickyrem | ohhh, gotcha | 01:35 |
| rickyrem | Where is that line located? | 01:35 |
| zaitcev | https://github.com/openstack/ceilometermiddleware/blob/master/ceilometermiddleware/swift.py#L111 | 01:35 |
| rickyrem | Agh, I was looking for that in the Swift project. | 01:36 |
| rickyrem | This makes more sense now | 01:36 |
| zaitcev | I wanted to ask to make sure if the middleware is actually in the pipeline in proxy-server.conf, but you said you compared all configs | 01:36 |
| rickyrem | Correct | 01:36 |
| zaitcev | python-ceilometermiddleware.noarch is a separate RPM | 01:37 |
| zaitcev | at this point I'd look at proxy processes with lsof, to see if it even tries | 01:38 |
| zaitcev | IIRC AMQP runs over TCP for us | 01:38 |
| zaitcev | s/for us/in OpenStack/ | 01:38 |
| kota_ | back to online :) | 01:39 |
| *** Jeffrey4l has quit IRC | 01:39 | |
| rickyrem | I verified that I didn't miss that RPM, and the egg is there. But I would assume it would have complained about the missing egg. | 01:41 |
| rickyrem | I'll see what I can find with lsof now | 01:41 |
| zaitcev | By it you mean proxy server process, right? | 01:41 |
| rickyrem | Correct | 01:42 |
| zaitcev | Sadly /info does not show currently active pipeline | 01:45 |
| rickyrem | I'm not seeing anything in lsof to show that it is attempting to bind to a socket. | 01:50 |
| zaitcev | Oslo could try once at start, then sit on the error silently. | 01:50 |
| *** lyrrad has quit IRC | 01:51 | |
| *** klrmn has quit IRC | 01:51 | |
| rickyrem | That's what i'm afraid of | 01:52 |
| *** mgarza_ has joined #openstack-swift | 01:55 | |
| zaitcev | Ceilometer does this _log_and_ignore_error() thing with except Exception. It's supposed to log "An exception occurred processing the API call: %s", but I hit situations wen log wasn't visible. And in fact Ceilometer does _LOG = logging.getLogger(__name__), which has a ton of reasons for not working. | 01:57 |
| zaitcev | In Swift all these things are muted unless they use get_logger() | 01:57 |
| zaitcev | There's some kind of patching through and setting priorities that getLogger alone does not do | 01:58 |
| rickyrem | I see what you're talking about | 02:00 |
| *** haomaiwang has quit IRC | 02:01 | |
| *** haomaiwa_ has joined #openstack-swift | 02:01 | |
| zaitcev | My preferred approach to this problem is to identify people who use except Exception, then meet up with them and beat them up with a PVC pipe | 02:02 |
| zaitcev | It's not solving the issue of getLogger not working though | 02:02 |
| zaitcev | Well, except in this case the exception would patch up from middleware into our code, where we log it properly. | 02:03 |
| *** klrmn has joined #openstack-swift | 02:04 | |
| zaitcev | One can verify if that thing fails by adding a little code snipped right ander _LOG, which writes the exception into /tmp/debugdump or whatnot... | 02:05 |
| rickyrem | Right, and I'm trying to trace down any remants of that | 02:06 |
| rickyrem | I'll see if I can do that | 02:06 |
| *** baojg has quit IRC | 02:07 | |
| zaitcev | John always tells me to use "import q" to do that, and I never get around to learn q. | 02:07 |
| openstackgerrit | James Nzomo proposed openstack/python-swiftclient: Fix segmented upload to pseudo-dir via <container> https://review.openstack.org/271728 | 02:07 |
| rickyrem | I had never seen q before. Looks cool | 02:09 |
| *** baojg has joined #openstack-swift | 02:10 | |
| *** klrmn has quit IRC | 02:12 | |
| *** Mingyu has quit IRC | 02:12 | |
| *** Mingyu has joined #openstack-swift | 02:13 | |
| rickyrem | Thanks a lot for your help zaitcev | 02:16 |
| zaitcev | rickyrem: sorry I can't do more, working here... | 02:16 |
| rickyrem | It's fine, I'll let you know if I find anything significant | 02:17 |
| *** alejandrito has joined #openstack-swift | 02:20 | |
| alejandrito | hi gyus, quick question, reading > https://blueprints.launchpad.net/swift/+spec/storage-tiers | 02:21 |
| alejandrito | its not clear if storage tiers is supported by swift through storage policies | 02:21 |
| alejandrito | can someone clarify ? | 02:21 |
| alejandrito | thx ^_^ | 02:21 |
| *** baojg has quit IRC | 02:23 | |
| *** baojg has joined #openstack-swift | 02:23 | |
| *** jamielennox|away is now known as jamielennox | 02:32 | |
| openstackgerrit | Merged openstack/python-swiftclient: Fix intermittent fail of test_delete_bulk_account https://review.openstack.org/274581 | 02:48 |
| *** baojg has quit IRC | 02:51 | |
| *** baojg has joined #openstack-swift | 02:52 | |
| *** thurloat_isgone is now known as thurloat | 02:54 | |
| *** davechen has joined #openstack-swift | 02:59 | |
| *** davechen has left #openstack-swift | 03:00 | |
| *** haomaiwa_ has quit IRC | 03:01 | |
| *** haomaiwang has joined #openstack-swift | 03:01 | |
| *** baojg has quit IRC | 03:02 | |
| *** baojg_ has joined #openstack-swift | 03:02 | |
| *** baojg_ has quit IRC | 03:05 | |
| *** jamielennox is now known as jamielennox|away | 03:06 | |
| *** baojg has joined #openstack-swift | 03:06 | |
| *** baojg has quit IRC | 03:09 | |
| *** baojg has joined #openstack-swift | 03:09 | |
| *** rickyrem has quit IRC | 03:09 | |
| *** jamielennox|away is now known as jamielennox | 03:11 | |
| *** sanchitmalhotra has joined #openstack-swift | 03:14 | |
| *** baojg has quit IRC | 03:16 | |
| *** baojg has joined #openstack-swift | 03:16 | |
| *** alejandrito has quit IRC | 03:17 | |
| janonymous | notmyname: Hi, i would work on it this weekend. I was very busy this month due to some releases in ofc so could not got much time to contribute, but from this weekend i am looking forward to start contributing :) | 03:18 |
| openstackgerrit | Merged openstack/swift: Allow IPv6 addresses/hostnames in StatsD target https://review.openstack.org/270991 | 03:23 |
| *** klrmn has joined #openstack-swift | 03:27 | |
| *** mrmoje has quit IRC | 03:33 | |
| *** thumpba has joined #openstack-swift | 03:34 | |
| *** Jeffrey4l has joined #openstack-swift | 03:42 | |
| *** daemontool has quit IRC | 03:46 | |
| *** daemontool has joined #openstack-swift | 03:46 | |
| *** baojg has quit IRC | 03:46 | |
| *** links has joined #openstack-swift | 03:49 | |
| *** thumpba has quit IRC | 03:53 | |
| *** thumpba has joined #openstack-swift | 03:54 | |
| *** baojg has joined #openstack-swift | 03:54 | |
| *** esker has quit IRC | 03:55 | |
| *** haomaiwang has quit IRC | 04:01 | |
| *** haomaiwang has joined #openstack-swift | 04:01 | |
| *** esker has joined #openstack-swift | 04:05 | |
| *** mgarza_ has quit IRC | 04:06 | |
| *** dmorita has quit IRC | 04:07 | |
| *** briancurtin has quit IRC | 04:11 | |
| *** thumpba_ has joined #openstack-swift | 04:12 | |
| *** cppforlife_ has quit IRC | 04:12 | |
| *** jamielennox is now known as jamielennox|away | 04:12 | |
| *** thumpba has quit IRC | 04:13 | |
| *** kaleta has quit IRC | 04:13 | |
| *** jroll has quit IRC | 04:13 | |
| *** CrackerJackMack has quit IRC | 04:13 | |
| *** kaleta has joined #openstack-swift | 04:13 | |
| *** chlong has quit IRC | 04:13 | |
| *** dosaboy_ has quit IRC | 04:14 | |
| *** CrackerJackMack has joined #openstack-swift | 04:14 | |
| *** jroll has joined #openstack-swift | 04:14 | |
| *** dosaboy has joined #openstack-swift | 04:15 | |
| *** silor has joined #openstack-swift | 04:18 | |
| *** amit213 has quit IRC | 04:19 | |
| *** BAKfr has quit IRC | 04:19 | |
| *** dfg_ has quit IRC | 04:19 | |
| *** tristanC has quit IRC | 04:19 | |
| *** dfg has joined #openstack-swift | 04:19 | |
| *** tristanC has joined #openstack-swift | 04:19 | |
| *** BAKfr has joined #openstack-swift | 04:20 | |
| *** briancurtin has joined #openstack-swift | 04:20 | |
| *** cppforlife_ has joined #openstack-swift | 04:23 | |
| *** mingdang1 has joined #openstack-swift | 04:24 | |
| *** thumpba_ has quit IRC | 04:25 | |
| *** chlong has joined #openstack-swift | 04:28 | |
| *** haomaiwang has quit IRC | 04:28 | |
| *** mingdang1 has quit IRC | 04:28 | |
| *** asettle has joined #openstack-swift | 04:40 | |
| *** bkumar has joined #openstack-swift | 04:50 | |
| openstackgerrit | Michael Barton proposed openstack/swift: go: fix some range parsing problems https://review.openstack.org/276039 | 05:08 |
| *** DericHorn-HP has joined #openstack-swift | 05:10 | |
| *** klrmn has quit IRC | 05:14 | |
| *** DericHorn-HP has quit IRC | 05:17 | |
| *** haomaiwa_ has joined #openstack-swift | 05:20 | |
| *** asettle has quit IRC | 05:26 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: Add --get <key> option to `swift capabilities` / `swift info` https://review.openstack.org/276043 | 05:27 |
| *** asettle has joined #openstack-swift | 05:28 | |
| *** baojg has quit IRC | 05:46 | |
| *** dmorita has joined #openstack-swift | 05:50 | |
| *** dmorita has quit IRC | 05:55 | |
| *** dmorita has joined #openstack-swift | 05:59 | |
| *** haomaiwa_ has quit IRC | 06:01 | |
| *** haomaiwang has joined #openstack-swift | 06:01 | |
| *** baojg has joined #openstack-swift | 06:02 | |
| *** zaitcev has quit IRC | 06:12 | |
| *** mingdang1 has joined #openstack-swift | 06:13 | |
| *** mingdang1 has quit IRC | 06:17 | |
| *** haomaiwang has quit IRC | 06:18 | |
| *** haomaiwang has joined #openstack-swift | 06:19 | |
| *** asettle has quit IRC | 06:26 | |
| *** proteusguy__ has quit IRC | 06:27 | |
| *** zhill has joined #openstack-swift | 06:29 | |
| *** tongli has quit IRC | 06:30 | |
| *** zhill has quit IRC | 06:39 | |
| *** proteusguy__ has joined #openstack-swift | 06:40 | |
| *** mingdang1 has joined #openstack-swift | 06:45 | |
| *** trifon has joined #openstack-swift | 06:49 | |
| *** SkyRocknRoll has joined #openstack-swift | 06:51 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Refactor versioned_writes middleware https://review.openstack.org/209315 | 06:57 |
| *** tdasilva has quit IRC | 07:00 | |
| *** haomaiwang has quit IRC | 07:01 | |
| *** haomaiwa_ has joined #openstack-swift | 07:01 | |
| *** Jeffrey4l has quit IRC | 07:19 | |
| *** tdasilva has joined #openstack-swift | 07:24 | |
| *** Jeffrey4l has joined #openstack-swift | 07:32 | |
| *** eranrom has joined #openstack-swift | 07:34 | |
| *** dmorita has quit IRC | 07:38 | |
| eranrom | Got a fast-post question. The spec says (under: proposed changes->Container-server): "Note that each object will continue to have only one row in the database table." Does this mean that if I delete an object merge_items would find the object row, mark it as deleted an 'move' its location timestamp-wise? | 07:39 |
| eranrom | If I understand the code correctly, this is indeed what is happening. That is, the original object row is deleted, and a new one is added at the 'bottom' of the table, where lower in the table means later timestamp. Can someone pls confirm? | 07:42 |
| *** zhill has joined #openstack-swift | 07:42 | |
| *** joeljwright has joined #openstack-swift | 07:50 | |
| *** ChanServ sets mode: +v joeljwright | 07:50 | |
| *** joeljwright has quit IRC | 07:50 | |
| *** gyee has quit IRC | 07:53 | |
| *** mvk_ has quit IRC | 07:57 | |
| *** cbartz has joined #openstack-swift | 07:58 | |
| *** haomaiwa_ has quit IRC | 08:01 | |
| *** haomaiwa_ has joined #openstack-swift | 08:01 | |
| *** ChubYann has quit IRC | 08:07 | |
| *** zhill has quit IRC | 08:09 | |
| *** rledisez has joined #openstack-swift | 08:10 | |
| *** mingdang1 has quit IRC | 08:17 | |
| *** mvk_ has joined #openstack-swift | 08:26 | |
| *** jmccarthy has joined #openstack-swift | 08:27 | |
| *** jistr has joined #openstack-swift | 08:29 | |
| *** geaaru has joined #openstack-swift | 08:34 | |
| *** nakagawamsa has joined #openstack-swift | 08:46 | |
| *** nakagawamsa has quit IRC | 08:52 | |
| *** jordanP has joined #openstack-swift | 08:52 | |
| *** lpabon has joined #openstack-swift | 09:00 | |
| *** haomaiwa_ has quit IRC | 09:01 | |
| *** haomaiwa_ has joined #openstack-swift | 09:01 | |
| *** baojg has quit IRC | 09:04 | |
| *** mrmoje has joined #openstack-swift | 09:07 | |
| *** baojg has joined #openstack-swift | 09:07 | |
| *** admin0 has joined #openstack-swift | 09:09 | |
| *** Mingyu has quit IRC | 09:13 | |
| *** admin0 has quit IRC | 09:13 | |
| *** ChanServ sets mode: +v tdasilva | 09:14 | |
| *** Mingyu has joined #openstack-swift | 09:14 | |
| *** mrmoje has quit IRC | 09:15 | |
| *** jistr has quit IRC | 09:16 | |
| *** openstackgerrit has quit IRC | 09:17 | |
| *** mrmoje has joined #openstack-swift | 09:17 | |
| *** SkyRocknRoll has quit IRC | 09:17 | |
| *** openstackgerrit has joined #openstack-swift | 09:17 | |
| *** asettle has joined #openstack-swift | 09:18 | |
| *** asettle has quit IRC | 09:24 | |
| *** asettle has joined #openstack-swift | 09:25 | |
| *** admin0 has joined #openstack-swift | 09:26 | |
| *** asettle_ has joined #openstack-swift | 09:32 | |
| *** asettle has quit IRC | 09:32 | |
| *** lpabon has quit IRC | 09:32 | |
| tdasilva | good morning | 09:35 |
| *** lpabon has joined #openstack-swift | 09:42 | |
| *** asettle_ is now known as asettle | 09:43 | |
| *** Jeffrey4l has quit IRC | 09:43 | |
| *** mrmoje has quit IRC | 09:47 | |
| *** mrmoje has joined #openstack-swift | 09:49 | |
| ntt | hi, I need to add a zone and a replica to a running swift cluster. Someone can help me please? I don't understand how can I add a replica | 09:52 |
| ntt | Actually I have only one node with one zone and replica = 1 | 09:52 |
| *** lpabon has quit IRC | 09:56 | |
| *** ppai has joined #openstack-swift | 09:56 | |
| *** haomaiwa_ has quit IRC | 10:01 | |
| *** haomaiwang has joined #openstack-swift | 10:01 | |
| *** Jeffrey4l has joined #openstack-swift | 10:01 | |
| *** acoles_ is now known as acoles | 10:02 | |
| acoles | eranrom: correct, delete creates a new | 10:06 |
| *** dudi has joined #openstack-swift | 10:08 | |
| acoles | eranrom: correct, delete creates a new row | 10:09 |
| *** lpabon has joined #openstack-swift | 10:10 | |
| dudi | hello, can anybody tell me what should be the permissions for /etc/swift/ dir | 10:11 |
| dudi | because its not creating /var/cache/swift folder when i start all swift services | 10:11 |
| *** mingdang1 has joined #openstack-swift | 10:12 | |
| dudi | account-server.conf : http://pastebin.com/y1tQq6AL | 10:12 |
| *** lpabon has quit IRC | 10:16 | |
| acoles | dudi /etc/swift should be owned by whatever user runs swift processes, in examples that is 'swift' user http://docs.openstack.org/developer/swift/development_saio.html#configuring-each-node | 10:20 |
| acoles | dudi: see also http://docs.openstack.org/developer/swift/development_saio.html#common-post-device-setup | 10:21 |
| dudi | swift creates it for me or should i create manually | 10:22 |
| dudi | ok iys in ur link..thanks | 10:23 |
| *** baojg_ has joined #openstack-swift | 10:24 | |
| *** baojg has quit IRC | 10:25 | |
| *** Jeffrey4l has quit IRC | 10:29 | |
| admin0 | ho wis the swift internal endpoint being used ? | 10:35 |
| admin0 | * how is | 10:35 |
| *** dudi has quit IRC | 10:36 | |
| *** baojg_ has quit IRC | 10:38 | |
| *** baojg has joined #openstack-swift | 10:45 | |
| eranrom | acoles: thanks | 10:50 |
| acoles | eranrom: hi! I just added further comment on patch 270961 | 10:51 |
| patchbot | acoles: https://review.openstack.org/#/c/270961/ - swift - Container-Sync to perform HEAD before PUT object o... | 10:51 |
| acoles | eranrom: I have been trying to think of any downside to the change (wrt using row timestamp as the condition to sync or not) other than a delay | 10:52 |
| *** baojg has quit IRC | 10:53 | |
| *** haomaiwang has quit IRC | 10:54 | |
| eranrom | acoles: looking | 11:02 |
| *** dabukalam has joined #openstack-swift | 11:10 | |
| *** etienneme has joined #openstack-swift | 11:11 | |
| *** baojg has joined #openstack-swift | 11:13 | |
| openstackgerrit | venkatamahesh proposed openstack/swift: Use uppercase 'S' in word "OpenStack" https://review.openstack.org/276164 | 11:18 |
| * tdasilva wonders if there will be a pep rule on how to spell "OpenStack" correctly :P | 11:20 | |
| acoles | tdasilva: s/pep/PEP/ :) | 11:26 |
| *** oshritf has joined #openstack-swift | 11:27 | |
| *** sanchitmalhotra has quit IRC | 11:29 | |
| tdasilva | lol | 11:29 |
| tdasilva | sorry! | 11:29 |
| eranrom | acoles: Reviewing your comment with oshritf we are not sure what is the bottom line - that is - do you think that the delete scenario exposes a correctness problem or merely an upoptimized order of replication | 11:29 |
| acoles | eranrom: no. my point is that as far as I can see the only change in "observed behaviour" with the patch is that some sync actions may be delayed until a container update occurs at the source, because the container's view of the latest object timestamp may be stale. | 11:33 |
| acoles | eranrom: but AFAICT the change would never cause data to be sync'd that would, on master, not be sync'd. i.e. even if a later delete has not yet updated the container, the sync process would still not actually transmit the older data to the destination. which is good! | 11:35 |
| oshritf | acoles: right, in the case of delete in t2, at t1 the object may be synced to remote (in case it was not synced before) and later deleted (since no get was performed for latest) | 11:36 |
| *** haomaiwa_ has joined #openstack-swift | 11:37 | |
| acoles | tdasilva: more seriously, I am forming the view that with a project so large and with so many contributors, there is little point in fixing things that are not verified by a test of some kind. Particularly things that make no difference to the behaviour of the code or the correctness of the documentation. | 11:37 |
| *** SkyRocknRoll has joined #openstack-swift | 11:39 | |
| tdasilva | acoles: yeah, that was sort of my point...this kinda of changes are only "valid" until the next contributor who has no idea of this "rule" writes some new doc and spells "Openstack" | 11:39 |
| tdasilva | am I supposed to -1 that?? sounds dumb | 11:39 |
| acoles | oshritf: well I read it differently...in the case of delete at t2, when the sync process finds the row at t1, it will issue the GETs for the object, but the the GETs will return 404 (because the object has been deleted at t2), so no data gets sync'd (which is a good thing) | 11:39 |
| acoles | tdasilva: it reminds me of the saying "tidying your house while the kids are growing is like clearing the drive while its still snowing" | 11:41 |
| acoles | tdasilva: i know what you mean...-1, ignore, defer to notmyname ?? | 11:41 |
| tdasilva | acoles: yeah, i mean...i won't even think of looking for that..it's just a waste of time | 11:42 |
| tdasilva | and i feel bad for whoever is sending the patch because he is sending for all projects | 11:42 |
| oshritf | acoles: so your suggestion is not to skip GET to "sync" local services before checking with remote? | 11:43 |
| acoles | tdasilva: looks like its all in doc files so probably little risk of generating merge conflicts, but then I see "swift" and "Swift" in same docs...so if we really care... | 11:46 |
| tdasilva | hehe...oh boy..here we go | 11:46 |
| eranrom | tdasilva,: How about -1 conditioned on adding a test? | 11:46 |
| oshritf | acoles: in phase 2 of container sync, where iterating over all objects, the "local sync" occurs #of services times. This is what we wanted to save - since they all check all objects, at least one will have the most updated data. However, "local sync" is better in cases where they are rapid changes, such as rapid updates or updates+deletes | 11:47 |
| acoles | eranrom: sounds very reasonable !) | 11:47 |
| tdasilva | eranrom: adding a test on word spelling? | 11:47 |
| *** jcook_ has quit IRC | 11:48 | |
| eranrom | tdasilva,: Is that such a bad idea? Wll not every word just - you know - OpenStack / Swift | 11:48 |
| eranrom | s/Wll/Well | 11:48 |
| eranrom | tdasilva,: PEP rule | 11:48 |
| acoles | eranrom: tdasilva the test would have to differentiate between text and urls | 11:48 |
| *** bkumar has quit IRC | 11:50 | |
| tdasilva | eranrom: PEP rule would make sense so we don't have to deal with it again, but yeah, acoles has a good point | 11:50 |
| acoles | oshritf: ok, I am concerned that my comment has caused confusion! I think I am agreeing with you. "local sync" (i.e. GET to all local objects before comparing local time with remote time, i.e. master branch) will result in a local change sometimes being sync'd sooner. But, yes, it is costly. And I think it is worth the introducing the potential delay in order to save the GETs. But I wanted to make sure you , eranrom and I | 11:52 |
| acoles | had the same understanding of that trade off. | 11:52 |
| acoles | tdasilva: eranrom I think it needs a PTL executive decision and we should wake him right now ;) | 11:53 |
| eranrom | :-) | 11:53 |
| tdasilva | lol | 11:53 |
| tdasilva | it's only 22:53 in melbourne, he should be around | 11:54 |
| tdasilva | :D | 11:54 |
| mahatic | hehe yes. but a spelling correct *and* a test, wouldn't that add to the code churn | 11:54 |
| mahatic | correction* | 11:55 |
| eranrom | aolces: Great, so we are on the same page, and continue to work on the patch with the current logic: Do not perform local GET at all times. | 11:55 |
| *** jistr has joined #openstack-swift | 11:56 | |
| acoles | eranrom: yep. of course I reserve the right to change my mind ;) | 11:57 |
| oshritf | :) | 11:57 |
| eranrom | :-) | 11:57 |
| *** resker has joined #openstack-swift | 11:58 | |
| *** haomaiwa_ has quit IRC | 12:01 | |
| *** esker has quit IRC | 12:01 | |
| *** haomaiwang has joined #openstack-swift | 12:01 | |
| *** ppai has quit IRC | 12:04 | |
| *** silor has quit IRC | 12:05 | |
| *** Guest44793 has quit IRC | 12:07 | |
| *** mingdang1 has quit IRC | 12:09 | |
| openstackgerrit | Merged openstack/swift: Fixing typo in unit test https://review.openstack.org/275917 | 12:10 |
| *** asettle has quit IRC | 12:27 | |
| *** baojg has quit IRC | 12:28 | |
| *** baojg has joined #openstack-swift | 12:31 | |
| *** dmorita has joined #openstack-swift | 12:39 | |
| *** Mingyu has quit IRC | 12:40 | |
| *** Mingyu has joined #openstack-swift | 12:41 | |
| *** dmorita has quit IRC | 12:44 | |
| *** admin0 has quit IRC | 12:44 | |
| *** daemontool_ has joined #openstack-swift | 12:48 | |
| *** natarej_ has joined #openstack-swift | 12:49 | |
| *** kaleta_ has joined #openstack-swift | 12:50 | |
| *** mrmoje_ has joined #openstack-swift | 12:51 | |
| *** rledisez1 has joined #openstack-swift | 12:52 | |
| *** kaleta has quit IRC | 12:53 | |
| *** mrmoje has quit IRC | 12:53 | |
| *** rledisez has quit IRC | 12:53 | |
| *** SkyRocknRoll has quit IRC | 12:53 | |
| *** daemontool has quit IRC | 12:53 | |
| *** natarej has quit IRC | 12:53 | |
| *** kaleta_ is now known as kaleta | 12:53 | |
| *** SkyRocknRoll has joined #openstack-swift | 12:56 | |
| openstackgerrit | Alistair Coles proposed openstack/python-swiftclient: Support --os-identity-api-version option https://review.openstack.org/275719 | 12:57 |
| *** daemontool_ has quit IRC | 12:58 | |
| *** mingdang1 has joined #openstack-swift | 12:58 | |
| oshritf | acoles: Does it make sense to upload patch set 2 to the timestamp fix with moving the headers and body setting lines to after the exception handling? it might be easier to follow the flow | 12:59 |
| *** haomaiwang has quit IRC | 13:01 | |
| *** haomaiwa_ has joined #openstack-swift | 13:01 | |
| *** mingdang1 has quit IRC | 13:02 | |
| *** admin0 has joined #openstack-swift | 13:03 | |
| acoles | oshritf: do you mean patch 275147? I'll take a look | 13:03 |
| patchbot | acoles: https://review.openstack.org/#/c/275147/ - swift - Fix time-shifting of objects PUT with container-sync | 13:03 |
| *** [1]eranrom has joined #openstack-swift | 13:03 | |
| oshritf | acoles: yes, it's minor, but relates to the change...# Populate with newest object data headers = source_obj_info body = source_obj_iter for key in ('date', 'last-modified'): | 13:04 |
| *** eranrom has quit IRC | 13:05 | |
| *** [1]eranrom is now known as eranrom | 13:05 | |
| *** kei_yama has quit IRC | 13:16 | |
| *** jcook_ has joined #openstack-swift | 13:19 | |
| *** baojg has quit IRC | 13:27 | |
| *** baojg has joined #openstack-swift | 13:28 | |
| mahatic | acoles: hi, could I bug for a min? | 13:29 |
| *** petertr7_away is now known as petertr7 | 13:34 | |
| acoles | mahatic: go ahead | 13:38 |
| *** david-lyle has quit IRC | 13:38 | |
| mahatic | acoles: thanks. for the incomplete read issue here: https://github.com/openstack/swift/blob/feature/crypto/swift/common/middleware/encrypter.py#L115-115 -> since a close method can't be called (because it is not an iterable), I was thinking could we check on the content length and use it the etag calculatations.. | 13:40 |
| mahatic | acoles: i.e. only when we hit the last chunk (determined by using content length), do we calculate etag and the subsequent things we are doing now. | 13:40 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Fix time-shifting of objects PUT with container-sync https://review.openstack.org/275147 | 13:42 |
| acoles | oshritf: ^^ I think that is better than moving the lines | 13:42 |
| *** Jeffrey4l has joined #openstack-swift | 13:48 | |
| acoles | mahatic: looking | 13:49 |
| mahatic | okay | 13:50 |
| oshritf | acoles: much better! thanks | 13:51 |
| *** lpabon has joined #openstack-swift | 13:55 | |
| *** links has quit IRC | 13:56 | |
| *** haomaiwa_ has quit IRC | 14:01 | |
| *** haomaiwa_ has joined #openstack-swift | 14:01 | |
| *** SkyRocknRoll has quit IRC | 14:02 | |
| *** barra204 has quit IRC | 14:04 | |
| *** shakamunyi has joined #openstack-swift | 14:06 | |
| *** daemontool has joined #openstack-swift | 14:09 | |
| *** peyton_ has joined #openstack-swift | 14:18 | |
| *** peyton has quit IRC | 14:20 | |
| *** acoles is now known as acoles_ | 14:23 | |
| petertr7 | Hi acoles_ , could I get your thoughts on something? | 14:27 |
| *** acoles_ is now known as acoles | 14:29 | |
| *** wshao has joined #openstack-swift | 14:29 | |
| *** wshao has left #openstack-swift | 14:29 | |
| *** admin0 has quit IRC | 14:33 | |
| *** resker has quit IRC | 14:36 | |
| acoles | mahatic: sorry I got called to a meeting | 14:37 |
| acoles | mahatic: its not a big deal, just slightly more defensive coding. Its perhaps unlikely, but if the reader counts bytes until they read exactly what they expect, then the reader may never call read() for the case when there is no more data to be read. In which case the code as it is never calculates the etag etc.Then the reader calls the footers_callback | 14:39 |
| mahatic | acoles: no problem! | 14:39 |
| acoles | and no footers have been prepared. So i just wonder if its safer to prepare the footers in the footers callback | 14:39 |
| acoles | mahatic: and maybe have a check there that all the data got consumed | 14:40 |
| acoles | petertr7: sure | 14:40 |
| tdasilva | wbhuber: around? | 14:40 |
| petertr7 | Thanks. I'm interested in working on a new feature for python-swiftclient. I prepared a blueprint-like/doc with my plan. Could you have a look? | 14:40 |
| petertr7 | https://docs.google.com/document/d/12jAgzSuoa3n57edVLAIecA_KTiSvHwusQS4XWjRScwc/edit | 14:40 |
| *** admin0 has joined #openstack-swift | 14:41 | |
| mahatic | acoles: alright, thanks. (will test it out by moving the code accordingly) | 14:44 |
| acoles | petertr7: so is the cache file stored locally or sent to swift? | 14:46 |
| petertr7 | sent to swift | 14:46 |
| petertr7 | I'm open to either | 14:46 |
| petertr7 | but I think the x-delete-after header would help keep things clean | 14:47 |
| petertr7 | otherwise the user's workspace will start filling up with cache files | 14:47 |
| *** taane is now known as tanee | 14:48 | |
| petertr7 | I'm not a huge fan of updating this cache file for each successful segment, as the default max # of segments is 1000 | 14:48 |
| petertr7 | that's 1000 uploads each time a segment upload is complete | 14:48 |
| petertr7 | perhaps it can be batched | 14:48 |
| acoles | petertr7: whats your plan if the local file has changed since the cache got created - so the cached state is invalid? | 14:49 |
| petertr7 | I'm not sure yet. perhaps save the timestamp in the cache when uploading starts | 14:50 |
| petertr7 | and compare that when the upload is "continued" | 14:50 |
| petertr7 | rewrite/delete the cache accordingly and start segmenting from 0 | 14:50 |
| acoles | right. are you aware of recent features added to retry uploads ? (commit d4157ce5) its a little different but related | 14:52 |
| petertr7 | No I'm not. Let me check it out | 14:52 |
| acoles | petertr7: i think timburke and joelwright may give better feedback than me. | 14:53 |
| petertr7 | Thanks acoles | 14:55 |
| acoles | petertr7: in terms of process, i would point out that launchpad blueprints don't work too well. you could write a swift-spec https://specs.openstack.org/openstack/swift-specs/readme_link.html - I advise keeping it short and simple :) | 14:56 |
| petertr7 | Thanks acoles, I will do that. Was the doc I created tmi? | 14:56 |
| acoles | petertr7: no not tmi. a spec is not a "must" but it can provide a place for discussion since it goes through the usual gerrit review process. If you have code already then you could just propose a patch and have the discussion there | 14:58 |
| petertr7 | acoles: oh I see. Thank you. I haven't started coding it yet. I wasn't sure how people felt about creating an extra file that gets put on swift | 14:58 |
| *** ppai has joined #openstack-swift | 14:59 | |
| acoles | petertr7: one thought - maybe if the cache file were local, it could be less automatic - like a "recovery file" that can be used with a --recovery option - so user gets more visibility of what's happening. but thats a CLI feature use case I guess. | 14:59 |
| petertr7 | acoles: hmmm that's a good idea. it would resolve the issue of the file changing after an initial upload has started | 15:00 |
| *** haomaiwa_ has quit IRC | 15:01 | |
| petertr7 | as the user will be able to simply delete the recovery file | 15:01 |
| *** haomaiwang has joined #openstack-swift | 15:01 | |
| acoles | petertr7: i think you have already identified one issue with caching the file in swift which is needing to do a read/modify/write of that object for every successful segment upload. | 15:01 |
| acoles | petertr7: i'm away for a bit | 15:02 |
| petertr7 | okay. Thanks so much for your feedback acoles, it is much appreciated! | 15:02 |
| openstackgerrit | Ondřej Nový proposed openstack/swift: Howto check sanity of manpage in devel guide https://review.openstack.org/276280 | 15:03 |
| *** trifon has quit IRC | 15:04 | |
| tdasilva | onovy: cool, TIL: http://fpaste.org/318456/98540145/ | 15:09 |
| *** Mingyu has quit IRC | 15:11 | |
| *** Mingyu has joined #openstack-swift | 15:12 | |
| onovy | hmm | 15:12 |
| onovy | https://review.openstack.org/#/c/275406/ fixing this :) | 15:13 |
| tdasilva | onovy: mmm, i broke the file on purpose, just so i could see warnings | 15:17 |
| openstackgerrit | Ondřej Nový proposed openstack/swift: Fixed manpages errors. https://review.openstack.org/275406 | 15:21 |
| onovy | tdasilva: ah! :) | 15:21 |
| *** acoles is now known as acoles_ | 15:23 | |
| *** eranrom has quit IRC | 15:23 | |
| *** acoles_ is now known as acoles | 15:24 | |
| openstackgerrit | Ondřej Nový proposed openstack/swift: Howto check sanity of manpage in devel guide https://review.openstack.org/276280 | 15:24 |
| *** mgarza has joined #openstack-swift | 15:29 | |
| *** esker has joined #openstack-swift | 15:32 | |
| *** baojg has quit IRC | 15:33 | |
| *** silor has joined #openstack-swift | 15:34 | |
| *** arch-nemesis has joined #openstack-swift | 15:35 | |
| *** baojg has joined #openstack-swift | 15:37 | |
| *** jcook_ has quit IRC | 15:37 | |
| *** baojg has quit IRC | 15:37 | |
| openstackgerrit | Merged openstack/swift: Stop nesting functions unnecessarily https://review.openstack.org/275920 | 15:43 |
| *** eranrom has joined #openstack-swift | 15:43 | |
| *** silor1 has joined #openstack-swift | 15:45 | |
| *** bjkeller has joined #openstack-swift | 15:47 | |
| *** silor has quit IRC | 15:47 | |
| *** silor1 is now known as silor | 15:47 | |
| oshritf | acoles: if-modified-since accepts only "%a, %d %b %Y %H:%M:%S GMT" format? | 15:54 |
| *** garthb has joined #openstack-swift | 15:55 | |
| *** eranrom has quit IRC | 15:59 | |
| *** haomaiwang has quit IRC | 16:01 | |
| *** haomaiwa_ has joined #openstack-swift | 16:01 | |
| *** petertr7 is now known as petertr7_away | 16:01 | |
| *** admin6 has joined #openstack-swift | 16:02 | |
| *** mrmoje_ has quit IRC | 16:03 | |
| *** petertr7_away is now known as petertr7 | 16:03 | |
| *** ppai has quit IRC | 16:03 | |
| *** mrmoje has joined #openstack-swift | 16:05 | |
| admin6 | Hi chat. I got hundreds of message like this one when rebuilding an erasure coding ring : RingValidationWarning: The partition 524287 has been assigned to duplicate devices [18, 19, 5, 12, 10, 8, 9, 7, 11, 4, 12, 5] | 16:05 |
| glange | how many devices do you have in your ring? | 16:06 |
| cschwede | i was wondering the same… | 16:06 |
| admin6 | Currently, only 20 | 16:06 |
| glange | it might not work very well with that few devices | 16:06 |
| glange | admin6: you might want to talk to clayg when he gets on -- he's worked on that | 16:07 |
| admin6 | glange: ok thanks. I’ll try to catch him. | 16:09 |
| *** lpabon has quit IRC | 16:10 | |
| *** chsc has joined #openstack-swift | 16:12 | |
| *** ppai has joined #openstack-swift | 16:16 | |
| *** nadeem has joined #openstack-swift | 16:21 | |
| *** admin6 has quit IRC | 16:21 | |
| *** esker has quit IRC | 16:23 | |
| *** david-lyle has joined #openstack-swift | 16:24 | |
| *** esker has joined #openstack-swift | 16:24 | |
| *** admin0 has quit IRC | 16:25 | |
| openstackgerrit | Shashirekha Gundur proposed openstack/swift: change default ports for servers https://review.openstack.org/274840 | 16:27 |
| *** mrmoje has quit IRC | 16:35 | |
| *** mrmoje has joined #openstack-swift | 16:36 | |
| *** mvk_ has quit IRC | 16:37 | |
| *** baojg has joined #openstack-swift | 16:38 | |
| openstackgerrit | Donagh McCabe proposed openstack/swift: Added links to API reference and usage guides https://review.openstack.org/276342 | 16:42 |
| *** baojg has quit IRC | 16:44 | |
| *** zaitcev has joined #openstack-swift | 16:44 | |
| *** ChanServ sets mode: +v zaitcev | 16:44 | |
| *** petertr7 is now known as petertr7_away | 16:46 | |
| *** petertr7_away is now known as petertr7 | 16:59 | |
| *** haomaiwa_ has quit IRC | 17:01 | |
| *** haomaiwang has joined #openstack-swift | 17:01 | |
| *** gyee has joined #openstack-swift | 17:02 | |
| *** jistr has quit IRC | 17:02 | |
| *** zhill has joined #openstack-swift | 17:03 | |
| *** silor has quit IRC | 17:04 | |
| *** jordanP has quit IRC | 17:06 | |
| *** nadeem has quit IRC | 17:09 | |
| *** nadeem has joined #openstack-swift | 17:10 | |
| *** cbartz has quit IRC | 17:13 | |
| *** petertr7 is now known as petertr7_away | 17:14 | |
| timburke | acoles: d4157ce5 wasn't a feature, it was a fix. it worked like a year ago :( | 17:16 |
| *** jistr has joined #openstack-swift | 17:16 | |
| *** jistr has quit IRC | 17:18 | |
| timburke | petertr7_away: i'm not opposed. how much can we trust that the file hasn't changed, though? if we can trust both content *and* mtime, it might be worth heading the segment destinations and comparing etag vs actual md5, similar to what's done for --skip-identical | 17:19 |
| *** ppai has quit IRC | 17:19 | |
| timburke | better yet: container listing with --prefix so we can get all the segments at once | 17:19 |
| wbhuber | tdsailva: i'm here | 17:27 |
| *** mrmoje has quit IRC | 17:32 | |
| *** lyrrad has joined #openstack-swift | 17:34 | |
| *** esker has quit IRC | 17:37 | |
| sgundur | anyone know of any reported delays in jenkins - gate checks ? | 17:38 |
| *** esker has joined #openstack-swift | 17:38 | |
| *** ho_away has joined #openstack-swift | 17:39 | |
| *** dmorita has joined #openstack-swift | 17:43 | |
| *** rledisez1 has quit IRC | 17:45 | |
| *** klrmn has joined #openstack-swift | 17:47 | |
| *** baojg has joined #openstack-swift | 17:50 | |
| *** amit213 has joined #openstack-swift | 17:55 | |
| *** amit213 has quit IRC | 17:55 | |
| *** amit213 has joined #openstack-swift | 17:56 | |
| *** eranrom has joined #openstack-swift | 17:58 | |
| *** baojg has quit IRC | 17:59 | |
| *** oshritf has quit IRC | 18:00 | |
| *** haomaiwang has quit IRC | 18:01 | |
| *** haomaiwang has joined #openstack-swift | 18:01 | |
| *** esker has quit IRC | 18:04 | |
| *** arch-nemesis has quit IRC | 18:05 | |
| *** arch-nemesis has joined #openstack-swift | 18:06 | |
| openstackgerrit | Richard Hawkins proposed openstack/swift: Add option for label to static web listings https://review.openstack.org/267123 | 18:15 |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: Add --json option to `swift capabilities` / `swift info` https://review.openstack.org/276382 | 18:29 |
| *** tongli has joined #openstack-swift | 18:31 | |
| *** Jeffrey4l has quit IRC | 18:35 | |
| *** ChubYann has joined #openstack-swift | 18:38 | |
| *** mrmoje has joined #openstack-swift | 18:40 | |
| *** ho_away has quit IRC | 18:45 | |
| *** petertr7_away is now known as petertr7 | 18:47 | |
| petertr7 | Hi timburke - thanks for the feedback | 18:48 |
| petertr7 | timburke: my one concern is that if you're uploading a 0.5TB file, doing md5 checks on segments will take forever | 18:48 |
| *** mrmoje has quit IRC | 18:50 | |
| timburke | that's fair. i suppose if we're doing listings like that, we're already assuming same mtime and same size (and same chunk size). might be enough to assume that if we see a segment, it's correct | 18:50 |
| petertr7 | timburke: Would be safe to assume, with proper documentation, that the user should know this "recovery" option is for the same file and that it would be invalid if the file has changed? | 18:52 |
| petertr7 | i.e. user attempts to upload file a.py using the --recovery flag. goes home for the night and the next morning sees the upload did not complete | 18:52 |
| petertr7 | then retries the upload on the same a.py | 18:52 |
| petertr7 | and the user would be or should be aware if a.py has changed since the original attempt to upload | 18:53 |
| timburke | so what all do we get in the cache object that we don't get in container listings? the chunk size & large-object type? how do we decide how long the .cache should hang around? | 18:54 |
| *** esker has joined #openstack-swift | 18:54 | |
| petertr7 | well if you do a container listing you need to know the location of the segments, which a user may not know | 18:55 |
| petertr7 | I'm not sure how long the .cache file should stay around. I'm open to suggestions | 18:55 |
| petertr7 | I guess it would vary depending on the size of the file | 18:56 |
| petertr7 | If a user is uploading an archived file that is multi GB or even 1 TB it could take a variable amount of time to complete the upload | 18:56 |
| timburke | user shouldn't need to know; during uploads, swiftclient decides it based on object attributes: https://github.com/openstack/python-swiftclient/blob/2.7.0/swiftclient/service.py#L1836-L1845 | 18:57 |
| *** admin0 has joined #openstack-swift | 18:59 | |
| petertr7 | Hmm so I guess instead of creating and managing a cache file, a "--recovery" flag could to a listing for the segments | 18:59 |
| petertr7 | and continue where it left off | 18:59 |
| petertr7 | the manifest only needs location and etag which is available when you do a listing | 18:59 |
| petertr7 | *and size | 18:59 |
| timburke | yep. on the plus side, we'd be able to update the x-delete-at whenever we update the cache. but i think the delay we want (if we follow this approach) will be dominated by user convenience, not upload time | 19:00 |
| *** haomaiwang has quit IRC | 19:01 | |
| petertr7 | thanks timburke | 19:01 |
| petertr7 | So just to clarify from the user perspective | 19:01 |
| *** haomaiwang has joined #openstack-swift | 19:01 | |
| petertr7 | user uploads a static large object, goes home for the night | 19:01 |
| petertr7 | the next morning realizes the upload did not finish. runs the upload again with --recovery | 19:01 |
| petertr7 | and python-swiftclient will do a listing and pick up where the file left off | 19:02 |
| petertr7 | no cache file created locally | 19:02 |
| timburke | or dlo. maybe leaves for the weekend, comes back to it Monday. but yeah, this all sounds like a good experience | 19:03 |
| petertr7 | awesome, thanks for your feedback, it's much appreciated | 19:04 |
| acoles | timburke: petertr7 when listing the segments container, how would the recovery phase differentiate old segments from a previous (successful) upload vs missing segments form failed upload? can we use the modified time for that? | 19:05 |
| timburke | petertr7: might be nice to display a warning if --recovery is passed and no segments are found (and we're only uploading one object). since it probably means the file has changed and there may be orphaned segments | 19:06 |
| petertr7 | thanks timburke, I will make a not of that | 19:06 |
| petertr7 | *note | 19:06 |
| timburke | acoles: mtime gets baked into the segment object name | 19:06 |
| acoles | the segments' x-timestamp may not be in segment order, so I'm not sure we can simply check for a chronological progression of segments | 19:06 |
| acoles | timburke: ah, right, that helps | 19:06 |
| petertr7 | is the header "x-object-meta-mtime" the mtime of the segment upload or the timestamp of the file being segmented? | 19:08 |
| timburke | so, if we're dealing with an actual file (so we can get & reasonably trust things like file size and mtime), and if we feel safe in assuming that the user would use the same segment size and large-object type, then we should be able to rely on just container listings | 19:09 |
| petertr7 | yes , of course we can raise warnings if the segment size doesn't match | 19:10 |
| *** petertr7 has left #openstack-swift | 19:10 | |
| *** david-lyle has quit IRC | 19:11 | |
| timburke | petertr7: the mtime from the file on disk, which will also be set as the x-object-meta-mtime for the manifest. i think (but need to confirm) that we don't bother to set x-object-meta-mtime for the segments. and it will have nothing to do with the x-timestamp for any of them | 19:11 |
| *** petertr7 has joined #openstack-swift | 19:14 | |
| *** acoles is now known as acoles_ | 19:14 | |
| petertr7 | Sorry timburke, anoles, I somehow deleted this channel from my irc bouncer and kicked myself out. I've missed any messages you sent in the last couple of minutes | 19:15 |
| timburke | i think just this: petertr7: the mtime from the file on disk, which will also be set as the x-object-meta-mtime for the manifest. i think (but need to confirm) that we don't bother to set x-object-meta-mtime for the segments. and it will have nothing to do with the x-timestamp for any of them | 19:16 |
| petertr7 | Thanks | 19:16 |
| petertr7 | right okay | 19:16 |
| *** admin0 has quit IRC | 19:16 | |
| *** barker has joined #openstack-swift | 19:19 | |
| *** barker has quit IRC | 19:25 | |
| *** baojg has joined #openstack-swift | 19:29 | |
| *** geaaru has quit IRC | 19:40 | |
| Mingyu | It seems that in some system, the environment markers don't support <, <= etc ? | 19:40 |
| *** diazjf has joined #openstack-swift | 19:42 | |
| wbhuber | how does x-newest header work on containers? i understand that x-newest makes sense when used - to get the latest version of an object by newest replica (out of two or three), but containers? how? | 19:43 |
| *** zhill has quit IRC | 19:46 | |
| *** mragupat has joined #openstack-swift | 19:48 | |
| *** admin0 has joined #openstack-swift | 19:48 | |
| *** admin0 has quit IRC | 19:49 | |
| *** admin0 has joined #openstack-swift | 19:49 | |
| openstackgerrit | Michael Barton proposed openstack/swift: go: make it compile on OSX https://review.openstack.org/276418 | 19:52 |
| *** diazjf has quit IRC | 19:52 | |
| *** diazjf has joined #openstack-swift | 19:55 | |
| *** stevemar has quit IRC | 19:56 | |
| *** SkyRocknRoll has joined #openstack-swift | 19:56 | |
| *** zhill has joined #openstack-swift | 19:58 | |
| *** haomaiwang has quit IRC | 20:01 | |
| *** haomaiwang has joined #openstack-swift | 20:01 | |
| *** baojg has quit IRC | 20:03 | |
| *** gyee has quit IRC | 20:05 | |
| *** gordc has joined #openstack-swift | 20:14 | |
| gordc | hi folks, just curious, but is there an easy way to define an offset when running get_container? | 20:15 |
| petertr7 | gordc: off by number? | 20:15 |
| *** diazjf has quit IRC | 20:15 | |
| petertr7 | i.e. list objects in <container> starting after X objects? | 20:16 |
| *** admin0 has quit IRC | 20:16 | |
| gordc | petertr7: correct | 20:16 |
| petertr7 | atm I don't think so. the only thing close to that is using the marker which requires you to know the names in the listing which is not helpful if you want exactly X offset | 20:17 |
| petertr7 | would be a really helpful feature for pagination though | 20:18 |
| gordc | petertr7: yeah. that's what i'm trying to accomplish... sucks. | 20:18 |
| gordc | petertr7: thanks for the help though | 20:18 |
| petertr7 | gordc: your welcome | 20:19 |
| petertr7 | gordc: I too was trying to figure out how to create pagination like feature | 20:19 |
| petertr7 | gordc: and i ended up giving up cause I read stuff into memory, created a stack and used object names as markers. it was too much of a mess | 20:19 |
| gordc | yeah, i found the ML topic where they decided to use marker. i guess it was chosen to make repeatable queries possible. | 20:20 |
| gordc | sucks that i have the opposite requirement. | 20:21 |
| *** diazjf has joined #openstack-swift | 20:31 | |
| *** fthiagogv_ has joined #openstack-swift | 20:37 | |
| *** eranrom has quit IRC | 20:37 | |
| *** trifon has joined #openstack-swift | 20:38 | |
| *** briancli1e has joined #openstack-swift | 20:41 | |
| *** wolsen_ has joined #openstack-swift | 20:41 | |
| *** briancurtin_ has joined #openstack-swift | 20:45 | |
| *** diazjf has quit IRC | 20:47 | |
| *** tdasilva has quit IRC | 20:47 | |
| *** briancurtin has quit IRC | 20:47 | |
| *** fthiagogv has quit IRC | 20:47 | |
| *** briancline has quit IRC | 20:47 | |
| *** rsFF has quit IRC | 20:47 | |
| *** wolsen has quit IRC | 20:47 | |
| *** pdardeau has quit IRC | 20:47 | |
| *** haomaiwang has quit IRC | 20:47 | |
| *** briancurtin_ is now known as briancurtin | 20:48 | |
| *** tdasilva has joined #openstack-swift | 20:49 | |
| *** diazjf has joined #openstack-swift | 20:49 | |
| *** pdardeau has joined #openstack-swift | 20:49 | |
| *** rsFF has joined #openstack-swift | 20:49 | |
| *** tdasilva has quit IRC | 20:49 | |
| *** tdasilva has joined #openstack-swift | 20:49 | |
| *** haomaiwang has joined #openstack-swift | 20:49 | |
| *** diazjf has quit IRC | 20:50 | |
| *** rsFF has quit IRC | 20:51 | |
| *** rsFF has joined #openstack-swift | 20:52 | |
| *** gordc has quit IRC | 20:52 | |
| *** dmorita has quit IRC | 20:58 | |
| *** haomaiwang has quit IRC | 21:01 | |
| *** haomaiwang has joined #openstack-swift | 21:01 | |
| *** baojg has joined #openstack-swift | 21:04 | |
| *** diazjf has joined #openstack-swift | 21:05 | |
| *** ChanServ sets mode: +v tdasilva | 21:05 | |
| *** lpabon has joined #openstack-swift | 21:05 | |
| *** SkyRocknRoll has quit IRC | 21:06 | |
| timburke | petertr7: the trouble with an absolute offset into container listings is that the objects in the container are *constantly* changing. if you get one page of, say, 50 items, when you go to get the next page at ?offset=50 you have *absolutely no idea* where that will be in relation to your original request. if enough writes came in (or if there was a partition, or if...) even the last object returned could come *before* the first objec | 21:15 |
| timburke | t from your previous listing | 21:15 |
| timburke | so for the longest time, pagination was one-way. you could start at the beginning, click next, click next, but never go back. only recently did we get reverse listings, which you might be interested in: https://github.com/openstack/swift/blob/2.6.0/CHANGELOG#L14-L19 | 21:17 |
| *** dmorita has joined #openstack-swift | 21:18 | |
| *** admin0 has joined #openstack-swift | 21:21 | |
| tdasilva | acoles_: https://twitter.com/EmilienMacchi/status/695194351224381440 | 21:22 |
| *** mvk has joined #openstack-swift | 21:22 | |
| torgomatic | FWIW, on a previous app I worked on, we had a large table (hundreds of millions of rows) that we'd iterate over piecewise, and we'd do it in pages of 10,000. | 21:23 |
| *** dmorita has quit IRC | 21:23 | |
| torgomatic | It ended up using limit/offset queries to do it. | 21:23 |
| torgomatic | The middle were always waaaay slower than the ends. The DB engine had to do a lot more work to get OFFSET 150000000 LIMIT 10000 than it did OFFSET 20000 LIMIT 10000. Orders of magnitude more work. | 21:24 |
| torgomatic | So, I guess what I'm saying is supporting limit/offset queries in Swift is basically like adding a DoS-the-cluster button. | 21:26 |
| timburke | good point! maybe solvable with order statistic trees, but that's really on the db engine | 21:27 |
| torgomatic | yep, and I'm not counting on sqlite to get that fancy any time soon :p | 21:28 |
| *** dmorita has joined #openstack-swift | 21:29 | |
| *** dmorita has quit IRC | 21:30 | |
| *** dmorita has joined #openstack-swift | 21:31 | |
| torgomatic | It's not generally solvable given the ability to specify arbitrary WHERE clauses, but my old app was really just doing a bunch of "SELECT * FROM big_old_table ORDER BY id LIMIT <N> OFFSET <M>" | 21:32 |
| *** baojg has quit IRC | 21:32 | |
| torgomatic | and still, the DB churned mightily serving up the stuff in the middle | 21:32 |
| torgomatic | it got better toward the end, so it was able to use the index on id (the primary key) in reverse, which was nice | 21:33 |
| *** zul has joined #openstack-swift | 21:42 | |
| *** Mingyu has quit IRC | 21:50 | |
| *** Mingyu has joined #openstack-swift | 21:50 | |
| *** lpabon has quit IRC | 21:51 | |
| *** Mingyu has quit IRC | 21:52 | |
| *** Mingyu has joined #openstack-swift | 21:52 | |
| *** trifon has quit IRC | 21:56 | |
| *** Mingyu has quit IRC | 21:59 | |
| *** Mingyu has joined #openstack-swift | 21:59 | |
| *** haomaiwang has quit IRC | 22:01 | |
| *** haomaiwa_ has joined #openstack-swift | 22:01 | |
| notmyname | good morning | 22:02 |
| *** dmorita has quit IRC | 22:05 | |
| *** dmorita has joined #openstack-swift | 22:07 | |
| *** rickyrem has joined #openstack-swift | 22:09 | |
| *** jamielennox|away is now known as jamielennox | 22:10 | |
| *** admin0 has quit IRC | 22:12 | |
| *** ajiang has left #openstack-swift | 22:12 | |
| *** petertr7 is now known as petertr7_away | 22:15 | |
| *** trifon has joined #openstack-swift | 22:26 | |
| *** arch-nemesis has quit IRC | 22:43 | |
| *** david-lyle has joined #openstack-swift | 22:48 | |
| *** gyee has joined #openstack-swift | 22:48 | |
| *** bjkeller has left #openstack-swift | 22:51 | |
| *** david-lyle has quit IRC | 22:52 | |
| *** diazjf has quit IRC | 22:57 | |
| *** haomaiwa_ has quit IRC | 23:01 | |
| *** daemontool has quit IRC | 23:01 | |
| *** haomaiwang has joined #openstack-swift | 23:01 | |
| *** chsc has quit IRC | 23:18 | |
| *** nadeem has quit IRC | 23:18 | |
| *** km has joined #openstack-swift | 23:26 | |
| *** mgarza has quit IRC | 23:27 | |
| *** kei_yama has joined #openstack-swift | 23:29 | |
| *** chlong has quit IRC | 23:30 | |
| *** jamielennox is now known as jamielennox|away | 23:31 | |
| *** mragupat_ has joined #openstack-swift | 23:45 | |
| *** mragupat has quit IRC | 23:49 | |
| *** mragupat_ has quit IRC | 23:49 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!