| *** eranrom has quit IRC | 00:02 | |
| kota_ | back to online at office | 00:03 |
|---|---|---|
| kota_ | and re-starting to add reconstructor tests and address to acoles comments | 00:04 |
| openstackgerrit | Merged openstack/swift master: IO Priority support on the AArch64 architecture https://review.openstack.org/423783 | 00:16 |
| *** stradling has joined #openstack-swift | 00:18 | |
| *** mmotiani_ has joined #openstack-swift | 00:21 | |
| *** mmotiani_ has quit IRC | 00:21 | |
| *** sams-gleb has joined #openstack-swift | 00:31 | |
| *** sams-gleb has quit IRC | 00:36 | |
| *** stradling has quit IRC | 00:38 | |
| *** mmotiani_ has joined #openstack-swift | 00:40 | |
| *** stradling has joined #openstack-swift | 00:46 | |
| *** caiobrentano has joined #openstack-swift | 00:48 | |
| *** caiobrentano has quit IRC | 00:49 | |
| notmyname | I'm working on a list of "top 3 features of ocata" right now. I'll drop something in here when I have it written up | 00:50 |
| notmyname | for comment | 00:50 |
| *** dja has quit IRC | 00:53 | |
| *** caiobrentano has joined #openstack-swift | 00:53 | |
| *** dja has joined #openstack-swift | 00:54 | |
| *** stradling has quit IRC | 01:19 | |
| *** _JZ_ has quit IRC | 01:33 | |
| *** caiobrentano_ has joined #openstack-swift | 01:47 | |
| *** winggundamth has joined #openstack-swift | 01:49 | |
| *** caiobrentano has quit IRC | 01:50 | |
| *** tanee_away is now known as tanee | 02:17 | |
| *** Jeffrey4l has quit IRC | 02:17 | |
| *** Jeffrey4l has joined #openstack-swift | 02:20 | |
| *** sams-gleb has joined #openstack-swift | 02:32 | |
| *** sams-gleb has quit IRC | 02:36 | |
| *** tongli has quit IRC | 02:50 | |
| *** caiobrentano has joined #openstack-swift | 02:51 | |
| *** caiobrentano_ has quit IRC | 02:51 | |
| *** openstackstatus has quit IRC | 02:58 | |
| *** htruta has quit IRC | 02:59 | |
| *** htruta has joined #openstack-swift | 02:59 | |
| *** mmotiani_ has quit IRC | 03:02 | |
| *** openstackstatus has joined #openstack-swift | 03:02 | |
| *** ChanServ sets mode: +v openstackstatus | 03:02 | |
| *** psachin has joined #openstack-swift | 03:03 | |
| *** mmotiani_ has joined #openstack-swift | 03:11 | |
| *** JimCheung has quit IRC | 03:11 | |
| *** JimCheung has joined #openstack-swift | 03:12 | |
| *** JimCheung has quit IRC | 03:16 | |
| *** bkopilov has quit IRC | 03:19 | |
| *** bkopilov_ has quit IRC | 03:19 | |
| *** caiobrentano has quit IRC | 03:20 | |
| sanchitmalhotra | Hi... Is anyone using ceilometermiddleware with swift here? | 03:21 |
| sanchitmalhotra | Can we specify more than one host for RabbitMQ, like we used to do it earlier using rabbit_hosts parameter? | 03:22 |
| clayg | notmyname: you should put "AArch64 support" on there | 03:26 |
| *** links has joined #openstack-swift | 03:33 | |
| *** sams-gleb has joined #openstack-swift | 03:34 | |
| jrichli | clayg: one of the troubled probetests succeeds if we simply remove the assertion here https://github.com/openstack/swift/blob/master/test/probe/test_reconstructor_revert.py#L211 | 03:38 |
| jrichli | but is there a reason that assertion is there? I dont really know why it was saying the x-backend-timestamp couldn't be present | 03:38 |
| *** sams-gleb has quit IRC | 03:39 | |
| jrichli | i looked through the changes that were involved in the merge to see if I could tell why that timestamp would now be present. it wasn't clear to me. | 03:40 |
| *** JimCheung has joined #openstack-swift | 03:43 | |
| *** JimCheung has quit IRC | 03:48 | |
| *** SkyRocknRoll has joined #openstack-swift | 03:58 | |
| *** mmotiani_ has quit IRC | 03:59 | |
| *** psachin has quit IRC | 04:00 | |
| *** mmotiani_ has joined #openstack-swift | 04:05 | |
| *** si1ver has quit IRC | 04:10 | |
| *** klrmn has quit IRC | 04:14 | |
| *** tqtran has quit IRC | 04:17 | |
| *** psachin has joined #openstack-swift | 04:24 | |
| *** sams-gleb has joined #openstack-swift | 04:37 | |
| *** sams-gleb has quit IRC | 04:42 | |
| *** bkopilov_ has joined #openstack-swift | 04:48 | |
| *** bkopilov has joined #openstack-swift | 04:49 | |
| *** dja has quit IRC | 04:52 | |
| *** mmotiani_ has quit IRC | 05:04 | |
| jrichli | clayg: in trying to understand the assertion that we *may* be able to remove now, I looked at the old patch that added it : patch 218023 | 05:07 |
| patchbot | https://review.openstack.org/#/c/218023/ - swift - Fix purge for tombstone only REVERT job (MERGED) | 05:07 |
| jrichli | and I noticed in the commit message it mentioned "... they sync tombstones with other handoffs to fill in for the primaries" | 05:09 |
| jrichli | which did sound conflicting with a recent change of patch 425441 | 05:10 |
| patchbot | https://review.openstack.org/#/c/425441/ - swift - Do not revert fragments to handoffs (MERGED) | 05:10 |
| jrichli | I was just curious about how sync'ing tombstones is related to the fragment reverts | 05:11 |
| *** mmotiani_ has joined #openstack-swift | 05:12 | |
| jrichli | well, what i mean is, it seems like in the past we depended on some reverting to 'other handoffs', but now we have removed that | 05:24 |
| *** SkyRocknRoll has quit IRC | 05:37 | |
| *** dmorita has quit IRC | 05:40 | |
| *** dmorita has joined #openstack-swift | 05:42 | |
| *** dmorita has quit IRC | 05:46 | |
| *** winggundamth has quit IRC | 06:04 | |
| *** dmorita has joined #openstack-swift | 06:05 | |
| *** dmorita has quit IRC | 06:09 | |
| *** tqtran has joined #openstack-swift | 06:15 | |
| *** tqtran has quit IRC | 06:19 | |
| *** kmARC_ has joined #openstack-swift | 06:29 | |
| *** ediardo_ has joined #openstack-swift | 06:30 | |
| *** ediardo has quit IRC | 06:33 | |
| *** kmARC has quit IRC | 06:33 | |
| *** ntata has quit IRC | 06:33 | |
| *** ediardo_ is now known as ediardo | 06:33 | |
| *** ntata has joined #openstack-swift | 06:33 | |
| *** kmARC_ is now known as kmARC | 06:33 | |
| *** sams-gleb has joined #openstack-swift | 06:38 | |
| *** SkyRocknRoll has joined #openstack-swift | 06:39 | |
| kota_ | jrichli: sounds take a progress | 06:42 |
| kota_ | jrichli: maybe as you notice, wheter the x-backend-timestmp is in or not depends on the ts is in the disk | 06:42 |
| kota_ | https://github.com/openstack/swift/blob/master/swift/obj/server.py#L922-L923 | 06:42 |
| *** sams-gleb has quit IRC | 06:43 | |
| kota_ | i don't look at the probe test yet but the expected situation would be deleting the ts but not -> causes the failure | 06:43 |
| kota_ | maybe? | 06:43 |
| kota_ | recently we have a patch to stop to revert to handoff agressively, IIRC | 06:44 |
| *** mmotiani_ has quit IRC | 06:50 | |
| *** foexle has joined #openstack-swift | 06:55 | |
| *** mmotiani_ has joined #openstack-swift | 06:58 | |
| *** rcernin has joined #openstack-swift | 07:00 | |
| *** tesseract has joined #openstack-swift | 07:11 | |
| *** ChubYann has quit IRC | 07:19 | |
| *** sams-gleb has joined #openstack-swift | 07:24 | |
| *** sams-gle_ has joined #openstack-swift | 07:25 | |
| *** sams-gleb has quit IRC | 07:25 | |
| *** foexle has quit IRC | 07:25 | |
| *** joeljwright has quit IRC | 07:27 | |
| *** ChubYann has joined #openstack-swift | 07:31 | |
| *** pcaruana has joined #openstack-swift | 07:36 | |
| *** MooingLemur has quit IRC | 07:44 | |
| *** ChubYann has quit IRC | 07:48 | |
| *** rledisez has joined #openstack-swift | 08:10 | |
| *** silor has joined #openstack-swift | 08:14 | |
| *** silor1 has joined #openstack-swift | 08:19 | |
| *** silor has quit IRC | 08:21 | |
| *** silor1 is now known as silor | 08:21 | |
| *** JimCheung has joined #openstack-swift | 08:34 | |
| *** mmotiani_ has quit IRC | 08:37 | |
| *** mmotiani_ has joined #openstack-swift | 08:38 | |
| *** geaaru has joined #openstack-swift | 08:42 | |
| *** mmotiani_ has quit IRC | 08:42 | |
| *** rcernin has quit IRC | 08:50 | |
| *** glb1 has joined #openstack-swift | 08:57 | |
| *** winggundamth has joined #openstack-swift | 08:58 | |
| *** openstackgerrit has quit IRC | 09:03 | |
| *** glb1 has quit IRC | 09:04 | |
| *** glb1 has joined #openstack-swift | 09:04 | |
| *** cbartz has joined #openstack-swift | 09:05 | |
| *** rcernin has joined #openstack-swift | 09:06 | |
| *** oshritf has joined #openstack-swift | 09:07 | |
| *** cbartz has quit IRC | 09:08 | |
| kota_ | It looks maybe I'm closing to finish to address all existing comments on ec duplication patch | 09:10 |
| kota_ | Now running tests in my local | 09:10 |
| psachin | ping #openstack-swift. I was trying 'swift tmpurl' I followed http://docs.openstack.org/developer/swift/api/temporary_url_middleware.html. But when I try to download object from tmpurl it throws '401 Unauthorized: Temp URL invalid' | 09:13 |
| *** Jeffrey4l has quit IRC | 09:17 | |
| *** Jeffrey4l has joined #openstack-swift | 09:17 | |
| *** jordanP has joined #openstack-swift | 09:21 | |
| *** reedip has joined #openstack-swift | 09:22 | |
| reedip | hi guys , I have a query | 09:23 |
| *** dja has joined #openstack-swift | 09:27 | |
| reedip | If Swift and Ceilometer are communicating, and if swift has a request, which it sends to ceilometer. But Ceilometer is not there on the network anymore therefore the Swift rquest is not forwarded to ceilometer. Would the request from Swift be hanged ? | 09:28 |
| psachin | reedip: It should timeout I guess | 09:30 |
| reedip | psachin : this is a behavior which one of our team members noticed in stable/mitaka | 09:31 |
| *** openstackgerrit has joined #openstack-swift | 09:32 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift master: EC Fragment Duplication - Foundational Global EC Cluster Support https://review.openstack.org/219165 | 09:32 |
| *** cbartz has joined #openstack-swift | 09:34 | |
| psachin | reedip: The request hangs without an error? | 09:35 |
| reedip | psachin : any idea about the timeout value? | 09:35 |
| *** JimCheung has joined #openstack-swift | 09:35 | |
| reedip | psachin: no the request hangs indefinetly, without an error, on the screen. Let me check the logs once though. | 09:35 |
| *** JimCheung has quit IRC | 09:35 | |
| *** Shashikant86 has joined #openstack-swift | 09:43 | |
| *** dja has quit IRC | 09:44 | |
| *** acoles_ is now known as acoles | 09:45 | |
| acoles | kota_: thanks for new version of global EC patch, I will do my best to review again today. | 09:45 |
| *** dja has joined #openstack-swift | 09:45 | |
| acoles | kota_: but sorry, after today I am on vacation until middle of next week. | 09:46 |
| kota_ | acoles: oh, okay. Thanks for great reviews and it's ok for your best. your eyes/comments always be helpful to me. | 09:51 |
| kota_ | acoles: and please enjoy your vacation ;-) | 09:51 |
| kota_ | acoles: are your going to somewhere to stay? | 09:52 |
| acoles | kota_: thanks! and like clayg said in meeting, we are committed to get this merged whether before or after release, thanks for all your work on it | 09:52 |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Fix misleading hash invalidations test comments https://review.openstack.org/427205 | 09:55 |
| *** csmart has joined #openstack-swift | 09:56 | |
| *** dja has quit IRC | 09:56 | |
| *** dosaboy has quit IRC | 09:56 | |
| csmart | \o | 09:57 |
| *** dja has joined #openstack-swift | 09:57 | |
| csmart | Finally found time to connect up via riot.im | 09:57 |
| *** dosaboy has joined #openstack-swift | 09:58 | |
| *** SkyRocknRoll has quit IRC | 09:59 | |
| *** dosaboy has joined #openstack-swift | 10:00 | |
| *** SkyRocknRoll has joined #openstack-swift | 10:02 | |
| *** dja has quit IRC | 10:03 | |
| *** Shashikant86 has quit IRC | 10:05 | |
| kota_ | ok, I'm going to be back home earlier than other working days in this week. but I can be responsive if my bouncer kicks me | 10:11 |
| kota_ | please feel free to comment/ping me. | 10:11 |
| *** Shashikant86 has joined #openstack-swift | 10:13 | |
| *** tqtran has joined #openstack-swift | 10:17 | |
| *** tqtran has quit IRC | 10:22 | |
| *** sams-gleb has joined #openstack-swift | 10:26 | |
| *** links has quit IRC | 10:27 | |
| *** sams-gle_ has quit IRC | 10:28 | |
| *** pcaruana has quit IRC | 10:32 | |
| *** pcaruana has joined #openstack-swift | 10:33 | |
| *** links has joined #openstack-swift | 10:40 | |
| cnf | csmart: riot is yet another slack / hipchat / ... ? | 10:50 |
| cnf | wow, what a bad website | 10:51 |
| *** sams-gleb has quit IRC | 11:02 | |
| *** sams-gleb has joined #openstack-swift | 11:02 | |
| *** SkyRocknRoll has quit IRC | 11:05 | |
| *** sams-gleb has quit IRC | 11:07 | |
| csmart | cnf: riot is an implementation of matrix.org - an open, distributed, federated (slack-like) chat protocol | 11:35 |
| *** kei_yama has quit IRC | 11:35 | |
| openstackgerrit | Joel Wright proposed openstack/swift master: Add Preamble and Postamble to SLO and SegmentedIterable https://review.openstack.org/365371 | 11:36 |
| *** caiobrentano has joined #openstack-swift | 11:39 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Fix race when consolidating new partition https://review.openstack.org/426336 | 11:47 |
| *** JimCheung has joined #openstack-swift | 11:51 | |
| *** sams-gleb has joined #openstack-swift | 11:55 | |
| *** links has quit IRC | 11:55 | |
| *** JimCheung has quit IRC | 11:55 | |
| *** cbartz has quit IRC | 11:56 | |
| *** cbartz has joined #openstack-swift | 11:58 | |
| *** oshritf_ has joined #openstack-swift | 11:59 | |
| kota_ | Gate failed!? | 12:00 |
| *** oshritf_ has quit IRC | 12:01 | |
| *** oshritf_ has joined #openstack-swift | 12:02 | |
| *** PavelK has joined #openstack-swift | 12:02 | |
| kota_ | Er, perhaps related to quorum changes | 12:03 |
| *** oshritf has quit IRC | 12:03 | |
| kota_ | No, it looks it gets on GET. | 12:04 |
| *** links has joined #openstack-swift | 12:07 | |
| *** SkyRocknRoll has joined #openstack-swift | 12:11 | |
| *** JimCheung has joined #openstack-swift | 12:11 | |
| *** psachin has quit IRC | 12:12 | |
| *** JimCheung has quit IRC | 12:16 | |
| *** psachin has joined #openstack-swift | 12:18 | |
| *** bkopilov has quit IRC | 12:24 | |
| *** bkopilov_ has quit IRC | 12:24 | |
| cnf | csmart: so yet another chat thing :P | 12:43 |
| *** links has quit IRC | 12:50 | |
| kota_ | back to online to check the gate failure | 12:54 |
| *** klamath has joined #openstack-swift | 13:01 | |
| *** klamath has quit IRC | 13:01 | |
| *** klamath has joined #openstack-swift | 13:02 | |
| *** mmotiani_ has joined #openstack-swift | 13:02 | |
| *** oshritf_ has quit IRC | 13:06 | |
| *** oshritf_ has joined #openstack-swift | 13:07 | |
| *** stradling has joined #openstack-swift | 13:09 | |
| *** psachin has quit IRC | 13:13 | |
| *** psachin has joined #openstack-swift | 13:33 | |
| *** Shashikant86 has quit IRC | 13:37 | |
| kota_ | ok, make sense | 13:38 |
| kota_ | the assertion should be wrong | 13:38 |
| kota_ | it could be less or equal to 14 in my calculation | 13:38 |
| kota_ | i think it's not related to my ec duplication, it should happen on master but it seems freaky | 13:40 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift master: EC Fragment Duplication - Foundational Global EC Cluster Support https://review.openstack.org/219165 | 13:41 |
| kota_ | my idea added to the gerrit. I think no more failure happens in the gerrit. It should be robust. | 13:52 |
| *** oshritf_ has quit IRC | 13:53 | |
| *** bkopilov_ has joined #openstack-swift | 14:01 | |
| *** bkopilov has joined #openstack-swift | 14:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 14:01 |
| *** mmotiani_ has quit IRC | 14:06 | |
| *** mmotiani_ has joined #openstack-swift | 14:17 | |
| *** dmorita has joined #openstack-swift | 14:18 | |
| *** tqtran has joined #openstack-swift | 14:19 | |
| *** mmotiani_ has quit IRC | 14:21 | |
| *** mmotiani_ has joined #openstack-swift | 14:21 | |
| jrichli | reedip: are saying that ceilometer has made a request to swift, and it disconnects before the response is returned from swift? | 14:22 |
| jrichli | reedip: if so, you may want to comment on https://bugs.launchpad.net/swift/+bug/1662159 | 14:22 |
| openstack | Launchpad bug 1662159 in OpenStack Object Storage (swift) "TCP leak between proxy and object-server on client disconnection" [Undecided,New] | 14:22 |
| *** tqtran has quit IRC | 14:23 | |
| *** dmorita has quit IRC | 14:24 | |
| *** mmotiani_ has quit IRC | 14:24 | |
| *** dmorita has joined #openstack-swift | 14:24 | |
| *** oshritf_ has joined #openstack-swift | 14:29 | |
| *** Shashikant86 has joined #openstack-swift | 14:30 | |
| jrichli | reedip: actually, I just read you comment again. I am confused because you were expecting a response, but it sounds as if ceilometer, the client, disconnected. | 14:31 |
| *** catintheroof has joined #openstack-swift | 14:31 | |
| *** ChatSharp has joined #openstack-swift | 14:40 | |
| *** ChatSharp has left #openstack-swift | 14:40 | |
| jrichli | kota_: thank you for the input! | 14:43 |
| *** JimCheung has joined #openstack-swift | 14:48 | |
| *** JimCheung has quit IRC | 14:48 | |
| *** dmorita_ has joined #openstack-swift | 15:00 | |
| *** dmorita_ has quit IRC | 15:00 | |
| *** dmorita has quit IRC | 15:04 | |
| *** ChatSharp has joined #openstack-swift | 15:12 | |
| *** ChatSharp has left #openstack-swift | 15:12 | |
| *** Shashikant86 has quit IRC | 15:12 | |
| *** sams-gleb has quit IRC | 15:13 | |
| *** sams-gleb has joined #openstack-swift | 15:14 | |
| *** sams-gleb has quit IRC | 15:18 | |
| *** _JZ_ has joined #openstack-swift | 15:24 | |
| *** sams-gleb has joined #openstack-swift | 15:35 | |
| *** Shashikant86 has joined #openstack-swift | 15:36 | |
| *** Shashikant86 has quit IRC | 15:38 | |
| *** abqkawi1000 has joined #openstack-swift | 15:59 | |
| abqkawi1000 | Quick question about *.conf, *.gz, *.builder files. Given a controller, a proxy/account/continer box and an object box: What should the contents of each /etc/swift be? Do all files need to be on all boxes? | 16:00 |
| timburke | clayg: you can't see me right now, but i'm squinting very hard at you | 16:04 |
| acoles | cschwede_: I left some comments on patch 337297, but I am now on vacation until after the release date so can't follow up - sorry! | 16:08 |
| patchbot | https://review.openstack.org/#/c/337297/ - swift - Add support to increase object ring partition power | 16:08 |
| cschwede_ | acoles: alright, just saw the mail notification - thanks a lot for reviewing it, i will follow up soon and push a new patchset! | 16:08 |
| cschwede_ | acoles: enjoy your vacation! | 16:09 |
| timburke | abqkawi1000: both boxes will need all *.ring.gz files. none of them *need* the builder files; you could actually do all of your ring-building on some separate admin box and push them out from there if you like. the conf files come down to whatever you need to have running -- so mostly the object-*.conf files on the object box, and {proxy,account,container}-*.conf on the other one. swift.conf has to be on all of them; memecache goes wi | 16:09 |
| timburke | th the proxy; others... i might need to defer to someone else on | 16:09 |
| cschwede_ | acoles: oh, will you join the Atlanta PTG? | 16:09 |
| abqkawi1000 | timburke: gotcha....makes sense. Also do I need to start any swift services on the controller? | 16:11 |
| timburke | abqkawi1000: at minimum, you'll want the proxy, account, container, and object servers running on the appropriate boxes. this allows the client-facing API to function at all. to ensure the proper health of your cluster, all the other background daemons should be running, too. so, account-auditor, account-reaper, account-replicator on any boxes running account-server; container-auditor, -reconciler, -replicator, -updater on the contain | 16:17 |
| timburke | er-server box, etc. | 16:17 |
| abqkawi1000 | timburke: Ok great, thanks for you help. | 16:18 |
| timburke | psachin: did you ever get help on your tempurl issue? i'm wondering if you may have been trying the new prefix-based tempurls, which were a rather recent addition -- not many clusters would support them yet, and i would expect the ones that don't to fail with that message | 16:19 |
| acoles | cschwede_: np, the nits really are nits, don't need to fix them, but I am curious about the strategy for knowing when relinking has been 100% successful | 16:19 |
| acoles | cschwede_: yes i'll be at PTG :) | 16:19 |
| cschwede_ | acoles: yay, best news of the day! | 16:20 |
| acoles | cschwede_: just out til middle of next week | 16:20 |
| cschwede_ | acoles: i'll need a new patchset anyways, because of the probe test issue. mattoliverau actually fixed it, it seems (thx Matt!) | 16:20 |
| acoles | nice | 16:21 |
| *** caiobrentano_ has joined #openstack-swift | 16:23 | |
| *** caiobrentano_ has quit IRC | 16:23 | |
| *** caiobrentano has quit IRC | 16:26 | |
| *** caiobrentano has joined #openstack-swift | 16:27 | |
| *** JimCheung has joined #openstack-swift | 16:31 | |
| *** klrmn has joined #openstack-swift | 16:31 | |
| *** rcernin has quit IRC | 16:32 | |
| *** JimCheung has quit IRC | 16:35 | |
| *** chsc has joined #openstack-swift | 16:37 | |
| *** chsc has joined #openstack-swift | 16:37 | |
| *** mmotiani_ has joined #openstack-swift | 16:52 | |
| *** PavelK has quit IRC | 16:53 | |
| notmyname | good morning | 16:56 |
| *** stradling has quit IRC | 16:59 | |
| *** tqtran has joined #openstack-swift | 17:01 | |
| *** dmorita has joined #openstack-swift | 17:01 | |
| *** glb has joined #openstack-swift | 17:04 | |
| *** dmorita has quit IRC | 17:05 | |
| *** catintheroof has quit IRC | 17:06 | |
| *** catintheroof has joined #openstack-swift | 17:07 | |
| *** catintheroof has quit IRC | 17:11 | |
| *** tesseract has quit IRC | 17:18 | |
| *** jordanP has quit IRC | 17:30 | |
| *** pcaruana has quit IRC | 17:30 | |
| *** geaaru has quit IRC | 17:31 | |
| *** joeljwright has joined #openstack-swift | 17:32 | |
| *** ChanServ sets mode: +v joeljwright | 17:32 | |
| *** caiobrentano has joined #openstack-swift | 17:35 | |
| *** silor has quit IRC | 17:36 | |
| clayg | kota_: patchset 55 looks good at first glance! I may get to spend more time with it this afternoon, or tomorrow | 17:43 |
| *** Administrator__ has joined #openstack-swift | 17:43 | |
| clayg | jrichli: sounds good to me - is there a fix for the probetest up? | 17:44 |
| * clayg hopes timburke is squinting because of my crack about AArch64 <giggles> | 17:46 | |
| *** zhugaoxiao has quit IRC | 17:46 | |
| jrichli | clayg: after learning some about this, I am not so sure we can simply remove that assertion. | 17:48 |
| jrichli | clayg: it seems like that is the way we determine if there was a tombstone or not: timestamp is added if there was a tombstone | 17:48 |
| clayg | jrichli: boooooo | 17:48 |
| jrichli | clayg: and in the test, it says that tombstone should have been reverted back to primary. this shows that is not happening. | 17:49 |
| clayg | sure that's true it's like 404 "with feeling" | 17:49 |
| *** mmotiani_ has quit IRC | 17:49 | |
| jrichli | clayg: but you think it's ok that this tombstone remains on the handoff even though the primary is back? | 17:49 |
| clayg | zohno! we can't just be leaving tombstones around on the floor!? is everything broke!N???!!!!11 | 17:49 |
| clayg | no, i mean if the primary is up i don't see why the tombstone wouldn't make it's way out to the primary | 17:50 |
| jrichli | clayg: I would need somebody to verify my understanding ... I am like just now parsing the code | 17:50 |
| clayg | jrichli: i think if there is *another* frag in the partition - then the sync job would just point to the primary for that index - if it's up - it should get the tombstone | 17:50 |
| clayg | jrichli: if there are *no* frags in the partition the handoff doesn't know *where* to send the tombstones in the part - so I think it sends it to all primaries | 17:51 |
| clayg | how did we get a tombstone on a handoff anyway? | 17:51 |
| clayg | jrichli: so probetests don't reset your environment when they fail | 17:52 |
| clayg | jrichli: after it stops - go poke around on disk | 17:52 |
| clayg | start up your object servers | 17:52 |
| clayg | run the reconstrucotor like `swift-init object-reconstructor once --no-daemon --verbose --config X' where X is a node#/config# | 17:52 |
| *** mmotiani_ has joined #openstack-swift | 17:53 | |
| clayg | or `swift-init object-reconstrucotr once -nv -c X` | 17:53 |
| clayg | then see if you think the damn thing is doing what it is supposed to | 17:53 |
| clayg | i like `find /srv/node*/sdb*/object* -name \*.data` to see wtf is going on | 17:53 |
| jrichli | clayg: how did we get a tombstone on handoff: my understanding is that the primary is taken down, then delete is requested. | 17:53 |
| clayg | or... \*.ts | 17:53 |
| clayg | delete doens't really do the connects the same way PUT does - the tombstone should just not get written to any downed primaries | 17:54 |
| jrichli | clayg: i have looked at disk afterwards. when the case succeeds, there is one less tombstone on disk than when the test fails. | 17:54 |
| clayg | as long as enought (k + 1) response so that a read shouldn't work - i think we 2XX | 17:54 |
| jrichli | clayg: but then how is the delete recorded? | 17:55 |
| clayg | this is EC right? am I crazy? | 17:55 |
| jrichli | oh, right | 17:55 |
| *** dmorita has joined #openstack-swift | 17:55 | |
| clayg | no - don't let me confuse you - i might be crazy! | 17:55 |
| *** dmorita has quit IRC | 17:56 | |
| jrichli | I think I gotta take a break and eat lunch! my brain needs sugar :-) i'll keep looking. | 17:56 |
| *** oshritf_ has quit IRC | 17:56 | |
| *** caiobrentano has quit IRC | 18:02 | |
| *** rledisez has quit IRC | 18:04 | |
| *** winggundamth has quit IRC | 18:04 | |
| *** jordanP has joined #openstack-swift | 18:11 | |
| *** jordanP has quit IRC | 18:12 | |
| *** zhugaoxiao has joined #openstack-swift | 18:15 | |
| *** Administrator__ has quit IRC | 18:18 | |
| notmyname | I'm struggling with the "marketing phrasing" of new features in swift. I don't feel like "things were pretty terrible and sometimes surprising it worked at all, but now things are probably a little bit less terrible" is a particularly good thing to say to press | 18:18 |
| *** SkyRocknRoll has quit IRC | 18:23 | |
| notmyname | when I write about closing bugs that corrupted data, I feel like it's similar to saying, "last week was great! I had a 37% lower chance of kicking puppies" (ok, that's great, but how often were you kicking puppies before??!) | 18:24 |
| *** joeljwright has quit IRC | 18:26 | |
| *** catintheroof has joined #openstack-swift | 18:26 | |
| *** dmorita has joined #openstack-swift | 18:26 | |
| *** dmorita has quit IRC | 18:28 | |
| *** dmorita has joined #openstack-swift | 18:28 | |
| *** psachin has quit IRC | 18:30 | |
| *** ChubYann has joined #openstack-swift | 18:40 | |
| *** cbartz has quit IRC | 18:54 | |
| *** cbartz has joined #openstack-swift | 18:54 | |
| *** cbartz has quit IRC | 18:54 | |
| *** caiobrentano has joined #openstack-swift | 19:06 | |
| * acoles away til 15th | 19:19 | |
| *** acoles is now known as acoles_ | 19:21 | |
| *** caiobrentano has quit IRC | 19:26 | |
| *** caiobrentano has joined #openstack-swift | 19:27 | |
| notmyname | this is an amazing email, and something the entire openstack community needs to consider http://lists.openstack.org/pipermail/openstack-dev/2017-February/111894.html | 19:30 |
| *** wer has quit IRC | 19:34 | |
| *** wer has joined #openstack-swift | 19:35 | |
| *** JimCheung has joined #openstack-swift | 19:40 | |
| *** tonanhngo has joined #openstack-swift | 19:43 | |
| *** dja has joined #openstack-swift | 19:48 | |
| openstackgerrit | Merged openstack/swift master: Fix misleading hash invalidations test comments https://review.openstack.org/427205 | 19:53 |
| *** JimCheung has quit IRC | 19:54 | |
| *** JimCheung has joined #openstack-swift | 19:55 | |
| notmyname | ok, I submitted the "top features and upcoming goals" survey for swift | 19:56 |
| notmyname | while I consider many of the internal improvements to be the most significant, much of what is asked for has to do with end-user impact | 19:56 |
| timburke | end user impact: your data are safe | 19:57 |
| *** Jeffrey4l has quit IRC | 19:57 | |
| notmyname | so what I put down is (1) "significant improvements to durability and correctness of data" (includes stuff like the repl/recon improvements, elimination if .durable, etc) (2) tempurl common prefix and (3) last-modified on container listings | 19:58 |
| *** Jeffrey4l has joined #openstack-swift | 19:58 | |
| notmyname | I wanted to put down the part-power increase, but I'm not 100% sure it will land (I never am until it actually *does* land), so I didn't mention it | 19:58 |
| notmyname | as part of this, and as normal, I was asked to identify the "themes" that are in ocata and what we'll be focusing on in pike and queens. | 19:59 |
| clayg | heyoh! | 19:59 |
| notmyname | and as normal, i put down that we focus on reliability, scalability, and manageability. and I de-emphasized interoperability, security (at least until we do more with crypto), and user experience | 20:00 |
| notmyname | (those are the 6 themes I can choose from. I can't say my own) | 20:00 |
| timburke | notmyname: slo concurrency? your PUTs are faster! | 20:00 |
| clayg | what? | 20:01 |
| notmyname | timburke: indeed. yet another that I like | 20:01 |
| timburke | oh, and hey! you can do multiple ranges now! | 20:01 |
| timburke | slos are great | 20:01 |
| clayg | queens :P | 20:01 |
| notmyname | timburke: but "things you expected to work well, now work well-er" is harder to explain to a non-tech audience than "here's a new shiny" ;-) | 20:01 |
| notmyname | timburke: I made a call :-) | 20:01 |
| *** JimCheung has quit IRC | 20:01 | |
| tdasilva | notmyname: that is a really nice email, especially the part about help. I think that can apply to all openstack projects | 20:10 |
| notmyname | tdasilva: I wonder how many wrong expectations we've set up in openstack. openstack started as this commercial joint venture thing, so projects are sortof measured (and live and die) by how much corporate patrons contribute to them | 20:12 |
| notmyname | tdasilva: which means it would be very very hard for a smaller project that is awesomely useful but doesn't have paid employees working on it to be considered successful (much less actually successful by the metrics the TC puts out) | 20:13 |
| notmyname | like, that doesn't seem "normal" in the broader open-source world | 20:13 |
| *** tonanhngo has quit IRC | 20:17 | |
| tdasilva | notmyname: OTOH, I can also understand how it is also a challenge. Since the big-tent, there are now 100s of projects in openstack and it is natural that not all of them "survive". So how does one differentiate a small but useful project, from one that is small and not very useful project | 20:17 |
| *** tonanhngo has joined #openstack-swift | 20:20 | |
| notmyname | that only matters if "openstack" is this one big thing that's gotta be deployed all together. in that case, keeping the "bad" projects in has significant negative cost | 20:20 |
| notmyname | however if openstack provides components for cloud infrastructure that don't need to be co-installed or can be consumed a-la-cart, then the cost is much less | 20:21 |
| -openstackstatus- NOTICE: Restarting gerrit due to performance problems | 20:21 | |
| *** tonanhngo_ has joined #openstack-swift | 20:21 | |
| notmyname | so if there's only a couple of people contributing to designate, but it's a stable project providing good utility, then that's great! keep it in! (and of course hope that as more people use it, more hands come on to help carry the load) | 20:22 |
| notmyname | but if there's only a couple of people contributing to designate and the project navigator (ie foundation marketing) therefore says "not mature" or it becomes impossible to meet TC "community goals", then that's bad IMO | 20:23 |
| *** tonanhngo has quit IRC | 20:24 | |
| *** tonanhngo_ has quit IRC | 20:26 | |
| notmyname | /lunchtime | 20:30 |
| *** tonanhngo has joined #openstack-swift | 20:35 | |
| *** MooingLemur has joined #openstack-swift | 20:37 | |
| csmart | /breakfast time | 20:38 |
| *** JimCheung has joined #openstack-swift | 20:40 | |
| *** JimCheung has quit IRC | 20:45 | |
| *** silor has joined #openstack-swift | 20:47 | |
| tdasilva | jrichli, clayg: i'm having to go back almost 30 days to see probe tests working :( | 20:48 |
| notmyname | tdasilva: in the qa cluster? | 20:48 |
| tdasilva | no, in my dev. environment | 20:49 |
| notmyname | oh ok | 20:49 |
| notmyname | still bad, just bad in a different way | 20:49 |
| jrichli | tdasilva: I have been focused on only one of the tests: test_delete_propagate. | 20:50 |
| jrichli | test_delete_propagate was introduced from fc09dda Merge "Better optimistic lock in get_hashes" | 20:50 |
| jrichli | is that what you are seeing for that one as well? | 20:50 |
| tdasilva | jrichli: i saw this fail: test_main (test.probe.test_replication_servers_working.TestReplicatorFunctions in b5530f4620a0a9ba6b9011c68a93b6e1ef928233 | 20:52 |
| tdasilva | jrichli: so I took a big jump back to 0590d20fccd0ca2202c6246eb6b213115427f237 and tests pass, so now i'm going to work my way back | 20:53 |
| jrichli | tdasilva: oh, ok. nice work! keep me posted. i am still focused on understanding/fixing test_delete_propagate :/ | 20:56 |
| clayg | jrichli: did you fix everything!? | 20:56 |
| jrichli | clayg: lol. I am glad you seem to have hope for me yet :-) | 20:57 |
| jrichli | I had a long lunch at the pancake house. is that a good excuse? | 20:57 |
| *** dja has quit IRC | 21:01 | |
| tdasilva | jrichli: the "funny" part is that when i checkout the "Better optimistic lock in get_hashes" commit, i saw 4 tests fail :/ | 21:03 |
| *** silor1 has joined #openstack-swift | 21:04 | |
| *** silor has quit IRC | 21:04 | |
| *** silor1 is now known as silor | 21:04 | |
| jrichli | tdasilva: the initial commit aa71d7e is different then the merged one at fc09dda. which one are you speaking of? | 21:06 |
| jrichli | it is the diff between those two that I am currently focused on. | 21:06 |
| jrichli | if i understand right, the patch as it was tested before it merged passed - at least this one test. but then when it was merged with what had transpired on master, that is when things didnt work together. | 21:07 |
| tdasilva | jrichli: yep, I tested fc09dda | 21:08 |
| jrichli | i guess that is why you cant just test a patch with git -d review <>. you have to rebase, then test. | 21:08 |
| tdasilva | yep | 21:09 |
| *** caiobrentano has quit IRC | 21:21 | |
| *** caiobrentano has joined #openstack-swift | 21:22 | |
| *** JimCheung has joined #openstack-swift | 21:26 | |
| *** caiobrentano has quit IRC | 21:27 | |
| *** JimCheung has quit IRC | 21:32 | |
| *** JimCheung has joined #openstack-swift | 21:33 | |
| *** Jeffrey4l_ has joined #openstack-swift | 21:35 | |
| *** Jeffrey4l has quit IRC | 21:35 | |
| *** JimCheung has quit IRC | 21:38 | |
| mattoliverau | Morning | 21:47 |
| clayg | mattoliverau: *incorrect* ;) | 22:00 |
| clayg | mattoliverau: Afternoon | 22:00 |
| notmyname | first line of an email I just got: "How would you like a free consult for an oil change for your IT systems?" | 22:08 |
| notmyname | I'll admit that I haven't changed my oil in *any* of my servers yet | 22:08 |
| *** silor has quit IRC | 22:08 | |
| mattoliverau | Lol | 22:09 |
| *** mmotiani_ has quit IRC | 22:13 | |
| *** sams-gleb has quit IRC | 22:21 | |
| *** sams-gleb has joined #openstack-swift | 22:22 | |
| *** sams-gleb has quit IRC | 22:26 | |
| *** abqkawi1000 has quit IRC | 22:34 | |
| *** dja has joined #openstack-swift | 22:39 | |
| jrichli | clayg: I think that for test_delete_propagate, its revealing that an extra tombstone file can be left on a handoff because delete_reverted_objs is not getting called when it should for a ts. | 22:41 |
| clayg | *weird* | 22:46 |
| clayg | so like you can watch that test put your saio in a weird state - then no matter how much you run your daemons it doens't get right? | 22:46 |
| clayg | like a for-realzy bad!? | 22:46 |
| *** mmotiani_ has joined #openstack-swift | 22:48 | |
| jrichli | ah: so the following condition will NEVER be true given the code: https://github.com/openstack/swift/blob/master/swift/obj/reconstructor.py#L656 | 22:59 |
| jrichli | syncd_with is only incremented in a loop over sync_to nodes now. the looping used to be over something different | 23:00 |
| jrichli | used to be "for node in dest_nodes" | 23:01 |
| timburke | jrichli: yeah, it'd chain that with the handoff iter | 23:01 |
| jrichli | clayg: i will run reconstructor | 23:01 |
| timburke | clayg: i guess https://review.openstack.org/#/c/425441/ should have taken out that test? | 23:02 |
| patchbot | patch 425441 - swift - Do not revert fragments to handoffs (MERGED) | 23:02 |
| timburke | or at least fixed it up :-( sorry i didn't catch that earlier | 23:02 |
| jrichli | i got to learn a lot from investigation :-) | 23:03 |
| clayg | timburke: jrichli: yeah I assumed something like that but I hadn't looked at the test yet (waiting to see what jrichli comes up with!) | 23:05 |
| clayg | oh... so it *is* a bug? | 23:05 |
| *** gabor_antal has quit IRC | 23:05 | |
| clayg | why wouldn't synced_with be == to len(job['sync_to']) | 23:06 |
| clayg | timburke: taken out the "test" or taken out the "conditional" | 23:06 |
| *** gabor_antal has joined #openstack-swift | 23:07 | |
| jrichli | clayg: oh, right. it could hit when = to. i am not saying it right. we miss 2 loops here. i gotta think more to explain that. | 23:08 |
| *** cebreidian has joined #openstack-swift | 23:10 | |
| timburke | in the test, we shut down two primaries, repair the first, run reconstructor. so there's still one down primary that we can't sync with -- previously we would send it to another handoff, but now it sits tight | 23:13 |
| *** chsc has quit IRC | 23:13 | |
| clayg | timburke: yeah - by design - stupid probetest needs updating - seems stright forward | 23:13 |
| *** mmotiani_ has quit IRC | 23:14 | |
| clayg | assert it *is* sitting tight - then fix the downed primary and assert the revert | 23:14 |
| clayg | sounds like a baller test to me | 23:14 |
| *** gabor_antal has quit IRC | 23:15 | |
| *** gabor_antal has joined #openstack-swift | 23:17 | |
| jrichli | yes, so the primaries being down is why we dont get == to | 23:19 |
| jrichli | i mean the one primary | 23:20 |
| jrichli | clayg: ok, i will work on that patch if its ok i might take longer than others | 23:21 |
| timburke | do it! it'll be great :-) | 23:24 |
| clayg | *so great* | 23:25 |
| *** nikivi has quit IRC | 23:32 | |
| *** nikivi has joined #openstack-swift | 23:34 | |
| jrichli | well, i guess that is just switching the assert, right? we arleady assert the revert in that test after primary is restored | 23:38 |
| *** mmotiani_ has joined #openstack-swift | 23:39 | |
| *** mmotiani_ has quit IRC | 23:39 | |
| *** mmotiani_ has joined #openstack-swift | 23:40 | |
| *** mmotiani_ has quit IRC | 23:42 | |
| *** mmotiani_ has joined #openstack-swift | 23:42 | |
| timburke | jrichli: yeah -- might be worth running reconstructor on that "second" handoff again and seeing that the tombstone goes away? i could go either way; we already watch that happen on the "first" handoff | 23:42 |
| jrichli | timburke, clayg: but what about the check that has >= when only == could happen? should that not be changed somehow? | 23:43 |
| timburke | i'm partial toward leaving it as >= -- seems less failure prone (a la https://review.openstack.org/#/c/399237/) if we go back and change our minds | 23:44 |
| patchbot | patch 399237 - swift - Avoid infinite loop while placing parts (MERGED) | 23:44 |
| jrichli | ok, i think i still have to stew over that part. but i will probably upload a patch before tomorrow. gonna break now though. | 23:50 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!