| *** catintheroof has joined #openstack-swift | 00:10 | |
| *** catintheroof has quit IRC | 00:11 | |
| -openstackstatus- NOTICE: Due to a problem with the cinder volume backing the log server, jobs are failing with POST_FAILURE. Please avoid issuing 'recheck' commands until the issue is resolved. | 00:13 | |
| *** ChanServ changes topic to "Due to a problem with the cinder volume backing the log server, jobs are failing with POST_FAILURE. Please avoid issuing 'recheck' commands until the issue is resolved." | 00:13 | |
| openstackgerrit | Merged openstack/liberasurecode: Fix reconstruct to return an error when memory allocation failed https://review.openstack.org/385240 | 00:18 |
|---|---|---|
| *** arch-nemesis has quit IRC | 00:31 | |
| *** arch-nemesis has joined #openstack-swift | 00:34 | |
| openstackgerrit | Merged openstack/swift: go: Update README.md to clarify policies allowed https://review.openstack.org/400286 | 00:42 |
| *** dmorita has quit IRC | 00:50 | |
| *** dmorita has joined #openstack-swift | 00:52 | |
| *** dmorita has quit IRC | 00:55 | |
| *** dmorita has joined #openstack-swift | 00:55 | |
| *** tqtran has quit IRC | 01:03 | |
| *** siva_krish has quit IRC | 01:06 | |
| *** diogogmt has quit IRC | 01:09 | |
| *** clu_ has quit IRC | 01:11 | |
| *** vint_bra has joined #openstack-swift | 01:23 | |
| *** nikivi has quit IRC | 01:31 | |
| *** arch-nemesis has quit IRC | 01:35 | |
| *** vint_bra has quit IRC | 01:38 | |
| *** vint_bra has joined #openstack-swift | 01:43 | |
| *** catintheroof has joined #openstack-swift | 01:50 | |
| timburke | mattoliverau: on the py3 thing, i was using ascii as a nothing-up-my-sleeves kind of a move -- it should only ever be used when dealing with data that should be a very limited set of chars, like numeric (for the policy index header or the expiration time in tempurl's hmac) or hex (i think that was generally for etag-related things?) | 01:52 |
| *** catintheroof has quit IRC | 01:55 | |
| *** catintheroof has joined #openstack-swift | 01:55 | |
| timburke | the latin1/utf8 stuff is a mess, though; pep-3333 seems to say env vars are native strings, and if that means unicode, then they have to be encodable to latin1. which means that when we finally have some py3 code speaking http, it'll be with some crazy "wsgi string" (which i have distinct in my head from a byte string, unicode string, or even native string) getting passed around as headers | 01:55 |
| timburke | (fwiw, "wsgi string" in my head means "native (byte) string for py2, or latin1-decoded utf8 string for py3") | 01:57 |
| *** catinthe_ has joined #openstack-swift | 01:58 | |
| timburke | it's gonna be a mess, and it's gonna touch most things. so may as well start with something vaguely small-ish, like making ring building work. but that dragged in utils... so here we are. | 01:59 |
| *** catintheroof has quit IRC | 02:00 | |
| *** vint_bra has quit IRC | 02:07 | |
| mattoliverau | timburke: ahh ok, makes more sense, yeah you are only touching a few things with ascii, so that makes sense. Yeah this if going to touch everything and could look a little messy, which is why a this is the plan/how it works docs might be in order so we're all on the same page when dev or reviewing :) | 02:13 |
| mattoliverau | Thanks for all the hard work here! | 02:13 |
| *** diogogmt has joined #openstack-swift | 02:15 | |
| timburke | mattoliverau: yeah, definitely. it's not a terribly high priority, though, so i'm not entirely sure when i'll get to it ;-) | 02:16 |
| mattoliverau | ;) | 02:19 |
| timburke | but on the plus side, that patch triples the number test files for the py3 jobs! (if you count the empty test/unit/common/ring/__init__.py) | 02:23 |
| timburke | bug 1644387 is unfortunate, though | 02:24 |
| openstack | bug 1644387 in OpenStack Object Storage (swift) "Ring builder files need a consistent format" [Undecided,New] https://launchpad.net/bugs/1644387 | 02:24 |
| mattoliverau | I know I count it.. we should add all the empty __init__.py's! | 02:38 |
| mattoliverau | yeah | 02:38 |
| openstackgerrit | chen.xing proposed openstack/swift: Update a more simple rabbitmq configuration https://review.openstack.org/392030 | 02:57 |
| openstackgerrit | chen.xing proposed openstack/swift: Make rabbitmq configuration much simpler https://review.openstack.org/392030 | 03:00 |
| *** catinthe_ has quit IRC | 03:14 | |
| *** _JZ_ has joined #openstack-swift | 03:32 | |
| *** EmilienM has quit IRC | 03:50 | |
| *** EmilienM has joined #openstack-swift | 03:53 | |
| *** psachin has joined #openstack-swift | 04:03 | |
| *** dmorita has quit IRC | 04:28 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:31 | |
| *** links has joined #openstack-swift | 04:45 | |
| *** smaktub has joined #openstack-swift | 05:14 | |
| smaktub | Hi! | 05:15 |
| smaktub | We have been previously using Icehouse setup of SWIFT and CEILOMETER. | 05:15 |
| smaktub | In Icehouse ceilometer, for incoming and outgoing bytes, Sample class structure with following fields was published onto ceilometer: name, type, unit, volume, user_id, project_id, resource_id, timestamp and resource_metadata. | 05:15 |
| smaktub | Is this structure changed in Mitaka version of ceilometer? | 05:15 |
| smaktub | I could see only the result(volume) and Metric being raised as an event and notified to ceilometer? | 05:15 |
| *** dmorita has joined #openstack-swift | 05:18 | |
| *** ppai has joined #openstack-swift | 05:23 | |
| *** dmorita has quit IRC | 05:23 | |
| *** ChanServ changes topic to "Let's talk, we're nice. | Ideas: https://wiki.openstack.org/wiki/Swift/ideas | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/ | Meetings: https://wiki.openstack.org/wiki/Meetings/Swift | Priority Reviews: https://wiki.openstack.org/wiki/Swift/PriorityReviews" | 05:39 | |
| -openstackstatus- NOTICE: The affected filesystems on the log server are repaired. Please leave 'recheck' comments on any changes which failed with POST_FAILURE. | 05:39 | |
| *** sams-gleb has joined #openstack-swift | 06:24 | |
| *** ChubYann has quit IRC | 06:34 | |
| *** tqtran has joined #openstack-swift | 07:01 | |
| *** tqtran has quit IRC | 07:05 | |
| *** tesseract has joined #openstack-swift | 07:18 | |
| *** tesseract is now known as Guest13491 | 07:19 | |
| *** rcernin has joined #openstack-swift | 07:21 | |
| *** hseipp has joined #openstack-swift | 07:38 | |
| openstackgerrit | Merged openstack/swift: Handle deleted files with if-none-match https://review.openstack.org/395582 | 08:11 |
| *** smaktub has quit IRC | 08:32 | |
| donagh | smaktub: Swift does not send notifications to Ceilimeter. Instead Ceilimiter polls Swift to get account (and if you really must) container size data. So I suggest you ask in ceilometer room if there have been any changes. | 08:42 |
| *** amoralej|off is now known as amoralej | 08:44 | |
| donagh | smaktub: Are you sure the fields you mention are related to Swift? One of the fields is "volume" --- that sounds more like Cinder than Swift. | 08:46 |
| *** rledisez has joined #openstack-swift | 08:48 | |
| *** geaaru has joined #openstack-swift | 09:01 | |
| *** cbartz has joined #openstack-swift | 09:14 | |
| *** asettle has joined #openstack-swift | 09:24 | |
| *** asettle has quit IRC | 09:26 | |
| *** asettle has joined #openstack-swift | 09:27 | |
| *** admin6 has joined #openstack-swift | 09:29 | |
| *** _JZ__ has joined #openstack-swift | 09:34 | |
| *** bkopilov has quit IRC | 09:37 | |
| *** _JZ_ has quit IRC | 09:37 | |
| *** acoles_ is now known as acoles | 09:38 | |
| *** abalfour has quit IRC | 09:53 | |
| *** abalfour has joined #openstack-swift | 09:54 | |
| *** sams-gleb has quit IRC | 10:14 | |
| *** sams-gleb has joined #openstack-swift | 10:15 | |
| *** kei_yama has quit IRC | 10:15 | |
| *** sams-gleb has quit IRC | 10:19 | |
| *** 17SAAM6B5 has joined #openstack-swift | 10:47 | |
| *** sams-gleb has joined #openstack-swift | 10:52 | |
| *** Jeffrey4l has quit IRC | 10:52 | |
| *** SkyRocknRoll has quit IRC | 10:53 | |
| *** Jeffrey4l has joined #openstack-swift | 11:04 | |
| *** Jeffrey4l has quit IRC | 11:05 | |
| *** Jeffrey4l has joined #openstack-swift | 11:06 | |
| *** Jeffrey4l has quit IRC | 11:07 | |
| *** Jeffrey4l has joined #openstack-swift | 11:07 | |
| *** laci has joined #openstack-swift | 11:24 | |
| laci | hi | 11:24 |
| laci | acoles, I managed my proxy<->keystone problem, thank you for the help | 11:25 |
| laci | acoles, I would like to use my swift as a s3 provider, how can I acquire access_key ans secret_key | 11:27 |
| *** dmorita has joined #openstack-swift | 11:29 | |
| onovy | cschwede: yep, as i sad: don't lost your data guys :) | 11:30 |
| *** geaaru has quit IRC | 11:31 | |
| cschwede | onovy: yes; i’m still looking into this, it seems to get even more worse - will update the burgreport soon | 11:31 |
| acoles | laci: glad you got keystone working. I'm not familiar with swift3 | 11:32 |
| onovy | i think every 10th run: https://github.com/openstack/swift/blob/stable/mitaka/swift/obj/replicator.py#L409 are synced all suffixies | 11:32 |
| onovy | laci: hi, try #swift3 channel, there are upstream authors of swift s3 plugin | 11:33 |
| laci | acoles, Thank you! | 11:33 |
| *** dmorita has quit IRC | 11:33 | |
| *** 17SAAM6B5 has quit IRC | 11:35 | |
| *** links has quit IRC | 11:39 | |
| psachin | ping #openstack-swift, can any please review https://review.openstack.org/#/c/349199/ ? | 11:41 |
| patchbot | patch 349199 - swift - Fix swift-get-nodes arg parsing for missing ring | 11:41 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Add pip and virtualenv upgrade to development guidelines https://review.openstack.org/401986 | 11:44 |
| *** geaaru has joined #openstack-swift | 11:45 | |
| onovy | cschwede: do you want any help from us? | 11:47 |
| cschwede | onovy: not yet, thx! | 11:48 |
| onovy | ok | 11:48 |
| cschwede | onovy: but it would be great if you could have a look later, if i’m right it’s even worse… | 11:48 |
| onovy | data lost already? | 11:48 |
| onovy | we didn't have disk/servers failure sync upgrade to mitaka... | 11:49 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Drop "--allow-insecure netifaces" from tox pip install https://review.openstack.org/401994 | 11:53 |
| *** links has joined #openstack-swift | 11:54 | |
| *** vint_bra has joined #openstack-swift | 12:00 | |
| *** davideagle_ has joined #openstack-swift | 12:01 | |
| *** mvk has quit IRC | 12:15 | |
| cschwede | acoles: Hi! If you have some time, please have a log at the following bug: https://bugs.launchpad.net/swift/+bug/1634967 | 12:17 |
| openstack | Launchpad bug 1634967 in OpenStack Object Storage (swift) "object-replicator in 2.7.0 and later doesn't replicate all suffixes" [Critical,In progress] | 12:17 |
| cschwede | acoles: this seems critical to me | 12:17 |
| *** catintheroof has joined #openstack-swift | 12:17 | |
| acoles | cschwede: ack. I was just reading the bug report | 12:20 |
| cschwede | thx | 12:21 |
| onovy | cschwede: is Pavel patch fixies it? according to 'redo.sh' | 12:23 |
| cschwede | onovy: yes, it fixes that for new suffixes, but not for missing suffixes which could take a long time | 12:23 |
| onovy | so applying patch + rm *.pkl will fix it :) | 12:24 |
| cschwede | onovy: sure, that’s the quickest way to fix this :) | 12:24 |
| cschwede | or maybe based on time? if hashes.pkl wasn’t changed for X hours/days, re-list the dir | 12:25 |
| acoles | cschwede: I can't look immediately, will try to look later today | 12:26 |
| onovy | cschwede: i think we could split it into two parts | 12:26 |
| onovy | 1. fix it for new upgrades | 12:26 |
| onovy | 2. found solution for already upgraded | 12:26 |
| *** catintheroof has quit IRC | 12:26 | |
| onovy | 1b. found workaround for alread upgraded (rm *.pkl) | 12:26 |
| cschwede | acoles: alright, thx a lot! i’ll be around today for some time, ping me if you want to discuss later on | 12:26 |
| cschwede | onovy: sure; i tend to +2 the patch for now, and then we’ll add some follow ups | 12:27 |
| *** amoralej is now known as amoralej|lunch | 12:29 | |
| *** catintheroof has joined #openstack-swift | 12:29 | |
| *** catintheroof has quit IRC | 12:31 | |
| *** tqtran has joined #openstack-swift | 13:04 | |
| *** geaaru has quit IRC | 13:05 | |
| *** oshritf has quit IRC | 13:08 | |
| *** tqtran has quit IRC | 13:09 | |
| *** links has quit IRC | 13:14 | |
| *** mvk has joined #openstack-swift | 13:17 | |
| *** geaaru has joined #openstack-swift | 13:22 | |
| openstackgerrit | Pavel Kvasnička proposed openstack/swift: WIP: optimize hash calculation when suffix hash invalidated https://review.openstack.org/402043 | 13:30 |
| openstackgerrit | Mathias Bjoerkqvist proposed openstack/swift: Storing encryption root secret in Barbican https://review.openstack.org/364878 | 13:38 |
| -openstackstatus- NOTICE: OpenStack CI has taken a Thanksgiving break; no new jobs are currently launched. We're currently hoping for a friendly admin to come out of Thanksgiving and fix the system. | 13:39 | |
| *** ppai has quit IRC | 13:55 | |
| davideagle_ | join /#openstack-swift | 13:56 |
| patchbot | Error: You don't have the admin capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified. | 13:56 |
| onovy | cschwede: are you going to do follow up for (2), or should we work on it? | 13:57 |
| *** amoralej|lunch is now known as amoralej | 14:00 | |
| *** oshritf has joined #openstack-swift | 14:03 | |
| *** laci has quit IRC | 14:06 | |
| davideagle_ | has anyone worked with swift3 (amazon s3 middleware) and tempauth, the proxy responds with a 403 Access denied error when I try to create a bucket, "s3curl --id credentials --createBucket --debug -- http://localhost:8080/bucket1" | 14:07 |
| *** Jeffrey4l has quit IRC | 14:13 | |
| *** Jeffrey4l has joined #openstack-swift | 14:17 | |
| *** Jeffrey4l has quit IRC | 14:18 | |
| davideagle_ | proxy-server.conf http://paste.openstack.org/show/590316/ and .s3curl file http://paste.openstack.org/show/590317/ | 14:18 |
| *** Jeffrey4l has joined #openstack-swift | 14:18 | |
| *** silor has joined #openstack-swift | 14:21 | |
| onovy | davideagle_: hi, try #swift3 channel :) | 14:23 |
| davideagle_ | thanks :) | 14:31 |
| *** silor1 has joined #openstack-swift | 14:31 | |
| cschwede | onovy: i’ll propose a patch for an enforced do_listdir if hashes.pkl has not been updated within X days | 14:31 |
| onovy | cschwede: cool! Pavel had same suggestion. "X days" should be lower than reclaim_age | 14:32 |
| onovy | ideally < 1/2 of reclaim_age | 14:33 |
| *** silor has quit IRC | 14:33 | |
| *** silor1 is now known as silor | 14:33 | |
| *** mvk has quit IRC | 14:39 | |
| -openstackstatus- NOTICE: OpenStack CI is processing jobs again. Thanks to the Canadian admin "team" that had their Thanksgiving holiday already ;) Jobs are all enqueued, no need to recheck. | 14:48 | |
| *** mvk has joined #openstack-swift | 14:53 | |
| *** oshritf has quit IRC | 14:58 | |
| *** oshritf has joined #openstack-swift | 14:59 | |
| *** oshritf has quit IRC | 15:01 | |
| *** oshritf has joined #openstack-swift | 15:02 | |
| *** openstackgerrit has quit IRC | 15:03 | |
| *** openstackgerrit has joined #openstack-swift | 15:04 | |
| *** _JZ__ has quit IRC | 15:05 | |
| *** sams-gleb has quit IRC | 15:05 | |
| *** sams-gleb has joined #openstack-swift | 15:06 | |
| *** oshritf has quit IRC | 15:07 | |
| *** oshritf has joined #openstack-swift | 15:08 | |
| *** PetrKubena has joined #openstack-swift | 15:09 | |
| *** sams-gleb has quit IRC | 15:11 | |
| *** PetrKubena has quit IRC | 15:12 | |
| *** oshritf has quit IRC | 15:13 | |
| *** sams-gleb has joined #openstack-swift | 15:24 | |
| *** silor1 has joined #openstack-swift | 15:27 | |
| *** silor has quit IRC | 15:27 | |
| *** silor has joined #openstack-swift | 15:28 | |
| *** diogogmt has quit IRC | 15:28 | |
| *** oshritf has joined #openstack-swift | 15:29 | |
| *** oshritf has quit IRC | 15:29 | |
| *** PetrKube_ has joined #openstack-swift | 15:30 | |
| *** Jeffrey4l has quit IRC | 15:30 | |
| *** PetrKube_ is now known as PetrKubena | 15:30 | |
| *** Jeffrey4l has joined #openstack-swift | 15:30 | |
| *** silor1 has quit IRC | 15:32 | |
| *** Jeffrey4l has quit IRC | 15:32 | |
| *** Jeffrey4l has joined #openstack-swift | 15:32 | |
| cschwede | onovy: so i have that patch ready, but i just realized this won’t fix it. the mtime of the hashes.pkl is likely to change quite often, and therefore the listdir will never be retriggered | 15:33 |
| cschwede | onovy: on the other side, if you have partitions with archive data that never changes, it will recompute the hashes.pkl every reclaim_age/2, which is totally overkill then | 15:33 |
| onovy | cschwede: ah, right | 15:34 |
| onovy | idea: add option: force_sync=True to config | 15:34 |
| cschwede | onovy: i think a better solution is to simply list suffixes, and if they are not yet contained in the hashes.pkl then add them - this won’t recompute existing ones | 15:34 |
| onovy | or forst_listdir | 15:34 |
| onovy | *force | 15:34 |
| onovy | cschwede: hmm | 15:35 |
| onovy | i think we need this | 15:35 |
| onovy | 1. hashes can be not-consistent | 15:35 |
| onovy | doesn't matter why | 15:36 |
| onovy | 2. sometimes recount it | 15:36 |
| onovy | sometimes is run % 10 == 0 now | 15:36 |
| onovy | which is not working in every case | 15:36 |
| onovy | we should recount 1/10 of partitions every replicator run, right? | 15:36 |
| cschwede | onovy: yes; however the problem right now with the issue you found is that the suffixes are totally unknown, and that needs to be fixed quickly imo | 15:36 |
| onovy | cschwede: yep, but we could fix both at once | 15:37 |
| cschwede | true | 15:37 |
| onovy | if we have counter of "replicator runs", why not recount partition_id%10=replication run? | 15:37 |
| *** Jeffrey4l has quit IRC | 15:37 | |
| onovy | we recount exact 1/10 of partitions every run | 15:37 |
| cschwede | yeah, i was thinking about this already | 15:37 |
| onovy | after 10th run, everything will be recounted | 15:37 |
| onovy | almost this is already there | 15:38 |
| onovy | but it's 1/10 of random partitions | 15:38 |
| onovy | add determinism here and we are done | 15:38 |
| cschwede | exactly | 15:38 |
| cschwede | let’s see | 15:38 |
| onovy | do_listdir=(self.replication_count % 10) == 0, | 15:39 |
| onovy | we need new counter, replication_phases | 15:39 |
| onovy | increment it every whole phase | 15:39 |
| cschwede | i think i have a solution, testing | 15:39 |
| onovy | ok :) | 15:40 |
| *** Jeffrey4l has joined #openstack-swift | 15:49 | |
| *** links has joined #openstack-swift | 15:52 | |
| *** links has quit IRC | 15:52 | |
| *** diogogmt has joined #openstack-swift | 16:12 | |
| acoles | onovy: cschwede I am looking at patch 410330. I can clearly see the bug but I want to make sure I understand the impact | 16:15 |
| patchbot | https://review.openstack.org/#/c/410330/ | 16:15 |
| acoles | hey patchbot didn't you used to print the patch subject? | 16:16 |
| cschwede | patch 401330 | 16:16 |
| patchbot | https://review.openstack.org/#/c/401330/ - swift - Fixed regression in consolidate_hashes | 16:16 |
| acoles | aaahh, thanks cschwede ! | 16:17 |
| cschwede | acoles: yw :) | 16:17 |
| acoles | so, on a node that has a replica of new object in a new suffix, consolidated_hashes loses the suffix for the new object (the bug), so the replicator is unaware it has a new object to replicate and its suffix hashes appears the same as a node that does not have the object :/ | 16:19 |
| *** Guest13491 has quit IRC | 16:19 | |
| acoles | but, every 10% or replication passes a recalculate is forced, at which point the replicator will spot the difference with the node that is missing the object, and will sync the object. | 16:20 |
| *** rcernin has quit IRC | 16:20 | |
| cschwede | acoles: well, it’s non-deterministic - and during my testing i spotted a partition that was due too bad luck only fixed after 32 replication cycles | 16:21 |
| acoles | cschwede: understood | 16:21 |
| cschwede | acoles: so in addition to the fixed bug, i’m working on making the process deterministic, to ensure after 10 replication cycles all hashes.pkl are updated once | 16:21 |
| acoles | then the REPLICATE after the rsync will force the receiving side to recalculate its suffixes so its hashes.pkl should be good, correct? | 16:22 |
| acoles | so what causes the reported increase in rsync traffic? | 16:23 |
| cschwede | acoles: i think it was not increased if you take into account the whole sum of traffic over a period of time. however, due to the bug the replication traffic was a bit lower, and every 10th replication pass was syncing a whole partition then, because its suffix wasn’t replicated for some time | 16:26 |
| *** oshritf has joined #openstack-swift | 16:26 | |
| cschwede | acoles: honestly, i didn’t check the rsync traffic during my tests, i was already worried enough about the missing suffixes and focused on that | 16:26 |
| *** oshritf has quit IRC | 16:28 | |
| *** dmorita has joined #openstack-swift | 16:29 | |
| cschwede | onovy: you spotted the increased rsync traffic, right? ^^ | 16:31 |
| *** geaaru has quit IRC | 16:32 | |
| acoles | cschwede: reading bug report again, the increase is only when there is a mix of object server versions - which would be because get_hashes reports incorrect results on newer nodes (when there is a "lost suffix") so replicator fires off an rsync process (even though objects are in sync). when all nodes are upgraded then they all report the wrong get_hashes :/ so they think they are in sync and rsync is called less often | 16:33 |
| acoles | ^^ that's my guess at least | 16:34 |
| *** dmorita has quit IRC | 16:34 | |
| cschwede | acoles: ah yes, that’s the one issue if running a mixed cluster. however the underlying issue happens also on master, without running a mixed cluster | 16:34 |
| acoles | so there's two bugs really - 1. inconsistent suffix hashing between versions 2. failure to replicate lost suffixes | 16:35 |
| *** oshritf has joined #openstack-swift | 16:35 | |
| acoles | #2 being worse than #1 :( | 16:35 |
| cschwede | acoles: yeah, i should have opened a second bug maybe… | 16:36 |
| acoles | cschwede: IDK, they are related and there's only one fix so i think it's ok to leave as one | 16:36 |
| acoles | cschwede: at first i didn't understand the rsync stats increase and was concerned it was a separate issue, but i think that explains it | 16:37 |
| acoles | cschwede: I'll comment on launchpad | 16:37 |
| *** geaaru has joined #openstack-swift | 16:44 | |
| *** mvk has quit IRC | 16:49 | |
| *** two_tired has joined #openstack-swift | 16:50 | |
| acoles | cschwede: I ownder if we could have a gate job that runs a script to create a bunch of objects then checkout the last N releases, call get_hashes and check all results are consistent? | 16:53 |
| acoles | wonder* | 16:54 |
| *** hseipp has quit IRC | 16:54 | |
| cschwede | acoles: thx for looking into the bug and responding on lp | 16:55 |
| cschwede | acoles: hmm, sounds a bit tricky due to different requirements? pyeclib comes to my mind. but might be possible? | 16:55 |
| acoles | could run the get_hashes in a venv?? anyway, add that to my list of "better testing" :) | 16:56 |
| acoles | cschwede: I'll continue with review of patch now... | 16:56 |
| *** asettle has quit IRC | 16:59 | |
| *** two_tired has quit IRC | 17:01 | |
| *** davideagle_ has quit IRC | 17:03 | |
| *** tqtran has joined #openstack-swift | 17:06 | |
| *** tqtran has quit IRC | 17:10 | |
| *** cbartz has quit IRC | 17:10 | |
| *** davideagle_ has joined #openstack-swift | 17:20 | |
| onovy | cschwede: yep | 17:31 |
| onovy | a attached new graph | 17:31 |
| onovy | we have 2.7.0 only in production, and traffic is still increased | 17:31 |
| onovy | that information about "drop to normal" in bug is wrong | 17:32 |
| onovy | bug updated | 17:33 |
| onovy | i think reason is simple: one of replica have different hashes.pkl (sender have wrong, receiver have right) | 17:34 |
| onovy | and hashes are fixed undeterministicly, so this situation is fixed in 1 of 10 runs | 17:36 |
| onovy | and i think it's one bug with one fix :) | 17:36 |
| onovy | afk, on the way to home | 17:37 |
| *** rledisez has quit IRC | 17:37 | |
| *** jordanP has joined #openstack-swift | 17:39 | |
| *** davideagle_ has quit IRC | 17:46 | |
| *** jordanP has quit IRC | 17:48 | |
| *** geaaru has quit IRC | 17:49 | |
| *** jordanP has joined #openstack-swift | 17:49 | |
| *** davideagle_ has joined #openstack-swift | 17:50 | |
| *** two_tired has joined #openstack-swift | 17:52 | |
| *** oshritf has quit IRC | 18:03 | |
| *** psachin has quit IRC | 18:07 | |
| onovy | back | 18:17 |
| *** davideagle_ has quit IRC | 18:17 | |
| *** amoralej is now known as amoralej|off | 18:47 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 18:51 |
| *** two_tired has quit IRC | 18:56 | |
| *** Jeffrey4l has quit IRC | 18:58 | |
| *** siva_krish has joined #openstack-swift | 19:07 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Add assertions to invalidate_hashes tests https://review.openstack.org/402199 | 19:11 |
| acoles | cschwede: onovy I'll be around tomorrow to discuss patch 401330 more | 19:17 |
| patchbot | https://review.openstack.org/#/c/401330/ - swift - Fixed regression in consolidate_hashes | 19:17 |
| * acoles afk now | 19:17 | |
| *** acoles is now known as acoles_ | 19:18 | |
| *** jordanP has quit IRC | 19:26 | |
| *** jordanP has joined #openstack-swift | 19:40 | |
| *** jordanP has quit IRC | 19:46 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Add missing suffixes to hashes.pkl during replication https://review.openstack.org/402204 | 19:48 |
| *** amoralej|off is now known as amoralej | 19:56 | |
| onovy | cschwede: here? | 19:57 |
| *** tqtran has joined #openstack-swift | 20:07 | |
| *** tqtran has quit IRC | 20:12 | |
| *** Renich has quit IRC | 20:38 | |
| *** siva_krish has quit IRC | 20:40 | |
| *** asettle has joined #openstack-swift | 20:48 | |
| *** asettle has quit IRC | 20:50 | |
| *** McMurlock1 has joined #openstack-swift | 21:07 | |
| *** McMurlock1 has quit IRC | 21:21 | |
| mattoliverau | morning | 21:22 |
| *** sams-gleb has quit IRC | 21:37 | |
| *** sams-gleb has joined #openstack-swift | 21:37 | |
| *** vint_bra has quit IRC | 21:42 | |
| *** sams-gleb has quit IRC | 21:42 | |
| *** amoralej is now known as amoralej|off | 21:49 | |
| *** silor has quit IRC | 21:50 | |
| *** diogogmt has quit IRC | 21:58 | |
| *** davideagle has joined #openstack-swift | 22:31 | |
| *** vint_bra has joined #openstack-swift | 22:43 | |
| *** mvk has joined #openstack-swift | 23:09 | |
| *** vint_bra has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!