Tuesday, 2017-02-21

*** delattec has quit IRC00:00
*** jerrygb has joined #openstack-swift00:08
*** IRCFrEAK has joined #openstack-swift00:13
*** IRCFrEAK has left #openstack-swift00:13
*** catinthe_ has quit IRC00:17
*** catintheroof has joined #openstack-swift00:19
*** jordanP has joined #openstack-swift00:19
*** dmorita has joined #openstack-swift00:20
*** dmorita has quit IRC00:22
*** dmorita has joined #openstack-swift00:23
*** catintheroof has quit IRC00:23
*** dmorita_ has joined #openstack-swift00:27
*** dmorita has quit IRC00:27
*** dmorita_ has quit IRC00:31
*** kei_yama_ has joined #openstack-swift00:32
*** kei_yama has quit IRC00:33
*** dja has quit IRC00:37
*** dja has joined #openstack-swift00:39
*** dmorita has joined #openstack-swift00:41
*** caiobrentano has quit IRC00:43
*** caiobrentano has joined #openstack-swift00:43
*** caiobrentano has quit IRC00:48
*** dmorita has quit IRC00:52
*** dmorita has joined #openstack-swift00:53
*** dmorita has quit IRC00:58
*** dmorita has joined #openstack-swift01:02
*** dja has quit IRC01:10
*** dmorita has quit IRC01:17
*** dmorita has joined #openstack-swift01:18
*** dja has joined #openstack-swift01:18
*** sams-gleb has joined #openstack-swift01:19
*** vint_bra has joined #openstack-swift01:19
*** dmorita has quit IRC01:20
*** dmorita has joined #openstack-swift01:20
*** sams-gleb has quit IRC01:24
*** dmorita has quit IRC01:25
*** dmorita has joined #openstack-swift01:48
*** dmorita has quit IRC01:49
*** dmorita has joined #openstack-swift01:49
*** dmorita_ has joined #openstack-swift01:51
*** dmorita has quit IRC01:51
*** dmorita_ has quit IRC01:55
*** tonanhngo has joined #openstack-swift01:59
*** dmorita has joined #openstack-swift02:05
*** jeblair_ has joined #openstack-swift02:07
*** jeblair has quit IRC02:09
*** dmorita_ has joined #openstack-swift02:09
*** dmorita has quit IRC02:09
*** briancline has quit IRC02:10
*** dmorita_ has quit IRC02:11
*** dmorita has joined #openstack-swift02:11
*** webmobi has quit IRC02:11
*** m_kazuhiro has joined #openstack-swift02:12
*** m_kazuhiro has quit IRC02:17
*** dmorita has quit IRC02:44
*** dmorita_ has joined #openstack-swift02:44
*** winggundamth has joined #openstack-swift02:48
*** jerrygb has quit IRC02:57
*** dmorita_ has quit IRC03:10
*** dmorita has joined #openstack-swift03:12
*** dmorita has quit IRC03:16
*** winggundamth has quit IRC03:19
*** dmorita has joined #openstack-swift03:21
*** sams-gleb has joined #openstack-swift03:22
*** dmorita_ has joined #openstack-swift03:23
*** dmorita has quit IRC03:23
*** winggundamth has joined #openstack-swift03:23
*** sams-gleb has quit IRC03:27
*** dmorita_ has quit IRC03:27
*** dmorita has joined #openstack-swift03:29
*** dmorita has quit IRC03:34
*** dmorita has joined #openstack-swift03:34
*** dmorita_ has joined #openstack-swift03:37
*** dmorita has quit IRC03:38
*** dmorita has joined #openstack-swift03:39
*** dmorita_ has quit IRC03:39
*** dmorita has quit IRC03:55
*** chlong has joined #openstack-swift04:03
*** dmorita has joined #openstack-swift04:04
*** dmorita has quit IRC04:05
*** dmorita has joined #openstack-swift04:05
*** tonanhngo has quit IRC04:08
*** dmorita has quit IRC04:10
*** dmorita has joined #openstack-swift04:10
*** dmorita has quit IRC04:12
*** dmorita has joined #openstack-swift04:12
*** dmorita_ has joined #openstack-swift04:15
*** dmorita has quit IRC04:16
*** psachin has joined #openstack-swift04:17
*** dmorita has joined #openstack-swift04:18
*** dmorita_ has quit IRC04:19
*** dmorita has quit IRC04:21
*** dmorita has joined #openstack-swift04:21
*** dmorita has quit IRC04:26
*** dmorita has joined #openstack-swift04:26
*** dmorita has quit IRC04:27
*** dmorita has joined #openstack-swift04:28
*** jordanP has quit IRC04:29
*** gatuus has quit IRC05:01
John341clayg: unfortunately, providing "X-Auth-Token: <ID>", where the id is the id for a fernet generated token.05:23
*** seongsoocho has quit IRC05:24
*** seongsoocho has joined #openstack-swift05:24
*** sams-gleb has joined #openstack-swift05:25
John341I still get 401 Not authorized :(05:26
*** sams-gleb has quit IRC05:30
openstackgerritTim Burke proposed openstack/swift master: We don't "yield" in collect_parts anymore  https://review.openstack.org/43632206:17
timburkejrichli: ^^^06:17
*** dja has quit IRC06:18
*** dmorita has quit IRC06:38
*** dmorita has joined #openstack-swift06:42
*** dmorita has quit IRC06:47
*** dmorita has joined #openstack-swift06:48
*** dmorita has quit IRC06:52
*** dmorita has joined #openstack-swift06:57
*** dmorita has quit IRC07:01
*** dmorita has joined #openstack-swift07:03
*** dmorita has quit IRC07:05
*** dmorita has joined #openstack-swift07:05
*** dmorita has quit IRC07:09
*** sanchitmalhotra has quit IRC07:10
*** sanchitmalhotra has joined #openstack-swift07:10
*** sams-gleb has joined #openstack-swift07:12
*** m_kazuhiro has joined #openstack-swift07:14
*** m_kazuhiro has quit IRC07:19
*** daidv has joined #openstack-swift07:22
daidvHi everybody, I wonder how can I get a trustly rolling upgrade steps from swift community documentation?07:23
daidvAcctually, I'm facing with error "No *** running" when I try to run "swift-init rest stop" command?07:24
*** tesseract has joined #openstack-swift07:25
*** dmorita has joined #openstack-swift07:32
*** ChubYann has quit IRC07:32
*** openstackgerrit has quit IRC07:32
*** dmorita has quit IRC07:36
*** pcaruana has joined #openstack-swift07:37
*** dmorita has joined #openstack-swift07:43
*** dmorita has quit IRC07:48
*** dmorita has joined #openstack-swift07:49
*** dmorita has quit IRC07:54
*** hseipp has joined #openstack-swift07:57
*** dmorita has joined #openstack-swift07:57
*** klrmn has quit IRC08:00
*** dmorita_ has joined #openstack-swift08:01
*** dmorita has quit IRC08:01
*** dja has joined #openstack-swift08:04
*** dmorita_ has quit IRC08:05
*** bob_cheesey has joined #openstack-swift08:13
*** amoralej|off is now known as amoralej08:27
*** dmorita has joined #openstack-swift08:28
*** dmorita_ has joined #openstack-swift08:30
*** dmorita__ has joined #openstack-swift08:32
*** dmorita_ has quit IRC08:32
*** dmorita has quit IRC08:32
*** dmorita__ has quit IRC08:33
*** dmorita has joined #openstack-swift08:34
*** bob_cheesey has quit IRC08:37
*** dja has quit IRC08:37
*** bob_cheesey has joined #openstack-swift08:38
*** dmorita_ has joined #openstack-swift08:38
*** dmorita has quit IRC08:39
*** dmorita_ has quit IRC08:42
*** dmorita has joined #openstack-swift08:42
*** dmorita has quit IRC08:47
*** dmorita has joined #openstack-swift08:50
*** dmorita_ has joined #openstack-swift08:51
*** dmorita has quit IRC08:54
*** dmorita_ has quit IRC08:55
*** dmorita has joined #openstack-swift08:59
*** dmorita has quit IRC09:03
*** dmorita_ has joined #openstack-swift09:03
*** dmorita_ has quit IRC09:07
*** dmorita has joined #openstack-swift09:08
*** dmorita has quit IRC09:12
*** dmorita has joined #openstack-swift09:13
*** dmorita has quit IRC09:15
*** m_kazuhiro has joined #openstack-swift09:15
*** dmorita has joined #openstack-swift09:15
*** klrmn has joined #openstack-swift09:16
*** logan- has quit IRC09:16
*** dmorita has quit IRC09:19
*** m_kazuhiro has quit IRC09:20
*** daidv has quit IRC09:21
*** dmorita has joined #openstack-swift09:26
*** dmorita has quit IRC09:27
*** dmorita has joined #openstack-swift09:28
*** dmorita_ has joined #openstack-swift09:31
*** dmorita__ has joined #openstack-swift09:32
*** dmorita_ has quit IRC09:32
*** dmorita has quit IRC09:33
*** cbartz has joined #openstack-swift09:33
*** dmorita has joined #openstack-swift09:35
*** dmorita__ has quit IRC09:37
*** Jeffrey4l_ has quit IRC09:38
*** dmorita has quit IRC09:39
*** dmorita has joined #openstack-swift09:46
*** dmorita has quit IRC09:51
*** Jeffrey4l_ has joined #openstack-swift09:51
*** dmorita has joined #openstack-swift09:52
*** dmorita_ has joined #openstack-swift09:54
*** dmorita_ has quit IRC09:57
*** dmorita_ has joined #openstack-swift09:57
*** dmorita has quit IRC09:57
*** dmorita has joined #openstack-swift09:59
*** dmorita_ has quit IRC09:59
*** dmorita_ has joined #openstack-swift10:01
*** dmorita has quit IRC10:01
*** oshritf has joined #openstack-swift10:02
*** dmorita_ has quit IRC10:05
*** dmorita has joined #openstack-swift10:07
*** dmorita has quit IRC10:09
*** dmorita has joined #openstack-swift10:10
*** balu_23 has joined #openstack-swift10:10
*** balu_23 has left #openstack-swift10:13
*** dmorita has quit IRC10:14
*** dmorita has joined #openstack-swift10:18
*** dmorita has quit IRC10:22
*** dmorita has joined #openstack-swift10:23
*** dmorita_ has joined #openstack-swift10:26
*** sams-gle_ has joined #openstack-swift10:27
*** dmorita has quit IRC10:27
*** dmorita_ has quit IRC10:28
*** dmorita has joined #openstack-swift10:28
*** sams-gleb has quit IRC10:29
*** dmorita has quit IRC10:30
*** dmorita has joined #openstack-swift10:31
*** dmorita has quit IRC10:32
*** dmorita has joined #openstack-swift10:32
*** dmorita has quit IRC10:34
*** dmorita has joined #openstack-swift10:35
*** dmorita has quit IRC10:39
*** dmorita has joined #openstack-swift10:49
*** dmorita_ has joined #openstack-swift10:51
*** dmorita has quit IRC10:51
*** openstackgerrit has joined #openstack-swift10:54
openstackgerritAlexandre Lécuyer proposed openstack/swift master: KV / small files patches. For discussion with Romain Le Disez at the PTG  https://review.openstack.org/43640610:54
*** dmorita_ has quit IRC10:55
*** dmorita has joined #openstack-swift10:55
*** dmorita has quit IRC10:59
*** dmorita has joined #openstack-swift11:05
*** oshritf has quit IRC11:06
*** oshritf has joined #openstack-swift11:08
*** dmorita has quit IRC11:10
*** klrmn has quit IRC11:13
*** m_kazuhiro has joined #openstack-swift11:16
*** m_kazuhiro has quit IRC11:20
*** dmorita has joined #openstack-swift11:22
*** NM has joined #openstack-swift11:25
*** sams-gle_ has quit IRC11:35
*** oshritf has quit IRC11:35
*** sams-gleb has joined #openstack-swift11:35
*** oshritf has joined #openstack-swift11:37
*** sams-gleb has quit IRC11:40
*** Jeffrey4l_ has quit IRC11:44
*** kei_yama_ has quit IRC11:46
*** Jeffrey4l_ has joined #openstack-swift11:47
*** Jeffrey4l_ has quit IRC11:49
*** oshritf has quit IRC11:49
*** Jeffrey4l_ has joined #openstack-swift11:50
*** sams-gleb has joined #openstack-swift11:53
*** mvk has quit IRC11:53
*** jerrygb has joined #openstack-swift11:59
*** jerrygb has quit IRC12:03
*** _JZ_ has joined #openstack-swift12:08
*** winggundamth has quit IRC12:16
*** winggundamth has joined #openstack-swift12:16
*** _JZ_ has quit IRC12:23
*** mvk has joined #openstack-swift12:29
*** oshritf has joined #openstack-swift12:29
*** oshritf has quit IRC12:32
*** oshritf has joined #openstack-swift12:33
*** logan- has joined #openstack-swift12:35
*** oshritf has quit IRC12:36
*** winggundamth has quit IRC12:36
*** caiobrentano has joined #openstack-swift12:38
*** oshritf has joined #openstack-swift12:51
*** McMurlock1 has joined #openstack-swift13:01
*** gabor_antal has joined #openstack-swift13:02
*** zul_ has joined #openstack-swift13:17
*** zul_ has quit IRC13:20
*** zul has joined #openstack-swift13:20
*** jerrygb_ has joined #openstack-swift13:21
*** geaaru has joined #openstack-swift13:22
*** klamath has joined #openstack-swift13:23
*** jerrygb_ has quit IRC13:30
*** dmorita has quit IRC13:36
*** amoralej is now known as amoralej|lunch13:39
*** dmorita has joined #openstack-swift13:40
*** dmorita_ has joined #openstack-swift13:41
*** dmorita has quit IRC13:41
*** winggundamth has joined #openstack-swift13:43
*** dmorita_ has quit IRC13:46
*** winggundamth has quit IRC13:47
claygJohn341: bummer!13:48
claygdaidv: you left!  That's a warning not an error.  When you ask services to stop some of them are not running - that's probably ok?13:48
claygdaidv you're correct though - as near as I can tell neither https://docs.openstack.org/developer/swift/ops_runbook/index.html nor https://docs.openstack.org/developer/swift/admin_guide.html say give best practices overview to perform a rolling swift upgrade - one of them should13:52
clayg... and briancline isn't around13:52
clayg... so I was hating on lp bug #1380815 ... and remembered13:53
openstackLaunchpad bug 1380815 in OpenStack Object Storage (swift) "swift.common.utils should not patch logging on import" [High,New] https://launchpad.net/bugs/138081513:53
claygI'm *always* promsing people beers for fixing bugs?  Does anyone need to collect on that at the PTG?13:53
*** vint_bra has joined #openstack-swift13:54
*** amoralej|lunch is now known as amoralej13:57
*** jerrygb has joined #openstack-swift13:57
kota_good morning14:05
*** jordanP has joined #openstack-swift14:06
*** jamielennox|away is now known as jamielennox14:09
*** dmorita has joined #openstack-swift14:09
*** m_kazuhiro has joined #openstack-swift14:11
*** david-lyle has joined #openstack-swift14:12
*** jordanP has quit IRC14:12
*** dmorita has quit IRC14:13
*** dmorita has joined #openstack-swift14:13
*** dmorita_ has joined #openstack-swift14:15
*** dmorita has quit IRC14:16
*** jeblair_ is now known as jeblair14:16
*** dmorita has joined #openstack-swift14:17
*** dmorita_ has quit IRC14:17
*** dmorita_ has joined #openstack-swift14:18
*** dmorita has quit IRC14:18
*** dmorita_ has quit IRC14:23
*** zaitcev has joined #openstack-swift14:24
*** ChanServ sets mode: +v zaitcev14:24
*** m_kazuhiro has quit IRC14:25
*** McMurlock2 has joined #openstack-swift14:26
*** McMurlock1 has quit IRC14:26
*** hseipp has quit IRC14:26
*** jordanP has joined #openstack-swift14:31
*** dmorita has joined #openstack-swift14:33
*** tqtran has joined #openstack-swift14:35
*** dmorita has quit IRC14:38
*** dmorita has joined #openstack-swift14:38
clayggood morning!14:39
*** dmorita_ has joined #openstack-swift14:40
*** dmorita has quit IRC14:40
notmynameI'm looking for eran, if anyone sees him14:40
notmynamehe found my laptop charger yesterday and turned it in, but no desk I ask knows about it14:40
*** dmorita_ has quit IRC14:42
*** dmorita has joined #openstack-swift14:42
*** dmorita has quit IRC14:44
*** dmorita has joined #openstack-swift14:44
notmynameI'm in the arch-WG capabilities discussion. (although it seems to be nova devs talking about which CPU features should be exposed in horizon)14:45
notmynamerelated to this doc https://review.openstack.org/#/c/386555/7/guidelines/capabilities.rst,unified14:45
patchbotpatch 386555 - api-wg - Add API capabilities discovery guideline14:45
*** david-lyle has quit IRC14:45
*** david-lyle has joined #openstack-swift14:46
*** david-lyle has quit IRC14:46
claygnotmyname: you're doing god's work my friend14:46
notmynameI've seen kota_ and zaitcev this morning, but nobody else14:47
*** cdelatte has joined #openstack-swift14:48
*** dmorita has quit IRC14:49
*** dmorita has joined #openstack-swift14:50
*** david-lyle has joined #openstack-swift14:53
*** sams-gleb has quit IRC14:55
*** dmorita has quit IRC14:55
*** sams-gleb has joined #openstack-swift14:55
*** dmorita has joined #openstack-swift14:57
*** david-lyle has quit IRC14:57
*** sams-gleb has quit IRC14:59
*** dmorita has quit IRC15:02
*** sanchitmalhotra has quit IRC15:05
*** sanchitmalhotra has joined #openstack-swift15:06
*** ouchkernel has joined #openstack-swift15:06
*** dmorita has joined #openstack-swift15:11
*** dmorita_ has joined #openstack-swift15:12
*** dmorita has quit IRC15:12
*** sams-gleb has joined #openstack-swift15:13
*** psachin has quit IRC15:14
*** m_kazuhiro has joined #openstack-swift15:15
*** dmorita_ has quit IRC15:16
jrichliI have been in macon at a talk about tempest plugins15:18
claygwhat time is he picture thing?  is that today?15:20
claygjrichli: does swift have a tempest plugin!?15:21
timburkeclayg: thursday, 9:20 iirc15:21
jrichliclayg: picture thing is thursday at 9, i believe.  no, we dont have a plugin. would have to be in a separate repo.15:21
*** dmorita has joined #openstack-swift15:28
*** m_kazuhiro has quit IRC15:28
*** david-lyle_ has joined #openstack-swift15:29
claygwhoot!  large objects ssync replication test in progress!15:30
claygthroughput is SO much higher than LOSF scenario!15:30
claygjrichli: let's move *everything* into seperate repos!?15:31
jrichlilol15:31
*** dmorita has quit IRC15:32
*** m_kazuhiro has joined #openstack-swift15:36
*** dmorita has joined #openstack-swift15:40
*** dmorita_ has joined #openstack-swift15:44
*** dmorita has quit IRC15:45
*** dmorita has joined #openstack-swift15:47
*** hseipp has joined #openstack-swift15:47
*** dmorita_ has quit IRC15:49
*** dmorita_ has joined #openstack-swift15:51
*** dmorita has quit IRC15:51
*** dmorita_ has quit IRC15:55
*** SkyRocknRoll has joined #openstack-swift15:56
*** dmorita has joined #openstack-swift15:58
*** jamielennox is now known as jamielennox|away15:59
*** jamielennox|away is now known as jamielennox15:59
*** dmorita_ has joined #openstack-swift16:01
*** david-lyle has joined #openstack-swift16:01
*** bob_cheesey has quit IRC16:02
*** dmorita has quit IRC16:02
*** david-lyle_ has quit IRC16:02
*** dmorita_ has quit IRC16:05
*** dmorita has joined #openstack-swift16:07
openstackgerritMahati Chamarthy proposed openstack/swift master: Test coverage for reconstructor  https://review.openstack.org/43652216:10
*** dmorita_ has joined #openstack-swift16:11
timburkemahatic: you're still awake? go to sleep!16:11
*** gatuus has joined #openstack-swift16:11
*** dmorita has quit IRC16:11
mahatictimburke: hey! I was about to ping you and clay16:11
mahatictimburke: clayg I noticed some missing coverage during this patch 425468 and some follow on I think16:12
patchbothttps://review.openstack.org/#/c/425468/ - swift - Shuffle disks and parts in reconstructor (MERGED)16:12
mahaticI attempted to cover it in patch 43652216:12
patchbothttps://review.openstack.org/#/c/436522/ - swift - Test coverage for reconstructor16:12
mahatictimburke: it's only 9:40 pm. Good time to push changes sitting in drafts ;)16:13
mahatichaving a good time I hope!16:13
* mahatic afk for a bit16:14
*** dmorita_ has quit IRC16:15
*** vinsh_ has quit IRC16:27
*** david-lyle has quit IRC16:27
*** vinsh has joined #openstack-swift16:27
*** david-lyle has joined #openstack-swift16:34
*** dmorita has joined #openstack-swift16:37
*** chsc has joined #openstack-swift16:38
*** chsc has joined #openstack-swift16:38
*** jamielennox is now known as jamielennox|away16:39
*** SkyRocknRoll has quit IRC16:41
*** dmorita_ has joined #openstack-swift16:41
*** dmorita has quit IRC16:42
*** oshritf has quit IRC16:43
*** dmorita has joined #openstack-swift16:44
*** dmorita_ has quit IRC16:44
*** dmorita has quit IRC16:48
*** garyj has joined #openstack-swift16:52
*** dmorita has joined #openstack-swift16:57
*** david-lyle has quit IRC16:58
*** m_kazuhiro has quit IRC17:01
*** zul has quit IRC17:01
*** zaitcev has quit IRC17:02
*** dmorita has quit IRC17:02
*** tqtran has quit IRC17:03
*** Jeffrey4l_ has quit IRC17:03
*** cbartz has quit IRC17:04
*** mvk has quit IRC17:06
*** Jeffrey4l has joined #openstack-swift17:06
*** pcaruana has quit IRC17:12
*** garyj has quit IRC17:14
*** david-lyle has joined #openstack-swift17:15
*** david-lyle has quit IRC17:16
*** silor has joined #openstack-swift17:16
*** geaaru has quit IRC17:20
*** silor has quit IRC17:23
*** silor has joined #openstack-swift17:23
ntatacan anyone help me understand what happens when a *.data file gets deleted  (by mistake) in a cluster?17:24
ntatareplicator looks for hashes.pkl (in our case is untouched) and replicates only if there's a change to hashes.pkl17:25
ntataauditor is taking care of the expired objects and the *.ts (deleted) objects17:25
*** vinsh_ has joined #openstack-swift17:25
*** vinsh_ has quit IRC17:26
*** vinsh has quit IRC17:28
*** silor1 has joined #openstack-swift17:31
*** silor has quit IRC17:32
*** silor1 is now known as silor17:32
*** jordanP has quit IRC17:35
*** dmorita has joined #openstack-swift17:36
*** tonanhngo has joined #openstack-swift17:38
*** tesseract has quit IRC17:41
claygntata: we miss you :'(17:47
*** vinsh has joined #openstack-swift17:48
claygntata: if a change happens in a suffix w/o invalidating the suffix - no one will really notice17:48
*** cdelatte has quit IRC17:48
claygit's like writing directly to a block device with O_DIRECT instead of going through the filesystem/linux-pagecache17:48
claygyou're tinkinering under the purvue of the system17:48
claygany place we udpate a suffix (write/delete/post/quarantine) we always invalidate the suffix - you HAVE to17:48
claygoh... or rsync - rsync can change a suffix - and it has to call REPLICATE with the explict hash invalidations17:49
claygif the suffix get's another update later - that would of course pick up any changes that happened underneath/out-of-band17:49
claygonovy: do you know Allison Randal - she works on debian (ubuntu?) packages/testing for openstack17:51
*** mmotiani_ has joined #openstack-swift17:51
onovyclayg: hi, no17:51
onovybut that name i already saw somewhere...17:51
claygonovy: yeah...17:51
*** zul has joined #openstack-swift17:53
*** jrichli_ has joined #openstack-swift17:55
*** jrichli_ has quit IRC17:55
onovytimburke: hi, thanks for working on stable branch!17:58
timburkeonovy: sure! and now we should even have a working gate again :-)17:59
onovynice!17:59
timburkemanaged to corner someone from infra over lunch -- thanks jhesketh!18:00
*** McMurlock2 has quit IRC18:01
notmynamentata: yes, your right. so you'll need to invalidate hashes.pkl to get replication to fix it. the "easy" way to do this is to delete hashes.pkl18:02
*** zul has quit IRC18:04
onovynotmyname: hi, btw: we really need to found solution for ntata question automagically in cluster18:05
onovythere are many many situation where pkl file is not consistent with directory content18:05
onovywe should recount it "sometimes"18:05
onovysame as we call listdir sometimes18:05
onovyexamples: power outage, small filesystem corruption, daemon crash, network outage (just after rsync call, before REPLICATE call), ...18:06
onovywe have done recount of part of production cluster and we really found few of inconsistent pkls18:07
notmynameonovy: I'm pretty sure that clayg has solved it (he had lots of stuff on a whiteboard, at least)18:07
onovyalready talked with clayg about this, not sure if solution is already written18:07
*** mvk has joined #openstack-swift18:10
*** zul has joined #openstack-swift18:11
*** openstack has joined #openstack-swift18:14
ntataclayg, glad to know I'm remembered and missed! Got pulled into some off-community work.. I'm going to lurk around more often now on :D18:16
ntataclayg, notmyname yes! that's what I observed.. removing hashes.pkl will recover everything that's lost.18:16
claygyeah we need to get off rsync for that reason - ssync invalidates as it goes - in the datapath ftw18:21
claygonovy: i want to add more context/data to hashes.pkl - i'm fine with a 'rehashed at time()'18:22
claygif that *really* manages to go a few days without any activity i'm cool to rehash it18:22
claygonovy: there was a number of bugs that PavelK fixed that could introduce bad pickles18:22
claygbut those are better18:22
claygacoles: says he knows of 1 more race18:22
*** klrmn has joined #openstack-swift18:24
*** mmotiani_ has quit IRC18:25
redbo_There can be billions of suffix dirs, they probably won't all get activity every few days.  Also, you don't really want to re-index them every few days.  Maybe every few weeks.18:28
redbo_If you have multiple replicas, at least one *should* get updated, and then replication will re-index until the hashes match.  Or that's how it was supposed to work.18:30
claygaren't thre like *exactly* 4096 * part_power unique suffixes?  We can monitor a rate at which they get invalidated and also pick a satisfactory frequency at which to reindex them "after a while when we're not busy"18:31
clayger... 4096 * 2 ** part_power /we18:31
*** zul has quit IRC18:36
*** redbo_ is now known as redbo18:38
*** ChanServ sets mode: +v redbo18:38
*** foexle has joined #openstack-swift18:39
*** jerrygb has quit IRC18:42
redboI'm sure you could pick some rate to re-index that doesn't have a giant impact.  But I think replication will fix it faster, since the chances that the hashes for all copies of a suffix will match and be wrong are low.  Unless there's some sort of horrible bug.18:43
*** david-lyle has joined #openstack-swift18:46
*** matt6434 is now known as mattoliverau18:48
*** ChanServ sets mode: +v mattoliverau18:48
*** zul has joined #openstack-swift18:54
*** tqtran has joined #openstack-swift18:54
*** mmotiani_ has joined #openstack-swift18:54
*** gatuus has quit IRC18:55
*** gatuus has joined #openstack-swift18:56
*** david-lyle has quit IRC18:57
*** dmorita has quit IRC19:00
*** vinsh has quit IRC19:00
*** m_kazuhiro has joined #openstack-swift19:01
*** jordanP has joined #openstack-swift19:01
onovyredbo: yep, theory. reality is, horrible bugs is there. many many race conditions. we should fix it: yes? can we be sure if we fixed all? no :).19:02
onovyreality is we have this inconsistency in production cluster and only way to fix is to recount that hashes by hand19:02
*** dmorita has joined #openstack-swift19:04
*** dmorita has quit IRC19:08
timburkenotmyname: you should go look at some of the newton patches now that the gate's fixed https://review.openstack.org/#/dashboard/?title=Open+Backports&foreach=is:open+branch:%255Estable/.*&Swift=project:openstack/swift&Swift+Client=project:openstack/python-swiftclient19:09
redboI'd rather at least know why replication isn't noticing the hashes differ and re-indexing them.  Or are both servers hitting the same bug so their hashes match but are both wrong.  Those questions should be sort of easy to answer.19:13
redboI think it's a bad idea to spackle over bugs without understanding them.19:13
redboMaybe I should go back, but I didn't even see what the problem is.  Are things not getting replicated?19:15
openstackgerritTim Burke proposed openstack/swift master: Add abort flag to replicator/reconstructor  https://review.openstack.org/42901519:15
notmynameredbo: I think there's two general cases. one is if a file gets deleted/corupted externally to swift processes (ie without hashes.okl getting invalidated). the other is bugs in the logic of when hashes.pkl gets updated normally. the second is what onovy and clayg have been looking in to lately19:17
onovyright19:18
onovysimple example: small XFS corruption and file gets lost19:19
onovylost=not in dir index19:19
onovyauditor and replicator will not notice it19:19
onovyand even if we find and fix all bugs for second case (bugs in logic of when pkl getd updated normally) we still needs to recount them automagically19:22
redboSure.  But we've talked about this 3 or 4 times before.  And the math on making sure the entire system is re-indexed every week or whatever is pretty depressing.19:22
onovyso don't recount it so often19:23
*** zaitcev has joined #openstack-swift19:23
*** ChanServ sets mode: +v zaitcev19:23
onovy+we can recount only first/master replica19:23
pdardeau_we're currently toying with an idea that might help in this regard (efficiently noticing the changes)19:24
*** tqtran has quit IRC19:25
redboHow is only re-indexing the first primary going to help?19:26
*** tqtran has joined #openstack-swift19:26
onovyredbo: right, sry19:27
*** jordanP has quit IRC19:32
*** amoralej is now known as amoralej|off19:34
timburkeonovy: on https://review.openstack.org/#/c/415556/, the failure's legit -- i think tonyb was going to look at whether it makes sense to bump oslosphinx lower bounds for newton, or maybe we'll want to just revert https://review.openstack.org/#/c/354426/19:38
patchbotpatch 415556 - python-swiftclient (stable/newton) - Require newer version of oslosphinx19:38
patchbotpatch 354426 - python-swiftclient - reenable sidebar links (MERGED)19:38
onovytimburke: yep. we should fix global requirements for stable/newton, than recheck 415556, or revert that 35442619:40
onovy*then19:40
redboI'm probably biased by people who used to want to re-index every 24 hours or something.  An auditor speed reindexing would be fine.19:41
*** Renich has joined #openstack-swift19:41
onovywe should reindex every < reclaim_age idealy. but because it's rare, i think it's acceptable higher interval19:42
onovyof course after all bugs in this part be fixed19:42
redboThough some people think the auditor needs to look at everything all the time super fast too.19:43
*** jordanP has joined #openstack-swift19:44
openstackgerritMerged openstack/swift master: We don't "yield" in collect_parts anymore  https://review.openstack.org/43632219:44
tonybtimburke: Yeah I don't really like it but the 2 options are 1) add olsosphinx>=$whatever and relying on u-c to select the version that works ; or 2) revert the add sidebar review19:45
tonybtimburke: it will get better in $future19:45
onovyredbo: we have really long almost never-ending cycle for auditor in production19:46
timburkeit's already better on master -- look at us, living in the future!19:46
tonybtimburke: :)19:46
onovyhttps://github.com/openstack/requirements/commit/e4b0883ad8bc95ed0a98f83dc1f095d086a0692e19:47
onovyyep, i define future!19:47
timburketonyb: any strong feelings on which way that ought to go? i'm kinda partial toward the revert, just because i work with a guy that can actually land everything necessary to get onovy's patch landed19:47
* onovy prefers to update g-r19:48
onovyso backport https://github.com/openstack/requirements/commit/e4b0883ad8bc95ed0a98f83dc1f095d086a0692e to stable/newton19:48
*** silor has quit IRC19:49
tonybtimburke: I think 1) is the least worst.  It's a lie but it's one we tell in many other places19:50
tonybtimburke: I can update the review to do that thing if you like19:52
onovytonyb: why not fix global requirements and keep my review as it's?19:52
onovyhey really, it's just formal change19:53
tonybonovy: becuase the stable policy prevents altering the minimum accepted version on stable branches19:53
onovyso that's deadlock19:53
onovyso 3) don't do anything19:54
tonybonovy: Sure19:54
tonybonovy: it isn't deadlock we have 2 ways to move forward19:54
onovyi don't see point for "1"19:54
tonybonovy: and this will get better in pike19:54
onovybetter?19:55
*** vinsh has joined #openstack-swift19:56
onovyu-c of oslosphinx for newton is 4.7.0 -> stable branch for swiftclient will build fine19:56
onovyif we can alter minimum accepted version in global req, there is only two solutions: revert, do nothing19:57
*** hseipp has quit IRC19:57
*** vinsh has quit IRC19:57
*** vinsh has joined #openstack-swift19:57
onovy*we can't19:57
tonybonovy: No option 1 will pass the gate and do what you expect but it'll be a lie but as I said it's one we tell elsewheer (that our minimums mean anything)19:58
tonyband that will get better in pike as we're writing tools to test minimums19:58
onovycool19:58
onovyyou said: 1) add olsosphinx>=$whatever and relying on u-c19:59
onovybut that's what we already have in stable/newton branch19:59
onovyso it's same as my "1" solution: do nothing19:59
onovyand accept lie :)19:59
*** mmotiani_ has quit IRC19:59
tonybSure, that's true I forgot that you already had oslosphinx in requirements.txt20:00
onovywe already accepted that lie20:00
*** mmotiani_ has joined #openstack-swift20:00
onovybtw: http://lists.openstack.org/pipermail/openstack-dev/2016-September/102870.html20:00
onovyi TRIED to fix it before release20:00
onovyand prevent this mess20:01
*** chlong has quit IRC20:01
*** gatuus has quit IRC20:01
*** david-lyle has joined #openstack-swift20:01
*** gatuus has joined #openstack-swift20:01
tonybonovy: Yeah okay IIRC that had large knock-on effects that were unacceptable at that stage of the release20:02
onovyso to finish it: do nothing. it's "fixed" in my debian package, in master and it's wrong in stable branch, which doesn't matter20:03
openstackgerritOpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements  https://review.openstack.org/8873620:03
tonybonovy: yeah20:04
*** mmotiani_ has quit IRC20:05
onovyabandoned, thanks :]20:06
patchbotError: Spurious "]".  You may want to quote your arguments with double quotes in order to prevent extra brackets from being evaluated as nested commands.20:06
*** gatuus has quit IRC20:06
*** chlong has joined #openstack-swift20:08
*** mmotiani_ has joined #openstack-swift20:08
openstackgerritShashirekha Gundur proposed openstack/swift master: Invalidate cached tokens api  https://review.openstack.org/37031920:08
openstackgerritThiago da Silva proposed openstack/swift master: WIP: centos func test  https://review.openstack.org/43664220:12
*** mmotiani_ has quit IRC20:14
*** NM has quit IRC20:17
claygonovy: redbo: the change to trigger a reindex when the auditor finds an expired tombstone probably helps too!20:18
onovylink?20:18
claygI think writing down the index time off suffixes is the first step and the only one I'm willing to commit to20:18
claygonovy: it was old-ish - I think mahatic did it20:18
claygso if I find there's suffixes in my cluster that aren't getting re-indexed after weeks or months ... I think that sort of defacto means that cluster has the spare time to do the reindex20:20
onovyclayg:  81d4673674febdbe0bba6e27a6d24739456eb3c4?20:20
claygonovy: yuppers - she's a good 'en20:21
*** mmotiani_ has joined #openstack-swift20:21
*** NM has joined #openstack-swift20:21
onovyah. it's from 2.10.0 and we have 2.7.0 in production!20:21
onovythat is nice solution for ts20:22
claygonovy: rather than all the backports - can you just get on the upgrade/follow-master train with us?20:22
claygI just... yesterdays bugs are so *boring* - we fixed those already!  ;)20:22
claygthink of all the bugs we can write *just this week*20:23
*** david-lyle has quit IRC20:23
onovyi'm trying to finish/cleanup newton20:23
onovyso only 1 release behind!20:23
timburketdasilva: that looks exciting...20:26
*** mmotiani_ has quit IRC20:27
onovyclayg: but really: we need to change our process for deploying. because stretch is frozen and i can't upgrade swift to something newer than newton for months20:27
*** mmotiani_ has joined #openstack-swift20:29
*** ChubYann has joined #openstack-swift20:30
tdasilvatimburke: but does it work??20:30
tdasilva:)20:30
*** tqtran has quit IRC20:32
tdasilvaacoles: hi! just noticed we still have gate jobs: gate-swift-tox-xfs-tmp-func-ubuntu-trusty and gate-swift-tox-xfs-tmp-func-fast-post-ubuntu-trusty20:32
tdasilvashould we get rid of  gate-swift-tox-xfs-tmp-func-fast-post-ubuntu-trusty since fast post is enabled by default?20:33
claygtdasilva: yes20:33
claygtdasilva: slow post is dead to me20:33
jrichlitdasilva: as long as older versions / backports are still going to have it as a test20:35
tdasilvajrichli: oh yeah, forgot about that, maybe that's why it is there, for the backports stuff that mattoliverau was doing20:36
tdasilvasorry, not backports, stable branches20:36
jrichliyes, stable branches :-)20:36
timburketdasilva: no, we should not! stable branches! mattoliverau and i only just got the newton gate working again!20:36
tdasilvatimburke: right right, sorry, forgot about that again20:37
timburkenow, if it's still triggering on master, that's a separate issue...and one we should fix :-)20:37
tdasilvawas about to break it again :P20:37
*** david-lyle has joined #openstack-swift20:38
*** Renich has quit IRC20:42
timburkeclayg: ha! sound familiar? #3 on HN: https://blog.packagecloud.io/eng/2017/02/21/set-environment-variable-save-thousands-of-system-calls/20:46
*** david-lyle has quit IRC20:49
*** _JZ_ has joined #openstack-swift20:50
mattoliverauwont everybobdy please think of the stable branches :P20:51
mattoliverautho if your were to break it then at least do it this week so I can trick jhesketh into reviewing infra patches :P20:52
openstackgerritKota Tsuyuzaki proposed openstack/liberasurecode master: jerasure: plug memory leaks  https://review.openstack.org/43469620:52
*** NM has quit IRC20:55
*** mmotiani_ has quit IRC20:55
*** david-lyle has joined #openstack-swift20:55
*** NM has joined #openstack-swift20:57
onovytimburke: thanks for that TZ link, that is really interesting20:59
*** david-lyle has quit IRC20:59
timburkeonovy: it very much made me think of https://review.openstack.org/#/c/331369/20:59
patchbotpatch 331369 - swift - Always set swift processes to use UTC20:59
timburke...which taught me (as if i needed any reminding) just how terrible time is. it's the worst21:00
notmynametimburke: eternally21:01
onovy...and you should probably be running your servers in UTC anyway. // we are running servers in GMT+1 :)21:01
notmynameeveryone should always run their servers in UTC ;-)21:01
onovyand during summer in +2!21:01
notmynamefun!21:01
onovyCET/CEST time is funny21:02
timburkeyou can run your servers in whatever timezone you like. i don't think swift will think it's in the same tz though :P21:02
openstackgerritMerged openstack/swift master: Prevent traceback in object-server on client disconnect  https://review.openstack.org/41718721:02
notmynameonovy: so.... when time falls back in autumn every year, do you have an hour where writes overwrite older data?21:03
onovyno, xfs is using timestamp21:03
claygzaitcev: tdasilva: why don't you have a liberasurecode spec file in rpm-packaging!?  https://github.com/openstack/rpm-packaging/tree/master/openstack21:03
zaitcevclayg: Perhaps because I did not hear of rpm-packaging until now.21:04
onovytz should be used only when presenting time to people. why any software should depend on it...21:04
*** foexle has quit IRC21:05
claygonovy: holy crap!  this is you -> https://github.com/openstack/deb-swift/tree/debian/newton21:05
onovyclayg: :]21:05
patchbotError: Spurious "]".  You may want to quote your arguments with double quotes in order to prevent extra brackets from being evaluated as nested commands.21:05
claygonovy: do you have one for liberasurecode too?  because *apparently* those get published in a mirrored repository in infra that we could point at and get "newer than current system packages"21:06
claygzaitcev: nice ;)21:06
onovyhttps://github.com/openstack/deb-liberasurecode21:06
onovyhttps://github.com/openstack/deb-liberasurecode/tree/debian/newton21:07
claygzaitcev: so I'm told... that in addition to getting all up in openstack/releases with the yaml - we *also* want to get in the openstack/rpm-packaging - so that when the "automatically built newer than current system packages" rpm analog of this debian thing is in place in upstream infra - it will be *that* repo that is automatically with the build testing21:07
zaitcevhumm21:08
zaitcev[remote "origin"]21:08
zaitcev        url = ssh://zaitcev@pkgs.fedoraproject.org/rpms/liberasurecode21:08
zaitcevapparently we don't have a public git?21:08
onovyclayg: but i think you should use jessie-backports instead of infra-build debian packages21:09
claygonovy: I don't think I know what all those words mean - but I've been told we should try to make the infra packages have the versions we want and use them21:11
onovyi can't recomment it21:11
onovybecause zigo will not maintain debian packages anymore21:11
onovyand i will not use infra for my packages21:11
claygonovy: as part of a larger openstack initiative to maybe someday drive the non-python dependencies the same why upstream openstack is already doing for python dependencies21:11
claygonovy: i'm only talking about using openstack infra packages *in* openstack infra21:12
onovyyep, i know21:12
onovybut i will not build deb-liberasurecode inside infra21:12
openstackgerritMerged openstack/liberasurecode master: jerasure: plug memory leaks  https://review.openstack.org/43469621:13
clayghere's the thing that I think weirds out downstream - *I* (SwiftStack employee, Swift deployer/operator) do not *ever* want downstream packages - I am expected by my customers - but my clusters users to own my stuff soup to nuts - for stuff that I depend on critically to function - i built myself or go home21:13
clayglike when we found the liberasurecode bugs - it was fixed when I built my packages - I was every part in the critical path - it can be only that way21:13
*** gatuus has joined #openstack-swift21:13
claygonovy: interesting!  you'll have to explain why you would refuse to participate in the larger upstream effort to handle driving dependency resolution in upstream packaging and also better support downstream packagers21:14
claygtelling openstack they are doing it wrong is a favorite pass time of mine21:14
claygbut in this case monty and other infra and packaging people seem to be telling me things that make sense and would solve my problems21:15
onovyi'm fine to do packaging work inside infra21:15
claygoh, sorry - i misunderstood21:16
onovyBUT nobody have time to make it working21:16
onovyfor example i can make packages for ocata21:16
onovybecause nobody created branch for it21:16
onovyand nobody have time to do it21:16
claygwat?  that's stupid - why can't you just have access to do it21:16
claygor you're saying *you* don't have time?21:17
onovyyep, i don't have time to do this21:17
onovyi want to do packaging, i don't want to make infrastructure for it :)21:17
claygwhat about liberasurecode - it doesn't have "an ocata" release - it just has a latest version - which was released months ago?21:17
claygbut it's not the version you get in dsvm jobs in the gate21:17
onovybut that is different story21:18
onovyi'm talking about "building deb packages inside infra", which we are doing now21:18
onovyand you want to use products of this building inside gate21:18
onovybut i think i will move building deb packages from infra back to alioth (old fashion way)21:19
onovybecause building deb packages in infra is not working correctly21:19
onovyit's buggy and nobody have time to maintain it21:19
clayg*why* does deb-liberasurecode have a debian/newton branch?21:19
onovybecause "zigo"21:19
claygi feel like I have gone into a crazy town21:19
claygoic21:19
claygyeah the rpm-packaging guys here are telling me they maybe didn't love the solutions developed in the upstream infra instructure - but more tactical than strategic disagreements21:20
claygthey *like* the idea of doing upstream packaing21:21
onovybut that's is something different. upstream packaging is not using upstream infra for downstream packaging21:21
onovywhich is what debian do now21:21
onovybut upstream packaging is not ideal too. it's not about "please give me debian/ dir for new upstream release". i need to manage/patch old version of already eol products, etc.21:23
onovythat's the reason why almost all distros have own repos for packaging, split from upsream21:23
*** tobberydberg has joined #openstack-swift21:28
*** tobberydberg has quit IRC21:31
*** Jeffrey4l has quit IRC21:35
*** Jeffrey4l has joined #openstack-swift21:36
*** gatuus has quit IRC21:36
*** gatuus has joined #openstack-swift21:36
*** NM has quit IRC21:39
*** mmotiani_ has joined #openstack-swift21:44
openstackgerritOpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements  https://review.openstack.org/8873621:49
claygonovy: so *generally* I found the *idea* of upstream packaging kinda weird :\21:52
claygonovy: but openstack infra needs a way to manage it's dependencies - and they think dependency mangagement (for non-python dependencies at least) makes sense to do with *packages* - so they want to do upstream packaging?21:52
claygseems not insane, I don't think anyone expects downstream packagers to use upstream infra for build21:53
claygin the case of rpm-packaging at least downstream seemed willing to collaborate upstream on the creation of .spec files - and so far they have all just been doing the downstream building in 3rd party CI21:53
claygchange goes into rpm-packaging - downstream build infrastructure kicks off - and in at least SUSE's case it seems those build artifacts are published in some sort of "unstable" sort of repo?21:54
claygI don't really know21:54
claygI would like it if you and zaitcev could talk to woudar (Allison deb) and dirk (Dirk rpm) - because you both undoubtably understand *all* of this at a mucher deeper level than I could hope to21:55
*** jordanP has quit IRC21:56
zaitcevI don't see how any of this helps liberasurecode, which is relatively stable _and_ is included into Fedora. Just do "dnf install" and voila.21:57
zaitcevI mean, at infra machines.21:57
zaitcevNow, if they want to drag liberasurecode 1.4.0 into CentOS 7, it's possible too... The CentOS release team should be responsble for it.21:57
*** zul has quit IRC21:58
*** tonanhngo has quit IRC22:01
*** caiobrentano has quit IRC22:02
*** tonanhngo has joined #openstack-swift22:02
*** caiobrentano has joined #openstack-swift22:02
*** peterlisak has quit IRC22:04
*** peterlisak has joined #openstack-swift22:04
*** vint_bra has quit IRC22:09
*** mmotiani_ has quit IRC22:19
*** gatuus has quit IRC22:23
*** sams-gleb has quit IRC22:23
*** gatuus_ has joined #openstack-swift22:23
*** m_kazuhiro has quit IRC22:24
*** zaitcev has quit IRC22:25
*** mmotiani_ has joined #openstack-swift22:34
*** mmotiani_ has quit IRC22:39
*** gatuus_ has quit IRC22:53
*** caiobrentano has quit IRC22:54
*** gatuus has joined #openstack-swift22:59
*** kei_yama has joined #openstack-swift23:27
*** chsc has quit IRC23:33
*** catintheroof has joined #openstack-swift23:53

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!