Tuesday, 2016-12-20

*** siva_krish has quit IRC00:11
openstackgerritTim Burke proposed openstack/swift: Always set swift processes to use UTC  https://review.openstack.org/33136900:23
*** juzuluag has joined #openstack-swift00:24
*** tqtran has quit IRC00:24
*** catintheroof has quit IRC00:29
*** dja has joined #openstack-swift00:42
kota_good morning00:58
*** chlong has quit IRC00:59
mattoliveraukota_: morning01:09
kota_mattoliverau: o/01:10
*** Jeffrey4l has joined #openstack-swift01:23
*** juzuluag has quit IRC01:26
*** two_tired has joined #openstack-swift01:33
*** clu_ has quit IRC01:44
*** Jeffrey4l has quit IRC02:22
*** trananhkma has joined #openstack-swift02:22
*** jamielennox is now known as jamielennox|away03:08
*** Jeffrey4l has joined #openstack-swift03:14
*** jamielennox|away is now known as jamielennox03:22
*** winggundamth has quit IRC03:39
*** winggundamth has joined #openstack-swift03:41
*** dmorita has quit IRC03:44
*** dja has quit IRC04:04
*** dmorita has joined #openstack-swift04:05
*** dmorita has quit IRC04:10
openstackgerritpangliye proposed openstack/swift: Replace assertTrue with assertIs.  https://review.openstack.org/41273404:14
*** psachin has joined #openstack-swift04:17
*** trananhkma has quit IRC04:39
mahatic_good morning04:51
*** links has joined #openstack-swift04:52
two_tiredAnybody using jclouds? I'm wondering if the project still has momentum... and if anyone's found a way to do a parallel, multi-threaded upload.04:58
*** SkyRocknRoll has joined #openstack-swift04:58
mattoliveraumahatic_: o/05:16
mahatic_mattoliverau: o/05:16
mahatic_mattoliverau: not taking off yet for christmas?05:17
mattoliveraumahatic_: not yet, will work until Friday, but then have 2 weeks off :)05:18
mahatic_mattoliverau: nice! :)05:18
mattoliverauHow you doing? Christmas probably isn't a big holiday there in India like it is here :) so working through or having some holidays?05:19
mahatic_mattoliverau: yeah, that's true, no party mood around here (we instead have quite a few holidays/festivals spread around the year ;) )05:28
mahatic_so I'm gonna work through and take a few days off for our "regional new year" coming up in Jan :)05:29
*** ppai has joined #openstack-swift05:29
*** dja has joined #openstack-swift05:30
*** netinfinity has joined #openstack-swift05:31
netinfinityQuick question about swift-recon. It seems i can't get any stats because of this error:     return self.greenlet.switch() socket.timeout: timed out05:35
netinfinityNot a single swift-recon command doesn't work because of this... even with added timeout05:36
netinfinitywith added timeout i get this: httplib.BadStatusLine: ''05:36
netinfinityAnyone?05:40
*** two_tired has quit IRC05:44
kota_netinfinity: please check your recon get valid host info from your swift's ring.05:47
netinfinitywhat is the command for that?05:48
kota_netinfinity: recon uses swift ring (e.g. objct.ring.gz) for your cluster05:48
netinfinityswift-recon --validate-servers05:49
netinfinitygot method 405 method not allowed05:49
kota_hm05:50
kota_netinfinity: what's about `swift-recon -r --verbose`05:51
netinfinitycluster works btw, currently a lot of files is being uploaded into the storage05:51
netinfinityah yes05:52
netinfinityseems like one of the storage nodes is the problem05:52
netinfinityverbose shows the data for all nodes except one that got me that 40505:52
netinfinityshould I rebalance?05:53
kota_netinfinity: good info to know what happens05:53
kota_hmm05:53
kota_depends on your use case for the problematic node05:53
kota_if it's ok to purge that (no use), you should run rebalance05:54
netinfinitygot 6 storage nodes with approx 470 TB of storage05:54
netinfinityit's in use05:54
netinfinity:/05:54
kota_perhaps, the node doesn't get the config for recon middleware?05:55
kota_could you check if the recon middleware is in the pipeline in the problematic server's config.05:55
kota_?05:55
netinfinitygive me few minutes05:55
netinfinitydon't go away ;)05:55
kota_i have time to stay for now.05:56
netinfinityaccount-server.conf (container and object) have these lines: pipeline = healthcheck recon account(container,object)-server05:58
netinfinityso recon is in the middleware05:59
kota_it sounds correct setting.06:00
netinfinityor recon middleware is in pipeline..sorry, 6am here, first coffee :D06:01
kota_k, cofee always helps us ;-)06:01
netinfinity System information disabled due to load higher than 4.006:03
netinfinitythe node has a high load06:04
netinfinityprobably why recon is timing out06:04
kota_oh06:04
netinfinity27067 swift     20   0  129908  25220   2524 R 100.0  0.1   7834:56 rsync06:05
kota_it's a little strange because recon shouldn't result in 405, iirc06:05
netinfinity100% cpu usage by rsync06:05
netinfinityon the particular storage node06:06
kota_rsync? curious :/06:06
netinfinityreplication06:06
openstackgerritAndreas Jaeger proposed openstack/swift: Move bandit to pep8  https://review.openstack.org/36489206:07
kota_i know but it should not be cpu bouded.06:07
netinfinityreplication is06:09
netinfinitybut why this node alone is the question06:09
kota_probably there are various resons for strange behavior so that it needs to resolve tangled situation anyway...06:12
netinfinityseems like that node is stuck in replication06:13
kota_are you seeing sort of failure logs somewhere like syslog?06:13
netinfinitylet me check06:14
patchbot(let <variable> = <value> in <command>) -- Defines <variable> to be equal to <value> in the <command> and runs the <command>. '=' and 'in' can be omitted.06:14
netinfinitycontainer-replicator: Skipping sdat as it is not mounted06:16
netinfinitycontainer-server: ERROR __call__ error with PUT /sdn/33606:17
netinfinityetc06:17
kota_hmm it looks like not healty stats06:18
netinfinitylooks like locked db06:20
*** dja has quit IRC06:21
*** dja has joined #openstack-swift06:23
*** netinfinity_ has joined #openstack-swift06:29
netinfinity_I think I understand the problem now06:29
*** netinfinity has quit IRC06:33
*** dja has quit IRC06:33
*** dja has joined #openstack-swift06:35
*** dja has quit IRC06:44
*** dmorita has joined #openstack-swift06:45
*** dmorita has quit IRC06:49
*** dja has joined #openstack-swift06:51
*** mmuffinman has joined #openstack-swift06:54
*** rcernin has quit IRC06:59
*** netinfinity_ has quit IRC07:00
*** McMurlock1 has joined #openstack-swift07:07
openstackgerritAndreas Jaeger proposed openstack/swift: Move bandit to pep8  https://review.openstack.org/36489207:09
*** tesseract has joined #openstack-swift07:11
*** rcernin has joined #openstack-swift07:11
*** tesseract is now known as Guest1497207:11
*** rcernin has quit IRC07:26
*** sams-gleb has joined #openstack-swift07:28
*** jamielennox is now known as jamielennox|away07:28
openstackgerritAndreas Jaeger proposed openstack/swift: Move bandit to pep8  https://review.openstack.org/36489207:30
*** silor has joined #openstack-swift07:33
*** silor1 has joined #openstack-swift07:41
*** silor has quit IRC07:42
*** silor1 is now known as silor07:42
*** rcernin has joined #openstack-swift07:43
openstackgerritpangliye proposed openstack/swift: Replace assertTrue with assertEqual .  https://review.openstack.org/41273407:44
*** hseipp has joined #openstack-swift07:45
*** ChubYann has quit IRC08:07
*** mmuffinman has quit IRC08:18
*** pcaruana has joined #openstack-swift08:19
*** rledisez has joined #openstack-swift08:20
*** mr_xer_ has joined #openstack-swift08:22
*** mr_xer_ has quit IRC08:23
*** amoralej|off is now known as amoralej08:35
*** rcernin has quit IRC08:36
*** SkyRocknRoll has quit IRC08:54
*** geaaru has joined #openstack-swift08:55
*** rcernin has joined #openstack-swift08:58
openstackgerrityunfeng zhou proposed openstack/swift-bench: add CONTRIBUTING.rst  https://review.openstack.org/41296809:23
*** jordanP has joined #openstack-swift09:24
*** SkyRocknRoll has joined #openstack-swift09:28
*** mmuffinman has joined #openstack-swift09:31
*** manous has joined #openstack-swift09:42
*** dja has quit IRC09:48
*** bkopilov_ has quit IRC10:17
*** sams-gleb has quit IRC10:18
*** sams-gleb has joined #openstack-swift10:19
*** sams-gleb has quit IRC10:23
*** SkyRocknRoll has quit IRC10:45
*** dmorita has joined #openstack-swift10:45
*** manous has quit IRC10:47
*** dmorita has quit IRC10:50
*** mvk has quit IRC10:53
*** asettle has quit IRC10:53
*** sams-gleb has joined #openstack-swift10:58
*** asettle has joined #openstack-swift10:58
*** acoles_ is now known as acoles11:07
*** amac has quit IRC11:17
*** mvk has joined #openstack-swift11:26
*** dja has joined #openstack-swift12:32
*** catintheroof has joined #openstack-swift12:37
*** ppai has quit IRC12:49
*** chlong has joined #openstack-swift12:53
*** asettle has quit IRC12:59
*** pcaruana has quit IRC13:01
*** pcaruana has joined #openstack-swift13:06
*** vint_bra has joined #openstack-swift13:09
*** kei_yama has quit IRC13:17
*** dja has quit IRC13:25
*** amoralej is now known as amoralej|lunch13:43
*** amac has joined #openstack-swift13:45
openstackgerritAlistair Coles proposed openstack/swift: Make tempurl functional tests clean up account keys  https://review.openstack.org/34675213:54
*** kelepirci has joined #openstack-swift14:02
kelepircihello all14:03
kelepircijust a quick question. what is the implications of deleting "quarantined/objects" directory?14:03
*** catinthe_ has joined #openstack-swift14:06
*** catintheroof has quit IRC14:08
*** chlong has quit IRC14:16
acoleskelepirci: the object auditor process moves object file to quarantined dir if they are found to be corrupt. No swift process subsequently reads from quarantined, so they can be deleted. I'm not sure what "best practice" would be w.r.t. how long to keep them - the corrupt files should be replaced anyway after a replication cycle. They do provide some insight into failures if you have tools to find and audit quarantined dir14:16
acoless.14:16
*** asettle has joined #openstack-swift14:17
kelepircithx acles. it was a greate insight14:21
*** pcaruana has quit IRC14:23
kelepirci*acoles14:25
*** geaaru has quit IRC14:28
acoleskelepirci: it may be worth checking that your cluster does still have good copies of quarantined objects before deleting - if the swift.conf hash prefix/suffix options get messed up then you may end up with a lot of quarantined objects like this bug describes https://bugzilla.redhat.com/show_bug.cgi?id=1321507 and in that case the bug author wanted to restore them14:28
openstackbugzilla.redhat.com bug 1321507 in openstack-swift "Recover quarantined objects" [Medium,Closed: notabug] - Assigned to zaitcev14:28
acolesbut note that swift does not provide any such restore process itself.14:29
*** dmorita has joined #openstack-swift14:29
*** links has quit IRC14:32
*** dmorita has quit IRC14:34
*** pcaruana has joined #openstack-swift14:37
*** psachin has quit IRC14:42
*** _JZ_ has joined #openstack-swift14:42
*** sams-gleb has quit IRC14:57
*** sams-gleb has joined #openstack-swift14:57
*** sams-gleb has quit IRC15:02
*** amoralej|lunch is now known as amoralej15:02
kelepircino I do not need to restore them.15:05
*** acoles is now known as acoles_15:08
*** acoles_ is now known as acoles15:15
*** sams-gleb has joined #openstack-swift15:16
*** vint_bra has quit IRC15:19
*** vint_bra has joined #openstack-swift15:26
*** AJaeger has joined #openstack-swift15:43
AJaegerswift cores, could you review https://review.openstack.org/364892 to merge bandit into the normal linter job - and thus follow best practices -, please?15:43
patchbotpatch 364892 - swift - Move bandit to pep815:43
openstackgerritMathias Bjoerkqvist proposed openstack/swift: Storing encryption root secret in Barbican  https://review.openstack.org/36487815:44
*** cbartz has joined #openstack-swift15:49
*** chlong has joined #openstack-swift15:53
*** tuan_luong has joined #openstack-swift16:01
*** chlong has quit IRC16:01
*** bkopilov has quit IRC16:09
*** chlong has joined #openstack-swift16:16
*** McMurlock1 has quit IRC16:17
*** bkopilov has joined #openstack-swift16:20
*** mvk has quit IRC16:22
*** tuan_luong has quit IRC16:26
*** chlong has quit IRC16:28
*** chsc has joined #openstack-swift16:31
*** chsc has joined #openstack-swift16:31
*** catintheroof has joined #openstack-swift16:39
*** catinthe_ has quit IRC16:42
*** chlong has joined #openstack-swift16:42
*** amac has quit IRC16:43
*** chsc has quit IRC16:45
*** amac has joined #openstack-swift16:48
*** asettle has quit IRC17:01
*** bkopilov_ has joined #openstack-swift17:01
*** jordanP has quit IRC17:02
*** bkopilov has quit IRC17:02
*** Guest14972 has quit IRC17:03
*** _JZ_ has quit IRC17:10
*** hseipp has quit IRC17:27
*** oshritf has quit IRC17:29
*** tqtran has joined #openstack-swift17:30
notmynamegood morning17:40
*** garyj has joined #openstack-swift17:41
*** dmorita has joined #openstack-swift17:44
*** garyj has quit IRC17:46
*** rledisez has quit IRC17:47
*** pcaruana has quit IRC17:59
*** acoles is now known as acoles_18:01
*** catinthe_ has joined #openstack-swift18:03
timburkegood morning18:03
*** catintheroof has quit IRC18:06
*** chsc has joined #openstack-swift18:08
*** chsc has joined #openstack-swift18:08
AJaegernotmyname, cschwede, could you review 364892 ,  please?18:13
*** klrmn has joined #openstack-swift18:22
notmynamepatch 36489218:23
patchbothttps://review.openstack.org/#/c/364892/ - swift - Move bandit to pep818:23
AJaegernotmyname: that follows best practice to run bandit as part of our linters job instead of a separate job.18:24
notmynameAJaeger: are the appropriate changes in the upstream jobs made? or what's the plan with the separate bandit job vs pep8 job?18:25
notmynameand what about the comment about not needing the bandit.yaml file any more? what's the recommended practice there?18:26
AJaegernotmyname: see the needed-by, I proposed a job for infra to remove the separate bandit job18:26
AJaegernotmyname: that will stop running the job on older branches but it's a non-voting job, I doubt you need it there - or you could backport the change18:26
notmynamenah18:26
AJaegerinitially every bandit invocatoin needed a bandit.yaml. Now it's optional.18:27
notmynamewhat'st he default it uses if the yaml file isn't there?18:27
AJaegerSo, you could remove it - but removing the file, the job fails.18:27
AJaegerbetter ask the security team about those details - travis is on the review18:27
notmynamebecause the job defines passing it in?18:27
AJaegerthe tox line includes it18:28
AJaeger"bandit -c bandit.yaml -r swift -n 5" - see the -c bandit.yaml18:28
notmynameright. that's what I meant18:28
AJaegera further cleanup would be removing bandit.yaml (or shrinking it) and fixing/whitelisting the problems18:28
notmynameok,thanks for the info18:29
*** rcernin has quit IRC18:29
*** mmuffinman has quit IRC18:36
*** amoralej is now known as amoralej|off18:40
*** asettle has joined #openstack-swift18:46
openstackgerritTim Burke proposed openstack/swift: Make tempurl functional tests clean up account keys  https://review.openstack.org/34675218:46
*** asettle has quit IRC18:54
*** mvk has joined #openstack-swift18:54
*** asettle has joined #openstack-swift18:55
*** asettle has quit IRC18:56
*** chlong has quit IRC18:56
*** asettle has joined #openstack-swift18:58
*** dmorita has quit IRC19:02
*** dmorita has joined #openstack-swift19:06
*** diogogmt has joined #openstack-swift19:10
*** cnf has quit IRC19:11
*** chlong has joined #openstack-swift19:13
*** hseipp has joined #openstack-swift19:17
*** silor has quit IRC19:19
*** cnf has joined #openstack-swift19:22
*** cnf has quit IRC19:30
*** catintheroof has joined #openstack-swift19:37
*** catinthe_ has quit IRC19:40
*** asettle has quit IRC19:40
*** asettle has joined #openstack-swift19:41
*** _JZ_ has joined #openstack-swift19:41
openstackgerritMerged openstack/swift: Move bandit to pep8  https://review.openstack.org/36489219:45
*** asettle has quit IRC19:45
*** AJaeger has left #openstack-swift19:46
*** oshritf has joined #openstack-swift19:46
*** oshritf has quit IRC19:56
*** oshritf has joined #openstack-swift19:56
*** cnf has joined #openstack-swift19:59
*** amac is now known as stradling20:01
*** stradling has quit IRC20:02
*** ChubYann has joined #openstack-swift20:08
*** hseipp has quit IRC20:10
*** chlong has quit IRC20:15
*** chlong has joined #openstack-swift20:27
*** jamielennox|away is now known as jamielennox20:44
*** catintheroof has quit IRC20:47
tdasilvaclayg, onovy: just FYI...currently working with infra team on how to merge the two repos and preserve history. For now I've created a throw-away repo to test the final result and this is what I have so far: https://github.com/thiagodasilva/libec20:49
tdasilvamind you this would probably be just the initial commit and then we could change as we see fit20:49
tdasilvaclayg, onovy: i'm taking off from the office now and will be back in about 30-40 min20:50
claygtdasilva: that's pretty cool - i didn't know you could combine the git history of pyeclib into the liberasure repo like that - pretty cool!20:58
*** dmorita has quit IRC21:04
openstackgerritMerged openstack/swift: Make tempurl functional tests clean up account keys  https://review.openstack.org/34675221:06
timburkei mean really, what fun is a singly-rooted DAG?21:07
claygtimburke: pfft - no fun i wanna have21:12
*** dmorita has joined #openstack-swift21:19
*** cbartz has quit IRC21:19
*** dmorita has quit IRC21:20
*** mrda is now known as michaeld21:22
*** michaeld is now known as mrda21:24
*** vint_bra has quit IRC21:33
*** oshritf has quit IRC21:33
*** Jeffrey4l has quit IRC21:35
*** dmorita has joined #openstack-swift21:39
tdasilvaheh21:42
*** chlong has quit IRC21:46
*** Jeffrey4l has joined #openstack-swift21:47
*** asettle has joined #openstack-swift21:50
mattoliverauMorning21:55
*** sams-gleb has quit IRC21:57
*** sams-gleb has joined #openstack-swift21:57
notmynametimburke: fun fact: swift itself doesn't have just one root in the git history22:00
notmyname001407b969bc12d48bd7f10960f84f519bb19111 and fe067413b1fb61cdfba97d543189c5a01230b45c22:01
notmynamemerged in 3a7f80814b0eb7c220cfbf82e1c5812e29d56c5522:02
*** sams-gleb has quit IRC22:02
notmynamehttp://d.not.mn/early_swift_history.png22:04
timburkenotmyname: 659382e466cb11df6f2c9d81e3e5f0d1bc2a4c5c, surely? or maybe 2ee9b837b5a1e13681ca9359138451719f8641dd?22:04
*** asettle has quit IRC22:05
timburkeyeah, that second one from July 7! *that* looks like a good merge commit to declare22:05
*** asettle has joined #openstack-swift22:05
*** nikivi has joined #openstack-swift22:07
tdasilvalol, make xfs-based tests optional22:07
tdasilvafeels like deja-vu22:08
*** nikivi has quit IRC22:08
*** asettle has quit IRC22:10
*** vint_bra has joined #openstack-swift22:26
*** vinsh has quit IRC22:35
*** diogogmt has quit IRC22:50
*** chlong has joined #openstack-swift22:57
*** dja has joined #openstack-swift23:37
*** chsc has quit IRC23:42
*** diogogmt has joined #openstack-swift23:42
*** cebruns has quit IRC23:44
*** kei_yama has joined #openstack-swift23:44
*** cebruns has joined #openstack-swift23:51
*** diogogmt has quit IRC23:56
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements  https://review.openstack.org/8873623:56
*** cebruns has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!