Friday, 2017-09-15

*** itlinux has joined #openstack-swift00:11
*** gyee has joined #openstack-swift00:14
*** gyee has quit IRC00:15
*** Sukhdev has joined #openstack-swift00:22
timburketorgomatic: so that was walking the entire index then? to give us an idea of how long it would take to assign pivots?00:36
*** itlinux has quit IRC00:51
*** tovin07_ has joined #openstack-swift00:56
*** lucasxu has joined #openstack-swift01:16
*** lucasxu has quit IRC01:19
*** Sukhdev has quit IRC01:21
*** itlinux has joined #openstack-swift02:36
itlinuxhello all.. I got Tripleo installed but swift is consuming most of my CPU .. any tips? I could turn off swift and move the images out of swift any tips on that.. Thanks03:01
itlinuxI have a flash storage in case03:01
*** links has joined #openstack-swift03:44
*** kong has quit IRC03:48
*** itlinux has quit IRC03:49
*** Sukhdev has joined #openstack-swift04:29
*** psachin has joined #openstack-swift05:11
*** Sukhdev has quit IRC05:41
*** skudlik has joined #openstack-swift05:52
*** rcernin has joined #openstack-swift06:07
*** SkyRocknRoll has joined #openstack-swift06:17
*** SkyRocknRoll has quit IRC06:19
*** klrmn has quit IRC06:31
openstackgerritTim Burke proposed openstack/swift master: Use check_drive consistently  https://review.openstack.org/50015806:35
openstackgerritTim Burke proposed openstack/swift master: Differentiate between a drive that's not mounted vs. not a dir more  https://review.openstack.org/50434106:35
*** pcaruana has joined #openstack-swift06:45
*** itlinux has joined #openstack-swift06:46
openstackgerritTim Burke proposed openstack/swift master: doc migration: update the doc link address[2/3]  https://review.openstack.org/50077606:47
openstackgerritMerged openstack/swift master: Always require device dir for containers  https://review.openstack.org/45807006:51
openstackgerritTim Burke proposed openstack/swift master: Move listing formatting out to proxy middleware  https://review.openstack.org/44939406:55
*** geaaru has joined #openstack-swift06:57
openstackgerritTim Burke proposed openstack/swift master: Respond 400 Bad Request when Accept headers fail to parse  https://review.openstack.org/50284507:07
*** skudlik has left #openstack-swift07:07
*** tesseract has joined #openstack-swift07:27
*** itlinux has quit IRC08:15
*** joeljwright has joined #openstack-swift08:27
*** ChanServ sets mode: +v joeljwright08:27
*** skudlik has joined #openstack-swift08:37
*** skudlik has left #openstack-swift08:39
*** psachin has quit IRC09:05
*** psachin has joined #openstack-swift09:05
*** xlucas has joined #openstack-swift09:19
*** xlucas has left #openstack-swift09:21
*** xlucas has joined #openstack-swift09:21
*** xlucas has left #openstack-swift09:22
*** silor has joined #openstack-swift09:34
*** jlutran has joined #openstack-swift09:37
*** ma9_1 has joined #openstack-swift09:40
*** esnyder has quit IRC10:10
*** tovin07_ has quit IRC10:14
*** jlutran has left #openstack-swift10:17
*** ma9_1 has left #openstack-swift10:30
*** skudlik has joined #openstack-swift10:43
*** skudlik has left #openstack-swift10:43
*** d0ugal has quit IRC11:04
*** silor has quit IRC11:05
openstackgerritMerged openstack/swift master: doc migration: update the doc link address[2/3]  https://review.openstack.org/50077611:09
*** mat128 has joined #openstack-swift11:31
*** mat128 has quit IRC11:33
*** mat128 has joined #openstack-swift11:35
*** psachin has quit IRC11:38
*** links has quit IRC11:41
*** mat128 has quit IRC11:46
*** mat128 has joined #openstack-swift11:52
*** links has joined #openstack-swift11:54
*** d0ugal has joined #openstack-swift12:14
*** catintheroof has joined #openstack-swift12:22
*** pit has joined #openstack-swift12:28
*** pit has quit IRC12:34
*** links has quit IRC12:48
*** Dinesh_Bhor has quit IRC12:58
*** chlong has joined #openstack-swift13:04
*** links has joined #openstack-swift13:07
*** cbartz has quit IRC13:55
*** mwheckmann has joined #openstack-swift14:05
*** saint_ has joined #openstack-swift14:06
*** links has quit IRC14:10
*** gyee has joined #openstack-swift14:40
*** klrmn has joined #openstack-swift14:58
*** aagrawal has quit IRC15:10
*** Sukhdev has joined #openstack-swift15:21
*** pcaruana has quit IRC15:24
torgomatictimburke: yes, that was the whole thing15:35
timburkegood morning15:37
timburkeso, long enough that it'd be nice to be dropping stats (so we can see that it isn't wedged), but probably not so long that we really need to start diverting updates immediately...15:39
*** rcernin has quit IRC15:44
*** EmilienM has quit IRC15:55
*** EmilienM has joined #openstack-swift15:57
*** itlinux has joined #openstack-swift16:02
*** links has joined #openstack-swift16:09
*** klrmn has quit IRC16:13
*** m_kazuhiro has joined #openstack-swift16:16
openstackgerritKota Tsuyuzaki proposed openstack/swift master: Remove all post_as_copy related code and configes  https://review.openstack.org/50406916:17
*** joeljwright has quit IRC16:24
*** SkyRocknRoll has joined #openstack-swift16:33
timburkeeasy merge! https://review.openstack.org/#/c/502893/16:47
patchbotpatch 502893 - swift - Add assertion about last-modified to object post test16:47
openstackgerritClay Gerrard proposed openstack/swift master: Accept a trade off of dispersion for balance  https://review.openstack.org/50315216:49
openstackgerritClay Gerrard proposed openstack/swift master: Accept a trade off of dispersion for balance  https://review.openstack.org/50315216:50
openstackgerritKota Tsuyuzaki proposed openstack/swift master: Remove all post_as_copy related code and configes  https://review.openstack.org/50406916:59
kota_make the patch 504069 to be back to patch set 2 that emits warning log17:04
patchbothttps://review.openstack.org/#/c/504069/ - swift - Remove all post_as_copy related code and configes17:04
*** chsc has joined #openstack-swift17:05
*** chsc has joined #openstack-swift17:05
*** tesseract has quit IRC17:06
*** klrmn has joined #openstack-swift17:09
*** SkyRocknRoll has quit IRC17:12
*** vint_bra has joined #openstack-swift17:15
*** vint_bra has quit IRC17:19
*** abhitechie has joined #openstack-swift17:20
*** SkyRocknRoll has joined #openstack-swift17:25
*** vint_bra has joined #openstack-swift17:29
openstackgerritSamuel Merritt proposed openstack/swift master: Shorten typical proxy pipeline.  https://review.openstack.org/50447217:34
torgomatictimburke: ^^17:35
notmynamerledisez: ^^17:36
timburketorgomatic: https://trello.com/b/z6oKKI4Q/container-sharding17:37
*** Sukhdev has quit IRC17:41
notmynamehttp://d.not.mn/swift_team_denver_ptg.jpeg17:43
*** links has quit IRC17:48
openstackgerritSamuel Merritt proposed openstack/swift master: Shorten typical proxy pipeline.  https://review.openstack.org/50447217:53
kota_timburke: https://review.openstack.org/50447917:59
patchbotpatch 504479 - swift3 - Change log updates for version 1.1217:59
*** Sukhdev has joined #openstack-swift18:07
*** Sukhdev has quit IRC18:07
openstackgerritKazuhiro MIYAHARA proposed openstack/swift master: WIP: Fix location header to be relative in 'leave_relative_location' environment  https://review.openstack.org/50448418:16
itlinuxhello all.. I have installed TripleO RDO but the controllers are getting abused by the swift processors .. any tips18:18
*** m_kazuhiro has quit IRC18:19
timburketorgomatic: huh... http://paste.openstack.org/show/621214/18:20
*** m_kazuhiro has joined #openstack-swift18:26
*** honga has joined #openstack-swift18:28
clarkbnotmyname: https://review.openstack.org/#/c/471057/4 and https://review.openstack.org/29677118:37
patchbotpatch 471057 - swift - Func test hacks to work under against apache218:37
patchbotpatch 296771 - openstack-infra/devstack-gate - Enable tlsproxy by default18:37
*** geaaru has quit IRC18:49
notmynameitlinux: oh hi. I saw your question earlier, but it seems we're all on and off at the wrong times19:08
itlinuxahh..19:08
itlinuxno worries..19:08
itlinuxany tips?19:09
notmynameitlinux: might be a good idea to talk with cschewede (who's in the room with me right now and promises that he'd love to talk to you next week)19:09
notmynamehe's got some experience with deploying swift+tripleo19:09
itlinuxsounds good!19:09
notmynameor swift on tripleo19:09
itlinuxvery good thanks..19:09
*** hseipp has joined #openstack-swift19:10
*** hseipp has quit IRC19:10
*** hseipp has joined #openstack-swift19:11
notmynameitlinux: oh, first thing he said earlier was maybe there's a misconfiguration with gnocchi? there's a way it can be configured to store data in swift. and swift has ceilometer installed. which sends data to gnocchi. which gets put in swift ...19:11
notmynameI don't know details, but it may be worth checking19:11
itlinuxahh yes I do have gnocchi and ceilometer19:12
itlinuxif he can pass some options to check then I will be happy to do it.. otherwise we can talk next week.19:13
torgomatictimburke: heh, that's hilarious. Fortunately, it seems like it's only gonna happen at process exit.19:13
notmynamehttps://postfacto.io/retros/swift19:23
itlinuxahh you are in Denver at the PTG meeting..19:29
notmynametdasilva: http://doodle.com19:40
*** mat128 has quit IRC19:41
*** SkyRocknRoll has quit IRC19:51
timburketorgomatic: hrm. also, http://paste.openstack.org/show/621220/ during unittests...19:53
timburke(other one was during probe tests, which isn't too surprising)19:54
torgomatictimburke: no idea about that second one; the stack doesn't have anything useful in it19:57
torgomaticcertainly suspicious though19:57
notmyname"For all XFS users out there, start planning a kernel upgrade in the near future" http://seclists.org/oss-sec/2017/q3/43620:05
openstackgerritKota Tsuyuzaki proposed openstack/swift master: Make gate keeper to save relative location header path  https://review.openstack.org/50450720:11
*** itlinux has quit IRC20:12
kota_m_kazuhiro, timburke: patch 504507 is the challenge to resolve a gate issue with dvsm for symlink20:18
patchbothttps://review.openstack.org/#/c/504507/ - swift - Make gate keeper to save relative location header ...20:18
m_kazuhirokota_: Thanks! I will check it.20:19
*** itlinux has joined #openstack-swift20:19
*** catintheroof has quit IRC21:26
tdasilvam_kazuhiro: https://review.openstack.org/#/c/449394/21:36
patchbotpatch 449394 - swift - Move listing formatting out to proxy middleware21:36
m_kazuhirotdasilva: Thanks!21:36
*** guimaluf has joined #openstack-swift21:50
*** saint_ has quit IRC21:51
guimalufhi guys, I've a swift cluster in production and one of my files are in 3 handoffs server but I can't list, download or do anything through swift API, I keep getting 404 error21:52
notmynameguimaluf: that's sounds unfortunate21:53
guimalufnotmyname, what may be happening?21:53
notmynameguimaluf: a 404 sounds like something I'd expect in that situation21:53
guimalufbut the file is in swift storages21:53
notmynameso there's a few things that may be going on21:53
guimalufplease clarify me! :)21:53
guimalufnotmyname, what may be happening?21:54
notmynameok, I was looking for some settings21:54
guimalufI though that if a file is present on handoffs swift should point to it, right?21:55
guimalufI could see the 3 copies in handoff servers21:55
notmynamefirst, tell me about your cluster. replicas or ec? how many servers? how many drives per server?21:55
notmynamewhat version of swift are you running?21:55
guimalufreplica. 2 proxys, 8 storages, 4 drivers per server, two regions.21:56
notmynameok21:56
notmynamehow many replicas?21:57
guimalufkilo version21:57
guimaluf3 replicas21:57
notmynamekilo? ie 2.3.0 released April 30, 2015?21:57
*** vint_bra has quit IRC21:57
notmynameok21:57
guimalufswift                                2.2.2-0ubuntu1.3~cloud121:58
guimaluf2.221:58
notmyname2.2.2 was released on feb 2 (2/2) ;-)21:58
notmynameI liked it when that lined up21:58
guimalufhahahahhahaha21:58
guimalufgood to know :P21:58
notmynameFWIW https://wiki.openstack.org/wiki/Swift/version_map21:58
guimalufI hope that mystic forces of this alignment does'nt be the cause of this!21:59
notmynameok, so you're running a very old version of swift, so some things may be slightly different from current docs and config files. but let's try anyway21:59
notmynamelet me dig into git history to see when this config variable was introduced...21:59
guimalufnotmyname, what kind of problem may be happning?22:00
notmynameok, it's there22:00
notmynamein the proxy server, you've got a config variable called "request_node_count"22:00
notmynamewell, do you have it set? and if so, what is it set to?22:00
guimalufi'll check22:01
guimalufthis setting is not present22:01
notmynameok22:01
notmynamenot a problem22:01
notmynamebasically, the proxy server looks up the object name in the request against the current ring to get drives. it then sends requests to the object servers that have those drives. if it gets all 404s, then it starts asking more object servers until it ask asked a total of "request_node_count" servers22:02
notmynameif it doesn't find an object, it returns 404 to the client22:02
notmynamethe default for request_node_count is "2 * replicas" so in your case that's 622:03
notmynameso there's 2 things I want to talk with you about: (1) how to get your cluster in a better shape so you can serve requests and (2) how to not let this happen again22:03
guimalufhahahah the object is far beyond the 6th handoff22:03
guimalufhahahaha22:03
guimalufnotmyname, the immediate fix would be set this to STORAGE_COUNTs - REPLICAS, right?22:04
guimalufso it would search in all servers22:04
notmynamehonestly, I think that's a terrible idea22:04
guimalufwhy?22:05
notmynamebut it will work, and it may be the fastest way22:05
notmynameit will make all requests that would otherwise 404 talk to every drive and that will be expensive22:06
notmynameI'm assuming "STORAGE_COUNT" is the number of drives you have in the system?22:06
guimalufbut is better then get 404 when reaching a file22:06
guimalufno, storage count = servers counts22:06
guimalufcount*22:06
notmynameoh22:06
notmynameyou you'd set it to 8 (for your 8 servers) and then that means you'd look in the first 5 handoffs (after the 3 primaries)22:07
guimalufyou're right... it should be the drivers number... cause every driver is present in handoff servers22:07
notmynamewhich means you could still get 404s (I don't know how bad the cluster is)22:07
guimalufexactly22:08
guimalufso, "how to get your cluster in a better shape so you can serve requests" ?22:08
guimaluf:D22:08
notmynamebut that simply papers over the problem and introduces a lot of work in the cluster22:08
notmynameie imagine if someone simply reuqests random uuids. they'd all 404, but it would cause the cluster a lot of work to talk to every disk22:08
*** vint_bra has joined #openstack-swift22:08
notmynameok, so to get it healthy...22:09
guimalufI would like to force object from handoff to its acctually server22:09
notmynameyou need to make sure that replication is running and completing as quickly as possible22:09
notmynamethe replication process is what does that22:09
guimalufbut why this object remain on handoff servers?22:09
notmynamethere should be an object-replicator process running on every server (potentially more than one, but I don't remember the details in 2.2.2--we've made a *lot* of improvements since then)22:10
notmynameit won't remain on handoff servers if replication is running22:10
notmynameI'm assuming you have the same object ring file on each of the servers22:10
guimalufnotmyname, yes, same ring.22:11
notmynameok, good22:11
guimaluf[2017-09-15 19:11:21] Checking ring md5sums22:11
guimaluf8/8 hosts matched, 0 error[s] while checking hosts.22:11
notmynamewhat are you using to monitor the cluster22:11
notmynameoh. that's swift-recon output :-)22:11
guimalufnotmyname, tail -f on swift.log :(22:11
notmynamewhat's your recent replication cycle time?22:11
guimaluf[2017-09-15 19:11:56] Checking on replication22:12
guimaluf[replication_time] low: 0, high: 81399, avg: 34650.6, total: 277204, Failed: 0.0%, no_result: 0, reported: 822:12
guimalufOldest completion was 2017-01-17 15:52:49 (241 days ago) by 192.168.55.109:6000.22:12
guimalufMost recent completion was 2017-04-18 09:56:12 (150 days ago) by 192.168.55.102:6000.22:12
notmynamesoo... yeah, that doens't look very good22:12
notmynameis the replication process actually running now?22:13
guimalufwow22:13
guimalufnow I realized those infos!22:13
*** vint_bra has quit IRC22:14
guimalufnotmyname, how can I check this?22:14
notmynameps -ef? top?22:14
notmynameswift-init object-replicator status22:14
guimalufswift-*-replicator is running22:14
notmynameswift-oldies -a022:15
notmyname`swift-oldies -a 0`22:15
notmynameI wonder if they're hung processes. could be interesting to strace them to see what's happening22:15
*** vint_bra has joined #openstack-swift22:15
guimaluf5789 26528 object-replicator /etc/swift/object-server.conf22:16
notmynameon one of the servers, kill it and restart it22:16
guimalufWOW22:18
guimalufomg!22:18
notmynameis that good? did you just get fired?22:18
guimalufhahahahahhahahaha22:19
guimalufno, but I can see lots of things happening on logs22:19
acolesgood things I hope22:19
guimalufindeed the proccess was hung for some reason22:19
guimalufI'll repeat the proccess22:19
guimalufswift-init stop object-replicator isn't killing it22:20
notmyname`kill -9`?22:20
notmynamehow full are your disks?22:20
guimaluf1%22:20
guimaluf:)22:20
notmynameyou have 100TB drives??!?22:21
*** vint_bra has quit IRC22:21
guimalufhahahahah no :)22:21
notmynameok, so replication running will get you into a better situation22:21
notmynamesince your drives aren't very full, it shouldn't take too long for that to settle down22:22
guimalufwow, that's was awesome! I really have no word to thanks your help, notmyname . You were really kind and helpfull! :)22:22
notmyname(assuming some things about your hardware)22:23
guimalufI'll cross my finger and let the replication happens22:23
notmynameoh, wait. we aren't done22:23
guimalufno?22:23
guimalufomg!22:23
guimalufahhahaha22:23
notmynameI don't want you to be in this situation again22:23
guimalufme neither!22:24
notmynameso, I know this is cliche, but you really should upgrade to the most recent version of swift, if at all possible22:24
notmynameyou can upgrade a live cluster with no downtime and do it all at once22:24
notmynamea rough guide is at https://www.swiftstack.com/blog/2013/12/20/upgrade-openstack-swift-no-downtime/22:25
guimalufnotmyname, yes, I know I should. I did once from havana, I think, but things are slow on this environment22:25
notmynameok, beyond that, you should make sure you have some good ops tools to monitor22:26
guimalufnotmyname, yes. I know this. but I'm the only one taking care of this cluster22:27
notmynameI was thinking of some other things (like making sure you don't push new rings befor a replication cycle has finished)22:27
guimalufI would love to do all the things I should do :( but I just can't get time enough22:27
guimalufnotmyname, I was not aware of this replication cycle22:27
guimalufincredible all files was available for so long time22:28
notmynamesure, but when you aren't monitoring, you end up in situations where replication isn't rnning for 6+ months ;-)22:28
guimaluf5 months! ;)22:29
guimalufhahaha22:29
notmyname241 days for the longest22:29
guimalufoh crap :(22:29
guimalufI have another issue. my rsync mask is messy... 644 for directories :/22:30
notmynameswift-recon is available, and there's swift-recon-cron that can be run periodically. might not be a bad idea to spend a few minutes to set that up to run and email you every day22:30
guimalufI'll do it. I'll take care of this cluster more closely22:31
notmyname(ie run `swift-recon --all` periodically. swift-recon-cron can populate what swift-recon reports on)22:31
guimalufI'm happy that swift is really robust... even with all this it worked very well22:32
notmynameyeah, that's great to hear :-)22:32
guimalufit is really good to have people like you in the community!22:33
guimalufI would never realize this by myself22:33
notmynameI'm glad you stopped by to ask. there's a lot of friendly swift experts in here22:34
guimalufnotmyname, I was neglecting this error for so long time22:45
guimalufError: /Stage[main]/Swift::Storage::Object/Swift::Storage::Generic[object]/Service[swift-object-replicator]: Failed to call refresh: Could not stop Service[swift-object-replicator]: Execution of '/sbin/stop swift-object-replicator' returned 1: stop: Job failed while stopping22:45
guimalufError: /Stage[main]/Swift::Storage::Object/Swift::Storage::Generic[object]/Service[swift-object-replicator]: Could not stop Service[swift-object-replicator]: Execution of '/sbin/stop swift-object-replicator' returned 1: stop: Job failed while stopping22:45
guimaluf:(22:45
notmynameguimaluf: can you share anything about the kind of data you're storing in swift?22:58
notmynameI'm always curious to hear how people are using it22:58
* notmyname may lose internet availability any time in the next 15 minutes22:59
guimalufnotmyname, I integrate swift with owncloud and offer a storage service for brazilian researchers. :)23:01
notmynamethat's cool!23:01
guimalufin general it is personal files, as photos, documents, and in a closer future web conferece videos23:02
notmynameis owncloud working well for you? I've considered using it at home23:04
notmyname(owncloud + swift)23:04
notmynamewifi is going down. good luck guimaluf23:08
guimalufnotmyname, yes it is. actually just now our users are really using it23:08
guimaluf:)23:08
guimalufnotmyname, no words to thank you :)23:08
*** hseipp has quit IRC23:15
*** m_kazuhiro has quit IRC23:28
openstackgerritKota Tsuyuzaki proposed openstack/swift master: Make gate keeper to save relative location header path  https://review.openstack.org/50450723:36
*** gyee has quit IRC23:42

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!