Sunday, 2015-08-16

*** haomaiwang has quit IRC00:01
*** 16WAAJRL0 has joined #openstack-swift00:02
*** ho has joined #openstack-swift00:17
*** ho_ has quit IRC00:17
*** lcurtis has joined #openstack-swift00:23
*** 16WAAJRL0 has quit IRC01:01
*** haomaiwang has joined #openstack-swift01:02
*** ho_ has joined #openstack-swift01:07
*** ho has quit IRC01:07
*** lcurtis has quit IRC01:09
*** lcurtis has joined #openstack-swift01:14
*** bapalm_ has joined #openstack-swift01:22
*** bapalm_ has quit IRC01:25
*** bapalm_ has joined #openstack-swift01:25
*** changbl has joined #openstack-swift01:36
*** ho has joined #openstack-swift01:43
*** ho_ has quit IRC01:43
*** haomaiwang has quit IRC01:51
*** bapalm_ has quit IRC02:04
*** ho has quit IRC02:12
*** ho has joined #openstack-swift02:12
openstackgerritKota Tsuyuzaki proposed openstack/swift: Support last modified on listing containers  https://review.openstack.org/19863402:28
*** hrou has joined #openstack-swift02:29
portantepeluse: so ... how is phoenix doing these days?02:34
portantejust read an article on sinking Phoenix, is it going to rise from the ashes ?  ;)02:35
*** dbarman has joined #openstack-swift02:51
dbarmananyone up for a swift problem question?02:51
*** logan2 has quit IRC02:57
*** ho has quit IRC02:59
*** ho has joined #openstack-swift02:59
*** dbarman has quit IRC03:04
openstackgerritKota Tsuyuzaki proposed openstack/swift-specs: Add global ec cluster improvement  https://review.openstack.org/20944703:14
*** ho_ has joined #openstack-swift03:21
*** ho has quit IRC03:21
*** SkyRocknRoll has joined #openstack-swift03:27
*** jrichli has joined #openstack-swift03:29
*** hezhiqiang has joined #openstack-swift03:29
*** chlong has joined #openstack-swift03:29
*** logan2 has joined #openstack-swift03:37
*** nadeem has joined #openstack-swift04:06
*** haigang has joined #openstack-swift04:13
*** haomaiwang has joined #openstack-swift04:26
*** hezhiqiang has quit IRC04:42
*** jrichli has quit IRC04:45
*** haomaiwang has quit IRC05:01
*** haomaiwa_ has joined #openstack-swift05:02
*** hezhiqiang has joined #openstack-swift05:42
*** hezhiqiang has quit IRC05:47
*** zul has quit IRC05:59
*** haomaiwa_ has quit IRC06:01
*** haomaiwang has joined #openstack-swift06:01
*** links has joined #openstack-swift06:08
*** jaison has joined #openstack-swift06:08
*** jaison has quit IRC06:10
openstackgerritVictor Stinner proposed openstack/swift: Fix tox -e py34  https://review.openstack.org/19903406:26
openstackgerritVictor Stinner proposed openstack/swift: pep8: Don't format using self.__dict__  https://review.openstack.org/20596606:32
openstackgerritVictor Stinner proposed openstack/swift: Fix hacking H232 warnings (octal)  https://review.openstack.org/20614906:40
*** hrou has quit IRC06:45
*** SkyRocknRoll has quit IRC06:49
*** lcurtis has quit IRC06:54
*** nadeem has quit IRC06:56
*** haomaiwang has quit IRC07:01
*** haomaiwang has joined #openstack-swift07:02
*** david-ly_ has joined #openstack-swift07:02
*** cebruns_ has joined #openstack-swift07:05
*** dhellmann_ has joined #openstack-swift07:05
*** raginbaj- has joined #openstack-swift07:05
*** dfg_ has joined #openstack-swift07:05
*** jamespag` has joined #openstack-swift07:05
*** jamespag` has joined #openstack-swift07:05
*** d0ugal_ has joined #openstack-swift07:06
*** jamespage has quit IRC07:06
*** david-lyle has quit IRC07:06
*** cschwede has quit IRC07:06
*** raginbajin has quit IRC07:06
*** pgbridge has quit IRC07:06
*** wer has quit IRC07:06
*** HenryG has quit IRC07:06
*** dfg has quit IRC07:06
*** dhellmann has quit IRC07:06
*** cebruns has quit IRC07:06
*** d0ugal has quit IRC07:06
*** torgomatic has quit IRC07:06
*** cschwede has joined #openstack-swift07:06
*** pgbridge has joined #openstack-swift07:06
*** torgomatic has joined #openstack-swift07:06
*** wer has joined #openstack-swift07:06
*** ChanServ sets mode: +v cschwede07:06
*** ChanServ sets mode: +v torgomatic07:06
*** d0ugal_ is now known as d0ugal07:07
*** HenryG has joined #openstack-swift07:07
*** d0ugal is now known as Guest4795107:07
*** raginbaj- is now known as raginbajin07:07
*** mfalatic has quit IRC07:08
*** dhellmann_ is now known as dhellmann07:08
*** haomaiwang has quit IRC08:01
*** haomaiwang has joined #openstack-swift08:02
*** sanchitmalhotra has joined #openstack-swift08:29
*** sanchitmalhotra1 has joined #openstack-swift08:31
*** sanchitmalhotra has quit IRC08:34
*** sanchitmalhotra has joined #openstack-swift08:38
*** sanchitmalhotra1 has quit IRC08:40
openstackgerritVictor Stinner proposed openstack/swift: pep8: Fix hacking H403 warnings (docstrings)  https://review.openstack.org/20615008:41
openstackgerritVictor Stinner proposed openstack/swift: pep8: Fix hacking H232 warnings (octal)  https://review.openstack.org/20614908:41
openstackgerritVictor Stinner proposed openstack/swift: pep8: Don't format using self.__dict__  https://review.openstack.org/20596608:41
openstackgerritVictor Stinner proposed openstack/swift: pep8: replace deprecated calls to assert_()  https://review.openstack.org/21349108:41
openstackgerritVictor Stinner proposed openstack/swift: Fix tox -e py34  https://review.openstack.org/19903408:57
*** haomaiwang has quit IRC09:01
*** haomaiwang has joined #openstack-swift09:02
*** haomaiwang has quit IRC09:23
openstackgerritVictor Stinner proposed openstack/swift: pep8: Remove leading space in docstrings  https://review.openstack.org/21349309:34
openstackgerritVictor Stinner proposed openstack/swift: pep8: Fix usage of the l10n _('...') function  https://review.openstack.org/21349409:34
openstackgerritVictor Stinner proposed openstack/swift: pep8: Don't override '_' symbol  https://review.openstack.org/21349509:34
openstackgerritVictor Stinner proposed openstack/swift: pep8: fix warning F812  https://review.openstack.org/21349609:34
*** sanchitmalhotra1 has joined #openstack-swift09:52
*** sanchitmalhotra has quit IRC09:54
*** ho has joined #openstack-swift10:03
*** ho_ has quit IRC10:03
*** haigang has quit IRC10:18
*** sanchitmalhotra has joined #openstack-swift10:18
*** sanchitmalhotra1 has quit IRC10:20
*** silor has joined #openstack-swift10:38
*** SkyRocknRoll has joined #openstack-swift10:38
*** sanchitmalhotra1 has joined #openstack-swift10:44
*** sanchitmalhotra has quit IRC10:46
*** sanchitmalhotra has joined #openstack-swift10:49
*** sanchitmalhotra1 has quit IRC10:51
*** ho_ has joined #openstack-swift10:53
*** ho has quit IRC10:53
*** sanchitmalhotra1 has joined #openstack-swift10:56
*** sanchitmalhotra has quit IRC10:58
*** sanchitmalhotra has joined #openstack-swift11:19
*** sanchitmalhotra1 has quit IRC11:21
*** sanchitmalhotra1 has joined #openstack-swift11:28
*** ho_ has quit IRC11:28
*** ho has joined #openstack-swift11:28
*** sanchitmalhotra has quit IRC11:30
*** sanchitmalhotra has joined #openstack-swift11:47
*** sanchitmalhotra1 has quit IRC11:49
*** sanchitmalhotra1 has joined #openstack-swift11:54
*** sanchitmalhotra has quit IRC11:56
*** sanchitmalhotra has joined #openstack-swift12:09
*** sanchitmalhotra1 has quit IRC12:11
*** sanchitmalhotra1 has joined #openstack-swift12:23
*** sanchitmalhotra has quit IRC12:25
*** sanchitmalhotra has joined #openstack-swift12:29
*** openstackgerrit has quit IRC12:31
*** sanchitmalhotra1 has quit IRC12:31
*** openstackgerrit has joined #openstack-swift12:31
*** dbarman has joined #openstack-swift12:32
dbarmanis anyone here?12:32
ctennismaybe12:36
dbarmanlol12:37
dbarmanI was having an issue with my swift.  Was hoping someone was wiling to lead some advice12:37
ctennispost it here, someone will answer eventually12:38
dbarmanok.  thanks.12:38
dbarmanI added more storage to my small swift environment since my disks were almost full.  However, the new storage node disks are being used.  The swift-recon shows the the drives are there and empty.  But nothing get's stored to them.  I thought when I added the new node, that swift would automatically shift some of the data to the new drives12:40
ctennisdid you recreate your ring files after adding the storage?12:40
dbarmanI didn't re-create them, but I added the new drives on the new node to the existing rings.  Then rebalanced and then copied the new ring files to all the storage nodes12:41
*** ho has quit IRC12:41
*** ho has joined #openstack-swift12:41
ctennisyep that sounds right12:41
ctennisyou copies the .gz files right, not the .builder files?12:41
dbarmanYes, I thought so too.  For the life of me I can't figure out what is going on.  Pretty new to openstack.  Not sure where to look for the problem12:41
dbarmancorrect, the three *.ring.gz files12:42
ctennisyes that sounds correct12:42
dbarmanI used the scp command from the controller node to copy them to the storage nodes12:42
ctennisI would double check that the .gz files from your controller node match the ones on the storage nodes as expected, in /etc/swift12:43
dbarmanI am in a pickle since my other drives are virtually full.  Really need to figure out a solution12:43
ctennisalso make sure the object-replicator is running on the storage nodes, as it is what will move the data around to the new storage12:43
dbarmanis there a compare command or something to verify that they are the same?12:43
ctennischeck the md5sum or the date and size is probably the easiest way12:43
dbarmanobject-replicator, that's a service?  Can I just service object-replicator restart?12:44
ctennisyeah12:44
ctennisswift-object-replicator12:44
dbarmanok, I will check the dates on the files12:44
ctennisit will periodically output to syslog, depending on where you ahve its output going...it will show some statistics of what it's doing, how hard it's working, etc.12:44
dbarmanthe service is on the object nodes, correct?12:45
ctennisas long as the ring files are correct, it will move the data around12:45
ctennisright12:45
ctennisI'd check and make sure it's not erroring out, perhaps rsync isn't running and it can't shuffle bits.12:45
dbarmanWhere are the log files to see what is going on?12:45
ctenniswell, it just sends to syslog, so you have to use the syslogger to view it12:45
ctennisto the LOCAL0 log endpoint I believe12:46
dbarmanok.  never used that before.  I will try it out12:46
ctennisyou might be able to manually start it and see output on the command line though12:46
ctennisjust temporarily to make sure it's working right12:46
dbarmanmanually start what exactly?12:47
ctennisjust run swift-object-replicator at the command line12:48
ctennisinstead of as a daemon12:48
dbarmanIt is asking for command line parameters12:48
ctennisit may still log to the syslogger, I'm not sure12:48
ctennisah ok12:48
ctennisnevermind on that then12:48
ctenniswhat OS?12:50
dbarmanso all the logs go to syslog.  So I need to start there?  I did look at syslog before with just text editor and I saw a ton of quarantine entries12:50
dbarmanUbuntu 14.0412:50
ctennishm, quarantine is not good12:50
ctennisyeah, I'm not sure if you're using rsyslogd or syslog-ng, but you should be able to add a config item for all log local0 entries to go to a single file, like /var/log/swift/all.log, then see all of the swift activity in that file.12:51
ctennisassuming your /etc/swift/*.conf files are set to log to log local0, which is the default.12:51
ctennisthis is a sample rsyslog config to send swift data to places, you can uncomment out what's interesting and then restart rsyslog: https://github.com/openstack/swift/blob/master/etc/swift-rsyslog.conf-sample12:52
dbarmanok.  I just saw this:   account-server: ERROR __call__ error with REPLICATE /sda1/93/17788d0904681bdecc5a6ee496c41532 : #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/dist-packages/swift/account/server.py", line 284,$12:52
ctennisprobably need to look into quarantine stuff though, that's usually an indicate of an issue12:52
ctennisis there more to that error?12:53
ctennismore lines I mean12:53
dbarmanline 157, in makedirs#012    mkdir(name, mode)#012OSError: [Errno 13] Permission denied: '/srv/node/sda1/tmp'12:53
dbarmanlet me check12:54
ctenniscould you run swift-ring-builder on your rings and post the output of what the structure looks like maybe in a pastie or a gist and share it?12:54
ctennisthat's on the .builder files12:54
ctennisany idea why it would get a permission denied to that directory?12:55
ctennisin trying to mkdir12:55
ctennisI presume /srv/node/sda1 is one of your disks in the swift system12:56
ctennismaybe the ownership of that directory isn't such that the swift processes can write to it?12:56
dbarmanHere a a handfull of lines12:56
dbarmanOSSN2 account-server: ERROR __call__ error with REPLICATE /sda1/93/17788d0904681bdecc5a6ee496c41532 : #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/dist-packages/swift/account/server.py", line 284, in __call__#012    res = method(req)#012  File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 2217, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.7/dist-packages/swift/c12:56
dbarmanThere aren't any tmp directories in those storage folders.  Does it make them automatically?12:58
*** haomaiwang has joined #openstack-swift12:58
ctennisit's trying to make them, but doesn't have permission to do so12:58
dbarmanThe parent folders look like they are owned by root12:59
ctennisare the swift processes running as the swift user?12:59
dbarmanhow do I tell?12:59
ctennisps aux | grep swift12:59
ctennisI think the left most column will list the owner of the process13:00
*** haomaiwang has quit IRC13:01
dbarmanI see lots of processes.  container-updater, account-auditor, etc.  Also object-replicator, and there are more13:01
ctennisyeah13:01
ctenniswhat user are they running as?13:01
ekarlsopeople working on a sunday ? :p13:01
ctennisI'm just having coffee and ignoring my kids.13:01
*** haomaiwa_ has joined #openstack-swift13:02
dbarmanfirst column is al swift13:03
ctennisok13:03
ctennisso it sounds like that you have a permission problem and it can't write data13:03
ctennisso what I would recommend doing is just checking that you have the drives mounted into the /srv/node directories as expected, then check each one for ownership, and if it's owned by root, change it to be owned by swift13:04
ctennisso the processes can write the data to it13:04
ctennisthat's likely what's preventing new data from making its way onto the drives13:04
dbarmanyes, the (2) drives are both in the /srv/node folder, but both say "root root"13:05
ctennisyeah I'd make them owned by swift:swift13:05
ctennisyou can watch then, almost immediately stuff should start being written to them13:05
dbarmanok, sounds reasonable.  I will give it a try.  If so, it should start replicating immediately wouldn't you think?13:06
ctennisyep13:08
dbarmanok, I changed the ownership.  Now trying to figure out if anything is happening13:09
ctennislook for some new subdiretories13:11
ctennisman I cannot spell this morning13:11
dbarmanin the srv/node folders?13:11
ctennisit data is moving around it will start going under directories like "accounts" or "objects"13:11
ctennisyeah, on the drive mount where you just changed ownership13:11
dbarmanthere are a few directories in there13:12
ctennissounds like a good sign13:14
dbarmanok. I guess I need to wait and see what happens.  I really appreciate your input on this13:15
ctennisI would check the logs again and see if errors have gone away.13:15
ctennisalso the used space on the disks should be going up13:15
ctennisyes no problem at all13:15
dbarmanI did a df-h and the two drives a few GB used each now.  So I think something is happening!13:15
dbarmanBut it's kind of weird.  The disk space goes up and then it seems like it goes down again.13:17
dbarmanI guess I will give it an hour or two and see what happens13:17
ctennisok, just let us know how it goes13:17
dbarmanwill do13:23
*** links has quit IRC13:29
*** barra204 has joined #openstack-swift13:35
*** shakamunyi has quit IRC13:36
*** SkyRocknRoll has quit IRC13:42
*** haomaiwa_ has quit IRC14:01
*** haomaiwang has joined #openstack-swift14:02
*** SkyRocknRoll has joined #openstack-swift14:02
*** sanchitmalhotra1 has joined #openstack-swift14:05
*** sanchitmalhotra has quit IRC14:07
*** SkyRocknRoll has quit IRC14:07
*** haomaiwang has quit IRC14:31
*** sanchitmalhotra has joined #openstack-swift14:34
*** sanchitmalhotra1 has quit IRC14:36
*** sanchitmalhotra1 has joined #openstack-swift14:51
*** sanchitmalhotra has quit IRC14:54
*** sanchitmalhotra has joined #openstack-swift15:07
*** sanchitmalhotra1 has quit IRC15:09
*** nadeem has joined #openstack-swift15:17
*** sanchitmalhotra1 has joined #openstack-swift15:30
*** sanchitmalhotra has quit IRC15:31
*** janonymous has joined #openstack-swift15:36
*** sanchitmalhotra has joined #openstack-swift15:41
dbarmanok.  things are looking better.  new node has over 30GB used now.15:42
*** sanchitmalhotra1 has quit IRC15:43
*** nadeem has quit IRC15:47
*** janonymous has quit IRC15:49
dbarmananyone know what this means in syslog:  account-replicator: Replication run OVER16:05
*** pbourke has quit IRC16:10
*** pbourke has joined #openstack-swift16:10
*** sanchitmalhotra1 has joined #openstack-swift16:17
dbarmananyone know what this means: Aug 16 06:54:47 OSSN1 object-auditor: ERROR Trying to audit /srv/node/sdb1/objects/272/659/4434c5946ef1aa00bd4b58a87c7e6659: #012Traceback (most recent call last):#012  File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 1 73, in failsafe_object_audit#012    self.object_audit(location)#012  File "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 191, in object_audit#01216:19
*** sanchitmalhotra has quit IRC16:20
*** silor has quit IRC16:20
*** sanchitmalhotra has joined #openstack-swift16:33
*** sanchitmalhotra1 has quit IRC16:35
*** hezhiqiang has joined #openstack-swift16:37
*** sanchitmalhotra1 has joined #openstack-swift17:14
*** sanchitmalhotra has quit IRC17:16
dbarmanany way to tell when new node has completely replicated?17:27
*** sanchitmalhotra has joined #openstack-swift17:54
dbarmanany way to tell when new node has completely replicated?17:55
*** sanchitmalhotra1 has quit IRC17:56
*** sanchitmalhotra1 has joined #openstack-swift18:08
*** sanchitmalhotra has quit IRC18:10
*** sanchitmalhotra has joined #openstack-swift18:10
*** sanchitmalhotra1 has quit IRC18:12
notmynamedbarman: you can use `swift-recon --replication` to see how long replication is taking in your cluster. replication is push-based, so you need to look at the replication cycle time on the rest of the cluster18:13
*** sanchitmalhotra1 has joined #openstack-swift18:34
*** sanchitmalhotra has quit IRC18:35
MooingLemurdbarman: also, replication may not complete in one pass, especially if your rsync connection limits are exceeded on the receiving nodes18:37
*** gaillard has joined #openstack-swift18:38
gaillardHi all. Can anybody tell me, why it is not possible to pass a header through the formpost middleware but through the tempurl middleware?18:39
dbarmanok.  do I run the recon on all nodes or just one?18:41
dbarmanI see it shows the oldest completion and the most recent completion.  What exactly does that do for me?18:43
*** sanchitmalhotra has joined #openstack-swift18:47
*** sanchitmalhotra1 has quit IRC18:50
*** sanchitmalhotra1 has joined #openstack-swift19:36
*** gaillard has quit IRC19:36
*** sanchitmalhotra has quit IRC19:38
*** sanchitmalhotra has joined #openstack-swift19:40
*** sanchitmalhotra1 has quit IRC19:42
*** sanchitmalhotra1 has joined #openstack-swift19:59
*** sanchitmalhotra has quit IRC20:02
*** hezhiqiang has quit IRC20:04
*** sanchitmalhotra has joined #openstack-swift20:09
*** sanchitmalhotra1 has quit IRC20:11
*** silor has joined #openstack-swift20:17
*** sanchitmalhotra1 has joined #openstack-swift20:24
*** sanchitmalhotra has quit IRC20:25
*** sanchitmalhotra has joined #openstack-swift20:28
*** silor has quit IRC20:30
*** sanchitmalhotra1 has quit IRC20:30
*** silor has joined #openstack-swift20:31
*** silor has quit IRC20:59
*** hezhiqiang has joined #openstack-swift21:05
*** sanchitmalhotra1 has joined #openstack-swift21:07
*** hezhiqiang has quit IRC21:09
*** sanchitmalhotra has quit IRC21:09
ctennisdbarman: you just need to run it on one node, it will show the same thing no matter where you run it21:21
*** sanchitmalhotra has joined #openstack-swift21:21
*** sanchitmalhotra1 has quit IRC21:23
*** sanchitmalhotra1 has joined #openstack-swift21:23
*** sanchitmalhotra has quit IRC21:26
*** sanchitmalhotra has joined #openstack-swift21:26
*** sanchitmalhotra1 has quit IRC21:27
*** sanchitmalhotra1 has joined #openstack-swift21:31
*** sanchitmalhotra has quit IRC21:33
*** sanchitmalhotra has joined #openstack-swift21:33
*** sanchitmalhotra1 has quit IRC21:36
*** wbhuber has joined #openstack-swift21:44
*** fbo has quit IRC21:44
*** Kennan has quit IRC22:01
*** Kennan has joined #openstack-swift22:02
*** changbl has quit IRC22:07
dbarmanok.  so what does the oldest and most recent completion tell me about the total progress of the replication?22:22
*** sanchitmalhotra1 has joined #openstack-swift22:28
*** sanchitmalhotra has quit IRC22:29
*** sanchitmalhotra has joined #openstack-swift22:39
*** sanchitmalhotra1 has quit IRC22:41
*** sanchitmalhotra1 has joined #openstack-swift22:52
*** sanchitmalhotra has quit IRC22:54
*** sanchitmalhotra has joined #openstack-swift23:08
*** sanchitmalhotra1 has quit IRC23:10
*** kei_yama has joined #openstack-swift23:19
hogood morning!23:36
*** hrou has joined #openstack-swift23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!