Wednesday, 2016-07-27

kota_clayg: maybe, https://review.openstack.org/#/c/317672/ ?00:00
patchbotkota_: patch 317672 - openstack-infra/project-config - add python jobs to pyeclib project00:00
claygboy that pyeclib_c_get_segment_info is a function now00:00
kota_yes, it's a function of python c-extention00:03
*** Suyi_ has quit IRC00:04
openstackgerritKota Tsuyuzaki proposed openstack/pyeclib: Ref count for dict item should be Py_DECREF  https://review.openstack.org/34406600:04
kota_fix a typo00:05
kota_wooo, no-op jenkis so fast...00:05
*** ukaynar has joined #openstack-swift00:08
kota_and I know, probably the python-gate should get a bunch of failures in... pep8 gate:/00:09
-kota_- tsuyuzaki@ubuntu:~/git_repos/pyeclib$ flake8 | wc -l00:09
-kota_- 20700:09
*** _JZ_ has quit IRC00:09
*** jamielennox is now known as jamielennox|away00:12
kota_:\00:12
*** diogogmt has quit IRC00:27
*** ukaynar has quit IRC00:42
*** vinsh has joined #openstack-swift00:50
*** m_kazuhiro has joined #openstack-swift00:53
*** tqtran has quit IRC00:56
*** vinsh has quit IRC01:02
*** dmorita_ has quit IRC01:28
*** dmorita has joined #openstack-swift01:29
*** openstackgerrit has quit IRC01:33
*** openstackgerrit has joined #openstack-swift01:34
*** mingyu has joined #openstack-swift01:43
*** trananhkma has joined #openstack-swift01:50
*** mingyu has quit IRC01:54
*** dmorita has quit IRC01:57
openstackgerritMerged openstack/swift: go: fix tests with old swift.conf  https://review.openstack.org/34690902:01
*** Suyi_ has joined #openstack-swift02:12
*** StevenK has joined #openstack-swift02:22
*** Jeffrey4l has joined #openstack-swift02:26
*** mathiasb has joined #openstack-swift02:27
*** mingyu has joined #openstack-swift02:39
*** jamielennox|away is now known as jamielennox02:41
*** tqtran has joined #openstack-swift02:55
*** tqtran has quit IRC02:59
*** gyee has quit IRC03:03
*** mingyu has quit IRC03:10
*** mingyu has joined #openstack-swift03:11
*** thumpba has joined #openstack-swift03:15
*** mingyu has quit IRC03:15
*** thumpba has quit IRC03:18
*** mingyu has joined #openstack-swift03:19
openstackgerritJanie Richling proposed openstack/swift: WIP - Add tests and docs for POST to DLO manifest file  https://review.openstack.org/34754503:23
*** sheel has joined #openstack-swift03:48
openstackgerritJanie Richling proposed openstack/swift: WIP - Add tests and docs for POST to DLO manifest file  https://review.openstack.org/34754503:51
*** abhitechie has joined #openstack-swift03:51
*** links has joined #openstack-swift03:53
claygboy that pyeclib_c_get_segment_info is a function now03:54
claygsrory03:54
*** dmorita has joined #openstack-swift03:57
*** thumpba has joined #openstack-swift04:00
*** dmorita has quit IRC04:02
*** thumpba has quit IRC04:06
*** links has quit IRC04:09
*** links has joined #openstack-swift04:12
*** thumpba has joined #openstack-swift04:15
*** asettle has joined #openstack-swift04:26
*** thumpba has quit IRC04:27
*** jamielennox is now known as jamielennox|away04:30
*** asettle has quit IRC04:31
*** chsc has joined #openstack-swift04:31
*** chsc has joined #openstack-swift04:31
*** thumpba has joined #openstack-swift04:36
*** janonymous has joined #openstack-swift04:45
*** mingyu has quit IRC04:47
*** chsc has quit IRC04:47
*** adu has joined #openstack-swift04:48
*** jamielennox|away is now known as jamielennox04:51
*** psachin has joined #openstack-swift04:52
*** chsc has joined #openstack-swift04:53
*** chsc has joined #openstack-swift04:53
*** flwang1 has quit IRC05:03
*** mingyu has joined #openstack-swift05:04
*** chsc has quit IRC05:06
*** SkyRocknRoll has joined #openstack-swift05:16
*** maestropandy has joined #openstack-swift05:21
*** zaitcev has quit IRC05:25
*** ChubYann has quit IRC05:31
*** adu has quit IRC05:35
*** dmorita has joined #openstack-swift05:42
*** mingyu has quit IRC05:45
*** McMurlock1 has joined #openstack-swift05:45
*** dmorita has quit IRC05:47
*** maestropandy has quit IRC05:54
*** ppai has joined #openstack-swift05:56
*** tqtran has joined #openstack-swift05:58
*** tqtran has quit IRC06:02
*** Suyi_ has quit IRC06:05
*** mingyu has joined #openstack-swift06:06
*** RonChiang has joined #openstack-swift06:09
*** RonChiang has quit IRC06:10
*** KriskK has joined #openstack-swift06:13
*** amitkqed has joined #openstack-swift06:35
*** amitkqed has quit IRC06:36
*** amitkqed has joined #openstack-swift06:37
*** asettle has joined #openstack-swift06:39
*** silor has joined #openstack-swift06:39
*** klrmn has quit IRC06:42
*** asettle has quit IRC06:43
*** tesseract- has joined #openstack-swift06:44
*** rcernin has joined #openstack-swift06:47
*** geaaru has joined #openstack-swift06:47
*** silor1 has joined #openstack-swift06:48
*** maestropandy has joined #openstack-swift06:48
*** derekjhyang has quit IRC06:49
*** zhiyan has quit IRC06:49
*** briancurtin has quit IRC06:49
*** cppforlife_ has quit IRC06:50
*** amit213 has quit IRC06:50
*** silor has quit IRC06:51
*** silor1 is now known as silor06:51
*** amit213 has joined #openstack-swift06:51
*** derekjhyang has joined #openstack-swift06:52
*** briancurtin has joined #openstack-swift06:52
*** zhiyan has joined #openstack-swift06:52
*** cppforlife_ has joined #openstack-swift06:52
*** thumpba has quit IRC06:53
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/swift: [WIP] Testing latest u-c  https://review.openstack.org/31844107:07
*** rledisez has joined #openstack-swift07:12
*** thumpba has joined #openstack-swift07:15
*** thumpba has quit IRC07:18
*** jordanP has joined #openstack-swift07:19
*** asettle has joined #openstack-swift07:21
*** mingyu has quit IRC07:22
*** asettle has quit IRC07:26
*** oshritf has joined #openstack-swift07:29
*** mtreinish has quit IRC07:36
*** mtreinish has joined #openstack-swift07:36
*** lickitysplitted has quit IRC07:37
*** lickitysplitted has joined #openstack-swift07:37
*** maestropandy has quit IRC07:38
*** saltsa has quit IRC07:39
*** saltsa has joined #openstack-swift07:39
*** acoles_ is now known as acoles07:41
acolesgood morning07:45
acolescschwede: did you see in scrollback that there is no meeting this evening?07:46
cschwedeacoles: good morning! not yet, but thx a lot for the reminder :D07:47
acolescschwede: <notmyname> [Tue 26 23:23] still nothing has been added to the meeting agenda for tomorrow, so I'll call it off. no meeting tomorrow07:48
mahaticacoles: good morning07:53
acolesmahatic: hi07:54
*** donagh has quit IRC07:54
mahaticworking on your comments now (on tomstones bug). Couldn't get to them yesterday07:54
kota_good morning, acoles and mahatic07:57
mahatickota_: hello, good evening07:57
kota_yup, I can drink a lot in this evening :-)07:58
acolesmahatic: sure, np07:58
kota_because of no meeting.07:58
acoleskota_: hehe07:59
mahatickota_: have fun07:59
kota_:-)07:59
openstackgerritzheng yin proposed openstack/swift: Use del replace pop  https://review.openstack.org/34735808:04
*** abhitechie has quit IRC08:07
*** abhitechie has joined #openstack-swift08:07
*** portante has quit IRC08:07
*** saltsa_ has joined #openstack-swift08:08
*** pgbridge has quit IRC08:09
*** portante has joined #openstack-swift08:09
openstackgerritDavanum Srinivas (dims) proposed openstack/swift: [WIP] Testing latest u-c  https://review.openstack.org/31844108:10
*** pgbridge has joined #openstack-swift08:10
*** saltsa has quit IRC08:11
*** rminmin has joined #openstack-swift08:12
*** hoonetorg has quit IRC08:13
openstackgerritAlistair Coles proposed openstack/swift: Additional test for container override etag preferences  https://review.openstack.org/34768408:14
*** hoonetorg has joined #openstack-swift08:17
*** asettle has joined #openstack-swift08:17
*** mingyu has joined #openstack-swift08:18
*** asettle has quit IRC08:21
*** acoles is now known as acoles_08:31
*** acoles_ is now known as acoles08:35
*** geaaru has quit IRC08:39
*** geaaru has joined #openstack-swift08:52
*** hseipp has joined #openstack-swift08:55
*** asettle has joined #openstack-swift09:13
*** asettle has quit IRC09:18
*** maestropandy has joined #openstack-swift09:19
*** rminmin has quit IRC09:25
*** asettle has joined #openstack-swift09:31
*** asettle has quit IRC09:36
*** maestropandy has quit IRC09:57
*** chlong_POffice has quit IRC09:57
*** oshritf_ has joined #openstack-swift10:05
*** acoles is now known as acoles_10:07
*** oshritf has quit IRC10:07
*** mvk has quit IRC10:07
*** chlong_POffice has joined #openstack-swift10:10
*** furlongm has quit IRC10:15
*** furlongm has joined #openstack-swift10:16
*** maestropandy has joined #openstack-swift10:27
*** ppai has quit IRC10:27
*** NM has joined #openstack-swift10:32
*** mvk has joined #openstack-swift10:38
*** ppai has joined #openstack-swift10:41
openstackgerritMerged openstack/swift: Mention SWIFT_TEST_DEBUG_LOGS in development guide  https://review.openstack.org/34748310:44
openstackgerritMerged openstack/swift: Document how to run a single functional test  https://review.openstack.org/34749210:46
*** maestropandy has quit IRC10:51
*** trananhkma has quit IRC10:55
*** m_kazuhiro has quit IRC10:57
*** m_kazuhiro has joined #openstack-swift11:02
*** mingyu has quit IRC11:03
*** m_kazuhiro has quit IRC11:03
*** mingyu has joined #openstack-swift11:05
*** mingyu has quit IRC11:10
*** NM has quit IRC11:14
*** kei_yama has quit IRC11:14
hosanainotmyname: ack no meeting tomorrow.11:36
hosanainotmyname, clayg: now i'm working on an eventlet's patch. https://github.com/eventlet/eventlet/commit/7e6561e33ffde42c44c9d49ee1b2a506a9c260b4#diff-d0a9454bc2395e809b9546e72118d8e5R296 i need to discuss it to a guy of eventlet.11:39
hosanainotmyname, clayg: try to revert the patch from tpool11:44
openstackgerritoshritf proposed openstack/swift: Add process level concurrency to container sync  https://review.openstack.org/21009911:46
tdasilvagood morning11:47
openstackgerritoshritf proposed openstack/swift: Add thread level concurrency to container sync  https://review.openstack.org/22533811:47
*** cdelatte has joined #openstack-swift11:48
*** abhitechie has quit IRC11:50
*** hosanai has quit IRC11:53
*** natarej__ has quit IRC11:56
*** asettle has joined #openstack-swift11:59
*** NM has joined #openstack-swift12:06
*** maestropandy has joined #openstack-swift12:16
openstackgerritlokesh s proposed openstack/swift: Py3: Fixes AttributeError: 'generator' object issue  https://review.openstack.org/34779912:16
*** chlong_POffice has quit IRC12:21
*** chlong has joined #openstack-swift12:29
*** klamath has joined #openstack-swift12:40
*** klamath has quit IRC12:41
*** klamath has joined #openstack-swift12:41
*** abhitechie has joined #openstack-swift12:46
*** sheel has quit IRC12:56
openstackgerritlokesh s proposed openstack/swift: Py3: Fixes TypeError:'str' does not support the buffer interface  https://review.openstack.org/34781212:56
*** links has quit IRC13:08
*** ppai has quit IRC13:10
*** rledisez has quit IRC13:10
*** rledisez has joined #openstack-swift13:11
*** ppai has joined #openstack-swift13:18
*** acoles_ is now known as acoles13:30
*** ppai has quit IRC13:31
*** david-lyle_ has joined #openstack-swift13:31
*** david-lyle_ has quit IRC13:31
*** mmcardle has joined #openstack-swift13:33
*** david-lyle has quit IRC13:34
*** abhinavtechie has joined #openstack-swift13:41
*** maestropandy has left #openstack-swift13:42
*** aagrawal has joined #openstack-swift13:43
*** abhitechie has quit IRC13:43
*** ppai has joined #openstack-swift13:44
*** abhinavtechie has quit IRC13:45
*** aagrawal has quit IRC13:48
jrichligerrit seems to be down13:52
-openstackstatus- NOTICE: Gerrit is being restarted now to relieve performance degradation13:53
*** mmcardle has quit IRC13:53
jrichliah, nice!13:54
haypojrichli: ah?13:54
haypojrichli: i just pushed a change and then approved it13:54
haypo(it worked for me)13:54
jrichlihaypo: all I know is that I got a 503, but now its fine13:55
jrichligood timing, i guess13:55
haypooh. in fact i got a 503 but it worked when i retried ;)13:56
haypojrichli: i heard that gerrit has some issues (memory leak? someything else?) and sometimes the infra "simply" restarts servers to "fix" these issues :)13:56
haypoi heard a similar story a few years ago by an admin of a ejabberd server, no one knew erlang and no one was able to fix memory leaks :-p13:56
jrichliheh.  i wonder how often the restart must happen.  i guess not too much, since i haven't run into many issues.  :-)13:58
*** dmorita has joined #openstack-swift13:58
openstackgerritMahati Chamarthy proposed openstack/swift: Delete old tombstones  https://review.openstack.org/34686513:59
*** chsc has joined #openstack-swift14:00
*** chsc has joined #openstack-swift14:00
haypojrichli: no one excuse to review my python3 patches :-D14:01
haypooops14:01
haypono more excuse*14:01
haypojrichli: no idea. maybe once a week14:01
*** ppai has quit IRC14:01
*** mmcardle has joined #openstack-swift14:02
*** dmorita has quit IRC14:03
*** SkyRocknRoll has quit IRC14:05
*** geaaru has quit IRC14:07
*** aswadr_ has joined #openstack-swift14:11
*** mingyu has joined #openstack-swift14:13
*** _JZ_ has joined #openstack-swift14:15
*** ppai has joined #openstack-swift14:15
*** KriskK has quit IRC14:16
*** geaaru has joined #openstack-swift14:19
notmynamegood morning14:24
mahaticnotmyname: hello, good morning14:26
mmotianigood morning14:34
*** ppai has quit IRC14:36
jrichlipatch 347101 was updated (gerrit didn't announce it)14:36
patchbotjrichli: https://review.openstack.org/#/c/347101/ - swift - Enable in-process func tests to optionally use enc...14:36
notmynamecoffee drunk, emails checked. time to contiue to get ready. be back online later14:36
openstackgerritlokesh s proposed openstack/swift: Py3: Fixes TypeError:'str' does not support the buffer interface  https://review.openstack.org/34781214:40
*** siva_krish has joined #openstack-swift14:40
*** mmcardle has quit IRC14:45
*** zaitcev has joined #openstack-swift14:46
*** ChanServ sets mode: +v zaitcev14:46
*** abhitechie has joined #openstack-swift14:48
pdardeaugood morning14:52
mahaticzaitcev: hi, I modified an old test of yours to address a change (patch 346865). I included you as a co-author, hope it's okay14:52
patchbotmahatic: https://review.openstack.org/#/c/346865/ - swift - Delete old tombstones14:52
zaitcevmahatic: absolutely, thanks14:52
*** hseipp has quit IRC14:58
*** jistr is now known as jistr|call15:00
*** psachin has quit IRC15:09
*** KriskK has joined #openstack-swift15:11
*** chsc has quit IRC15:11
*** pcaruana has quit IRC15:17
acolesoshritf_: timburke: I'd be interested if you have an opinion on https://bugs.launchpad.net/swift/+bug/1605597 - I plan to fix the bug so that the SLO manifest is copied.15:23
openstackLaunchpad bug 1605597 in OpenStack Object Storage (swift) "container sync cannot copy SLOs" [High,Confirmed] - Assigned to Alistair Coles (alistair-coles)15:23
openstackgerritEllen Leahy proposed openstack/swift: Fix inline tempurl/formpost signature examples  https://review.openstack.org/33504415:23
*** rcernin has quit IRC15:24
oshritf_acoles: taking a look15:26
timburkeacoles: remind me, why do we 400 if the client sends X-Static-Large-Object? why not ignore it, like we would if they sent X-Backend-* or X-Object-Sysmeta-* or most (any?) of the other internal-only headers?15:34
*** jistr|call is now known as jistr15:35
timburkedefinitely want the manifest rather than the large object; we'll get a bunch of 413s otherwise15:36
*** tqtran has joined #openstack-swift15:38
timburkei expect some difficulties, though: what if the container name is different between source and destination? how do we know that the segments have all transferred before we attempt the manifest? how do we handle currently-broken manifests? what if the segments are in a separate container (which may or may not be synced)?15:39
*** KriskK has quit IRC15:43
*** diogogmt has joined #openstack-swift15:44
acolestimburke: re 400, I can't remind you, I don't know why ;) the rest are all good questions. It is the case with container sync that end users have to worry about cross container dependencies (something kota_ mentions in patch 304382), which is hard because the user gets no feedback as to the progress of sync.15:44
patchbotacoles: https://review.openstack.org/#/c/304382/ - swift - Add Pros/Cons table for global clusetr consideration15:44
oshritf_acoles: timbre: I'm not so familiar with SLO, but I assume transferring the manifest only wouldn't help much if the source container is no longer available?15:46
*** KriskK has joined #openstack-swift15:46
acolesoshritf_: if the segments container (where the pieces of the SLO are) is not available or finished sync'ing then the manifest copy would fail because the manifest is validated when PUT.15:48
oshritf_acoles: so basically when copying the manifest - it automatically generates multiple PUT requests for all the segments? causing the entire large object to replicate to the destination?15:51
oshritf_(assuming SLO middleware in internal client)15:51
acolesoshritf_: now, on master, if slo is in the client, then slo will re-assemble the segments and sync will attempt (but fail) to put that large object to the destination.15:53
acoleshmmm, so I think the same will happen for DLOs and work, i.e. they would be sync'd as reassembled objects, not manifests. I wonder if this is a regression, we have docs that suggest LO should work with sync but manifests and segments are sync'd separately.15:54
oshritf_acoles: thanks, now I understand timburke points above :)15:54
oshritf_acoles: regarding regression - were DLO/SLO ever in Container sync internal client pipeline?15:58
*** hseipp has joined #openstack-swift16:00
acolesoshritf_: not as far as I know, and I just realised that dlo is not in the client so what I said just before was wrong.16:00
oshritf_acoles: np, happy it's not a regression :)16:00
*** klrmn has joined #openstack-swift16:00
acolessync will copy a dlo manifest as a manifest because there is no DLO middleware to reassemble the object in the sync client pipeline.16:00
acolesconversely, without slo middleware sync tries to copy an slo manifest as a manifest, but whose content and headers are illegal to PUT :/16:01
acolesoshritf_: also not that DLO does not validate the manifest when PUT, so its ok to copy a DLO manifest before the segments have been copied16:02
acoless/not/note/16:02
openstackgerritOpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements  https://review.openstack.org/8925016:04
oshritf_acoles: right, but until they are all copy - there's a DLO manifest at the destination, but better not invoke any operations on it? what causes the segments to be copied? are they guarantee to be in the same container?16:04
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements  https://review.openstack.org/8873616:05
acolesoshritf_: 1. right! 2. the user must setup sync for the segments container (if different) 3. no guarantee, it is user's choice where DLO segments are16:06
acolesoshritf_: so am I correct to think that there is no way that a user, via the Swift API, can learn about the progress of container sync?16:07
oshritf_acoles: sounds like user responsibility, better use the same approach for DLO and SLO, so adapt SLO to follow the same behaviour or change both? What way is there to send SLO manifest other than PUT?16:08
oshritf_acoles: logs keep track of progress.. I don't recall any API for that16:09
acolesoshritf_: SLO manifest must be PUT, so somehow user needs to know that segments have been sync'd before the manifest. or, rely on container sync retrying failed manifest PUTs.16:09
*** KriskK has quit IRC16:09
*** itlinux has quit IRC16:13
timburkeand again, you need to worry about name changes. suppose you have containers foo and foo_segments that you want to sync to another account on another cluster. but you already have a foo container there, so you set it up to sync to foo_remote and foo_segments_remote ... and none of your large objects will work until we make container sync smart enough to start manipulating manifests16:14
*** nadeem has joined #openstack-swift16:15
openstackgerritThiago da Silva proposed openstack/swift: Symlink implementation.  https://review.openstack.org/23216216:17
oshritf_acoles:  I guess for a meaningful error message to the user would be a good idea - instead of reserved header error, one that would encourage the user to manually make sure all segments are at destination and container sync PUT of manifest would be helpful then16:18
acolestimburke: true. but same goes for DLO, so the note at start of http://docs.openstack.org/developer/swift/overview_container_sync.html should be improved.16:19
acolesand symlinks16:21
timburkeyup. and if you have fairly simple naming conventions for large objects, you may still get data, just not the data you expect :-/16:21
*** asettle has quit IRC16:23
*** SkyRocknRoll has joined #openstack-swift16:23
*** asettle has joined #openstack-swift16:27
acolestimburke: "Swift does not, however, stop the user from harming themselves."16:27
notmynameheh16:27
acolesIts in the docs docs.openstack.org/developer/swift/overview_large_objects.html (page all the way down)16:28
timburkethat's a lot of my argument around the toggleability of versioning mode in patch 214922 :-)16:28
patchbottimburke: https://review.openstack.org/#/c/214922/ - swift - Add "history" mode to versioned_writes middleware16:28
oshritf_:)16:30
*** dmorita has joined #openstack-swift16:35
klamathi have a few questions about concurrent_gets, can someone help answer?16:36
*** MadOtis has joined #openstack-swift16:37
klamathis a 404 a "valid" response? so if a handoff node is queried and a 404 is generated, the proxy will return a 404 to the client?16:37
*** mvk has quit IRC16:39
*** rledisez has quit IRC16:39
*** KriskK has joined #openstack-swift16:40
*** Suyi has joined #openstack-swift16:45
claygthere was a thread on python-ideas today "let's make stdlib consistent!  alias all the MethodNames and fix the classnames!"  Python core was like "yeah we've thought about this a *lot* - and always reach the same conclusion - too much effort and risk to justify the value, no thanks"16:54
claygso.. anytime you want to blow off review of "make all the codez better!" - know you're not alone16:55
MadOtisCan anyone help me get swift running on a Centos 7 controller?  The proxy fails to start and the journalctl output is here: http://paste.openstack.org/show/542665/17:03
MadOtisI have no /var/log/swift/ logs at all (directory is empty)17:03
*** gyee has joined #openstack-swift17:03
claygMadOtis: your proxy-server.conf is misconfigured, you have a reference to the authtoken middleware in your pipeline:main - but no filter:authtoken section defined17:04
notmynameMadOtis: check out the sample configs in https://github.com/openstack/swift/tree/master/etc17:05
MadOtisHmmm.... there should be...17:05
*** KriskK has quit IRC17:05
MadOtisthe [filter:authtoken] line itself is commented out... I assume that should be uncommented, so trying that.17:06
tdasilvaMadOtis: if you want a "template" to compare too, this project https://github.com/thiagodasilva/ansible-saio sets up a saio on a centos 7.2 VM17:06
MadOtismaking progress... swift proxy starts, but now when I run "swift stat", I get: Account HEAD failed: http://controller:8080/v1/AUTH_b9115c3d6e8649889b487174873a7c40 503 Service Unavailable17:11
MadOtisLogs say Identity server rejected authorization17:12
notmynamebut that shouldn't give you a 503. should be a 40117:13
*** tesseract- has quit IRC17:13
notmynameso there's something else going on17:13
MadOtisYeah, journalctl says: http://paste.openstack.org/show/542669/17:14
*** links has joined #openstack-swift17:15
MadOtisI'm looking at my object1 and object2 nodes, now.17:15
*** asettle has quit IRC17:15
*** siva_krish has left #openstack-swift17:16
notmynamethe connection refused messages are the important ones17:16
notmyname(1) are the storage node servers running? (account container and object)17:16
notmyname(2) if they are running, try `swift-recon object container account --validate` to check them17:18
notmynamethe second suggestion should find the most common issues in ring files17:18
MadOtisI think I missed a step in the installation docs... services are not running on the object nodes.  I'm starting them now.17:19
MadOtis<facepalm>17:19
*** chsc has joined #openstack-swift17:22
MadOtisok, I don't feel so bad... they were enabled already, and tried to start on both nodes, but failed with the libisal.so missing errors.  I'll just retry installing libisal on the nodes and see what happens.17:22
notmynamereminder to all, no swift team meeting today17:30
*** oshritf_ has quit IRC17:31
*** KriskK has joined #openstack-swift17:40
MadOtisLooks like there may be an issue with the "object" swift config or something... I get this with the "swift-recon XXXX --validate":  http://paste.openstack.org/show/542673/17:46
MadOtis...on both object nodes.17:47
openstackgerritCharles Hsu proposed openstack/swift: Retrieve recon data from base port and add --exclude-zeroweight.  https://review.openstack.org/34351417:48
notmynameMadOtis: that means that the IP:port listed in the object ring is actually pointing to servers that are self-identifying as container servers17:49
notmynameMadOtis: normally it's because of a fat-finger entry on the port when building rings17:49
MadOtisSo, I should rebuild the rings and redeploy the .gz files?17:50
MadOtis...for the object server?17:50
MadOtisI don't doubt that I fat-fingered something... I did this at like 2am this morning.17:51
*** mingyu has quit IRC17:52
*** links has quit IRC17:54
*** mingyu has joined #openstack-swift17:54
pdardeaui added a bunch of wishlist bugs for missing man pages (3 conf files and 10 command-line utilities)17:58
*** mvk has joined #openstack-swift17:58
notmynameMadOtis: yeah, rebuilding them is the simplest. you could remove the bad entries, add the good entries, and rebalance, but since you're just starting out with your cluster, simply remaking the object ring would probably be simplest17:58
*** acoles is now known as acoles_17:58
notmynamepdardeau: thanks17:58
notmynamepdardeau: did you ever find any tool or syntax description for man pages?17:59
pdardeaunotmyname: kind of. not really knowing groff/troff, it's a deep rabbit hole17:59
notmynameclayg: my first improvement to vSAIO was going to be adding xenial support. alas, it's not working upstream. but the bug report comment from 4 hours ago says "should get the fixes real soon now (I hope.)"18:00
pdardeaunotmyname: i toyed with the idea of a utility to take a more friendly format (.ini) and then generating man page from it18:00
pdardeaunotmyname: https://github.com/pauldardeau/cli-docs18:01
notmynameinteresting18:02
MadOtisok, nodes are up and recon validates on all 3 (container, account, and object).  Yet, running "swift stat" on the controller still returns a "401 unauthorized" error.18:07
notmynameMadOtis: that's good news! sounds like you've got (nearly) all the pieces in place18:08
MadOtisYeah, just need to find out why auth is being rejected18:08
notmynameMadOtis: there's 2 possibilities with the 401. first is simply that the creds you're using are bad18:08
notmynameMadOtis: that's the easy one18:08
notmynameMadOtis: the more challenging possibility is that you've got a config issue between swift and your auth system. what auth system are you using?18:09
MadOtiskeystone18:09
notmynameso it could be a keystone setup thing18:09
MadOtishere's a paste from the /var/log/swift/swift.log on the controller: http://paste.openstack.org/show/542677/18:09
notmynameso (1) creds (2) communication between swift and keystone (3) keystone config issue18:10
MadOtisok, let me look there18:10
*** jamielennox is now known as jamielennox|away18:10
notmynameMadOtis: hang on, there might be one simple config option18:10
MadOtisok18:10
notmynameMadOtis: https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L312-L31518:11
notmynamecheck that in your proxy config file18:11
MadOtisok, checking18:11
notmynameif it's already set to true, then carry on with your keystone config checking18:11
MadOtisyes, delay_auth_decision = true18:12
*** mingyu has quit IRC18:16
MadOtisArgh!  checked the proxy config line-for-line with the docs (I even remove the space between 'admin, user' on the 'operator_roles' section and restarted the proxy.  Same damn authentication error.18:24
notmynameMadOtis: try `swift stat -v`18:25
*** zul_ has joined #openstack-swift18:25
notmynameno, sorry18:25
*** zul has quit IRC18:25
notmyname`swift auth -v`18:25
MadOtisok18:25
notmynamethe hope being it might help isolate the issue18:25
MadOtisAll looks good, and is the same as I use for other services18:26
notmynamenormally it's at this point I defer to someone with keystone config experience :-/18:27
*** abhitechie has quit IRC18:27
notmynameMadOtis: oh wait!18:28
notmynamewhat command, exactly, are you using?18:28
MadOtis. admin-openrc; swift stat18:28
notmynameare you explicitly telling the swift cli tool to use auth version 2.0?18:28
MadOtisOS_AUTH_VERSION=318:29
MadOtisshould I try 2?18:29
notmynamesure. 2.018:29
notmynameI mean, it's not like the current situation will get worse that way ;-)18:30
MadOtiscomes back almost immediately with a 404 error18:30
notmynamenot sure. next step is to go over your keystone config, I guess18:30
notmynameand unfortunately, i need to step away for a little bit18:31
MadOtisno worries, thanks for at least getting me this far!18:31
MadOtisHowever... someone may want to update the official docs to include some sort of sidebar note at Centos 7 users may need to manually install libisal (opensource v 2.13 from source available at intel), symlink the .so's into /usr/lib64, ldconfig, before installing openstack-swift.  I would be happy to put together a doc on the steps I performed.18:35
*** geaaru has quit IRC18:36
*** KriskK has quit IRC18:38
tqtranhello, i have a question concerning account acl. lets say i have 2 (sue and cindy) people under project awesome. right now i can give members of project awesome access to my account via the acl. but is it possible to give sue access without giving cindy access?18:39
*** silor has quit IRC18:40
claygMadOtis: oh interesting, we package and install intel's isa-l liberasure library on our cent7 machines, there's something about adding something to /etc/ld.so.conf.d - but I'm not sure how generally applicable that is...18:40
claygMadOtis: may be more of an issue with the libisal package you were using?  documenting the work around on the internet *somewhere* (so it's googleable) would be *helpful* but maybe not sufficient?18:41
claygif there's a bug in the cent7 package - you should probably try to report it to the package maintainers?18:41
claygtqtran: you're using Keystone?  because I don't think that platform supports account acl's?18:42
claygtqtran: i'm not sure exactly what's lacking in swift keystone authorization model to support that?18:43
MadOtisI can do that.  I was just getting errors of unable to find any erasure so's at all.18:43
*** ChubYann has joined #openstack-swift18:45
claygyeah, sounds like the isa-l package isn't getting where ever it's installed fully discoverable...18:45
tqtranclayg: using tempauth atm18:47
tqtranim trying to understand how the json format is being done for tempauth so i can do propose something similar for keystoneauth18:47
claygtqtran: ah neat!18:48
tqtranso i guess my question is.... is there a way to assign access to an individual member of a project?18:48
tqtranor would i have to create one project per user in order to give access to an individual member?18:48
claygtqtran: so I don't think account acl's can *deny* access - so there's have to be something about sue that allows her to be indentified as a swift_ower that's not true for cindy18:49
tqtranright, deny was the wrong word. basically, i want to give sue access but not cindy18:49
claygyes, probably all members of project would have swift_owner to their project18:49
tqtranlets assume projectA contains sue and cindy and they are both admin of that project18:50
*** hseipp has quit IRC18:50
tqtranlets assume I am admin of projectB18:50
tqtranI want to give sue in ProjectA access to projectB18:50
tqtranbut i do NOT want to give cindy in ProjectA access to projectB18:51
tqtranis there a way to do this atm?18:51
*** haypo has quit IRC18:52
claygsure, you don't have to grant access on B to all of A, you can just do it for the specific member of A sue18:52
tqtranwhat does the syntax for that look like? currently i have {read-only:[AUTH_sue] ?18:53
tqtranthat didnt work for me, i had to use {read-only:[AUTH_projectA]}18:54
claygtqtran: I believe the fully qualified name of sue in A in tempauth is "AUTH_projectA:sue"18:55
tqtrani think it tried that too, but let me try again and confirm18:56
tqtranyeah it doesnt work for me, says forbidden18:57
claygidk, wfm -> https://gist.github.com/clayg/4d83ac62846750e22bfd2a3dc9e68b4019:02
claygmy account is AUTH_test, X-Account-Access-Control: {"read-only":["test2:tester2"]}19:02
clayg... granted access to the user tester2 in the account test219:02
claygand that user can stat my AUTH_test account and otherwise do account read-only things19:03
claygmaybe a syntax issue?19:04
claygor sometimes using the swift cli across accounts is burdensome?19:04
*** fifieldt has quit IRC19:08
tqtranlet me try omitting the AUTH_ prefix and see if that work19:09
claygoh... interesting, yeah i'm not sure why AUTH_test2:tester2 doesn't wfm?19:10
*** ukaynar has joined #openstack-swift19:16
*** fifieldt has joined #openstack-swift19:18
tqtranyeah, seems to work without the AUTH prefix19:19
tqtrani guess maybe you only need the prefix for project level? *shrugs*19:20
claygi guess in tempauth users are 'test2:tester2' - and they get automatically added to the groups 'test2' and 'AUTH_test2' - but users don't get their own group 'AUTH_test2:tester2'19:21
claygin tempauth the group extension would allow you to make up a group name "foo" and grant access to that group on your account, then any user you want to read-only you just add to the group "foo"19:22
claygbut for tempauth at least the fully qualified name of sue, is projectA:sue not AUTH_projectA:sue - who knew19:22
claygI'm not sure exactly why the AUTH_test2 group is even added, maybe for backwards compat with something?  There's always some hope that tempauth (and other auth systems modeled after it) have some capability of working pleastly with *other* auth systems in the pipeline by namespacing to the reseller prefix19:23
*** SkyRocknRoll has quit IRC19:24
clayg... but I'm not really sure that holds up if there's a name collision at the project level19:24
tqtranhm.. thanks for the explanation19:26
*** asettle has joined #openstack-swift19:41
*** KriskK has joined #openstack-swift19:47
*** KriskK has quit IRC19:54
*** silor has joined #openstack-swift20:13
*** cdelatte has quit IRC20:17
*** silor1 has joined #openstack-swift20:20
*** silor has quit IRC20:21
*** silor1 is now known as silor20:21
*** ober37 has quit IRC20:27
*** ningliu has quit IRC20:27
*** mminesh has quit IRC20:27
*** bkeller` has quit IRC20:27
openstackgerritMerged openstack/swift: go: make a rate limiter for account per disk/server  https://review.openstack.org/34621120:28
*** mminesh has joined #openstack-swift20:30
*** ningliu has joined #openstack-swift20:30
*** aswadr_ has quit IRC20:31
*** ober37 has joined #openstack-swift20:31
*** bkeller` has joined #openstack-swift20:32
*** ukaynar has quit IRC20:34
*** silor has quit IRC20:39
notmynameMadOtis: ok, so what's the 503 from?20:43
openstackgerritMichael Barton proposed openstack/swift: go: make container timeout configurable  https://review.openstack.org/34332020:43
MadOtisnotmyname: I do an ". demo-openrc", followed by a "swift stat", and the error I pasted in the keystone channel shows up.20:45
notmynameMadOtis: take the transaction ID that's logged (eg tx0f341615abdd4fd09f582-0057991ba1 in the first one) and grep your logs on all servers for it20:46
MadOtisok20:46
MadOtisok, here are several repeated log entries from all 3 nodes (the last one is from the controller, the 1st 2 are from the object nodes): http://paste.openstack.org/show/542693/20:50
*** cdelatte has joined #openstack-swift20:51
*** cutforth has joined #openstack-swift20:51
*** McMurlock1 has quit IRC20:54
*** asettle has quit IRC20:55
*** cutforth has quit IRC20:56
*** ukaynar has joined #openstack-swift20:57
notmynameMadOtis: can you pastebin your proxy server pipeline?20:58
MadOtissure, one sec20:59
MadOtishere you go: http://paste.openstack.org/show/542694/21:00
notmynameMadOtis: what does `swift auth -v` give you?21:04
MadOtispasted here: http://paste.openstack.org/show/542696/21:05
notmynameso that's good21:05
notmynamethat's telling you that it did get to auth21:06
MadOtisok, that's good21:06
notmynamewell, check that `swift auth` (ie no -v) works21:06
MadOtisok21:06
MadOtisI get a URL and TOKEN back, want me to paste them?21:07
notmynamenah21:07
notmynamethat's good though21:07
MadOtisok21:08
notmynameseems you're using local IPs and dummy creds, right?21:08
MadOtisYes, correct.21:08
notmynameok, hten pastebin `swift --debug stat`21:09
MadOtisk21:09
MadOtishttp://paste.openstack.org/show/542698/21:10
notmynamehow are you doing logging? where are the swift logs going?21:12
MadOtisjournalctl and /var/log/swift/swift.log21:13
notmynamepastebin `grep tx8e55822fc10a4011ac02d-0057992302 /var/log/swift/swift.log` (from every server) please21:13
MadOtisok21:13
MadOtisController here: http://paste.openstack.org/show/542699/21:15
MadOtisobject server 2: http://paste.openstack.org/show/542700/21:16
*** jamielennox|away is now known as jamielennox21:17
*** mingyu has joined #openstack-swift21:17
MadOtisobject server 1: http://paste.openstack.org/show/542701/21:17
notmynamedo you have https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L134 set to true or false? please make sure it is set to true21:18
MadOtistrue21:19
*** ukaynar has quit IRC21:19
*** mingyu has quit IRC21:21
notmynameMadOtis: the issue is that the account servers aren't finding the account (which is normal at first) but it's also not getting created automatically21:25
*** flwang1 has joined #openstack-swift21:25
MadOtisI'll paste my entire proxy-server.conf file, if it will help.21:26
notmynamenext thing I'd do is check the account ring to make sure it looks right. then check the account drives.21:27
MadOtisI've copied/pasted my entire proxy config: http://paste.openstack.org/show/542704/  I only changed any password fields (not that it matters).21:28
MadOtisI have no data, so I can delete and re-create all rings, if need be.21:29
notmynamesure why not. TBH I'm not sure at this point21:31
MadOtisis there an easy way to re-create the rings?21:31
MadOtisdo I just delete the .gz files?21:32
*** NM has quit IRC21:33
claygI think allow_account_management has to be true with account_autocreate?21:41
claygMadOtis: ^21:42
MadOtisits worth a try21:42
MadOtisno joy.  Same error21:45
claygare you sure you restarted your proxies ;)21:45
MadOtisyes.  I have only 1 controller, and I did a systemctl restart openstack-swift-proxy.service21:46
MadOtis...or do I have to restart it on the object nodes, too?21:46
claygyou might stop, make sure all the pids are dead, then start21:46
claygno, that setting only effects the proxy servers21:46
MadOtisok21:46
MadOtisI'm just going to be totally sure... I'm going to reboot the server itself.21:47
claygrofl21:48
MadOtissame thing when it came back up.  Swift just hates me.21:51
claygwhy would RL get a 204 on the HEAD and the swiftclient gets the 503...21:52
claygalso, why isn't there more backend 404s?21:54
MadOtisSorry, I can't help you there.  ;)21:54
claygcan you do a cluster wide grep for a transaction id from a 503 again and try to get everything from all the nodes21:54
MadOtissure21:54
MadOtis...I think... not sure what logs to grep on the non-object nodes.  Everything else, so far, is working flawlessly.21:55
mattoliveraumorning21:57
MadOtisscrew it... I'm going to start all over with Ubuntu, even though it's a really old supported version (14.04)21:58
claygi'm not sure the problem is the OS tho...21:58
claygi means swift is running and return an error status code - so... the explination is somewhere in the logs21:59
MadOtisI doubt it, too... I'm just MUCH more familiar with Ubuntu.  I only tried it with Centos because it was a newer OS release, but I've struggled with selinux, and other missing dependencies on Centos so far.22:00
MadOtisI'll just keep plugging along, then.  If for no other reason than to help identify a possible bug, somewhere.22:00
MadOtisI don't even know what logs to look in on the other nodes at this point.  The only nodes with swift on them are the controller, and the two object nodes.22:06
timburkei think it might be time to investigate those 401s. i suspect that the 'swift' keystone user in your authtoken config isn't allowed to validate tokens22:06
notmynameMadOtis: memcache is running, right?22:08
MadOtisYes, it is.22:08
MadOtishow do I see what roles the swift user has?  I did recreate it at one point, wondering if I had the wrong password specified for it.22:10
notmynameMadOtis: that's set in the keystone db. I think there's a command to list it from keystone22:11
MadOtisok, let me see if I can find it.22:11
timburke`openstack role list --user swift` i think22:13
MadOtisno, I get a 404 error when I try that.  And the keystone database is pretty empty with very little data in any of the tables.22:18
MadOtisI see users, roles, domains, etc... but there is very little on any of the other tables, so no real easy way to see who has what role.22:19
*** diogogmt_ has joined #openstack-swift22:33
*** diogogmt has quit IRC22:34
*** diogogmt_ is now known as diogogmt22:34
timburkewhen it 404ed, was that on port 5000 (public) or 35357 (admin)? you'll probably want to through a --debug to confirm (osc will use whatever you've got for your OS_AUTH_URI to get back a catalog of endpoints, then picks whatever's labeled as the admin identity endpoint)22:34
timburkeif your keystone endpoints are misconfigured (like mine are! yay!) osc will then try to do admin operations on the public api port, which leads to a bunch of 404s :-/22:35
MadOtisIIRC, every "xxxx_uri" has been http://controller:5000, and every "xxxx_url" has been http://controller:3535722:36
MadOtisI even checked that to make sure both were correct.22:37
*** jamielennox is now known as jamielennox|away22:45
*** asettle has joined #openstack-swift22:46
claygnotmyname: doug just hit me up with a usecase for an enhancement to encryption - timburke thinks its reasonable too - i was thinking about adding it to lp - but is ideas better?  it's pretty small...22:47
timburkeclayg: lp/wishlist seems fine to me22:48
notmynameclayg: probably a wishlist bug, if you've got neat ideas on how to do it22:48
timburkedoesn't seem like it'll require a whole lot of design or anything22:49
notmynameclayg: IMO ideas is great when it's more like the notes or general thinking on a topic. eg doug would write an ideas thing and clayg would turn it into a wishlist bug ;-)22:49
*** asettle has quit IRC22:51
*** klamath has quit IRC22:54
*** diogogmt has quit IRC22:55
*** diogogmt has joined #openstack-swift22:56
clayglp bug #160711623:01
openstackLaunchpad bug 1607116 in OpenStack Object Storage (swift) "Encryption Feature: Expose 'X-Encrypted-Data' header" [Wishlist,New] https://launchpad.net/bugs/160711623:01
*** siva_krish has joined #openstack-swift23:02
*** siva_krish has quit IRC23:03
*** siva_krish has joined #openstack-swift23:03
*** dmorita_ has joined #openstack-swift23:05
openstackgerritMerged openstack/swift: go: get rid of BackendError  https://review.openstack.org/33465523:06
*** dmorita has quit IRC23:07
*** siva_krish has quit IRC23:08
*** NM has joined #openstack-swift23:09
claygtimburke: tdasilva: I think either of one you would be best equipped to handle a review on patch 27272423:13
patchbotclayg: https://review.openstack.org/#/c/272724/ - swift - Copy headers correctly when copying object23:13
claygit fixes TWO bugs!23:13
claygjrichli and mattoliverau say it's good2go23:13
claygso there's a good chance you'll be able to +A - race!23:14
*** uCHobby has joined #openstack-swift23:14
*** MadOtis has quit IRC23:22
*** kei_yama has joined #openstack-swift23:24
*** nadeem has quit IRC23:29
notmyname"are you speaking 4th dimensionally about an upcoming change?" :-)23:34
*** bill_az has joined #openstack-swift23:38
*** chlong has quit IRC23:38
*** chsc has quit IRC23:39
*** NM has quit IRC23:42
openstackgerritTim Burke proposed openstack/swift: Store SLO Etag and swift_bytes in sysmeta  https://review.openstack.org/34753823:48
*** urth has quit IRC23:51
*** urth has joined #openstack-swift23:53
*** diogogmt has quit IRC23:55
*** adu has joined #openstack-swift23:57
*** asettle has joined #openstack-swift23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!