Thursday, 2019-07-11

openstackgerritTim Burke proposed openstack/python-swiftclient master: Authors/changelog for 3.8.0  https://review.opendev.org/67021500:01
openstackgerritTim Burke proposed openstack/python-swiftclient master: Authors/changelog for 3.8.0  https://review.opendev.org/67021500:02
*** tdasilva has quit IRC00:08
*** gyee has quit IRC00:47
*** zaitcev has quit IRC01:17
*** zaitcev has joined #openstack-swift01:29
*** ChanServ sets mode: +v zaitcev01:29
*** irclogbot_1 has joined #openstack-swift02:06
*** irclogbot_1 has quit IRC02:13
*** irclogbot_3 has joined #openstack-swift02:17
*** altlogbot_2 has joined #openstack-swift02:18
*** irclogbot_3 has quit IRC02:19
*** mvkr_ has joined #openstack-swift02:23
*** mvkr has quit IRC02:25
*** zaitcev_ has joined #openstack-swift02:25
*** ChanServ sets mode: +v zaitcev_02:25
*** zaitcev has quit IRC02:29
*** rcernin has quit IRC02:29
*** BjoernT has quit IRC02:34
*** altlogbot_2 has quit IRC02:34
*** rcernin has joined #openstack-swift02:46
*** abelur has joined #openstack-swift03:20
*** irclogbot_3 has joined #openstack-swift03:20
*** psachin has joined #openstack-swift03:21
abelurAnyone knows how to set read-acl ".r:*,.rlistings" on a container with a few object? I want to have public http access to the container with a few objects created on the openstack cloud.03:21
abelurI am trying this with python openstackclient, latest version. Does not seem like the container create has "--read-acl"  option?03:24
*** irclogbot_3 has quit IRC03:25
*** psachin has quit IRC03:38
*** rcernin has quit IRC03:39
*** altlogbot_0 has joined #openstack-swift03:54
*** rcernin has joined #openstack-swift03:56
*** altlogbot_0 has quit IRC03:59
*** viks___ has joined #openstack-swift04:14
*** BjoernT has joined #openstack-swift04:16
*** pcaruana has joined #openstack-swift04:55
openstackgerritMerged openstack/swift master: py3: add swift-dsvm-functional-py3 job  https://review.opendev.org/65354805:02
abelurok, I was able to create a container and set the required acls to get public access ...05:04
abelurhttps://object-storage-ca-ymq-1.vexxhost.net/swift/v1/12c36e260d8e4bb2913965203b1b491f/test-container/05:04
abeluranyone around to anwer a few questions on container/objects and how we can view the object on the bowser. For ex, if I have a bunch of .html|.txt files (static content), I dont want this to be download locally but viewable on the browser05:06
openstackgerritMerged openstack/swift master: Run all func tests under py2 against services running py3  https://review.opendev.org/66694205:08
*** zaitcev_ has quit IRC05:09
*** zaitcev_ has joined #openstack-swift05:21
*** ChanServ sets mode: +v zaitcev_05:21
openstackgerritMerged openstack/swift feature/losf: Get new unit tests running under py3  https://review.opendev.org/66442405:25
*** ccamacho has quit IRC05:33
*** zaitcev_ has quit IRC05:36
openstackgerritMerged openstack/swift master: py3: port staticweb and domain_remap func tests  https://review.opendev.org/64792305:40
*** BjoernT has quit IRC05:49
*** zaitcev_ has joined #openstack-swift05:49
*** ChanServ sets mode: +v zaitcev_05:49
timburkeabelur, looks like the cluster has staticweb enabled -- you should be able to POST X-Container-Meta-Web-Listings: true to the container to enable listings05:53
timburkesee https://docs.openstack.org/swift/latest/middleware.html#staticweb -- there are a few other options to customize the listing05:53
openstackgerritTim Burke proposed openstack/swift feature/losf: Merge branch 'master' into feature/losf  https://review.opendev.org/66800706:00
abelurtimburke: can I do this thru the openstack client CLI?06:01
timburke...maybe? i usually do it via swiftclient or curl06:02
timburke`openstack container set --property web-listings=true test-container` seems promising06:04
abelurtimburke: that changed the web browser listing ... however06:10
abelurwhat I am trying to do is more like https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/odlparent-releases-verify-master/9/06:10
abelurwe want ot view the robot tests or even the console log in the browser.06:10
abelurrather than downloading the logs06:11
openstackgerritTim Burke proposed openstack/swift master: Authors/changelog for 2.22.0  https://review.opendev.org/66899006:11
abelurtimburke: is there anything that can be enabled on the container or swift to make things work for view the logs on the browser.06:12
abelurtimburke: what I am looking at is something similar to the S3 bucket + cloud front integration ?06:14
*** ccamacho has joined #openstack-swift06:14
timburkeabelur, what are the content-types of the objects? i know if i set them to something like text/plain, they'll display in browser fine06:17
*** altlogbot_2 has joined #openstack-swift06:18
abelurtimburke: there are mostly .html/.log files06:20
timburkealternatively, you could try explicitly setting content-disposition to inline -- but i think some content-types (like application/octet-stream) will still cause most browsers to prompt to download06:20
abelurhttps://www.irccloud.com/pastebin/xqaI83t1/06:20
timburkeok, so looking at the error.log.gz you've got in that test-container, i see 'Content-Type: binary/octet-stream'06:21
timburkewe want Content-Type: text/plain and Content-Encoding: gzip06:22
abeluris this something we can set with --property on CLI?06:23
*** altlogbot_2 has quit IRC06:23
*** altlogbot_3 has joined #openstack-swift06:25
abelurhttps://www.irccloud.com/pastebin/9PsSRXVh/06:25
* abelur cut'n paste, formatting got messed :(06:26
timburkeso it wouldn't be set at the container level, but rather per-object. i'm not sure that openstack cli supports object metadata that isn't X-Object-Meta-* though06:28
*** altlogbot_3 has quit IRC06:29
timburkemost direct thing might be to get a hold of a raw auth token, then send some curl requests like `curl -H 'X-Auth-Token: <token>' -X POST -H 'Content-Type: text/plain' -H 'Content-Encoding: gzip' <url>`06:33
abeluryeah, there is quite a bit of diff in the way openstack CLI vs swift cli does/outputs the container properties06:36
abelurtimburke: I give this a try and get back06:36
abelurtimburke: thanks a ton :-))06:36
timburkei'll likely be asleep -- getting late for me. but i'll be interested in reading how it went! good luck!06:37
abelursure thing, i'll keep you posted.06:38
abelurtimburke: what TZ are you in? I'm mostly in aest tz, wondering what is the best time to reachout06:38
timburkePDT, so getting on towards midnight now06:39
timburkemattoliverau might be a good person to talk to though -- he's near melbourne if i remember right06:40
abelurcool 👍06:41
timburkeor kota_ in japan06:41
mattoliverauoh hey, someone used my name :) o/06:48
mattoliverauI'm in Aest too \o/06:48
mattoliverauSorry was distracted with work.06:49
openstackgerritTim Burke proposed openstack/swift master: reconciler: Enqueue right work for shard containers  https://review.opendev.org/67023906:53
*** stamps has joined #openstack-swift06:56
*** altlogbot_2 has joined #openstack-swift06:57
openstackgerritTim Burke proposed openstack/swift master: py3: port dlo func tests  https://review.opendev.org/64292006:58
*** altlogbot_2 has quit IRC07:01
*** ianychoi_ is now known as ianychoi07:01
openstackgerritzengjia proposed openstack/swift master: Update auth_url in install docs  https://review.opendev.org/67000507:04
*** ccamacho has quit IRC07:12
*** ccamacho has joined #openstack-swift07:12
*** irclogbot_3 has joined #openstack-swift07:17
*** rcernin has quit IRC07:17
*** irclogbot_3 has quit IRC07:21
*** irclogbot_1 has joined #openstack-swift07:25
*** rdejoux has joined #openstack-swift07:25
*** irclogbot_1 has quit IRC07:27
*** irclogbot_2 has joined #openstack-swift07:53
*** irclogbot_2 has quit IRC07:55
*** Fidde has joined #openstack-swift08:06
*** irclogbot_2 has joined #openstack-swift08:09
*** tkajinam has quit IRC08:10
*** irclogbot_2 has quit IRC08:13
*** gkadam has joined #openstack-swift08:28
*** ianychoi has quit IRC08:54
*** FlorianFa has joined #openstack-swift09:36
*** ianychoi has joined #openstack-swift09:38
abelurmattoliverau: cool 👍09:39
abelurmattoliverau: I've a few questions on the containers/object I've tried what tim.burke suggested, however this returns a `ERR_CONTENT_DECODING_FAILED` while trying to access the container after setting the content-type and content-encoding.09:40
*** irclogbot_2 has joined #openstack-swift09:49
*** irclogbot_2 has quit IRC09:51
abelurUnexpected exception for https://object-storage-ca-ymq-1.vexxhost.net/v1/12c36e260d8e4bb2913965203b1b491f/test-container: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',))09:58
*** psachin has joined #openstack-swift10:13
*** altlogbot_2 has joined #openstack-swift10:27
*** altlogbot_2 has quit IRC10:29
*** altlogbot_1 has joined #openstack-swift10:35
*** altlogbot_1 has quit IRC10:39
*** altlogbot_0 has joined #openstack-swift10:41
*** irclogbot_3 has joined #openstack-swift10:41
*** altlogbot_0 has quit IRC10:45
*** irclogbot_3 has quit IRC10:45
*** altlogbot_2 has joined #openstack-swift10:57
*** altlogbot_2 has quit IRC11:01
*** tesseract has joined #openstack-swift11:21
*** altlogbot_0 has joined #openstack-swift11:57
abelurtim.burke: just for the record, it worked https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/12c36e260d8e4bb2913965203b1b491f/test-container-3/error.log.gz12:00
*** altlogbot_0 has quit IRC12:01
*** zaitcev__ has joined #openstack-swift12:06
*** ChanServ sets mode: +v zaitcev__12:06
*** zaitcev_ has quit IRC12:09
*** henriqueof has joined #openstack-swift12:23
*** henriqueof has quit IRC12:28
*** irclogbot_3 has joined #openstack-swift12:33
*** irclogbot_3 has quit IRC12:35
*** zaitcev_ has joined #openstack-swift12:41
*** ChanServ sets mode: +v zaitcev_12:41
*** zaitcev__ has quit IRC12:44
*** jdewald has quit IRC12:51
*** altlogbot_0 has joined #openstack-swift12:52
*** altlogbot_0 has quit IRC12:53
*** zaitcev_ has quit IRC13:13
*** zaitcev_ has joined #openstack-swift13:25
*** ChanServ sets mode: +v zaitcev_13:25
*** BjoernT has joined #openstack-swift13:31
*** irclogbot_2 has joined #openstack-swift13:41
*** new_student1411 has joined #openstack-swift13:43
*** irclogbot_2 has quit IRC13:45
*** BjoernT_ has joined #openstack-swift13:46
*** BjoernT has quit IRC13:47
*** Fidde has quit IRC13:51
new_student1411timburke thanks for your input. I am able to find out the account server, but a POST request fails with bad request. I am using:14:03
*** ccamacho has quit IRC14:03
new_student1411`curl -g -I -X POST "http://<server>:6202/<device>/<partition>/<account_id>" -H "X-Account-Sysmeta-Global-Write-Ratelimit: 2"`14:04
*** baojg has quit IRC14:28
new_student1411My bad timburke, I am able to set it. Was missing `X-Timestamp`14:51
*** pcaruana has quit IRC15:11
*** gyee has joined #openstack-swift15:23
*** tdasilva has joined #openstack-swift15:41
*** ChanServ sets mode: +v tdasilva15:41
*** pcaruana has joined #openstack-swift15:57
*** Fidde has joined #openstack-swift16:04
*** altlogbot_2 has joined #openstack-swift16:06
*** Fidde has quit IRC16:06
*** altlogbot_2 has quit IRC16:07
*** zaitcev_ has quit IRC16:15
timburkenew_student1411, oh good! sorry; i don't often make direct requests like that, so i'd forgotten to mention it16:16
timburkewe should really *really* make that something that reseller admins can do through a normal proxy :-/16:16
*** Fidde has joined #openstack-swift16:22
timburkeabelur, whoa! what!? you can set content-encoding on a *container*? i'd meant it for each *object* -- i didn't even realize you *could* set that on a container!16:24
timburkei certainly can't do that on my all-in-one...16:26
timburkeah... heh. https://object-storage-ca-ymq-1.vexxhost.net/info claims swift version 12.2.11 -- which is to say, it's not really swift, but speaks the swift api ;-)16:28
timburkeat least it seems to support staticweb!16:28
tdasilvatimburke: yeah, that's ceph rgw16:32
*** tesseract has quit IRC16:36
*** altlogbot_0 has joined #openstack-swift16:45
*** gkadam has quit IRC16:46
timburketime to blow some minds! https://review.opendev.org/#/c/670353/16:49
patchbotpatch 670353 - devstack - Remove Swift from default DISABLED_PYTHON3_PACKAGES - 1 patch set16:49
*** altlogbot_0 has quit IRC16:49
*** psachin has quit IRC16:52
*** d34dh0r53 has quit IRC17:05
*** altlogbot_3 has joined #openstack-swift17:24
*** tdasilva has quit IRC17:24
*** altlogbot_3 has quit IRC17:25
*** altlogbot_2 has joined #openstack-swift17:31
*** sasregulus has joined #openstack-swift17:33
*** altlogbot_2 has quit IRC17:35
*** altlogbot_1 has joined #openstack-swift17:37
openstackgerritTim Burke proposed openstack/swift master: sharding: Cache shard ranges for object writes  https://review.opendev.org/66720317:40
*** altlogbot_1 has quit IRC17:41
*** d34dh0r53 has joined #openstack-swift17:42
*** tdasilva has joined #openstack-swift17:43
*** ChanServ sets mode: +v tdasilva17:43
*** altlogbot_2 has joined #openstack-swift17:45
*** altlogbot_2 has quit IRC17:45
*** Fidde has quit IRC18:07
*** rdejoux has quit IRC18:45
openstackgerritMerged openstack/swift master: Update auth_url in install docs  https://review.opendev.org/67000519:00
*** irclogbot_0 has joined #openstack-swift19:29
*** irclogbot_0 has quit IRC19:33
*** altlogbot_2 has joined #openstack-swift19:40
*** altlogbot_2 has quit IRC19:41
*** jistr has quit IRC19:43
*** jistr has joined #openstack-swift19:45
*** new_student1411 has quit IRC19:54
*** zaitcev has joined #openstack-swift19:58
*** ChanServ sets mode: +v zaitcev19:58
*** jistr has quit IRC20:03
*** jistr has joined #openstack-swift20:05
*** zaitcev_ has joined #openstack-swift20:06
*** ChanServ sets mode: +v zaitcev_20:06
*** zaitcev has quit IRC20:10
*** irclogbot_0 has joined #openstack-swift20:14
*** irclogbot_0 has quit IRC20:16
*** pcaruana has quit IRC20:31
*** BjoernT_ has quit IRC20:39
*** altlogbot_0 has joined #openstack-swift21:01
*** altlogbot_0 has quit IRC21:05
*** irclogbot_0 has joined #openstack-swift21:24
*** irclogbot_0 has quit IRC21:26
*** irclogbot_0 has joined #openstack-swift21:29
*** zaitcev_ has quit IRC21:30
*** irclogbot_0 has quit IRC21:32
claygtimburke: how do you do srings again?21:36
claygJul 11 21:35:06 saio proxy-server: '\xe8\x94\x8a\xe9\x90\x8a\xe3\x9c\x86\xe8\x94\x8b\xeb\x8c\x8e\xe5\x98\x85\xe8\x94\x8a\xef\x84\x91\xe8\x94\x8b\xe5\x98\x88-versions' (txn: tx612eefb2f9114559bf936-005d27ab8a)21:36
claygJul 11 21:35:06 saio proxy-server: '%E8%94%8A%E9%90%8A%E3%9C%86%E8%94%8B%EB%8C%8E%E5%98%85%E8%94%8A%EF%84%91%E8%94%8B%E5%98%88-versions' (txn: tx612eefb2f9114559bf936-005d27ab8a)21:36
claygthe top one is wsgi_unquote(<json>.encode('utf8'))21:36
claygthe second one is a header21:37
timburkeeh? that... looks right... what about them?21:37
claygI need to say 'if a == b' do the right thing21:37
timburkeif it's in a header, it oughta be quoted -- otherwise you're gonna have a hard time with container names that have newlines21:37
claygand currently i'm only doing the right thing when it's not UTF821:37
timburkewsgi_unquote the header, too21:38
timburkei *think* that's what you want, anyway. why are we unquoting something coming out of json though?21:39
timburkeand wait, does wsgi_unquote *work* on an encoded value on py3? i don't *think* it should...21:39
claygyeah that was it21:40
timburkewhat i'd probably do: wsgi_to_bytes(wsgi_unquote(header)).decode('utf-8') == json.loads(value)21:40
timburkeso get me to bytes so i can decode and compare everything in unicode21:41
claygok, so that *also* worked21:44
*** zaitcev_ has joined #openstack-swift21:44
*** ChanServ sets mode: +v zaitcev_21:44
claygis equivilence better in unicode?21:44
claygI still don't think I understand the "type" of "wsgi strings" - or i just can't hold the difference between them from py2 to py3 in my head...21:46
timburkebytes on py2, unicode on py3. only, not useful unicode :P21:46
*** zaitcev_ has quit IRC22:13
*** altlogbot_3 has joined #openstack-swift22:19
*** irclogbot_0 has joined #openstack-swift22:19
*** altlogbot_3 has quit IRC22:23
*** irclogbot_0 has quit IRC22:24
*** zaitcev_ has joined #openstack-swift22:26
*** ChanServ sets mode: +v zaitcev_22:26
openstackgerritTim Burke proposed openstack/python-swiftclient master: Authors/changelog for 3.8.0  https://review.opendev.org/67021522:27
*** zaitcev_ has quit IRC22:44
*** tkajinam has joined #openstack-swift22:52
*** altlogbot_2 has joined #openstack-swift22:58
*** altlogbot_2 has quit IRC22:59
*** irclogbot_1 has joined #openstack-swift23:01
abelurtimburke: actually, dint set thoes on the container, just did it on the file, like you suggested.23:02
abelurtimburke: setting it on the container returned a content decoding errors, and make the containers inaccessable23:03
timburkeabelur, yeah, that was the unexpected part for me: i didn't think you *could* set content-encoding for containers ;-)23:05
abelurtimburke: Is there a TTL option on the container or objects? this is to say that the container should be cleaned up at 3 months automagically without having another cronjob do this.23:05
*** irclogbot_1 has quit IRC23:06
abelurtimburke: the problem was after setting and unsetting the content headers on the container level ... this made the container inaccessable. not sure if its a bug or feature :)23:06
timburkeglad to see test-container-3 got you squared, though. you might be able to fix up the other guys by POSTing an empty content-encoding, or specify 'identity'23:06
abeluraah, gotcha23:07
timburkewith normal swift, you can set X-Delete-At and X-Delete-After on objects to have them get cleaned up. presumably rgw supports that, too?23:08
timburkesee https://docs.openstack.org/swift/latest/overview_expiring_objects.html23:08
*** irclogbot_3 has joined #openstack-swift23:09
*** rcernin has joined #openstack-swift23:13
mattoliveraumorning23:14
abelurmattoliverau: GM :)23:15
*** irclogbot_3 has quit IRC23:16
abelurtimburke: can I do a set X-Delete-At and X-Delete-After on the entirity of the container recursively, assuming there are objects?23:16
timburkenot really -- though i've played around with the idea of having it get set automatically as data is uploaded with https://review.openstack.org/#/c/342857/23:18
patchbothttps://review.opendev.org/#/c/342857/ - swift - Add defaulter middleware - 7 patch sets23:18
abelurtimburke: the reason is we've cron jobs that remove entire folder pertaining to job logs and artifacts after a time period, and having another cron job to do a container delete is more jobs to be maintained :/23:19
openstackgerritMerged openstack/swift master: functests: Make test_PUT_metadata less flakey  https://review.opendev.org/67020923:19
openstackgerritMerged openstack/swift feature/losf: Merge branch 'master' into feature/losf  https://review.opendev.org/66800723:19
timburkeabelur, how much control do you have over the uploader? can you get it to set an expiration date then, or is that an unknown at upload time?23:21
mattoliverauwell you can't set containers to have an X-Delete-At, though I guess if you used a container and put logs under a psudo container (ie prefix all the objects), then when all the objects are removed so would the pseudo container.. though that isn't pretty. Setting X-Delete-At or After as you ingest makes more sense. Then have a reaper cron that looks for and cleans up empty containers. That's only 1 cronjob at least.23:23
abelurtimburke: we can set an expire time during the upload of the object and container (if the feature is available), not being able to set the X-del after time period on the container level makes additional admin tasks, since our env would end up creating about 5k-10k containers everyday.23:24
abelurin that case wondering if containers are time stampted, so that they can be removed?23:28
*** sasregulus has quit IRC23:29
mattoliveraunothing stopping you from timestamping them in the name, and then get the container listing and filter by prefix to get the containers you want. Get the response in json so you can parse it and you should get a number of objects in the container too. NOTE: deleting a container with objects still in it will fail. you can only delete an empty container.23:37
abelurmattoliverau: oh yeah, realized that yesterday :(23:40
abelurnon-empty container deleting straight away does not work. I do see that there is recursive option for this23:40
abelurnot sure what is the use of that23:40
abelurmattoliverau: what is the life of a container once create and populated with some objects? are these syncd to the storage everytime new objects are added?23:45

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!