| *** dasp has joined #openstack-swift | 00:16 | |
| *** dasp_ has quit IRC | 00:18 | |
| *** zaitcev has quit IRC | 01:04 | |
| kota_ | good morning | 01:09 |
|---|---|---|
| kota_ | Jesus, commuting has been restricted. | 01:11 |
| * kota_ is working at home | 01:11 | |
| *** tkajinam has quit IRC | 01:24 | |
| *** tkajinam has joined #openstack-swift | 01:25 | |
| *** rledisez has quit IRC | 02:26 | |
| *** alecuyer has quit IRC | 02:26 | |
| *** alecuyer has joined #openstack-swift | 02:27 | |
| *** rledisez has joined #openstack-swift | 02:28 | |
| *** ChanServ sets mode: +v rledisez | 02:28 | |
| *** alecuyer has quit IRC | 02:30 | |
| *** rledisez has quit IRC | 02:30 | |
| mattoliverau | kota_: oh really. welcome to my world.. though I'm actaully set up for it | 02:31 |
| *** alecuyer has joined #openstack-swift | 02:36 | |
| *** rledisez has joined #openstack-swift | 02:37 | |
| *** ChanServ sets mode: +v rledisez | 02:37 | |
| *** psachin has joined #openstack-swift | 03:34 | |
| *** evrardjp has quit IRC | 05:35 | |
| *** evrardjp has joined #openstack-swift | 05:35 | |
| *** threestrands has joined #openstack-swift | 06:04 | |
| *** threestrands has quit IRC | 06:16 | |
| *** rcernin has quit IRC | 06:47 | |
| *** rdejoux has joined #openstack-swift | 07:29 | |
| *** rdejoux has quit IRC | 07:45 | |
| *** rdejoux has joined #openstack-swift | 07:46 | |
| *** tesseract has joined #openstack-swift | 07:59 | |
| *** ccamacho has joined #openstack-swift | 08:13 | |
| *** manuvakery has quit IRC | 08:14 | |
| *** rpittau|afk is now known as rpittau | 08:18 | |
| *** tkajinam has quit IRC | 08:24 | |
| *** tkajinam has joined #openstack-swift | 09:39 | |
| *** tkajinam has quit IRC | 10:18 | |
| *** mvkr has joined #openstack-swift | 10:28 | |
| *** rpittau is now known as rpittau|bbl | 11:32 | |
| *** tkajinam has joined #openstack-swift | 11:47 | |
| *** manuvakery has joined #openstack-swift | 11:57 | |
| pawan-gupta | Hi, I would like to know, if implementation of BUCKET_POLICY is planned in near future? It would be helpful for me to have an idea about timelines on this. Thanks in advance. | 11:58 |
| *** rdejoux has quit IRC | 12:20 | |
| *** tkajinam has quit IRC | 12:45 | |
| *** rdejoux has joined #openstack-swift | 12:56 | |
| *** rpittau|bbl is now known as rpittau | 13:34 | |
| *** psachin has quit IRC | 13:35 | |
| *** jvisser has joined #openstack-swift | 14:01 | |
| *** jvisser has quit IRC | 14:01 | |
| *** jvisser has joined #openstack-swift | 14:01 | |
| clayg | timburke: p 710492 looks good to me, solid bug fix, thanks for the help! | 14:56 |
| patchbot | https://review.opendev.org/#/c/710492/ - swift - Apply limit to list versioned containers - 4 patch sets | 14:56 |
| clayg | pawan-gupta: there's a number of different s3 style bucket policies - support probably won't be universal initially | 14:57 |
| clayg | pawan-gupta: version expiration is the one that's top of mind for me - what sort of support are you looking for? | 14:57 |
| *** gyee has joined #openstack-swift | 16:15 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Use limit-helper in container-server, too https://review.opendev.org/710815 | 16:20 |
| *** diablo_rojo has quit IRC | 16:40 | |
| *** diablo_rojo has joined #openstack-swift | 16:40 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Use float consistently for proxy timeout settings https://review.opendev.org/710475 | 16:44 |
| *** rpittau is now known as rpittau|afk | 16:59 | |
| timburke | clayg, looking at p 710475 -- is monkey-patching the actual policies like that going to taint any other tests? | 17:09 |
| patchbot | https://review.opendev.org/#/c/710475/ - swift - Use float consistently for proxy timeout settings - 2 patch sets | 17:09 |
| *** tesseract has quit IRC | 17:16 | |
| *** rdejoux has quit IRC | 17:24 | |
| clayg | timburke: I don't think so? teardown resets them to None | 17:30 |
| clayg | oh, and the whole class is using patch'd policies helper | 17:30 |
| clayg | but, yeah I could easily be wrong | 17:30 |
| *** evrardjp has quit IRC | 17:35 | |
| *** evrardjp has joined #openstack-swift | 17:35 | |
| *** rdejoux has joined #openstack-swift | 18:02 | |
| *** mahatic has joined #openstack-swift | 18:34 | |
| *** ChanServ sets mode: +v mahatic | 18:34 | |
| timburke | mahatic! o/ | 18:49 |
| timburke | have a good flight! | 18:49 |
| mahatic | hey timburke, thanks! :) | 18:49 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Clean up concurrent get requests once we have a response https://review.opendev.org/706361 | 19:33 |
| *** guimaluf has joined #openstack-swift | 20:22 | |
| guimaluf | Hi all, I'm struggling with something really wierd. 3 weeks ago I started upgrading my environment from Kilo to Queens. I have only Keystone and Swift. I think between Mitaka and Newton my storages started to throw Permission Denied error while writing metadata. http://paste.openstack.org/show/790217/ I've check everything I could, user, permissions, mount options. But I still getting this error. I wrote a | 20:31 |
| guimaluf | python script for creating files and writing metadata within the swift mount and everything works fines. Don't know what to do anymore. | 20:31 |
| timburke | guimaluf, can you still read existing data? you checked user/group/perms -- was that at the root of the drive, or down in the object directory that gets created? | 20:37 |
| timburke | you can find the object dir with something like `swift-get-nodes /etc/swift/object.ring.gz AUTH_b462852304414da5a0c507ad035717e6/teste/drawing.pdf` | 20:37 |
| guimaluf | timburke, yes, everything is working fine. I can even download objects. but uploading no more. Users and perms were checked many time. I can do everything manually with swift user. This happens in the object directory | 20:41 |
| guimaluf | timburke, I've wrote a small python script to create file and write metadata in the specific object dir | 20:42 |
| clayg | do people still use nose to run swift tests? https://gist.github.com/clayg/c275732de47d64c6f608cc1e63701b45 | 20:48 |
| clayg | ^ I'm having an encoding problems with the test.unit.proxy.controller.test_container module under py2 😢 | 20:49 |
| timburke | so what do the perms look like? 0755, or something else? is there a umask set? have you spot-checked that the swift processes really are running as the expected user? | 20:50 |
| guimaluf | timburke, directory stats: Access: (0744/drwxr--r--) Uid: ( 106/ swift) Gid: ( 113/ swift) | 20:51 |
| tdasilva | clayg: weird, I don't run into that in my env: http://paste.openstack.org/show/790218/ | 20:52 |
| timburke | clayg, what's your $LANG look like? | 20:52 |
| guimaluf | timburke, yes. process are running as swift | 20:53 |
| clayg | wow good thinking @timburke !!! I was LANG=C.UTF-8 and `LANG=C nosetests ...` worked | 20:53 |
| guimaluf | swift 2370 4.3 1.5 1283060 46408 ? Sl 16:27 3:44 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf | 20:53 |
| clayg | @tdasilva @timburke what is your LANG set to? | 20:53 |
| tdasilva | en_US.UTF-8 | 20:53 |
| timburke | clayg, that's like... the opposite of what i was expecting :-/ | 20:54 |
| guimaluf | timburke, I've tried to reformat a full storage node to see what could it be... but with no clue... | 20:55 |
| guimaluf | format the disks I mean | 20:55 |
| guimaluf | not the full node... yet... | 20:55 |
| clayg | @guimaluf and it happened on all disks on all the nodes you upgraded? | 20:59 |
| guimaluf | clayg, AFAIKT yes. because I'm getting the same error on the 3 copies tries in all upload | 21:00 |
| *** zaitcev has joined #openstack-swift | 21:02 | |
| *** ChanServ sets mode: +v zaitcev | 21:02 | |
| timburke | i wonder if it's something to do with O_TMPFILE support... what kernel are you on? | 21:02 |
| guimaluf | timburke, Linux storage-dev-01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | 21:02 |
| timburke | does your script use O_TMPFILE like the object-server would? | 21:03 |
| guimaluf | no | 21:03 |
| guimaluf | but I'm kin to reproduce in that way | 21:03 |
| timburke | just thinking about how https://github.com/openstack/swift/blob/master/CHANGELOG#L1323-L1327 came in newton... | 21:04 |
| guimaluf | hmmm | 21:04 |
| guimaluf | I'll try that | 21:04 |
| guimaluf | timburke, can you point out where is the snippet of code that use that flag>? | 21:05 |
| timburke | might want to check out https://github.com/openstack/swift/blob/stable/queens/swift/obj/diskfile.py#L2600-L2620 | 21:05 |
| guimaluf | trying to figure out how to reproduce that... if you have a easy handy way on mind | 21:12 |
| guimaluf | please share ;| | 21:12 |
| guimaluf | timburke, if you there is an way of debug direct on swift code I'm willing to change here for debug purpose | 21:15 |
| guimaluf | know* | 21:15 |
| clayg | guimaluf: maybe just re-write utils.o_tmpfile_support to always return False? | 21:15 |
| clayg | I guess maybe one question is what does `utils.o_tmpfile_supported` return currently? | 21:16 |
| guimaluf | clayg, it wrote the file | 21:20 |
| guimaluf | it worked... I'll change in both storages to see if upload happens | 21:21 |
| clayg | guimaluf: I bet @tdasilva fixed you -> https://review.opendev.org/#/c/633646/ | 21:22 |
| patchbot | patch 633646 - swift - Change how O_TMPFILE support is detected (MERGED) - 5 patch sets | 21:22 |
| clayg | I mean... that is to say you're not the only person to run in to problems how the o_tmpfile support was detected originally | 21:22 |
| timburke | clayg, idk about that... i don't see EACCES in that list :-( | 21:23 |
| guimaluf | clayg, upload done. it worked | 21:23 |
| timburke | guimaluf, what OS is this? are you using xfs? i'm glad we've got a workaround, but i'd love to find out more about what's causing the error... | 21:24 |
| guimaluf | timburke, ubuntu 18.04 upgraded from 14.04. yes xfs | 21:25 |
| clayg | timburke: yeah maybe you're right! could be un-related | 21:25 |
| clayg | or it could express different on ubuntu - which might be worse - in that maybe this bug still exists? | 21:25 |
| guimaluf | I'm really keen to help on report this | 21:25 |
| clayg | I have 18.04 in dev, my kernel is like 4.15 - so guimaluf probably should be able to use o_tmpfile | 21:27 |
| clayg | maybe turning it off isn't the greatest workaround 🤔 | 21:27 |
| timburke | what's your swift user's umask? what are the mount options you're using? | 21:27 |
| clayg | still it's hard/annoying to debug old code, the most useful think beyond being able to reproduce would be checking if the problem is expressed when using the latest tip of master swift | 21:28 |
| guimaluf | timburke, umask 0022; xfs noatime,nodiratime,logbufs=8 | 21:29 |
| guimaluf | I'm on Rocky on that storage... I can move forward the upgrade and check if it's fixed | 21:29 |
| guimaluf | but may be related with this ubuntu upgrade process... | 21:30 |
| guimaluf | clayg, I don't trust that a upgraded ubuntu is the same as a brand new 18.04 :/ | 21:31 |
| guimaluf | clayg, this patch you mentioned is present in what release? | 21:34 |
| timburke | stein | 21:34 |
| timburke | 2.21.0 | 21:34 |
| guimaluf | ok! I'll move to it | 21:36 |
| clayg | guimaluf: well, I guess timburke is right that the patch may not be what fixes you - we're not even sure if you should or should not be using linkat? I guess ... currently using linkat didn't work (so there's that) | 21:37 |
| clayg | what about just rebooting so all the kernel upgrady bits can fall into place? | 21:37 |
| guimaluf | I'm on Rocky version... I'll try to move to stein | 21:38 |
| guimaluf | clayg, I rebooted many times in this finding for the reason of this problem :/ so all kernel upgrades are done | 21:39 |
| guimaluf | clayg, timburke thank you very much, fellows, for helping me with this, I hope after the upgrade things get back to normal, without hacking the swift code. | 21:46 |
| mattoliverau | Morning | 21:49 |
| timburke | guimaluf, clayg: fwiw, here's what i came up with as a quick write-tester: http://paste.openstack.org/show/790221/ | 21:56 |
| guimaluf | timburke, bang! same error | 22:00 |
| guimaluf | timburke, http://paste.openstack.org/show/790223/ | 22:01 |
| guimaluf | clayg, timburke, it haven't fixed on stein :( I'm leaving now, but I'll be back tomorrow and I can give you all information about my environment so we can nail this down. thank you very much for all help and effort | 22:02 |
| *** guimaluf has quit IRC | 22:09 | |
| timburke | tomorrow guimaluf oughta add a `st = os.fstat(dfw._fd); print('%r (mode=0o%o)' % (st, st.st_mode))` between the write() and the put()... | 22:27 |
| *** tkajinam has joined #openstack-swift | 22:28 | |
| *** rcernin has joined #openstack-swift | 23:01 | |
| *** DHE has quit IRC | 23:28 | |
| seongsoocho | morning | 23:51 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!