| *** gyee has quit IRC | 00:19 | |
| *** DHE has joined #openstack-swift | 00:34 | |
| *** two_tired2 has joined #openstack-swift | 00:42 | |
| openstackgerrit | Sam Morrison proposed openstack/swift master: WIP: s3 secret caching https://review.openstack.org/603529 | 01:35 |
|---|---|---|
| kota_ | good morning | 02:09 |
| *** jistr has quit IRC | 03:00 | |
| *** jistr has joined #openstack-swift | 03:00 | |
| kota_ | mattoliverau: btw do you have time to look at p 447129 ? I and rledisez like to make it progressed but me also worried you may find any issue with your good eyes. If you have any concerns so far, I'd like to help your reviews. | 03:31 |
| patchbot | https://review.openstack.org/#/c/447129/ - swift - Configure diskfile per storage policy - 20 patch sets | 03:31 |
| openstackgerrit | Sam Morrison proposed openstack/swift master: s3 secret caching https://review.openstack.org/603529 | 03:51 |
| *** two_tired2 has quit IRC | 04:21 | |
| *** rcernin has quit IRC | 05:30 | |
| *** rcernin has joined #openstack-swift | 05:30 | |
| *** silor has joined #openstack-swift | 05:35 | |
| *** geaaru has joined #openstack-swift | 05:44 | |
| *** rcernin_ has joined #openstack-swift | 06:04 | |
| *** rcernin has quit IRC | 06:06 | |
| *** kirubak has joined #openstack-swift | 06:08 | |
| kirubak | ?help | 06:11 |
| kirubak | i am trying to build swift-rocky for xenial, do we have the location of the debian package that i can use to build rocky on xenial ? | 06:12 |
| *** rcernin has joined #openstack-swift | 06:34 | |
| *** rcernin_ has quit IRC | 06:36 | |
| kota_ | kirubak: I'm not sure ubuntu/debian package is actively maintained. Instead, you can find tar ball of python package around http://tarballs.openstack.org/swift/ | 06:48 |
| kota_ | kirubak: specifically, http://tarballs.openstack.org/swift/swift-stable-rocky.tar.gz is the rocky stable one. | 06:49 |
| sorrison | kirubak: ubuntu package up 2.19.0 in their cloud archive for bionic | 06:57 |
| *** rcernin has quit IRC | 07:06 | |
| zigo | notmyname: Around? Or sleeping already? | 07:10 |
| *** hoonetorg has quit IRC | 07:17 | |
| *** mikecmpbll has joined #openstack-swift | 07:21 | |
| *** hoonetorg has joined #openstack-swift | 07:30 | |
| *** gkadam has joined #openstack-swift | 07:36 | |
| *** e0ne has joined #openstack-swift | 07:36 | |
| *** mikecmpbll has quit IRC | 07:43 | |
| *** e0ne has quit IRC | 07:48 | |
| *** gkadam has quit IRC | 07:57 | |
| *** mikecmpbll has joined #openstack-swift | 08:01 | |
| *** e0ne has joined #openstack-swift | 08:25 | |
| *** e0ne has quit IRC | 08:26 | |
| *** e0ne has joined #openstack-swift | 08:33 | |
| *** e0ne has quit IRC | 09:02 | |
| seongsoocho | :-( When I add a new node, object-replicator(and rsync) degrades the performance of object-server... | 09:22 |
| seongsoocho | There are a lots of small size files in object server and utilization of disks are almost 80 | 09:22 |
| seongsoocho | 80%.. Could this be the cause? | 09:23 |
| alecuyer | seongsoocho: It could be. How many files do you have per object-server, and how much RAM ? if all inodes cannot be cached, you may see performance degradation | 09:27 |
| *** e0ne has joined #openstack-swift | 10:47 | |
| *** kirubak has quit IRC | 12:04 | |
| *** patchbot has quit IRC | 12:07 | |
| *** patchbot has joined #openstack-swift | 12:08 | |
| *** e0ne has quit IRC | 12:20 | |
| *** DHE has quit IRC | 12:46 | |
| *** silor has quit IRC | 12:48 | |
| seongsoocho | alecuyer: sorry for late reply. It is almost 100milion files per object servers and 64GB RAM | 12:51 |
| *** DHE has joined #openstack-swift | 12:55 | |
| *** kukacz_ is now known as kukacz | 13:00 | |
| *** geaaru has quit IRC | 13:41 | |
| notmyname | zigo: good morning. what's up? | 14:43 |
| zigo | notmyname: Everything's fine ! :) | 14:44 |
| notmyname | great! | 14:44 |
| zigo | notmyname: I got my cluster setting-up correctly ... | 14:44 |
| zigo | notmyname: I'm writting a provisionning tool, which is puppet-openstack. | 14:44 |
| zigo | notmyname: The first thing that I'm doing is a swift cluster. | 14:44 |
| zigo | So far, so good ... | 14:44 |
| zigo | :) | 14:44 |
| zigo | notmyname: I just discovered this though: https://bugs.debian.org/909171 | 14:45 |
| notmyname | cool | 14:45 |
| openstack | Debian bug 909171 in python3-swiftclient "python3-swift never exits" [Important,Open] | 14:45 |
| zigo | Any idea what could be doing this? | 14:45 |
| zigo | onovy: ^ | 14:46 |
| notmyname | no that sounds odd. may be good to test with curl to see if it's a particular server-side thing or something in the client | 14:46 |
| zigo | notmyname: That's not the issue. It works perfectly with openstackclient or the python 2 version of swiftclient. | 14:47 |
| zigo | (Debian has both clients...) | 14:47 |
| zigo | So, this looks like a Py3 problem with the cli. | 14:47 |
| notmyname | ok. just trying to find a way to make it simpler to debug. IIRC there's a --debug flag you can pass to the CLI. maybe that will reveal something | 14:48 |
| zigo | Nothing that I saw helped with --debug. | 14:48 |
| notmyname | that's unfortunate | 14:48 |
| zigo | I'm doing a redeployment from scratch, I can maybe give you the output of --debug later when it's deployed again in my PoC. | 14:51 |
| onovy | as I wrote to bug, I can't reproduce it | 14:52 |
| onovy | BUT I tested agains swauth, not keystone | 14:52 |
| onovy | so maybe it's keystoneauth related? | 14:52 |
| onovy | AND I tested @sid, zigo probably @stretch | 14:53 |
| zigo | yeah, stretch | 15:00 |
| zigo | Using keystoneauth indeed. | 15:00 |
| *** gyee has joined #openstack-swift | 15:37 | |
| notmyname | AJaeger: done | 15:46 |
| *** pcaruana has joined #openstack-swift | 15:47 | |
| *** e0ne has joined #openstack-swift | 16:07 | |
| *** e0ne has quit IRC | 16:17 | |
| *** pcaruana has quit IRC | 16:27 | |
| *** e0ne has joined #openstack-swift | 16:31 | |
| *** mikecmpbll has quit IRC | 16:34 | |
| *** e0ne has quit IRC | 16:35 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3 secret caching https://review.openstack.org/603529 | 16:53 |
| AJaeger | thanks, notmyname | 17:52 |
| AJaeger | swift cores, small cleanup for zuul.yaml, could you import https://review.openstack.org/#/c/600732/ , please? | 17:54 |
| patchbot | patch 600732 - swift - Use templates for cover and lower-constraints - 1 patch set | 17:54 |
| tdasilva | AJaeger: just for the sake of my understanding, in this case a template is a "zuul job template", not to be confused with an 'ansible template', correct? | 18:08 |
| AJaeger | tdasilva: yes | 18:09 |
| AJaeger | tdasilva: one as listed here: https://docs.openstack.org/infra/openstack-zuul-jobs/project-templates.html | 18:09 |
| notmyname | AJaeger: I'm looking forward to having these sort of patches settle down. I love the move to put stuff in each project repo. maybe these patches are necessary because of that. but there's no real way for me to validate them or know what's right or wrong. so all I can really do is think "oh, some -infra proposed this, gate seems to pass, I guess I should +A". | 18:13 |
| notmyname | I'll land this, but I'm worried about maintainability | 18:13 |
| notmyname | tdasilva: timburke: clayg: I don't know if you feel the same way ^^ | 18:15 |
| AJaeger | notmyname: I'm happy to answer questions - now or also later... | 18:15 |
| AJaeger | notmyname: I understand your concerns, my change was made to make maintainability easier for you. | 18:15 |
| AJaeger | notmyname: feel free to drop in anytime in #openstack-infra | 18:16 |
| notmyname | yeah, I understand. I'm just hoping the setup phase is nearly done :-) | 18:16 |
| AJaeger | notmyname: I'm not aware of anything else coming (famous last words ;) | 18:16 |
| notmyname | lol | 18:16 |
| *** silor has joined #openstack-swift | 18:17 | |
| *** zaitcev has joined #openstack-swift | 18:17 | |
| *** ChanServ sets mode: +v zaitcev | 18:17 | |
| tdasilva | AJaeger: for this particular patch, what's the benefit of the template? over listing the actual job? | 18:18 |
| tdasilva | i think i got it: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n503 | 18:18 |
| timburke | i just realized: we almost certainly shouldn't be using the generic openstack-tox-lower-constraints job -- it isn't going to get us an xfs tmpdir, so we skip ~1400 tests | 18:19 |
| AJaeger | tdasilva: cosmetics ;) It gives us the chance to change jobs in one place instead of in each repo | 18:19 |
| AJaeger | timburke: oh, indeed - yes | 18:19 |
| tdasilva | basically the template already lists the job(s) to run, but also where to run (i.e., check, gate, etc) | 18:19 |
| AJaeger | tdasilva: yes, exactly | 18:19 |
| notmyname | timburke: good call. but shoudl we replace the other one? or just add a new one? | 18:20 |
| AJaeger | want me to propose a followup? or will one of you do it? | 18:20 |
| notmyname | I guess the generic one doesn't do much that's useful if we've got another that uses xfs | 18:20 |
| AJaeger | notmyname: agreed | 18:20 |
| tdasilva | zaitcev: just a reminder about swift community meeting later today... | 18:24 |
| zaitcev | tdasilva: thanks | 18:25 |
| zaitcev | timburke: I always do TMPDIR=/var/tmp tox -e py27 | 18:25 |
| notmyname | timburke: are you in the middle of stuff? will you be able to make a better constraints job for us? or should I work on it? | 18:25 |
| zaitcev | Someone wrote me about my kernel patch to have xattr on tmpfs while we were in PTG | 18:26 |
| notmyname | I /think/ all that's needed is to inherit from the upstream on, set the envvar in .zuul.yaml then use that job instead of the generic one in the queues | 18:27 |
| zaitcev | People still need that, but nobody is willing to put in enough work. Kinda like container sharding used to be. | 18:27 |
| notmyname | zaitcev: everything needs a sponsor | 18:27 |
| zaitcev | notmyname: I can work out a transfer to kernel team... | 18:28 |
| zaitcev | now that Tim is on py3 and Clay is rewriting PUT+POST with verbs | 18:28 |
| zaitcev | not much left to do | 18:28 |
| notmyname | nah, we still *really* need you with py3 work | 18:28 |
| timburke | notmyname: i'm working on something else at the moment; if you could work on it, that'd be great | 18:29 |
| notmyname | timburke: kk | 18:29 |
| *** mikecmpbll has joined #openstack-swift | 18:29 | |
| notmyname | AJaeger: did my comment earlier about how to make a better coverage job sound right? | 18:29 |
| AJaeger | notmyname: yeah, should work - give it a try ;) feel free to add me as reviewer | 18:30 |
| notmyname | AJaeger: thanks | 18:30 |
| timburke | zaitcev: speaking of py3, i might have you take a look at my alternative approach to p 592761 -- see p 602822 / p 602823 | 18:30 |
| patchbot | https://review.openstack.org/#/c/592761/ - swift - py3: Adapt direct_client and internal_client - 7 patch sets | 18:30 |
| patchbot | https://review.openstack.org/#/c/602822/ - swift - py3: port direct_client - 2 patch sets | 18:30 |
| patchbot | https://review.openstack.org/#/c/602823/ - swift - py3: port internal_client - 2 patch sets | 18:30 |
| notmyname | AJaeger: should the "irrelevant-files" directive match for the check and gate queues? I see some things where it doesn't right now | 18:38 |
| AJaeger | notmyname: your call ;) I would match them in general so that the same jobs really runs in same situations. | 18:40 |
| timburke | notmyname: pretty sure i caused some of that. i tried to capture my thinking in https://github.com/openstack/swift/blob/2ddc979e1d6fa680e8867249e11224d2bb1ca477/.zuul.yaml#L183-L186 | 18:48 |
| notmyname | timburke: I see where you're coming from. was wondering from AJaeger on the aesthetics or The Way It Should Be (tm) view ;-) | 18:49 |
| openstackgerrit | John Dickinson proposed openstack/swift master: Use templates for cover and lower-constraints https://review.openstack.org/600732 | 18:53 |
| openstackgerrit | John Dickinson proposed openstack/swift master: set up a lower constraints job that uses an XFS tmpdir https://review.openstack.org/603870 | 18:53 |
| openstackgerrit | John Dickinson proposed openstack/swift master: set up a lower constraints job that uses an XFS tmpdir https://review.openstack.org/603870 | 18:55 |
| *** silor has quit IRC | 19:16 | |
| openstackgerrit | Fatema Khalid Sherif proposed openstack/swift master: Enabling direct_client users to overwrite X-Timestamp https://review.openstack.org/603261 | 19:45 |
| notmyname | FYI the CI jobs are super backed up. clarkb sent an email about what's going on and some ways to help. put simply, there's flaky tests and a bunch of repeated failures tracked on http://status.openstack.org/elastic-recheck/index.html that aren't resolved. I've checked the various gate tracking pages, and there's not really anything that affects swift right now | 20:11 |
| notmyname | (well, one transient zuul error two days ago) | 20:12 |
| notmyname | the team meeting is still happening today (in 37 minutes from now. I won't be leading it, but tdasilva will | 20:24 |
| notmyname | I've put patchbot in there already | 20:24 |
| notmyname | I'll be around before and after the meeting, but I have a conflicting personal commitment I need to take care of today | 20:24 |
| notmyname | fatema_: oh hi! you're pushing patches. how's it going? any questions? | 20:30 |
| mattoliverau | kota_: I'll try and find time to look today sometime. I'm on leave this week, but will try to give it another pass before I can review better first thing next week. If you can wait that long. | 20:31 |
| fatema_ | notmyname: well I am still trying to understand the context of the bug as the implementation is there. But there has been objections about the context | 20:34 |
| fatema_ | thank you for checking :D | 20:34 |
| notmyname | fatema_: please do not hesitate to ask in here or in gerrit when you have questions. | 20:35 |
| notmyname | fatema_: and if you're not always logged into irc, services like https://www.irccloud.com can make it easier. clayg uses that one | 20:36 |
| timburke | idk about "objections" -- but it'd definitely help to have clayg chime in with some more guidance (since he reported the bug) | 20:36 |
| clayg | which bug | 20:36 |
| clayg | irccloud.com is great - and I can use it on my phone and I don't have to pay for a bouncer | 20:37 |
| timburke | https://launchpad.net/bugs/1757250 | 20:37 |
| openstack | Launchpad bug 1757250 in OpenStack Object Storage (swift) "direct_client gen_headers is inconsistent" [Wishlist,New] | 20:37 |
| clayg | I used it free for awhile, but ultimately the $5/mo was cheaper than running irssi in a cloud vm so I'm very happy | 20:37 |
| clayg | there's even a desktop client I use on my ubuntu laptop! | 20:37 |
| timburke | so like, how valid is the use-case "i don't want to send any x-timestamp and let the server figure it out instead"? | 20:37 |
| fatema_ | notmyname: I am using irccloud now and it's very nice. | 20:38 |
| timburke | clayg: personally, i kinda wanna get rid of both add_ts and add_user_agent | 20:39 |
| * notmyname is out for a bit | 20:39 | |
| notmyname | I'll catch up on meeting logs later | 20:39 |
| timburke | and have the helper always add those headers if absent | 20:39 |
| notmyname | thansk tdasilva! | 20:39 |
| clayg | yeah I think the intent was more clear with "ensure_ts" | 20:40 |
| fatema_ | if I am understanding this rightously, then I see that no need for the user to be able to override those two variable as timburke says | 20:40 |
| clayg | I believe there was a specific add to direct_client recenty-ish where acoles had to twist gen_headers because it wasn't letting him say what he needed explicitly | 20:41 |
| clayg | i'd be around the same time the bug was filed | 20:41 |
| clayg | if there a need for more context | 20:41 |
| clayg | anyway, I think half the job was not just re-write gen_headers - but make it called in all the direct_client methods | 20:41 |
| clayg | some methods like "get_XXX" can ensure_ts=False as they don't require a X-Timestamp and shouldn't add one unless the caller passes through headers | 20:42 |
| timburke | yeah, i still don't want that :P i want a no-arg helper so x-timestamp is always in the header dict. and if we *really* want to support not sending an x-timestamp, make the caller pass it as None or something | 20:42 |
| clayg | which I think is the big motivation for the change - not all direct_client methods support headers kwarg because of this opinionated stuff in gen_headers | 20:43 |
| clayg | maybe i'm mis-remembering | 20:43 |
| timburke | seems to make sense. maybe i could be talked out of my stance. but passing x-timestamp can cause differences even on GET... | 20:44 |
| timburke | (iirc) | 20:44 |
| clayg | timburke: it's not obvious to me why you care if an ignored header get's sent along with the safe http requests ... oh ... CAN IT? | 20:44 |
| fatema_ | ok I was still on partial-bug fix but still not getting why woul the user need to override x-timestamp ? | 20:44 |
| clayg | maybe that was the problem... let me find that other review | 20:45 |
| timburke | well, i'm thinking of expired-but-not-yet-reaped objects... i *think* changing x-timestamp can turn that 404 back to a 200 | 20:45 |
| fatema_ | ok this seems legit timburke | 20:46 |
| clayg | meh, looks like the issue was more about the user_agent p 565745 | 20:47 |
| patchbot | https://review.openstack.org/#/c/565745/ - swift (feature/deep-review) - Add shard range support to container server (MERGED) - 11 patch sets | 20:47 |
| clayg | IIRC the thinking at the time was - why should the method ever overwrite the existing ts/agent - shouldn't the kwarg be more like "ensure_XXX" | 20:47 |
| clayg | 🤷♂️ | 20:48 |
| clayg | @timburke that's surprising to me if that's true | 20:48 |
| clayg | disk_file.open(current_time=req_timestamp) ... interesting | 20:50 |
| timburke | clayg: i think it might be needed for reconstruction? in case there's a POST that kicked out the expiration... or something | 20:51 |
| clayg | p 532595 | 20:51 |
| patchbot | https://review.openstack.org/#/c/532595/ - swift - Use X-Timestamp when checking object expiration (MERGED) - 7 patch sets | 20:51 |
| clayg | There was no associated bug, look like it made sense to Sam & Tim tho | 20:51 |
| clayg | i guess it's stupid for one server to 404 and another 200 so... that's fine | 20:52 |
| clayg | so maybe remove add_ts and instead just `if 'x-timestamp' not in ...` that's fine | 20:52 |
| fatema_ | ok how is the bug and the patches related ? | 20:53 |
| clayg | we still have methods like e.g. `direct_head_container` that don't use hdrs_in when they call gen_headers - I think that's mostly what the bug is about ifxing | 20:54 |
| clayg | fatema_: read the bug and commit messages very carefully and ask a more specific question please. 😁 | 20:54 |
| *** m_kazuhiro has joined #openstack-swift | 20:55 | |
| fatema_ | clayg: ok will do | 20:55 |
| clayg | 👍 sorry!!! 😊 | 20:55 |
| kota_ | morning | 20:57 |
| timburke | fatema_: so some background on x-timestamp and how it relates to eventual consistency. 'cause i think it might be handy regardless :-) | 20:57 |
| timburke | x-timestamp is how swift orders events. a client does PUT, POST, PUT to the same object, but because of failures individual object servers may only see PUT, POST or PUT, PUT and find out about the missing events during replication | 20:57 |
| timburke | so think about the latter one: server sees a PUT, stores it. sees another PUT, overwrites it. asynchronously, it finds out there was a POST -- but should it actually apply the metadata? | 20:57 |
| timburke | by assigning timestamps at the proxy for each request, we get a more-or-less well-ordered system: PUT at t0, POST at t1, PUT at t2. the object-server can see that the POST at t1 was superseded by the PUT at t2 and basically ignores it during replication | 20:57 |
| tdasilva | meeting time @ #openstack-meeting | 21:00 |
| tdasilva | clayg, zaitcev joining us? | 21:02 |
| clayg | zaitcev: I'd be cool to discuss more immediately after the meeting - but like I said, I don't have any ideas I'm in love with right now | 21:37 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Use latest eventlet in probe tests https://review.openstack.org/602526 | 21:59 |
| clayg | zaitcev: so I have a few - but honesty it's 5pm here so I need to run in a bit - we could schedule something for later in the week - or get an email thread going? | 22:01 |
| kota_ | btw, anyone knows where we can find our team photos? | 22:02 |
| clayg | I'm commited to finishing STAGE+UPDATE+COMMIT to the point where all the tests pass, but less concerned about the polish considering that I don't really think it's the the right path forward over all | 22:02 |
| tdasilva | kota_: yeah i had seen a link for them | 22:03 |
| tdasilva | and added to my swift photo album | 22:03 |
| *** m_kazuhiro has quit IRC | 22:03 | |
| kota_ | tdasilva: nice, could you share it? | 22:03 |
| kota_ | i wanna get it | 22:04 |
| tdasilva | kota_: https://photos.app.goo.gl/CwuHgqN6bJMpGzBw8 | 22:04 |
| kota_ | tdasilva: much appreciated! | 22:04 |
| tdasilva | i'm trying to find team photos from 2017 ptg denver to add there | 22:05 |
| zaitcev | clayg: just hit me with a review link... add me as a reviewer | 22:05 |
| clayg | zaitcev: rock on - as soon as I get it up | 22:05 |
| clayg | then we can regroup | 22:05 |
| tdasilva | ok, gotta run too | 22:06 |
| * notmyname back | 22:06 | |
| notmyname | tdasilva: thanks! | 22:06 |
| tdasilva | notmyname: yw | 22:07 |
| kota_ | tdasilva: i may have the picture on my lap top in my office, trying to find out to share | 22:09 |
| kota_ | probably tomorrow in my time because of a dayoff today. | 22:09 |
| notmyname | tdasilva: kota_: http://d.not.mn/swift_team_denver_ptg.jpeg and http://d.not.mn/swift_team_denver_ptg_hats.jpeg | 22:10 |
| kota_ | nice work notmyname | 22:10 |
| fatema_ | according to this patch 532595, if I understand correctly this means it is safer/ preferred to send x-timestamp, is this right ? | 22:16 |
| patchbot | https://review.openstack.org/#/c/532595/ - swift - Use X-Timestamp when checking object expiration (MERGED) - 7 patch sets | 22:16 |
| openstackgerrit | Sam Morrison proposed openstack/swift master: s3 secret caching https://review.openstack.org/603529 | 22:26 |
| sorrison | timburke: now with unit tests :-) | 22:26 |
| timburke | thanks sorrison! i'll try to take a look... soon. not quite sure when, unfortunately. kota_ might be interested too, though | 22:31 |
| sorrison | cheers | 22:32 |
| *** hoonetorg has quit IRC | 22:41 | |
| *** rcernin has joined #openstack-swift | 22:44 | |
| *** mikecmpbll has quit IRC | 22:51 | |
| *** hoonetorg has joined #openstack-swift | 22:58 | |
| *** mvkr has joined #openstack-swift | 23:10 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
