| torgomatic_ | yep, py3 won't decode that, so on py3, check_utf8 simplifies to try: candidate.decode("utf-8") except UnicodeDecodeError: return False | 00:03 |
|---|---|---|
| *** lifeless_ has joined #openstack-swift | 00:06 | |
| *** lifeless has quit IRC | 00:06 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Have check_drive raise ValueError on errors https://review.openstack.org/504341 | 00:15 |
| mattoliverau | timburke: lol, I spend a bunch of yesterday reading the same thing about wsgi strings, and been hacking the contraints patch to do almost the same, though yours is a little smarter (re:latin1 encoding), from an quick initial look. I'll play with it today. | 00:15 |
| timburke | mattoliverau: sorry, i didn't mean to trample all over your patch -- i just sometimes find it easier to start from master and see where i get with py3 :-/ | 00:17 |
| mattoliverau | timburke: no its fine, I'm still new to all this py3 stuff. And the patch was stuck. | 00:19 |
| mattoliverau | it was zaitcev that pinged the patch, and so I had another go and did more reading rather then just getting tests to pass. | 00:19 |
| mattoliverau | but was still unsure about the latin stuff. so I appreciate the time you put in. But confirms to me we're on the right track :) | 00:20 |
| timburke | have i mentioned before how weird it feels to be paying *so much* attention to types in a dynamically typed language? | 00:21 |
| timburke | it's all getting better as a result... it's just painful. especially since we (apparently) have to think in terms of FOUR different types of strings | 00:22 |
| mattoliverau | +100, all the unicode, bytestring, native string in 2 different versions does my head in sometimes :P | 00:22 |
| torgomatic_ | seriously; writing code that works with both bytes/str in py3 and contagious Unicode in py2 is a royal PITA | 00:23 |
| timburke | (and that's not even getting into the quoted vs unquoted debacle that lead me down the https://review.openstack.org/#/c/571905/ patch chain...) | 00:23 |
| patchbot | patch 571905 - swift - Fix quoting for large objects | 00:23 |
| torgomatic_ | Agreed. I either don't want to think about the types of things and just jam string-like things together and have it work, *or* I want explicit, compiler-enforced types on these four string-ish things so I'm not playing whack-a-mole with type errors. | 00:49 |
| zaitcev | https://github.com/zaitcev/slasti/commit/42f73d700604ff5533e25633a0c49e9c288e6589#diff-6b651c1bae4c691328dbf80764c69372R155 | 00:50 |
| *** two_tired has joined #openstack-swift | 00:54 | |
| openstackgerrit | Merged openstack/swift master: specify test selector with py3 packages in bindep https://review.openstack.org/576993 | 01:05 |
| *** gyee has quit IRC | 01:25 | |
| *** two_tired has quit IRC | 01:27 | |
| *** links has joined #openstack-swift | 01:33 | |
| vpc | +acloes | 01:50 |
| vpc | +acoles are you there | 01:50 |
| vpc | +mattolievarau | 01:57 |
| vpc | +kota_ | 01:57 |
| *** lifeless_ has quit IRC | 02:26 | |
| *** vpc has quit IRC | 02:39 | |
| *** kei_yama has quit IRC | 02:39 | |
| *** itlinux has joined #openstack-swift | 02:43 | |
| *** lifeless has joined #openstack-swift | 02:43 | |
| *** psachin has joined #openstack-swift | 02:46 | |
| *** spsurya has joined #openstack-swift | 02:51 | |
| *** rcernin has quit IRC | 02:52 | |
| *** kei_yama has joined #openstack-swift | 02:53 | |
| *** rcernin has joined #openstack-swift | 03:10 | |
| *** mwheckmann has quit IRC | 03:24 | |
| *** vpc has joined #openstack-swift | 03:24 | |
| vpc | hi guys | 03:24 |
| mattoliverau | vpc: o/ | 03:30 |
| vpc | o/ | 03:32 |
| vpc | Hi | 03:32 |
| vpc | +mattoliverau | 03:32 |
| vpc | My swift_proxy_server container cant start due to error: No module named webhook | 03:33 |
| mattoliverau | have you installed the webhook middleware? | 03:35 |
| mattoliverau | the proxy needs to be able to find the middleware via an extry point, so needs to be installed in your python path somehow. where is the webhook middleware coming from | 03:36 |
| mattoliverau | vpc: ^ | 03:36 |
| *** kei_yama has quit IRC | 03:45 | |
| vpc | ahm no | 03:47 |
| vpc | i didnt install it | 03:47 |
| vpc | on swift_proxy_Server container | 03:47 |
| vpc | i added it on proxy-server.conf | 03:47 |
| vpc | +mattoliverau how can i install if it if i cannot access the container? | 03:55 |
| *** rcernin has quit IRC | 03:55 | |
| mattoliverau | why can't you access the container? you should be able to ssh or join or something to the container right? I guess I don't understand enough about the environment you've set up | 03:57 |
| *** kei_yama has joined #openstack-swift | 04:02 | |
| *** germs has quit IRC | 04:07 | |
| *** lifeless has quit IRC | 04:08 | |
| *** rcernin has joined #openstack-swift | 04:14 | |
| *** lifeless has joined #openstack-swift | 04:14 | |
| *** lifeless has quit IRC | 04:20 | |
| *** lifeless has joined #openstack-swift | 04:20 | |
| *** prasen has quit IRC | 04:22 | |
| *** lifeless has quit IRC | 04:25 | |
| *** lifeless has joined #openstack-swift | 04:26 | |
| *** torgomatic_ has quit IRC | 04:27 | |
| *** StevenK has quit IRC | 04:29 | |
| *** lifeless has quit IRC | 04:30 | |
| *** cshastri has joined #openstack-swift | 04:30 | |
| *** lifeless has joined #openstack-swift | 04:31 | |
| *** cshastri_ has joined #openstack-swift | 04:33 | |
| *** lifeless has quit IRC | 04:36 | |
| *** cshastri has quit IRC | 04:36 | |
| *** cshastri_ has quit IRC | 04:36 | |
| *** lifeless has joined #openstack-swift | 04:37 | |
| *** cshastri has joined #openstack-swift | 04:39 | |
| *** lifeless has quit IRC | 04:41 | |
| *** lifeless has joined #openstack-swift | 04:43 | |
| *** lifeless has quit IRC | 04:47 | |
| *** lifeless has joined #openstack-swift | 04:48 | |
| *** lifeless has quit IRC | 04:52 | |
| *** lifeless has joined #openstack-swift | 04:54 | |
| *** prasen has joined #openstack-swift | 04:55 | |
| *** lifeless has quit IRC | 04:58 | |
| *** lifeless has joined #openstack-swift | 04:59 | |
| *** lifeless has quit IRC | 05:04 | |
| *** lifeless has joined #openstack-swift | 05:04 | |
| *** lifeless has quit IRC | 05:09 | |
| *** lifeless has joined #openstack-swift | 05:10 | |
| *** lifeless has quit IRC | 05:14 | |
| *** lifeless has joined #openstack-swift | 05:15 | |
| *** ccamacho has quit IRC | 05:21 | |
| *** silor has joined #openstack-swift | 05:21 | |
| *** torgomatic has joined #openstack-swift | 05:31 | |
| *** ChanServ sets mode: +v torgomatic | 05:31 | |
| *** links has quit IRC | 05:32 | |
| *** cshastri has quit IRC | 05:38 | |
| vpc | i disable first the webhook in pipeline +mattoliverau. How can i install the webhook | 05:38 |
| *** psachin has quit IRC | 05:38 | |
| mattoliverau | first where are you getting it from? what was the link to the guide your following again? | 05:39 |
| vpc | here | 05:39 |
| vpc | https://github.com/stmuraka/OpenStackSwift-OpenWhisk/tree/master/OpenStackSwift/Webhook | 05:39 |
| *** vpc_ has joined #openstack-swift | 05:42 | |
| vpc_ | hi +mattoliverau sorry i dc | 05:42 |
| vpc_ | here is the link https://github.com/stmuraka/OpenStackSwift-OpenWhisk/tree/master/OpenStackSwift/Webhook | 05:42 |
| *** vpc has quit IRC | 05:44 | |
| mattoliverau | vpc_: ok, so how is swift installed in the proxy container? is the source repo checked out somewhere and then pip installed? | 05:47 |
| mattoliverau | the howto is adding the middleware directly to the swift source. So if your following that, you need to add it to the source, then use something like pip to install it. | 05:48 |
| vpc_ | im using kolla | 05:55 |
| vpc_ | and it have image | 05:55 |
| vpc_ | docker image | 05:55 |
| *** links has joined #openstack-swift | 05:56 | |
| *** silor has quit IRC | 05:57 | |
| vpc_ | where can i find the middleware source code? | 05:57 |
| vpc_ | since i cannot put in the dokcer image | 05:58 |
| mattoliverau | ok, I haven't used docker, but the container image has swift installed inside it. you'll need to get the webhook middleware in there as part of the code or the proxy wont be able to find it as it isn't apart of swift. | 05:58 |
| mattoliverau | sorry I mean I haven't used kolla | 05:58 |
| mattoliverau | when deploying with kolla, does it deploy it's services git code, ie git? and can you change this repository? | 05:59 |
| mattoliverau | becuase you could get into the container and attempt to update the code there, or if you can give kolla another repo, you can clone the swift repo and add the code and then point kolla to it. | 06:00 |
| vpc_ | im just pulling the image from dockerhub | 06:01 |
| *** cshastri has joined #openstack-swift | 06:01 | |
| vpc_ | can i just install the code via pip? | 06:01 |
| mattoliverau | that will install the code, but pip has upstream code, so without the webhook. | 06:02 |
| mattoliverau | the webhook middleware you want to use, isn't apart of upstream swift, so you have to get it in there | 06:02 |
| mattoliverau | https://docs.openstack.org/kolla/latest/admin/image-building.html#build-openstack-from-source | 06:02 |
| mattoliverau | ^ that seems to suggest you could point it at some git repo. | 06:03 |
| mattoliverau | so you can use kolla build. | 06:03 |
| vpc_ | what is the git repo the middleware? | 06:05 |
| mattoliverau | which middleware? the webhook? its the link you sent me :) | 06:06 |
| mattoliverau | looks like that kolla build link says you can point it local. So you could checkout the upstream swift code somethere, add the webhook middleware then point kolla at it to build the image | 06:07 |
| *** StevenK has joined #openstack-swift | 06:07 | |
| vpc_ | i already added the webhook.py on swift/common/middleware folder | 06:07 |
| vpc_ | is that what you mean | 06:08 |
| vpc_ | ? | 06:08 |
| mattoliverau | to where? in the proxy container? | 06:09 |
| vpc_ | yes | 06:09 |
| vpc_ | on all the swift container | 06:09 |
| mattoliverau | in site-packages? | 06:10 |
| *** psachin has joined #openstack-swift | 06:10 | |
| mattoliverau | or some other folder (noting I don't use kolla so not sure of their folder layout) | 06:10 |
| vpc_ | here | 06:11 |
| vpc_ | swift/common/middleware | 06:11 |
| mattoliverau | full path? | 06:11 |
| vpc_ | thats where all the py script is | 06:12 |
| vpc_ | http://paste.openstack.org/show/723995/ | 06:13 |
| mattoliverau | in the /swift folder can you try and run: pip install . | 06:14 |
| mattoliverau | or something. | 06:14 |
| mattoliverau | if the proxy isn't finding it, then it isn't either in the python patch (hopefully pip install will put it there) or the proxy hasn't been restarted to look for it | 06:15 |
| vpc_ | pip install what | 06:16 |
| mattoliverau | . | 06:16 |
| vpc_ | Directory '.' is not installable. File 'setup.py' not found. | 06:16 |
| mattoliverau | ok so it isn't the repo, just the source code. | 06:16 |
| mattoliverau | what does ls /swift show (in a paste) | 06:16 |
| vpc_ | http://paste.openstack.org/show/723996/ | 06:17 |
| mattoliverau | that is the repo | 06:17 |
| mattoliverau | so: cd /swift && pip install . | 06:17 |
| mattoliverau | because there is a setup.py there | 06:18 |
| vpc_ | http://paste.openstack.org/show/723997/ | 06:18 |
| mattoliverau | nice, looks like it installed it. so try and restart the proxy and see if it can find the webhook | 06:19 |
| vpc_ | wait | 06:22 |
| *** cshastri has quit IRC | 06:23 | |
| *** ccamacho has joined #openstack-swift | 06:24 | |
| *** cshastri has joined #openstack-swift | 06:27 | |
| *** mvk has joined #openstack-swift | 06:44 | |
| *** gkadam has joined #openstack-swift | 07:05 | |
| *** tesseract has joined #openstack-swift | 07:11 | |
| *** bharath1234 has joined #openstack-swift | 07:12 | |
| *** armaan has joined #openstack-swift | 07:20 | |
| *** links has quit IRC | 07:21 | |
| *** links has joined #openstack-swift | 07:21 | |
| *** pcaruana has joined #openstack-swift | 07:22 | |
| *** geaaru has joined #openstack-swift | 07:42 | |
| *** links has quit IRC | 07:42 | |
| *** geaaru_ has joined #openstack-swift | 07:43 | |
| *** geaaru_ has quit IRC | 07:46 | |
| *** geaaru_ has joined #openstack-swift | 07:47 | |
| *** rcernin has quit IRC | 07:47 | |
| *** geaaru_ has quit IRC | 07:49 | |
| *** links has joined #openstack-swift | 07:55 | |
| *** bharath1234 has quit IRC | 08:03 | |
| *** cbartz has joined #openstack-swift | 08:04 | |
| *** hseipp has joined #openstack-swift | 08:07 | |
| *** vpc_ has quit IRC | 08:14 | |
| *** bharath1234 has joined #openstack-swift | 08:28 | |
| *** vpc has joined #openstack-swift | 09:24 | |
| vpc | +mattoliverau im reconfigure sorry wait | 09:24 |
| bharath1234 | timburke: All the functional tests are running fine when i run them now. But yesterday for some reason, they didnt work. i am not entirely sure if i can reproduce them also | 09:33 |
| bharath1234 | i have eventlet 0.22.1 | 09:34 |
| acoles | bharath1234: some of the tox test envs run the functional tests against an 'in process' swift i.e. the test sets up an instance of a swift cluster using a predetermined config, so does not require you to have a running swift cluster. For example, [testenv:func-ec]. That can be a useful cross check if the func tests are failing on your SAIO. | 09:39 |
| *** rcernin has joined #openstack-swift | 09:39 | |
| bharath1234 | I also get broken pipe errors when running the probe tests. They are traced to the eventlet package. | 09:39 |
| bharath1234 | acoles:yup i ll run tox tests now | 09:40 |
| acoles | note that probe tests require a very specific cluster setup i.e. nodes, policies and rings as per vanilla SAIO installation | 09:42 |
| acoles | if you change your SAIO rings then probe tests may fail | 09:42 |
| openstackgerrit | Yash Murty proposed openstack/slogging master: Fix log source issue of Public and Service category https://review.openstack.org/577112 | 09:44 |
| bharath1234 | acoles: I did play around a bit with the ring-builder commands and created some new rings outside the SAIO installation | 09:44 |
| acoles | not sure that would explain broken pipe errors though | 09:45 |
| bharath1234 | acoles: I only got 3 broken pipe errors. I ll post the entire error message on the launchpad and put the link here | 09:45 |
| acoles | ok | 09:45 |
| bharath1234 | https://bugs.launchpad.net/swift/+bug/1778028 | 09:52 |
| openstack | Launchpad bug 1778028 in OpenStack Object Storage (swift) "Broken pipe errors during probetests" [Undecided,New] | 09:52 |
| bharath1234 | please do tell if more context is required | 09:52 |
| *** rcernin has quit IRC | 09:53 | |
| *** silor has joined #openstack-swift | 09:54 | |
| *** spsurya has quit IRC | 09:58 | |
| *** silor has quit IRC | 10:05 | |
| *** silor has joined #openstack-swift | 10:07 | |
| acoles | bharath1234: I may not get time to look into that today | 10:08 |
| bharath1234 | acoles: cool cool. whenever you have the time | 10:15 |
| *** silor has quit IRC | 10:30 | |
| *** mpasserini has joined #openstack-swift | 10:40 | |
| *** rcernin has joined #openstack-swift | 10:55 | |
| *** bharath1234 has quit IRC | 11:04 | |
| *** bharath1234 has joined #openstack-swift | 11:14 | |
| *** mpasserini has quit IRC | 11:15 | |
| *** vpc has quit IRC | 11:16 | |
| *** rcernin has quit IRC | 11:19 | |
| *** armaan_ has joined #openstack-swift | 11:24 | |
| *** armaan has quit IRC | 11:27 | |
| *** bharath1234 has quit IRC | 11:38 | |
| *** rcernin has joined #openstack-swift | 11:39 | |
| *** rcernin has quit IRC | 11:44 | |
| *** bharath1234 has joined #openstack-swift | 12:03 | |
| *** mpasserini has joined #openstack-swift | 12:15 | |
| *** prasen has quit IRC | 12:16 | |
| *** mpasserini1 has joined #openstack-swift | 12:17 | |
| *** mpasserini has quit IRC | 12:20 | |
| *** mpasserini1 has left #openstack-swift | 12:24 | |
| *** cshastri has quit IRC | 12:41 | |
| *** links has quit IRC | 12:43 | |
| *** kei_yama has quit IRC | 12:43 | |
| *** mwheckmann has joined #openstack-swift | 12:46 | |
| *** armaan_ has quit IRC | 13:06 | |
| *** armaan has joined #openstack-swift | 13:08 | |
| bharath1234 | the value of the swift version has not changed in the swift-ring-builder file in my /usr/local/bin (being specific) folder | 13:36 |
| bharath1234 | after pulling the master code to update the version, when i ran ring-builder, i got a distribution error. This was solved by changing the __requires__ = 'swift==2.18.1.dev97'. Is that something we have to deal with at every new release? | 13:37 |
| *** bharath1234 has quit IRC | 13:37 | |
| acoles | you probably need to run 'pip install -e .' (note the . at end) | 13:39 |
| acoles | after pulling new version | 13:40 |
| *** bharath1234 has joined #openstack-swift | 13:43 | |
| *** bharath1234 has quit IRC | 13:43 | |
| *** spsurya has joined #openstack-swift | 13:44 | |
| *** jistr is now known as jistr|mtg | 13:49 | |
| *** ccamacho1 has joined #openstack-swift | 13:54 | |
| *** ccamacho has quit IRC | 13:54 | |
| *** armaan has quit IRC | 14:00 | |
| *** armaan has joined #openstack-swift | 14:00 | |
| *** silor has joined #openstack-swift | 14:00 | |
| *** bharath1234 has joined #openstack-swift | 14:03 | |
| *** links has joined #openstack-swift | 14:17 | |
| *** links has quit IRC | 14:23 | |
| *** itlinux has quit IRC | 14:31 | |
| *** armaan has quit IRC | 14:37 | |
| *** armaan has joined #openstack-swift | 14:37 | |
| *** zhongjun__ has joined #openstack-swift | 14:55 | |
| *** pcaruana has quit IRC | 14:56 | |
| *** hseipp has quit IRC | 14:57 | |
| *** hseipp has joined #openstack-swift | 15:03 | |
| *** silor has quit IRC | 15:03 | |
| *** silor has joined #openstack-swift | 15:04 | |
| *** gkadam has quit IRC | 15:17 | |
| *** itlinux has joined #openstack-swift | 15:19 | |
| *** silor has quit IRC | 15:19 | |
| *** hseipp has quit IRC | 15:22 | |
| *** bharath1234 has quit IRC | 15:23 | |
| *** silor has joined #openstack-swift | 15:33 | |
| *** jistr|mtg is now known as jistr | 15:36 | |
| *** jistr is now known as jistr|off | 15:47 | |
| *** ianychoi has joined #openstack-swift | 15:48 | |
| *** geaaru has quit IRC | 15:51 | |
| *** tesseract has quit IRC | 16:09 | |
| *** cbartz has left #openstack-swift | 16:11 | |
| *** gyee has joined #openstack-swift | 16:12 | |
| *** silor has quit IRC | 16:16 | |
| timburke | good morning | 16:18 |
| notmyname | good morning | 16:21 |
| *** d0ugal has quit IRC | 16:23 | |
| *** spsurya has quit IRC | 16:28 | |
| *** mwheckmann has quit IRC | 16:31 | |
| *** ccamacho1 has quit IRC | 16:36 | |
| acoles | clayg: zaitcev torgomatic : some thoughts re object server version 'discovery' https://etherpad.openstack.org/p/swift-put-post | 16:38 |
| *** armaan has quit IRC | 16:38 | |
| zaitcev | acoles: I'll have a look. At present I do not understand just what you don't like about expanding the path. | 16:39 |
| acoles | clayg: option 4 is what I was describing last night but since then a simple option 3 occurred to me | 16:39 |
| *** armaan has joined #openstack-swift | 16:39 | |
| acoles | zaitcev: fair point, I'll add some comment to the etherpad | 16:40 |
| *** raopajay has joined #openstack-swift | 16:46 | |
| *** armaan has quit IRC | 16:54 | |
| *** armaan has joined #openstack-swift | 16:54 | |
| *** armaan has quit IRC | 16:57 | |
| *** psachin has quit IRC | 16:58 | |
| *** mwheckmann has joined #openstack-swift | 17:08 | |
| *** geaaru has joined #openstack-swift | 17:18 | |
| *** hseipp has joined #openstack-swift | 17:27 | |
| *** bharath1234 has joined #openstack-swift | 17:44 | |
| *** germs has joined #openstack-swift | 18:05 | |
| *** germs has quit IRC | 18:05 | |
| *** germs has joined #openstack-swift | 18:05 | |
| *** hseipp has quit IRC | 18:05 | |
| *** germs has quit IRC | 18:10 | |
| timburke | hmm... it occurs to me, log analysis may be an argument in favor of new verbs... at least for the POSTs? though, i suppose if you're interested in what *clients* are doing, you're probably just going to look at proxy-server logs, not object-server... | 18:20 |
| *** gkadam has joined #openstack-swift | 18:36 | |
| torgomatic | timburke: good point; you'd also get statsd logging so you could see the timings of the different steps of an encrypted-EC-object-PUT request. Re-using PUT and POST means we'd have to do extra work to break the stats out. | 18:37 |
| torgomatic | Probably not much extra work, but a non-zero amount. | 18:37 |
| timburke | i feel like reusing PUT may not be the worst thing? *shrug* | 18:39 |
| *** notkevin has joined #openstack-swift | 18:42 | |
| notkevin | I have noticed errors like the following on my swift cluster: | 18:45 |
| notkevin | rsyncd[313]: rsync: mkstemp "/37/objects/612077/eed/956ed42383a57ba5d8925be3c60c6eed/.1528878191.30118.ts.tEdzpD" (in object) failed: No such file or directory (2) | 18:45 |
| torgomatic | I mean, it's not bad, it's just that there may be something slightly better. I don't have a strong opinion either way, but I have a weak preference for different verbs. | 18:45 |
| notkevin | I look on the sending server and that file does not exist, and it doesn't exist on the receiving server, but /objects/612077/eed/956ed42383a57ba5d8925be3c60c6eed/1528878191.30118.ts exists on both | 18:45 |
| notkevin | Any idea why I am getting those error messages? | 18:46 |
| zaitcev | Hmm. I'd look at path= in rsyncd.conf first. What if you middle-buttoned something there. | 18:48 |
| notkevin | default_perms_for_dir: sys_acl_get_file(eed/956ed42383a57ba5d8925be3c60c6eed, ACL_TYPE_DEFAULT): No such file or directory, falling back on umask | 18:49 |
| notkevin | Is also on the sending side in the log files | 18:50 |
| notkevin | the path in /etc/rsync.conf is /srv/node | 18:51 |
| zaitcev | Also, do you really have a device called "37"? | 18:51 |
| notkevin | Yeah, 60 disk jbods | 18:51 |
| notkevin | The file /srv/node/37/objects/612077/eed/956ed42383a57ba5d8925be3c60c6eed/1528878191.30118.ts exists | 18:52 |
| *** gkadam has quit IRC | 18:55 | |
| zaitcev | Mysterious. | 18:57 |
| notkevin | Does the object-replicator make a .filename.random-text file before rsyncing it to the new node? | 19:03 |
| notkevin | and is that file getting removed from the source before it is copied to the destination? | 19:03 |
| notkevin | Because a few lines up, this is in the logs object-replicator[1976]: <f+++++++++ eed/956ed42383a57ba5d8925be3c60c6eed/1528878191.30118.ts | 19:04 |
| zaitcev | I'm pretty sure it's how rsync itself works. | 19:04 |
| notkevin | Which I thought meant that the rsync was successful | 19:04 |
| notkevin | Here is the logs from that period: https://pastebin.com/tN8xERVP | 19:21 |
| *** geaaru_ has joined #openstack-swift | 19:21 | |
| notkevin | It looks like the command is set to exclude "--exclude=.*.[0-9a-zA-Z][0-9a-zA-Z][0-9a-zA-Z][0-9a-zA-Z][0-9a-zA-Z][0-9a-zA-Z]" which should ignore the temp files? | 19:23 |
| *** geaaru has quit IRC | 19:23 | |
| *** bharath1234 has quit IRC | 19:50 | |
| zaitcev | This may happen if 37 is suddenly unmounted by the drive-audit. | 20:07 |
| *** mwheckmann has quit IRC | 20:40 | |
| timburke | am i reading this right? does https://github.com/openstack/swift/blob/2.18.0/test/unit/container/test_updater.py#L217 depend on the particular choices of hash prefix/suffix up at https://github.com/openstack/swift/blob/2.18.0/test/unit/container/test_updater.py#L40-L41 ? | 20:49 |
| notmyname | timburke: would seem so, at first glance | 20:53 |
| timburke | that makes me a little grumpy :-/ | 20:53 |
| timburke | but i don't feel like fixing it right now... | 20:54 |
| openstackgerrit | Tim Burke proposed openstack/swift master: container-updater: Always report zero objects/bytes used for shards https://review.openstack.org/577297 | 20:54 |
| notmyname | oh, yikes. that's an important patch timburke | 21:06 |
| timburke | maybe? we'd kinda acknowledged that during review and said "eh, it probably doesn't matter..." | 21:07 |
| timburke | but now i've got a complaint from an actual user who's collecting account stats across the whole cluster, so... yeah, patch. | 21:08 |
| timburke | it's easy enough to just ignore .shards_* accounts, i guess? but it'd be nicer if we didn't do it to begin with, IMO | 21:09 |
| notmyname | bah. bad on me for not noticing that :-( | 21:09 |
| torgomatic | interesting, so the propagation of stats from shard to root is done by the container sharder, not the container updater? | 21:13 |
| notkevin | zaitcev: The errors appear to only show up when replacing a failed drive or rebalancing the ring | 21:33 |
| notkevin | Based on reviewing the logs from 2018 | 21:34 |
| zaitcev | notkevin: I don't have a better idea. Just noticed that 35 works great, 37 fails. | 21:35 |
| timburke | torgomatic: yeah, ultimately, i'd be rather happy to have the stats-reporting move to the updater, the moving-object-rows (both as misplaced objects and as cleaving) move to the replicator, and have the sharder only live on to identify shard ranges (if at all; maybe this could move to the replicator, too?) | 21:36 |
| timburke | oh, and the shard-audit stuff move to the auditor | 21:36 |
| *** d0ugal_ has joined #openstack-swift | 21:44 | |
| torgomatic | right; that all sounds like a good long-term plan. Probably the stats-reporting move is the highest priority since that's affecting steady-state containers. | 21:50 |
| *** itlinux has quit IRC | 22:04 | |
| torgomatic | timburke: I'm looking at patch 577014 and thinking about the surrogateescape parts. I've convinced myself that the to/from_wsgi stuff is necessary, but I'm not convinced about the need for dodging errors. | 22:09 |
| patchbot | https://review.openstack.org/#/c/577014/ - swift - py3: Port common/constraints | 22:09 |
| torgomatic | Don't we already enforce that header names and values are UTF-8, in which case there's no error-handling required? | 22:10 |
| timburke | not for objects :-( | 22:11 |
| timburke | https://review.openstack.org/#/c/577014/1/test/unit/common/test_constraints.py@63 in particular is a problem | 22:12 |
| patchbot | patch 577014 - swift - py3: Port common/constraints | 22:12 |
| torgomatic | timburke: ah, so then all the surrogateescape stuff is necessary since we're not guaranteed anything but "here are some bytes" | 22:17 |
| timburke | yup | 22:17 |
| timburke | b'\xff' -> u'\xdcff' and we can *kinda* work with it | 22:17 |
| *** germs has joined #openstack-swift | 22:18 | |
| *** germs has quit IRC | 22:18 | |
| *** germs has joined #openstack-swift | 22:18 | |
| torgomatic | and that's why we need surrogateescape on encode as well, since otherwise it won't eat those surrogates | 22:18 |
| timburke | https://en.wikipedia.org/wiki/Universal_Character_Set_characters#Surrogates calls out python for taking an "interesting" approach to undecodable bytes | 22:19 |
| *** notkevin has quit IRC | 22:22 | |
| torgomatic | so that mostly looks good, but now I'm wondering if wsgi_to_str() returning wacky strs is going to be a problem. Generally, if I've got some string S in py3, I expect that S.encode('utf-8') will work, but that's not the case here. | 22:24 |
| torgomatic | I mean, it has to have wackiness in it since you can use arbitrary bytes as object names, but it still sort of violates my expectations | 22:25 |
| *** rcernin has joined #openstack-swift | 22:26 | |
| torgomatic | then again, Swift did make a choice to allow those arbitrary byte strings in object names; maybe it's just time to pay the piper | 22:29 |
| *** germs has quit IRC | 22:31 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: py36: Fix test_get_logger_sysloghandler_plumbing https://review.openstack.org/577313 | 22:43 |
| openstackgerrit | Samuel Merritt proposed openstack/swift master: Add clarifying comment about surrogate check https://review.openstack.org/577315 | 22:45 |
| timburke | you haven't wrestled with sys.argv on py3 enough yet ;-) | 22:56 |
| timburke | python3 -c 'import sys; print(sys.argv[1])' $(echo -e '\xff') | 22:57 |
| timburke | doesn't work either --> python3 -c 'import sys; print(sys.argv[1].encode("utf8"))' $(echo -e '\xff') | 22:58 |
| timburke | "works" --> python3 -c 'import sys; print(sys.argv[1].encode("utf8", "surrogateescape"))' $(echo -e '\xff') | 22:59 |
| timburke | actually round-trips --> python -c 'import sys; print(sys.argv[1])' $(echo -e '\xff') | 22:59 |
| timburke | round-tripping with py3 --> python3 -c 'import sys, os; stdout=os.fdopen(sys.stdout.fileno(), "wb"); stdout.write(sys.argv[1].encode("utf8", "surrogateescape"))' $(echo -e '\xff') | 23:02 |
| timburke | *obviously* | 23:02 |
| torgomatic | oof | 23:09 |
| *** itlinux has joined #openstack-swift | 23:10 | |
| *** kei_yama has joined #openstack-swift | 23:18 | |
| mattoliverau | morning | 23:20 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Reverse shard ranges later https://review.openstack.org/570265 | 23:36 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!