| *** saschpe- has quit IRC | 00:00 | |
| *** saschpe has joined #openstack-swift | 00:07 | |
| *** sungju has joined #openstack-swift | 00:24 | |
| *** matsuhashi has joined #openstack-swift | 00:25 | |
| *** sungju has quit IRC | 00:34 | |
| *** d89 has joined #openstack-swift | 00:38 | |
| *** nosnos has joined #openstack-swift | 01:36 | |
| *** d89 has quit IRC | 02:00 | |
| *** redbo_ is now known as redbo | 02:01 | |
| *** ChanServ sets mode: +v redbo | 02:02 | |
| *** d89 has joined #openstack-swift | 02:18 | |
| *** matsuhashi has quit IRC | 02:55 | |
| *** matsuhashi has joined #openstack-swift | 03:21 | |
| *** ekarlso has quit IRC | 03:48 | |
| *** ekarlso has joined #openstack-swift | 03:48 | |
| *** Trixboxer has joined #openstack-swift | 04:40 | |
| *** ppai has joined #openstack-swift | 04:49 | |
| *** chandankumar has quit IRC | 05:14 | |
| *** Dieterbe has quit IRC | 05:15 | |
| *** nosnos_ has joined #openstack-swift | 05:16 | |
| *** nosnos has quit IRC | 05:19 | |
| *** chandan_kumar has joined #openstack-swift | 05:25 | |
| *** nshaikh has joined #openstack-swift | 05:58 | |
| *** matsuhas_ has joined #openstack-swift | 06:00 | |
| *** matsuhashi has quit IRC | 06:01 | |
| *** nosnos has joined #openstack-swift | 06:19 | |
| *** nosnos_ has quit IRC | 06:22 | |
| *** dvas has joined #openstack-swift | 06:25 | |
| *** psharma has joined #openstack-swift | 06:28 | |
| *** Longgeek_ has joined #openstack-swift | 06:29 | |
| *** matsuhas_ has quit IRC | 06:31 | |
| *** matsuhashi has joined #openstack-swift | 06:31 | |
| *** saju_m has joined #openstack-swift | 06:35 | |
| *** saju_m has quit IRC | 06:59 | |
| *** tom_ has joined #openstack-swift | 07:03 | |
| *** tom_ is now known as Guest93130 | 07:04 | |
| *** saju_m has joined #openstack-swift | 07:12 | |
| *** saju_m has quit IRC | 07:17 | |
| *** saju_m has joined #openstack-swift | 07:17 | |
| *** redbo has quit IRC | 07:19 | |
| *** redbo has joined #openstack-swift | 07:20 | |
| *** ChanServ sets mode: +v redbo | 07:20 | |
| *** saju_m has quit IRC | 07:22 | |
| *** saju_m has joined #openstack-swift | 07:23 | |
| *** dvas has quit IRC | 07:29 | |
| *** saju_m has quit IRC | 07:30 | |
| *** saju_m has joined #openstack-swift | 07:31 | |
| *** mlipchuk has joined #openstack-swift | 07:35 | |
| *** dvas has joined #openstack-swift | 07:40 | |
| *** mlipchuk has left #openstack-swift | 07:43 | |
| *** yuanz has quit IRC | 08:06 | |
| *** yuan has joined #openstack-swift | 08:07 | |
| *** jamie_h has joined #openstack-swift | 08:34 | |
| *** dvas has quit IRC | 08:37 | |
| *** mmcardle has joined #openstack-swift | 08:46 | |
| *** Guest93130 has quit IRC | 08:55 | |
| *** jamie_h has quit IRC | 08:58 | |
| *** jamie_h_ has joined #openstack-swift | 08:59 | |
| *** jamie_h_ has quit IRC | 09:00 | |
| *** nacim has joined #openstack-swift | 09:04 | |
| *** mkollaro has joined #openstack-swift | 09:11 | |
| *** ppai has quit IRC | 09:14 | |
| *** tanee-away is now known as tanee | 09:14 | |
| *** tanee is now known as tanee-away | 09:14 | |
| *** tanee-away is now known as tanee | 09:15 | |
| *** nprivalova has joined #openstack-swift | 09:23 | |
| nprivalova | hi guys! I have a question about rsyslog config in devstack. Could you please take a look here https://review.openstack.org/#/c/82131/1 ? | 09:27 |
|---|---|---|
| *** nacim has quit IRC | 09:28 | |
| *** Midnightmyth has joined #openstack-swift | 09:32 | |
| *** ppai has joined #openstack-swift | 09:33 | |
| *** nacim has joined #openstack-swift | 09:36 | |
| *** d89 has quit IRC | 09:39 | |
| *** matsuhashi has quit IRC | 10:33 | |
| *** matsuhas_ has joined #openstack-swift | 10:35 | |
| openstackgerrit | Ghe Rivero proposed a change to openstack/python-swiftclient: Add pbr to requirements.txt https://review.openstack.org/82465 | 10:37 |
| *** Longgeek_ has quit IRC | 10:44 | |
| *** dvas has joined #openstack-swift | 10:53 | |
| *** mkollaro has quit IRC | 10:56 | |
| *** matsuhas_ has quit IRC | 11:07 | |
| *** dvas_ has joined #openstack-swift | 11:17 | |
| *** mkollaro has joined #openstack-swift | 11:19 | |
| *** jasondotstar has quit IRC | 11:20 | |
| *** dvas has quit IRC | 11:20 | |
| *** nosnos has quit IRC | 11:23 | |
| *** chandankumar_ has joined #openstack-swift | 11:29 | |
| *** dvas_ has quit IRC | 11:33 | |
| *** dvas_ has joined #openstack-swift | 11:33 | |
| *** ppai has quit IRC | 11:40 | |
| *** ppai has joined #openstack-swift | 11:52 | |
| *** matsuhashi has joined #openstack-swift | 11:53 | |
| *** nosnos has joined #openstack-swift | 11:54 | |
| *** chandankumar_ has quit IRC | 12:13 | |
| *** IvanBerezovskiy1 has joined #openstack-swift | 12:13 | |
| *** IvanBerezovskiy1 has left #openstack-swift | 12:13 | |
| *** matsuhashi has quit IRC | 12:14 | |
| *** matsuhas_ has joined #openstack-swift | 12:17 | |
| *** jasondotstar has joined #openstack-swift | 12:32 | |
| *** grapsus__ has quit IRC | 12:36 | |
| *** tdasilva has joined #openstack-swift | 12:36 | |
| *** nshaikh has quit IRC | 12:36 | |
| *** grapsus_ has joined #openstack-swift | 12:42 | |
| *** mmcardle has quit IRC | 12:45 | |
| *** nshaikh has joined #openstack-swift | 12:47 | |
| *** seandrummond has joined #openstack-swift | 12:49 | |
| *** mmcardle has joined #openstack-swift | 13:08 | |
| *** matsuhas_ has quit IRC | 13:12 | |
| *** matsuhashi has joined #openstack-swift | 13:13 | |
| *** matsuhashi has quit IRC | 13:17 | |
| *** psharma has quit IRC | 13:20 | |
| *** rustlebee is now known as russellb | 13:23 | |
| *** nosnos has quit IRC | 13:24 | |
| *** pberis has joined #openstack-swift | 13:25 | |
| *** nosnos has joined #openstack-swift | 13:25 | |
| *** ppai has quit IRC | 13:25 | |
| *** nosnos has quit IRC | 13:29 | |
| *** pberis has quit IRC | 13:38 | |
| *** jasondotstar has quit IRC | 13:43 | |
| *** cutforth has joined #openstack-swift | 13:51 | |
| *** cutforth has quit IRC | 13:52 | |
| *** nosnos has joined #openstack-swift | 13:58 | |
| *** dmsimard has joined #openstack-swift | 13:59 | |
| *** matsuhashi has joined #openstack-swift | 13:59 | |
| *** nosnos has quit IRC | 14:00 | |
| *** nosnos has joined #openstack-swift | 14:01 | |
| *** nosnos has quit IRC | 14:05 | |
| *** jasondotstar has joined #openstack-swift | 14:10 | |
| *** nosnos has joined #openstack-swift | 14:11 | |
| *** nosnos has quit IRC | 14:14 | |
| *** krtaylor has quit IRC | 14:15 | |
| *** matsuhashi has quit IRC | 14:16 | |
| *** ChanServ sets mode: +v dfg | 14:20 | |
| *** piyush1 has joined #openstack-swift | 14:28 | |
| *** lpabon has joined #openstack-swift | 14:29 | |
| *** matsuhashi has joined #openstack-swift | 14:32 | |
| *** tanee is now known as tanee-away | 14:35 | |
| *** dvas_ has quit IRC | 14:36 | |
| *** saju_m has quit IRC | 14:39 | |
| openstackgerrit | Yuan Zhou proposed a change to openstack/swift: Fixes versioning function tests with non-zero default policy https://review.openstack.org/82515 | 14:40 |
| openstackgerrit | Yuan Zhou proposed a change to openstack/swift: Fixes versioning function tests with non-zero default policy https://review.openstack.org/82515 | 14:42 |
| nprivalova | hi guys! ping somebody from core-team...need to discuss https://bugs.launchpad.net/ceilometer/+bug/1294789 | 14:43 |
| *** krtaylor has joined #openstack-swift | 14:44 | |
| *** ChanServ sets mode: +v notmyname | 14:45 | |
| *** dvas_ has joined #openstack-swift | 14:52 | |
| notmyname | good morning. /me is reading buffer playback while finishing coffee | 14:58 |
| *** Dieterbe has joined #openstack-swift | 15:01 | |
| *** Dieterbe has joined #openstack-swift | 15:01 | |
| dmsimard | notmyname: I knew that name rang a bell, re: https://github.com/victorlin/swiftsense/issues/9 :P | 15:02 |
| notmyname | dmsimard: likewise :-) | 15:02 |
| *** seandrummond has left #openstack-swift | 15:07 | |
| luisbg | notmyname, morning :) | 15:08 |
| luisbg | notmyname, have an internal training this week plus the usual client work, so very busy and will be a MIA this week :( | 15:11 |
| notmyname | luisbg: any chance you'll be going to the atlanta summit? | 15:11 |
| luisbg | notmyname, discussed it with my company and can't take the time off, unfortunately :( | 15:12 |
| luisbg | I really wanted to go | 15:12 |
| luisbg | notmyname, coming to Montreal for PyCon? | 15:12 |
| notmyname | luisbg: no, I'll be at the Red Hat summit in San Francisco that week | 15:12 |
| luisbg | if anybody is coming to Montreal let me know, I will show you some good places/restaurants :) | 15:13 |
| creiht | aww man... it's the first pycon that I will be missing in a while | 15:13 |
| creiht | I will likely go next year though | 15:13 |
| Alex_Gaynor | luisbg: Sorry, my entire week is already scheduled going to Schwartz's and eating Montreal bagels :P | 15:15 |
| *** dvas_ has quit IRC | 15:16 | |
| luisbg | Alex_Gaynor, I live around the block of St Viateur bagels, I hope you mean *those* iconic bagels | 15:17 |
| luisbg | Alex_Gaynor, you miss La Banquise in that list, and Dieu Du Ciel | 15:18 |
| *** dvas_ has joined #openstack-swift | 15:19 | |
| *** dvas_ has quit IRC | 15:19 | |
| *** byeager has joined #openstack-swift | 15:20 | |
| *** tanee-away is now known as tanee | 15:20 | |
| notmyname | portante: creiht: clayg: please pay attention to https://bugs.launchpad.net/ceilometer/+bug/1294789. It turns out that something like 60-80% of _all_ of the jenkins log lines are coming from ceilometer's swift middleware, and as a consequence there are proposals to significantly change swift's logging functionality/framework | 15:21 |
| creiht | lol | 15:21 |
| portante | oy | 15:22 |
| * portante looks | 15:22 | |
| Alex_Gaynor | Just a dude with zero context here, but if it's ceilometer's swift middleware, isn't that what should be changed? | 15:22 |
| creiht | Alex_Gaynor: yeah my thoughts as well, and the reason for the lol :) | 15:22 |
| notmyname | Alex_Gaynor: the current reasoning is that it's swift's fault because we don't use oslo logging | 15:23 |
| *** dvas_ has joined #openstack-swift | 15:23 | |
| portante | Alex_Gaynor, notmyname, creiht: and because of they way they configure logging, it always goes to syslog and the proxy-s log file, so a double whammy | 15:23 |
| * creiht sighs | 15:24 | |
| portante | I believe our logging can be fixed up for other reasons, like not logging to two places, but still not convinced yet about moving to oslo.logging | 15:25 |
| notmyname | so I'd appreciate feedback there so that the issues can be resolved calmly | 15:25 |
| portante | ;) | 15:25 |
| *** matsuhashi has quit IRC | 15:25 | |
| *** nshaikh has quit IRC | 15:27 | |
| *** dvas_ has quit IRC | 15:27 | |
| dmsimard | notmyname: Have you had any luck visualizing a large ring with *anything* ? | 15:28 |
| *** Midnightmyth has quit IRC | 15:28 | |
| *** matsuhas_ has joined #openstack-swift | 15:28 | |
| dmsimard | swift-ring-builder and swift-recon are not very… visual | 15:28 |
| notmyname | dmsimard: no. while I'd love to see swiftsense fleshed out a bit to be a little better, I currently use swiftsense for presentations and meetups. for that, it's great (and I normally use a ring power of 8 or something similarly low) | 15:29 |
| notmyname | ok, coffee cup is empty. time to get ready and go to the office | 15:30 |
| notmyname | I'll be back online later | 15:30 |
| portante | creiht: is anybody from rackspace looking into that logging problem above? | 15:30 |
| creiht | portante: I'm looking at it | 15:30 |
| portante | k | 15:30 |
| *** matsuhas_ has quit IRC | 15:33 | |
| *** dvas_ has joined #openstack-swift | 15:34 | |
| creiht | portante: it is a bit confusing though | 15:35 |
| creiht | hard to tell what the exact issue is | 15:35 |
| portante | yes, agreed | 15:35 |
| creiht | it seems like a combo of things | 15:35 |
| portante | from what I can tell, the ceilometer code is configured to log a lot of debug stuff, and that ends up duplicated | 15:36 |
| portante | I am guessing it is also using oslo.logging, but not sure | 15:36 |
| portante | creiht: | 15:36 |
| portante | I am hoping it is just using the provided logger from the middleware invocation | 15:37 |
| creiht | holy cow ceilometer logs a lot | 15:38 |
| portante | debug, too, right? | 15:39 |
| *** gyee has joined #openstack-swift | 15:39 | |
| creiht | yeah | 15:39 |
| creiht | well most of the openstack services log a lot actually | 15:39 |
| creiht | From the other hand, we use pipeline.setup_pipeline inside Swift. And all swift-logging configs | 15:41 |
| creiht | become applied to Ceilometer's pipeline. As a result (in case of DEBUG is on) we see too many | 15:41 |
| creiht | Ceilometr-related messages in syslog. | 15:41 |
| creiht | https://review.openstack.org/#/c/82526/ | 15:41 |
| * portante looks | 15:41 | |
| creiht | I'm not entirely sure what that means but sounds broken | 15:42 |
| portante | hardcoding the log level just seems to reduce the volume, but not the double logging | 15:43 |
| creiht | since they are using get_logger, they shouldn't have to hard code it | 15:44 |
| portante | yes | 15:44 |
| creiht | why does the ceilometer middleware call setup_pipeline? | 15:45 |
| creiht | that seems like a really bad idea | 15:45 |
| portante | not sure, that seems odd | 15:45 |
| creiht | that's something core to ceilometer | 15:46 |
| * portante goes and checks out ceilometer | 15:47 | |
| creiht | oh maybe that's something different than a pipeline for wsgi | 15:50 |
| portante | okay, reached me limit without food, will back | 15:58 |
| *** cyril_ has joined #openstack-swift | 16:02 | |
| *** judd7_ has joined #openstack-swift | 16:16 | |
| *** j_king_ is now known as j_king | 16:16 | |
| *** madhuri has joined #openstack-swift | 16:18 | |
| creiht | yeah I need some food as well | 16:21 |
| creiht | hard to get thinking quickly on a Monday morning :) | 16:21 |
| *** piyush1 has quit IRC | 16:22 | |
| *** piyush has joined #openstack-swift | 16:22 | |
| *** piyush1 has joined #openstack-swift | 16:26 | |
| *** haypo has joined #openstack-swift | 16:27 | |
| haypo | hi. i'm working on porting OpenStack to Python3. i would like to know if swiftclient is now accepted patches to port it to Python3 | 16:27 |
| creiht | haypo: yes, several python3 related patches have already landed in swiftclient | 16:28 |
| notmyname | yay python 3 | 16:29 |
| notmyname | ...or something.... ;-) | 16:29 |
| creiht | hehe | 16:29 |
| cyril_ | Isn't there some sort of featrure freeze currently ? | 16:29 |
| * notmyname is back online now | 16:29 | |
| *** piyush has quit IRC | 16:29 | |
| notmyname | cyril_: "freeze" is such a strong word | 16:29 |
| haypo | creiht: a colleague told me that python3 patches must enhance the test coverage | 16:29 |
| haypo | creiht: does it mean that a python3 patch will not get accepted if it doesn't add new tests? | 16:30 |
| notmyname | haypo: yes, that is something that's been a key factor in merging some of the other py3 patches | 16:30 |
| notmyname | haypo: basically the concern is ensuring that (1) there are no regressions and (2) we don't inadvertently have a py3 regression | 16:30 |
| creiht | notmyname: it still isn't clear to me what is going on with the ceilometer stuff, and I'm headed to lunch | 16:31 |
| creiht | I'll poke at it some more later | 16:31 |
| notmyname | creiht: thanks for looking :-) | 16:31 |
| notmyname | haypo: basic good hygiene stuff, and sincy py3 will cause more code churn than "normal" patches, better testing is going to be a key part of reviews | 16:31 |
| *** madhuri has quit IRC | 16:33 | |
| *** jasondotstar has quit IRC | 16:39 | |
| *** shri has joined #openstack-swift | 17:01 | |
| portante | creiht: it looks like there are two loggers being used for the ceilometer stuff | 17:04 |
| portante | most of the code seems to use openstack.common.log, and I believe that is logging to a file, then our stdout adapter picks that up and logs it as well | 17:05 |
| *** tanee is now known as tanee-away | 17:06 | |
| portante | if the logger was provided to the publisher object, I am guessing we could avoid all that | 17:06 |
| portante | but still looking ... | 17:06 |
| *** tanee-away is now known as tanee | 17:12 | |
| *** zaitcev has joined #openstack-swift | 17:12 | |
| *** ChanServ sets mode: +v zaitcev | 17:12 | |
| *** piousbox has joined #openstack-swift | 17:14 | |
| openstackgerrit | Victor Stinner proposed a change to openstack/python-swiftclient: Python 3: Get compatible types from six https://review.openstack.org/82552 | 17:17 |
| openstackgerrit | Victor Stinner proposed a change to openstack/python-swiftclient: Python 3: cast map() result to list https://review.openstack.org/82553 | 17:18 |
| openstackgerrit | Victor Stinner proposed a change to openstack/python-swiftclient: Python 3: Fix tests using temporary text files https://review.openstack.org/82554 | 17:19 |
| openstackgerrit | Victor Stinner proposed a change to openstack/python-swiftclient: Python 3: Replace iter.next() with six.next(iter) https://review.openstack.org/82555 | 17:19 |
| openstackgerrit | Victor Stinner proposed a change to openstack/python-swiftclient: Python 3: Fix module names in import https://review.openstack.org/82556 | 17:20 |
| notmyname | oh my | 17:21 |
| openstackgerrit | Victor Stinner proposed a change to openstack/python-swiftclient: Python 3: use six.iteritems() https://review.openstack.org/82557 | 17:21 |
| haypo | i don't see how to improve the test coverage since it is even not possible to import swiftclient in python3 right now :-( | 17:22 |
| haypo | don't hesitate if you have suggestions to improve my patches | 17:23 |
| haypo | notmyname: you prefer one patch instead of 6? | 17:23 |
| *** nacim has quit IRC | 17:26 | |
| notmyname | haypo: I haven't looked, so I don't have a good answer to that right now :-) | 17:26 |
| haypo | notmyname: ok. i have to leave. see you tomorrow | 17:28 |
| *** mlipchuk has joined #openstack-swift | 17:29 | |
| *** bada has joined #openstack-swift | 17:29 | |
| *** Midnightmyth has joined #openstack-swift | 17:32 | |
| *** flashgordon is now known as jogo | 17:34 | |
| *** piyush1 has quit IRC | 17:37 | |
| *** erlon has joined #openstack-swift | 17:39 | |
| *** cyril_ has quit IRC | 17:41 | |
| portante | creiht: I tried to run the unit tests for ceilometer on my box, a F19 setup, but ran into a number of package dependency issues | 17:43 |
| *** mmcardle has quit IRC | 17:43 | |
| *** dvas_ has quit IRC | 17:47 | |
| *** dvas_ has joined #openstack-swift | 17:48 | |
| *** dvas_ has quit IRC | 17:52 | |
| notmyname | I'm looking at what's in master since 1.13.0 (seeing what's going to be in Icehouse) | 18:04 |
| notmyname | FYI https://gist.github.com/notmyname/560733484877ffd8a091 | 18:05 |
| clayg | dencaval: the object datafile changes are rolled up by hash suffix - there's a little pickle sittin' in each part that keep track of which hash suffixes have changes - if you delete a .data file and don't invalidate that .pkl (like the auditor does) replication will by-pass it until another write comes into that hash suffix | 18:05 |
| notmyname | so it seems that 1.13.1 is probably more likely than 1.14.0 as of right now | 18:05 |
| notmyname | clayg: ah, thanks. I remember seeing that question, but I forgot to respond | 18:05 |
| clayg | it's a pretty good gotcha as far as gotcha's go | 18:06 |
| pandemicsyn | clayg: my scrollback rolled off right before your message ....what did you want the auditor to throw in recon ? | 18:06 |
| pandemicsyn | a swift-recon -d will show ya drive fullness | 18:07 |
| clayg | pandemicsyn: i can't recall - number of ojbects in a part maybe? | 18:08 |
| clayg | per drive? | 18:08 |
| *** piyush has joined #openstack-swift | 18:08 | |
| clayg | pandemicsyn: someone was worried about a hash attack or something maybe... - that was like last week bro! | 18:09 |
| clayg | :P | 18:09 |
| *** dvas_ has joined #openstack-swift | 18:09 | |
| pandemicsyn | lol sorry, allergies been kicking my ass the last week (yay Oak!) | 18:09 |
| *** piyush1 has joined #openstack-swift | 18:10 | |
| clayg | torgomatic: peluse: re account-head policy index vs. name - I don't see the value in the http response referring to policy by name, but I like it very much that internall the account-backend knows of and speaks only index | 18:10 |
| notmyname | pandemicsyn: looks like it's been light and moderate for a week. ;-) http://saallergy.info | 18:11 |
| clayg | if you pass the response back to the proxy via index then it has to do a bunch of string munging to get the headers in shape for the client - seems easier just to translate to name in the account server and let the proxy stay out of the way? | 18:11 |
| torgomatic | clayg: if it makes life easier to have the account server refer to things both by index and name, then that'd be okay by me too | 18:11 |
| torgomatic | (although then you run into trouble with a policy whose *name* is "1") | 18:11 |
| notmyname | pandemicsyn: you gotta get scotticus to add a multiplier for everyone on his team who isn't at work for a day | 18:11 |
| pandemicsyn | lol | 18:11 |
| torgomatic | ...so forget everything I just said. It's crap. :) | 18:11 |
| clayg | torgomatic: if an http request comes to the account server refering to a storage policy (i.e. container update) - it's by index | 18:12 |
| pandemicsyn | clayg: i think the auditor already tracks bytes processes on the last pass or something, might be easy to hack in a file count | 18:12 |
| clayg | it's only in the GETorHEAD response where it translates policy index (integers) to name (for headers) | 18:12 |
| clayg | pandemicsyn: right on | 18:12 |
| *** zul has quit IRC | 18:13 | |
| *** piyush has quit IRC | 18:13 | |
| notmyname | pandemicsyn: and since objects are directories, then you can pull your neat XFS trick like with async-pendings (# of links - 2). that's still a cool trick, IMO | 18:13 |
| *** zul has joined #openstack-swift | 18:16 | |
| *** mmcardle has joined #openstack-swift | 18:21 | |
| *** piousbox has quit IRC | 18:25 | |
| clayg | torgomatic: peluse: so the 409 if you put to a non-default-sp container w/o storage policy header is sorta annoying | 18:25 |
| clayg | torgomatic: peluse: it was introduced with the merge for the _update_or_create change and I fixed in the account-head thing - but I'm thinking of unrolling it since that is two changes deep in the patch queue? | 18:26 |
| clayg | yeah i'ma do that | 18:26 |
| *** haomaiwang has quit IRC | 18:27 | |
| *** dvas__ has joined #openstack-swift | 18:29 | |
| clayg | torgomatic: I just pulled down the x-storage-policy-index-override thingy - but didn't see any gatekeeper changes? like it goes all they way out to the client on PUT - I think that's sorta not-so-super? | 18:29 |
| torgomatic | clayg: yeah, I need to rename it now that the gatekeeper change landed | 18:29 |
| clayg | oh wah? | 18:30 |
| *** dvas___ has joined #openstack-swift | 18:32 | |
| *** mmcardle has quit IRC | 18:32 | |
| *** dvas_ has quit IRC | 18:32 | |
| torgomatic | clayg: change to master where X-Backend* is blocked by gatekeeper, so if we call our override thingy X-Backend-Whatevermabobber, we get the blocking for free | 18:32 |
| clayg | yay free! | 18:32 |
| torgomatic | :) | 18:32 |
| portante | creiht: just to get my feet wet with ceilometer: https://review.openstack.org/82572 | 18:33 |
| creiht | portante: cool | 18:33 |
| creiht | so talking to the ceilometer guys, they just want to lower the log level | 18:34 |
| creiht | they don't see any other changes needed | 18:34 |
| *** dvas__ has quit IRC | 18:36 | |
| *** mkollaro has quit IRC | 18:36 | |
| portante | really, wow, cause wouldn't we want middleware to log under the context of the proxy server rather than using some other loggin? | 18:36 |
| portante | this smacks of global variable problem | 18:37 |
| portante | creiht: do they realize this won't reduce the double logging? | 18:44 |
| creiht | well that is a separate issue | 18:45 |
| *** pberis has joined #openstack-swift | 18:46 | |
| portante | hmm | 18:46 |
| zaitcev | creiht: What does it mean, "lower the log level"? | 18:46 |
| creiht | zaitcev: from DEBUG to WARNING or ERROR for example | 18:47 |
| zaitcev | creiht: where do we load as debug today? It's all info. | 18:47 |
| portante | devstack makes it debug | 18:47 |
| portante | so they are hard-coding in their middleware to change log level to WARN | 18:47 |
| portante | but only for the swift logger | 18:48 |
| portante | which does not affect much, so not sure how this actually helps | 18:48 |
| creiht | portante: I'm helping them to change the logging just for their middleware | 18:49 |
| portante | okay, I'll wait to see what comes out of that | 18:49 |
| *** tanee is now known as tanee-away | 18:52 | |
| creiht | notmyname: ok, so who is the right person to talk to about devstack? | 18:53 |
| creiht | about what/where they want logs, so I can make sure devstack is set up correctly | 18:53 |
| notmyname | creiht: either sdague or maybe jeblair (because CI) | 18:54 |
| notmyname | at least that's where I'd start. I'm not actually sure | 18:54 |
| creiht | k | 18:55 |
| creiht | I think the ceilometer team is good now | 18:55 |
| notmyname | creiht: thanks for looking and working with them | 18:55 |
| creiht | np | 18:55 |
| notmyname | portante: that looks like a reasonable patch to ceilometer. thanks | 18:55 |
| portante | welcome | 18:56 |
| creiht | jeblair: ping | 18:56 |
| creiht | portante: yes, thanks :) | 18:56 |
| notmyname | creiht: I don't really understand the "hard-code to a different log level", but I'm sure I'm missing some context on why setting log_level=WARN in a config is hard | 18:56 |
| creiht | notmyname: yeah I walked them through how to set it up for the config so no code changes will be required | 18:57 |
| notmyname | nice | 18:57 |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Allow overriding of storage policy index. https://review.openstack.org/82211 | 18:57 |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Make the container replicator notice misplaced objects https://review.openstack.org/80731 | 18:57 |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Add object-reconciler daemon https://review.openstack.org/80730 | 18:57 |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Make object updater reconcile on 409s https://review.openstack.org/80732 | 18:57 |
| creiht | just most likely an update to the devstack setup | 18:58 |
| zaitcev | portante: BTW, here's the double-logging bug and a suggestion to use local0.debug: https://bugzilla.redhat.com/show_bug.cgi?id=997983 | 18:58 |
| zaitcev | portante: not necessarily a good idea at all, but eh... it's either that or modify /etc/rsyslog.conf with a trick sed(1) invocation. | 18:59 |
| *** byeager has quit IRC | 18:59 | |
| *** byeager has joined #openstack-swift | 18:59 | |
| portante | zaitcev: doesn't the SAIO documentation also describes how to log to separate files? | 19:01 |
| *** mlipchuk has quit IRC | 19:01 | |
| zaitcev | portante: it does, but does not explain how to cork the default with local0.none in /etc/rsyslog.conf. | 19:03 |
| *** byeager has quit IRC | 19:04 | |
| creiht | portante: sorry :) | 19:04 |
| portante | zaitcev: we should update and cork that. :) | 19:04 |
| creiht | yeah it isn't clear to me if they don't want it to log at all or to log to a separate location for syslog | 19:05 |
| *** gvernik has joined #openstack-swift | 19:05 | |
| creiht | notmyname: do you know what joe gordon's nick is? | 19:07 |
| creiht | ahh | 19:08 |
| creiht | jogo: ping :) | 19:08 |
| *** dvas____ has joined #openstack-swift | 19:25 | |
| *** dvas___ has quit IRC | 19:29 | |
| *** dvas_____ has joined #openstack-swift | 19:30 | |
| *** dvas____ has quit IRC | 19:30 | |
| *** byeager has joined #openstack-swift | 19:30 | |
| *** dvas_____ has quit IRC | 19:31 | |
| *** pberis has quit IRC | 19:32 | |
| *** pberis has joined #openstack-swift | 19:33 | |
| creiht | notmyname: btw, do you have an svg of the swift logo? | 19:46 |
| creiht | I would like to try 3d printing something with it | 19:46 |
| notmyname | creiht: cool | 19:46 |
| notmyname | I have a tiff I think | 19:46 |
| creiht | that would work | 19:46 |
| notmyname | looking... | 19:46 |
| creiht | thanks | 19:46 |
| notmyname | ah. I've got small medium and large jpegs and I've got it in eps in black and 4 colors (pantone and cmyk) | 19:47 |
| notmyname | I'm assuming you want the black eps | 19:48 |
| creiht | black eps would be great | 19:48 |
| notmyname | uploading it to cloud files now... | 19:48 |
| creiht | cool | 19:49 |
| notmyname | http://d.not.mn/Swift_logo_stroke_black.eps | 19:49 |
| creiht | tracking your upload... | 19:49 |
| creiht | :) | 19:49 |
| notmyname | heh. clearly it would move the graphs | 19:49 |
| creiht | hehe | 19:49 |
| creiht | cool | 19:50 |
| creiht | I think I can make that work | 19:50 |
| creiht | thanks | 19:50 |
| notmyname | so as a formal disclaimer, that's not official, trademarked, etc. it's simply a cool thing to tie together visual elements and provide a visual identity. | 19:50 |
| creiht | lol | 19:50 |
| notmyname | also, for the record, it in now way is in competition with http://www.swift.com/index.page?lang=en | 19:51 |
| notmyname | s/now/nbo/ | 19:51 |
| notmyname | err | 19:51 |
| notmyname | no way | 19:51 |
| creiht | heh... that's still a thing? :) | 19:51 |
| notmyname | their lawyers say it is :-) | 19:52 |
| *** Trixboxer has quit IRC | 19:53 | |
| *** gvernik has quit IRC | 19:59 | |
| *** shri has quit IRC | 20:02 | |
| jeblair | creiht: pong | 20:06 |
| creiht | jeblair: howdy | 20:07 |
| creiht | notmyname: uh oh | 20:07 |
| creiht | jeblair: so not sure if you are the right person to ask | 20:08 |
| creiht | https://bugs.launchpad.net/ceilometer/+bug/1294789?comments=all | 20:08 |
| creiht | mentions that swift in devstack logging to both console and syslog is an issue | 20:08 |
| creiht | do you know what the desired behavior would be for devstack? | 20:08 |
| creiht | jogo: the same question to you if you get a moment :) | 20:10 |
| jeblair | creiht: that seems reasonable to me, but my experience in this area is limited; you might want to ask sdague, dtroyer and (as you have been) jogo to weigh in. | 20:14 |
| jeblair | creiht: (sorry to be specific, the thing that seems reasonable is configuring the ceilometer logs in swift to be at WARN level in devestack, as suggested in the review) | 20:14 |
| *** shri has joined #openstack-swift | 20:15 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Fix PUT to existing container w/o specific storage policy https://review.openstack.org/82590 | 20:15 |
| jeblair | basically, my thinking is that the current level is too verbose to be useful to anyone, so it's reasonable to reduce the verbosity | 20:15 |
| portante | jeblair: so the double logging is not a concern? | 20:16 |
| jeblair | portante: double logging? that sounds concerning :) | 20:16 |
| portante | same entry lands in syslog.txt and proxy-s-*.txt | 20:16 |
| jogo | creiht: so the way I see it there are a few factors in this bug | 20:17 |
| portante | notmyname: can that openstackgerrit bot include the branch in the message? | 20:17 |
| notmyname | portante: that would be nice (/cc jeblair) | 20:17 |
| jogo | 1) swift supports logging to syslog, and presumably that is a common use case so we want to run that in devstack | 20:18 |
| jogo | 2) ceilometer logs useless junk when its running as a middleware for swift | 20:18 |
| portante | notmyname, jeblair: if it is not master, that is? | 20:18 |
| jogo | so dropping ceilometers log level sounds reasonable | 20:18 |
| creiht | jeblair: yeah I have already worked with them on how to set the config right so that they will only log WARN | 20:18 |
| jogo | FWIW we dropped the cheilometer plugin for swift in gate until this is resolved | 20:19 |
| jeblair | portante: unfortunately, since swift proxy logs don't have timestamps, we can't put them in logstash, so until that changes, we need swift logs going to syslog to get the timestamps | 20:19 |
| jeblair | portante: our preference would be to not log swift to syslog in the gate | 20:19 |
| jogo | creiht: another factor here is we don't parse swift logs and store them in elasticsearch | 20:19 |
| portante | jeblair: but the proxy file does not have any timestamps, so is it worth even using that for now? | 20:20 |
| notmyname | jogo: cheilometer? is that ceilometer written by sean connery? | 20:20 |
| jogo | so we reley on syslog logs for that, and we when syslog blew up our ES clusterw as too small | 20:20 |
| jogo | notmyname: haha yup | 20:20 |
| jeblair | notmyname, portante: i don't think including the branch name is currently a feature, but the data are definitely there for an enhancement | 20:20 |
| portante | jeblair: thanks | 20:20 |
| jogo | how is swift usually deployed log wise? | 20:22 |
| creiht | jogo: always syslog | 20:22 |
| *** piyush1 has left #openstack-swift | 20:22 | |
| jogo | creiht: any syslog side config to split out the swift logs so they don't overwhelm the other syslog entries? | 20:23 |
| portante | yes | 20:23 |
| portante | we document that in our SAIO | 20:23 |
| creiht | jogo: yeah, the rsyslog config should do that | 20:23 |
| *** piyush has joined #openstack-swift | 20:23 | |
| *** piyush has left #openstack-swift | 20:23 | |
| jogo | also any way to to identify log level ? | 20:23 |
| notmyname | jeblair: we did add high-resolution timestamps to the proxy logs (partially in response to the request from -infra) six months ago. is it not sufficient? | 20:23 |
| jogo | portante: link? | 20:24 |
| portante | sec | 20:24 |
| *** piyush has joined #openstack-swift | 20:24 | |
| portante | jogo: http://docs.openstack.org/developer/swift/development_saio.html | 20:24 |
| jogo | clarkb: ^ | 20:24 |
| portante | http://docs.openstack.org/developer/swift/development_saio.html#optional-setting-up-rsyslog-for-individual-logging | 20:24 |
| creiht | jogo: I'm not running devstack right now, but there is a files/swift/rsyslog.conf | 20:25 |
| portante | that is a better link | 20:25 |
| creiht | in devstack | 20:25 |
| creiht | that looks like gets used by lib/swift | 20:25 |
| portante | one ammendment needs to be made to that | 20:25 |
| notmyname | portante: best link: https://github.com/openstack/swift/blob/master/doc/saio/rsyslog.d/10-swift.conf | 20:26 |
| notmyname | ;-) | 20:26 |
| portante | ;) | 20:26 |
| portante | good one! | 20:26 |
| jeblair | notmyname: well that sounds swell. looking at http://logs.openstack.org/92/82392/3/check/check-tempest-dsvm-full/4567a13/logs/screen-s-proxy.txt.gz it looks like that isn't used yet... | 20:26 |
| jeblair | notmyname: maybe devstack needs updating to turn that on? | 20:27 |
| jeblair | clarkb: ^ | 20:27 |
| notmyname | jeblair: it's the last 2 fields of the log line | 20:27 |
| portante | jeblair, and one can change the rsyslog format to match what you need for timestamp patterns for logstash | 20:27 |
| creiht | jogo: but more specifically, why would you set up any production system *not* using syslog? ;) | 20:28 |
| jogo | jeblair: we aren't collecting /var/log/swift either | 20:28 |
| jogo | creiht: to make your life harder of course | 20:29 |
| jeblair | so i originally set up devstack-gate to _only_ log to syslog and collect syslog | 20:29 |
| creiht | we actually use syslog-ng instead of rsyslog | 20:29 |
| jeblair | creiht: because of exactly that feeling | 20:29 |
| openstackgerrit | Victor Stinner proposed a change to openstack/python-swiftclient: Python 3: use six.iteritems() https://review.openstack.org/82557 | 20:30 |
| jeblair | however, a long time ago it was pretty clear that openstack did not agree with that, as it was impossible to debug anything (other than perhaps swift) using only syslogs | 20:30 |
| jogo | jeblair: well it sounds like we have a road forward on this | 20:30 |
| jeblair | so that decision was reversed, and instead everything (except swift) is configured _not_ to log to syslog in order that we are certain we get the logs we need | 20:30 |
| creiht | swift can be configured to log just about however you want | 20:31 |
| jogo | jeblair: sounds like a good topic for the a cross project summit session or something | 20:31 |
| creiht | so if you let us know how you want us to log, we can help you make that happen ;) | 20:31 |
| notmyname | creiht: as long as it's a SysLogHanlder ;-) | 20:32 |
| clayg | yeah don't get it; syslog is great for redirecting log messages to different files based on "things" | 20:32 |
| creiht | well true | 20:32 |
| *** lpabon has quit IRC | 20:32 | |
| creiht | but syslog is *very* configurable :) | 20:32 |
| jogo | creiht: timestamped logs that we can filter out debug or higher logs | 20:32 |
| jeblair | creiht: groovy; clarkb is the expert on "what we want", so we should ask him. | 20:32 |
| notmyname | I'm with clayg on this one. it seems that the issue is one of syslog configs | 20:32 |
| creiht | jogo: right, does the patch that notmyname mentioned above fix that for yall? | 20:33 |
| jogo | notmyname: well that and ceilometer logging useless stuff | 20:33 |
| creiht | jogo: and I think we have ceilometer on the right track now | 20:33 |
| openstackgerrit | Victor Stinner proposed a change to openstack/python-swiftclient: Python 3: Get compatible types from six https://review.openstack.org/82552 | 20:34 |
| jogo | creiht: link? | 20:34 |
| jogo | and yup | 20:34 |
| notmyname | jogo: git show d4e02a2e8b50ae9272efc07cd44217333b4aacd6 | 20:34 |
| notmyname | however that's linkified | 20:34 |
| notmyname | jogo: the (current) last 2 log fields are high-res timestamps for the request | 20:35 |
| creiht | jogo: http://paste.openstack.org/show/74186/ | 20:35 |
| notmyname | actually, isn't it nice that someone documented the swift log formats? http://docs.openstack.org/developer/swift/logs.html | 20:35 |
| creiht | there's an example log line from my saio, where you can see the timestamps at the end | 20:35 |
| portante | notmyname, also from within the server itself, not when the logger gets the message | 20:36 |
| portante | jeblair, jogo: on my F19 laptop, I comment out the rsyslog.conf line: $ActionFileDefaultTemplate RSYSLOG_... | 20:37 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Container Confict on obj update with wrong Storage-Policy-Index https://review.openstack.org/79731 | 20:37 |
| haypo | oh, i should not send patches in a hurry. i introduced bugs when i splitted my changes in 6 small patches | 20:37 |
| jogo | http://logs.openstack.org/55/82255/1/check/check-tempest-dsvm-full/88d1e36/logs/screen-s-proxy.txt.gz | 20:37 |
| portante | and I get log lines like: "2014-03-23T04:29:51.463145-04:00 frodo NetworkManager[724]: <warn> ..." | 20:37 |
| *** iccha has left #openstack-swift | 20:37 | |
| notmyname | portante: does that server get weak and tired when carrying the swift rings? | 20:38 |
| jogo | is what we have been trying to use. but the syslog output is better | 20:38 |
| portante | notmyname: cute | 20:38 |
| portante | ;) | 20:39 |
| jogo | so we don't add every debug level log into elasticSearch, but it looks like your rsyslog config can help with that | 20:39 |
| notmyname | and for today's humorous LinkedIn email: "I had reached out to see if SwiftStack may have an interest in Paragon Software Group’s image based backup and disaster recovery technology for a SwiftStack backup offering to complement your Private Cloud Storage offerings." <-- have you heard of Swift? | 20:39 |
| portante | ;) | 20:39 |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Make the container replicator notice misplaced objects https://review.openstack.org/80731 | 20:40 |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Add object-reconciler daemon https://review.openstack.org/80730 | 20:40 |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Make object updater reconcile on 409s https://review.openstack.org/80732 | 20:40 |
| jogo | portante notmyname creiht: anyway clarkb is the resident devstack-gate log wizard | 20:40 |
| notmyname | portante: that's feature/ec #notmynamebot | 20:40 |
| creiht | jogo: what you could probably do to simplify everything, is just do the /var/log/swift/all.log rule | 20:40 |
| creiht | ignore the others | 20:41 |
| creiht | and index that | 20:41 |
| creiht | you could even make all the swift services log to the same syslog target | 20:41 |
| jogo | creiht: that would be too big of a file, I think we want a all.error log as well | 20:42 |
| clayg | torgomatic: have you talked to peluse today? | 20:42 |
| notmyname | creiht: actually, since devstack is just running one of each process with one replica, that would make sense (and no need to index stuff) | 20:42 |
| torgomatic | clayg: no | 20:42 |
| * clayg thinks torgomatic has his autoresponder on | 20:42 | |
| clayg | ... that was too quick | 20:42 |
| creiht | jogo: oh sure | 20:42 |
| torgomatic | clayg: no ;) | 20:42 |
| clayg | torgomatic: do you automatically respond "no" to anything I ask you? | 20:42 |
| jogo | creiht: anyway we should wait for clarkb to chime in | 20:42 |
| creiht | jogo: sure | 20:42 |
| clayg | whoa! | 20:42 |
| torgomatic | clayg: mu? | 20:43 |
| clayg | lol | 20:43 |
| notmyname | jogo: for the record, you're comment about doing syslog in devstack because that's what's in production is funny since devstack has just one replica and one set of services | 20:43 |
| jogo | thanks for being so proactive on this btw | 20:43 |
| jogo | notmyname: heh babysteps | 20:43 |
| creiht | heh | 20:43 |
| notmyname | jogo: also that the devstack single drive is 1GB | 20:45 |
| notmyname | ah, under tempest it's 6GB | 20:45 |
| notmyname | jogo: creiht: thanks for helping to make devstack better :-) | 20:46 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add Storage Policy Support to Account HEAD https://review.openstack.org/73747 | 20:46 |
| clarkb | jogo: what do you need me to chime in on? | 20:56 |
| portante | clarkb: we were discussing swift and logging, use of rsyslog config to separate out swift logs from /var/log/messages, and how to remove duplication in proxy-s*.txt file | 20:57 |
| jogo | clarkb: see backlog on sorting out swift logs and sociology | 20:57 |
| clarkb | individual files for each service with timestamps is the sanest thing | 20:59 |
| clarkb | fwiw syslog doesn't hold up to the amount of data thrown at it during gating in addition to the other things jeblair mentioned | 20:59 |
| clarkb | there is a lot of tuning that must be done and no one has done it | 20:59 |
| portante | clarkb: so ... this is a no go? | 21:04 |
| clarkb | portante: no I think for swift it is probably ok | 21:05 |
| clarkb | portante: the problem is when we start throwing all of the other services at it too | 21:05 |
| portante | poor little VMs ... | 21:06 |
| notmyname | torgomatic: 5368709122000000000000000000000000000000 bytes | 21:25 |
| torgomatic | yeah, that should be enough :) | 21:26 |
| notmyname | https://twitter.com/notmyname/status/448209057908654080 | 21:26 |
| notmyname | and that's just with defaults | 21:26 |
| torgomatic | yeah, let your SLOs have 10000 segments instead of 1000, and now you're talking real storage | 21:27 |
| *** fifieldt has quit IRC | 21:30 | |
| *** tdasilva has left #openstack-swift | 21:32 | |
| *** piyush has quit IRC | 21:43 | |
| *** fifieldt has joined #openstack-swift | 21:46 | |
| *** dmsimard has quit IRC | 21:59 | |
| *** byeager has quit IRC | 22:02 | |
| *** byeager has joined #openstack-swift | 22:03 | |
| *** byeager has quit IRC | 22:07 | |
| *** Midnightmyth has quit IRC | 22:07 | |
| *** byeager has joined #openstack-swift | 22:24 | |
| *** gyee has quit IRC | 22:40 | |
| openstackgerrit | A change was merged to openstack/swift: Allow overriding of storage policy index. https://review.openstack.org/82211 | 23:05 |
| *** tdasilva has joined #openstack-swift | 23:10 | |
| *** byeager has quit IRC | 23:10 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Bump up sleep when expecting a timeout https://review.openstack.org/82664 | 23:18 |
| notmyname | so to test this I've gotta write an objest-server middleware that passes headers through fine but mutates the response body? | 23:21 |
| torgomatic | notmyname: ? | 23:21 |
| notmyname | torgomatic: it's your fault :-) | 23:21 |
| torgomatic | notmyname: oh, that one... yeah, just mangle the etag on the way back out, it'll be easeier | 23:21 |
| torgomatic | *easier | 23:21 |
| clayg | notmyname: +1 | 23:21 |
| notmyname | torgomatic: no, that won't work. that will be caught in the conditional at line 309 | 23:22 |
| torgomatic | notmyname: oh, okay then... yeah, it's bit-flipping time :) | 23:23 |
| notmyname | I need the same header etag as in the manifest, but a different computed etag in the proxy | 23:23 |
| torgomatic | the etag mangling will work with DLOs, though, and that's the same code path | 23:23 |
| clayg | i want a object-server file trans corrupting middleware neway - for science | 23:24 |
| clayg | notmyname: i'm telling you man "if random() > 0.7: chunk[-1] = 'X' # trolololo" | 23:42 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!