| openstackgerrit | gordon chung proposed openstack/ceilometer: do not configure worker specific items in init https://review.openstack.org/273792 | 00:08 |
|---|---|---|
| openstackgerrit | gordon chung proposed openstack/ceilometer: raise coordination error if not registered https://review.openstack.org/268292 | 00:11 |
| *** rbak has quit IRC | 00:19 | |
| *** cdent has joined #openstack-telemetry | 00:20 | |
| *** cdent has quit IRC | 00:21 | |
| *** diogogmt has quit IRC | 00:21 | |
| *** cdent has joined #openstack-telemetry | 00:24 | |
| *** cdent has quit IRC | 00:27 | |
| *** thorst has joined #openstack-telemetry | 00:39 | |
| *** ljxiash_ has quit IRC | 00:41 | |
| *** thorst has quit IRC | 00:43 | |
| *** Ephur has quit IRC | 00:46 | |
| *** chlong has joined #openstack-telemetry | 00:49 | |
| *** rbak has joined #openstack-telemetry | 00:53 | |
| *** rbak has quit IRC | 00:56 | |
| *** ljxiash has joined #openstack-telemetry | 01:05 | |
| *** cheneydc has joined #openstack-telemetry | 01:13 | |
| *** mragupat has quit IRC | 01:18 | |
| *** AndChat|224721 has joined #openstack-telemetry | 01:32 | |
| *** nicodemus_ has quit IRC | 01:32 | |
| *** yarkot has quit IRC | 01:32 | |
| *** mgagne has quit IRC | 01:32 | |
| *** mgagne has joined #openstack-telemetry | 01:32 | |
| *** mgagne has quit IRC | 01:32 | |
| *** mgagne has joined #openstack-telemetry | 01:32 | |
| openstackgerrit | Lianhao Lu proposed openstack/python-ceilometerclient: Enhances client to support unique meter retrieval https://review.openstack.org/272633 | 01:33 |
| *** pradk has quit IRC | 01:35 | |
| *** vishwanathj has quit IRC | 01:48 | |
| *** pradk has joined #openstack-telemetry | 01:51 | |
| *** changbl has quit IRC | 02:09 | |
| *** Ephur has joined #openstack-telemetry | 02:15 | |
| *** ljxiash has quit IRC | 02:25 | |
| *** ljxiash has joined #openstack-telemetry | 02:26 | |
| *** ljxiash has quit IRC | 02:26 | |
| *** ljxiash has joined #openstack-telemetry | 02:26 | |
| *** nicodemus_ has joined #openstack-telemetry | 02:28 | |
| *** prashantD has quit IRC | 02:29 | |
| *** AndChat|224721 has quit IRC | 02:30 | |
| *** AndChat|224721 has joined #openstack-telemetry | 02:43 | |
| *** nicodemus_ has quit IRC | 02:43 | |
| *** yarkot has joined #openstack-telemetry | 02:48 | |
| *** mgagne has quit IRC | 02:51 | |
| *** mgagne has joined #openstack-telemetry | 02:51 | |
| *** achatterjee has joined #openstack-telemetry | 02:51 | |
| *** pradk has quit IRC | 03:16 | |
| *** pradk has joined #openstack-telemetry | 03:28 | |
| *** datravis has quit IRC | 03:29 | |
| *** datravis has joined #openstack-telemetry | 03:30 | |
| *** nicodemus_ has joined #openstack-telemetry | 03:33 | |
| *** AndChat|224721 has quit IRC | 03:33 | |
| openstackgerrit | liusheng proposed openstack/aodh: Log deprecation message if users use nosql backend https://review.openstack.org/273865 | 03:43 |
| *** liamji has joined #openstack-telemetry | 03:46 | |
| *** datravis has quit IRC | 04:19 | |
| *** nicodemus_ has quit IRC | 04:30 | |
| *** nicodemus_ has joined #openstack-telemetry | 04:31 | |
| *** prashantD_ has joined #openstack-telemetry | 04:34 | |
| *** nicodemus_ has quit IRC | 04:40 | |
| *** nicodemus_ has joined #openstack-telemetry | 04:41 | |
| *** ljxiash has quit IRC | 04:42 | |
| *** ljxiash has joined #openstack-telemetry | 04:42 | |
| *** ljxiash has quit IRC | 04:47 | |
| *** thorst has joined #openstack-telemetry | 04:59 | |
| *** thorst has quit IRC | 04:59 | |
| *** ljxiash has joined #openstack-telemetry | 04:59 | |
| *** prashantD_ has quit IRC | 05:00 | |
| *** thorst has joined #openstack-telemetry | 05:00 | |
| *** thorst has quit IRC | 05:04 | |
| *** chlong has quit IRC | 05:12 | |
| *** cheneydc has quit IRC | 05:15 | |
| *** cheneydc has joined #openstack-telemetry | 05:18 | |
| *** yprokule has joined #openstack-telemetry | 05:29 | |
| *** thorst has joined #openstack-telemetry | 05:30 | |
| *** chlong has joined #openstack-telemetry | 05:32 | |
| *** thorst has quit IRC | 05:41 | |
| openstackgerrit | Lianhao Lu proposed openstack/gnocchi: Added original_resource_id field into resource https://review.openstack.org/273008 | 05:50 |
| *** liamji has quit IRC | 05:57 | |
| *** ljxiash has quit IRC | 05:57 | |
| *** ljxiash has joined #openstack-telemetry | 05:58 | |
| *** ljxiash_ has joined #openstack-telemetry | 06:00 | |
| *** ljxiash__ has joined #openstack-telemetry | 06:01 | |
| *** ljxiash has quit IRC | 06:02 | |
| *** ljxiash has joined #openstack-telemetry | 06:03 | |
| *** ljxiash__ has quit IRC | 06:03 | |
| *** thorst has joined #openstack-telemetry | 06:04 | |
| *** ljxiash_ has quit IRC | 06:04 | |
| *** thorst has quit IRC | 06:18 | |
| *** cheneydc_ has joined #openstack-telemetry | 06:22 | |
| *** cheneydc_ has left #openstack-telemetry | 06:24 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ceilometer: Imported Translations from Zanata https://review.openstack.org/273346 | 06:26 |
| *** chlong has quit IRC | 06:28 | |
| *** rcernin has quit IRC | 06:29 | |
| *** chlong has joined #openstack-telemetry | 06:39 | |
| openstackgerrit | Merged openstack/aodh: threshold: fix statistics empty case https://review.openstack.org/273531 | 06:53 |
| *** ljxiash has quit IRC | 06:54 | |
| *** ljxiash has joined #openstack-telemetry | 06:55 | |
| *** ljxiash has quit IRC | 06:59 | |
| *** _nadya_ has joined #openstack-telemetry | 07:00 | |
| *** _nadya_ has quit IRC | 07:01 | |
| *** _nadya_ has joined #openstack-telemetry | 07:10 | |
| *** vishwanathj has joined #openstack-telemetry | 07:19 | |
| *** vishwanathj is now known as vishwanathj_zzz | 07:24 | |
| *** rcernin has joined #openstack-telemetry | 07:26 | |
| *** chlong has quit IRC | 07:32 | |
| *** belmoreira has joined #openstack-telemetry | 07:33 | |
| *** ljxiash has joined #openstack-telemetry | 07:33 | |
| *** Liuqing has joined #openstack-telemetry | 07:38 | |
| *** safchain has joined #openstack-telemetry | 07:44 | |
| openstackgerrit | Merged openstack/ceilometer: Enhances get_meters to return unique meters https://review.openstack.org/259626 | 07:47 |
| *** boris-42 has joined #openstack-telemetry | 07:58 | |
| *** _nadya_ has quit IRC | 08:16 | |
| openstackgerrit | Merged openstack/python-ceilometerclient: Updated from global requirements https://review.openstack.org/269641 | 08:17 |
| *** shardy has joined #openstack-telemetry | 08:55 | |
| *** diogogmt has joined #openstack-telemetry | 09:00 | |
| *** shardy has quit IRC | 09:01 | |
| *** shardy has joined #openstack-telemetry | 09:02 | |
| *** _nadya_ has joined #openstack-telemetry | 09:07 | |
| *** mattyw has joined #openstack-telemetry | 09:08 | |
| *** sanjana has quit IRC | 09:10 | |
| *** liamji has joined #openstack-telemetry | 09:13 | |
| *** yassine__ has joined #openstack-telemetry | 09:13 | |
| openstackgerrit | Ren Qiaowei proposed openstack/ceilometer: xenapi: support the session when xenserver is master https://review.openstack.org/215393 | 09:16 |
| *** efoley has joined #openstack-telemetry | 09:24 | |
| *** liamji has quit IRC | 09:32 | |
| *** Liuqing has quit IRC | 09:34 | |
| *** links has joined #openstack-telemetry | 09:38 | |
| *** links has quit IRC | 09:38 | |
| *** cheneydc has quit IRC | 09:59 | |
| *** efoley_ has joined #openstack-telemetry | 10:05 | |
| *** efoley has quit IRC | 10:08 | |
| *** ildikov has quit IRC | 10:14 | |
| *** sanjana has joined #openstack-telemetry | 10:20 | |
| *** sanjana has quit IRC | 10:37 | |
| *** mattyw has quit IRC | 10:37 | |
| *** efoley__ has joined #openstack-telemetry | 10:39 | |
| *** thorst has joined #openstack-telemetry | 10:42 | |
| *** efoley_ has quit IRC | 10:42 | |
| *** mattyw has joined #openstack-telemetry | 10:44 | |
| *** Liuqing has joined #openstack-telemetry | 10:48 | |
| *** jaypipes has joined #openstack-telemetry | 10:50 | |
| *** efoley__ has quit IRC | 10:58 | |
| *** ildikov has joined #openstack-telemetry | 11:03 | |
| *** efoley__ has joined #openstack-telemetry | 11:04 | |
| *** _nadya_ has quit IRC | 11:06 | |
| *** jaypipes is now known as leakypipes | 11:10 | |
| *** Liuqing has quit IRC | 11:15 | |
| *** efoley__ is now known as efoley | 11:23 | |
| *** thorst has quit IRC | 11:24 | |
| *** shardy has quit IRC | 11:27 | |
| *** shardy has joined #openstack-telemetry | 11:28 | |
| *** thorst has joined #openstack-telemetry | 11:29 | |
| *** hparekh_ has joined #openstack-telemetry | 11:46 | |
| *** thorst has quit IRC | 11:47 | |
| *** mattyw has quit IRC | 11:57 | |
| *** thorst has joined #openstack-telemetry | 12:03 | |
| *** thorst has quit IRC | 12:08 | |
| *** _nadya_ has joined #openstack-telemetry | 12:13 | |
| *** rcernin has quit IRC | 12:18 | |
| *** rcernin has joined #openstack-telemetry | 12:18 | |
| jd__ | lolilol gate BROKEN | 12:19 |
| jd__ | openstack sdk client changes its output | 12:20 |
| jd__ | #fun | 12:20 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: devstack: fix publicURL retrieval https://review.openstack.org/274043 | 12:22 |
| *** efoley has quit IRC | 12:27 | |
| *** cdent has joined #openstack-telemetry | 12:38 | |
| *** mattyw has joined #openstack-telemetry | 12:40 | |
| *** julim has joined #openstack-telemetry | 12:47 | |
| *** efoley has joined #openstack-telemetry | 12:47 | |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: devstack: fix publicURL retrieval https://review.openstack.org/274043 | 13:05 |
| *** kragniz has quit IRC | 13:12 | |
| *** efoley_ has joined #openstack-telemetry | 13:15 | |
| *** gordc has joined #openstack-telemetry | 13:16 | |
| *** kragniz has joined #openstack-telemetry | 13:18 | |
| *** efoley has quit IRC | 13:19 | |
| *** changbl has joined #openstack-telemetry | 13:27 | |
| *** leitan has joined #openstack-telemetry | 13:41 | |
| *** datravis has joined #openstack-telemetry | 13:41 | |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: devstack: fix publicURL retrieval https://review.openstack.org/274043 | 13:44 |
| *** diogogmt_ has joined #openstack-telemetry | 13:47 | |
| *** diogogmt has quit IRC | 13:48 | |
| *** diogogmt_ is now known as diogogmt | 13:48 | |
| *** datravis has quit IRC | 13:49 | |
| *** pradk has quit IRC | 13:51 | |
| *** hparekh_ has quit IRC | 13:52 | |
| *** datravis has joined #openstack-telemetry | 14:02 | |
| *** leitan has quit IRC | 14:06 | |
| *** diogogmt has quit IRC | 14:06 | |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: setup: include config file at install https://review.openstack.org/274078 | 14:11 |
| *** rbak has joined #openstack-telemetry | 14:13 | |
| *** ljxiash has quit IRC | 14:15 | |
| gordc | cdent: we angered the overlords. they blacklisted our release. | 14:17 |
| gordc | their vengence is swift. | 14:17 |
| cdent | saw that. pbr and pip's fault yet? | 14:17 |
| cdent | s/yet/yeah/ | 14:17 |
| gordc | apparently. | 14:17 |
| gordc | but all that was managed by global-reqs... so... | 14:18 |
| gordc | now jd__ will come in and say 'i told you so' | 14:18 |
| jd__ | hihi | 14:18 |
| *** ljxiash has joined #openstack-telemetry | 14:18 | |
| gordc | get out! | 14:18 |
| *** diogogmt has joined #openstack-telemetry | 14:21 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer: rewriting history https://review.openstack.org/273049 | 14:22 |
| *** diogogmt has quit IRC | 14:25 | |
| *** _nadya_ has quit IRC | 14:28 | |
| *** ljxiash has quit IRC | 14:33 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer: cleanup devstack plugin for gnocchi https://review.openstack.org/270818 | 14:43 |
| *** nicodemus_ has joined #openstack-telemetry | 14:44 | |
| nicodemus_ | hello sileht, do you have a moment? | 14:45 |
| *** efoley_ has quit IRC | 14:55 | |
| *** cdent has quit IRC | 14:57 | |
| *** leitan has joined #openstack-telemetry | 14:59 | |
| *** ljxiash has joined #openstack-telemetry | 14:59 | |
| *** liamji has joined #openstack-telemetry | 15:00 | |
| *** achatterjee has quit IRC | 15:00 | |
| *** cdent has joined #openstack-telemetry | 15:01 | |
| *** efoley_ has joined #openstack-telemetry | 15:07 | |
| *** krotscheck has quit IRC | 15:10 | |
| *** pradk has joined #openstack-telemetry | 15:18 | |
| *** pradk has quit IRC | 15:18 | |
| *** krotscheck has joined #openstack-telemetry | 15:21 | |
| *** yprokule has quit IRC | 15:22 | |
| *** ildikov has quit IRC | 15:24 | |
| *** pradk has joined #openstack-telemetry | 15:26 | |
| *** mragupat has joined #openstack-telemetry | 15:47 | |
| openstackgerrit | Xia Linjuan proposed openstack/python-aodhclient: Fix aodh client fails when command with the arg --time-constraint https://review.openstack.org/272946 | 15:57 |
| *** safchain has quit IRC | 16:04 | |
| *** leakypipes has quit IRC | 16:04 | |
| *** belmoreira has quit IRC | 16:08 | |
| gordc | hm... aodh break? | 16:13 |
| cdent | My new mission in life is to -1 jd__'s +2s | 16:13 |
| cdent | It's a fun game. | 16:13 |
| *** mgarza has joined #openstack-telemetry | 16:14 | |
| *** ljxiash has quit IRC | 16:15 | |
| gordc | he will return fire | 16:15 |
| *** ljxiash has joined #openstack-telemetry | 16:15 | |
| gordc | cdent: wait. you -1 my patch!? | 16:15 |
| gordc | dammit | 16:15 |
| * gordc puts cdents name on the list | 16:16 | |
| cdent | It's really good stuff but not good enough! | 16:16 |
| gordc | *mumbles* yeah... you just keep talking | 16:16 |
| cdent | It actually makes it rather clear that suff has changed lately. Which is nice to see. | 16:16 |
| cdent | (jay was grousing about ceilometer today, and I had to fight back a bit) | 16:16 |
| gordc | what's ceilometer? | 16:16 |
| cdent | some legacy thing | 16:17 |
| gordc | *shrugs* i assume everyone is talking shit about another project behind their back | 16:18 |
| gordc | it's the openstack way | 16:18 |
| gordc | presentation idea? | 16:19 |
| gordc | just a few slides to "how to make yourself feel better in opentack" | 16:19 |
| *** _nadya_ has joined #openstack-telemetry | 16:19 | |
| *** alejandrito has joined #openstack-telemetry | 16:19 | |
| cdent | "how to talk shit about some other project without knowing much about it" | 16:19 |
| *** ljxiash has quit IRC | 16:19 | |
| gordc | naw knowing about it is effort. | 16:20 |
| gordc | my projects broke, let's redirect focus to another | 16:20 |
| gordc | cdent: how goes midcycle? | 16:21 |
| *** renatoarmani has joined #openstack-telemetry | 16:21 | |
| cdent | midcycle is done and we've moved on to a hotel near london to have a little mirantis meetup | 16:22 |
| gordc | cool cool | 16:24 |
| cdent | midcycle was described by most people as "productive" | 16:24 |
| gordc | sure. why not. your company paid a few thousand | 16:26 |
| *** ildikov has joined #openstack-telemetry | 16:26 | |
| gordc | cdent: patting yourself on the back is still a pat on the back.lol | 16:27 |
| cdent | hah! | 16:27 |
| cdent | _I_ didn't describe it as productive | 16:27 |
| gordc | cdent: not you per se | 16:27 |
| *** rcernin has quit IRC | 16:27 | |
| gordc | jd__: something merge today? seems like integration gate is busted... no alarms. | 16:28 |
| gordc | pradk: you have time for this https://bugs.launchpad.net/aodh/+bug/1539685? | 16:31 |
| openstack | Launchpad bug 1539685 in Aodh "zaqar not found error on startup" [High,Triaged] | 16:31 |
| pradk | gordc, looking | 16:32 |
| gordc | pradk: thanks | 16:32 |
| pradk | gordc, hmm do we need to add it in the requires? i thought the plugin wont be loaded unless its explicitly called | 16:33 |
| gordc | pradk: i think it's being loaded. but zaqar isn't enabled | 16:34 |
| gordc | so we need to not get zaqar on init but when needed | 16:35 |
| pradk | gordc, yea reason i changed it to init is so its loaded only once .. i guess that wont fly here | 16:35 |
| gordc | well you can load it the first time you use it | 16:35 |
| pradk | gordc, originally i had it called in notify call | 16:36 |
| gordc | pradk: sure. that's probably a good place | 16:38 |
| pradk | gordc, k i'll push a fix in a bit | 16:40 |
| *** cdent has quit IRC | 16:48 | |
| *** cdent has joined #openstack-telemetry | 16:54 | |
| *** efoley_ has quit IRC | 16:55 | |
| *** cdent has quit IRC | 16:55 | |
| mnaser | does the pipeline.yaml file have to be located on all agents? or only on the nodes that handle notificaitons? | 16:56 |
| gordc | mnaser: all agents. the polling agents still use the definition to decide what to poll (we didn't get around to creating a polling specific file) | 16:57 |
| gordc | but the actual transformations/pipline work is only on notification agent | 16:58 |
| mnaser | gordc: ok sounds good, trying to reduce the amount of data we're retrieving, a lot of it we're not using | 16:58 |
| mnaser | thank you | 16:58 |
| gordc | mnaser: that's highly recommended. (but rarely people actually do it) | 16:59 |
| mnaser | yep, you kinda don't have a choice (unless you want to throw more $$$ at db) | 16:59 |
| gordc | mnaser: i find ususally blaming scalability is a good alternative ;) | 17:00 |
| mnaser | so you're saying ceilometer cant run in a 1gb vps and handle data for 1000 vms? toss it out | 17:01 |
| gordc | hahah. we're working on it. magic dust is stuck at customs | 17:02 |
| mnaser | gordc: is there any way from the API to remove the old samples/resources/meters that aren't used anymore or do we need to do some db magic | 17:04 |
| openstackgerrit | gordon chung proposed openstack/aodh: change to keystonev3 https://review.openstack.org/274158 | 17:05 |
| gordc | mnaser: ceilometer or gnocchi? ceilometer, you can set a ttl... but it's global | 17:05 |
| gordc | and if you run sql, you need to run it as a cron job | 17:06 |
| mnaser | gordc: ceilometer, just looking to remove data we don't need right now (instead of waiting for TTL) | 17:06 |
| gordc | mnaser: i see. no there's no 'delete this now' command. | 17:06 |
| mnaser | we have 35 metrics per VM, we need 3. so cleaning this up would be nice | 17:07 |
| mnaser | i guess im on my own to investigate this and hope not to break the db :< | 17:07 |
| *** liamji has quit IRC | 17:11 | |
| gordc | mnaser: which db? | 17:11 |
| mnaser | mongo for this setup | 17:11 |
| *** _nadya_ has quit IRC | 17:12 | |
| mnaser | it seems pretty straight forward though | 17:12 |
| jd__ | gordc: openstack client broke gnocchi gate | 17:12 |
| mnaser | removing the data for it is straightforward, cleaning up the meters' existance seems more dificult | 17:12 |
| gordc | jd__: they changed to v3 also https://github.com/openstack-dev/devstack/commit/f4ce44bf3fbf06e53c2ae3ec6aa4996831cf4605 | 17:13 |
| jd__ | gordc: ah that might be that then, idk | 17:13 |
| jd__ | the return from openstack client changed, that's all I know | 17:13 |
| gordc | jd__: all our devstack configurations are straight busted | 17:13 |
| jd__ | lol. | 17:14 |
| jd__ | my fix is at https://review.openstack.org/#/c/274043/ | 17:14 |
| jd__ | but it seems it's not enough | 17:14 |
| gordc | jd__: i have https://review.openstack.org/#/c/274158/ | 17:15 |
| gordc | but we also need to define a domain | 17:15 |
| gordc | i've no idea what exactly is neede | 17:15 |
| jd__ | rofl | 17:15 |
| jd__ | this is really hilarious, somehow | 17:15 |
| jd__ | why the hell do we have to set a version in those url | 17:16 |
| jd__ | we don't give a !*(@! about the verseion | 17:16 |
| *** ljxiash has joined #openstack-telemetry | 17:17 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/aodh: Load zaqar client outside init https://review.openstack.org/274170 | 17:18 |
| gordc | jd__: i'm going to disable that test. we can't fix them one by one. | 17:18 |
| gordc | https://github.com/openstack-dev/devstack/blob/master/stack.sh#L1018-L1028 | 17:18 |
| jd__ | gordc: why can't we fix it? | 17:19 |
| jd__ | oh you mean that we need to merge multiple patches at once? | 17:19 |
| gordc | right | 17:19 |
| jd__ | I think it'd be better to raise the issue on the ml | 17:19 |
| jd__ | this is not really "acceptable", even if i'm not blaming anyone | 17:20 |
| jd__ | shit happens | 17:20 |
| jd__ | or maybe we were doing something "wrong" since the beginning without knowing, but since we were the first plugins, we can't be blamed ;) | 17:21 |
| gordc | jd__: yeah. everyone doesn't have additional service_credentials | 17:26 |
| gordc | they all seem to just use keystone_authtoken and configure_auth_token_middleware from devstack | 17:26 |
| gordc | not really sure | 17:26 |
| *** mattyw has quit IRC | 17:33 | |
| nicodemus_ | hello jd__, do you have a moment? | 17:37 |
| jd__ | nicodemus_: a short one :) | 17:37 |
| openstackgerrit | gordon chung proposed openstack/aodh: change to keystonev3 https://review.openstack.org/274158 | 17:38 |
| nicodemus_ | I'll be brief then | 17:38 |
| nicodemus_ | I'm testing metricd with a small ceph pool, about 80k of objects | 17:38 |
| nicodemus_ | and works like a charm | 17:38 |
| nicodemus_ | I'm going to gradually increase the number of unprocessed measures in the pool, to see exactly when it can no longer cope with the amount of data | 17:39 |
| nicodemus_ | plus I'm gathering info about the number of measures processed per minute, and how many of them are being skipped (already processed) | 17:40 |
| gordc | nice. that'd be good info | 17:40 |
| nicodemus_ | with 80k of rados objects, metricd is using under 1gb of memory... with 7M it eats up all ram | 17:42 |
| nicodemus_ | I'll let you know when I find something conclusive. | 17:42 |
| jd__ | nicodemus_: ok | 17:44 |
| jd__ | good to know | 17:44 |
| jd__ | feed me with information! :) | 17:44 |
| nicodemus_ | haha will do | 17:44 |
| jd__ | nicodemus_: using sileht thread disabling patch? | 17:44 |
| jd__ | because that should multiply the perf by 40x :) | 17:44 |
| jd__ | also what does that mean 80k objects vs 7M? what are these objects? | 17:45 |
| nicodemus_ | I applied patch set 3 (the latest), however as per the logs it's still using threads... I didn't see the "rados.run_in_thread is monkeypatched" line in the logs | 17:46 |
| nicodemus_ | the objects in the gnocchi pool in ceph, the output of "rados df" | 17:46 |
| nicodemus_ | right now I have two labs, one in which I have over 7M objects and four metricd instances qith 16gb RAM, and a second separate lab with 80k of objects and two metricd instances with 4gb RAM | 17:47 |
| *** guillaume_ has joined #openstack-telemetry | 17:58 | |
| jd__ | hum I wonder if the logging system is initialized when the code is logging that thread are disabled | 18:02 |
| jd__ | nicodemus_: ok, I'm not sure how the number of objects can affect memory usage, but we'll see what you got | 18:02 |
| nicodemus_ | perhaps memory usage is not related to the num of objects, it's just a guess | 18:05 |
| nicodemus_ | I'm letting the pool to grow now, when there are 200-300k objects I'll start metricd again and see | 18:05 |
| gordc | nicodemus_: you mean the backlog right? | 18:10 |
| *** _nadya_ has joined #openstack-telemetry | 18:11 | |
| gordc | it makes sense since we're grabbing everything | 18:11 |
| nicodemus_ | gordc: I'm not sure about the backlog... What I do is start all ceilometer-agents and stop metricd so that new measures are pushed to gnocchi without metricd processing them | 18:12 |
| nicodemus_ | when "rados df" shows about 300k of objects in the ceph pool, I'll stop the agents and start metricd | 18:13 |
| nicodemus_ | I'm trying to replicate what happened in my other lab (the one with 7M of rados objects in the pool) in a controlled fashion | 18:14 |
| gordc | yeah, that's the backlog. if you have metricd disabled you basically have a bunch of unprocessed datapoints sitting around. | 18:15 |
| gordc | and right now, the logic is grab it all and process it. | 18:15 |
| gordc | so if you grab 7M points, you can imagine how much data you are pulling into memory | 18:16 |
| *** prashantD has joined #openstack-telemetry | 18:20 | |
| *** renatoarmani has quit IRC | 18:21 | |
| *** diogogmt has joined #openstack-telemetry | 18:21 | |
| *** diogogmt has quit IRC | 18:26 | |
| jd__ | oh yeah that's completely logical like gordc explained | 18:32 |
| jd__ | that's interesting, but it should be easily fixable | 18:32 |
| jd__ | I'll give it a thought and will write a patch :) | 18:32 |
| * jd__ never thought people would let grow the backlog on purpose lol | 18:32 | |
| nicodemus_ | well, at the beginning it wasn't intentional :P | 18:36 |
| nicodemus_ | and now I find myself with a huge backlog waiting to be processed | 18:37 |
| nicodemus_ | taking into account that I don't see the "rados.run_in_thread is monkeypatched" line in metricd log (as per sileht's patch)... how could I be sure if metricd is using a single librados thread? | 18:40 |
| openstackgerrit | gordon chung proposed openstack/ceilometer: integration-gate: fix publicURL retrieval https://review.openstack.org/274197 | 18:40 |
| *** prashantD has quit IRC | 18:40 | |
| *** prashantD has joined #openstack-telemetry | 18:41 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer: integration-gate: fix publicURL retrieval https://review.openstack.org/274197 | 18:41 |
| *** datravis has quit IRC | 18:41 | |
| gordc | nicodemus_: it can recover when there's 80k objects? | 18:41 |
| nicodemus_ | with 80k objects it's working like a charm, however the number of objects decreased by 2000 on one hour (give or take) | 18:43 |
| gordc | nicodemus_: right now i think the issue is gnocchi is kinda behaving like a MQ. as the backlog grows, the performance drops. so it's a double hit... ideally there should be enough workers to keep backlog low | 18:43 |
| gordc | 2000 in one hour is not a lot. this is while new ceilometer agents are still active or have they been shut down? | 18:44 |
| nicodemus_ | all of them were stopped, only metricd is accesing the ceph pool | 18:44 |
| *** lsmola has quit IRC | 18:45 | |
| gordc | yeah, that's not really that good tbh. | 18:45 |
| gordc | i'd imagine any small size environment is sending more than 2000 datapoints per hour | 18:46 |
| nicodemus_ | correct, in this test environment the ceilometer agent send between 30k-40k datapoints | 18:47 |
| nicodemus_ | that's three ceilometer-agent-compute and one ceilometer-agent-central | 18:47 |
| gordc | that said, pulling 80k objects in one go. is not that great either. | 18:47 |
| gordc | nicodemus_: polling interval set at? | 18:48 |
| nicodemus_ | 60 seconds | 18:48 |
| gordc | nicodemus_: something to investigate. 2k/hr is slow. very very slow. | 18:48 |
| gordc | something is up. | 18:48 |
| nicodemus_ | http://paste.openstack.org/show/485473/ that's 40 minutes of "rados df", command executed at one minute intervals | 18:51 |
| nicodemus_ | it starts with 87k objects, and finishes with 85k | 18:51 |
| gordc | nicodemus_: is that just number of objects in ceph in general? | 18:52 |
| gordc | nicodemus_: can you try running gnocchi status instead? | 18:52 |
| nicodemus_ | so the rate would be maybe 3k/hour | 18:52 |
| nicodemus_ | yes, just number of objects | 18:52 |
| nicodemus_ | ok, I'll start metricd again and gather info using gnocchi status | 18:53 |
| gordc | nicodemus_: oh. ok. yeah, well that shouldn't necessary decrease, for every item processed, it will still write back the aggregated datapoint back to ceph | 18:53 |
| gordc | and depending on how many aggregates/granularities you are calculating, it could increase. | 18:54 |
| *** pcaruana has joined #openstack-telemetry | 18:57 | |
| nicodemus_ | gordc: is it possible for gnocchi status to take a while? I executed the command about five minutes ago, and still no output whatsoever | 18:58 |
| gordc | um. possibly? i never had an 80k backlog to be honest. | 19:00 |
| gordc | biab. have a meeting | 19:00 |
| *** gordc has quit IRC | 19:00 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/aodh: Load zaqar client outside init https://review.openstack.org/274170 | 19:05 |
| *** krotscheck has quit IRC | 19:17 | |
| *** renatoarmani has joined #openstack-telemetry | 19:21 | |
| *** pcaruana has quit IRC | 19:24 | |
| *** gordc has joined #openstack-telemetry | 19:26 | |
| *** renatoarmani has quit IRC | 19:26 | |
| *** renatoarmani has joined #openstack-telemetry | 19:28 | |
| *** yassine__ has quit IRC | 19:28 | |
| gordc | nicodemus_: back. any updates? on running 'gnocchi status'? | 19:34 |
| *** pcaruana has joined #openstack-telemetry | 19:37 | |
| nicodemus_ | gnocchi status timeouts :( | 19:44 |
| nicodemus_ | gordc: Request to http://192.168.100.113:8041/v1/status timed out | 19:44 |
| mnaser | right. possible reasons for why ceilometer is not processing notifications...? http://i.imgur.com/sIV5yVF.png | 19:45 |
| gordc | nicodemus_: strange. let me check the command | 19:46 |
| mnaser | i have 2x ceilometer-collector processes running on two different servers | 19:46 |
| gordc | mnaser: notifications.* queues are handled by notification agent | 19:46 |
| mnaser | got two of those running too :< | 19:46 |
| gordc | any errors? liberty? | 19:47 |
| gordc | do you have workers enabled? or two separate processes? | 19:47 |
| mnaser | liberty, running in DEBUG it just seems idle | 19:47 |
| mnaser | single process on two different servers | 19:47 |
| gordc | so no workload_partitioning? | 19:47 |
| mnaser | http://pastebin.com/nhxct9uU | 19:47 |
| mnaser | i do have workload partitioning | 19:47 |
| gordc | ack. | 19:48 |
| mnaser | oh wait | 19:48 |
| mnaser | its set to false? | 19:48 |
| gordc | you should probably have it enabled (but that's not the reason) | 19:48 |
| mnaser | could it be possible that | 19:48 |
| mnaser | 2016-01-29 19:26:26.193 32054 DEBUG oslo_service.service [-] coordination.backend_url = redis://172.17.100.1 log_opt_values /usr/lib/python2.7/dist-packages/oslo_config/cfg.py:2233 | 19:48 |
| mnaser | combined with notification.workload_partitioning = False | 19:48 |
| mnaser | means it does nothing? | 19:48 |
| gordc | it should just continue but not actually coordinate | 19:49 |
| gordc | so all your transformations may be wrong. | 19:49 |
| mnaser | let me fix that then | 19:49 |
| gordc | mnaser: check sudo rabbitmqctl list_consumers. it seems nothing is reading messages | 19:50 |
| mnaser | yeah, nothing (i see it here infront of me from the admin panel) | 19:50 |
| gordc | also, i should add, that queue is going to take forever to clear... rabbit does not do well when queues get large | 19:50 |
| gordc | try adding a third agent and see what happens | 19:51 |
| *** datravis has joined #openstack-telemetry | 19:51 | |
| mnaser | gordc: i dont really easily have access for hw to setup a third agent for right now | 19:53 |
| mnaser | will adding workers help? | 19:53 |
| gordc | workers is slightly broken right now* | 19:55 |
| gordc | by that i mean if you start service with --worker param | 19:55 |
| mnaser | let me try to stop the other notification and remove all the coordination code.. and see if i tdoes something | 19:55 |
| mnaser | whah wait | 19:56 |
| mnaser | 2016-01-29 19:53:50.432 1624 DEBUG ceilometer.pipeline [-] Pipeline config file: /etc/ceilometer/pipeline.yaml _setup_pipeline_manager /usr/lib/python2.7/dist-packages/ceilometer/pipeline.py:792 ... few lines later ... 2016-01-29 19:53:50.519 1624 DEBUG ceilometer.pipeline [-] Pipeline config file: None _setup_pipeline_manager /usr/lib/python2.7/dist-packages/ceilometer/pipeline.py:792 | 19:56 |
| mnaser | oh it looks like i have my pipeline config file but no event pipeline config file | 19:57 |
| gordc | mnaser: https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_pipeline.yaml this? | 19:58 |
| mnaser | yeah im adding it to /etc and trying to restart | 19:58 |
| gordc | yeah, you need that. | 19:58 |
| gordc | mnaser: if that's the case, maybe raise a bug saying it didn't give you a good warning | 19:58 |
| mnaser | that was it | 19:58 |
| mnaser | its processing data now | 19:59 |
| mnaser | i guess when i enabled the event saving too | 19:59 |
| mnaser | it resulted in this | 19:59 |
| gordc | mnaser: yeah. if you have store_events off, it might ignore it. | 19:59 |
| gordc | mnaser: that should've raise an error i think. | 20:00 |
| mnaser | yeah now there's a fun 2.8 million records to go through.. | 20:01 |
| gordc | mnaser: i wouldn't be surprise if it didn't drop. it's going to chug along for a while. | 20:02 |
| mnaser | the data isn't important right now so i think im going to purge the queue | 20:02 |
| gordc | mnaser: good choice | 20:03 |
| mnaser | now to report a bug about this too | 20:03 |
| *** prashantD has quit IRC | 20:05 | |
| *** pcaruana has quit IRC | 20:06 | |
| gordc | mnaser: thanks | 20:06 |
| *** pcaruana has joined #openstack-telemetry | 20:09 | |
| *** shardy has quit IRC | 20:18 | |
| gordc | nicodemus_: it seems to be a large query but not any larger than query to metricd runs... | 20:19 |
| *** pcaruana has quit IRC | 20:19 | |
| gordc | do you see your call in your gnocchi api logs? | 20:19 |
| openstackgerrit | gordon chung proposed openstack/ceilometer: rewriting history https://review.openstack.org/273049 | 20:25 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ceilometer: Updated from global requirements https://review.openstack.org/272780 | 20:44 |
| nicodemus_ | gordc: yes | 20:57 |
| nicodemus_ | but still no reply :( | 20:57 |
| *** _nadya_ has quit IRC | 21:09 | |
| *** leitan has quit IRC | 21:18 | |
| *** vishwanathj_zzz has quit IRC | 21:23 | |
| *** thorst_ has joined #openstack-telemetry | 21:36 | |
| *** rcernin has joined #openstack-telemetry | 21:36 | |
| *** thorst_ has quit IRC | 21:39 | |
| *** thorst_ has joined #openstack-telemetry | 21:41 | |
| *** thorst__ has joined #openstack-telemetry | 21:43 | |
| *** thorst_ has quit IRC | 21:46 | |
| openstackgerrit | Merged openstack/ceilometer: integration-gate: fix publicURL retrieval https://review.openstack.org/274197 | 21:47 |
| openstackgerrit | gordon chung proposed openstack/ceilometer: test https://review.openstack.org/274262 | 21:49 |
| *** rvasilets__ has joined #openstack-telemetry | 21:50 | |
| rvasilets__ | Hi is ceilometer client working? | 21:51 |
| *** diogogmt has joined #openstack-telemetry | 21:51 | |
| *** zqfan has quit IRC | 21:51 | |
| *** diogogmt has quit IRC | 21:52 | |
| rvasilets__ | We got all ceilometer jobs failed in rally gates | 21:52 |
| rvasilets__ | http://logs.openstack.org/59/274059/1/check/gate-rally-dsvm-rally/c484675/rally-plot/results.html.gz#/CeilometerEvents.create_user_and_get_event/failures | 21:52 |
| rvasilets__ | with such error | 21:52 |
| rvasilets__ | Do you know what is it? | 21:52 |
| *** diogogmt has joined #openstack-telemetry | 21:53 | |
| *** alejandrito has quit IRC | 21:54 | |
| *** pradk has quit IRC | 22:01 | |
| gordc | they changed devstack to default to keystonev3? are you passing in a domain? | 22:11 |
| gordc | rvasilets__: i'm heading offline. please check you are passing in correct v3 params... if yes, please open a bug. | 22:12 |
| gordc | rvasilets__: https://bugs.launchpad.net/ceilometer/+bug/1539728 that exists currently | 22:12 |
| openstack | Launchpad bug 1539728 in Ceilometer "keystone v3 breaks integration gate" [High,Triaged] | 22:12 |
| gordc | heading off | 22:12 |
| *** gordc has quit IRC | 22:14 | |
| *** thorst_ has joined #openstack-telemetry | 22:14 | |
| *** thorst__ has quit IRC | 22:17 | |
| *** changbl has quit IRC | 22:18 | |
| *** rbak has quit IRC | 22:21 | |
| *** diogogmt has quit IRC | 22:25 | |
| *** thorst_ has quit IRC | 22:28 | |
| *** mragupat has quit IRC | 22:34 | |
| *** renatoarmani has quit IRC | 22:43 | |
| jd__ | nicodemus_: if sileht patch is not working, clearly it's normal, it's 40x times slower… | 22:45 |
| *** alejandrito has joined #openstack-telemetry | 23:01 | |
| *** cdent has joined #openstack-telemetry | 23:07 | |
| openstackgerrit | Merged openstack/gnocchi: devstack: fix publicURL retrieval https://review.openstack.org/274043 | 23:17 |
| *** leitan has joined #openstack-telemetry | 23:23 | |
| *** mgarza has quit IRC | 23:30 | |
| *** prashantD has joined #openstack-telemetry | 23:33 | |
| *** prashantD has quit IRC | 23:37 | |
| *** alejandrito has quit IRC | 23:41 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
