| *** vint_bra has quit IRC | 01:01 | |
| *** vint_bra has joined #openstack-telemetry | 01:05 | |
| *** zhurong has joined #openstack-telemetry | 01:19 | |
| *** zhangguoqing has joined #openstack-telemetry | 01:21 | |
| *** vint_bra has quit IRC | 01:34 | |
| *** dave-mccowan has joined #openstack-telemetry | 01:34 | |
| *** vint_bra has joined #openstack-telemetry | 01:36 | |
| *** ljxiash_ has quit IRC | 01:37 | |
| *** swamireddy has joined #openstack-telemetry | 01:45 | |
| *** diogogmt has quit IRC | 01:46 | |
| *** ljxiash has joined #openstack-telemetry | 01:56 | |
| *** ljxiash has quit IRC | 01:57 | |
| *** ljxiash has joined #openstack-telemetry | 01:57 | |
| *** vint_bra has quit IRC | 01:59 | |
| openstackgerrit | Su Zhengwei proposed openstack/ceilometer: raise InstanceShutOffException if vm power off when get_vm_moid use map of vm manage obj to reduce vmware api calls https://review.openstack.org/389493 | 02:06 |
|---|---|---|
| *** dave-mcc_ has joined #openstack-telemetry | 02:11 | |
| *** lhx_ has joined #openstack-telemetry | 02:11 | |
| *** dave-mccowan has quit IRC | 02:12 | |
| *** dave-mccowan has joined #openstack-telemetry | 02:13 | |
| *** dave-mcc_ has quit IRC | 02:16 | |
| *** dave-mccowan has quit IRC | 02:18 | |
| *** gongysh has joined #openstack-telemetry | 02:24 | |
| *** fguillot has quit IRC | 02:27 | |
| *** thorst has joined #openstack-telemetry | 02:32 | |
| *** thorst has quit IRC | 02:32 | |
| *** thorst has joined #openstack-telemetry | 02:33 | |
| *** dave-mccowan has joined #openstack-telemetry | 02:39 | |
| *** thorst has quit IRC | 02:40 | |
| *** thorst has joined #openstack-telemetry | 02:41 | |
| *** donghao has joined #openstack-telemetry | 02:42 | |
| *** thorst has quit IRC | 02:45 | |
| *** dave-mccowan has quit IRC | 02:48 | |
| *** diogogmt has joined #openstack-telemetry | 02:56 | |
| *** adriant has quit IRC | 03:03 | |
| *** zhurong has quit IRC | 03:04 | |
| *** zhurong has joined #openstack-telemetry | 03:07 | |
| *** thorst has joined #openstack-telemetry | 03:09 | |
| *** thorst has quit IRC | 03:10 | |
| *** jwcroppe has joined #openstack-telemetry | 03:23 | |
| *** jwcroppe has quit IRC | 03:24 | |
| *** jwcroppe has joined #openstack-telemetry | 03:24 | |
| *** jwcroppe has quit IRC | 03:24 | |
| *** jwcroppe has joined #openstack-telemetry | 03:25 | |
| *** catintheroof has joined #openstack-telemetry | 03:27 | |
| *** liusheng has quit IRC | 03:33 | |
| *** lhx_ has quit IRC | 03:35 | |
| *** zhurong has quit IRC | 03:35 | |
| *** zhurong has joined #openstack-telemetry | 03:36 | |
| *** thorst has joined #openstack-telemetry | 03:36 | |
| *** thorst has quit IRC | 03:37 | |
| *** adriant has joined #openstack-telemetry | 03:37 | |
| *** catintheroof has quit IRC | 03:40 | |
| *** links has joined #openstack-telemetry | 03:42 | |
| *** ljxiash has quit IRC | 03:50 | |
| *** zhangguoqing has quit IRC | 04:02 | |
| *** flwang1 has quit IRC | 04:08 | |
| *** lhx_ has joined #openstack-telemetry | 04:11 | |
| *** ljxiash has joined #openstack-telemetry | 04:11 | |
| *** ljxiash has quit IRC | 04:15 | |
| *** zhangguoqing has joined #openstack-telemetry | 04:31 | |
| *** ljxiash has joined #openstack-telemetry | 05:03 | |
| *** lhx_ has quit IRC | 05:38 | |
| *** gongysh has quit IRC | 05:49 | |
| *** yprokule has joined #openstack-telemetry | 06:01 | |
| *** adriant has quit IRC | 06:02 | |
| *** lhx_ has joined #openstack-telemetry | 06:04 | |
| *** liusheng has joined #openstack-telemetry | 06:07 | |
| *** xiaofandh12 has joined #openstack-telemetry | 06:37 | |
| *** nadya has joined #openstack-telemetry | 06:37 | |
| *** nadya has quit IRC | 06:38 | |
| *** donghao has quit IRC | 06:38 | |
| openstackgerrit | gecong proposed openstack/python-ceilometerclient: Add __ne__ built-in function https://review.openstack.org/401088 | 06:53 |
| *** qwertyco has joined #openstack-telemetry | 07:08 | |
| *** nadya has joined #openstack-telemetry | 07:12 | |
| *** nadya has quit IRC | 07:12 | |
| *** tesseract has joined #openstack-telemetry | 07:12 | |
| *** tesseract is now known as Guest59528 | 07:13 | |
| *** nadya has joined #openstack-telemetry | 07:27 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: indexer: list_metric(), skip sql if names is empty https://review.openstack.org/400762 | 07:57 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: file: new paths configuration https://review.openstack.org/401114 | 07:58 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: file: new paths configuration https://review.openstack.org/401114 | 07:59 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: rest: wait for the thread pool executor result https://review.openstack.org/401116 | 08:09 |
| *** zhangguoqing has quit IRC | 08:19 | |
| *** yprokule_ has joined #openstack-telemetry | 08:20 | |
| *** yprokule has quit IRC | 08:22 | |
| *** yprokule_ is now known as yprokule | 08:22 | |
| *** zhangguoqing has joined #openstack-telemetry | 08:23 | |
| *** shardy has joined #openstack-telemetry | 08:30 | |
| *** amoralej|off is now known as amoralej | 08:43 | |
| *** yprokule_ has joined #openstack-telemetry | 08:52 | |
| *** yprokule has quit IRC | 08:53 | |
| *** yprokule_ is now known as yprokule | 08:53 | |
| *** gongysh has joined #openstack-telemetry | 09:01 | |
| *** xiaofandh12 has quit IRC | 09:04 | |
| openstackgerrit | Merged openstack/aodh: [instll] Update a more simple rabbitmq configuration https://review.openstack.org/392018 | 09:15 |
| sileht | jd__, I have something that almost works for ttnn | 09:22 |
| sileht | jd__, I have fixed all my issues | 09:22 |
| sileht | jd__, but metricd is still a bit slow to consume all my measurements | 09:23 |
| sileht | jd__, I'm not IO bound, measurements are read from tmpfs and written to a SDD | 09:29 |
| sileht | jd__, metricd looks do nothing most of the times... | 09:30 |
| jd__ | sileht: 0% cpu? | 09:40 |
| sileht | jd__, 0% most of the times then 100% during some seconds and 0% again | 09:41 |
| sileht | jd__, I'm adding some trace to see where it wait | 09:41 |
| lhx_ | sileht, does it related to the issues what you said? | 09:42 |
| lhx_ | http://paste.openstack.org/show/590176/ | 09:42 |
| sileht | lhx_, perhaps I don't known | 09:45 |
| *** yassine has joined #openstack-telemetry | 09:46 | |
| jd__ | sileht: and your backlog is full? | 09:46 |
| sileht | jd__, oh yeah :) | 09:46 |
| *** yassine is now known as Guest58074 | 09:46 | |
| sileht | bbm | 09:47 |
| sileht | bbl | 09:47 |
| lhx_ | sileht, the metricd log prints the error repeatedly. | 09:51 |
| lhx_ | what issue is it? | 09:52 |
| sileht | lhx_: i would say a tooz bug | 09:57 |
| sileht | lhx_: it checks life of a deleted/released Iock I think | 09:57 |
| lhx_ | sileht, is this serious? | 09:59 |
| sileht | lhx_: I don't think so, but I haven't into that one yet | 10:00 |
| sileht | *dig* | 10:00 |
| *** zhurong has quit IRC | 10:01 | |
| *** donghao has joined #openstack-telemetry | 10:04 | |
| lhx_ | sileht, hum..., I will open a bug to tooz and set a trace | 10:06 |
| *** donghao has quit IRC | 10:09 | |
| *** gongysh has quit IRC | 10:15 | |
| *** vint_bra has joined #openstack-telemetry | 10:23 | |
| *** vint_bra has quit IRC | 10:39 | |
| *** qwertyco has quit IRC | 10:40 | |
| *** cdent has joined #openstack-telemetry | 10:48 | |
| *** nadya has quit IRC | 10:53 | |
| *** lhx__ has joined #openstack-telemetry | 10:55 | |
| *** lhx_ has quit IRC | 10:58 | |
| *** nadya has joined #openstack-telemetry | 11:04 | |
| *** donghao has joined #openstack-telemetry | 11:21 | |
| *** Guest58074 has quit IRC | 11:41 | |
| *** openstackgerrit has quit IRC | 12:03 | |
| *** openstackgerrit has joined #openstack-telemetry | 12:03 | |
| *** yassine has joined #openstack-telemetry | 12:08 | |
| *** catintheroof has joined #openstack-telemetry | 12:09 | |
| *** yassine is now known as Guest22022 | 12:09 | |
| *** catintheroof has quit IRC | 12:15 | |
| *** gongysh has joined #openstack-telemetry | 12:15 | |
| *** gongysh has quit IRC | 12:17 | |
| *** vint_bra has joined #openstack-telemetry | 12:17 | |
| *** magg has joined #openstack-telemetry | 12:17 | |
| magg | hello | 12:17 |
| magg | i already have a devstack multi node installation in my lab... i'm wondering if I can integrate gnocchi in another node | 12:18 |
| magg | I have ceilometer running on the controller and compute nodes, with a mongo backend | 12:18 |
| magg | i just heard about gnocchi and wanted to try it out, but i don't want to mess too much with my current installation | 12:19 |
| *** catintheroof has joined #openstack-telemetry | 12:20 | |
| *** nadya has quit IRC | 12:37 | |
| *** shardy is now known as shardy_lunch | 12:39 | |
| *** catintheroof has quit IRC | 12:44 | |
| *** nadya has joined #openstack-telemetry | 12:45 | |
| *** nadya has quit IRC | 12:46 | |
| *** links has quit IRC | 12:54 | |
| *** zhangguoqing has quit IRC | 13:00 | |
| *** zhangguoqing has joined #openstack-telemetry | 13:01 | |
| *** ljxiash has quit IRC | 13:02 | |
| *** lhx__ has quit IRC | 13:12 | |
| *** Guest59528 has quit IRC | 13:19 | |
| *** amoralej is now known as amoralej|lunch | 13:20 | |
| *** ljxiash has joined #openstack-telemetry | 13:28 | |
| *** shardy_lunch is now known as shardy | 13:28 | |
| *** ljxiash has quit IRC | 13:31 | |
| *** ljxiash has joined #openstack-telemetry | 13:32 | |
| *** lhx__ has joined #openstack-telemetry | 13:39 | |
| *** nadya has joined #openstack-telemetry | 13:46 | |
| *** nadya has quit IRC | 13:51 | |
| *** gordc has joined #openstack-telemetry | 13:57 | |
| *** ramishra has quit IRC | 13:59 | |
| *** donghao has quit IRC | 13:59 | |
| *** ramishra has joined #openstack-telemetry | 14:01 | |
| *** fguillot has joined #openstack-telemetry | 14:04 | |
| *** thorst has joined #openstack-telemetry | 14:08 | |
| *** pcaruana has quit IRC | 14:14 | |
| *** thorst has quit IRC | 14:19 | |
| *** lhx__ has quit IRC | 14:19 | |
| *** nicodemus_ has joined #openstack-telemetry | 14:22 | |
| nicodemus_ | hello | 14:22 |
| *** amoralej|lunch is now known as amoralej | 14:22 | |
| *** thorst has joined #openstack-telemetry | 14:22 | |
| openstackgerrit | Merged openstack/gnocchi: indexer: list_metric(), skip sql if names is empty https://review.openstack.org/400762 | 14:24 |
| *** nadya has joined #openstack-telemetry | 14:24 | |
| *** lhx__ has joined #openstack-telemetry | 14:24 | |
| nicodemus_ | quick question: does gnocchi have some sort of "garbage collector"? I mean: if a user terminates an instance and the archive policy says the metrics are to be stored for a month, when the month is over does the resource disappear? Or does it stay until one deletes it? | 14:25 |
| gordc | nicodemus_: there is a 'garbage collector' worker but it doesn't behave as you mentioned | 14:27 |
| gordc | since gnocchi isn't relevant to 'now', it only cleans up from the last known datapoint it has | 14:27 |
| gordc | so if you store a month, it will keep a month, even if that month is a year ago (assuming you don't have any new datapoints in a year) | 14:28 |
| *** yprokule_ has joined #openstack-telemetry | 14:29 | |
| nicodemus_ | gordc, understood. Therefore, resources are never deleted | 14:29 |
| gordc | resources themselves will never disappear unless you delete them... | 14:29 |
| nicodemus_ | right? | 14:29 |
| nicodemus_ | I see. Thanks!! :) | 14:29 |
| gordc | right. only measurements are expired and removed by policy | 14:29 |
| gordc | s/relevant/relative/ you get the point :) | 14:30 |
| *** yprokule has quit IRC | 14:30 | |
| *** yprokule_ is now known as yprokule | 14:30 | |
| *** thorst has quit IRC | 14:31 | |
| *** tesseract has joined #openstack-telemetry | 14:32 | |
| *** tesseract is now known as Guest48653 | 14:33 | |
| sileht | jd__, the title of my next commit is 'Don't sleep if we have stuff to do' | 14:35 |
| * sileht is teasering its works :p | 14:35 | |
| jd__ | haha | 14:36 |
| jd__ | aka "no rest for the wicked" | 14:36 |
| * gordc likes 'does this work?' title | 14:37 | |
| sileht | gordc, I have found how you cheat your chart ! | 14:38 |
| gordc | sileht: lol ssshhhh | 14:39 |
| gordc | what did i do? | 14:39 |
| *** Guest48653 has quit IRC | 14:39 | |
| *** tesseract- has joined #openstack-telemetry | 14:39 | |
| gordc | i just auto return 201... that's not cheating | 14:40 |
| sileht | gordc, 202 to be exact | 14:40 |
| sileht | gordc, it's the correct HTTP code for 'I take it, but who known if I can handle it' | 14:41 |
| gordc | :) that's someone elses problem. | 14:41 |
| gordc | what's the bug i made? | 14:41 |
| sileht | gordc, I don't known who introduce the bug and I don't care, but effect on bench can be funny | 14:42 |
| sileht | but also depends on the webserver | 14:42 |
| gordc | oh. the threading patch you have? | 14:43 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Don't sleep if we have stuff to do https://review.openstack.org/401291 | 14:44 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: storage: split process_new_measures() https://review.openstack.org/401292 | 14:44 |
| gordc | i thought the threadin waited... maybe i never checked. *shrugs* | 14:44 |
| sileht | gordc, this one: https://review.openstack.org/#/c/401116/ | 14:44 |
| gordc | i didn't make charts on that endpoint though... i should've now that i know it's super quick. ;) | 14:44 |
| sileht | :) | 14:44 |
| gordc | which webserver does it not wait? | 14:45 |
| sileht | gordc, werkzeug | 14:45 |
| *** tesseract- has quit IRC | 14:49 | |
| openstackgerrit | Merged openstack/ceilometer: drop image pollster https://review.openstack.org/400885 | 14:50 |
| sileht | jd__, I think metricd is working now: load average: 74.17, 65.95, 43.46 | 14:52 |
| jd__ | haha | 14:52 |
| sileht | jd__, but it still can't replace my pnp4nagios, metricd is still much slower than npcd | 14:53 |
| jd__ | sileht: maybe you can remove some agg methods too? | 14:54 |
| sileht | jd__, yeah pnp4nagios have only min/max/avg | 14:55 |
| jd__ | if you have more intel on what might be slow that help | 14:55 |
| jd__ | sileht: ah that's good first lead! | 14:55 |
| *** zhangguoqing has quit IRC | 14:55 | |
| jd__ | that should reduce your load by ~70% | 14:55 |
| sileht | :D | 14:55 |
| sileht | I have already simplify the archive policy with one def | 14:56 |
| *** paw has quit IRC | 14:57 | |
| jd__ | how big is your metricd server in # cpu? | 14:57 |
| *** rbak has joined #openstack-telemetry | 15:00 | |
| sileht | jd__, 4 vcpus | 15:02 |
| jd__ | not that bad I guess | 15:03 |
| sileht | jd__, nagios run on it too and it's cpu intensive | 15:04 |
| jd__ | sileht: ENLARGE YOUR INSTANCE | 15:04 |
| sileht | jd__, but it works well with npcd (that create/update rdd files) | 15:04 |
| sileht | jd__, it's baremetal | 15:04 |
| jd__ | u said vcpu : | 15:04 |
| jd__ | :p | 15:04 |
| *** thorst has joined #openstack-telemetry | 15:05 | |
| sileht | jd__, yeah because it's 1 cpu with 2 hyper threaded cores :) | 15:05 |
| jd__ | ah :D | 15:05 |
| jd__ | so it's really 2,2 CPU or something like that lol | 15:05 |
| sileht | jd__, it looks to work very well with the new agg methods | 15:06 |
| sileht | jd__, what are those that take all the cpu time ? std , | 15:06 |
| sileht | ? | 15:06 |
| jd__ | sileht: pct I think | 15:06 |
| jd__ | and probably median too since it's 50pct | 15:06 |
| jd__ | which reminds me that I think gordc wrote an optimization for several pct | 15:07 |
| jd__ | we could use that for median | 15:07 |
| gordc | jd__: i did an optimisation? | 15:09 |
| gordc | sileht: when i did profile, the 95 percentile agg method took by far the most cpu | 15:10 |
| jd__ | or someone that looks like you | 15:10 |
| gordc | jd__: then it's just me. :P | 15:10 |
| jd__ | I cannot find that in the log maybe it's not done | 15:12 |
| jd__ | I don't know how it works anymore :P | 15:12 |
| *** thorst has quit IRC | 15:12 | |
| sileht | jd__, now measurements are lagging at gnocchi-nagios side, but that just the times it creates the 6000 resources :) | 15:15 |
| jd__ | \o/ | 15:16 |
| jd__ | maybe we should remove 95pct and median from the default archive policies | 15:16 |
| jd__ | WDYT gordc ? | 15:16 |
| *** thorst has joined #openstack-telemetry | 15:16 | |
| gordc | sure. i'm ok with that. | 15:17 |
| gordc | jd__: i should mention the 95% agg isn't taking the majority of time. it's just out of all the aggregations, the 95% takes the most | 15:18 |
| jd__ | yup got that | 15:18 |
| jd__ | but removing 2 on 8 should improve default perf by 25% so… | 15:18 |
| gordc | default perf (of aggregation portion of code)* by 25% | 15:20 |
| *** nicodemus_ has quit IRC | 15:20 | |
| *** thorst has quit IRC | 15:20 | |
| gordc | let me check my profile, i think a lot of it was just grouping unagg data for aggregations. | 15:20 |
| gordc | is the issue that the default policy is too verbose? | 15:21 |
| *** diogogmt has quit IRC | 15:22 | |
| gordc | but yeah, i think drop median, 95% are good. maybe stdev if you want to be really aggressive with cutting | 15:23 |
| *** chlong has joined #openstack-telemetry | 15:25 | |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: Remove 95pct and median from default archive policies https://review.openstack.org/401315 | 15:27 |
| *** nicodemus_ has joined #openstack-telemetry | 15:33 | |
| sileht | jd__, gnocchi-metricd is slow again :( | 15:33 |
| *** chlong has quit IRC | 15:34 | |
| *** yprokule_ has joined #openstack-telemetry | 15:35 | |
| *** yprokule has quit IRC | 15:36 | |
| *** yprokule_ is now known as yprokule | 15:36 | |
| *** paw has joined #openstack-telemetry | 15:36 | |
| sileht | jd__, my don't sleep patch is buggy | 15:38 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Don't sleep if we have stuff to do https://review.openstack.org/401291 | 15:40 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: rest: wait for the thread pool executor result https://review.openstack.org/401116 | 15:40 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: storage: split process_new_measures() https://review.openstack.org/401292 | 15:40 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: file: new paths configuration https://review.openstack.org/401114 | 15:40 |
| sileht | jd__, funny bug when metricd says 'you should add workers', it sleeps 10 seconds :) | 15:41 |
| *** chlong has joined #openstack-telemetry | 15:49 | |
| magg | hello, i already have a devstack multi node installation in my lab... i'm wondering if I can integrate gnocchi in another node. I have ceilometer running on the controller and compute nodes, with a mongo backend. i just heard about gnocchi and wanted to try it out, but i don't want to mess too much with my current installation | 15:50 |
| *** narasimha_SV has joined #openstack-telemetry | 15:52 | |
| narasimha_SV | After integration ceilometer with gnocchi as its backend | 15:52 |
| narasimha_SV | how much it will take to get the metrics collected to gnocchi database ? | 15:52 |
| *** nadya has quit IRC | 15:53 | |
| *** nadya has joined #openstack-telemetry | 15:54 | |
| gordc | magg: set meter_dispatchers = gnocchi in addition to existing meter_dispatchers = database you have in ceilometer.conf | 15:54 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: Don't sleep if we have stuff to do https://review.openstack.org/401291 | 15:54 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: storage: split process_new_measures() https://review.openstack.org/401292 | 15:54 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/gnocchi: file: new paths configuration https://review.openstack.org/401114 | 15:54 |
| gordc | magg: http://docs.openstack.org/developer/ceilometer/install/dbreco.html#moving-from-ceilometer-to-gnocchi | 15:54 |
| gordc | narasimha_SV: not sure what your question is, can you ask again? | 15:55 |
| narasimha_SV | gordc: I have integrated ceilometer to gnocchi | 15:56 |
| *** nadya has quit IRC | 15:56 | |
| *** nadya_ has joined #openstack-telemetry | 15:56 | |
| narasimha_SV | I can see the logs that ceilometer is properly integrated to gnocchi and meter_dispatchers and event_dispatchers are set to gnocchi | 15:56 |
| narasimha_SV | but I dont see metrics collected in gnocchi database | 15:57 |
| narasimha_SV | gnocchi metric list or gnocchi resource list gives me empty list | 15:57 |
| narasimha_SV | i created a VM, downloaded glance image | 15:57 |
| *** nadya_ has quit IRC | 15:57 | |
| narasimha_SV | but no metric detailsin gnocchi | 15:57 |
| *** nadya has joined #openstack-telemetry | 15:58 | |
| gordc | narasimha_SV: do you have errors in ceilometer-collector? do you see requests in gnocchi-api logs? | 15:58 |
| gordc | that should guarantee you get something from those cmomands (assuming your policy allows it) | 15:59 |
| narasimha_SV | thanks gordc. there are few errors in ceilometer collector | 16:00 |
| narasimha_SV | thanks for pointing me to right direction | 16:00 |
| sileht | jd__, guess when I have started metricd: https://dl.sileht.net/gnocchi.png | 16:02 |
| *** nadya has quit IRC | 16:02 | |
| sileht | jd__, I'm at 27000 metrics and the number continue to grow | 16:04 |
| *** cdent has quit IRC | 16:05 | |
| gordc | sileht: i don't think this patch does what you think it will do. https://review.openstack.org/#/c/401291 | 16:06 |
| gordc | it's just going to keep grabbing and shoving the same tasks onto queue over and over again | 16:06 |
| sileht | gordc, perhaps this is not the correct way to fix my issue | 16:07 |
| sileht | gordc, so the fact : | 16:07 |
| narasimha_SV | gordc: http://paste.openstack.org/show/590227/ | 16:07 |
| narasimha_SV | getting this error | 16:07 |
| sileht | gordc, my issue: metricd processor got N metrics, processes them into 1 seconds and wait 60 seconds to got the N next | 16:08 |
| narasimha_SV | http://paste.openstack.org/show/590228/ this is my ceilometer.conf file | 16:08 |
| gordc | sileht: i set the scheduler interval really low. to like 4s but it's still not ideal beacuse it always grabs keys from front of giant list. | 16:09 |
| sileht | gordc, I can put 'metric_processing_delay = 0' but that just crappy work around | 16:10 |
| sileht | gordc, I think list_metric_with_measures_to_process() should not return 'in process' metrics | 16:10 |
| sileht | gordc, that was my old npcd does :p | 16:10 |
| sileht | gordc, it rename file with a suffix '-taken-by-PID-XXXXXX' and this files are ignored for the next run | 16:11 |
| sileht | the drawback is when npcd crash I lost measures | 16:12 |
| sileht | (and I have to cleanup this files) | 16:12 |
| gordc | sileht: lol yeah. and you're just making processnig a little slower | 16:12 |
| gordc | sileht: that's why i asked on ML about creating buckets for new measrues | 16:13 |
| gordc | then maybe we can just have it loop through each bucket and dump it's contents in | 16:13 |
| gordc | minimise chance of scheduling same task | 16:14 |
| gordc | i don't know if that's a good solution. we just have an issue with how we retrieve new measures currently. | 16:19 |
| *** rbak has quit IRC | 16:19 | |
| narasimha_SV | is there any issue in the ceilometer.conf of mine http://paste.openstack.org/show/590228/ | 16:20 |
| gordc | narasimha_SV: what version of ceilometer are you using? | 16:20 |
| narasimha_SV | i am building code from the master branch code | 16:21 |
| narasimha_SV | ceilometer code around 2 weeks old | 16:22 |
| gordc | you should enable debug to get more info | 16:23 |
| narasimha_SV | ok | 16:23 |
| gordc | i also have cache but it should not be related: http://paste.openstack.org/show/590232/ | 16:23 |
| gordc | you need to make sure you have dispatcher finds your gnocchi_resources.yaml file correctly | 16:23 |
| *** fguillot has quit IRC | 16:24 | |
| *** fguillot has joined #openstack-telemetry | 16:26 | |
| sileht | gordc, the question is how many measures per bucket :) | 16:28 |
| gordc | sileht: ¯\_(ツ)_/¯ | 16:29 |
| gordc | sileht: i thought 100 at first but i was targeting maybe 100k new measures | 16:30 |
| gordc | ~1K new measures/bucket | 16:31 |
| gordc | but if it's a small test environment, it may end up looping through a lot of empty buckets. | 16:31 |
| *** ramishra has quit IRC | 16:34 | |
| sileht | gordc, I think we can coordonate scheduler to adjust the number of needed buckets | 16:35 |
| sileht | gordc, and store in indexer the number of bucket the backend currently have | 16:36 |
| sileht | gordc, scheduler read the value from BD, creates a hashring of who is taking care of part of the buckets | 16:37 |
| gordc | yeah. but i imagine we don't want number of buckets to change a lot or else we'll need to rehash/move things each time. | 16:37 |
| sileht | gordc, let put 100 and double it if we reach it :) | 16:37 |
| *** thorst has joined #openstack-telemetry | 16:38 | |
| gordc | :) 'agile' | 16:38 |
| sileht | a bit like application are creating connections pool | 16:38 |
| sileht | pool_size=100, with a overhead of 200 to absorb spike | 16:38 |
| sileht | and if this is not a spike, oduble pool_size | 16:39 |
| *** thorst has quit IRC | 16:39 | |
| gordc | when do we spike? if backlog is above 100K? | 16:39 |
| gordc | sileht: one thing i didn't look at is how much overhead it is to create 100 objects/containers/folders. | 16:39 |
| sileht | gordc, for ceph it's better that the current 1 object I think :) | 16:40 |
| gordc | although in big picture, maybe not that much | 16:40 |
| sileht | that/than | 16:40 |
| gordc | yeah, definitely 1 obj/container/folder solution is not working right now. | 16:40 |
| openstackgerrit | Julien Danjou proposed openstack/gnocchi: Remove 95pct and median from default archive policies https://review.openstack.org/401315 | 16:41 |
| *** nadya has joined #openstack-telemetry | 16:43 | |
| *** yprokule has quit IRC | 16:48 | |
| jd__ | sileht: if you're interested, I sent a bunch of patches on tooz to review last week | 17:01 |
| *** magg has quit IRC | 17:01 | |
| jd__ | sileht: I don't know if you monitor it, but nobody is except harlowja :( | 17:01 |
| jd__ | gordc: sileht: as I said in sileht's review, I think the first step that we want is to anyway split that measure storage in its own driver system | 17:02 |
| jd__ | that should help also to experiment with new drivers and techniques :) | 17:02 |
| gordc | jd__: i already tried it. i wrote to ramdisk. very good perf :P | 17:04 |
| jd__ | I guess so | 17:04 |
| jd__ | there are people wanting to do that :) | 17:04 |
| gordc | :) i wouln't recommend it. it was super hard to manage across multiple nodes. | 17:06 |
| *** thorst has joined #openstack-telemetry | 17:06 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer: use domainListGetStats to get cputime https://review.openstack.org/401354 | 17:07 |
| *** thorst_ has joined #openstack-telemetry | 17:07 | |
| *** lhx__ has quit IRC | 17:11 | |
| *** thorst has quit IRC | 17:11 | |
| *** lhx__ has joined #openstack-telemetry | 17:11 | |
| *** thorst_ has quit IRC | 17:11 | |
| gordc | i can look at that split probably next week if no one does anything. | 17:12 |
| *** lhx__ has quit IRC | 17:21 | |
| *** magg has joined #openstack-telemetry | 17:22 | |
| *** diogogmt has joined #openstack-telemetry | 17:29 | |
| harlowja | jd__ i monitor it sorta :-P | 17:37 |
| *** shardy has quit IRC | 17:43 | |
| openstackgerrit | Merged openstack/gnocchi: rest: wait for the thread pool executor result https://review.openstack.org/401116 | 17:49 |
| *** chlong has quit IRC | 17:50 | |
| *** dave-mccowan has joined #openstack-telemetry | 17:56 | |
| *** dave-mccowan has quit IRC | 17:57 | |
| *** amoralej is now known as amoralej|off | 18:02 | |
| *** nadya has quit IRC | 18:04 | |
| *** nadya has joined #openstack-telemetry | 18:04 | |
| *** nadya has quit IRC | 18:08 | |
| openstackgerrit | Merged openstack/gnocchi: Remove 95pct and median from default archive policies https://review.openstack.org/401315 | 18:12 |
| narasimha_SV | gordc: I am not using any cache mechanism now | 18:33 |
| *** flwang1 has joined #openstack-telemetry | 18:41 | |
| narasimha_SV | http://paste.openstack.org/show/590246/ | 18:41 |
| narasimha_SV | getting this error after integrating gnocchi with ceilometer | 18:41 |
| narasimha_SV | i have those resources present which it is saying as not exist | 18:43 |
| narasimha_SV | these are ceilometer collector logs | 18:43 |
| *** dave-mccowan has joined #openstack-telemetry | 18:49 | |
| *** dave-mcc_ has joined #openstack-telemetry | 18:59 | |
| *** dave-mccowan has quit IRC | 19:02 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: libvirt: rewrite the error handling https://review.openstack.org/398297 | 19:26 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: Don't poll nova with compute agent https://review.openstack.org/397654 | 19:26 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: tests: switch devstack/gate to libvirt_metadata https://review.openstack.org/398268 | 19:26 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: Remove useless code https://review.openstack.org/398870 | 19:26 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: Remove duplicated code https://review.openstack.org/398871 | 19:26 |
| *** catintheroof has joined #openstack-telemetry | 19:27 | |
| openstackgerrit | Merged openstack/python-gnocchiclient: fix 409 exception regex https://review.openstack.org/400683 | 19:29 |
| *** flwang1 has quit IRC | 19:34 | |
| *** catintheroof has quit IRC | 19:34 | |
| *** dave-mcc_ has quit IRC | 19:49 | |
| *** nicodemus__ has joined #openstack-telemetry | 19:51 | |
| *** nicodemus_ has quit IRC | 19:53 | |
| *** nadya has joined #openstack-telemetry | 19:56 | |
| *** paw has quit IRC | 19:56 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: libvirt: rewrite the error handling https://review.openstack.org/398297 | 19:58 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: Remove useless singleton check https://review.openstack.org/386022 | 20:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: intel_node_manager: stop using global conf https://review.openstack.org/386023 | 20:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: register keystoneauth options at runtime https://review.openstack.org/386020 | 20:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: tools: stop using global conf https://review.openstack.org/387358 | 20:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: register oslo_db options at runtime https://review.openstack.org/386021 | 20:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: test_storage_scenarios: stop using global conf https://review.openstack.org/385933 | 20:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: cmd.storage: stop using global conf https://review.openstack.org/385934 | 20:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: utils: stop using global conf https://review.openstack.org/386024 | 20:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: storage: stop using global conf https://review.openstack.org/385935 | 20:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: register ceilometer options at runtime https://review.openstack.org/386025 | 20:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: gnocchi: Allow to set a different creds section https://review.openstack.org/387306 | 20:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: Print ceilometer configuration on startup https://review.openstack.org/387193 | 20:03 |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer: Allow to configure the triplo network name https://review.openstack.org/394452 | 20:03 |
| *** narasimha_SV has quit IRC | 20:11 | |
| *** nicodemus__ has quit IRC | 20:21 | |
| *** nicodemus__ has joined #openstack-telemetry | 20:35 | |
| *** swamireddy has quit IRC | 20:37 | |
| *** flwang1 has joined #openstack-telemetry | 20:39 | |
| *** nadya has quit IRC | 20:50 | |
| *** catintheroof has joined #openstack-telemetry | 21:08 | |
| -openstackstatus- NOTICE: Due to a problem with the cinder volume backing the log server, jobs are failing with POST_FAILURE. Please avoid issuing 'recheck' commands until the issue is resolved. | 21:10 | |
| *** ChanServ changes topic to "Due to a problem with the cinder volume backing the log server, jobs are failing with POST_FAILURE. Please avoid issuing 'recheck' commands until the issue is resolved." | 21:10 | |
| *** fguillot has quit IRC | 21:24 | |
| openstackgerrit | Merged openstack/ceilometer: Remove useless code https://review.openstack.org/398870 | 21:24 |
| openstackgerrit | Merged openstack/ceilometer: Remove duplicated code https://review.openstack.org/398871 | 21:25 |
| openstackgerrit | gordon chung proposed openstack/ceilometer: drop instance meter https://review.openstack.org/400973 | 21:33 |
| openstackgerrit | gordon chung proposed openstack/ceilometer: use domainListGetStats to get cputime https://review.openstack.org/401354 | 21:34 |
| *** catintheroof has quit IRC | 21:40 | |
| *** pradk has quit IRC | 21:43 | |
| *** thorst_ has joined #openstack-telemetry | 21:47 | |
| *** nicodemus__ has quit IRC | 22:00 | |
| *** adriant has joined #openstack-telemetry | 22:17 | |
| *** chlong has joined #openstack-telemetry | 22:19 | |
| *** thorst_ has quit IRC | 22:20 | |
| *** chlong has quit IRC | 22:30 | |
| *** swamireddy has joined #openstack-telemetry | 22:45 | |
| *** catintheroof has joined #openstack-telemetry | 22:48 | |
| *** catintheroof has quit IRC | 22:52 | |
| *** ChanServ changes topic to "#openstack-telemetry is OpenStack Telemetry | http://wiki.openstack.org/Telemetry" | 22:55 | |
| -openstackstatus- NOTICE: The affected filesystems on the log server are repaired. Please leave 'recheck' comments on any changes which failed with POST_FAILURE. | 22:55 | |
| *** catintheroof has joined #openstack-telemetry | 23:00 | |
| *** catinthe_ has joined #openstack-telemetry | 23:23 | |
| *** catintheroof has quit IRC | 23:26 | |
| *** vint_bra has quit IRC | 23:30 | |
| *** catinthe_ has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
