| *** http_GK1wmSU has joined #openstack-telemetry | 00:17 | |
| *** http_GK1wmSU has left #openstack-telemetry | 00:17 | |
| *** rwsu has quit IRC | 00:38 | |
| *** zhurong has joined #openstack-telemetry | 00:46 | |
| *** ddyer has quit IRC | 00:53 | |
| *** ddyer has joined #openstack-telemetry | 00:57 | |
| *** dixiaoli has joined #openstack-telemetry | 00:58 | |
| *** gongysh has joined #openstack-telemetry | 01:30 | |
| *** thorst_afk has quit IRC | 01:37 | |
| *** lhx__ has joined #openstack-telemetry | 01:42 | |
| *** joadavis has quit IRC | 01:43 | |
| *** abhi89 has joined #openstack-telemetry | 01:50 | |
| openstackgerrit | liuwei proposed openstack/ceilometer master: Add meters for libvirt: disk.device.read.latency, disk.device.write.latency https://review.openstack.org/489910 | 01:51 |
|---|---|---|
| openstackgerrit | xianbin xie proposed openstack/ceilometer master: fix cpu_util precision is too precise https://review.openstack.org/490278 | 01:58 |
| *** abhi89 has quit IRC | 01:59 | |
| *** joadavis has joined #openstack-telemetry | 02:48 | |
| *** dikonoor has joined #openstack-telemetry | 02:54 | |
| *** abhi89 has joined #openstack-telemetry | 03:10 | |
| *** masuberu has joined #openstack-telemetry | 03:18 | |
| *** masber has quit IRC | 03:22 | |
| *** dikonoo has joined #openstack-telemetry | 03:24 | |
| *** dikonoor has quit IRC | 03:28 | |
| *** dixiaoli has quit IRC | 03:28 | |
| *** thorst_afk has joined #openstack-telemetry | 03:38 | |
| *** thorst_afk has quit IRC | 03:52 | |
| *** links has joined #openstack-telemetry | 03:54 | |
| *** gongysh has quit IRC | 04:15 | |
| *** dikonoo has quit IRC | 04:57 | |
| *** dikonoo has joined #openstack-telemetry | 05:12 | |
| openstackgerrit | xianbin xie proposed openstack/ceilometer master: fix cpu_util precision is too precise https://review.openstack.org/490278 | 05:42 |
| *** lhx__ has quit IRC | 05:46 | |
| *** thorst_afk has joined #openstack-telemetry | 05:49 | |
| *** psachin has joined #openstack-telemetry | 05:51 | |
| *** dims has quit IRC | 05:52 | |
| *** lhx__ has joined #openstack-telemetry | 05:53 | |
| *** thorst_afk has quit IRC | 05:53 | |
| *** dims has joined #openstack-telemetry | 05:54 | |
| *** larainema has quit IRC | 05:56 | |
| *** tovin07 has quit IRC | 06:02 | |
| *** dikonoo has quit IRC | 06:02 | |
| *** hoonetorg has quit IRC | 06:12 | |
| *** dikonoor has joined #openstack-telemetry | 06:14 | |
| *** rcernin has joined #openstack-telemetry | 06:19 | |
| *** hoonetorg has joined #openstack-telemetry | 06:25 | |
| *** pcaruana has joined #openstack-telemetry | 06:48 | |
| *** tesseract has joined #openstack-telemetry | 07:17 | |
| *** thorst_afk has joined #openstack-telemetry | 07:22 | |
| *** yprokule has joined #openstack-telemetry | 07:25 | |
| *** thorst_afk has quit IRC | 07:27 | |
| openstackgerrit | Debo Zhang proposed openstack/ceilometer master: Update API link in README https://review.openstack.org/490371 | 07:39 |
| *** tovin07_ has joined #openstack-telemetry | 07:45 | |
| *** larainema has joined #openstack-telemetry | 07:54 | |
| *** efoley has joined #openstack-telemetry | 08:02 | |
| *** sileht has joined #openstack-telemetry | 08:12 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** flg has joined #openstack-telemetry | 08:43 | |
| *** flg_ has joined #openstack-telemetry | 08:44 | |
| *** dims has quit IRC | 09:04 | |
| *** dims has joined #openstack-telemetry | 09:08 | |
| *** thorst_afk has joined #openstack-telemetry | 09:23 | |
| *** dikonoor has quit IRC | 09:25 | |
| *** thorst_afk has quit IRC | 09:28 | |
| *** dikonoor has joined #openstack-telemetry | 09:52 | |
| *** tovin07_ has quit IRC | 10:09 | |
| *** dikonoor has quit IRC | 10:25 | |
| *** zhurong has quit IRC | 10:38 | |
| *** abhi89 has quit IRC | 10:41 | |
| *** dikonoor has joined #openstack-telemetry | 10:55 | |
| *** dixiaoli has joined #openstack-telemetry | 10:59 | |
| *** dixiaoli has quit IRC | 11:04 | |
| *** openstackgerrit has joined #openstack-telemetry | 11:04 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/aodh master: Imported Translations from Zanata https://review.openstack.org/484720 | 11:04 |
| *** dikonoor has quit IRC | 11:53 | |
| *** thorst_afk has joined #openstack-telemetry | 12:03 | |
| *** catintheroof has joined #openstack-telemetry | 12:36 | |
| *** zhurong has joined #openstack-telemetry | 12:49 | |
| *** gordc has joined #openstack-telemetry | 12:50 | |
| *** abhi89 has joined #openstack-telemetry | 13:06 | |
| *** links has quit IRC | 13:06 | |
| *** lhx__ has quit IRC | 13:08 | |
| *** psachin has quit IRC | 13:13 | |
| *** dikonoor has joined #openstack-telemetry | 13:22 | |
| *** psachin has joined #openstack-telemetry | 13:24 | |
| *** lhx_ has joined #openstack-telemetry | 13:30 | |
| *** dikonoor has quit IRC | 13:34 | |
| *** dikonoor has joined #openstack-telemetry | 13:35 | |
| openstackgerrit | Merged openstack/ceilometer master: Update and replace http with https for doc links in ceilometer https://review.openstack.org/488324 | 13:59 |
| *** dikonoor has quit IRC | 14:03 | |
| *** zhurong has quit IRC | 14:17 | |
| openstackgerrit | Merged openstack/ceilometer master: vmware: ignore -1 values https://review.openstack.org/482659 | 14:19 |
| *** thorst_afk has quit IRC | 14:30 | |
| *** thorst_afk has joined #openstack-telemetry | 14:32 | |
| *** thorst_afk has quit IRC | 14:37 | |
| *** thorst_afk has joined #openstack-telemetry | 14:42 | |
| *** thorst_afk has quit IRC | 15:00 | |
| *** thorst_afk has joined #openstack-telemetry | 15:01 | |
| *** yprokule has quit IRC | 15:20 | |
| akrzos | gordc: yeah to change 3.0 scheduling to hash based looks a like a lot of work | 15:35 |
| akrzos | need to access coordination driver to know how many members | 15:35 |
| akrzos | and not even sure if it would apply to the other drivers then | 15:36 |
| akrzos | do you think it's worth it? | 15:36 |
| *** psachin has quit IRC | 16:01 | |
| *** thorst_afk has quit IRC | 16:08 | |
| *** rcernin has quit IRC | 16:12 | |
| *** dave-mccowan has joined #openstack-telemetry | 16:14 | |
| *** thorst_afk has joined #openstack-telemetry | 16:15 | |
| gordc | akrzos: sorry, was afk. | 16:16 |
| gordc | personally, don't think it's worth it. especially since we have v4 which address this stuff already. | 16:16 |
| *** jmlowe has quit IRC | 16:20 | |
| *** vishwana_ has joined #openstack-telemetry | 16:20 | |
| *** vishwanathj has quit IRC | 16:24 | |
| *** lhx__ has joined #openstack-telemetry | 16:40 | |
| *** donghao has joined #openstack-telemetry | 16:41 | |
| *** jmlowe has joined #openstack-telemetry | 16:41 | |
| *** lhx_ has quit IRC | 16:43 | |
| *** efoley has quit IRC | 16:49 | |
| *** pcaruana has quit IRC | 16:54 | |
| *** rcernin has joined #openstack-telemetry | 17:02 | |
| *** donghao has quit IRC | 17:24 | |
| akrzos | gordc: got it so you think generall with the current sort all, then basiclly throw away thats good enough for that patch? | 17:28 |
| gordc | akrzos: i guess that's the most inline with what exists. | 17:33 |
| gordc | akrzos: i think you'll still have that memory issue though. | 17:33 |
| akrzos | yeah i can share what the memory looks like under stress | 17:34 |
| gordc | i think it's the fact that 3.0 pulls in entire list that's causing memory issue. which is why we stopped it in 3..1 | 17:34 |
| akrzos | that was a previous issue though? | 17:34 |
| gordc | akrzos: let me dig. it was jmlowe who ran into it. | 17:34 |
| gordc | (i think( | 17:35 |
| gordc | ) | 17:35 |
| gordc | i will say, by sorting, it does make it less likely we'll have the backlog grow as big. | 17:35 |
| jmlowe | huh what? | 17:35 |
| gordc | i'm naming names | 17:35 |
| akrzos | let me grab a quick screen scrape to show you | 17:36 |
| jmlowe | entirely possible, I tend to break things | 17:36 |
| gordc | akrzos: this was discussion: http://eavesdrop.openstack.org/irclogs/%23openstack-telemetry/%23openstack-telemetry.2017-01-27.log.html#t2017-01-27T16:57:42 | 17:38 |
| akrzos | https://snapshot.raintank.io/dashboard/snapshot/SgKcct5xym8a4ZeSY9LFYVO8CuPKdWFC | 17:39 |
| gordc | i don't think we have the original issue. it was a timeout issue. | 17:39 |
| akrzos | so let me explain that graph | 17:39 |
| akrzos | i have the sorting patch on 8/2 1800 | 17:40 |
| akrzos | i switched back to the old method | 17:40 |
| akrzos | 8/3 11ish | 17:40 |
| akrzos | basicly when you see the measures grow | 17:40 |
| akrzos | turn it back on | 17:40 |
| akrzos | and boom backlog drops to a steady state | 17:40 |
| gordc | yeah, it's because some of the metrics are unprocessed because we only chunk from front of alphanumerically sorted result | 17:41 |
| akrzos | yes | 17:41 |
| gordc | so i guess you're not hitting the timeout that https://github.com/gnocchixyz/gnocchi/commit/cd8d85e9d5505667802f8a17db5fa361521714a4 addresses | 17:42 |
| akrzos | so sorting prevents that unbounded growth on the untouched metrics due to how the work is distributed | 17:42 |
| akrzos | now i'd love to argue lets just change how the keys are named but you mentioned basiclly how data is pulled you need the metric uuid first | 17:43 |
| akrzos | so changing names is not an option | 17:43 |
| akrzos | i'm gonna push that to a whole lot more instances | 17:44 |
| akrzos | but it's limited in size | 17:44 |
| akrzos | i migth be able to get some time on a bigger cluster to prove this patch out | 17:44 |
| akrzos | but basiclly your issues then become how slow ceilometer collector is on getting the data into the backlog | 17:44 |
| gordc | akrzos: well i know your patch works. because that's what we tried to fix in v4 | 17:44 |
| akrzos | even after we fixed that for batching | 17:44 |
| gordc | my main concern was that if you have very long processing_delay value, you'll run into a scenario where you're sorting millions of records potentially | 17:45 |
| akrzos | yeah i can see that scenario | 17:46 |
| akrzos | technically if you have several million keys in that omap your probably in a world of hurt already | 17:47 |
| akrzos | which is why gnocchi v4 is better | 17:47 |
| akrzos | multiple omaps | 17:47 |
| gordc | yeah. true. | 17:47 |
| gordc | you'll probably need to apply your patch against 3.1 as well. | 17:48 |
| gordc | but yeah, you've convinced me with "several million keys in that omap your probably in a world of hurt already" | 17:49 |
| *** tesseract has quit IRC | 17:49 | |
| akrzos | i can investigate memory a bit more | 17:50 |
| akrzos | maybe there is a different use case with how gnocchi would be setup that i'm not aware of that would allow a much larger backlog? | 17:51 |
| gordc | all my benchmarks have metricd aggregating constantly just to test max throughput | 17:52 |
| akrzos | my other thought is letting a backlog grow a bit, and processing is more samples processed per interval for each metrics which is probably more efficent than just processing the latest sample | 17:53 |
| gordc | but i imagine most deployments will have more relaxed processing_delay | 17:53 |
| akrzos | but at the expense of delay | 17:54 |
| gordc | akrzos: like modifying the TASKS_PER_WORKER param? | 17:54 |
| gordc | biab. going to grab lunch. | 17:56 |
| akrzos | ok no worries | 17:56 |
| akrzos | my opinion is we need a two fold approach | 17:56 |
| akrzos | expose TASKS_PER_WORKER and essentially cap backlog growth by sorting | 17:56 |
| *** lhx__ has quit IRC | 18:17 | |
| *** vint_bra1 has joined #openstack-telemetry | 18:20 | |
| gordc | yeah, both could work. but we also have v4.x :p | 18:20 |
| openstackgerrit | Samuel Pilla proposed openstack/ceilometer master: Add bandit to ceilometer tests https://review.openstack.org/490621 | 18:30 |
| *** abhi89 has quit IRC | 18:32 | |
| *** masber has joined #openstack-telemetry | 18:57 | |
| *** masuberu has quit IRC | 18:59 | |
| *** vint_bra1 has quit IRC | 19:17 | |
| *** vint_bra has joined #openstack-telemetry | 19:18 | |
| *** harlowja has quit IRC | 19:57 | |
| *** flg_ has quit IRC | 20:39 | |
| *** flg has quit IRC | 20:39 | |
| *** thorst_afk has quit IRC | 21:10 | |
| *** thorst_afk has joined #openstack-telemetry | 21:12 | |
| *** joadavis has quit IRC | 21:12 | |
| *** harlowja has joined #openstack-telemetry | 21:15 | |
| *** thorst_afk has quit IRC | 21:17 | |
| *** jmlowe has quit IRC | 21:18 | |
| *** efoley has joined #openstack-telemetry | 21:20 | |
| *** joadavis has joined #openstack-telemetry | 21:26 | |
| *** thorst_afk has joined #openstack-telemetry | 21:30 | |
| *** thorst_afk has quit IRC | 21:34 | |
| *** thorst_afk has joined #openstack-telemetry | 21:50 | |
| *** thorst_afk has quit IRC | 21:55 | |
| *** gordc has quit IRC | 22:05 | |
| *** vint_bra has quit IRC | 22:37 | |
| *** dave-mccowan has quit IRC | 22:51 | |
| *** rcernin has quit IRC | 22:58 | |
| *** thorst_afk has joined #openstack-telemetry | 23:26 | |
| *** jmlowe has joined #openstack-telemetry | 23:27 | |
| *** thorst_afk has quit IRC | 23:32 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!