| *** vint_bra has joined #openstack-telemetry | 00:18 | |
| *** jmlowe has quit IRC | 00:19 | |
| *** liusheng has quit IRC | 00:23 | |
| *** psachin has joined #openstack-telemetry | 00:32 | |
| *** rmcallis has joined #openstack-telemetry | 00:33 | |
| -openstackstatus- NOTICE: due to unscheduled restart of zuulv3.o.o you will need to 'recheck' your jobs that were last running. Sorry for the inconvenience. | 00:33 | |
| *** liusheng has joined #openstack-telemetry | 00:34 | |
| *** tovin07_ has joined #openstack-telemetry | 00:40 | |
| *** thorst has joined #openstack-telemetry | 00:57 | |
| *** thorst has quit IRC | 01:02 | |
| *** thorst has joined #openstack-telemetry | 01:03 | |
| *** thorst has quit IRC | 01:08 | |
| *** Tommy-Tom has joined #openstack-telemetry | 01:11 | |
| *** Tommy-Tom has quit IRC | 01:11 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 01:23 | |
| *** AlexeyAbashkin has quit IRC | 01:27 | |
| *** thorst has joined #openstack-telemetry | 02:03 | |
| *** vint_bra has quit IRC | 02:07 | |
| *** thorst has quit IRC | 02:08 | |
| openstackgerrit | Dai Dang Van proposed openstack/ceilometer master: Use generic user for both zuul v2 and v3 https://review.openstack.org/512913 | 02:14 |
|---|---|---|
| openstackgerrit | Dai Dang Van proposed openstack/panko master: Use generic user for both zuul v2 and v3 https://review.openstack.org/512914 | 02:15 |
| *** lhx_ has joined #openstack-telemetry | 02:22 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 02:23 | |
| *** AlexeyAbashkin has quit IRC | 02:27 | |
| *** dave-mccowan has quit IRC | 02:44 | |
| *** chlong has quit IRC | 02:47 | |
| *** tovin07 has quit IRC | 02:47 | |
| *** tovin07 has joined #openstack-telemetry | 02:48 | |
| *** dave-mccowan has joined #openstack-telemetry | 02:48 | |
| *** tovin07 has quit IRC | 02:56 | |
| *** tovin07 has joined #openstack-telemetry | 02:57 | |
| *** thorst has joined #openstack-telemetry | 03:04 | |
| *** thorst has quit IRC | 03:09 | |
| *** tianqing has joined #openstack-telemetry | 03:11 | |
| tianqing | hello... | 03:11 |
| *** jmlowe has joined #openstack-telemetry | 03:12 | |
| *** openstackgerrit has quit IRC | 03:22 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 03:23 | |
| *** AlexeyAbashkin has quit IRC | 03:27 | |
| *** liusheng has quit IRC | 03:28 | |
| *** liusheng has joined #openstack-telemetry | 03:29 | |
| *** tianqing has quit IRC | 03:31 | |
| *** tomtomtom has quit IRC | 03:31 | |
| *** rmcallis has quit IRC | 03:37 | |
| *** links has joined #openstack-telemetry | 03:46 | |
| *** gkadam has joined #openstack-telemetry | 03:55 | |
| *** rmcallis has joined #openstack-telemetry | 04:11 | |
| *** rmcallis has quit IRC | 04:44 | |
| *** dave-mccowan has quit IRC | 04:47 | |
| *** fguillot has joined #openstack-telemetry | 04:51 | |
| *** fguillot has left #openstack-telemetry | 04:52 | |
| *** thorst has joined #openstack-telemetry | 05:05 | |
| *** hyakuhei has quit IRC | 05:05 | |
| *** thorst has quit IRC | 05:10 | |
| *** shardy has quit IRC | 05:23 | |
| *** shardy has joined #openstack-telemetry | 05:25 | |
| *** jmlowe has quit IRC | 05:28 | |
| *** Tommy-Tom has joined #openstack-telemetry | 05:51 | |
| *** iranzo has joined #openstack-telemetry | 05:53 | |
| *** Tom has joined #openstack-telemetry | 05:56 | |
| *** Gautam_ has joined #openstack-telemetry | 06:10 | |
| *** lhx__ has joined #openstack-telemetry | 06:15 | |
| *** lhx_ has quit IRC | 06:17 | |
| *** Tom has quit IRC | 06:19 | |
| *** liusheng has quit IRC | 06:30 | |
| *** liusheng has joined #openstack-telemetry | 06:31 | |
| *** iranzo has quit IRC | 06:36 | |
| *** Tom has joined #openstack-telemetry | 06:39 | |
| *** iranzo has joined #openstack-telemetry | 06:42 | |
| *** Tom has quit IRC | 06:42 | |
| *** Tom has joined #openstack-telemetry | 06:42 | |
| *** pcaruana has joined #openstack-telemetry | 06:45 | |
| lhx__ | tianqi | 06:56 |
| *** thorst has joined #openstack-telemetry | 06:58 | |
| *** thorst has quit IRC | 07:03 | |
| *** openstackgerrit has joined #openstack-telemetry | 07:09 | |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer master: Remove deprecated storage drivers https://review.openstack.org/512700 | 07:09 |
| lhx__ | jd_, around? | 07:15 |
| jd_ | lhx__: don't ask to ask, just ask, like they say, sure around :p | 07:17 |
| *** tesseract has joined #openstack-telemetry | 07:17 | |
| lhx__ | jd_, hah, I want to make sure all datbase dispatcher stuff should be removed because I saw all about ceilometer api are being removed :) | 07:19 |
| lhx__ | jd_, I found I went wrong way and it needn't deprecate database dispatcher before removing. | 07:21 |
| jd_ | lhx__: no need | 07:23 |
| jd_ | lhx__: you had a patch to deprecate them? | 07:23 |
| jd_ | lhx__: I've got a patch to remove them allhttps://review.openstack.org/#/c/512700/ | 07:23 |
| jd_ | lhx__: btw if you have time to dig into https://review.openstack.org/#/c/512286 it fails on the integration job and I don't know why :( maybe your eyes would help | 07:24 |
| lhx__ | jd_, https://review.openstack.org/#/c/505585/ | 07:24 |
| jd_ | lhx__: oh yeah… well that was good before we decided to go ahead and remove everything lol | 07:24 |
| *** yprokule has joined #openstack-telemetry | 07:24 | |
| jd_ | lhx__: it won't be needed if we merge API/storage removal this cycle | 07:25 |
| lhx__ | jd_, okay, I will invest the gate failure | 07:25 |
| jd_ | lhx__: <3 | 07:25 |
| *** shardy is now known as shardy_afk | 07:26 | |
| lhx__ | jd_, this is really a huge patch ;) https://review.openstack.org/#/c/512700/ | 07:26 |
| jd_ | lhx__: it's only removal :) | 07:27 |
| *** tovin07_ has quit IRC | 07:28 | |
| lhx__ | jd_, seems I should abandon my patch ;) | 07:28 |
| jd_ | lhx__: yeah I think you can, we can restore if anything goes wrong anyway | 07:28 |
| lhx__ | jd_, no problem | 07:29 |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer master: Remove Ceilometer API https://review.openstack.org/512286 | 07:30 |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer master: Remove deprecated storage drivers https://review.openstack.org/512700 | 07:30 |
| *** hoonetorg has joined #openstack-telemetry | 07:32 | |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer master: Move oslo.cache to gnocchi flavor https://review.openstack.org/512952 | 07:32 |
| *** Tommy-Tom has quit IRC | 07:33 | |
| *** Tommy-Tom has joined #openstack-telemetry | 07:34 | |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer master: Remove Ceilometer API https://review.openstack.org/512286 | 07:36 |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer master: Remove deprecated storage drivers https://review.openstack.org/512700 | 07:36 |
| *** Tommy-Tom has quit IRC | 07:38 | |
| *** Gautam_ has quit IRC | 07:41 | |
| *** Tommy-Tom has joined #openstack-telemetry | 07:42 | |
| *** Gautam_ has joined #openstack-telemetry | 07:42 | |
| *** lhx__ has quit IRC | 07:46 | |
| *** Gautam_ has quit IRC | 07:47 | |
| *** lhx__ has joined #openstack-telemetry | 07:47 | |
| *** iranzo has quit IRC | 07:48 | |
| *** lhx__ has quit IRC | 07:49 | |
| *** lhx__ has joined #openstack-telemetry | 07:49 | |
| *** iranzo has joined #openstack-telemetry | 07:50 | |
| *** iranzo has quit IRC | 07:50 | |
| *** iranzo has joined #openstack-telemetry | 07:50 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 07:57 | |
| *** shardy_afk is now known as shardy | 08:00 | |
| *** akrzos has quit IRC | 08:07 | |
| *** arxcruz has quit IRC | 08:07 | |
| *** weshay|ruck has quit IRC | 08:07 | |
| *** akrzos has joined #openstack-telemetry | 08:09 | |
| *** weshay has joined #openstack-telemetry | 08:09 | |
| *** arxcruz has joined #openstack-telemetry | 08:10 | |
| *** Tommy-To_ has joined #openstack-telemetry | 08:12 | |
| *** Tommy-Tom has quit IRC | 08:15 | |
| *** yassine has quit IRC | 08:15 | |
| *** yassine has joined #openstack-telemetry | 08:30 | |
| *** yprokule_ has joined #openstack-telemetry | 08:32 | |
| *** lhx__ has quit IRC | 08:35 | |
| *** yprokule has quit IRC | 08:35 | |
| *** yprokule_ is now known as yprokule | 08:35 | |
| *** lhx__ has joined #openstack-telemetry | 08:35 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/ceilometer master: snmp: warn if snmp call timeout https://review.openstack.org/511499 | 08:36 |
| *** Gautam_ has joined #openstack-telemetry | 08:39 | |
| *** joadavis has quit IRC | 08:39 | |
| *** yprokule_ has joined #openstack-telemetry | 08:46 | |
| *** joadavis has joined #openstack-telemetry | 08:47 | |
| *** yprokule has quit IRC | 08:48 | |
| *** yprokule_ is now known as yprokule | 08:48 | |
| *** sileht has quit IRC | 08:50 | |
| *** sileht has joined #openstack-telemetry | 08:50 | |
| *** thorst has joined #openstack-telemetry | 08:59 | |
| *** thorst has quit IRC | 09:04 | |
| *** Tommy-To_ has quit IRC | 09:09 | |
| *** Tommy-Tom has joined #openstack-telemetry | 09:09 | |
| *** Tommy-Tom has quit IRC | 09:11 | |
| *** Tommy-To_ has joined #openstack-telemetry | 09:11 | |
| *** Tom has quit IRC | 09:18 | |
| *** Tommy-To_ has quit IRC | 09:18 | |
| *** Tommy-Tom has joined #openstack-telemetry | 09:19 | |
| *** Tommy-Tom has quit IRC | 09:22 | |
| *** Tommy-Tom has joined #openstack-telemetry | 09:23 | |
| *** Tom____ has joined #openstack-telemetry | 09:24 | |
| *** Tommy-Tom has quit IRC | 09:27 | |
| *** Tommy-Tom has joined #openstack-telemetry | 09:27 | |
| *** Tom____ has quit IRC | 09:28 | |
| openstackgerrit | Merged openstack/ceilometer master: pipeline: remove polling fallback support https://review.openstack.org/511897 | 09:42 |
| openstackgerrit | Merged openstack/ceilometer master: event: move models out of storage https://review.openstack.org/511891 | 09:42 |
| *** AlexeyAbashkin has quit IRC | 09:49 | |
| *** Gautam__ has joined #openstack-telemetry | 09:52 | |
| openstackgerrit | Merged openstack/panko master: Move legacy jobs to project https://review.openstack.org/510024 | 09:53 |
| *** Gautam_ has quit IRC | 09:56 | |
| *** Gautam__ has quit IRC | 09:57 | |
| *** iranzo has quit IRC | 09:57 | |
| *** iranzo has joined #openstack-telemetry | 09:59 | |
| *** iranzo has quit IRC | 09:59 | |
| *** iranzo has joined #openstack-telemetry | 09:59 | |
| *** Tommy-Tom has quit IRC | 10:00 | |
| *** Tommy-Tom has joined #openstack-telemetry | 10:00 | |
| *** Tommy-Tom has quit IRC | 10:05 | |
| *** gmann is now known as gmann_afk | 10:07 | |
| *** Tommy-Tom has joined #openstack-telemetry | 10:08 | |
| *** efoley has joined #openstack-telemetry | 10:10 | |
| *** Tommy-Tom has quit IRC | 10:19 | |
| *** Tommy-Tom has joined #openstack-telemetry | 10:23 | |
| *** Tommy-Tom has quit IRC | 10:27 | |
| *** Gautam_ has joined #openstack-telemetry | 10:36 | |
| *** Gautam_ has quit IRC | 10:40 | |
| *** Gautam_ has joined #openstack-telemetry | 10:43 | |
| *** Gautam_ has quit IRC | 10:47 | |
| *** lhx__ has quit IRC | 10:48 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 10:55 | |
| *** lhx__ has joined #openstack-telemetry | 10:59 | |
| *** thorst has joined #openstack-telemetry | 11:00 | |
| *** chlong has joined #openstack-telemetry | 11:01 | |
| *** efoley has quit IRC | 11:02 | |
| *** efoley_ has joined #openstack-telemetry | 11:02 | |
| *** sileht has quit IRC | 11:04 | |
| *** thorst has quit IRC | 11:05 | |
| *** sileht has joined #openstack-telemetry | 11:07 | |
| *** Gautam_ has joined #openstack-telemetry | 11:17 | |
| *** dave-mccowan has joined #openstack-telemetry | 11:31 | |
| *** dave-mcc_ has joined #openstack-telemetry | 11:35 | |
| *** dave-mccowan has quit IRC | 11:37 | |
| *** thorst has joined #openstack-telemetry | 11:52 | |
| *** efoley has joined #openstack-telemetry | 11:56 | |
| *** efoley_ has quit IRC | 11:56 | |
| *** edmondsw has joined #openstack-telemetry | 12:21 | |
| *** Gautam_ has quit IRC | 12:27 | |
| *** Gautam_ has joined #openstack-telemetry | 12:27 | |
| *** dave-mcc_ is now known as dave-mccowan | 12:29 | |
| *** Gautam_ has quit IRC | 12:32 | |
| *** gkadam has quit IRC | 12:34 | |
| *** gordc has joined #openstack-telemetry | 12:41 | |
| *** lhx__ has quit IRC | 12:42 | |
| *** swamireddy has quit IRC | 12:43 | |
| *** links has quit IRC | 12:44 | |
| *** gmann_afk is now known as gmann | 12:52 | |
| *** Gautam_ has joined #openstack-telemetry | 12:57 | |
| *** shardy has quit IRC | 13:00 | |
| *** shardy has joined #openstack-telemetry | 13:00 | |
| *** sileht has quit IRC | 13:01 | |
| *** sileht has joined #openstack-telemetry | 13:02 | |
| *** efoley has quit IRC | 13:10 | |
| *** shardy has quit IRC | 13:11 | |
| *** Gautam_ has quit IRC | 13:13 | |
| *** shardy has joined #openstack-telemetry | 13:13 | |
| *** efoley has joined #openstack-telemetry | 13:13 | |
| *** rmcallis has joined #openstack-telemetry | 13:15 | |
| *** rmcallis has quit IRC | 13:19 | |
| openstackgerrit | gordon chung proposed openstack/ceilometer master: common notification endpoint https://review.openstack.org/509612 | 13:19 |
| openstackgerrit | gordon chung proposed openstack/ceilometer master: rename sample handler https://review.openstack.org/509648 | 13:19 |
| openstackgerrit | gordon chung proposed openstack/ceilometer master: move listener targets to notification agent https://review.openstack.org/509668 | 13:19 |
| openstackgerrit | gordon chung proposed openstack/ceilometer master: separate base manager from pipeline https://review.openstack.org/509826 | 13:19 |
| openstackgerrit | gordon chung proposed openstack/ceilometer master: separate polling code https://review.openstack.org/510124 | 13:19 |
| *** Gautam_ has joined #openstack-telemetry | 13:20 | |
| *** efoley has quit IRC | 13:21 | |
| gordc | jd_: want to approve? https://review.openstack.org/#/c/510403/ | 13:23 |
| *** leitan has joined #openstack-telemetry | 13:23 | |
| *** efoley has joined #openstack-telemetry | 13:27 | |
| *** Tommy-Tom has joined #openstack-telemetry | 13:29 | |
| jd_ | gordc: sure | 13:31 |
| *** lhx_ has joined #openstack-telemetry | 13:32 | |
| *** lhx__ has joined #openstack-telemetry | 13:32 | |
| *** lhx_ has quit IRC | 13:32 | |
| gordc | merci | 13:35 |
| *** Gautam_ has quit IRC | 13:41 | |
| *** MVenesio has joined #openstack-telemetry | 13:42 | |
| *** Gautam_ has joined #openstack-telemetry | 13:42 | |
| MVenesio | Hi guys, we're having some issues with out of memory in gnocchi metricd and gnocchi api instances, we're using the official gnocchi4 container, is there any recommendation about the amount of memory to set in our instances regarding the amount of metrics to process, to avoid this issue ? | 13:45 |
| *** Gautam_ has quit IRC | 13:46 | |
| *** Gautam_ has joined #openstack-telemetry | 13:47 | |
| gordc | MVenesio: anything useful in the error? | 13:48 |
| *** efoley has quit IRC | 13:48 | |
| gordc | MVenesio: any idea how much memory it's using? we have some memory tweaks in upcoming gnocchi release but it's not to address OOM error | 13:49 |
| MVenesio | gordc: so far just an OOM from the SO | 13:49 |
| MVenesio | gordc: we are using 8GB instances | 13:50 |
| *** Tom____ has joined #openstack-telemetry | 13:50 | |
| *** tomtomtom has joined #openstack-telemetry | 13:50 | |
| *** vint_bra has joined #openstack-telemetry | 13:50 | |
| gordc | how many api/metricd workers you have on those instances or do you mean each worker has 8GB? | 13:51 |
| gordc | i think when i checked, each worker used about ~135MB of memory (that will shrink to 90MB in next release) | 13:52 |
| gordc | i haven't noticed any memory leak in my testing but it's possible. | 13:52 |
| *** efoley has joined #openstack-telemetry | 13:54 | |
| *** Tom____ has quit IRC | 13:54 | |
| tomtomtom | @gordc, so yesterday you mentioned running 4.0, is that going to have any compatibility issues with ceilometer in newton? | 13:56 |
| MVenesio | gordc: whe're using 4 workers for api and metricd all running in the same container | 13:57 |
| *** vint_bra has quit IRC | 13:57 | |
| gordc | tomtomtom: you would need to backport gnocchi dispatcher code: https://github.com/openstack/ceilometer/tree/stable/pike/ceilometer/dispatcher | 13:58 |
| MVenesio | gordc: so 8 workers using 135MB of memory give us a total of 1GB and the containers are taking 3GB right now | 13:59 |
| gordc | the newton gnocchi dispatcher code won't work against gnocchi 4. (or i don't believe it will) | 13:59 |
| tomtomtom | ok so sticking with my current gnocchi and then downgrading client is probably fastest at this point. | 13:59 |
| gordc | MVenesio: yeah, that doesn't seem like it would use up the memory. | 13:59 |
| gordc | does it throw OOM on query or just while it's processing in background? | 14:02 |
| gordc | i imagine you could run out of memory if you run a query on very large dataset OR you have a big back window defined | 14:03 |
| MVenesio | gordc: in this case we are using uwsgi and is configured with 6 processes and 32 threads, is this right ? | 14:04 |
| *** psachin has quit IRC | 14:05 | |
| gordc | so when you say 4 api, you mean 4 uwsgi with 6 processes? | 14:05 |
| *** aagate has joined #openstack-telemetry | 14:08 | |
| MVenesio | gordc: nono just one container running api and metricd, in gnocchi.conf we have configured 8 workers for api and metricd. but the uwsgi is configured to use 6 processes and 32 threads | 14:08 |
| jd_ | MVenesio: can you monitor the memory usage of each process to see which one grows and maybe guess why? | 14:10 |
| gordc | wasn't even aware you could configure workers for api.lol | 14:11 |
| MVenesio | gordc: yes of course we can, we're doing it right now to see how much memory is consuming each process about its about 500MB, but it will increase with the time | 14:14 |
| MVenesio | gordc: in uwsgi.ini you can configure processes and threads for the api and the metricd is configured using gnocchi.conf | 14:17 |
| gordc | MVenesio: oh, i always used uwsgi/apache to define workers. | 14:17 |
| gordc | i don't understand "8 workers for api... but the uwsgi is configured to use 6 processes" | 14:18 |
| *** Gautam_ has quit IRC | 14:18 | |
| gordc | but yeah, i think the big concern is "increase with the time" that shouldn't be happening | 14:18 |
| gordc | as jd_ mentioned, it'll help to check which of the processes are increasing. | 14:19 |
| gordc | i haven't really noticed (but i also have 256GB memory on my test node) | 14:19 |
| MVenesio | gordc: yes sorry my mistake there. the api is configured with 6 processes and 32 threads regarding uwsgi.ini and the metricd is using the number of cores of the vm | 14:20 |
| jd_ | well the memory used by metricd is not fixed as it grows when it processes metrics, and if your backlog is HUGE, it can grow big | 14:20 |
| jd_ | but if you have a small backlog that you processe steadily, that should not be a problem | 14:20 |
| leitan | im taking over MVenesio hes out for the beauty salon | 14:22 |
| gordc | lol | 14:22 |
| *** sergio_nubeliu has joined #openstack-telemetry | 14:23 | |
| leitan | here's sergio_nubeliu thats actually whos is following the issue | 14:23 |
| leitan | do you think our API+metricd vm is undersized for our metric flow ? | 14:24 |
| *** sergio_nubeliu has quit IRC | 14:24 | |
| leitan | gordc jd_ | 14:24 |
| *** sergio_nubeliu_ has joined #openstack-telemetry | 14:24 | |
| gordc | jd_: that backlog comment might be something to consider. | 14:25 |
| gordc | leitan: how many sacks do you have? | 14:25 |
| jd_ | leitan: we need more info, if you don't give more info, we can't say :) | 14:25 |
| leitan | jd_: sorry i thought that MVenesio already told you that | 14:26 |
| gordc | so yeah, 3 potential places: if you make a query on very large dataset; if you have very few sacks and very large backlog; or if you have a very big back window. | 14:26 |
| openstackgerrit | Merged openstack/ceilometer master: Use generic user for both zuul v2 and v3 https://review.openstack.org/512913 | 14:27 |
| openstackgerrit | Merged openstack/ceilometer master: Move oslo.cache to gnocchi flavor https://review.openstack.org/512952 | 14:27 |
| leitan | gordc: the backlog was almost empty at the time the server died | 14:27 |
| leitan | were processing steady | 14:27 |
| openstackgerrit | Merged openstack/aodh master: Move legacy jobs to project https://review.openstack.org/510403 | 14:28 |
| leitan | we are using an AWS c4.xlarge C4 High-CPU Extra Largec4.xlarge7.5 GiB4 vCPUs | 14:29 |
| leitan | im suspicious about the amount of threads were using with uwsgi here, maybe metricd is not the one to blame | 14:29 |
| gordc | tbh, i don't even have threads enabled in my uwsgi setup | 14:30 |
| gordc | or it's set to 1 | 14:30 |
| leitan | gordc: here is our uwsgi config https://pastebin.com/fqYYc3Cz | 14:30 |
| gordc | the parallel work the api does, it already creates it's own theads. | 14:30 |
| gordc | threads* | 14:30 |
| leitan | gordc: for what im looking, its exactly the same as the docs here: http://gnocchi.xyz/operating.html | 14:31 |
| leitan | and the docs also says there that you need to take care of adjusting accordingly to the cpu | 14:32 |
| leitan | so maybe our cpu / ram ratio is not correct here | 14:32 |
| leitan | and we should take care of, or adjusting uwsgi config, or changing the flavor | 14:33 |
| *** Gautam_ has joined #openstack-telemetry | 14:33 | |
| *** efoley has quit IRC | 14:33 | |
| gordc | i have: http://paste.openstack.org/show/623990/ | 14:33 |
| *** efoley has joined #openstack-telemetry | 14:33 | |
| gordc | but i also just set it once and have never changed it | 14:33 |
| *** gmann is now known as gmann_afk | 14:33 | |
| gordc | i do remember when i used threads (a long time ago) it didn't do anything... but this was back when i first started benchmarking gnocchi2 | 14:34 |
| leitan | gordc: so, maybe commenting out the threads 32 option on my uwsgi can be a good starting point | 14:35 |
| gordc | you can try. i'm not sure why it increases in memory though. | 14:37 |
| gordc | something seems wrong there. | 14:37 |
| *** Gautam_ has quit IRC | 14:37 | |
| gordc | do you know how many sacks you have configured? | 14:38 |
| gordc | i should probably mention i have 48 hyper threaded cores on my server which is why i have put 32 processes | 14:43 |
| leitan | ours its an AWS vm | 14:47 |
| leitan | 4 cpu | 14:47 |
| leitan | so im seeing the 4vcpu | 14:47 |
| *** yprokule has quit IRC | 14:47 | |
| gordc | i see, i also have ceph and 18 metricd on the same server. | 14:48 |
| gordc | leitan: but yeah, how many sacks? or you using default? | 14:48 |
| leitan | sorry gordc the only sack i now are the tcp selective acknoledges, what sacks are you referring to | 14:49 |
| *** chlong has quit IRC | 14:52 | |
| gordc | leitan: in gnocchi4 the data gets sharded into sacks/buckets. | 14:53 |
| leitan | gordc: i have no further config than my s3_bucket_prefix on gnocchi.conf | 14:55 |
| leitan | so i guess were using default | 14:55 |
| gordc | yeah, https://github.com/gnocchixyz/gnocchi/blob/master/gnocchi/cli/manage.py#L46 | 14:55 |
| gordc | in theory, i imagine if you had set a really high number it could use up memory because of the hashring built off of it. | 14:56 |
| leitan | gordc: the only thing i have on the incoming section is our redis url | 14:56 |
| gordc | but 128 is not a lot. and you have little/no backlog | 14:56 |
| gordc | leitan: it won't be in conf, it's a conf written to your storage when you upgrade | 14:56 |
| leitan | gordc: i see, our storage backend is s3 nnow | 14:57 |
| gordc | but yeah, not a reason for OOM. i suggest just monitoring processes to see which ones are increasing in memory... and we can look at that. | 14:57 |
| leitan | gordc: yes, we will need further monitoring on this to have a better idea of whats going on | 14:58 |
| leitan | thanks | 14:58 |
| gordc | np | 15:02 |
| *** efoley has quit IRC | 15:02 | |
| *** efoley has joined #openstack-telemetry | 15:02 | |
| *** Tommy-Tom has quit IRC | 15:12 | |
| *** Tommy-Tom has joined #openstack-telemetry | 15:13 | |
| *** catintheroof has joined #openstack-telemetry | 15:13 | |
| *** Tommy-Tom has quit IRC | 15:17 | |
| *** pcaruana has quit IRC | 15:19 | |
| *** tesseract has quit IRC | 15:20 | |
| lhx__ | gordc, sileht, I'm not clear why the error took effect on autoscaling tests, could you have a look? http://logs.openstack.org/86/512286/9/check/telemetry-dsvm-integration-ceilometer/c4165a0/logs/tempest.txt.gz#_2017-10-18_08_36_00_659 | 15:26 |
| lhx__ | for this patch https://review.openstack.org/#/c/512286/ | 15:26 |
| *** rmcallis has joined #openstack-telemetry | 15:27 | |
| *** rmcallis has quit IRC | 15:27 | |
| *** MVenesio has quit IRC | 15:28 | |
| sileht | /metric/v1/aggregation/resource/instance/metric/cpu_util?aggregation=mean&granularity=60&needed_overlap=0&refresh=False&start=2017-10-18T08%3A32%3A47.794264&stop=2017-10-18T08%3A34%3A47.794264 => generated 135 bytes in 47 msecs (HTTP/1.1 404) 3 headers in 98 bytes (1 switches on core 0) | 15:42 |
| sileht | in http://logs.openstack.org/86/512286/9/check/telemetry-dsvm-integration-ceilometer/c4165a0/logs/screen-gnocchi-api.txt.gz | 15:43 |
| sileht | that's the cause | 15:43 |
| sileht | lhx__, ^ | 15:43 |
| sileht | why is that returning 404 | 15:43 |
| sileht | We need more log of gnocchi-api to understand why it return 400 | 15:44 |
| sileht | 404 | 15:44 |
| *** vint_bra has joined #openstack-telemetry | 15:44 | |
| lhx__ | sileht, agreed :) | 15:45 |
| *** Gautam_ has joined #openstack-telemetry | 15:46 | |
| sileht | usualy that's because the granularity or aggregation method doesn't exists | 15:50 |
| *** jmlowe has joined #openstack-telemetry | 15:55 | |
| *** links has joined #openstack-telemetry | 15:55 | |
| jd_ | lhx__: got any hint on that integration failure? | 15:59 |
| jd_ | lhx__: ah ok maybe I should have read the backlog first lol | 15:59 |
| *** gkadam has joined #openstack-telemetry | 16:02 | |
| lhx__ | jd_, sorry, I have no idea, maybe it's good way to add more logs to show how it fails | 16:09 |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer master: Remove Ceilometer API https://review.openstack.org/512286 | 16:15 |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer master: Remove deprecated storage drivers https://review.openstack.org/512700 | 16:15 |
| *** gkadam has quit IRC | 16:22 | |
| *** Gautam_ has quit IRC | 16:28 | |
| *** links has quit IRC | 16:28 | |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer master: Remove deprecated storage drivers https://review.openstack.org/512700 | 16:32 |
| openstackgerrit | Julien Danjou proposed openstack/ceilometer master: Replace jsonutils by ujson https://review.openstack.org/513098 | 16:43 |
| *** AlexeyAbashkin has quit IRC | 16:47 | |
| *** iranzo has quit IRC | 16:48 | |
| *** sergio_nubeliu_ has quit IRC | 16:54 | |
| *** jmlowe has quit IRC | 16:58 | |
| *** lhx__ has quit IRC | 17:19 | |
| *** efoley has quit IRC | 17:30 | |
| *** catinthe_ has joined #openstack-telemetry | 17:53 | |
| *** catintheroof has quit IRC | 17:53 | |
| *** catintheroof has joined #openstack-telemetry | 17:59 | |
| *** catinthe_ has quit IRC | 18:01 | |
| *** catinthe_ has joined #openstack-telemetry | 18:05 | |
| openstackgerrit | gordon chung proposed openstack/aodh stable/pike: Move legacy jobs to project https://review.openstack.org/513153 | 18:05 |
| openstackgerrit | gordon chung proposed openstack/aodh stable/ocata: Move legacy jobs to project https://review.openstack.org/513154 | 18:05 |
| *** catintheroof has quit IRC | 18:07 | |
| *** catintheroof has joined #openstack-telemetry | 18:14 | |
| *** catinthe_ has quit IRC | 18:15 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 18:19 | |
| *** AlexeyAbashkin has quit IRC | 18:24 | |
| *** rwsu has quit IRC | 18:25 | |
| *** Tom__ has joined #openstack-telemetry | 18:27 | |
| *** weshay is now known as weshay|ruck|brb | 18:30 | |
| *** efoley has joined #openstack-telemetry | 18:57 | |
| *** efoley has quit IRC | 18:57 | |
| *** efoley has joined #openstack-telemetry | 18:58 | |
| *** weshay|ruck|brb is now known as weshay|ruck | 19:18 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 19:19 | |
| *** AlexeyAbashkin has quit IRC | 19:24 | |
| *** rwsu has joined #openstack-telemetry | 19:37 | |
| *** openstackgerrit has quit IRC | 20:17 | |
| *** openstackgerrit has joined #openstack-telemetry | 20:30 | |
| openstackgerrit | gordon chung proposed openstack/panko stable/pike: Move legacy jobs to project https://review.openstack.org/513178 | 20:30 |
| openstackgerrit | gordon chung proposed openstack/panko stable/ocata: Move legacy jobs to project https://review.openstack.org/513179 | 20:31 |
| *** catintheroof has quit IRC | 20:53 | |
| *** leitan has quit IRC | 20:53 | |
| *** catintheroof has joined #openstack-telemetry | 20:54 | |
| *** sbezverk has quit IRC | 20:54 | |
| *** catintheroof has quit IRC | 20:58 | |
| *** thorst has quit IRC | 21:04 | |
| *** john5223 has quit IRC | 21:14 | |
| *** john5223 has joined #openstack-telemetry | 21:15 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 21:19 | |
| openstackgerrit | Merged openstack/ceilometer master: agent: log pollster skipping to debug https://review.openstack.org/510173 | 21:21 |
| openstackgerrit | Merged openstack/ceilometer master: Imported Translations from Zanata https://review.openstack.org/510336 | 21:21 |
| *** AlexeyAbashkin has quit IRC | 21:24 | |
| *** gordc has quit IRC | 21:27 | |
| *** jmlowe has joined #openstack-telemetry | 21:39 | |
| *** edmondsw has quit IRC | 21:44 | |
| openstackgerrit | Merged openstack/python-ceilometerclient master: import content from cli-reference in openstack-manuals https://review.openstack.org/482204 | 21:45 |
| *** openstackgerrit has quit IRC | 21:48 | |
| *** edmondsw has joined #openstack-telemetry | 21:50 | |
| *** edmondsw has quit IRC | 21:54 | |
| *** edmondsw has joined #openstack-telemetry | 21:56 | |
| *** efoley has quit IRC | 21:56 | |
| *** edmondsw_ has joined #openstack-telemetry | 21:58 | |
| *** edmondsw has quit IRC | 22:00 | |
| *** edmondsw_ has quit IRC | 22:02 | |
| *** leitan has joined #openstack-telemetry | 22:04 | |
| *** rwsu has quit IRC | 22:05 | |
| *** leitan has quit IRC | 22:08 | |
| *** dave-mccowan has quit IRC | 22:11 | |
| *** Tommy-Tom has joined #openstack-telemetry | 22:13 | |
| *** jmlowe has quit IRC | 22:16 | |
| *** Tommy-Tom has quit IRC | 22:18 | |
| *** edmondsw has joined #openstack-telemetry | 22:32 | |
| *** jmlowe has joined #openstack-telemetry | 22:33 | |
| *** edmondsw has quit IRC | 22:36 | |
| *** Tom__ has quit IRC | 23:00 | |
| *** rwsu has joined #openstack-telemetry | 23:01 | |
| *** dave-mccowan has joined #openstack-telemetry | 23:06 | |
| *** vint_bra has quit IRC | 23:12 | |
| *** AlexeyAbashkin has joined #openstack-telemetry | 23:19 | |
| *** AlexeyAbashkin has quit IRC | 23:23 | |
| *** thorst has joined #openstack-telemetry | 23:32 | |
| *** thorst has quit IRC | 23:36 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!