Friday, 2017-01-27

*** rcernin has joined #openstack-telemetry00:00
*** harlowja has quit IRC00:04
*** thorst_ has joined #openstack-telemetry00:45
*** hfu has joined #openstack-telemetry00:47
*** hfu has quit IRC00:49
*** rcernin has quit IRC00:51
*** rcernin has joined #openstack-telemetry00:54
*** thorst_ has quit IRC00:55
*** rcernin has quit IRC00:55
*** rcernin has joined #openstack-telemetry00:55
*** leitan has joined #openstack-telemetry01:22
*** leitan has quit IRC01:27
*** thorst_ has joined #openstack-telemetry01:27
*** hfu has joined #openstack-telemetry02:05
*** dims_ has joined #openstack-telemetry02:14
*** dims has quit IRC02:14
*** thorst_ has quit IRC02:18
*** thorst_ has joined #openstack-telemetry02:20
*** dims_ has quit IRC02:47
*** dims has joined #openstack-telemetry02:50
*** Jack_Iv has joined #openstack-telemetry03:07
*** Jack_Iv has quit IRC03:11
*** leitan has joined #openstack-telemetry03:23
*** leitan has quit IRC03:28
*** yarkot has joined #openstack-telemetry03:31
*** thorst_ has quit IRC03:40
*** r-mibu has quit IRC05:03
*** r-mibu has joined #openstack-telemetry05:09
*** nadya has joined #openstack-telemetry05:20
*** leitan has joined #openstack-telemetry05:25
*** leitan has quit IRC05:29
*** Jack_Iv has joined #openstack-telemetry05:33
*** david-lyle has quit IRC05:46
*** thorst_ has joined #openstack-telemetry05:56
*** Jack_Iv_ has joined #openstack-telemetry05:59
openstackgerritaditi sharma proposed openstack/ceilometermiddleware: Remove link to modindex  https://review.openstack.org/42607405:59
*** Jack_Iv_ has quit IRC06:01
*** thorst_ has quit IRC06:01
*** Jack_Iv_ has joined #openstack-telemetry06:01
*** Jack_Iv has quit IRC06:01
*** Jack_Iv has joined #openstack-telemetry06:02
*** nadya has quit IRC06:03
*** Jack_Iv_ has quit IRC06:06
*** Jack_Iv has quit IRC06:09
*** Jack_Iv has joined #openstack-telemetry06:10
openstackgerritGhanshyam Mann proposed openstack/ceilometer: Switch to decorators.idempotent_id  https://review.openstack.org/42608406:12
openstackgerritGhanshyam Mann proposed openstack/aodh: Switch to decorators.idempotent_id  https://review.openstack.org/42608506:12
*** donghao has joined #openstack-telemetry06:41
*** Jack_Iv has quit IRC06:44
*** Jack_Iv has joined #openstack-telemetry06:48
*** Jack_Iv has quit IRC06:53
*** hfu has quit IRC07:09
*** adriant has quit IRC07:11
*** hfu has joined #openstack-telemetry07:17
*** rcernin has quit IRC07:24
*** leitan has joined #openstack-telemetry07:26
*** donghao has quit IRC07:26
*** leitan has quit IRC07:30
*** donghao has joined #openstack-telemetry07:56
*** donghao has quit IRC07:56
*** thorst_ has joined #openstack-telemetry07:57
*** nadya has joined #openstack-telemetry07:59
*** thorst_ has quit IRC08:01
*** donghao has joined #openstack-telemetry08:27
*** donghao has quit IRC08:31
*** elemoine has quit IRC08:31
*** shardy_afk is now known as shardy08:52
*** efoley has joined #openstack-telemetry09:22
*** leitan has joined #openstack-telemetry09:27
*** donghao has joined #openstack-telemetry09:28
*** leitan has quit IRC09:32
*** nadya has quit IRC09:32
*** donghao has quit IRC09:33
*** nadya has joined #openstack-telemetry09:38
*** hfu has quit IRC09:47
*** hfu has joined #openstack-telemetry09:47
*** hfu has quit IRC09:48
*** hfu has joined #openstack-telemetry09:48
*** hfu has quit IRC09:49
*** hfu has joined #openstack-telemetry09:49
*** hfu has quit IRC09:49
*** hfu has joined #openstack-telemetry09:50
*** hfu has quit IRC09:50
*** hfu has joined #openstack-telemetry09:51
*** hfu has quit IRC09:52
*** thorst_ has joined #openstack-telemetry09:58
*** nadya has quit IRC09:59
*** thorst_ has quit IRC10:02
*** cdent has joined #openstack-telemetry10:23
*** shardy has quit IRC10:45
*** shardy has joined #openstack-telemetry10:46
*** vint_bra has joined #openstack-telemetry11:08
*** vint_bra has quit IRC11:12
*** hfu has joined #openstack-telemetry11:13
*** Jack_Iv has joined #openstack-telemetry11:17
*** Jack_Iv has quit IRC11:18
*** Jack_Iv has joined #openstack-telemetry11:19
*** Jack_Iv_ has joined #openstack-telemetry11:21
*** Jack_Iv_ has quit IRC11:22
*** Jack_Iv_ has joined #openstack-telemetry11:22
*** Jack_Iv_ has quit IRC11:24
*** chlong has joined #openstack-telemetry11:24
*** Jack_Iv has quit IRC11:24
*** nadya has joined #openstack-telemetry11:26
*** leitan has joined #openstack-telemetry11:28
*** donghao has joined #openstack-telemetry11:29
*** sanchitmalhotra has quit IRC11:29
*** sanchitmalhotra has joined #openstack-telemetry11:29
*** yassine has joined #openstack-telemetry11:31
*** yassine is now known as Guest2190911:32
*** leitan has quit IRC11:33
*** donghao has quit IRC11:33
*** Jack_Iv has joined #openstack-telemetry11:36
*** thorst_ has joined #openstack-telemetry11:59
*** Jack_Iv has quit IRC12:00
*** Jack_Iv has joined #openstack-telemetry12:00
*** leitan has joined #openstack-telemetry12:02
*** Jack_Iv_ has joined #openstack-telemetry12:02
*** thorst_ has quit IRC12:03
*** catintheroof has joined #openstack-telemetry12:04
*** khushbu has joined #openstack-telemetry12:05
*** Jack_Iv_ has quit IRC12:05
*** Jack_Iv has quit IRC12:06
*** shardy is now known as shardy_lunch12:18
*** vint_bra has joined #openstack-telemetry12:30
*** thorst__ has joined #openstack-telemetry12:43
*** cdent has quit IRC13:04
*** cdent has joined #openstack-telemetry13:04
*** catintheroof has quit IRC13:05
*** gordc has joined #openstack-telemetry13:11
*** khushbu has quit IRC13:13
*** nadya has quit IRC13:14
*** Jack_Iv has joined #openstack-telemetry13:14
*** catintheroof has joined #openstack-telemetry13:16
*** shardy_lunch is now known as shardy13:17
*** Jack_Iv has quit IRC13:19
*** pradk has joined #openstack-telemetry13:21
*** pradk has quit IRC13:31
*** khushbu_ has joined #openstack-telemetry13:37
*** thorst__ is now known as thorst_13:42
*** rbak_ has joined #openstack-telemetry13:50
*** pradk has joined #openstack-telemetry14:05
*** Jack_Iv has joined #openstack-telemetry14:08
*** nadya has joined #openstack-telemetry14:10
*** Jack_Iv has quit IRC14:12
*** nadya has quit IRC14:23
*** nadya has joined #openstack-telemetry14:28
*** fguillot has joined #openstack-telemetry14:31
openstackgerritgordon chung proposed openstack/gnocchi: fix live gabbi test  https://review.openstack.org/42596214:32
*** sanfern has joined #openstack-telemetry14:34
*** efoley has quit IRC14:37
*** efoley has joined #openstack-telemetry14:38
*** efoley has quit IRC14:39
*** tlian has joined #openstack-telemetry14:39
*** efoley has joined #openstack-telemetry14:39
*** khushbu_ has quit IRC14:59
*** efoley_ has joined #openstack-telemetry15:05
*** efoley has quit IRC15:09
jd__pradk: u had a fix for that in tripleo right? https://bugs.launchpad.net/aodh/+bug/165986715:24
openstackLaunchpad bug 1659867 in Aodh "aodh-dbsync fails on missing table 'ceilometer.alarm_change'" [Undecided,New]15:24
jd__pradk: should we reassign?15:25
openstackgerritJulien Danjou proposed openstack/gnocchi: rest: string → UUID conversion for resource.id to be unique per user  https://review.openstack.org/41301715:25
openstackgerritJulien Danjou proposed openstack/gnocchi: indexer: make sure original_resource_id is never NULL  https://review.openstack.org/42582315:25
openstackgerritJulien Danjou proposed openstack/gnocchi: tests: factorize measure generation to speed up test  https://review.openstack.org/42626815:25
pradkjd__, well yea ths fix is really to move aodh to its own db , so i fixed from tripleo side. you can close this15:25
jd__pradk: k15:26
*** Jack_Iv has joined #openstack-telemetry15:27
*** Jack_Iv has quit IRC15:27
*** Jack_Iv has joined #openstack-telemetry15:28
*** eglynn has joined #openstack-telemetry15:29
*** Jack_Iv has quit IRC15:32
*** nadya has quit IRC15:37
*** Guest21909 has quit IRC15:43
gordcshame. looks like the devstack patch is going to fail again. time to nap15:55
*** Guest21909 has joined #openstack-telemetry15:58
dimsgordc : CI needs the weekend to catch up :(15:58
gordcdims: the weekend: the secret to merging on the last week of the cycle. :)16:04
*** efoley__ has joined #openstack-telemetry16:06
*** Jack_Iv has joined #openstack-telemetry16:06
*** Jack_Iv has quit IRC16:07
openstackgerritJulien Danjou proposed openstack/gnocchi: rest: string → UUID conversion for resource.id to be unique per user  https://review.openstack.org/41301716:08
openstackgerritJulien Danjou proposed openstack/gnocchi: indexer: make sure original_resource_id is never NULL  https://review.openstack.org/42582316:08
openstackgerritJulien Danjou proposed openstack/gnocchi: tests: do not create a resource with a custom resource type in 2.2  https://review.openstack.org/42628916:08
*** Jack_Iv has joined #openstack-telemetry16:08
*** efoley_ has quit IRC16:09
*** donghao has joined #openstack-telemetry16:11
*** efoley__ has quit IRC16:12
*** donghao has quit IRC16:13
*** Jack_Iv has quit IRC16:14
*** Jack_Iv has joined #openstack-telemetry16:15
openstackgerritJoseph Davis proposed openstack/ceilometer: cap python-novaclient/oslo.vmware/gabbi  https://review.openstack.org/42500516:30
*** Jack_Iv has quit IRC16:38
*** Jack_Iv has joined #openstack-telemetry16:39
*** Jack_Iv has quit IRC16:39
*** Jack_Iv has joined #openstack-telemetry16:40
openstackgerritgordon chung proposed openstack/ceilometer: drop api and storage references from install-guide  https://review.openstack.org/42630416:49
*** jmlowe1 has joined #openstack-telemetry16:56
jmlowe1ok, so I think my problem is that my omap is way too large to return all the metrics to be processed before the ceph osd times out and dies16:57
jmlowe1Kraken was recently release with a change in omap behavior16:58
jmlowe1"The librados omap get_keys and get_vals operations include a start key and a limit on the number of keys to return. The OSD now imposes a configurable limit on the number of keys and number of total bytes it will respond with, which means that a librados user might get fewer keys than they asked for. This is necessary to prevent careless users from requesting an unreasonable amount of data from the cluster in a single operat16:58
jmlowe1will gnocchi handle this change well if there is a very large backlog?  will only getting a partial list of keys from omap be a problem?16:59
gordcjmlowe1: hmm.. we should only be grabbing the first x key/val in omap... although maybe it's pulling in all and than grabbing first x?17:00
gordclet me take a quick look17:00
gordcregardless, you should be keeping backlog relatively low until we get new scheduling out17:01
gordc(get started)17:01
jmlowe1I'm looking at getting a couple of nvme ssd's and putting dedicated osd's on them for gnocchi so that I won't knock over the osd's that back my cinder and glance17:03
jmlowe1I figure I might have a chance at clearing my backlog as well17:03
jmlowe1it's about 3.5M measurements17:03
jmlowe1it looks like people have had similar trouble by putting 300M files in a single directory on cephfs which puts metadata in omap17:06
jd__those are good questions for sileht17:07
gordcwe can set a limit which we don't... that might help17:09
gordcit won't fix reporting though17:09
gordcjmlowe1: you can try something like http://paste.openstack.org/show/596756/17:13
gordcbut i feel like you will still have issues if you don't turn off reporting processing.17:14
gordcbecause the process requires it to list all 3.5M measures.17:14
jmlowe1ok, is that in the docs somewhere?17:14
gordcdisable reporting process? i don't think you can without hacking :/17:15
*** nadya has joined #openstack-telemetry17:15
gordcjmlowe1: you can comment out https://github.com/openstack/gnocchi/blob/master/gnocchi/cli.py#L30117:16
gordcthere is also a opt to configure how often it runs `metric_reporting_delay` but i've no idea if it runs right away and then waits, or waits and then runs.17:17
*** nadya has quit IRC17:17
openstackgerritgordon chung proposed openstack/gnocchi: ceph: set return limit  https://review.openstack.org/42631417:22
jmlowe1shoved in those patches and so far so good17:23
jmlowe1I may get out of this mess yet17:23
gordcjmlowe1: interesting... my hail mary worked?lol17:24
gordcjmlowe1: i missed an if in there. so it might17:24
gordcerror17:24
jmlowe1so far so good17:25
jmlowe1I'm guessing just turning off metric reporting might have been enough17:25
jd__good to know17:26
jmlowe1I'm down about 100k objects in the gnocchi pool so far17:28
gordcthat is definitely going to be problematic... can't really hack a fix for that17:28
jd__the reporting?17:28
gordcjd__: yeah17:28
jmlowe1it will remove stuff from omap as it processes no?17:29
jd__gordc: is there a "paging" api in Ceph?17:29
gordcjd__: well if we do the bucketise scheduling stuff, it'll help17:29
jd__jmlowe1: yes17:29
gordcjd__:  yeah, there is i believe.17:29
jmlowe1so just a disable reporting option that you can turn on when you get in trouble might be enough17:30
stevellecurrent docs suggest setting gnocchi as an event_dispatcher? http://docs.openstack.org/developer/ceilometer/install/dbreco.html#moving-from-ceilometer-to-gnocchi17:30
jmlowe1in kraken there is an offset17:30
jd__stevelle: yes17:31
jd__jmlowe1: that's what we'd need then17:31
jmlowe1give me all keys starting with x17:31
*** shardy has quit IRC17:31
jd__gordc: that'd be helpful in your patch with the limit17:31
*** Jack_Iv has quit IRC17:31
gordcjd__: i don' think that api exists. (also is kraken released?17:31
jmlowe1you have no idea how long I've been waiting for this, like 6 years http://ceph.com/releases/v11-2-0-kraken-released/17:32
jd__gordc: jmlowe1 just said it does17:32
*** Jack_Iv has joined #openstack-telemetry17:33
gordcjd__: hmm... been there for a while. https://github.com/ceph/ceph/blame/master/src/pybind/rados/rados.pyx#L323817:35
gordci have no idea if it's an offset or a marker...17:35
gordcseems like marker17:35
gordcwhich means it could disappear...17:36
jmlowe1release notes lead me to say marker17:36
gordcjmlowe1: yeah, i'm guessing that too... it expects string17:36
jmlowe1I don't want to even contemplate how to handle that17:36
jmlowe1on the other hand, does it matter if you get all of them?17:37
jmlowe1nm, order probably counts17:37
*** nadya has joined #openstack-telemetry17:38
jd__the problem is that getting all of them makes it crash no?17:39
gordcjmlowe1: in theory the reporting number isn't completely accurate anyways.17:39
jd__if the key does not exist it just returns everything?17:39
gordcjd__: right. but if it's a marker and it disappears, i think it could end up in some massive loop trying to page to the end17:39
gordcit returns everthing if you don't give marker and don't give limit17:40
gordci would think if the marker doesnt exist, it would start from beginning17:40
jmlowe1the problem is that there is a heartbeat and if it times out the osd dies, grabbing all of the keys exceeds the heartbeat time out and failover begins, you get flapping osd's17:41
jd__hehe17:41
gordcjd__: i think we need to bucketise our incoming measures regardless. right now all our incoming measures hit the same osd+filestore. same for swift i believe according to notmynames ML17:43
jd__yup17:43
gordcor well i guess ceph hits different osds but same filestore... something for gnocchi version next17:44
*** nadya has quit IRC17:44
jd__yeah17:45
openstackgerritgordon chung proposed openstack/gnocchi: ceph: set return limit  https://review.openstack.org/42631417:50
*** nicodemus_ has joined #openstack-telemetry18:00
*** harlowja has joined #openstack-telemetry18:02
*** hfu has quit IRC18:02
*** cdent has quit IRC18:02
*** chlong has quit IRC18:03
*** nadya has joined #openstack-telemetry18:11
*** donghao has joined #openstack-telemetry18:14
*** donghao has quit IRC18:18
openstackgerritMerged openstack/ceilometer: cap python-novaclient/oslo.vmware/gabbi  https://review.openstack.org/42500518:22
*** khushbu has joined #openstack-telemetry18:28
*** khushbu has quit IRC18:38
*** gordc has quit IRC18:50
*** Jack_Iv has quit IRC19:03
*** Jack_Iv has joined #openstack-telemetry19:04
nicodemus_hello19:04
nicodemus_one question... if a Nova instance has X metadata key/values, is ceilometer able to gather those metadata values?19:06
*** gordc has joined #openstack-telemetry19:07
nicodemus_I'm wondering if I could create a resource_type containing for example 10 different possible metadata values, and have ceilometer pushing to gnocchi each resource with that metadata (if it exists)19:07
*** Jack_Iv_ has joined #openstack-telemetry19:11
*** Jack_Iv has quit IRC19:14
*** donghao has joined #openstack-telemetry19:15
*** Jack_Iv_ has quit IRC19:18
*** Gleb has joined #openstack-telemetry19:19
*** Jack_Iv has joined #openstack-telemetry19:19
*** donghao has quit IRC19:19
GlebHello! I am in the process of configuring the HTTP dispatcher in Ceilometer and I have a question. In ceilometer.conf, I have the option to set two different targets under [http_dispatcher]: event_target and target. What is the difference between these two?19:20
*** nadya has quit IRC19:21
*** Jack_Iv has quit IRC19:21
*** Jack_Iv has joined #openstack-telemetry19:21
*** nadya has joined #openstack-telemetry19:22
jmlowe1guessing (I'm curious to know if I'm right) event_target is for events (this thing happened at this time like successful auth or instance launch) and target is for measurements (think samples)19:22
*** Jack_Iv has quit IRC19:24
GlebRight now I set both targets to different ports, and I only receive http requests on the address set in "target". I seem to obtain events at the address set by "target".19:25
GlebThe requests only get sent when something happens (e.g. instance launch), and have a event_type field.19:26
GlebHowever there are also periodical requests (every 10 mins) that describe the state of all the resources.19:27
gordcnicodemus_: we can now... but not before ocata19:35
gordcor maybe you can. it's just not as easy19:35
nicodemus_gordc, so... I should look at current master and backport some code if I wanted to make it work with a Newton deploy, right?19:36
gordcnicodemus_: we added this https://review.openstack.org/#/c/413236/19:36
gordci believe in theory you could add metadata before this but you need to know exact value and add it to event_definitions19:37
gordcand then you'd need to create equivalent resource type in gnocchi to capture the data19:38
gordcand probbaly update teh gnocchi mappings https://github.com/openstack/ceilometer/blob/master/ceilometer/dispatcher/data/gnocchi_resources.yaml19:38
gordc(i've never done it recently so i'm just yelling things out as they come into my head)19:39
nicodemus_gordc, I see. I imagined that I would need to modify gnocchi_resources.yaml19:39
gordcnicodemus_: yep19:39
nicodemus_The use case would be to create a resource_type with X metadata keys that are required:false, so if a certain instance has the metadata fine, if not, no problem19:40
nicodemus_I'll take a look to see how complex it would be to backport it19:40
nicodemus_thanks gordc !19:40
gordcnicodemus_: kk, you can specify metadata in meters.yaml file prior to ocata. but you need to specify each attribute specifically... and it might not necessarily be there either... but yeah, you'll need to debug that.19:42
*** eglynn has quit IRC19:42
gordcGleb: the periodic 10min stuff is from polling agents19:43
*** rbak_ has quit IRC19:43
*** rbak_ has joined #openstack-telemetry19:43
gordci don't actually use the http dispatcher but what jmlowe1 said was right (or should be)19:43
GlebOk. Do you have any suggestions about how to enable the event dispatcher? I only get http requests from the meter dispatcher.19:47
gordcGleb: what data are you looking for?19:47
*** Jack_Iv has joined #openstack-telemetry19:48
GlebI just noticed that ceilometer event-list returns an empty list, so I guess I need to set that up first. I'm trying to monitor the creation and deletion of resources (instances, volumes, images, etc)19:48
gordcyou should be able to just define meter_dispatchers=http and event_dispatchers=http and then set the appropriate in http_dispatchers section19:48
*** nadya has quit IRC19:49
GlebIt seems that it's just a matter of setting up events in ceilometer to get the event dispatcher working. Thanks for the help! Will check back in if I have any other questions.19:50
gordcyou should probably be a aware that we pushed all the ceilometer event-* storage to a new service named Panko19:50
GlebYes I know, but I'm stuck on an old release (kilo), so I think that predates Panko.19:51
gordcGleb: yeah :( although you in theory could still use it since it's just an endpoint (although i've never tested that so don't quote me)19:52
gordcprobably best to delete this entire history.19:53
*** david-lyle has joined #openstack-telemetry20:01
openstackgerritgordon chung proposed openstack/ceilometer: update verify to use gnocchi  https://review.openstack.org/42638920:23
*** nicodemus_ has quit IRC20:38
*** pradk has quit IRC20:47
*** nicodemus_ has joined #openstack-telemetry20:50
*** fguillot has quit IRC20:53
*** donghao has joined #openstack-telemetry21:16
*** leitan has quit IRC21:19
*** nicodemus_ has quit IRC21:19
*** donghao has quit IRC21:20
*** pradk has joined #openstack-telemetry21:41
*** catintheroof has quit IRC21:44
*** catintheroof has joined #openstack-telemetry21:45
*** catintheroof has quit IRC21:49
*** vint_bra has quit IRC21:54
*** Guest21909 has quit IRC22:00
*** pradk has quit IRC22:02
*** gordc has quit IRC22:12
*** vishwanathj has joined #openstack-telemetry22:13
*** vishwanathj has quit IRC22:14
*** vishwanathj has joined #openstack-telemetry22:15
*** thorst_ has quit IRC22:16
*** vishwanathj has quit IRC22:17
*** Jack_Iv has quit IRC22:20
*** rbak_ has quit IRC23:19
openstackgerritKen'ichi Ohmichi proposed openstack/ceilometer: WIP: Use Tempest stable library interfaces  https://review.openstack.org/42642723:22

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!