Thursday, 2017-06-22

*** jmlowe has quit IRC00:14
*** jmlowe has joined #openstack-telemetry00:15
openstackgerritzhangdaolong proposed openstack/ceilometer master: Fix can not find VMwareAPISession when import oslo_vmware  https://review.openstack.org/47598300:31
*** tovin07_ has joined #openstack-telemetry00:43
*** zhangguoqing has joined #openstack-telemetry01:15
*** chlong has joined #openstack-telemetry01:43
*** zhurong has joined #openstack-telemetry01:52
*** chlong has quit IRC02:04
*** gongysh has joined #openstack-telemetry02:24
*** chlong has joined #openstack-telemetry02:36
*** lhx__ has joined #openstack-telemetry03:00
*** zhangguoqing has quit IRC03:38
*** gongysh has quit IRC03:39
*** zhangguoqing has joined #openstack-telemetry03:40
*** gongysh has joined #openstack-telemetry03:42
*** rwsu has joined #openstack-telemetry03:49
*** zhurong has quit IRC03:49
*** gongysh has quit IRC03:53
*** dave-mccowan has quit IRC03:56
*** chlong has quit IRC04:02
*** chlong has joined #openstack-telemetry04:15
*** links has joined #openstack-telemetry04:21
*** zhangguoqing has quit IRC04:22
*** rwsu has quit IRC05:19
*** gongysh has joined #openstack-telemetry05:47
*** zhangguoqing has joined #openstack-telemetry05:58
*** yprokule has joined #openstack-telemetry06:01
*** zhangguoqing has quit IRC06:06
*** zhangguoqing has joined #openstack-telemetry06:06
*** zhangguoqing has quit IRC06:26
*** rcernin has joined #openstack-telemetry06:30
*** rcernin has quit IRC06:37
*** rcernin has joined #openstack-telemetry06:50
*** tesseract has joined #openstack-telemetry07:02
*** pcaruana has joined #openstack-telemetry07:05
*** yprokule_ has joined #openstack-telemetry07:18
*** yprokule has quit IRC07:21
*** yprokule_ is now known as yprokule07:21
*** liusheng has quit IRC07:36
*** liusheng has joined #openstack-telemetry07:37
*** zhangguoqing has joined #openstack-telemetry08:02
*** yprokule_ has joined #openstack-telemetry08:13
*** yprokule has quit IRC08:16
*** yprokule_ is now known as yprokule08:16
*** zhurong has joined #openstack-telemetry08:27
korean101anyone knows this ERRORs? (ERROR ceilometer.dispatcher.gnocchi [-] Resource ec9daaf0-501b-433a-a465-152e85ab5c88 does not exist (HTTP 404))08:53
korean101i use Newton and Ocata releases08:53
openstackgerritLingyong Xu proposed openstack/aodh master: Enable some off-by-default checks  https://review.openstack.org/47642609:09
*** efoley has joined #openstack-telemetry09:13
*** zhurong has quit IRC09:16
*** zhangguoqing has quit IRC09:22
*** zhangguoqing has joined #openstack-telemetry09:52
*** zhangguoqing has quit IRC09:56
*** tovin07_ has quit IRC10:01
*** zhangguoqing has joined #openstack-telemetry10:29
*** zhangguoqing has quit IRC10:29
*** zhangguoqing has joined #openstack-telemetry10:30
*** zhurong has joined #openstack-telemetry10:32
*** gongysh has quit IRC10:47
*** pradk has joined #openstack-telemetry11:49
*** rwsu has joined #openstack-telemetry11:50
*** zhurong has quit IRC12:01
*** zhurong has joined #openstack-telemetry12:16
*** zhurong has quit IRC12:18
*** chlong has quit IRC12:26
*** gongysh has joined #openstack-telemetry12:27
*** gordc has joined #openstack-telemetry12:39
*** dave-mccowan has joined #openstack-telemetry12:43
*** cristicalin has joined #openstack-telemetry12:43
cristicalindo I still need to deploy the ceilometer-api if I want to use gnocchi as the timeseries database ?12:44
cristicalinthe docs say it will respond with a 410 if gnocchi is configured12:45
cristicalinso in this case which components do i still need to deploy from ceilometer ?12:45
cristicalinI'm doing this with mitaka12:45
*** lhx__ has quit IRC12:48
*** lhx__ has joined #openstack-telemetry12:48
gordccristicalin: you don't need to deploy ceilometer-api if you're using gnocchi. unless you want event storage.12:54
cristicalingordc, no, I only need the samples part12:55
cristicalinisn't event storage done by panko these days ?12:55
gordcyou will probably want to backport the ceilometer+gnocchi dispatcher code from newer code though if you're using mitaka12:55
gordccristicalin: yes, but panko didn't official get release until newton12:56
cristicalinok, I'll keep that in mind for the ocata shift12:56
gordcso you can use event storage in ceilometer in mitaka12:56
cristicalinwe'll probably skip newton12:56
gordccristicalin: makes sense12:56
cristicalinthanks12:56
gordcnp12:57
*** efoley has quit IRC12:57
*** pradk has quit IRC13:02
cristicalingordc, do I need any ceilometer-dbsync in mitaka when running with gnocchi ?13:02
cristicalinas far as I can tell that one is only reserved for mysql/pg/es/mongo scenarii13:03
*** lhx__ has quit IRC13:03
gordccristicalin: you're correct13:07
gordcno need for ceilometer-upgrade/dbsync13:08
cristicalinI get a lot of these:13:10
cristicalin2017-06-22 13:10:12.018 14 WARNING ceilometer.coordination [req-101ba2f5-c609-4f8a-a699-acf038e332b1 - - - - -] Cannot extract tasks because agent failed to join group properly. Rejoining group.13:10
cristicalin2017-06-22 13:10:12.020 14 INFO ceilometer.coordination [req-101ba2f5-c609-4f8a-a699-acf038e332b1 - - - - -] Joined partitioning group ceilometer.notification13:10
cristicalinany idea what it's complaining about ?13:11
cristicalinI use a twemproxy instance in front of 2 memcaches for coordination13:11
*** vint_bra has joined #openstack-telemetry13:11
gordcwhich log is that from? you have workload_partitioning enabled for some service13:11
cristicalinyes, I do13:12
cristicalini's the agent-notification log13:12
gordcdoes the log keep throwing the warning or does it become an error at some point?13:13
gordci do recall in previous builds there was some connection/racing issues if you had workload_partitionnig enabled and set multiple workers13:13
gordcbut it should resolve itself13:14
cristicalinnope, doesn't throw an error13:14
cristicalinjust these warnings13:14
gordccristicalin: i'm guessing you don' thave debug on?13:15
gordci think it should throw an error if it actually fails.13:16
cristicalinyes, I do13:16
cristicalinthis is a staging build13:16
gordccristicalin: do you see https://github.com/openstack/ceilometer/blob/stable/mitaka/ceilometer/coordination.py#L19513:16
gordcsomething that says "my subset"13:16
cristicalinyes13:16
*** jmlowe has quit IRC13:16
gordcok, as long as the list isn't empty, it's ok.13:16
cristicalin2017-06-22 13:16:31.767 13 DEBUG ceilometer.coordination [req-28b7dd49-c223-4235-b884-e786d7e15e46 - - - - -] Members of group: ['b928918c-08c5-451e-bac1-aa4bbfcb5acc', 'e1ef82a6-02af-4f15-a40f-1320b2630c70', 'e9328785-1cec-43e5-9839-409089633ce3', '08d7588a-64ea-4934-90dc-624c79226f2f', '768a6cc2-f180-444f-bc97-77f50bc43c6e'], Me: 9c144019-32de-46eb-8d2f-ad0d2367fdc4 extract_my_subset /opt/ceilometer/lib/python213:16
cristicalin2017-06-22 13:16:31.768 13 WARNING ceilometer.coordination [req-28b7dd49-c223-4235-b884-e786d7e15e46 - - - - -] Cannot extract tasks because agent failed to join group properly. Rejoining group.13:17
cristicalin2017-06-22 13:16:31.770 13 INFO ceilometer.coordination [req-28b7dd49-c223-4235-b884-e786d7e15e46 - - - - -] Joined partitioning group ceilometer.notification13:17
gordcor you can just verify that all the pipeline queues have a consumer13:17
gordcwe fixed this properly in ocata i believe so it doesn't do the retries13:17
gordcit retries because when we start multiple workers in older builds, it gets sort of racy when trying to register to coordination service.13:18
cristicalinmaybe its my temproxy+memcache setup for coordination though13:19
cristicalinhave you tested workload partitioning with memcache ?13:19
cristicalinto docs only speak about redis13:19
cristicalinas a coordinator13:19
gordci've only used redis. and i don't operate a real cloud (handful of nodes).13:19
*** catintheroof has joined #openstack-telemetry13:20
*** zhangguoqing has quit IRC13:20
*** pradk has joined #openstack-telemetry13:22
cristicalinok, going to deploy a redis cluster then13:23
cristicalinindeed group membership looks kind of wakky13:25
cristicalinI have 4 instances of the service with 1 worker each13:25
cristicalinthere are 4 members in the group but the local member id is not in there13:25
cristicalinso yeah, probably memcache does not cut it13:26
gordccristicalin: probably both. the group membership stuff was kind of hacky previously as well.13:33
cristicalinis the ocata patch backportable to mitaka ?13:33
cristicalinthe one you memtioned fixes this hacky part13:34
gordccristicalin: um.. there was two patches. one is kind of big and requires a fairly recent tooz release.13:35
cristicalincan you point me to the gerrit for them ?13:35
gordcthe other fixed issue which relates to multiple workers in single service instance. so not really your scenario13:35
gordccristicalin: the easiest 'fix' might be to try a more recent tooz to see if it handles concurrent member joins better.13:36
cristicalinlike how recent ?13:37
cristicalinI have 1.3413:37
gordccristicalin: no idea. i'm not sure if there was a patch in tooz that directly addresssed this.13:37
gordccristicalin: https://github.com/openstack/ceilometer/commit/2726419a11fd3206c2dc75f1aa4639a42e0546b5 is the big patch13:37
cristicalinso head tooz is 1.57.2 ... so mine is quite the antique13:38
*** chlong has joined #openstack-telemetry13:38
gordccristicalin: lesser change. https://github.com/openstack/ceilometer/commit/9fcd66ec66657c4b597096a946635956c2b493f513:40
cristicalincan I do without workload partitioning ?13:40
gordccristicalin: if you don't have transformers in your pipeline.yaml13:41
cristicalinI'm using the default pipeline.yaml which calculates some rates but I don't do anything with those rate parameters so I guess I could discard them13:43
gordcthen in theory, you don't need it partitioning13:45
cristicalinon second thought , cpu_util might be used by some users of heat13:46
cristicalinbut I'll explore the redis option first and the second patch is trivial to backport so maybe that provides enough to have a working version13:47
gordcususally the issue is when you have multiple services/workers register at the same time.13:48
cristicalinok, new error13:52
*** lhx_ has joined #openstack-telemetry13:52
cristicalin NoMatches: No 'ceilometer.publisher' driver found, looking for 'gnocchi'13:52
gordccristicalin: i assume you're using ocata instructions?13:53
cristicalini think so13:53
gordccristicalin: you'll need to backport that if you want to run gnocchi as a publisher (i think)13:53
cristicalinwhat's on the developer site13:53
gordcyeah.13:53
gordcor something close to ocata13:53
cristicalinso gnocchi wasn't supported as a publisher in mitaka ?13:54
gordci'm not sure when.13:54
gordcyou can follow https://docs.openstack.org/project-install-guide/telemetry/ocata/install-base-rdo.html13:54
gordcwhich still references dispatcher (even though you don't need to)13:54
*** zhangguoqing has joined #openstack-telemetry13:54
cristicalinany chance this might work with the rest of the cloud being at mitaka level ?13:55
gordcyou'll need https://github.com/openstack/ceilometer/commit/f843b7882fb806cf564c5b3106f601815a48c93b if you want to drop collector service and use publisher directly13:55
cristicalinlike have ceilometer ocata + aodh ocata running on top of a mitaka cloud ?13:55
gordci think we only tested mitaka against gnocchiv2... which imo is not a pleasant experience13:56
gordcgnocchiv3/v4 are signficantly more useable. and the ceilometer+gnocchi integration is a bit more streamlined in newer builds to reduce number of http requests.13:56
cristicalinyeah, I've just built a gnocchi v4 with ceph backend13:56
cristicalinso I would need to talk to that13:57
cristicalinok, lets see if I can get it to play nice with ocata13:57
gordcyou should be able to just grab the gnocchi stuff here https://github.com/openstack/ceilometer/tree/master/ceilometer/dispatcher to make mitaka useable with gnocchiv413:57
gordcalso, this https://github.com/openstack/ceilometer/blob/master/ceilometer/gnocchi_client.py13:58
gordcand this https://github.com/openstack/ceilometer/tree/master/ceilometer/dispatcher/data13:58
cristicalinhmm,  that's a big chunk of code13:59
gordcyeah... i'm not sure how nubeliu did it... i imagine they just swapped the newer code over.14:00
cristicalinany chance that was supported in newton ?14:01
gordcpossibly? but you're skipping newton anyways no? :P14:04
cristicalinyeah, its just that the ocata upgrade is not yet planned14:05
gordcthe dispatcher code is in theory the last piece of code in ceilometer before it hands off to something else so in theory it should be less issues.14:05
gordci added in theory a lot because i've never verified this.lol14:06
cristicalinOK, i'll dig into that backporting14:07
gordckk14:07
cristicalinmaybe I can get it to work on top of mitaka without moving everything else with it14:07
cristicalinwas the collector able to publish to gnocchi in mitaka ?14:11
cristicalinmaybe I can narrow down the piece of code to backport14:12
gordcthe collector by default will be what pushes to storage (until master)14:13
gordci think in eitehr newton or ocata, you can just use notification agent to push to minimise rabbitmq load.14:14
*** zhangguoqing has quit IRC14:14
*** jmlowe has joined #openstack-telemetry14:33
cristicalingordc, with the collector I got it to speak to gnocchi14:36
cristicalinbut now it complains that there is no specific resource_type14:36
gordccristicalin: awesome14:36
cristicalinis there any prepping that I need to do to make gnocchi know about these resource types ?14:36
gordcyou'll need to run ceilometer-upgrade --skip-metering-database14:36
cristicalinerm, there is no ceilometer-upgrade in mitaka14:37
gordcit will  build ceilomter resourece_types in gnocchi for you14:37
gordcoh... what'd you end up backporting?14:37
cristicalinnothing14:38
cristicalinI have mitaka ceilometer speaking to gnocchi v4.014:38
cristicalinmaybe I need to get that ceilometer-upgrade tool backported14:38
cristicalinbut the rest seems to work ... at least to the point that it talks to gnocchi over http :)14:38
gordcword? i feel like you're probably going to hit something further along... *shrugs*14:39
gordci don't know if gnocchiv4 can still create ceilometer resources.14:39
gordcyou can try running https://bugzilla.redhat.com/show_bug.cgi?id=134438014:40
openstackbugzilla.redhat.com bug 1344380 in openstack-tripleo-heat-templates "Run gnocchi-upgrade with '--create-legacy-resource-types' during deployment" [Urgent,Closed: errata] - Assigned to pkilambi14:40
cristicalinreally ? wasn't gnocchi borne to serve ceilometer ? I know now it's split from openstack but really ?14:40
gordccristicalin: we expect the producer to create resources. gnocchi is just the consumer.14:41
cristicalinok, so ceilometer should be the one creating the resources, that I understand14:41
cristicalinI just need to get it to create them14:41
cristicalinthere was a gnocchi_resources.yaml file in mitaka14:42
gordcif the legacy thing doesn't work, you'll need to backport ceilometer-upgrade and this https://github.com/openstack/ceilometer/blob/master/ceilometer/gnocchi_client.py#L42-L10014:42
cristicalinso I just need to find out how to use it to create the resources14:42
gordcor you can manually build them yourself.14:42
cristicalinI would go with the manual one-off tool14:42
gordchave a look at http://gnocchi.xyz/rest.html#resource-types14:43
gordcor if you want to use python client, the code upgrade code is https://github.com/openstack/ceilometer/blob/master/ceilometer/gnocchi_client.py#L150-L17414:44
*** links has quit IRC14:45
akrzosgordc and jd,how many tasks/metrics does a metric processor decide to do each wake up? (or does it now continous process then go onto a processing delay until new measures show up?)14:54
akrzos(gnocchi v4) ^^14:54
akrzosI'm running Gnocchi v4 right now and it's looking nicer than Gnocchi v3.1 with the mutli-sack and scheduler no longer there14:54
gordcakrzos: each worker is assigned a set of buckets and every metricd_processing_delay seconds it will scan through the sacks to see if there's work.14:56
gordcideally, it shouldn't have to scan and should be able to detect that some other way but that's what it is for now.14:56
akrzosso more workers than sacks would be basiclly useless?14:56
gordcakrzos: there is some overlap by default. so a given sack (by default) is handled by 3 workers14:57
gordcbut yes, in general, you shouldn't have more workers than sacks14:59
gordcyou should have significantly more sacks than workers in practice (i imagine)14:59
akrzosso if there is overlap is it a race who picks up all the work out of a sack?14:59
gordcakrzos: yep, one of the workers will lock the entire sack and it alone will work on it15:00
gordcakrzos: https://github.com/gnocchixyz/gnocchi/blob/c78fc744385ef1304a3bc766f094112642effc41/gnocchi/opts.py#L137-L143 that's the option regarding worker overlap15:01
*** rcernin has quit IRC15:09
*** yprokule has quit IRC15:10
*** yprokule_ has joined #openstack-telemetry15:11
*** yprokule_ has quit IRC15:11
*** yprokule_ has joined #openstack-telemetry15:11
*** yprokule_ is now known as yprokule15:12
cristicalingordc, quick and dirty but works like a charm15:19
cristicalinhttp://paste.openstack.org/show/613408/ - if anybody else needs it15:20
gordccristicalin: nice.15:27
cristicalinthanks for all the great help!15:32
*** jmlowe_ has joined #openstack-telemetry15:36
*** jmlowe has quit IRC15:38
gordcnot a problem.15:39
*** pcaruana has quit IRC15:42
*** yprokule has quit IRC15:48
*** yprokule has joined #openstack-telemetry15:48
*** cristicalin has quit IRC16:00
*** gongysh has quit IRC16:05
*** yprokule has quit IRC16:31
pradkhey gordc  ..any idea why i would get this when gnocchi api  http://paste.openstack.org/show/613414/ .. recently started showing up, so wonder if something changed in gnocchi16:40
*** lhx_ has quit IRC16:44
*** thorst has joined #openstack-telemetry16:44
gordcpradk: never seen that personally. something not timing out prooperly?16:46
*** thorst has quit IRC16:57
*** thorst has joined #openstack-telemetry16:57
*** hyakuhei has quit IRC17:00
*** hyakuhei has joined #openstack-telemetry17:00
*** hyakuhei has quit IRC17:00
*** hyakuhei has joined #openstack-telemetry17:00
*** thorst_ has joined #openstack-telemetry17:00
*** vint_bra has quit IRC17:02
*** thorst has quit IRC17:02
*** hughhalf has quit IRC17:05
*** thorst_ has quit IRC17:05
*** hughhalf has joined #openstack-telemetry17:06
*** vint_bra has joined #openstack-telemetry17:08
*** thorst has joined #openstack-telemetry17:20
pradkgordc, yea not sure.. first time i'm seeing this as well17:21
pradkgordc, here is the full wsgi error log http://chunk.io/f/628e71b749d04fc8b0205cb8700d6a7217:22
*** thorst has quit IRC17:24
gordcpradk: but it works sometimse?17:41
pradkwell i just ran into it today17:42
gordcin that error log, does it actually accept requests at some points or it's just 40mins of constant errors?17:43
pradkgordc, in the error log its basically what i pasted .. in access log i see some 200's17:46
*** tesseract has quit IRC17:59
gordcword? yeah i have no idea.18:00
*** thorst has joined #openstack-telemetry18:23
*** thorst has quit IRC18:27
*** thorst has joined #openstack-telemetry18:35
*** thorst has quit IRC18:40
*** thorst has joined #openstack-telemetry18:50
*** thorst has quit IRC18:54
openstackgerritgordon chung proposed openstack/ceilometer master: expect oslo.messaging messages  https://review.openstack.org/47666419:17
*** rbak_ has joined #openstack-telemetry19:33
*** rbak has quit IRC19:34
*** thorst has joined #openstack-telemetry19:40
*** thorst has quit IRC19:44
*** thorst has joined #openstack-telemetry19:46
*** yassine has quit IRC19:50
*** thorst has quit IRC19:50
*** yassine has joined #openstack-telemetry19:56
openstackgerritMerged openstack/ceilometer master: post_test_hook: stop sourcing gnocchi/devstack/settings  https://review.openstack.org/47612320:28
*** rwsu has quit IRC20:28
*** jmlowe_ has quit IRC20:41
*** thorst has joined #openstack-telemetry21:04
*** vint_bra has quit IRC21:13
*** catintheroof has quit IRC21:18
*** vint_bra has joined #openstack-telemetry21:32
openstackgerritgordon chung proposed openstack/ceilometer master: stop converting to old message format  https://review.openstack.org/47671521:39
*** thorst has quit IRC21:47
*** chlong has quit IRC21:57
*** dave-mccowan has quit IRC22:15
*** vint_bra has quit IRC22:20
*** thorst has joined #openstack-telemetry22:52
*** gordc has quit IRC22:57
*** thorst has quit IRC23:09
*** chlong has joined #openstack-telemetry23:10
*** gongysh has joined #openstack-telemetry23:31
*** gongysh has quit IRC23:33
*** gongysh has joined #openstack-telemetry23:37
*** gongysh has quit IRC23:38
*** aignatov has quit IRC23:41
*** aignatov has joined #openstack-telemetry23:43

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!