Wednesday, 2017-10-04

*** lnxnut has joined #openstack-telemetry00:11
*** lnxnut has quit IRC00:21
*** itlinux has joined #openstack-telemetry00:36
*** itlinux has quit IRC00:58
*** vint_bra has quit IRC01:07
*** vint_bra has joined #openstack-telemetry01:07
*** lnxnut has joined #openstack-telemetry01:18
*** aagate has quit IRC01:23
*** Kevin_Zheng has joined #openstack-telemetry01:27
*** lnxnut has quit IRC01:29
*** itlinux has joined #openstack-telemetry01:36
*** vint_bra has quit IRC01:40
*** itlinux has quit IRC01:42
*** Tom has joined #openstack-telemetry01:43
*** tovin07_ has joined #openstack-telemetry01:51
*** catintheroof has joined #openstack-telemetry02:03
*** Tommy-Tom has joined #openstack-telemetry02:10
*** catintheroof has quit IRC02:19
*** lnxnut has joined #openstack-telemetry02:26
*** catintheroof has joined #openstack-telemetry02:29
*** lnxnut has quit IRC02:36
*** Tommy-Tom has quit IRC02:41
*** Tommy-Tom has joined #openstack-telemetry02:42
*** Tommy-Tom has quit IRC02:43
*** Tommy-Tom has joined #openstack-telemetry02:43
*** Tommy-Tom has quit IRC02:48
*** Tommy-Tom has joined #openstack-telemetry02:49
*** Tommy-Tom has quit IRC02:53
*** Tommy-Tom has joined #openstack-telemetry02:57
*** Tommy-Tom has quit IRC03:04
*** Tommy-Tom has joined #openstack-telemetry03:05
*** catintheroof has quit IRC03:09
*** Tommy-Tom has quit IRC03:10
*** Tommy-Tom has joined #openstack-telemetry03:11
*** Tommy-Tom has quit IRC03:14
*** Tommy-Tom has joined #openstack-telemetry03:14
*** Tommy-Tom has quit IRC03:16
*** Tommy-Tom has joined #openstack-telemetry03:16
*** Tommy-Tom has joined #openstack-telemetry03:17
*** Tommy-Tom has quit IRC03:21
*** itlinux has joined #openstack-telemetry03:23
*** itlinux has quit IRC03:31
*** itlinux has joined #openstack-telemetry03:32
*** lnxnut has joined #openstack-telemetry03:34
*** Kevin_Zheng has quit IRC03:37
*** gkadam has joined #openstack-telemetry03:39
*** itlinux has quit IRC03:41
*** lnxnut has quit IRC03:44
*** boris_42_ has quit IRC03:44
*** lnxnut has joined #openstack-telemetry04:40
*** lnxnut has quit IRC05:06
*** psachin has joined #openstack-telemetry05:15
*** lnxnut has joined #openstack-telemetry05:33
*** lnxnut has quit IRC05:37
*** f0o has quit IRC05:48
*** akrzos has quit IRC05:54
*** jmlowe has quit IRC05:55
*** jmlowe has joined #openstack-telemetry05:55
*** akrzos has joined #openstack-telemetry06:22
*** Tommy-Tom has joined #openstack-telemetry06:25
*** rcernin has joined #openstack-telemetry06:30
*** yprokule has joined #openstack-telemetry06:31
*** edmondsw has joined #openstack-telemetry06:33
*** lnxnut has joined #openstack-telemetry06:34
*** edmondsw has quit IRC06:37
*** Tommy-Tom has quit IRC06:43
*** Tommy-Tom has joined #openstack-telemetry06:43
*** Tommy-Tom has quit IRC06:45
*** Tommy-Tom has joined #openstack-telemetry06:45
*** pcaruana has joined #openstack-telemetry06:48
openstackgerritJulien Danjou proposed openstack/ceilometer master: Replace the Gnocchi dispatcher by a publisher  https://review.openstack.org/50058606:49
openstackgerritJulien Danjou proposed openstack/ceilometer master: Remove deprecated pollster-list option  https://review.openstack.org/50718306:51
jd_sileht: http://logs.openstack.org/32/508132/1/check/gate-ceilometer-dsvm-tempest-plugin-mongodb-ubuntu-xenial/23f734d/logs/devstacklog.txt.gz#_2017-10-03_19_37_12_17806:53
jd_wtf?06:53
silehtjd_, zuul v3 !06:54
*** spectr has quit IRC06:54
*** spectr has joined #openstack-telemetry06:55
jd_sileht: seriously?06:55
jd_should we do something?06:55
jd_except, resigning06:55
silehtjd_, I think most gates are dead except nova :p06:56
jd_sigh06:56
silehtjd_, just recheck06:57
*** lnxnut has quit IRC06:57
silehtjd_, the fix is already merged in devstack06:57
jd_ok06:57
silehthttps://review.openstack.org/#/c/509307/06:57
*** shardy_afk is now known as shardy07:01
*** Tommy-Tom has quit IRC07:08
silehtjd_, just read that on ML:07:08
silehtAs a result, Zuul v3 submitted "SUCCESS" to gerrit when gerrit event occurred.07:08
silehtBut, error logs were generated. And my job was not run.07:09
*** Tommy-Tom has joined #openstack-telemetry07:09
sileht#fear!07:09
silehtjd_, lol: "With that in mind, we are going to perform a partial rollback to Zuul v2"07:10
jd_omg07:11
silehtjd_, we propose pastamaker perhaps ?07:12
jd_sileht: <307:12
*** nrado has joined #openstack-telemetry07:14
*** tesseract has joined #openstack-telemetry07:16
openstackgerritVu Cong Tuan proposed openstack/ceilometer master: Replace the usage of some aliases in tempest  https://review.openstack.org/50936007:17
*** Tommy-Tom has quit IRC07:28
*** Tommy-Tom has joined #openstack-telemetry07:28
nradoHi, can someone help me please with ceilometer/gnocchi/panko? I noticed that some of the events are not stored in the mysql database, but I can't find any error messages in the services' logs. Found some warnings (CAST problems in panko/mysql and ceilometer collector warnings (Connection pool is full, discarding connection))07:32
nradoI'm using the services in the Ocata release. Is this a known issue?07:32
jd_hi nrado07:32
jd_there is no known issue07:33
*** Tommy-Tom has quit IRC07:33
jd_can you be more explicit and precise about your problem?07:33
nradoyes of course, I can give you some examples:07:36
*** Gautam_ has joined #openstack-telemetry07:44
nradohttps://gist.github.com/neki06/6212677c9bcba0b6eba6d30c50bdc1f507:48
nradoI created three volumes a few minutes ago. And the last thing I can see when I execute "openstack event list" is a create.start event from yesterday07:48
nradoThis is also a problem. Sometimes I just see a create.start without a create.END. Or I even see no event produced07:50
nradoIt's not just the case with the volumes, it was just an example07:50
nradoWhat's also interesting: when I take a look into the panko database in mysql, I see that the ID of the events is always increased by three:07:52
nrado| 1941 |07:52
nrado| 1944 |07:52
nrado| 1947 |07:52
nrado| 1950 |07:52
nrado| 1953 |07:52
nradoIs this normal?07:52
*** f0o|away has joined #openstack-telemetry07:54
nradojd_: ?08:00
*** f0o|away is now known as f0o08:00
jd_nrado: I barely know anything about Panko to be hnoest08:01
jd_nrado: but looks like Ceilometer does not get all the events, so maybe it has a config issue or Cinder is not sending all notifications08:01
jd_are all your cinder instances correctly configured to send notif?08:02
nradoall cinder-services have this in their configs:08:03
nrado[oslo_messaging_notifications]08:03
nradodriver = messagingv208:03
nradotopics = notifications08:03
nradojd_: Do you know the reason behind this message perhaps?:08:06
nrado==> /var/lib/docker/volumes/kolla_logs/_data/ceilometer/ceilometer-collector.log <==08:06
nrado2017-10-04 08:05:02.065 21 WARNING requests.packages.urllib3.connectionpool [-] Connection pool is full, discarding connection: 10.0.1.408:06
jd_nrado: your connection pool is full08:06
jd_:)08:06
nrado:)08:07
jd_You can increase max_parallel_requests08:07
nradook, I'll give it a try. Perhaps I can fix this warning after all :D08:09
nradoBut this is also the right chat when I have problems with Panko, right?08:10
*** Tommy-Tom has joined #openstack-telemetry08:10
*** Tommy-Tom has quit IRC08:21
*** Tommy-Tom has joined #openstack-telemetry08:21
*** edmondsw has joined #openstack-telemetry08:21
*** Gautam_ has quit IRC08:24
*** edmondsw has quit IRC08:26
*** lnxnut has joined #openstack-telemetry08:28
*** Tommy-Tom has quit IRC08:41
*** Tommy-Tom has joined #openstack-telemetry08:41
*** Tommy-Tom has quit IRC08:46
*** lnxnut has quit IRC08:47
*** yprokule has quit IRC08:52
*** yprokule_ has joined #openstack-telemetry08:52
*** yprokule_ is now known as yprokule08:53
openstackgerritJulien Danjou proposed openstack/ceilometer master: Replace the Gnocchi dispatcher by a publisher  https://review.openstack.org/50058609:16
openstackgerritNam Nguyen Hoai proposed openstack/ceilometer master: [WIP] Migrate to Zuul v3  https://review.openstack.org/50938509:35
*** lnxnut has joined #openstack-telemetry09:45
*** lnxnut has quit IRC09:56
*** tovin07_ has quit IRC10:06
*** edmondsw has joined #openstack-telemetry10:09
*** edmondsw has quit IRC10:13
*** tesseract has quit IRC10:40
silehtjd_, this change is a joke ^10:41
*** tesseract has joined #openstack-telemetry10:44
silehtzuulv2 is also broken now10:46
silehtthe rollback was not a rollback...10:46
*** lnxnut has joined #openstack-telemetry10:53
*** spectr-RH has joined #openstack-telemetry10:59
*** dave-mccowan has joined #openstack-telemetry11:01
*** spectr has quit IRC11:02
*** lnxnut has quit IRC11:03
*** dave-mcc_ has joined #openstack-telemetry11:10
*** shardy is now known as shardy_afk11:12
*** dave-mccowan has quit IRC11:13
*** lnxnut has joined #openstack-telemetry12:00
jd_baby rollback tudoudoudou12:00
*** Tom has quit IRC12:08
*** lnxnut has quit IRC12:11
*** Tom_____ has joined #openstack-telemetry12:14
*** edmondsw has joined #openstack-telemetry12:16
*** edmondsw_ has joined #openstack-telemetry12:17
*** nrado1 has joined #openstack-telemetry12:18
*** nrado has quit IRC12:21
*** edmondsw has quit IRC12:21
*** gordc has joined #openstack-telemetry12:30
*** gkadam has quit IRC12:50
*** spectr-RH has quit IRC13:00
*** pcaruana has quit IRC13:02
*** lnxnut has joined #openstack-telemetry13:08
*** spectr-RH has joined #openstack-telemetry13:13
*** leitan has joined #openstack-telemetry13:14
*** pcaruana has joined #openstack-telemetry13:15
*** spectr-RH has quit IRC13:16
*** yprokule has quit IRC13:16
*** lnxnut has quit IRC13:17
*** spectr-RH has joined #openstack-telemetry13:17
*** chlong_ has joined #openstack-telemetry13:22
*** lnxnut has joined #openstack-telemetry13:23
*** catintheroof has joined #openstack-telemetry13:24
*** pradk has quit IRC13:26
*** alexchadin has joined #openstack-telemetry13:30
*** shardy_afk is now known as shardy13:42
*** edmondsw_ is now known as edmondsw14:07
*** vint_bra has joined #openstack-telemetry14:09
*** pradk has joined #openstack-telemetry14:12
*** maestropandy has joined #openstack-telemetry14:25
maestropandyHi all14:25
maestropandydevstack error "No matching distribution found for gnocchiclient>=3.1.0 (from aodh==5.0.1.dev11)" http://paste.openstack.org/show/622667/14:26
jd_maestropandy: https://pypi.python.org/pypi/gnocchiclient it exists14:28
jd_maestropandy: likely global requirement stuff blocked it14:28
jd_have fun with that14:28
jd_or   Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7feba34ad790>: Failed to establish a new connection: [Errno 111] Connection refused',)': /simple/gnocchiclient/14:28
jd_internet is down14:28
maestropandyjd_ internet is working fine, I can able to download package wise, but getting conflict with gnocchi client 4.0 with aodh 5.0.1..14:29
jd_maestropandy: well pip does not seem to agree with you14:30
*** pcaruana has quit IRC14:33
maestropandyjd_: u are right, my proxy doesn't allowing to download few packages, i can download those using root, but for the above error i have done " pip install gnocchiclient" after cloning again.. getting error as "No matching distribution found for gnocchiclient>=4.0.0 (from aodh==5.0.1.dev11)".. so which version of gnocchiclient and aodh need to install .. pls suggest14:33
jd_gnocchiclient>=3.1.014:33
maestropandyhow about aodh version ? for correponding "gnocchiclient>=3.1.0" ?..14:34
jd_you're trying to install master no?14:35
maestropandyjd_:  yes doing master14:36
jd_so master aodh wants gnocchiclient>=3.1.014:36
*** spectr-RH has quit IRC14:41
*** spectr has joined #openstack-telemetry14:42
*** aagate has joined #openstack-telemetry14:44
*** pcaruana has joined #openstack-telemetry14:45
*** maestropandy has quit IRC14:48
*** alexchadin has quit IRC14:50
*** lnxnut_ has joined #openstack-telemetry14:56
*** lnxnut has quit IRC14:58
*** rcernin has quit IRC15:13
*** joadavis has quit IRC15:22
*** donghao has joined #openstack-telemetry15:22
*** psachin has quit IRC15:25
*** pcaruana has quit IRC15:28
*** tesseract has quit IRC15:29
*** joadavis has joined #openstack-telemetry15:35
*** nrado1 has quit IRC15:48
*** larainema has quit IRC16:32
*** shardy has quit IRC16:47
*** rwsu has quit IRC17:03
*** donghao has quit IRC17:05
*** pradk has quit IRC17:22
*** pradk has joined #openstack-telemetry17:38
*** Tommy-Tom has joined #openstack-telemetry17:42
*** pradk has quit IRC17:46
*** pradk has joined #openstack-telemetry17:46
*** Tommy-Tom has quit IRC17:47
*** itlinux has joined #openstack-telemetry17:49
*** itlinux has quit IRC17:53
*** nrado has joined #openstack-telemetry18:04
*** itlinux has joined #openstack-telemetry18:04
openstackgerritOpenStack Proposal Bot proposed openstack/panko master: Imported Translations from Zanata  https://review.openstack.org/50957718:08
*** itlinux has quit IRC18:12
*** pradk has quit IRC18:16
*** edmondsw has quit IRC18:20
*** edmondsw has joined #openstack-telemetry18:23
*** edmondsw has quit IRC18:28
*** edmondsw has joined #openstack-telemetry18:48
*** edmondsw has quit IRC19:06
*** edmondsw has joined #openstack-telemetry19:06
*** chlong_ has quit IRC19:16
openstackgerritgordon chung proposed openstack/ceilometer master: remove ironic exchange  https://review.openstack.org/50960219:27
nradoHi, is ceilometer-collector needed when deploying telemetry with gnocchi and panko in the Ocata release?19:30
gordcnrado: nope. you just need to change your publisher right target19:31
gordcchange it from notifier:// to gnocchi://19:32
nradoin the pipeline.yaml?19:34
gordcpanko is a little diff in ocata, you'll need to put direct://?dispatcher=panko if using ocata19:35
gordcnrado: yep, pipeline.yaml19:36
gordcand event_pipeline.yaml19:36
gordcpanko is only applicable for event_pipline19:36
nradoAnd where do I have to insert dispatcher=panko? in ceilometer.conf?19:41
gordcnrado: event_pipeline.yaml19:41
gordcnrado: like this https://github.com/openstack/ceilometer/blob/master/ceilometer/pipeline/data/pipeline.yaml19:42
gordcand https://github.com/openstack/ceilometer/blob/master/ceilometer/pipeline/data/event_pipeline.yaml19:42
gordcexcept in event_pipeline.yaml you should add direct://?dispatcher=panko in addition to existing gnocchi19:43
gordcor if you have pike panko, use panko://19:43
openstackgerritgordon chung proposed openstack/ceilometer master: common endpoint  https://review.openstack.org/50961219:48
nradoIt's really confusing for me gordc, sorry.19:50
nradoYou know, my main problem is that I'm not getting all of the events I expect to see. When I create a volume for example, I either get a create.start or a create.end, but never both19:50
gordcnrado: weird. in panko?19:51
nradoyes… started to debug my rabbitmq cluster19:52
nradoand the problem is still there19:52
nradoI can see the notifiy exchanges in the queue19:52
nradobut at the end, I just got a part of the event in the mysql19:52
nradoI have been struggling with this problem since Monday19:53
nradognocchi and metrics are not so important for me, I just want to see the event, not more :D19:53
gordcyeah, working on enabling just samples or just event. seems everyone wants one or the other19:54
gordcdo you see errors? it's strange you only get start and end but never together19:55
gordcif it was just start or just end, that would make more sense19:55
nradoyou what is more weird :D sometimes I see just the end, but not the start19:57
nradoit's chaning from time to time19:57
nradoit's really strange19:57
nradoI don't see any error messages19:57
gordc:(19:58
nradoI can show you my cinder and ceilo/panko loga when creating a volume19:58
nradogive me a second19:58
gordcmy only guess would be that somehting else is grabbing from same queue19:58
gordcso ceilometer only gets some of the messages depending on who reads first19:58
nradohttps://gist.github.com/neki06/c99d193573fd767a05243aa95426192820:00
nradoI'm missing in the ceilo logs a 2nd message received with the ID: 1c00a4b97baf42db89baaca690db0b9e20:02
nradothis COULD be the missing end event20:02
nradoI'm not sure20:02
nradoBut this is all I get when creating a volume20:02
gordcmaybe shut off notificaiton agent and see if there's anything else consuming from cinder queue?20:03
nradohmm, why cinder queue? I thought it is the queue "notifications.info" which is important for the events?20:14
openstackgerritOpenStack Proposal Bot proposed openstack/aodh master: Imported Translations from Zanata  https://review.openstack.org/50962820:20
gordcnrado: right. well i mean whatever queue you have cinder pushing notificaitons too20:20
gordcbut you're right, ceilometer by default will normalise stuff it gets off notificaitons.info and notifications.error queues20:20
nradoI can see two consumers, when I stop ceilometer_notification there is one left20:37
gordci think you need to figure out what that other one is. that's probably grabbing all the message you're missing20:40
nradoit's something with the port 55642. Would be nice if I would know how to kick the consumer just out of this queue :D20:41
*** yassine has quit IRC20:44
nradoThank you very much, gordc! I'll continue tomorrow20:46
nradoOh20:47
nradoI've found the consumer…20:47
nradoIt was barbican-keystone-listenter20:47
nradooh man, crazy20:48
gordcah. yeah, i have no idea what services are grabbing off the queue20:48
gordci imagine you'd need to configure keystone to publish to another topic if you want barbican service running20:49
gordcs/another/multiple/20:49
nradoyeah, now I'm getting my events20:50
nradooh man, I have to laugh :D20:50
gordcbeats crying20:51
*** nrado has quit IRC21:01
*** leitan has quit IRC21:01
*** rwsu has joined #openstack-telemetry21:02
*** donghao has joined #openstack-telemetry21:14
openstackgerritgordon chung proposed openstack/ceilometer master: remove ironic exchange  https://review.openstack.org/50960221:25
openstackgerritgordon chung proposed openstack/ceilometer master: common notification endpoint  https://review.openstack.org/50961221:25
openstackgerritgordon chung proposed openstack/ceilometer master: remove unused ExchangeTopics  https://review.openstack.org/50964221:25
*** edmondsw has quit IRC21:32
*** edmondsw has joined #openstack-telemetry21:32
openstackgerritgordon chung proposed openstack/ceilometer master: rename sample handler  https://review.openstack.org/50964821:36
*** edmondsw has quit IRC21:37
*** catintheroof has quit IRC21:41
openstackgerritMerged openstack/panko master: Imported Translations from Zanata  https://review.openstack.org/50957721:43
*** edmondsw has joined #openstack-telemetry22:41
*** edmondsw has quit IRC22:46
openstackgerritgordon chung proposed openstack/ceilometer master: common notification endpoint  https://review.openstack.org/50961222:57
openstackgerritgordon chung proposed openstack/ceilometer master: rename sample handler  https://review.openstack.org/50964822:57
openstackgerritgordon chung proposed openstack/ceilometer master: move listener targets to notification agent  https://review.openstack.org/50966822:57
*** edmondsw has joined #openstack-telemetry23:13
*** vint_bra has quit IRC23:17
*** vint_bra has joined #openstack-telemetry23:17
*** vint_bra has quit IRC23:21
*** edmondsw has quit IRC23:35
*** gordc has quit IRC23:43

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!