Wednesday, 2017-11-08

*** AlexeyAbashkin has quit IRC00:03
*** hieulq has quit IRC00:22
*** hieulq has joined #openstack-performance00:22
*** markvoelker has joined #openstack-performance00:26
*** markvoelker has quit IRC00:35
*** markvoelker has joined #openstack-performance00:40
*** markvoelker_ has joined #openstack-performance00:42
*** markvoelker has quit IRC00:44
*** masber has joined #openstack-performance00:48
*** markvoelker_ has quit IRC00:49
*** markvoelker has joined #openstack-performance00:58
*** markvoelker has quit IRC01:09
*** jkilpatr has quit IRC02:21
*** openstackgerrit has joined #openstack-performance02:30
openstackgerritMerged openstack/osprofiler master: Extend messaging driver to support reporting  https://review.openstack.org/51435902:30
openstackgerritMerged openstack/osprofiler master: Handle and report SQLAlchemy errors  https://review.openstack.org/51458702:55
*** klindgren has joined #openstack-performance02:58
*** AlexeyAbashkin has joined #openstack-performance03:01
*** AlexeyAbashkin has quit IRC03:05
*** klindgren has quit IRC03:33
*** klindgren has joined #openstack-performance03:42
*** aojea has joined #openstack-performance03:53
*** aojea has quit IRC03:57
*** klindgren has quit IRC04:34
*** klindgren has joined #openstack-performance04:59
*** klindgren has quit IRC05:21
*** markvoelker has joined #openstack-performance06:42
*** markvoelker_ has joined #openstack-performance06:43
*** markvoelker_ has quit IRC06:45
*** markvoelker has quit IRC06:46
*** brault has joined #openstack-performance06:47
*** markvoelker has joined #openstack-performance06:47
*** markvoelker has quit IRC06:52
*** markvoelker has joined #openstack-performance06:54
*** aojea has joined #openstack-performance06:58
*** aojea has quit IRC08:02
openstackgerritIlya Shakhat proposed openstack/osprofiler master: Add Zuul job for functional testing  https://review.openstack.org/51510808:06
*** aojea has joined #openstack-performance08:31
*** klindgren has joined #openstack-performance08:37
*** klindgren has quit IRC08:43
*** linpopilan has joined #openstack-performance08:44
*** AlexeyAbashkin has joined #openstack-performance08:59
*** ldesimone has joined #openstack-performance09:10
ldesimonehi all, I'm starting enabling osprofiler on devstack testbed...but something wrong. I've already set the local.conf with enable_plugin panko, enable_plugin ceilometer, enable_plugin osprofiler, pointing on stable/pike. Furthermore, I've used mongodb as backend (CEILOMETER_BACKEND=mongodb). The strange thing is that during a profiling, only glance api calls are traces!!! Any suggestions?09:14
*** akrzos has quit IRC09:43
*** Aju has quit IRC09:43
*** serverascode has quit IRC09:43
*** Guest66873 has joined #openstack-performance09:46
*** akrzos has joined #openstack-performance09:49
*** Aju has joined #openstack-performance09:49
*** serverascode has joined #openstack-performance09:49
*** serverascode has quit IRC09:50
*** ldesimon_ has joined #openstack-performance09:50
*** ldesimone has quit IRC09:50
*** serverascode has joined #openstack-performance09:51
*** ldesimone has joined #openstack-performance09:53
*** ldesimon_ has quit IRC09:53
*** Guest66873 has quit IRC10:05
*** ldesimon_ has joined #openstack-performance10:15
*** ldesimone has quit IRC10:15
ilyashakhatldesimon_, what OpenStack command do you run?10:17
ldesimon_Hi @ilyashakhat, the command is:  openstack --os-profile SECRET_KEY server create --flavor m1.tiny --image cirros-0.3.5-x86_64-disk PROVA110:21
ldesimon_then, to show the trace: osprofiler trace show --html 8448886a-6bef-45fa-a471-dafc888fb0d610:21
ldesimon_but by simply grep the nova keyword, there is no trace for nova10:22
ldesimon_maybe is the wrong command?10:22
ilyashakhatplease verify that profiling is enabled in Nova10:26
ilyashakhatconnection_string = messaging://10:26
ilyashakhathmac_keys = swordfish,SECRET_KEY10:26
ilyashakhattrace_sqlalchemy = True10:26
ilyashakhatenabled = True10:26
ilyashakhatin /etc/nova/nova.conf10:26
ilyashakhatand for Neutron you will need to add "osprofiler" into pipelines in /etc/neutron/api-paste.ini10:29
ilyashakhate.g. keystone = cors http_proxy_to_wsgi request_id osprofiler catch_errors authtoken keystonecontext extensions neutronapiapp_v2_010:29
ilyashakhatldesimon_ ^^^10:30
ldesimon_the following is in my /etc/nova/nova.conf:10:30
ldesimon_[profiler]10:30
ldesimon_connection_string = messaging://10:30
ldesimon_hmac_keys = SECRET_KEY10:30
ldesimon_trace_sqlalchemy = True10:30
ldesimon_enabled = True10:30
ilyashakhatlooks correct10:31
ldesimon_maybe is missing into the pipeline for api-paste.ini for nova component10:31
ilyashakhatdo you have any traces when executing "openstack --os-profile SECRET_KEY flavor list"?10:31
ldesimon_stack@ubuntu:~$ osprofiler trace show --html d1117578-9469-4ff0-b771-837819ae5a5710:33
ldesimon_Error occurred while connecting to Ceilometer: Authentication failed. Missing options: token, endpoint, tenant_id, tenant_name.10:33
ldesimon_stack@ubuntu:~$ osprofiler --os-tenant-name admin trace show --html d1117578-9469-4ff0-b771-837819ae5a5710:33
ldesimon_Trace with UUID d1117578-9469-4ff0-b771-837819ae5a57 not found. Please check the HMAC key used in the command.10:33
*** arcolife has joined #openstack-performance10:33
ldesimon_that is for "openstack --os-profile SECRET_KEY flavor list" in my env10:33
ldesimon_it is strange...10:33
ilyashakhatit complains about missing auth params...10:33
ldesimon_but I've sourced the admin.sh...10:34
ilyashakhatthe problem is that Ceilometer driver is a bit outdated.... not sure whether it works or not...10:34
ldesimon_an alternative to ceilometer? what I have to do to have a properly tracing within devstack? any suggestions?10:35
ilyashakhatoption 1) is to continue using RMQ driver for notification, disable Ceilometer and use osprofiler ability to collect traces10:37
ilyashakhatbut this requires patch https://review.openstack.org/#/c/514359/ which was merged this morning into master10:37
ilyashakhatoption 2) is to use database driver10:38
ilyashakhatfor example elasticsearch10:38
ilyashakhatit can be installed directly to DevBox VM, and profiler connection_string changed to elasticsearch://<host:port>10:40
ilyashakhataah.. you probably already have MongoDB for Ceilometer, right?10:41
ilyashakhatthen you can use mongodb driver10:41
ldesimon_yes...I've installed devstack with this conf:10:46
ldesimon_CEILOMETER_BACKEND=mongodb10:46
ldesimon_enable_plugin panko https://git.openstack.org/openstack/panko stable/pike10:46
ldesimon_enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer stable/pike10:46
ldesimon_enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler stable/pike10:46
ldesimon_but osprofiler is not tightly linked to ceilometer?10:46
ilyashakhatnot linked... Ceilometer was used initially at it provided an easy way to collect events from profiler queue10:47
ldesimon_Ah ok, so can be bypassed easily with your 1st option10:49
ldesimon_because my need is actually understand all the communication intra- and inter- services10:49
ilyashakhatyes, but 1) requires master, and you have pike10:49
ilyashakhatso I would go for 2) with mongodb driver10:50
ldesimon_mmm...I can rebuild, it is not a problem10:50
*** aojea has quit IRC10:50
ilyashakhatit's only change of connection_string parameter in all services10:50
ilyashakhatshould be much faster :)10:50
ldesimon_mmm...so in order to use mongodb connection, it is wrong to have such conf?:10:51
ldesimon_[profiler]10:51
ldesimon_connection_string = messaging://10:51
ldesimon_hmac_keys = SECRET_KEY10:51
ldesimon_trace_sqlalchemy = True10:51
ldesimon_enabled = True10:51
ilyashakhatwith this config RMQ driver will be used10:52
*** AlexeyAbashkin has quit IRC10:53
ilyashakhatyou need to change connection_string =10:53
ldesimon_like this: connection_string = mongodb:// ?10:53
ilyashakhatmongodb://localhost:<PORT>/10:54
ilyashakhatit can be copied from ceilometer conf (I guess)10:55
ilyashakhatin case there's a special user with password10:56
ldesimon_[database]10:56
ldesimon_metering_connection = mongodb://localhost:27017/ceilometer10:56
ldesimon_in ceilometer I've this one:10:56
ilyashakhatmongodb://localhost:27017/profiler10:57
ldesimon_ok....another doubt is about /etc/nova/api-paste.ini11:00
ldesimon_[composite:osapi_compute]11:00
ldesimon_use = call:nova.api.openstack.urlmap:urlmap_factory11:00
ldesimon_/: oscomputeversions11:00
ldesimon_/v2: openstack_compute_api_v21_legacy_v2_compatible11:00
ldesimon_/v2.1: openstack_compute_api_v2111:00
ldesimon_[composite:openstack_compute_api_v21]11:00
ldesimon_use = call:nova.api.auth:pipeline_factory_v2111:00
ldesimon_noauth2 = cors http_proxy_to_wsgi compute_req_id faultwrap request_log sizelimit osprofiler noauth2 osapi_compute_app_v2111:00
ldesimon_keystone = cors http_proxy_to_wsgi compute_req_id faultwrap request_log sizelimit osprofiler authtoken keystonecontext osapi_compute_app_v2111:00
ldesimon_[composite:openstack_compute_api_v21_legacy_v2_compatible]11:00
ldesimon_use = call:nova.api.auth:pipeline_factory_v2111:00
ldesimon_noauth2 = cors http_proxy_to_wsgi compute_req_id faultwrap request_log sizelimit osprofiler noauth2 legacy_v2_compatible osapi_compute_app_v2111:00
ldesimon_keystone = cors http_proxy_to_wsgi compute_req_id faultwrap request_log sizelimit osprofiler authtoken keystonecontext legacy_v2_compatible osapi_compute_app_v2111:00
ldesimon_I've noticed that for [composite:osapi_compute] there is no "osprofiler" field....is it normal?11:00
ilyashakhatit should be enough to have it in [composite:openstack_compute_api_v21] / keystone11:02
ldesimon_I've tried now within the modification about mongodb connection in nova conf, and the trace include in addition to glance also keystone traces....O.o11:06
ldesimon_the command is: openstack --os-profile SECRET_KEY server create --flavor m1.tiny --image cirros-0.3.5-x86_64-disk PROVA211:07
ilyashakhatthat's better :)11:09
ilyashakhatcould you check Nova logs -- does it tell that profiler is enabled?11:09
ilyashakhatthe message should be close to service start11:10
*** aojea has joined #openstack-performance11:12
ldesimon_i've restarted nova services...and I see something related to osprofiler11:16
ldesimon_but I can't state if it is working or not...and I cant paste here all the log file :D11:17
ldesimon_there is some way to send you the log?11:17
ilyashakhathttp://paste.openstack.org/11:17
ldesimon_greate....http://paste.openstack.org/show/625821/11:17
ldesimon_for my knowledge seems that for some component the connection string is correct11:18
ilyashakhatstrange that it differs... nova-compute has an old one11:20
ilyashakhatand there's no nova-api...11:20
ilyashakhathave you restarted devstack@n-api ?11:20
ldesimon_I guess yes, the command launched is => systemctl restart devstack@n*11:21
ilyashakhatlet me check on my devstack...11:25
ilyashakhat... and I don't have anything from Nova too11:28
ldesimon_can you show me the output for this command in your env?11:29
ldesimon_grep -rnw /etc/ -e "connection_string"11:29
ilyashakhathttp://paste.openstack.org/show/625824/ but it's configured semi-manually11:32
ldesimon_so by default you don't use any db11:33
ilyashakhatthe problem with Nova API is that now it is running under uwsgi, and most probably uses different settings for paste.ini11:33
ilyashakhatyes, but I have special reader from RMQ to collect traces (that's why I'm not using DB)11:34
ldesimon_Can I enable the same for your env? For me is not a constraint to have a DB11:36
*** AlexeyAbashkin has joined #openstack-performance11:38
ilyashakhatwith DB it's easier...11:40
ilyashakhatthe problem is in nova-api service11:40
ilyashakhatI'm looking into it11:40
*** ldesimone has joined #openstack-performance11:41
*** ldesimon_ has quit IRC11:41
*** ldesimone has quit IRC11:43
*** ldesimone has joined #openstack-performance11:43
*** ldesimon_ has joined #openstack-performance11:52
*** ldesimone has quit IRC11:52
*** ldesimone has joined #openstack-performance11:59
*** ldesimon_ has quit IRC11:59
ldesimone@ilyashakhat, other doubts are about the proper config for /etc/ceilometer/event_pipeline.yaml and /etc/ceilometer/ceilometer.conf12:02
ldesimone[database]12:03
ldesimonemetering_connection = mongodb://localhost:27017/ceilometer12:03
ldesimonefor ceilometer.conf I have:12:03
ldesimoneand for event_pipeline.yaml:12:03
ldesimone---12:03
ldesimonesources:12:03
ldesimone    - name: event_source12:03
ldesimone      events:12:03
ldesimone          - "*"12:03
ldesimone      sinks:12:03
ldesimone          - event_sink12:03
ldesimonesinks:12:03
ldesimone    - name: event_sink12:03
ldesimone      transformers:12:03
ldesimone      publishers:12:03
ldesimone          - database://12:03
ldesimone          - panko://12:03
ldesimone~12:03
*** aojea has quit IRC12:05
*** ldesimon_ has joined #openstack-performance12:15
*** ldesimone has quit IRC12:15
ilyashakhati'm not an expert in this... but after switching of osprofiler driver ceilometer is not used anymore12:21
*** catintheroof has joined #openstack-performance12:27
*** aojea has joined #openstack-performance12:28
*** jkilpatr has joined #openstack-performance12:32
*** catintheroof has quit IRC12:33
*** catintheroof has joined #openstack-performance12:40
*** brault has quit IRC12:40
*** brault has joined #openstack-performance12:41
*** catintheroof has quit IRC12:43
*** markvoelker has quit IRC12:50
*** brault has quit IRC13:03
ldesimon_@ilyashakhat, I've rebuild devstack env with this specific local.conf13:07
ldesimon_CEILOMETER_BACKEND=mongodb13:07
ldesimon_enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler stable/pike13:07
ldesimon_OSPROFILER_CONNECTION_STRING=mongodb://localhost:27017/profiler13:07
ldesimon_it seems to give same results about instance creation command13:07
ldesimon_that is, only glance traces are stored13:07
ilyashakhatnova is broken13:08
ilyashakhatlooks like that happened after moving API service under uwsgi13:08
*** ldesimon_ has quit IRC13:08
ilyashakhatand profiling is not initialized anymore13:09
*** ldesimone has joined #openstack-performance13:09
ldesimoneI can try with cinder?13:10
ldesimoneit could be work?13:10
*** brault has joined #openstack-performance13:11
ilyashakhatfor me Cinder doesn't work too...13:12
ldesimonemmm :D13:12
ilyashakhatonly keystone, glance and neutron13:12
ldesimonemmm ok13:12
ldesimoneI've found on youtube a workshop video and actually osprofiler works with vmware driver (https://www.youtube.com/watch?v=57fTdE1hjeo)13:13
ldesimonethere are some motivations?13:13
ilyashakhatdon't know..13:16
ilyashakhatbtw, if you goal is to see how opnestack services communicate you may use Packstack-based installation13:17
ilyashakhatit's Pike, but services run differently and profiling works13:18
ldesimonein packstack installation we can see the full traces for rpc, sql, rest api, communication between services?13:18
ldesimoneand, for older version of devstack (e.g., ocata) it could be work osprofiler+ceilometer?13:21
*** ldesimon_ has joined #openstack-performance13:23
*** ldesimone has quit IRC13:23
*** ldesimone has joined #openstack-performance13:24
ilyashakhatin Ocata Devstack it worked, but there's no guaranties that it still true13:25
ilyashakhatPackStack Pike profiling works13:25
ilyashakhatand it has the same amount of instrumentation points as current master13:26
*** ldesimon_ has quit IRC13:28
ldesimoneok, so packstack use osprofiler too13:29
ilyashakhatyes, but you will have to enable it manually13:30
ldesimonethere is some how-to to install it? is enough the README under git repo?13:36
ilyashakhatREADME should be enough13:37
ilyashakhatI usually install it from packages... will make a short instruction soon13:40
*** rmart04 has joined #openstack-performance13:40
*** ldesimon_ has joined #openstack-performance13:40
*** ldesimone has quit IRC13:42
*** ldesimone has joined #openstack-performance13:43
*** aojea has quit IRC13:45
*** aojea has joined #openstack-performance13:45
*** ldesimon_ has quit IRC13:46
ilyashakhat@ldesimone http://paste.openstack.org/show/625844/13:51
ldesimonethat is your config?13:53
ilyashakhatreplace ip address with yours in packstack-answers.txt13:53
ilyashakhatyes13:53
ilyashakhatso you need to install 3 packages13:53
ilyashakhatthen copy config13:53
ilyashakhatchange addresses13:53
ilyashakhatand then run packstack (under root)13:54
ldesimoneyep....only for centos machine? is it suitable for ubuntu env?13:54
ilyashakhatno :(13:54
ilyashakhatit's based on RDO packages13:54
ldesimoneok13:54
*** aojea has quit IRC14:05
*** aojea has joined #openstack-performance14:06
*** aojea has quit IRC14:17
*** aojea has joined #openstack-performance14:43
*** aojea has quit IRC15:07
*** aojea has joined #openstack-performance15:11
*** AlexeyAbashkin has quit IRC15:19
*** aojea has quit IRC15:22
*** aojea has joined #openstack-performance15:24
*** aojea has quit IRC15:34
*** arcolife has quit IRC15:39
*** aojea has joined #openstack-performance15:41
*** aojea has quit IRC15:45
*** aojea has joined #openstack-performance15:51
*** aojea has quit IRC16:04
*** aojea has joined #openstack-performance16:07
*** aojea has quit IRC16:19
*** rmart04 has quit IRC16:23
*** jkilpatr has quit IRC16:41
*** openstackgerrit has quit IRC16:48
*** arcolife has joined #openstack-performance16:49
*** jkilpatr has joined #openstack-performance16:55
*** AlexeyAbashkin has joined #openstack-performance16:56
*** rmart04 has joined #openstack-performance16:58
*** AlexeyAbashkin has quit IRC17:00
*** rmart04_ has joined #openstack-performance17:02
*** rmart04 has quit IRC17:02
*** rmart04_ is now known as rmart0417:02
*** jkilpatr has quit IRC17:05
*** arcolife has quit IRC17:12
*** jkilpatr has joined #openstack-performance17:18
*** AlexeyAbashkin has joined #openstack-performance17:28
*** AlexeyAbashkin has quit IRC17:32
*** rmart04 has quit IRC17:37
*** rmart04 has joined #openstack-performance17:39
*** ldesimone has quit IRC17:40
*** aojea has joined #openstack-performance17:55
*** rmart04 has quit IRC18:01
*** msimonin has joined #openstack-performance18:04
*** jkilpatr has quit IRC18:05
*** rmart04 has joined #openstack-performance18:06
*** rmart04 has quit IRC18:09
*** rmart04 has joined #openstack-performance18:10
*** jkilpatr has joined #openstack-performance18:18
*** aojea has quit IRC18:21
*** rmart04 has quit IRC18:26
*** ldesimone has joined #openstack-performance18:30
*** aojea has joined #openstack-performance18:37
*** msimonin1 has joined #openstack-performance18:45
*** msimonin has quit IRC18:48
*** aojea has quit IRC18:54
*** aojea has joined #openstack-performance19:00
*** jkilpatr has quit IRC19:00
*** jkilpatr has joined #openstack-performance19:13
*** aojea has quit IRC19:18
*** aojea has joined #openstack-performance19:31
*** ldesimone has quit IRC19:40
*** aojea has quit IRC19:41
*** aojea has joined #openstack-performance19:41
*** rmart04 has joined #openstack-performance19:52
*** aojea has quit IRC19:53
*** rmart04_ has joined #openstack-performance19:59
*** rmart04 has quit IRC19:59
*** rmart04_ is now known as rmart0419:59
*** aojea has joined #openstack-performance20:04
*** rmart04 has quit IRC20:13
*** jkilpatr_ has joined #openstack-performance20:32
*** jkilpatr has quit IRC20:35
*** aojea has quit IRC20:44
*** msimonin1 has quit IRC21:32
*** jkilpatr_ has quit IRC21:32
*** msimonin has joined #openstack-performance21:33
*** msimonin has quit IRC21:34
*** msimonin has joined #openstack-performance21:34
*** jkilpatr has joined #openstack-performance21:57
*** aojea has joined #openstack-performance22:44
*** aojea has quit IRC22:49
*** msimonin has quit IRC23:39
*** msimonin has joined #openstack-performance23:39
*** msimonin has quit IRC23:39
*** msimonin has joined #openstack-performance23:40
*** msimonin has quit IRC23:40
*** msimonin has joined #openstack-performance23:40
*** msimonin has quit IRC23:41
*** msimonin has joined #openstack-performance23:41
*** msimonin has quit IRC23:42

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!