| mkarpiarz | Thank you, leifmadsen and mrunge! | 12:41 |
|---|---|---|
| mrunge | mkarpiarz, my pleasure | 12:42 |
| mkarpiarz | So with Panko no longer being an option and support for MongoDB long gone, how are people storing/processing events these days? | 12:42 |
| mrunge | elasticsearch/loki | 12:43 |
| mrunge | err, I don't know how many people store..., but my understanding is, they are using different tools | 12:43 |
| mrunge | like elasticsearch or Grafana Loki | 12:44 |
| mkarpiarz | Ah, interesting! I see neither of them on the list of supported publishers here: https://docs.openstack.org/ceilometer/zed/admin/telemetry-data-pipelines.html#publishers. Do you mean there is a way to convert to ES/Loki using, for example, the "http" publisher? | 12:45 |
| mrunge | yes, you could be using the http publisher | 12:47 |
| mkarpiarz | OK, so is the basic idea to treat events like logs, send them to logstash for parsing and backing to ElasticSearch? | 12:52 |
| mrunge | in that case, you'd send them directly to ES, if you want to deal with elastic | 12:54 |
| mkarpiarz | So just send events as payload of requests to ElasticSearch's APIs? | 12:57 |
| mrunge | exactly | 12:58 |
| mrunge | however, you don't get an integration in the openstack CLI with this | 12:59 |
| mrunge | and I am not sure what you'd get out of events in this case anyways | 12:59 |
| mkarpiarz | Ceilometer appears to be sending lists of JSON-formatted objects through http. | 13:01 |
| mkarpiarz | Is there a better solution then? | 13:01 |
| mrunge | let me ask you the other way around: what do you want to get out of it? | 13:02 |
| mrunge | ... which events are important that you don't want to miss | 13:02 |
| mkarpiarz | Ah, a good question! All I need (at least for now) is events relating to instance creation and deletion. | 13:04 |
| mrunge | they would be sent when a user creates or deletes an instance | 13:04 |
| mrunge | iirc, they are not sent when an instance died e.g | 13:04 |
| mkarpiarz | That's fine. All we need is to be able to tell that this many VMs have been running at any given time. | 13:06 |
| mkarpiarz | We also want to know their flavours. | 13:06 |
| mrunge | that is something you should be able to get out of nova | 13:07 |
| mrunge | without any events | 13:07 |
| mkarpiarz | Yeah, just tap into the notification bus, right? | 13:07 |
| mrunge | uhm, it's been a little since I looked into that. iirc you can directly ask for the number of running vms | 13:08 |
| mkarpiarz | Sure, I can poll APIs for this. | 13:09 |
| mkarpiarz | Also, the Prometheus exporter for OpenStack keeps a tally of VMs in different states. | 13:10 |
| mrunge | is there an exporter? | 13:10 |
| mrunge | the one I've seen was not maintained for a long time | 13:11 |
| mkarpiarz | https://github.com/openstack-exporter/openstack-exporter | 13:11 |
| mrunge | oh, nice! | 13:13 |
| mkarpiarz | The `openstack_nova_server_status` metric is what I've been using. | 13:13 |
| mkarpiarz | The problem with a metrics-based approach is that the state of VMs doesn't update in real time. | 13:15 |
| mkarpiarz | But I guess for such a simple use case it might not make sense to maintain all the components required for processing/storing events. | 13:17 |
| mrunge | hmm, so how much real time do you need? | 13:17 |
| mrunge | you'd be using a lot of resources to keep track of the changes otherwise in a different data store | 13:18 |
| mkarpiarz | Very true. | 13:25 |
| mkarpiarz | However, from my experience a question about measurements in real time or with very fine resolution pops up every time sooner or later. | 13:27 |
| mkarpiarz | Also, personally I like the idea of having an alternative for polling APIs all the time just to get information which rarely changes. | 13:29 |
| mrunge | the question of getting metrics in a high resolution comes up. and then you'll discover that you rarely have a monitor to display the collected data | 13:30 |
| mrunge | we had the request to collect (and display) data in sub-second interval. And suddenly 10 minutes of data will require a larger screen, or you realize, that you could also sample at a lower rate without changing anything on the screen | 13:32 |
| mkarpiarz | Haha, that's a good one! In the end all people care is aggregates. xD | 13:32 |
| mrunge | yes. I agree, if you want events and reaction on events in real-time, then you don't want metrics | 13:32 |
| mrunge | "my network connection died" -> bring up a new one | 13:33 |
| mrunge | that's something you want instantly | 13:33 |
| mrunge | cpu count on server machines: maybe interesting, but not likely to change in a fast pace | 13:33 |
| mkarpiarz | Awesome! Thank you again for your time and all the food for thought! | 13:35 |
| mrunge | thank *you* for the questions and the link to the exporter. I really thought it was dead. | 13:36 |
| mrunge | but apparently it's not | 13:36 |
| mkarpiarz | You are welcome! :) | 13:40 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!