| *** eernst has joined #kata-dev | 00:32 | |
| *** eernst has quit IRC | 00:36 | |
| *** zerocoolback has joined #kata-dev | 01:54 | |
| *** zerocoolback has quit IRC | 02:08 | |
| *** zerocoolback has joined #kata-dev | 02:13 | |
| *** zerocoolback has quit IRC | 02:17 | |
| *** zerocoolback has joined #kata-dev | 02:17 | |
| *** yingjun has joined #kata-dev | 03:15 | |
| *** yingjun has quit IRC | 04:15 | |
| *** yingjun has joined #kata-dev | 05:39 | |
| *** marcov has joined #kata-dev | 05:42 | |
| kata-irc-bot | <niteshkonkar007> After enabling full debug, where do the agent logs get logged? Inside the VM or they are redirected outside for analysis? ` sudo journalctl -t kata-agent` did not show me anything. | 06:52 |
|---|---|---|
| *** jodh has joined #kata-dev | 07:08 | |
| *** zerocoolback has quit IRC | 07:39 | |
| kata-irc-bot | <caoruidong> output by proxy. I think you can journalctl -t kata-proxy | 07:45 |
| kata-irc-bot | <niteshkonkar007> @caoruidong: Yea, then we should document somewhere that agent logs can be seen via `journalctl -t kata-proxy` | 07:48 |
| *** yingjun has quit IRC | 07:48 | |
| *** davidgiluk has joined #kata-dev | 08:03 | |
| *** yingjun has joined #kata-dev | 08:19 | |
| kata-irc-bot | <james.o.hunt> @niteshkonkar007 - see https://github.com/kata-containers/tests/blob/master/cmd/log-parser/README.md :) | 08:29 |
| kata-irc-bot | <james.o.hunt> (which is referenced from https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#troubleshoot-kata-containers) | 08:29 |
| kata-irc-bot | <niteshkonkar007> @james.o.hunt: Ahh... that way its documented. :) | 08:32 |
| *** zerocoolback has joined #kata-dev | 08:59 | |
| *** zerocoolback has quit IRC | 09:04 | |
| *** zerocoolback has joined #kata-dev | 09:26 | |
| *** zerocoolback has quit IRC | 09:36 | |
| *** yingjun has quit IRC | 09:40 | |
| *** zerocoolback has joined #kata-dev | 09:54 | |
| *** zerocoolback has quit IRC | 09:58 | |
| *** zerocoolback has joined #kata-dev | 10:03 | |
| kata-irc-bot | <graham.whaley> If anybody is interested, I slapped the kata github org into cauldron.io to have a peek at github stats. It afaik does not auto-update, so is just a snapshot, but fun to stare at for a few minutes: http://cauldron.io/dashboards/kata-containers @anne ;) | 10:11 |
| *** zerocoolback has quit IRC | 10:11 | |
| kata-irc-bot | <mvedovati> @graham.whaley that's very cool! | 10:30 |
| kata-irc-bot | <graham.whaley> oh, note, the domains of contribs does not 100% reflect the orgs who contributed, as we know some folks submit from @gmail domain, so that all gets lumped together. I believe there is a layer/tool in either grimoire or biergia that allows you to do mappings, but I don't know if that is on the free view version, and I don't have time to track them down and type them in ;) | 10:38 |
| *** zerocoolback has joined #kata-dev | 10:39 | |
| kata-irc-bot | <niteshkonkar007> @graham.whaley: That's really awesome! :D | 10:42 |
| *** zerocoolback has quit IRC | 11:08 | |
| *** zerocoolback has joined #kata-dev | 11:10 | |
| *** zerocoolback has quit IRC | 11:21 | |
| *** zerocoolback has joined #kata-dev | 11:22 | |
| *** yingjun has joined #kata-dev | 12:36 | |
| *** marcov has quit IRC | 12:41 | |
| *** devimc has joined #kata-dev | 12:43 | |
| *** marcov has joined #kata-dev | 12:44 | |
| *** yingjun has quit IRC | 13:32 | |
| *** yingjun has joined #kata-dev | 13:43 | |
| *** annabelleB has joined #kata-dev | 14:07 | |
| *** annabelleB has quit IRC | 14:22 | |
| *** annabelleB has joined #kata-dev | 14:28 | |
| kata-irc-bot | Action: eric.ernst looks @ weizhang555's cgroups PR. Nice PR - I'm a bit swamped but will spend some time with this. | 14:37 |
| *** yingjun has quit IRC | 15:11 | |
| *** marcov has quit IRC | 15:14 | |
| *** marcov has joined #kata-dev | 15:20 | |
| *** jodh has quit IRC | 15:21 | |
| kata-irc-bot | <anne> @graham.whaley that looks great! @mrhillsman has been working on using grimoire as well and has something similar spun up. Perhaps we should combine efforts... :slightly_smiling_face: | 16:09 |
| kata-irc-bot | <anne> @salvador.fuentes @sebastien.boeuf @eric.ernst @graham.whaley The openstack-infra team is meeting in ~3 hrs (1900 UTC) if you all want to swing by and chat about the issues you've hit with the Zuul transition. I know that's quite late for @graham.whaley, but figured I'd ask! | 16:10 |
| kata-irc-bot | <graham.whaley> @anne sure. Hi @mrhillsman - the above is just the default that you get from cauldron.io - I did no work apart from rattle the org name into a dialog box and let it do 'its thing'. But, sure, if we are tailoring something, let's discuss. | 16:11 |
| kata-irc-bot | <graham.whaley> @anne heh, yeah, normally I will have dropped, but on Tuesdays I'm already in another call at that time ;) thx for the invite tho | 16:11 |
| kata-irc-bot | <mrhillsman> @anne i pinged you yesterday but you were in DnD mode | 16:12 |
| kata-irc-bot | <anne> ah sorry! | 16:12 |
| kata-irc-bot | <mrhillsman> @graham.whaley @anne if we get a node to setup grimoirelab on i can make that happen | 16:12 |
| kata-irc-bot | Action: graham.whaley wonders if mrhillsman will be using the same ELK infra the zuul folks use for the elastic view of zuul progress. I am working towards trying to deploy kata metrics gathering and view in that same setup. cauldron.io is using an ELK afaict. I've not dug into details of the underlying layers though | 16:13 |
| kata-irc-bot | <mrhillsman> unfortunately the pickle of multiple accounts appears to only be resolvable by using their profiles concept but should not be too difficult to maintain i think | 16:13 |
| kata-irc-bot | <mrhillsman> cauldron.io appears to be the same as bitergia; grimoirelab | 16:14 |
| kata-irc-bot | <graham.whaley> @mrhillsman yep, I believe that is the way to untangle the real user email/company matrix. Just requires setup and some typing... and possibly maintenance. | 16:14 |
| kata-irc-bot | <mrhillsman> which is yes, ELK :slightly_smiling_face: | 16:14 |
| kata-irc-bot | <mrhillsman> yes, it is not overly difficult especially since they have a python library to maintain it | 16:15 |
| kata-irc-bot | <mrhillsman> so individuals as well as folks on either committee can resolve conflicts | 16:15 |
| kata-irc-bot | <mrhillsman> i am ok with helping set it up wherever the community wants to host it | 16:16 |
| kata-irc-bot | <mrhillsman> it takes about 30 minutes or so to get it all working generally | 16:17 |
| kata-irc-bot | <mrhillsman> if that long | 16:17 |
| kata-irc-bot | <mrhillsman> i am not sure how much flexibility there is with using bitergia or cauldron but hosting it is not much of a burden it seems | 16:17 |
| *** zerocoolback has quit IRC | 16:24 | |
| kata-irc-bot | <graham.whaley> @mrhillsman right. that'd be great. I think the crux here is where it would be hosted and *who* would maintain/update it and look after security stuff if necessary. The OSF infra guys note that install and maintain of ELK is quite costly and complex, and thus they probably don't want to have another instance beyond the infra that already runs http://logstash.openstack.org/#/dashboard/file/logstash.json (see | 16:33 |
| kata-irc-bot | https://docs.openstack.org/infra/system-config/logstash.html) | 16:33 |
| kata-irc-bot | <mrhillsman> well that is quite an old version installed iirc :slightly_smiling_face: | 16:33 |
| kata-irc-bot | <graham.whaley> thus, for my jenkins/metrics ELK needs, I am looking to integrate with that existing infra. (I just need to figure out how to get json transparently through filebeat and logstash) :slightly_smiling_face: | 16:34 |
| kata-irc-bot | <graham.whaley> ah, that is something that would have to be discussed with the @osf-infra and @zuul folks I suspect. | 16:34 |
| kata-irc-bot | <mrhillsman> in terms of getting community stats i think it should be rather simple which is the use case i spoke with @anne about | 16:34 |
| kata-irc-bot | <graham.whaley> sure, np. /me interested to see where it goes :slightly_smiling_face: | 16:35 |
| kata-irc-bot | <mrhillsman> you could probably lean on grimoirelab tool or their code for getting straight json from jenkins - https://github.com/chaoss/grimoirelab-elk/blob/master/grimoire_elk/raw/jenkins.py | 16:36 |
| kata-irc-bot | <mrhillsman> schema is here - https://github.com/chaoss/grimoirelab-elk/blob/master/schema/jenkins.csv | 16:37 |
| *** annabelleB has quit IRC | 16:38 | |
| kata-irc-bot | <graham.whaley> @mrhillsman ah, sorry, this is for something slightly different. I run a bunch of metrics tests that generate json results, and have the ability to CURL it into a DB. I had it direct injecting into elastic and stuck kibana on top, and that PoC'd great - but the osf infra has the logstash layer as well that I will have to route though. looks like filebeat has a raw socket endpoint now, I just need to update my local PoC instance and | 16:38 |
| kata-irc-bot | test it out - and write the passthrough filter configs. Let me grab a github PR ref for what I'm trying to build... | 16:38 |
| kata-irc-bot | <mrhillsman> if you want the payload to be mapped a certain way rather than just pushed to a single key like _source or data and it is all there | 16:38 |
| *** davidgiluk has quit IRC | 16:39 | |
| kata-irc-bot | <graham.whaley> overview here: https://github.com/kata-containers/ci/issues/60 | 16:39 |
| *** davidgiluk has joined #kata-dev | 16:39 | |
| kata-irc-bot | <graham.whaley> but ... if you are experienced with elastic, then I reserve the right to ask you for input on the schema etc. - as, yeah, the whole mapping to a key etc. I think will blow in my face one day ;) | 16:39 |
| *** annabelleB has joined #kata-dev | 16:40 | |
| kata-irc-bot | <mrhillsman> looking | 16:41 |
| * davidgiluk tries to upgrade his hacky world from 1.1ish to 1.3-rc1 - has /run/kata-containers/shared/sandboxes moved to shared/contaienrs - and if so hmm | 16:41 | |
| *** devimc has quit IRC | 16:43 | |
| *** devimc has joined #kata-dev | 16:44 | |
| kata-irc-bot | <mrhillsman> so it sounds like @graham.whaley your focus is more on data from the CI system (jenkins and jenkins nodes)? | 16:45 |
| kata-irc-bot | <mrhillsman> if that is the case, i think logstash.o.o is only providing info for the CI system | 16:45 |
| kata-irc-bot | <graham.whaley> yep. specifically trying to detect any performance regressions and notify that back to github pull requests. yes, logstash should be a pretty transparent data port from my tests into the elastic db. But, in the osf infra, the logstash is required as part of the security/firewall routing and authorisation, so I have to go through it - and hence I think a filebeat with a raw socket and then a transparent direct path to elastic | 16:46 |
| kata-irc-bot | through logstash may work for me. that is my current thinking | 16:46 |
| kata-irc-bot | <mrhillsman> that sounds reasonable | 16:52 |
| kata-irc-bot | <mrhillsman> i think the only concern you will have is with structuring the data as you need | 16:53 |
| kata-irc-bot | <mrhillsman> should what you see in infra not be sufficient | 16:54 |
| kata-irc-bot | <raravena80> @graham.whaley do you have an idea of the amount of data? and retention? a small ELK cluster shouldn't be that bad to maintain. (We run a few clusters) | 16:56 |
| davidgiluk | oh, I see I think, one of those is host shared and the other is guest shared path - so different things | 16:56 |
| kata-irc-bot | <graham.whaley> @mrhillsman we'd use a different index in the osf infra for the metrics data, so we can define as we need. My main concern on my data is that if I import pretty raw json as it comes out at the minute, then afaik, if I then try to change that later I can only extend - if I make a non-compat change then elastic will reject the data as it will be in 'a different format' than any existing data of the same type (tag?). right now I think | 16:59 |
| kata-irc-bot | that means I just have to be very careful about data format upgrades and potentially versioning them. I may have to read more into elastic best practices of data transformation though. | 16:59 |
| kata-irc-bot | <mrhillsman> a different index is fine as you said and yes, i mean, you will need to be concerned about how you import your data | 17:00 |
| kata-irc-bot | <mrhillsman> the power of elasticsearch is in how you format your data | 17:01 |
| kata-irc-bot | <graham.whaley> @raravena80 I don't have an idea on the data size too much right now. I'd love to be able to keep it 'forever' ;). The data you can see attached to the Jenkins builds - such as the fragments attached to : http://jenkins.katacontainers.io/job/kata-metrics-agent-ubuntu-16-04-PR/ - so that looks like ~22kb of json - per CI build. We are averaging maybe 24 build/runs a day - so ~528k of json a day right now. I'd expect that to grow, a | 17:01 |
| kata-irc-bot | lot, though - both as we collect more data (I predict at least double that), and we get more PRs to process... | 17:01 |
| kata-irc-bot | <mrhillsman> if you expect to benefit from your json rather than just throwing it as is into a single key, you will need to use logstash to massage it into the structure you want to send it to elasticsearch afaik | 17:02 |
| kata-irc-bot | <mrhillsman> iiuc that is what the jenkins.py and jenkins.csv help to make happen | 17:03 |
| kata-irc-bot | <graham.whaley> in my PoC I stored it raw. I have a suitable timestamp field in there and the fields I want to index - I now also save raw results data so we can cook them as we want later. But, yes, I need to work out if I need to massage with logstash with the idea that I can futureproof myself from later data format changes. Here is a link to one of my jsons, just for reference: | 17:03 |
| kata-irc-bot | http://jenkins.katacontainers.io/job/kata-metrics-agent-ubuntu-16-04-PR/lastSuccessfulBuild/artifact/go/src/github.com/kata-containers/tests/metrics/results/boot-times.json | 17:03 |
| kata-irc-bot | <raravena80> @graham.whaley that's not that much :slightly_smiling_face: sounds like if you double 528k to 1000k (approximately) time 365 that'd give you 365m a year. Less than 1g for 3 years. | 17:06 |
| kata-irc-bot | <mrhillsman> @graham.whaley it has been a while since i have looked at elasticsearch and how it stores raw json. previously it just dumped it all into one field like message without any way of indexing/searching on it but we should definitely stay in touch | 17:19 |
| kata-irc-bot | <raravena80> elk, is generally pretty good for searching. Through Kibana you can search for different fields, time segments, etc. | 17:27 |
| *** zerocoolback has joined #kata-dev | 17:29 | |
| kata-irc-bot | <mrhillsman> agreed, but if i send raw json does it format into fields or just one field | 17:31 |
| *** marco_ has joined #kata-dev | 17:41 | |
| *** zerocoolback has quit IRC | 17:42 | |
| *** marcov has quit IRC | 17:44 | |
| *** marcov__ has joined #kata-dev | 17:51 | |
| *** marco_ has quit IRC | 17:55 | |
| kata-irc-bot | <graham.whaley> @mrhillsman @raravena80 when I ran up the PoC where the screenshot on that PR above came from, it showed up the individual json fields as elements I could search/sort on and also do math on - so I could calculate say the memory footprint or boot time averages etc. What I expect we'll do is set up an ELK with a test index, stash some data, and then see what we can do in the GUI - and then tweak from there. thanks for your input - | 17:57 |
| kata-irc-bot | once I've got enough in place to get some infra and an index set up then I'll let you know so we can brainstorm | 17:57 |
| kata-irc-bot | <mrhillsman> ++ | 17:59 |
| *** marcov__ has quit IRC | 18:03 | |
| *** annabelleB has quit IRC | 18:56 | |
| *** annabelleB has joined #kata-dev | 19:00 | |
| *** davidgiluk has quit IRC | 19:11 | |
| kata-irc-bot | <rreshef> @graham.whaley @raravena80 my 2 cents on metrics: for time-series metrics I prefer the TIG(K) stack to ELK stack. ELK uses Elastic’s close format, and milliseconds timestamps (I was using it for networking metrics where millisecond granularity is too course). Elastic uses NoSQL JSON datastore, not TSDB, so less suitable for TS metrics. And Logstash is a resource hog. Telegraf has open architecture and many many input and output plugins | 19:20 |
| kata-irc-bot | For time-series dashboards Grafana is IMO superior to Chronograf in the original TICK stack, and it can connect to Prometheus as well. | 19:20 |
| kata-irc-bot | <graham.whaley> hi @rreshef - ah, useful. I did start with influx and grafana. One reason I moved to K is that for some views I want a non-time X axis - what I actually want is git commits over time on the X axis :slightly_smiling_face: I found a way to half-hack that in Kibana, but could not find a way to do that in Grafana. So, I'd be happy to look at TIG(K), but likely with the K to ensure I am not glued to a purely time drawn X-axis. Yeah, I | 19:23 |
| kata-irc-bot | ended up on elastic as it was the 'natural' move to K at the time. Ultimately the choice might come down to what the infra team either have available or are willing to install/support | 19:23 |
| kata-irc-bot | Action: graham.whaley sees that the I is Influx :slightly_smiling_face:, so at least that is not a new surprise to me :-) | 19:24 |
| kata-irc-bot | <rreshef> hi @graham.whaley by K I meant Kapacitor (the optional component from InfluxData in the TICK stack, https://www.influxdata.com/time-series-platform/), not Kibana. I have never tried to connect Kibana to InfluxDB. | 19:26 |
| kata-irc-bot | <graham.whaley> ah, OK. I will have a look over TICK tomorrow. thx | 19:27 |
| kata-irc-bot | <rreshef> wlc. in http://docs.grafana.org/features/panels/graph/#x-axis-mode you can see examples of using a non-time-based x-axis | 19:28 |
| *** oikiki has joined #kata-dev | 19:48 | |
| kata-irc-bot | <raravena80> @rreshef yeah, I've used TIGK. I guess for the quantity of data that we are looking at it should be enough, but when I used it we were storing too much data on Influxdb making us go to their cluster solution which requires a paid license. Influxdb is pretty similar to Prometheus imo. | 19:48 |
| *** eernst has joined #kata-dev | 19:52 | |
| *** david-lyle is now known as dklyle | 20:02 | |
| kata-irc-bot | <rreshef> @raravena80 yeah Prometheus has built-in TSDB too (though in milliseconds I believe). If you “only” deal with aggregated metrics, Prometheus is a good choice. If you need to do event sourcing from your system and your metrics are the results of processing these events, then Prometheus is not suitable (I learned it the hard way :slightly_smiling_face: ); in such a case you are down to TIG(K). | 20:28 |
| *** oikiki has quit IRC | 21:47 | |
| *** devimc has quit IRC | 22:03 | |
| *** annabelleB has quit IRC | 22:36 | |
| *** annabelleB has joined #kata-dev | 22:37 | |
| *** annabelleB has quit IRC | 22:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!