| *** egernst has quit IRC | 01:53 | |
| *** egernst has joined #kata-dev | 01:54 | |
| *** egernst has quit IRC | 01:59 | |
| *** egernst has joined #kata-dev | 01:59 | |
| *** egernst has quit IRC | 02:04 | |
| *** egernst has joined #kata-dev | 02:37 | |
| *** egernst has quit IRC | 02:46 | |
| kata-irc-bot | <eric.ernst> anyone notice issues when building with latest golang release(s)? I see we have "14.4" in our versions.yaml. | 02:50 |
|---|---|---|
| *** egernst has joined #kata-dev | 03:11 | |
| *** egernst has quit IRC | 03:15 | |
| *** egernst has joined #kata-dev | 03:51 | |
| *** egernst has quit IRC | 03:55 | |
| *** egernst has joined #kata-dev | 04:22 | |
| *** egernst has quit IRC | 04:27 | |
| *** egernst has joined #kata-dev | 04:38 | |
| *** egernst_ has joined #kata-dev | 04:38 | |
| *** egernst_ has quit IRC | 04:40 | |
| *** egernst has joined #kata-dev | 04:40 | |
| *** egernst has quit IRC | 05:12 | |
| *** egernst has joined #kata-dev | 05:13 | |
| *** egernst has quit IRC | 05:18 | |
| *** egernst has joined #kata-dev | 05:40 | |
| *** egernst has quit IRC | 05:45 | |
| *** egernst has joined #kata-dev | 05:54 | |
| *** egernst has quit IRC | 05:59 | |
| *** sameo has joined #kata-dev | 06:05 | |
| *** egernst has joined #kata-dev | 06:26 | |
| *** egernst has quit IRC | 06:31 | |
| *** egernst has joined #kata-dev | 06:58 | |
| *** egernst has quit IRC | 07:04 | |
| *** egernst has joined #kata-dev | 07:20 | |
| *** egernst has quit IRC | 07:25 | |
| kata-irc-bot | <fidencio> I had some issues related to Ctty with golang 1.15, but I submited patches upstream to fix that. Apart from that, no. What's the issue you're facing? | 07:35 |
| *** brueckner has joined #kata-dev | 07:41 | |
| *** sgarzare has joined #kata-dev | 07:50 | |
| *** egernst has joined #kata-dev | 07:56 | |
| *** fgiudici has joined #kata-dev | 07:57 | |
| *** egernst has quit IRC | 08:01 | |
| *** egernst has joined #kata-dev | 08:19 | |
| *** jodh has joined #kata-dev | 08:19 | |
| *** egernst has quit IRC | 08:20 | |
| *** th0din has quit IRC | 10:05 | |
| *** th0din has joined #kata-dev | 10:07 | |
| *** fgiudici has quit IRC | 10:19 | |
| *** fgiudici has joined #kata-dev | 10:26 | |
| *** dklyle has quit IRC | 10:58 | |
| *** devimc has joined #kata-dev | 12:42 | |
| kata-irc-bot | <eric.ernst> Yeah, saw your PRs. That was it! | 13:18 |
| kata-irc-bot | <fidencio> \o/ | 13:21 |
| *** fuentess has joined #kata-dev | 14:17 | |
| *** fuentess has quit IRC | 15:23 | |
| *** fuentess has joined #kata-dev | 15:24 | |
| *** dklyle has joined #kata-dev | 15:39 | |
| *** fuentess has quit IRC | 16:01 | |
| *** fuentess has joined #kata-dev | 16:04 | |
| kata-irc-bot | <eric.ernst> I’m seeing CLH build failures in CI: error: failed to get `kvm-ioctls` as a dependency of package `hypervisor v0.1.0 (/cloud-hypervisor/hypervisor)` | 16:05 |
| kata-irc-bot | <eric.ernst> @chen.bo @sebastien.boeuf anything change here? | 16:06 |
| *** sameo has quit IRC | 16:24 | |
| fidencio | devimc: jodh: you guys who are part of the project since the very beginning, I'd like to ask an "archaeological" question | 16:34 |
| fidencio | devimc: jodh: did Kata ever worked together with cAdvisor? I understand it works quite well with CRI stats and I even see some parts of the code where we change cgroups namespaces prefixes to ensure cAdvisor won't check for that | 16:35 |
| fidencio | also, I know using cAdvisor would be a step backwards, but I'm interested to know the backgroud of not supporting that, if you guys happen to remember | 16:37 |
| devimc | fidencio, I don't remember, may be Eric knows ^^ | 16:38 |
| kata-irc-bot | <fidencio> @eric.ernst ^^ :-) | 16:39 |
| kata-irc-bot | <eric.ernst> I guess I don't fully know what you mean by work with | 16:40 |
| kata-irc-bot | <eric.ernst> cadvisor would need to speak to Kata directly to get guest stats for it to work. | 16:40 |
| fidencio | @eric.ernst, why do we do this? https://github.com/kata-containers/kata-containers/blob/e830192fcab199dac9c7c2166182bd28adb797ce/src/runtime/virtcontainers/pkg/cgroups/utils.go#L18-L22 | 16:43 |
| kata-irc-bot | <eric.ernst> Its more of a kube issue than cadvisor problem, per se. | 16:44 |
| kata-irc-bot | <fidencio> IIUC what you're saying, cAdvisor would need to be tweaked to talk to Kata directly. If that's the case, why do we care about preventing it look at those namespaces? | 16:45 |
| kata-irc-bot | <eric.ernst> Kubelet will use CRI if there aren't cadvisor stats | 16:45 |
| *** devimc has quit IRC | 16:45 | |
| *** devimc has joined #kata-dev | 16:45 | |
| kata-irc-bot | <eric.ernst> so, for some things cadvisor has the 'better stats' so Kubelet will defer to cadvisor *if* available. If not it'll use the values it gets from CRI. | 16:45 |
| kata-irc-bot | <eric.ernst> So, if kubelet sees container level stats, it'll throw away the cri ones. | 16:46 |
| kata-irc-bot | <eric.ernst> To facilitate that, we make sure not to create any cgroups on the host that cadivsor/kubelet assume to be a container cgroup. | 16:46 |
| kata-irc-bot | <fidencio> Hmm. Okay, this whole thing has been proved way, way more complex than I'd like to. :-). You, and also folks from containerd (@pavlenko.maksym) may want to take a look at: https://github.com/kubernetes/enhancements/pull/2364 | 16:47 |
| kata-irc-bot | <eric.ernst> Yeah, saw that. | 16:47 |
| kata-irc-bot | <eric.ernst> I think dashpole is the one who'd have the best history on it, as he implemented/maintains. | 16:48 |
| kata-irc-bot | <fidencio> ack, just trying to get all the pieces together with my limited knowledge of this whole environment | 16:48 |
| kata-irc-bot | <fidencio> @eric.ernst, thanks! | 16:49 |
| kata-irc-bot | <eric.ernst> agreed re: complexity. When it comes down to it, we aren't really creating "container cgrousp" that should be consumed by the upper layer. | 16:49 |
| kata-irc-bot | <eric.ernst> Down the road, 'container cgroup' should *only* be used for the scenario where we start handling cpusets *inside* the guest. | 16:49 |
| kata-irc-bot | <eric.ernst> no problem; it could be clearer in the code; I think we spent more effort describing in the commit than in the comment. | 16:50 |
| kata-irc-bot | <eric.ernst> There's a lot of projects in the ecosystem that assume if there's a directory that has a container ID in it, it is where we should get monitoring info on the behavior of that container. | 16:52 |
| kata-irc-bot | <eric.ernst> kind of related, and maybe more straightforward -- do you know if crio was going to enable using cri stats in kubelet? | 16:55 |
| kata-irc-bot | <eric.ernst> @fidencio ^ | 16:55 |
| kata-irc-bot | <eric.ernst> For the longest time there was a "if CRIO" check in Kubelet that bypassed CRI stats in its entirety, which means you'll never get accurate accounting info when using CRIO in Kubernestes w/ Kata. | 16:56 |
| kata-irc-bot | <eric.ernst> I assume that's part of what Peter is trying to resolve? | 16:56 |
| kata-irc-bot | <fidencio> Yep, that's exactly what Peter is trying to resolve | 16:56 |
| kata-irc-bot | <fidencio> And I got patches on CRI-O that know take the Kata information properly into consideration | 16:57 |
| kata-irc-bot | <eric.ernst> And the stance is he'd rather make sure the configured cadvisor doesn't scrape this in the first place for those specific stats? | 16:57 |
| kata-irc-bot | <eric.ernst> naively, you could just remove that "if crio"check in kubelet and all would be good, right? | 16:58 |
| kata-irc-bot | <fidencio> Ah, Eric, there's no much questions involved there :-) | 16:58 |
| kata-irc-bot | <eric.ernst> s/good/ functional but not ideal | 16:58 |
| kata-irc-bot | <fidencio> Monitoring tools are still consuming cAdvisor endpoints (AFAIU) | 16:58 |
| kata-irc-bot | <eric.ernst> Yerah. | 16:58 |
| kata-irc-bot | <fidencio> so, the switch is slightly more complex than just removing that code | 16:58 |
| kata-irc-bot | <eric.ernst> Which I thought they would still want for a lot of stats still? | 16:59 |
| kata-irc-bot | <fidencio> I think they will, for some. | 16:59 |
| kata-irc-bot | <fidencio> But this has to be quite well co-ordinated | 16:59 |
| kata-irc-bot | <fidencio> And I womn | 16:59 |
| kata-irc-bot | <eric.ernst> hm. it "works" w/ conatinerd fine today. And not becuase contaienrd does something, its becuase there isn't a special path in kubelet to prevent it from getting asked for cri stats. | 17:00 |
| kata-irc-bot | <fidencio> So, I got it to work properly with CRI-O, just by renaming the CRI-O socket (that's the check on the kubelet side) and using this series here: https://github.com/cri-o/cri-o/pull/4470 | 17:01 |
| kata-irc-bot | <eric.ernst> oh lord. | 17:02 |
| kata-irc-bot | <eric.ernst> Yeah, that'll work! | 17:02 |
| kata-irc-bot | <eric.ernst> it passes the "if socket name == crio" part. | 17:02 |
| kata-irc-bot | <eric.ernst> Anyway, this is super interesting, and I shoudln't waste your time with my banter here -- I should read the KEP! | 17:02 |
| kata-irc-bot | <fidencio> Well, it'll work as long as we calculate the things properly, which we didn't before my patch /o\ | 17:03 |
| kata-irc-bot | <eric.ernst> that'll help get the context. | 17:03 |
| kata-irc-bot | <eric.ernst> yeah. I guess i'm curious why not just fix kubelet to remove the if crio check? | 17:04 |
| kata-irc-bot | <eric.ernst> as a easier first step, then look at peter's KEP as a longer term cleaning? | 17:04 |
| kata-irc-bot | <fidencio> Removing the check is the first step, but that will not do much, as kubelet nowadays just get all the CRI stats and throw them away if cAdvisor stats are around (as you mentioned, as we found out in the hard way) | 17:05 |
| kata-irc-bot | <chen.bo> I saw this issue once yesterday from one of my PRs to CLH yesterday, but restarting the GH actions resolved my issue. | 17:07 |
| kata-irc-bot | <fidencio> metrics-server is ready to just consume cri stats, but other tools simply are not (prometheus-adapter, for instance) | 17:07 |
| kata-irc-bot | <fidencio> And, AFAIU, that would simply break them | 17:07 |
| kata-irc-bot | <chen.bo> Strange though, the CLH CI in kata kept failing with restarting CI couple of times. | 17:07 |
| *** brueckner has quit IRC | 17:07 | |
| kata-irc-bot | <eric.ernst> I don't think it'll throw away (again, this is working today?) | 17:08 |
| kata-irc-bot | <chen.bo> Let me ask Seb on IRC. | 17:08 |
| kata-irc-bot | <fidencio> but hey, take this with a grain of salt, read the KEP and the comments, as my brain is burning trying to get the whole picture | 17:08 |
| kata-irc-bot | <eric.ernst> I see, yeah, we updated kubelet-resources to expose these | 17:08 |
| kata-irc-bot | <eric.ernst> so if you prom scrapes these, you get the cri stats (that's what we do today) | 17:08 |
| kata-irc-bot | <eric.ernst> i think Peter's stuff is good, but again with your workaround or with removing special case in kubelet, today you can get the pod and container cgroup stats w/ kata. | 17:09 |
| kata-irc-bot | <fidencio> Yep, at least upstrem, yep. | 17:10 |
| *** fgiudici has quit IRC | 17:20 | |
| *** sgarzare has quit IRC | 17:57 | |
| *** jodh has quit IRC | 18:07 | |
| *** sameo has joined #kata-dev | 18:32 | |
| kata-irc-bot | <chen.bo> Heads-up that the issue is related to the changes from the dependency repo (cloud-hypervisor/kvm-ioclts). I am sending a patch to kata-containers/packaging which should fixed the issue. | 19:19 |
| kata-irc-bot | <chen.bo> Basically, the `kvm-ioctls` repo is now longer carrying the old commits that the clh is building as a dependency. Maybe there was a force push, or something similar. | 19:20 |
| kata-irc-bot | <eric.ernst> devimc -- hola -- around? | 19:25 |
| kata-irc-bot | <chen.bo> PR is here: https://github.com/kata-containers/packaging/pull/1192. PTAL. @eric.ernst @jose.carlos.venegas.m | 19:34 |
| kata-irc-bot | <jose.carlos.venegas.m> @chen.bo thanks | 19:36 |
| kata-irc-bot | <jose.carlos.venegas.m> I see snap CI failing, seems that is not related with your PR | 19:37 |
| kata-irc-bot | <chen.bo> Great. THanks for running the pipes for me. | 19:37 |
| kata-irc-bot | <chen.bo> Looks like the `jenkins-metrics-ubuntu-18-04 _Expected_ — _Waiting for status to be reported_` is not running? | 19:38 |
| kata-irc-bot | <chen.bo> Do we need special trigger for this CI? @jose.carlos.venegas.m | 19:38 |
| kata-irc-bot | <jose.carlos.venegas.m> no, should be `/test` | 19:39 |
| kata-irc-bot | <jose.carlos.venegas.m> you can try again, but I am afraid will restart the other job as well | 19:40 |
| kata-irc-bot | <chen.bo> alright, then. Let's just wait. Maybe the metrics machine is really busy. ;) | 19:47 |
| kata-irc-bot | <chen.bo> I will come back and check it after lunch. | 19:47 |
| kata-irc-bot | <chen.bo> Go out to pickup my lunch, and walk on the snow first time for the year. :) | 19:47 |
| kata-irc-bot | <jose.carlos.venegas.m> xD | 19:55 |
| kata-irc-bot | <jose.carlos.venegas.m> Enjoy | 19:55 |
| *** fuentess has quit IRC | 19:56 | |
| *** fuentess has joined #kata-dev | 20:16 | |
| *** fuentess has quit IRC | 20:21 | |
| kata-irc-bot | <chen.bo> Just manually triggered the metrics CI "/test-ubuntu-qemu-metrics" | 20:57 |
| *** sameo has quit IRC | 21:54 | |
| *** devimc has quit IRC | 21:58 | |
| *** fuentess has joined #kata-dev | 22:02 | |
| *** fuentess has quit IRC | 22:07 | |
| *** fuentess1 has joined #kata-dev | 22:08 | |
| *** fuentess1 has quit IRC | 22:26 | |
| *** fuentess has joined #kata-dev | 22:27 | |
| *** fuentess has quit IRC | 22:29 | |
| *** fuentess has joined #kata-dev | 22:30 | |
| *** fuentess has quit IRC | 22:32 | |
| *** fuentess1 has joined #kata-dev | 22:32 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!