07:06:26 <tobberydberg> #startmeeting publiccloud_sig
07:06:26 <opendevmeet> Meeting started Wed Aug 30 07:06:26 2023 UTC and is due to finish in 60 minutes.  The chair is tobberydberg. Information about MeetBot at http://wiki.debian.org/MeetBot.
07:06:26 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
07:06:26 <opendevmeet> The meeting name has been set to 'publiccloud_sig'
07:06:28 <tobberydberg> Slow bot today or did I misspell? ;-)
07:06:32 <tobberydberg> There it is :-)
07:07:12 <tobberydberg> Agenda to be found: #link https://etherpad.opendev.org/p/publiccloud-sig-meeting
07:07:21 <fkr> ok. first item on the agenda: "0. Quick introduction of any new people"
07:07:25 <tobberydberg> Please put yourname in there
07:08:10 <fkr> my hope is still strong (new people), since I did send out a reminder to the scs list as well :)
07:08:19 <gtema> :)
07:08:27 <tobberydberg> Do we have any new people in here? :-)
07:08:29 <zigo> o/
07:08:39 <fkr> hey zigo!
07:08:47 <joek-office> i'm new, but not brand new ;-)
07:09:07 <fkr> joek-office: do you want to intro yourself?
07:09:18 <joek-office> ok.
07:09:24 <zigo> frickler: Hey! :)
07:09:48 <zigo> whoops, meant for fkr...
07:09:57 <fkr> ;)
07:10:08 <joek-office> Like i said in the past, we are at the moment building up a mixed public/private Cloud infrastructure based on openstack.
07:10:59 <joek-office> destination is running services in the logistcs area with the software that is build inhouse.
07:11:11 <tobberydberg> Welcome (again) joek-office ;-)
07:11:29 <fkr> joek-office: what was the reason for choosing openstack?
07:11:40 <joek-office> at the moment we are running from testing to preprod stage.
07:13:51 <joek-office> fkr. i'm new in the company since a year. the decision is made before. but i think is the same as everywhere. Make the things more inhouse than just leave on the public hyper scalers.
07:14:27 <tobberydberg> Sounds like a sound choice :-)
07:15:01 <tobberydberg> If no more intros, lets move on in the agenda...
07:15:01 <fkr> ack
07:15:10 <fkr> (to both ;)
07:15:15 <tobberydberg> #topic 1. Topic for lightning talk next time?
07:15:38 <joek-office> the "make more inhouse" mindset is very common in the company. there are divisions from building up warehouse to running the complete lifecycle
07:15:59 <joek-office> sorry, what is a lightning talk?
07:16:21 <fkr> a short (5 - 15 ? minute) talk
07:16:28 <tobberydberg> Tha plan is to have a "video call" next time, and the proposal is to have a lightning talk if we have a topic and someone willing to present something
07:16:46 <fkr> and the idea was to make it a lightning talk to have it short and entertaining
07:16:53 <fkr> as well as less burden to prepare :)
07:17:02 <tobberydberg> Anyone having an interesting topic they would like to present?
07:17:06 <joek-office> ok, thank you
07:18:34 <gtema> maybe I have something from SDK/CLI area to present what might be interesting for the cloud operators: OpenAPI support for OpenStack and making a new faster CLI on Rust
07:18:44 <fkr> woot
07:18:47 <fkr> +1
07:18:51 <gtema> not sure whether I would have enough time to prepare though
07:19:15 <joek-office> sounds good and interesting
07:19:23 <fkr> from my perspective, it does not need to be with slides, bells and whistles
07:20:01 <fkr> the idea that puck, tobberydberg and me had was to let this be up for the person giving the lightning talk so demo, code, etherpad contents whatever is imho perfectly fine
07:20:18 <gtema> sure
07:20:26 <tobberydberg> Agreed, super interesting and bells and whistles not needed :-)
07:20:28 <tobberydberg> +1
07:20:49 <tobberydberg> Cool! Lets go with that then, great!
07:21:21 <tobberydberg> Will you create a Jitsi meeting that is open under scs jitsi fkr?
07:21:48 <fkr> yes
07:22:08 <fkr> https://conf.scs.koeln:8443/OIF-public-cloud-sig
07:22:15 <fkr> that is going to be the place
07:22:27 <tobberydberg> Perfect, thanks!
07:22:30 <fkr> (this way we already have it in the meeting log)
07:22:36 <tobberydberg> next topic...
07:22:48 <fkr> Metering & Billing
07:22:55 <tobberydberg> #topic 2. Meetering and billing...
07:23:14 <tobberydberg> (oooops, too many eee's)
07:23:19 <fkr> last time I had to depart early and after I left there was a discussion on metering and billing with distil being thrown into the conversation
07:23:41 <fkr> that I found very interesting, since as part of SCS this topic was raised a year ago as well
07:23:58 <joek-office> yes, as alternative or side by side of cloudkitty
07:24:10 <fkr> and as such there was a tender being created (VP13 - https://scs.community/tenders/lot13)
07:24:44 <fkr> and Cloud & Heat successfully bid on that tender and is currently working with gnocchi and ceilometer to provide something that emits json to a billing stack
07:25:06 <fkr> https://github.com/SovereignCloudStack/issues/issues/269
07:25:14 <fkr> that is the epic with the user stories referenced
07:25:45 <fkr> they've been reaching out to mrunge to coordinate this with ceilometer as well
07:26:07 <fkr> why I wanted to bring it back to the table here: I'm sure you do billing :)
07:26:27 <fkr> how are you (looking at this nice round ;) / how have you been doing this?
07:26:46 <zigo> FYI, we use cloudkitty (plus https://salsa.debian.org/openstack-team/services/ceilometer-instance-poller/ that I wrote to bill Windows images...).
07:27:18 <joek-office> we use the combination prometheus/cloudkitty
07:28:25 <tobberydberg> We did use ceilometer in the past, but had to much issues with that, loosing events etc that caused the "billing" to be incorrect. This is some time ago though....
07:28:37 <joek-office> have tested the combination of ceilometer/gnocchi/cloudkitty as well, but this stack looks like not so easy and wide supported as the easy combination of the prometheus exporters
07:28:37 <tobberydberg> I'm keen on somthing that is more "prometheus based", but I haven't dug my self deep into it for quite some time
07:29:11 <zigo> The main issue we have, is the Gnocchi API and MySQL being super busy even with as few as 3K VMs. We have dedicated 3x 128 core server for it, and a dedicated Ceph cluster too, but even that isn't enough...
07:30:27 <tobberydberg> I know we had a discussion regarding kind of forming a new metering project within openstack a bunch of years ago, prometheus based was already then a strong contender
07:30:35 <joek-office> the 3x128 core servers for compute or just for gnocchi/mysql? i hope for compute
07:31:00 <tobberydberg> performance was a huge issue for us as well
07:31:02 <fkr> jeiks (3x128 core servers)
07:31:13 <zigo> For the role we call "messaging" node. It contains: MySQL, Gnocchi-api, and cloudkitty-processor.
07:32:47 <joek-office> ok, thanks zigo. that is much ressources for "just" computing billing. but is a nice forecast for my future resource demand
07:33:22 <zigo> joek-office: I'm telling you, that is *not* overprovision ... :P
07:34:14 <fkr> tobberydberg: interesting (initiative of new metering, prometheus based). is there any trace of that?
07:34:51 <zigo> Correction: 64 cores, not 128, and 784 GB of RAM, but still... for "only" 3K VMs...
07:34:56 <tobberydberg> Didn't kick off at all...just one meeting kind of...
07:34:57 <fkr> but it reads to me as "everyone does it themselves by using similar building blocks but there is nothing that really works out of the box"
07:35:08 <joek-office> yes belief it. in the past i have seen in our test env that the cloudkitty processor shoot out the processors with just some (10-15) vms that have deployed and destroyed in short time
07:35:15 <zigo> Sometimes, there's so many Gnocchi threads that there's no source port left to connect !!!
07:35:20 <fkr> maybe not "works out of the box" but is "suiteable out of the box"
07:36:15 <zigo> How does the prometheus solution scales? Better?
07:36:54 <fkr> joek-office: can you share a bit more on how you built that?
07:37:25 <joek-office> like i mentioned. i have at the moment no stage/production systems with many vms deployed. but in the past i see many load for just ten to fifteen vms.
07:38:26 <tobberydberg> Haven't played with it in this use case, but I would assume so since it scales super good for "other" metrics
07:38:48 <joek-office> we have build up the mobnitoring cluster before based on prometheus. then we evaluated the combinations of gnocchi/ceilometer/cloudkitty and prometheus/cloudkitty
07:38:59 <joek-office> we deployed both with kolla-ansible
07:39:46 <joek-office> at the end we decide to use prometheus/cloudkitty because of the fewer services that had to run
07:40:02 <joek-office> but at the moment no scale projection can be made
07:41:41 <joek-office> tobberydberg: my feeling sounds also good for scaling with prometheus
07:42:55 <tobberydberg> I hope to get my team to prioritize looking in to that at some point... (point = year) ;-)
07:43:38 <fkr> is there an interest in this group to have a breakout to dive into more of this subject?
07:44:33 <joek-office> for me: yes. But the broader view of the billing infrastructure is very interesting.
07:44:57 <tobberydberg> Yes, if there is "enough" operators in the "call". But, I don't think I can pull the resources to to actual work unfortunate
07:45:07 <joek-office> are there best practices for the billing journey with OpenStack
07:45:44 <fkr> tobberydberg: I can go fishing for operators that join such a breakout
07:45:48 <fkr> and collect interest beforehand
07:46:07 <fkr> action item for me would be to also write a mail to openstack-discuss then
07:46:25 <tobberydberg> From what I know, the ceilometer/gnocchi/cloudkitty is the "official" and only recommendation upstream
07:46:45 <tobberydberg> How many operators that actually do it that way, not sure...
07:47:02 <tobberydberg> We don't, we are not using any of those projects
07:47:14 <joek-office> what do you use?
07:47:20 <joek-office> or how do you solve this
07:48:09 <joek-office> fkr: sounds good for me.
07:48:27 <tobberydberg> Self developed hack if I'm going to honest. It is solid and well proven for our use case and how we charge our customers. But, not perfect for that either
07:48:43 <fkr> tobberydberg: my gut feeling is that you're in good company with that
07:49:01 <tobberydberg> Yes, that is my understanding as well
07:49:31 <tobberydberg> Catalyst is probably an exception, at least from public cloud perspective.
07:50:28 <tobberydberg> As public cloud it is super important to be fully accurate with the numbers (which we had issues to reach with ceilometer)
07:52:07 <joek-office> tobberydberg: yes, this is it. would be nice to hear from puk how they work with that.
07:52:46 <fkr> i'll re-add it to the agenda for next time as well (maybe puck is around then)
07:52:57 <joek-office> and how it scale
07:53:12 <tobberydberg> They seam to be successful using that though, no clue about the scale there.
07:55:12 <tobberydberg> But a separate session regarding this is probably a good starting point, at least to share experiences
07:56:25 <tobberydberg> Would be awesome if that leads to something concrete, but sharing experience is always good if it stops with that :-)
07:56:45 <tobberydberg> Almost out of time.... Should we postpone the last topic until next time or start it now? :-)
07:56:55 <fkr> i'll add it for next time
07:57:07 <fkr> tobberydberg: +1 on the concrete!
07:58:24 <tobberydberg> Perfect! Something else we should bring up now or suggestion for topics moving ahead?
07:59:15 <joek-office> here in germany are several providers with public clouds out there that have to do billing and use OpenStack. Like Syseleven or Open Telekom Cloud. Anyone knows someone from this companys?
07:59:46 <fkr> yes
08:00:17 <joek-office> tobberydberg: from my side there is nothing.
08:00:35 <tobberydberg> Some representatives are already in here, but lets continue to try to get more of them in here on a regular basis...
08:02:25 <fkr> i'll now dive into the SCS Team IaaS call.
08:02:26 <tobberydberg> Ok, so time is unfortunate up here. We look forward to your lightning talk in two weeks gtema :-)
08:02:28 <tobberydberg> Thanks everyone for today and we will see (!) each other in two weeks :-)
08:02:35 <gtema> :)
08:02:37 <fkr> was a really nice meeting today, thanks for the participation!
08:02:54 <gtema> see ya
08:03:11 <tobberydberg> Have a great rest of the day!
08:03:15 <tobberydberg> #endmeeting