13:00:12 <chaconpiza> #startmeeting monasca
13:00:13 <openstack> Meeting started Tue May 26 13:00:12 2020 UTC and is due to finish in 60 minutes.  The chair is chaconpiza. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:17 <openstack> The meeting name has been set to 'monasca'
13:00:25 <chaconpiza> Hello everyone
13:00:39 <bandorf> hello
13:00:57 <chaconpiza> Hi Matthias
13:01:42 <chaconpiza> The agenda as usual https://etherpad.opendev.org/p/monasca-team-meeting-agenda
13:02:24 <witek> hi
13:02:36 <chaconpiza> I have changed a little bit the order in the agenda to start with the action items from previous meetings
13:02:40 <chaconpiza> Hi Witek
13:03:05 <chaconpiza> witek, thanks for leading last week.
13:03:09 <chaconpiza> #topic Action items from previous meeting
13:03:30 <witek> I think there was one for me, which I forgot...
13:04:01 <chaconpiza> From 2 weeks ago, I promised: chaconpiza Set minimum version for InfluxDB and influxdb-python-client. Clean the influxdb/metrics_repository
13:04:46 <chaconpiza> It will be on this change https://review.opendev.org/#/c/725779/ Still to do.
13:05:37 <chaconpiza> I think witek means: #action create a story to fix Cassandra installation in DevStack plugin
13:05:48 <chaconpiza> and #action witek mark monasca-tempest-python2-cassandra job as non-voting
13:06:00 <Wasaac> o/
13:06:05 <witek> right, will do right after the meeting, promise :)
13:06:18 <witek> hi Wasaac
13:06:28 <chaconpiza> Hi Isaac, welcome
13:06:38 <chaconpiza> thanks witek
13:07:05 <chaconpiza> alright, let's move to the first technical topic
13:07:17 <chaconpiza> #topic monasca-docker and zookeeper
13:07:31 <bandorf> OK, I'll try to explain quickly
13:07:49 <bandorf> We had a customer issue: disk space usage is grwoing for monasca-thresh.
13:08:26 <bandorf> chaconpiza's investigation has shown: Caused by snapshots and logs from zookeeper
13:08:47 <bandorf> Many steps of investigation later:
13:09:35 <bandorf> It looks like we're using in-process zookeeper in monasca-thresh for nimbus and supervisor (as mentioned in thresh log)
13:09:55 <bandorf> And, we're using zookeeper container for event_spout and metric_spout.
13:10:04 <bandorf> This sounds very strange to me.
13:10:15 <bandorf> Does anybody know a reason for this?
13:11:01 <bandorf> My guess: This happens unintentionally; ideally, only zookeeper container should be used.
13:11:33 <bandorf> However, I haven't figured out how to use zookeeper container for supervisor and nimbus
13:11:56 <bandorf> Any thoughts about this?
13:12:18 <witek> here is a note on using monasca-thresh with Storm
13:12:20 <witek> https://github.com/monasca/monasca-docker/tree/master/monasca-thresh#running-with-and-without-storm
13:12:34 <chaconpiza> The issue is easy to reproduce: Start the containers from monasca/monasca-docker and check the space used in /tmp inside the thresh container. It will slowly grow until no space left.
13:13:39 <bandorf> witek: OK, thanks, that might help! I will check for this
13:14:13 <chaconpiza> From Strom documentation: "It's critical that you set up a cron to compact Zookeeper's data and transaction logs. The Zookeeper daemon does not do this on its own, and if you don't set up a cron, Zookeeper will quickly run out of disk space. See here for more details." http://storm.apache.org/releases/current/Setting-up-a-Storm-cluster.html
13:14:20 <bandorf> Now, topic 2.6: I assume, nobody has come across the issue with huge disk usage?
13:14:21 <witek> I reckon it might be using localhost, because monasca-docker runs one-node installation
13:15:10 <witek> one more note on container configuration, you could compare the Kolla setup as well
13:15:20 <bandorf> Unfortunately, the functionality "purgeTxn..." (don't remember the name exactly, doesn't seem to be available in "in-process zookeeper"
13:15:58 <bandorf> Current approach: chaconpiza is going to write a script that deletes these files
13:16:46 <Wasaac> Can confirm monasca-thresh volume disk usage problem in kolla-ansible too
13:17:10 <witek> Wasaac: thanks
13:17:17 <bandorf> Aah, ok. Did you resolve the issue?
13:18:01 <chaconpiza> Wassac, or how do you workaround it?
13:18:08 <Wasaac> Have been looking at containers to mount and clean the volume
13:18:33 <Wasaac> But it would be nicer to approach it from the source of the issue
13:19:26 <bandorf> chaconpiza: Could you pls. quickly explain the solution with zookeeper 3.4.14 - autopurge?
13:20:03 <chaconpiza> Other way to workaround it for the case of https://github.com/monasca/monasca-docker  is to stop, remove containers, recreate containers and restart them.
13:21:09 <chaconpiza> bandorf: Sure, official zookeeper docker image 3.4.14 implements the autopurge with an env. variable
13:22:19 <chaconpiza> zookeeper 3.4.5 implements this code: https://github.com/krux/zookeeper-pkg/blob/3.4.5/src/java/main/org/apache/zookeeper/server/PurgeTxnLog.java
13:23:13 <chaconpiza> But as Matthias said, the reduced zookeeper provided by storm doesn't includes this part
13:24:55 <chaconpiza> Ok, we will keep you informed about this topic.
13:25:07 <bandorf> OK, thanks for your input anyway. Wasaac: We can stay in touch, regarding the temporary solution, we're going to provide. We can keep you updated.
13:25:17 <bandorf> OK, chaconpiza was faster :-)
13:25:32 <chaconpiza> :)
13:25:36 <chaconpiza> #topic kafka version in monasca-docker
13:26:08 <bandorf> Pls. check just the agenda. monasca-docker is stil using kafka 0.9. Is there any reason for this?
13:26:32 <witek> I think not, it just hasn't been updated
13:26:49 <bandorf> OK, thanks - topic done :-)
13:27:09 <witek> from what I recall we tested it in monasca-docker env
13:27:34 <bandorf> Yes, we executed system tests @est
13:27:45 <chaconpiza> #action Update kafka in monasca/monasca-docker
13:27:48 <witek> +1
13:27:57 <bandorf> +1
13:28:04 <chaconpiza> bandorf, would you like to own this action?
13:29:46 <chaconpiza> Let's continue with the last topic
13:29:49 <bandorf> I suggest, that we (EST) will take care of it. Assignment to an individual can be done later. But it will take time.
13:29:59 <chaconpiza> bandorf, thanks
13:30:12 <chaconpiza> #topic PTG planning
13:30:44 <chaconpiza> Kendall and Kendall contacted me to choose the option for the virtual PTG
13:30:56 <witek> haha
13:31:13 <witek> or short, Kendalls :)
13:31:14 <chaconpiza> one option was zoom and the second jitsi running on opendev
13:31:24 <chaconpiza> right, the Kendalls
13:31:53 <chaconpiza> I decided jitsi because it includes an etherpad
13:32:19 <chaconpiza> It will look like: https://meetpad.opendev.org/TestJitsiOnOpendev
13:32:45 <witek> do we have an option to have zoom as backup solution?
13:32:53 <chaconpiza> The final channel is not yet defined
13:33:21 <chaconpiza> #action Ask The Kendalls for a zoom backup channel
13:33:33 <chaconpiza> good point, I will ask them.
13:34:42 <chaconpiza> alright, concerning the content and topics for the vPTG
13:35:07 <chaconpiza> witek have already created this etherpad https://etherpad.opendev.org/p/monasca-ptg-victoria
13:36:30 <witek> I think it would be helpful to create Stories for each topic shortly describing the motivation and possible implementation direction
13:37:24 <chaconpiza> witek, do we have already the two time slots reserved? Tuesday June 2nd (7 - 8 UTC and 13 - 17 UTC)
13:37:32 <witek> yes
13:37:39 <chaconpiza> Great, thanks
13:38:20 <witek> we're in the Folsom room
13:38:26 <witek> https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/6e4619c416ff4bd19e1c087f27a43eea/www-assets-prod/ptg/PTG2020-Schedule.pdf
13:38:58 <witek> oh, I think all participants should register!
13:39:08 <witek> it's for free
13:39:24 <chaconpiza> +1
13:40:04 <chaconpiza> Wasaac feel free to add the topics you consider important
13:40:48 <Wasaac> chaconpiza: thanks
13:41:10 <chaconpiza> Wasaac do you know if Doug Szumski can join the PTG?
13:41:18 <bandorf> Is there a link to register?
13:41:34 <witek> https://www.openstack.org/ptg
13:41:35 <openstackgerrit> Merged openstack/monasca-common master: Bump sublibraries  https://review.opendev.org/730749
13:41:50 <bandorf> Thanks!
13:42:11 <witek> I think we could send an email to the mailing list advertising our sessions
13:42:51 <chaconpiza> #action advertising our sessions. I will take this action
13:42:52 <witek> also, there is the Automation SIG session on Thursday
13:43:29 <Wasaac> chaconpiza: I don't know what dougz schedule is like but he'll make it if able I'm sure
13:43:52 <witek> the agenda is here
13:43:53 <witek> https://etherpad.opendev.org/p/Automation-2020-VPTG
13:45:25 <witek> I've just noticed our Etherpad is not linked correctly
13:45:30 <witek> http://ptg.openstack.org/etherpads.html
13:45:30 <chaconpiza> Thanks for the SIG agenda
13:46:16 <witek> I can update it with openstackptg bot
13:46:22 <witek> instructions at the bottom
13:46:49 <chaconpiza> thanks
13:48:23 <chaconpiza> Ok, let's continue this week extending the Monasca PTG topics.
13:48:49 <chaconpiza> #topic AOB
13:49:27 <chaconpiza> I have a small technical question
13:49:46 <chaconpiza> In monasca-notification and monasca-persister still some traces of ujson
13:49:46 <chaconpiza> http://codesearch.openstack.org/?q=ujson&i=nope&files=&repos=openstack/monasca-agent,openstack/monasca-analytics,openstack/monasca-api,openstack/monasca-ceilometer,openstack/monasca-common,openstack/monasca-events-api,openstack/monasca-grafana-datasource,openstack/monasca-kibana-plugin,openstack/monasca-log-api,openstack/monasca-notification,openstack/monasca-persister,openstack/monasca-specs,openstack/mon
13:49:46 <chaconpiza> asca-statsd,openstack/monasca-tempest-plugin,openstack/monasca-thresh,openstack/monasca-transform,openstack/monasca-ui,openstack/python-monascaclient
13:50:18 <chaconpiza> do we need them?
13:50:52 <witek> it's only lower-constraints, right?
13:50:55 <chaconpiza> yes
13:51:20 <witek> no, should be dropped
13:51:49 <chaconpiza> #action remove ujson traces from notificatin and persister
13:51:59 <chaconpiza> witek, would you like to send the change?
13:52:05 <witek> oki
13:52:09 <chaconpiza> thanks
13:52:50 <chaconpiza> Ok, those are all topics from the agenda.
13:53:45 <witek> another AI for everyone
13:53:57 <witek> think of topics for PTG
13:54:11 <witek> already posted and new
13:54:54 <Wasaac> Will discuss with the StackHPC team
13:55:10 <witek> thanks
13:55:33 <chaconpiza> Thanks for joining today's meeting see you next week in the vPTG.
13:55:42 <Wasaac> o/
13:55:44 <witek> thanks chaconpiza
13:55:48 <witek> thanks everyone
13:55:53 <witek> see you next week
13:56:03 <chaconpiza> o/
13:56:05 <bandorf> Tnaks, good-bye
13:56:08 <chaconpiza> #endmeeting