08:00:02 #startmeeting large_scale_sig 08:00:03 Meeting started Wed Jun 24 08:00:02 2020 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:00:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 08:00:06 #topic Rollcall 08:00:07 The meeting name has been set to 'large_scale_sig' 08:00:14 Who is here for the Large Scale SIG meeting ? 08:00:23 hello! 08:00:42 belmoreira: ping 08:00:53 o/ 08:01:06 ttx: thanks for the ping 08:01:09 huyupeng: hi! 08:01:27 hello 08:01:46 huyupeng: first time joining? 08:01:51 yes 08:02:01 \o 08:02:20 huyupeng: care to introduce yourself, and let us know what you're interested in? 08:03:40 Or we can jump right in today's meeting agenda: 08:03:43 #link https://etherpad.openstack.org/p/large-scale-sig-meeting 08:04:07 ok, i am intersted in concurrency, resource utilize , data plane performance when we deploy large scale openstack cloud 08:04:41 huyupeng: sounds like you found the right group to discuss it. I think those are concerns that the group shares 08:04:59 #topic 'Large scale operations' OpenDev event next week 08:05:04 #link https://www.openstack.org/events/opendev-2020/ 08:05:09 Are y'all ready for the event next week? 08:05:34 Remember that our goal is to start new discussions, and motivate new organizations to engage publicly 08:05:48 The Large Scale SIG should be used to continue those discussions, we don't want this to be a one-off thing without follow-up 08:06:00 So the idea will be to heavily promote the SIG during all those discussions 08:06:08 We should point to our wiki page 08:06:14 #link https://wiki.openstack.org/wiki/Large_Scale_SIG 08:06:24 We should also point to our next meeting, July 8, 8utc on #openstack-meeting-3 08:06:34 And also to the meeting etherpad for people to add items to the agenda. 08:06:47 belmoreira: Anything else we need to prepare before next week event? 08:07:23 we have the "Upgrades" session on Monday. It would be great to have all your experience there for discussion 08:09:07 Also don;t forget to register, it's free 08:09:15 i care about how rabbitmq behavie when there are a lot client publish message and receive message. how to tunning the rabbitmq 08:09:41 sorry got disconnected... not sure if you got my msg 08:09:45 we have the "Upgrades" session on Monday. It would be great to have all your experience there for discussion 08:09:56 belmoreira: ack 08:10:11 huyupeng: we all struggle on rabbitmq tuning :p 08:10:16 huyupeng: we identified this as a scaling bottleneck too. We are working to instrument oslo.messaging to get better data 08:10:30 in a new library called oslo.metrics 08:10:35 we'll talk about it in a minute 08:10:42 #topic Progress on "Documenting large scale operations" goal 08:10:46 #link https://etherpad.openstack.org/p/large-scale-sig-documentation 08:10:56 For the newcomers, this is one of our current goals for the SIG - produce better documentation to help operators setting up large deployments. 08:11:09 In particular, documenting better configuration values when you start hitting issues with default values at a certain scale 08:11:19 amorin has been leading that effort 08:11:24 I havn't be able to move forward on that topic in the past 2 weeks 08:11:39 OK I'll push the action item back 08:11:43 so if we can copy back my action, 08:11:43 #action amorin to add some meat to the wiki page before we push the Nova doc patch further 08:11:44 thanks 08:11:54 We also had a TODO for all to describe briefly how you solved metrics/billing in your deployment 08:11:58 But not much contribution to that yet 08:12:16 completely forgot about that 08:12:18 huyupeng: one objective of the SIG is to share experience 08:12:31 do we have a etherpad to fill? 08:12:33 like how various groups solved common problems 08:12:46 I added it into https://etherpad.openstack.org/p/large-scale-sig-documentation 08:12:53 ack 08:12:56 feel free to drop a line there 08:13:10 I'll keep that work item open too 08:13:17 #action all to describe briefly how you solved metrics/billing in your deployment in https://etherpad.openstack.org/p/large-scale-sig-documentation 08:13:56 There seems to be a lot of custom solution in that space, which points to shortcomings in the openstack services offering 08:14:02 Anything else on this topic? 08:15:08 #topic Progress on "Scaling within one cluster" goal 08:15:11 #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling 08:15:21 For the newcomers, this is the other current goal of the SIG - identify, measure and push back common scaling limitations within one cluster 08:15:31 Like RabbitMQ starting to fall off first 08:16:14 One key effort within that goal is to work on bringing LINE's work on instrumenting oslo.messaging into upstream OpenStack 08:16:22 I had a TODO on solving merge conflict and push initial oslo.metrics code in 08:16:28 It's now in at: https://opendev.org/openstack/oslo.metrics/ 08:17:01 It's mostly at POC stage, but would be great if this group could look it up and tell us if that sounds like a good idea, and how to improve it 08:17:06 We also had a bunch of patches in to make the code more OpenStacky, thanks to hberaud 08:17:13 smcginnis and I did the reviews. 08:17:26 I think for the more feature-oriented reviews coming in the future, it would be good to have one of the original authors from LINE in the core review team for oslo-metrics 08:17:42 I'll try to see with masahito if he would be interested in being added 08:18:02 The next step will be to add basic tests so that we are reasonably confident we do not introduce regressions 08:18:18 I suspect masahito will mention it next week during the Opendev event 08:19:44 Anything else on that topic? 08:19:59 does oslo.metrics has something to do with rabbitmq tunning ? 08:20:30 yes, let me point to teh presentation LINE did at the Summit in Shanghai 08:21:14 https://www.openstack.org/videos/shanghai-2019/how-we-used-rabbitmq-in-wrong-way-at-a-scale 08:21:30 That will give you the background 08:21:46 #topic Large Scale SIG slide for OpenStack 10th birthday campaign 08:21:52 Helena on the OSF staff reached out to SIG with the following information: 08:21:53 thanks, i will see later 08:21:58 "OpenStack will be turning 10 years old in less than a month! We are sending out a slide deck to the event organizers who are organizing the 10 years of OpenStack celebration. For this slide deck, is there anything about the Large Scale SIG you would like to include?" 08:22:06 I was thinking we could make a slide with high-level goals for the SIG and pointers to our meeting time 08:22:14 Anyone interested in working on that? Or I can make a draft and propose it next meeting 08:22:59 #action ttx to produce a draft of the 10th birthday slide for the SIG 08:23:52 huyupeng: in the etherpad links I provided above, you will find other ways to get involved in those SIG goals, like providing scaling stories 08:24:32 Scaling stories are accounts of scaling experiences in your deployment -- what broke first, how did it break, and how you moved past it 08:24:36 ok,copy that 08:24:51 We hope to find common pain points this way (RabbitMQ being an obvious one) 08:25:13 We document those in https://etherpad.openstack.org/p/scaling-stories 08:25:28 and then copy them to https://wiki.openstack.org/wiki/Large_Scale_Scaling_Stories for long-term storage 08:25:54 We hope to have more starting next week after the Opendev event! 08:26:00 #topic Next meeting 08:26:08 Next meeting in two weeks... That's the week just after the Opendev event, so if we get contacts of people excited to join the SIG, we can strike that iron while it's hot. 08:26:18 #info next meeting: Jul 8, 8:00UTC 08:26:23 ok 08:26:32 Anything else before we close ? 08:27:04 If there is a topic you want to discuss next time, make sure to add it in advance to the meeting agenda on https://etherpad.opendev.org/p/large-scale-sig-meeting ! 08:28:22 Alright, if nothing else for today, have a great day, and hope to see you all virtually at Opendev next week 08:28:33 Thanks everyone! 08:28:55 #endmeeting