Thursday, 2021-02-04

*** pojadhav|afk is now known as pojadhav03:57
*** dklyle has quit IRC04:06
*** ricolin has quit IRC05:28
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-tc05:33
*** ricolin_ has joined #openstack-tc05:45
*** lpetrut has joined #openstack-tc07:11
*** slaweq has joined #openstack-tc07:15
*** e0ne has joined #openstack-tc07:26
*** ralonsoh has joined #openstack-tc07:36
*** rpittau|afk is now known as rpittau07:39
*** cgoncalves has quit IRC08:31
*** cgoncalves has joined #openstack-tc08:33
*** tosky has joined #openstack-tc08:46
*** andrewbonney has joined #openstack-tc08:50
*** ricolin_ has quit IRC10:05
*** e0ne has quit IRC10:14
*** e0ne_ has joined #openstack-tc10:14
*** ricolin_ has joined #openstack-tc11:15
*** e0ne_ has quit IRC13:42
*** ricolin_ has quit IRC14:04
*** ricolin has joined #openstack-tc14:05
*** marios has joined #openstack-tc14:48
*** weshay|ruck has joined #openstack-tc14:57
*** belmoreira has joined #openstack-tc15:00
mnaser#startmeeting tc15:01
openstackMeeting started Thu Feb  4 15:01:14 2021 UTC and is due to finish in 60 minutes.  The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
*** openstack changes topic to " (Meeting topic: tc)"15:01
openstackThe meeting name has been set to 'tc'15:01
mnaser#topic rollcall15:01
*** openstack changes topic to "rollcall (Meeting topic: tc)"15:01
mnasero/15:01
gmanno/15:01
dansmitho/15:01
jungleboyjo/15:01
belmoreirao/15:01
*** eglynn_ has joined #openstack-tc15:02
mnaser#topic Follow up on past action items15:02
*** openstack changes topic to "Follow up on past action items (Meeting topic: tc)"15:02
*** rosmaita has joined #openstack-tc15:02
mnaser#link http://eavesdrop.openstack.org/meetings/tc/2021/tc.2021-01-28-15.00.html15:03
mnaserdiablo_rojo complete retirement of karbor15:03
* mnaser looks up review15:03
mnaser#link https://review.opendev.org/c/openstack/governance/+/76705615:03
mnaser#link https://review.opendev.org/c/openstack/project-config/+/76705715:04
mnaserthat seems to have landed15:04
mnaserany other tc members want to vote on the governance repo change and we can land that cange15:04
gmannyeah, few usage one is the only left one #link https://review.opendev.org/q/topic:%2522retire-karbor%2522+status:open15:04
gmannespecially system-config and openstack-map15:04
mnaserosf/openstack-map i think ttx can help with that15:05
mnaserand opendev/system-config maybe we can ask for fungi or clarkb help for that one15:05
gmannrecheck them to have zuul green15:05
gmannah we have rebase issue https://review.opendev.org/c/osf/openstack-map/+/767236/2/deployment_tools.yaml15:06
mnasertc-members: please review https://review.opendev.org/c/openstack/governance/+/767056 so we can land this15:06
gmanni can fix that quickly15:06
fungii'll take a look at the opendev patches for that now15:06
mnaserwe can remove that action item for the upcoming meetings, i think it's just an open review topic for now15:06
mnaserthank you fungi !15:06
gmann+115:06
mnasernext up was tc-members review and leave comments on the etherpad for SIG updates <= i think we can leave that for the discusion item later15:07
mnasermnaser continue to sync with tripleo team to reduce usage < also can be left for the discussion point later15:07
mnaserqa team to land parallelizing changes for devstack < we can probably leave this also for teh gate performance discussion point if y'all are okay with it?15:07
jungleboyj++15:08
mnaserok cool15:08
mnaser#topic Audit SIG list and chairs (diablo_rojo)15:08
*** openstack changes topic to "Audit SIG list and chairs (diablo_rojo) (Meeting topic: tc)"15:08
mnasertc-members: https://etherpad.opendev.org/p/2021-SIG-Updates -- please review this to help diablo_rojo and ricolin go over auditing those15:09
mnaseras well as the attached reviews15:09
mnaser#link http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019994.html15:09
mnaserok, time to get into the heavier subjects :)15:10
mnaser#topic Gate performance and heavy job configs (dansmith)15:10
*** openstack changes topic to "Gate performance and heavy job configs (dansmith) (Meeting topic: tc)"15:10
dansmithyo15:10
mnaseron my site, I reached out to the TripleO team and they most definitely are working on improving/reducing the footprint -- it was brought up in their meeting a few days ago15:11
dansmithso, I've been working (with gracious help from fungi and clarkb) on trying to figure out what metrics we can generate and use to try to quantify the relative weights of jobs in each project15:11
dansmithand the results are pretty interesting15:11
dansmiththe two worst offenders, by far, are tripleo and neutron and I too have had good response from both teams in my reaching out15:12
mnaser#link http://eavesdrop.openstack.org/meetings/tripleo/2021/tripleo.2021-02-02-14.00.log.html#l-3715:12
dansmithI've got a hacky tool to do some measurement and I'm planning an email to the list after this meeting to share some of the important numbers15:12
dansmithI hope that some awareness will yield some initial good-faith trimming of jobs15:12
gmann+115:12
dansmithI think the TC could also use this data and try to determine some target "weights" for jobs to try to keep things reasonable,15:13
dansmithbecause again this week, lag times have been pretty bad.. not horrific, but definitely more than a day to get anything useful done15:13
fungiin particular, i like that your tool can show the aggregate node utilization for a single build, which makes it more striking in side-by-side comparisons between different jobs what the relative costs for them are15:13
gmannalso this can be good thing to monitor in Tact SIG15:13
gmannsomething like weekly or monthly usage on ML15:14
fungigmann: yep, discussions so far have been in the context of the tact sig, in our irc channel15:14
gmann+115:14
dansmithyeah, I'd like to revisit this periodically and look at trends and identify things that make jobs suddenly worse15:14
dansmithbut we also really need to get people to work on fixing the spurious failures, of which we have a lot these days15:14
dansmithas well as turning an eye to improving efficiency of what we're testing15:14
mnaseron that note too, the tripleo team brought up a really good point which was -- having a target set for how much utilization they should have15:15
mnaser"you're using too much" -- "how much should we be using" is a very fair question to ask15:15
fungiyeah, i get the impression a lot of these jobs test mostly the exact same things, rather than only testing what other jobs aren't covering15:15
dansmithyeah, and this script gives us a way to weigh a patch, wehich we can then say "should be X instead of currently Y"15:15
dansmithfungi: yes, we really need to spend time making sure that's not too overlappy15:15
dansmithI think tripleo is probably better than most at only testing a smaller subset, especially after they make their proposed changes15:16
gmannand also combining the jobs when testing with same config can helpful . there are lot of jobs separated out per set of tests not config15:16
dansmithpartially because one run is so heavy that they're not inclined to add more of those15:16
dansmithbut nova has a ton of overlap,15:16
dansmithand I've already proposed some reduction there and more can be done15:16
dansmithso anyway, that's my status for the moment15:17
mnaseri do think it does still bring back a really good point of "how much usage should we be at?"15:17
mnaserprojects have historically never had any 'quotas'15:17
gmannfor tripleo question: can we set a cap for each project utilization ? it should be done based on nature of testing at project not equal to each project15:17
gmannyeah, that one15:17
dansmithgmann: we can't, currently15:18
dansmiththere is fair queuing across projects,15:18
dansmithbut projects like tripleo have many trees, and it's definitely not a quota15:18
dansmithI have to step away for something important right now, so we can move on15:18
gmannyeah but if we set current-working-time(after optimization when we say current is fine now) as quota so that it does not goes high in future or can be seen if it goes15:19
mnaserit looks like we do have some actionable things out of this which is reaching out to the ML and bringing this up and it seems within the tact sig there is work and efforts15:20
mnaserin order to try and watch what is happening15:20
dansmithwell, a target quota is good, but doesn't enforce anything, and enforcing quotas are not an option at the moment15:20
dansmithyup yup15:20
dansmithlook for my email tome soon15:20
mnaseryes, i agree, just a target that we can all agree on :)15:20
jungleboyjIt would seem having a benchmark so we see when a project is suddenly consuming more is at least a good starting point.15:20
jungleboyjRaise a flag before things get bad.15:21
gmannyeah15:21
mnaser#action dansmith reach out to ML regarding node usage15:22
mnaser#action tact-sig review/trend project usage to raise flags before things "get bad"15:22
mnaser#topic Dropping lower-constraints testing from all projects (gmann)15:22
*** openstack changes topic to "Dropping lower-constraints testing from all projects (gmann) (Meeting topic: tc)"15:22
jungleboyj++15:23
*** lpetrut has quit IRC15:23
mnaser#link http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019672.html15:23
mnaseri know there was a few alternatives proposed which were being implemented in nova15:23
gmannI am testing with direct deps only in nova and placement15:24
gmann#link https://review.opendev.org/q/topic:%22l-c-direct-deps-only%22+(status:open%20OR%20status:merged)15:24
gmannit is under progress as it is not yet green15:24
gmannonce I finish that then i can push result on ML15:24
mnasershould we keep this a topic to continue to discuss in the TC meeting?15:25
fungijust be prepared to rip those out when stable branches are created. if you're not constraining your indirect dependencies then they will change over time and become incompatible15:25
gmannfor stable, I think we should remove it15:25
gmannmnaser: we need to get consensus in TC but may be after we finish the discussion in ML15:26
mnaserok so i think it is still productive to keep it open15:26
gmannmay be we can drop it for now and once we are ready to discuss in TC then we can add15:26
mnaseri'm okay with that too15:26
gmannyeah either way is ok,15:26
rosmaitaso it sounds like the direction is keep the job in master only and for direct dependencies only?15:26
gmannrosmaita: I am testing that if that work fine or need more work. will post the result on ML15:27
rosmaitaok, great15:27
gmannlet me add test for cinder too15:27
rosmaitai have no objection to that idea15:27
gmannyeah that surly can reduce the maintenance  effort15:28
mnaserok cool, gmann: your call on keeping it on the discussion list15:29
mnasersorry, agenda rathert15:29
gmannok15:29
gmannI can drop/keep it based on ML discussion. like nothing to discuss then drop15:30
*** diablo_rojo has joined #openstack-tc15:30
gmannbefore next Wed15:30
mnaserok that sounds like a good idea15:30
* diablo_rojo sneaks in late15:30
mnaser#action gmann continue to follow-up on using direct dependencies for l-c jobs15:30
mnasero/ diablo_rojo15:30
mnaser#topic Mistral Maintenance (gmann)15:31
*** openstack changes topic to "Mistral Maintenance (gmann) (Meeting topic: tc)"15:31
mnaser#link http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020137.html15:31
* jungleboyj marks diablo_rojo tardy15:31
mnaserdidnt.. or does.. tripleo use mistral?15:31
gmannhumm do not know that.15:31
dansmithnot anymore I don't think15:31
dansmithcould be wrong15:31
fungii thought they did at one point15:31
fungibut yeah, i vaguely recall they stopped a while back15:31
mnaseri think they dropped mistral in favour of using ansible15:32
mnaseri.. think15:32
mnaserhttps://specs.openstack.org/openstack/tripleo-specs/specs/ussuri/mistral-to-ansible.html15:32
mnaserwhich was implemetned in ussuri-315:33
mnaser#link https://blueprints.launchpad.net/tripleo/+spec/tripleo-mistral-to-ansible15:33
mnaserso that was a big consumer that droped off15:33
mnaser#link https://review.opendev.org/q/project:openstack/mistral15:33
mnaser~100 changes or so in the past year15:33
gmannother project using it is Tacker which is optional integration afaik and work without mistral too. As discussed with one of the Tacker member they will discuss it and see what they can do, either help in mistral or deprecate the support.15:34
mnaseri think it will become more tricky when we'll have to figure out a ptl for the project15:34
mnaserbut maybe a distributed leadership model could work because it is a very low volume project right now15:34
gmannyeah Renat  mentioned he will not be able to run as PTL in next election15:35
gmannmnaser: but do we have any other maintainer there other than Renat ?15:35
gmannmay be there are.15:36
mnaseri see some patches by `ali.abdelal@nokia.com`15:36
mnaserhttps://review.opendev.org/c/openstack/mistral/+/75548115:36
mnaserand i see Eyal too making reviews15:36
gmannyeah15:36
mnaser#link https://review.opendev.org/admin/groups/19588f2c593d37749e32bc5400365f654c683d19,members15:36
gmannDistributed leadership will be helpful but let's see15:36
belmoreirado we know the user adoption of Mistral?15:37
gmannother thing i was thinking to add it in newsletter to convey about it need maintainer...15:37
gmannbelmoreira: good point.15:37
mnaserbelmoreira: that's a good question, maybe user survey data can help with this15:37
fungimight be able to figure out some percentage of user survey respondents who claimed to be using it?15:38
mnasergmann: maybe we can reach out to mistral-core before that15:38
fungijinx15:38
gmannone I know is via Tacker in NEC/NTT japan which we are checking option to go woth non-mistral way15:38
belmoreiramnaser *I'm trying to find out*15:38
gmannmnaser: +1,15:38
* mnaser looks at aprice 15:38
mnaseroh wait i'm mblind, there is a 'projects used in production deployments'15:39
mnaserunder 'deployments' https://www.openstack.org/analytics15:39
fungiit's amazing how she answers your questions before you even know you have them15:39
mnaserso this shows .. i think .. 8% proudction, 4% testing, 13% interested15:39
gmannyeah15:40
belmoreirathis is 2019 data it will great to get the latest15:40
belmoreiramaybe aprice can help15:40
fungiall it shows to me is that we need to drop a few projects so that graph won't be unreadably crowded ;)15:40
apriceo/15:40
jungleboyjfungi:  ++15:41
belmoreirafungi :)15:41
apricein another meeting simultaneously - is there a tl;dr on the current user survey request?15:41
mnaseraprice: 2020 data :)15:41
gmannfungi: or divide into separate category based on usage like high vs low or so?15:42
apriceah yes, i didnt realize that wasn't on o.o/analytics yet. i can work with jimmy to get that live soon15:42
apriceprobably in the next 1-2 days15:42
mnasercool, awesome!15:42
apricebut if there's a specific data point yall need sooner, lmk15:42
mnaseraprice: % of usage of mistral in prod environments15:42
apricemnaser: ok - i can pull that in like an hour or so and drop it back in this channel15:42
mnaseraprice: yay thank you!15:42
apricenp!15:42
mnaserso we'll have that info, anyone would like to volunteer to reach out to the current maintainers?15:43
gmannfor 'you need help' ?15:43
mnaseryes, to try and see if any of them would like to run into ptl next release15:44
gmannok15:44
mnaserso we can try to get ahead of the troubled waters ahead :-)15:44
gmanni can do that15:44
mnaserthanks gmann15:44
mnaser#action gmann reach out to mistral maintainers about PTL for next cycle15:44
ricolingmann, let me help you with that too15:44
mnaser#undo15:44
openstackRemoving item from minutes: #action gmann reach out to mistral maintainers about PTL for next cycle15:44
mnaser#action gmann/ricolin reach out to mistral maintainers about PTL for next cycle15:44
mnaser:>15:44
gmannsure15:44
mnaserso with the combination of anyone stepping up + usage in the wild as suggested by belmoreira, we should be able to take a decision with the project's direction15:45
gmannyeah15:45
mnasercool, thanks for bringing up gmann15:46
mnaser#topic Open Reviews15:46
*** openstack changes topic to "Open Reviews (Meeting topic: tc)"15:46
mnaser#link https://review.opendev.org/q/project:openstack/governance+is:open15:46
mnaseri'm going to go over a few that need to be merged noew15:47
mnaserhttps://review.opendev.org/c/openstack/governance/+/77061615:50
mnaserthis really could use more votes15:50
mnaserhttps://review.opendev.org/c/openstack/governance/+/773383 also can help the osa team land things15:50
gmanndone15:52
jungleboyjDone.15:52
jungleboyjI want to read the cool down one again.  Will do that.15:52
*** tdasilva_ has joined #openstack-tc15:53
*** rosmaita has quit IRC15:53
mnasercool, dansmith had some concerns too, so appreciate a review on https://review.opendev.org/c/openstack/governance/+/770616 when possible15:54
jungleboyjDone.  I think the wording is positive enough while communicating our reasons.15:54
dansmithyeah I don't think my concerns were addressed, but they weren't super important15:55
jungleboyjDon't quit, but work on what you want to do.15:55
*** thiago__ has quit IRC15:55
mnaserdansmith: i'm happy to revise if you want if you feel strongly about the concerns :)15:56
dansmithI know, I'll try to hit it again after this15:56
mnasercool, no problem15:56
mnaserand also some of those applications for projects that have historically been very interop-y -- https://review.opendev.org/c/openstack/governance/+/773684 (cinder) and https://review.opendev.org/c/openstack/governance/+/773090 (neutron)15:57
mnaserthose cant land yet but earlier reviews is helpful15:57
mnasergmann: this might be at a standstill https://review.opendev.org/c/openstack/governance/+/771785 -- i think we'll have to ping the monasca team to properly deprecate it15:58
*** openstackgerrit has joined #openstack-tc15:58
openstackgerritMerged openstack/governance-sigs master: Add Dirk Mueller as Packaging SIG co-chair  https://review.opendev.org/c/openstack/governance-sigs/+/77288115:58
gmannyeah, I will check with them or if they need help15:58
mnasercool, awesome, we wer eable to clear out most of our queue15:58
gmann\o/15:59
ricolin:)15:59
mnaserthanks for coming everyone!15:59
openstackgerritMerged openstack/governance master: Add rbd-iscsi-client to cinder project  https://review.opendev.org/c/openstack/governance/+/77259715:59
mnaservery productive15:59
mnaser#endmeeting15:59
*** openstack changes topic to "OpenStack Technical Committee office hours: Tuesdays at 09:00 UTC, Wednesdays at 01:00 UTC, and Thursdays at 15:00 UTC | https://governance.openstack.org/tc/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-tc/"15:59
openstackMeeting ended Thu Feb  4 15:59:54 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tc/2021/tc.2021-02-04-15.01.html15:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tc/2021/tc.2021-02-04-15.01.txt15:59
diablo_rojoThanks mnaser!16:00
openstackLog:            http://eavesdrop.openstack.org/meetings/tc/2021/tc.2021-02-04-15.01.log.html16:00
ricolinthanks mnaser16:00
gmannthanks16:00
gmanndiablo_rojo: ttx resolved the merge conflict in retire-karbor openstack-map patch, its green now - https://review.opendev.org/c/osf/openstack-map/+/76723616:00
jungleboyjYay!  Thanks mnaser !16:00
openstackgerritMerged openstack/governance master: Define 2021 upstream investment opportunities  https://review.opendev.org/c/openstack/governance/+/77170716:00
diablo_rojoThanks ttx!16:00
openstackgerritMerged openstack/governance master: Define Xena release testing runtime  https://review.opendev.org/c/openstack/governance/+/77086016:00
openstackgerritMerged openstack/governance master: Remove Karbor project team  https://review.opendev.org/c/openstack/governance/+/76705616:01
openstackgerritMerged openstack/governance-sigs master: Update automation and multi-arch SIGs  https://review.opendev.org/c/openstack/governance-sigs/+/77248316:02
openstackgerritMerged openstack/governance master: Create ansible-role-pki repo  https://review.opendev.org/c/openstack/governance/+/77338316:06
openstackgerritMerged openstack/governance master: [manila] add assert:supports-api-interoperability  https://review.opendev.org/c/openstack/governance/+/77085916:07
*** dklyle has joined #openstack-tc16:39
ttxthanks gmann !17:00
*** marios is now known as marios|out17:15
*** openstack has joined #openstack-tc17:24
*** ChanServ sets mode: +o openstack17:24
*** marios|out has quit IRC17:28
*** bodgix has quit IRC17:35
*** bodgix has joined #openstack-tc17:36
*** rpittau is now known as rpittau|afk18:10
*** timburke_ is now known as timburke18:15
*** ralonsoh has quit IRC18:18
apricemnaser: (and others who were asking) - here is the mistral data for production environments in 2020: 5% in production, 6% testing and 21% interested18:50
apriceive put in a ticket to publish the 2020 data on openstack.org/analytics and can let you all know when that's live18:50
apricefor now, im happy to pull any data yall need. and mnaser, please tag anyone who may be looking for that :)18:50
*** belmoreira has quit IRC19:13
*** andrewbonney has quit IRC19:20
jungleboyjdansmith:  Good note on the resource usage.  Thanks!19:22
dansmiththanks19:22
*** belmoreira has joined #openstack-tc19:25
*** belmoreira has quit IRC19:34
mnaseraprice: thank you!19:59
*** lbragstad_ has joined #openstack-tc22:22
*** lbragstad has quit IRC22:24
*** tkajinam has quit IRC22:59
*** tkajinam has joined #openstack-tc22:59
*** slaweq has quit IRC23:03
*** lbragstad_ is now known as lbragstad23:03
*** tdasilva_ has quit IRC23:21
*** lbragstad has quit IRC23:21
*** tdasilva_ has joined #openstack-tc23:22
*** lbragstad has joined #openstack-tc23:23

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!