15:01:51 #startmeeting tc 15:01:52 Meeting started Thu Jun 14 15:01:51 2018 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:53 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:56 The meeting name has been set to 'tc' 15:01:56 tc-members office hours 15:01:57 #topic Office Hour 15:02:11 Been spending a few cycles this afternoon reviewing https://docs.openstack.org/project-team-guide/ 15:02:13 we've been using "tc" because that's what the index on eavesdrop links to for our office hour logs 15:02:31 One things that it's missing is a plug for basic design tenets, like the fact that you can assume teh base services will be around 15:02:48 Anyone interested to etherpadding that with me ? 15:03:15 o/ 15:03:28 cdent/zaneb: that could address some of the things we wanted to add to technical vision too 15:03:36 yeah, agree 15:04:07 o/ 15:04:11 I can't help etherpadding today but either can help tweak, or participate tomorrow afternoon/monday 15:04:23 sure, i've had my head in the base services space recently, happy to spitball some content for that 15:04:31 Will be at https://etherpad.openstack.org/p/project-team-guide-basic-design-tenets-chapter 15:05:43 in related news, the technical vision thing is getting closer to the top of my to-do list 15:05:58 I know y'all will be excited to hear this update about the state of my to-do list 15:06:05 :) 15:06:13 it's the only thing I think about 15:07:31 If we've got nothing else in particular to talk about today, I'd like to selfishly draw people's attention to jay's rant in http://lists.openstack.org/pipermail/openstack-dev/2018-June/131498.html 15:07:44 zaneb: couldn't sleep, waiting for it 15:07:53 that's a long standing nova problem that we keep talking about in a sideways fashion 15:07:56 but it never goes away 15:07:59 zaneb: you have the etherpad handy right? 15:08:12 (not talking in this instance about the number of cores, but rather the volume of change in a complex setting) 15:08:51 EmilienM: always, I keep it next to my bed. https://etherpad.openstack.org/p/tech-vision-2018 15:09:41 cdent: that was a good read. i'd love to see nova do something like split into smaller teams/packages for drivers for example 15:09:41 cdent: I missed that one. 15:10:03 mnaser: yeah. like jay's comments, your response is not new :) 15:10:04 nova-libvirt, nova-vmware, etc. rather than having the entire tree handled by the core team for changes that may not really touch the internals 15:10:04 o/ 15:10:24 It really does seem like nova should be decomposed somehow. Whether that's structurally or just process/team-wise. 15:10:29 mnaser: which I think means it is probably the right thing to do, but it never happens 15:10:55 can gerrit work with some sort of acl that you can only vote on files within a specific path? 15:10:58 (as a start) 15:11:40 @all did everyone had a chance to put their names on https://wiki.openstack.org/w/index.php?title=OpenStack_health_tracker ? 15:11:52 I think dhellmann wanted to do it for today 15:12:01 mnaser : unfortunately, no. we have had to rely on teams to trust each other for that level of granularity 15:12:18 yep, i did EmilienM 15:12:28 EmilienM: I put my name on a few but wanted to go back later to do more once others had a chance to grab some. 15:12:42 dhellmann: i see. it does seem a lot of work to split those things out 15:12:50 and affects a lot of downstream things 15:13:17 cdent: as an outsider to Nova, it's bewildering/frustrating that everyone inside Nova understands perfectly the problem and exactly how to fix it, and they still won't do it. for many years now. 15:13:32 i think it's more of a lack of resources 15:13:39 the placement work is a step in that direction, isn't it, zaneb ? 15:13:52 ++ 15:14:05 zaneb: I think it depends on how your define "inside", for one thing 15:14:06 mnaser: in tripleo, we use "Trust" (crazy right?) to make Gerrit ACLs 15:14:07 dhellmann: yes, that's true 15:14:09 But even with that, there seems to be some reluctance to actually split out placement. 15:14:11 "still won't do it" is pretty strong ... current experiments are the runway and the placement possibly moving out 15:14:18 mnaser: we promote people "core" on certain files/topics in tripleo 15:14:24 mnaser: and you know what? it works :( 15:14:28 But also the splitting is _hard_. nova is very coupled with itself, with a wiggle over here cascading to impact way over there 15:14:54 looking in from the outside it seems nothing has changed, but when I talk to the nova team I learn about things that are happening. I think it's just going more slowly than anyone likes. Which makes me wonder if we can somehow give them support to say "no" to feature requests that are not related to decomposing the project, to speed things along. 15:14:55 there was a previous failed attempt to move placement out too though, wasn't there? 15:14:59 smcginnis : i was in the placement break out session, there's work to be done to split it out. need hands now i think 15:15:01 mnaser: (not trying to compare, but giving an example where we also wanted Gerrit ACLs on files) 15:15:06 as far as I can tell the reason that placement extraction is happening at all is because a) i'm stubborn, b) i've been exceptionally lucky to have three employers in a row who let me do much of what I want. Other people are not so lucky 15:15:08 (not suggesting this one is going to fail) 15:15:23 EmilienM: i think that can be something that would be proposed "you are a nova-vmware-core, while you can +2 everything, you can only +2 vmware changes" 15:15:26 cdent: i saw that discussion beginning to build, and can't say i'm surprised at the discovery that you can't force people to review what you identify as important (i think all current/former ptls probably know this already) 15:15:27 zaneb : my understanding is that previous approach attempted to forklift it without a transition plan, but I may be mixing up bits of history there 15:15:28 ++ cdent 15:15:53 there was gantt which was sort of different 15:16:02 fungi : totally (jay's email from this morning) 15:16:32 and extracting placement won't really change anything other than bring things back to their 2016 state and these coversations go back further than that (recall dan berrange's giant thread in 2014) 15:16:46 * TheJulia feels like we're also semi-rehashing some of the discussions from yvr and wonders what the next steps would be to promote trust 15:16:59 TheJulia: agreed 15:17:11 I still hope they split functionality into separate review domains before they run out of "supercores" (the limited number of Nova cores that have the full picture in their brains and can really approve changes) 15:17:21 cdent: yeah, that thread is what I keep thinking back to 15:17:25 we have several different proposed solutions, each of them taking a different approach. Which one does the nova team support? 15:17:33 because it's hard to raise new ones 15:17:37 yeah, TheJulia, my intent here was not to really address the problem here, just draw people to the thread 15:18:26 Yeah, I hate to say it, but the only way movement is going to be made on the various issues is to toss concepts against a wall and see what sticks (for lack fo a better term to use) 15:18:33 ttx: yes, supercores are expiring 15:18:40 some of those concepts might be code, and from there sell everyone on it 15:18:57 cdent: That sounds grim. :) 15:19:09 * cdent is a grim guy 15:19:09 cdent: well their jobs aren't getting any easier as time passes 15:19:18 tru 15:20:08 what if we applied the maintenance-mode tag to nova? 15:20:18 why don't we add another 'voting' category in gerrit 15:20:24 cdent: no, your realistic. There is a huge difference :) 15:20:33 just like we have code review and rollcall-vote 15:20:36 dhellmann: that is an interesting idea. It basically maps to what jay is saying. 15:20:40 mnaser: I wanted to discuss next steps on the diversity thing with you, maybe off meeting though 15:20:57 dhellmann: I don't think that would solve anything 15:21:12 we can have a +2 in "domain-core" and then +2 in "super-core" and only a super-core can +W 15:21:31 so a supercore doesnt have to do a very deep dive if he knows that it doesn't touch anything beyond that specific domain 15:21:37 mnaser: I fear that would insitutionalize the wrong approach 15:21:47 do we expect some of the starlingx's nova folks to show up anytime soon? 15:21:56 dtroyer ^^ 15:22:01 You can't extend trust beyond a certain number anyway 15:22:11 yeah, I don't want us to build more complicated voting solutions to deal with the fact that we have a highly coupled system. we should look for ways to help the team have the space to fix the code. 15:22:12 mnaser: technical solution to a social problem 15:22:20 but i don't think trust scales 15:22:26 btw, talking about this amongst ourselves is probalby the wrong approach. we should at least invite melwitt, mriedem, dansmith, jaypipes into this instead of speculating 15:22:32 The only solution is to split into separate trust domains 15:22:48 i.e review domains 15:22:58 But that involves letting go of some control 15:23:07 (some would say some overall quality) 15:23:10 * cdent in neither core nor supercore 15:23:12 which, as they've noted, is hard with nova because of a lot of tight coupling between various parts of the service 15:23:25 ttx: exactly. nothing scales, and that's why we need to avoid projects that are Too Big To Fail 15:23:46 based on what jay said in that thread, and I've heard from other folks, the rate of change in the code makes it difficult for them to change processes or tools or have time to grow the team. can we address that rate of change in the code somehow? 15:23:50 (ttx: yes, let's talk about organizational diversity after if you have sometime) 15:24:10 dhellmann: saying nova is in maintenance mode isn't going to slow things down 15:24:18 if that's what you're suggesting 15:24:30 yeah I think it would only have negative effects 15:24:32 dansmith : yeah. I'm throwing out ideas to find ways to give you options to say "no" 15:24:43 dhellmann: I say no a lot :) 15:24:44 mnaser: trust scales, when you setup the right culture 15:24:55 dansmith : heh, ok 15:25:03 mnaser: https://review.openstack.org/#/admin/groups/190,members 15:25:15 EmilienM: it's really difficult. 15:25:21 cdent: You're a super non-core. ;) 15:25:22 dhellmann: but I agree, I'd like more people to say no more often.. I don't think artificially labeling nova as in maintenance mode really does anything though 15:25:26 One would argue that this group is tied by external trust 15:25:28 I don't think I said it was easy :) 15:25:30 I said it's possible 15:25:34 EmilienM : check number of redhat.com on that :) 15:25:34 Like read the email domain 15:25:43 it's not about redhat come on 15:25:55 I don't care in Gerrit if this reviewer is redhat or not 15:25:56 ;p 15:26:07 I'm just saying the trust is sustained by external constructs 15:26:07 what I care is who is doing it, and the knowledge/capacity to review 15:26:29 ttx: while i agree with you, i think we can bring those external constructs to the open regardless of the org 15:26:34 It's really difficult to trust someone to review the way you would have reviewed. Works if you have enough shared culture 15:26:46 That shared culture can come from company culture too 15:26:47 EmilienM : there's extra processes / hierarchy that helps too 15:26:48 "if i merge things in the wrong place, it will make my peers (other nova devs/cores) unhappy and i might lose my core" 15:26:52 based on the *many* conversations I've had with the nova team about this, I do not think the issue is just "trust". Nova *is* complex. Learning that takes time. Investing in reviewers to build them to core takes time. 15:27:15 Dunbar tribe bumber 15:27:16 dansmith: you seem to be the only one whos mostly around.. are most changes going in specific domain? 15:27:19 number* 15:27:20 the fact tripleo is redhat only doesn't mean it's easy for us to trust each others; but it's in our (project) culture to setup trust and we move forward 15:27:28 mnaser: we've talked about the domain core thing so many times 15:27:34 so instead of saying the nova team should already have fixed this by just trusting more people isn't really digging into the underlying problems 15:27:36 EmilienM: and yet most other team stalls around 15 in core teams 15:27:36 dims: like all projects 15:27:42 right 15:27:43 mnaser: and I really really think that it's not that easy 15:27:59 dansmith: i understand but i'm just trying to get a grasp if things mostly seem to be going in one direction 15:28:00 believe me or not it's not just because it's a redhat only project. 15:28:08 aka, is it a lot of the libvirt stuff that moves most of the time and others are mostly idle 15:28:11 EmilienM : i trust you on that 15:28:16 I was PTL for Puppet OpenStack and we promoted a bunch of non redhat core 15:28:30 we were diverse during some time (well not anymore) 15:28:34 yeah, it involves trust. and sometimes people break things. and you tell them, and they learn :) 15:28:52 and sometimes even long term cores will break things when missing something, and i think that's *ok* 15:28:53 EmilienM: agree that there are things you can do to set up project culture that allow to scale to larger groups 15:28:57 EmilienM : question is what did we learn from there that can be applied to nova 15:29:01 anyway I was not trying to give a lesson on something, just giving some examples based on my personal experience. 15:29:28 But it still generally stalls at some point, unless sustained by some other cultural alignment 15:29:29 dansmith : are there things the nova team wishes they could do (or maybe just you wish) that the TC could help you with to address the situation? 15:29:45 i'm around, but not engaging... 15:29:46 ++ ttx 15:30:09 mriedem : I'll ask the same question of you then :-) 15:30:15 Tossing in a disruptive thought into this track. People don't want to give up control in many cases because of responsibility or perception of responsibility. Fear of making a mistake, if that is reduced, would people not be able to be more trusting because they wouldn't have to have their guard up constantly? 15:30:19 Like if you set up a "ask for forgiveness, not for permission" culture you can grow review teams from 15 to ~25 15:30:23 * dims drags mriedem to the table 15:30:54 ttx: maybe this links to the idea of "master should be CD-able" 15:31:01 could we, maybe, stop rehashing the same arguments about this we've had for 8 years and talk to the folks directly involved about the help they need? 15:31:13 mnaser: good point yes 15:31:17 ah, yes, good point dhellmann. 15:31:31 dhellmann: I don't really think the TC is going to fix anything like this from the outside, I guess is my summary 15:31:34 since mriedem and dansmith are actually here 15:31:41 ++ dhellmann 15:31:49 ttx: yes, that's the kind of culture I'm talking about 15:31:49 dansmith: agree with you on that 15:31:51 * mriedem remembers barcelona 15:31:55 dansmith : let's not assume that. If you could change something, what would that be? 15:31:57 and atlanta 15:32:01 and every retrospective 15:32:12 dhellmann: let's not assume what? 15:32:17 that the tc can't help 15:32:23 assuming the tc can help, what do we need? 15:32:39 dansmith : that no one outside the team can help 15:32:40 or the tc in particular 15:32:52 mirantis and osic layoffs not happening? :) 15:33:25 once placement splits out it should help some, but the core team to start is still likely going to be similar to what it is now 15:33:34 i don't know how that works when cinder split out, i wasn't around that early 15:33:44 *worked 15:33:53 mriedem: should be easier to become core on placement than to become core on Nova I guess ? 15:34:08 yeah 15:34:10 cinder was an obvious piece that could be cleaved off, 15:34:10 it's a much smaller footprint 15:34:15 is the neutron project a good example as well? afik they've split a bunch of things as well 15:34:16 and not having to review tons of nova-network features reduced our workload for sure 15:34:21 is there another obvious piece, after placement? 15:34:22 so maybe the teams would start to diverge soon enough (hope) 15:34:51 placement hasn't yet reduced the amount of complexity or code in the rest of nova, but will eventually 15:35:10 looks like dtroyer is not around to say something about nova folks from starlingx ... 15:35:12 but not by as much asthe volume or network splits 15:35:51 I still have that hope that hypervisor drivers could be split off. If teh API there could be declared stable 15:35:59 Sounds domain-specific enough 15:36:15 * mnaser doesn't know if there is enough work on other drivers that is affecting cores that much 15:36:15 we certainly could do that, 15:36:23 But I know we've been chasing that stabel API for a long time and nobody really has time to work on it 15:36:34 but hypervisor drivers are way more integrated and complicated than network drivers and volume drivers 15:36:49 ttx: we're actively changing right now.. it's always changing.. substantially 15:37:07 what part of the code causes the most review burden? 15:37:14 it's not that nobody is pursuing a stable virt api, it's that it's something that is pretty tightly coupled and complex 15:37:24 runways has helped focus efforts on getting non-libvirt driver blueprint changes in 15:37:30 xen and powervm got several bps merged in rocky b/c of runways 15:37:38 yup 15:37:44 xen for the first time in a while 15:37:44 cool. 15:37:45 so runways is a new thing 15:37:54 i thiink pike saw a lot of ironic driver features get in 15:38:19 I think we had also been working on those for >1 year 15:38:31 TheJulia: also lots of those were blocked on getting the ironic stuff done first 15:38:35 * EmilienM likes the name "runway" fwiw, it reminds me "checkpoint" that we use in my internal team, which is a similar process that we're doing 15:38:39 so it's not like nova was just sitting there 15:38:45 indeed 15:38:45 TheJulia: and to the point, those changes were tied to the virt driver interface, compute manager, scheduler, and placement 15:38:55 EmilienM: of course you like the name "runway" 15:38:57 so you know, they were complicated to get right and land without breaking people too much 15:39:06 ttx: ;) 15:39:08 I like runways in general, always better than a field :P 15:39:08 dansmith: completely agree 15:39:20 * EmilienM puts his pilot hat 15:39:20 I'm much less interested in having things go faster than I am in making it so the team doesn't burn out. 15:39:21 also, nova tends to ask for actual CI coverage of features when possible if they are big enough, and that delays things 15:39:26 i keep reading runway as runaway! 15:39:31 dansmith: I was only providing context for the discussion in that some of these things take a long long time 15:39:38 dhellmann: okay I want to revise my answer 15:39:44 dims: just don't runaway on a runway plz 15:40:01 Yeah, my main concern at this point is losing the last supercores faster than we can decompose to simplify their work 15:40:07 dansmith : I'm all ears 15:40:10 queens, as noted in our retrospective at the ptg, was much less hectic in part b/c we had fewer approved blueprints to focus on 15:40:17 dhellmann: one thing that really makes me feel like ragequitting is feeling like everyone is complaining all the time about us merging a ton of code, but not enough code. I would love to feel like the TC was supporting me more and trying to fix me less 15:40:27 dhellmann: brutal honesty there, please take it in good faith 15:40:33 ^ same 15:41:07 dansmith : yes, I'm trying to make this conversation about helping you rather than fixing you, and I hope that's coming across. 15:41:11 we took on less in queens and got more done, which was good, 15:41:23 and b/c of that, it seems we dipped way into the debt tank in rocky and are way overcommitted agin 15:41:24 *again 15:41:24 dansmith : mriedem : +100 on what dhellmann said above 15:42:08 lots of big things take project management help too, 15:42:11 dhellmann: dims: well, it doesn't feel like that a lot of the time, which is just being honest, but I'm glad to hear that 15:42:13 sdague used to do that for things he cared about 15:42:14 dhellmann: thank you for making that effort, because I have to say I feel similar. we merge a _lot_ of code, yet are constantly the subject of "problems" and "not enough cores" 15:42:17 dansmith : IIRC from looking at the stats, nova is doing at least an order of magnitude more review work than other teams. The rate of code churn is incredible. So I'd like to help you lower your stress levels, because you're already doing more than is reasonable. 15:42:17 ildikov did it for multiattach 15:42:41 i think we're looking to find ways to reduce the load on you because you're valuable members with huge knowledge 15:42:53 dhellmann: thank you for saying that 15:42:57 * dansmith prints and frames 15:42:59 mnaser: ++ 15:44:31 melwitt : if I say there's a problem, I mean that the team is exhibiting signs of stress which leads to bad interactions. I think that's a problem for the long term health of the people, and the project. I want people to stay happy working on OpenStack in general and nova in particular. 15:44:34 i totally appreciate the work done by the nova core team, it's been invaluable as an operator :> 15:44:35 I'd just like to make it clear that this topic came up because I drew people's attention to jay's message today about feeling overstretched and people's sympathy about that is what is driving this talk. 15:44:37 dansmith: as a likely culprit in that perception, I'd like to acknowledge for the record that Nova is merging an enormous amount of code and that the current cores are working *incredibly* hard. I guess we all just wish you didn't _have_ to. 15:44:58 zaneb++ 15:45:05 Yeah at this point my concern is much more "how do we make the work of the Nova team sustainable", not "how do we make Nova approve $foo faster" 15:45:15 so, to be honest, 15:45:27 I can handle the workload to a large degree 15:45:43 it's the workload plus the feeling that nobody will ever be happy that really wears me out 15:45:57 is it fair to say, dansmith, that your ability to handle the workload is fairly unique? 15:46:00 IMO the feelings some have about Nova (too much, not enough, fix it, etc) have become entrenched folklore 15:46:06 as in dan is special (in the good way)? 15:46:21 and get passed on to new generations of people coming to OpenStack 15:46:22 cdent: I can only speak for myself, which is why I said "I" 15:46:33 dtroyer: very much agree 15:46:44 dtroyer: and that very much contributes to my cynicism about "help" being offered 15:47:06 dansmith : does that mean maybe we could help with setting expectations somehow? 15:47:26 I feel the same, it's all the complaining and criticism from outside that gets me down. nova is a lot of work but I like working on it and I like helping operators, users, and devs 15:47:35 this feels like one place we need to make a decision to intentionally change the culture 15:47:49 honestly i'm going to put in the same amount of time and effort regardless 15:48:30 maybe we should categorize that criticism and address it. I hope that none of this discussion is seen as criticism 15:48:45 dhellmann: I don't know the answer to that question. I have a cynical feeling about it, which may not match the reality or possibility :) 15:49:01 dims: ((your nova people from starlingx question: that is unlikely to be a sudden change)) 15:49:16 ack dtroyer. thanks 15:49:21 dims: also they won't become supercores tomorrow. or cores 15:49:21 i would question how much review happens on the starlingx stuff either 15:49:35 internal forks and stuff are generally done by teams that are paid to just get stuff done, not maintain it 15:49:44 and therefore don't care as much about the long-term costs or quality 15:49:49 hence why maintaining forks is SUPER HARD 15:50:24 dansmith: mriedem: melwitt: also sometimes Nova is taken as the example, while the criticism is actually on OpenStack in general 15:50:25 mriedem: that is why this is challenging to everyone involved :) 15:50:27 and throwing people at this problem doesn't really help 15:50:35 dansmith : ok. I'm reluctant to say that because no one has figured out how to help with that in the past we can't come up with any new ideas today. 15:50:57 Nova is singled out very often. Like in this discussion 15:50:57 dhellmann: ack, which is why I hedged and exposed my cynicism :) 15:51:29 i need a "my sensitive ego has been bruised" alert bot 15:51:42 dansmith : mriedem : melwitt : totally love and respect your work and try to hang out with you all every chance i get. So please don't feel your work is not appreciated. let's try an emoji - 💖 15:52:18 * mriedem sees a lego brick 15:52:19 yeah honestly my experience with the nova team is so awesome as an operator 15:52:21 dhellmann: I think handling Nova through the TC liaison mechanism rather than make it an all-TC exercise will help, too. 15:52:24 ++ lot of respect for nova and other projects 15:52:32 yeah, given the feedback i heard from the nova cores who skimmed some of the divergence in the starlingx nova fork, it doesn't sound like the choices they made are especially compatible with the level of review the current nova core reviewers would expect (no offense meant to the starlingx crew who were maintaining those) 15:52:45 mriedem: it's a heart 15:52:51 at the summit i was stopped twice by two different cores to ask about how impactful a change nova was doing was going to be for us. that's awesome 15:53:02 they totally didn't have to do that but the did 15:53:05 One of the things jay mentions in his email is an "endless stream of feature requests". Can someone give any detail about the source of those requests? 15:53:12 mnaser: brag all you want, everyone knows the nova team <3's mnaser :) 15:53:34 :) 15:53:36 vexxhost (mnaser) and cern get special treatment b/c of their level of openness and involvement in the project 15:53:46 dansmith : i mean ... who doesn't? 15:53:52 dims: indeed 15:54:03 i really want to do a talk on "how to work with upstream" and add more orgs to that list 15:54:05 mriedem: and they actually contribute positively 15:54:10 cburgess and med were also in that boat 15:54:17 RIP! 15:54:20 SAD! 15:54:23 i think if people understand the value of working with upstream, internal culture will change massively 15:55:05 mnaser: true story, one of our guys from china that was in vancouver for the first time commented after the nova cells v2 / cern session, 'wow you guys are very involved in helping cern and they are with making sure their needs are met' 15:55:08 it was like a revelation 15:55:30 well, they do also have a very big electron gun... 15:55:44 lols 15:55:53 i'm just personally interested in helping the people that show up 15:55:57 yeah, don't wanna be on the wrong end of that sucker 15:56:00 mgagne and the godaddy folks also come to mind 15:56:03 jmlowe too 15:56:12 maybe as the tc we should also look at our projects and acknowledge the good progress they've done so far 15:56:12 fungi: the choices made in starlingx were for expediency and had the advantage of a highly opinionated configuration rather than having to work with N possibilities. overcoming that is going to be one of our biggest challenges in clearing the debt backlog 15:56:18 rather than always trying to say "hey so let's fix this" 15:56:31 it might give the messaging of "we're fixing you because you don't have your stuff together" in a way 15:56:44 mriedem : helping folks who show up seems like exactly the right approach to day. That's how open source is supposed to work, imo. 15:56:50 mnaser: it totally does, 15:56:51 ugh, *to take 15:56:55 at least it always has to me 15:57:12 the culture in openstack has changed recently, 15:57:19 we used to celebrate the development side 15:57:27 i think we had 3 nova cores pretty much mention that they felt the messaging is "we're fixing you" so maybe we should look into changing that 15:57:28 focus on the wins 15:57:35 dtroyer: yeah, i get that. just don't want people getting the impression that they were doing the same sort of work the nova team needs now, and the learning curve for them is likely to be as steep as it is for most people coming to the nova team for the first time 15:57:45 nowadays, it feels like the developers are the ones in the back room not doing enough, 15:58:00 fungi: ++ (more for the record than anything :) 15:58:10 dansmith : I've noticed that a bit, too 15:58:12 which means we focus on the projects getting the most complaints 15:58:16 ++ fungi 15:59:12 this was good and i'd like to thank dansmith, melwitt and mriedem for speaking up 15:59:31 having said that, i was happy with the press in queens for multiattach and vgpus and the like 15:59:34 dansmith: I agree that could be, but I also think that as time has evolved, we've lost valuable people, people have become more spread thin... so perceptions might not match reality in terms of their ability to contribute 15:59:34 that was nice to see 15:59:51 cern was extremely gracious in vancouver 15:59:56 dansmith: dhellmann: in recent years i've seen lots of feedback that celebrating developer activity caused other parts of the community to feel like second-class citizens and start lobbying to stop doing things they saw as idolizing developers. i don't agree with that assessment but it seems to have pervaded a lot of the extended community 16:00:07 I guess I want to say quickly that I appreciate the concern from all of you. we are a team of human beings and sometimes we get stressed and exhausted and I think it's okay to be open and honest about that, IMHO. at the same time, I don't want it to cause a rush of concern and fixing if someone opens up about something. we open up to try to make things better and truly evaluate the processes we're trying out 16:00:15 fungi: the core party aside 16:00:34 glad the core reviewer parties are a thing of the past, yes 16:00:44 i expect those fueled a lot of that sentiment 16:00:49 I was glad melwitt had a chance to talk about the work in nova on the keynote stage. I wish there had been more time for that sort of thing. 16:01:11 That was really nice 16:01:12 agreed 16:01:16 it was awesome 16:01:38 med mentioned during the summit feedback session something like that for superusers during keynotes but the community contributor awards 16:01:46 that would be a solid 'hey we actually care about contributors' thing from the foundatoin 16:01:56 melwitt: agreed. at the same time, we'd like to try to help before the pressure gets so bad, so maybe we should talk about this more often in smaller doses 16:02:16 mriedem : yeah, that's high on my list of things 16:02:35 i also can't help but feel like there must be something the tc could do to help have the right conversations to stem the overwhelming tide of feature requests jaypipes mentioned and i've seen others (not even just in nova) bring up from time to time 16:02:39 I started a conversation with mark c. about that after dublin, but didn't follow through in time to make it happen for yvr 16:02:41 Unpopular opinions, but I actually liked the core parties. Maybe I didn't get exposure to the downsides of that. 16:02:58 fungi : right, that's what I was hoping we'd get to 16:03:23 dhellmann: I think the assumption that we need to be parented before we get into a fight is the wrong approach, FWIW 16:03:29 fungi: shameless plug: I think this is something that a technical vision could help with 16:03:42 indeed! 16:03:43 dansmith : oh, I communicated poorly, then, because that's not what I mean 16:04:09 dhellmann: well, that's what it feels like whenever someone venting turns into the TC showing up to separate the children on the playground :) 16:04:31 (to me) 16:04:35 smcginnis: the opportunity for a quiet place to get away and have laid-back conversations was nice. the exclusivity if it was sort of divisive and reinforced a caste system dynamic 16:04:40 dansmith : but as an example, if there's a flood of feature requests that won't stop, maybe that's something we could deal with at the TC/board level so you don't have to have the fight repeatedly with every vendor who comes along 16:04:48 we've done things in the past to try and stem the tide of feature requests, 16:05:00 but the thing is, then you get nailed for not landing enough features that people want 16:05:07 it's a damned if you do, damned if you don't situation 16:05:12 mriedem: define nailed 16:05:27 dansmith : you may be referring to one specific instance where I did feel a discussion had deteriorated to a point where a third-party could help mediate 16:05:29 constant critisizm? 16:05:42 constant "how do we fix nova?" 16:05:45 fungi: I liked the quiet place to have conversations aspect of it. 16:05:46 mriedem: could it just still then be a perceptions issue? 16:06:01 FTR we like to fix Swift too. 16:06:06 fungi: And felt like it was a good way to foster communication between the people in different teams that could work together to actually get things done. 16:06:08 so I have a theory that is probably unpopular :) 16:06:24 dhellmann: I dunno that the board/TC really have the ability to stem that tide I guess 16:06:38 TheJulia: idk, i think some people just don't care about anything but their special snowflake 16:06:38 I guess maybe I genuinely think people are well intentioned, and maybe criticism comes from a lack of context 16:06:40 dhellmann: I'm not referring to any one incident, fwiw 16:06:54 fungi: And if having a party incentivized some people to get more involved and become cores to be able to go to it, that's good in my book. 16:07:14 dansmith : if we said openstack was going to focus on technical debt during 2019 and not accept new features, and you received new feature requests, you could just send people to me instead of having to talk to them yourself. Would that help? 16:07:19 mriedem: true, or they don't have the capacity to care about anything outside of the overall venn diagram defining their role 16:08:02 zaneb: ? 16:08:05 dansmith: there is the idea that if the vendors pushing for new features are represented on the board or in the set of foundation member companies, then there's the potential for product management and executive management direction conversations to stop trying to cram new features into a project which is already under stress to keep up 16:08:11 dhellmann: that doesn't seem realistic to me, but if it did happen I guess it would be nice :) 16:08:14 the theory goes that due in part to the hype cycle and in part to the lack of a detailed technical vision, OpenStack became a thing that tried to be all things to all people. It tried to be both a cloud offering and a self-service full VMWare/oVirt replacement complete with live migration, shelve/unshelve, and generally a lot of stuff that increased the complexity of the code and led to a steady torrent of n 16:08:15 ew feature requests 16:09:10 dansmith: we kinda said Ocata could be used to to a stable release... not sure that worked really though 16:09:12 zaneb: I agree with that theory, but what point did it support? 16:09:17 dansmith : I would support that and fight for it if it would help you have time/space to do more work to split parts out of nova or take whatever other steps you need to help resolve the technical issues that make it hard for you to bring in new folks to the core team 16:09:38 if we could have picked one then things would have been easier to manage 16:09:57 zaneb: but that ship has sailed 16:10:07 yeah we kinda have users now 16:10:10 and the hordes are now doing it to k8s is my understanding 16:10:12 dhellmann: fwiw, I think that people external to the TC generally don't believe that the TC/board have influence on what contributing companies do. I follow the TC's work fairly closely and it isn't even clear to me that there is influence there 16:10:21 dhellmann: if there is influence, it isn't very visible 16:10:25 mriedem: yes, in many ways it's too late to fix that now because we have existing users to support 16:10:26 jroll: agree 16:10:37 jroll: ++ 16:10:41 dhellmann: I wouldn't say that a year of tech debt is going to accomplish or even work towards that goal 16:10:56 jroll : we have tried to be very hands-off in the past, but the bylaws of the project give us basically unlimited power if we choose to use it 16:11:02 but we can still do better for the future by saying clearly what we do and do not want OpenStack to be 16:11:05 i'm pretty sure my employer still wants/needs to see features going in every release regardless of what upstream does, so that doesn't help me really 16:11:14 mriedem: mine as well 16:11:35 dhellmann: he said influence over the companies, not the developers 16:11:54 dhellmann: I don't think the bylaws give you unlimited power over, say, huawei's actions 16:12:08 dansmith : if we have to make it so that the TC has to approve patches, we could do that. I wouldn't want to, but we could. 16:12:22 my point is that we have more power than we use 16:12:24 okay I guess I'm missing something 16:12:32 and then nova will just lobby to be a top level project in the foundatoin and outside the tc pervue 16:12:43 *purview? 16:12:47 when is swift going to do that btw? 16:13:01 we say no one has any influence over these companies, but the community sets the contribution rulues 16:13:32 dhellmann: my point being you can offer the TC to help with the negativity or turning down feature proposals, but most people won't believe that the TC can successfully help there 16:13:46 jroll : we can try ... right? 16:14:02 sure 16:14:19 jroll : walk softly and carry a big stick 16:14:32 not saying it isn't possible 16:14:34 * mriedem looks at the clock and realizes he hasn't reviewed anything he wanted to review yet today 16:14:44 I think we need to better define the problem (if there is a problem) before wielding a big hammer in any direction 16:14:47 mriedem , dansmith , melwitt : thanks for your time today 16:14:57 +1000 16:15:07 ttx: definitely 16:15:09 jroll: in fairness, i also have limited expectations in our ability to get board members to take specific actions, or that the board members even necessarily have the ear of the right people in their own organizations to make such things happen, much less in other member companies. still, not trying guarantees it won't happenm 16:15:32 well this is an interesting buffer scrollback to start the day with 16:15:55 I'll just point to thingee's work on forcing third-party ci through for the cinder team as an example of how, when we set the rules to support the teams doing the work, we can make big changes happen. 16:15:59 I'm just offering a POV from outside of the TC, especially as to why people generally are meh on the TC trying to take action here 16:16:01 notmyname: hi! 16:17:06 fungi: should we close the hour? 16:17:13 seems we've wound down 16:17:19 unless notmyname has something? ;) 16:17:29 If we continue I need another drink 16:17:31 it's especially difficult to answer the question "what can the TC do to help here?" if one doesn't think the TC has the influence or power to do much 16:17:41 * jroll is done, I think 16:18:16 do much with toning down the slew of patches or negativity from contributors* 16:18:24 jroll: yeah, we've been mostly loathe to exercise any nuclear options 16:19:03 in part, because those options put the project teams in a position of having to decide whether they want to continue to be part of openstack or go it on their own 16:19:25 the TC has a lot of influence, but mostly only power that is too strong to use 16:19:28 fungi: nah, still reading scrollback. don't wait for me for anything 16:19:33 kind of like the Queen 16:19:50 we could, however, back up teams that want to exercise some of that power themselves 16:20:47 though i think that comes down to those teams feeling comfortable to exercise that power, and knowing we've got their back if their decisions get escalated to us 16:21:44 which is exactly why I keep asking the team what would help 16:22:15 well, if we stick to the specific nova thing 16:22:17 what we probably can't be much help with is taking away the stress the employers of team members may place on them 16:22:21 we concluded the major part of the problem is that the constant negativity is what really drags down the folks working on it 16:22:29 at least as far as said nuclear options are concerned 16:22:30 fungi: Should you #endmeeting? 16:22:31 yeah, i was going to say, 16:22:36 how can the TC reduce people saying "nova doesn't want my features!!" 16:22:51 "nova sux they won't merge this completely out of scope thing" 16:22:54 etc 16:23:13 stop talking about 'fixing' things, more emotional support, and understand we're trying - we realize there are issues, and we're generally our own worst critics 16:23:23 jroll: it sounds like the bigger problem is "my employer will reassign me and i won't have time to be a nova core reviewer any longer if i can't at least also make some progress on features they've requested" 16:23:42 mriedem: That should be a presentation for berlin 16:23:51 TheJulia: did that in boston 16:23:53 it was the ptl talk 16:23:59 I know :) 16:24:23 was a very excellent talk, btw 16:24:25 it should be in the playlist on repeat of sorts 16:24:27 fungi: really depends on what the core is focusing on i think 16:24:31 fungi: I think that is a smaller risk than the cores totally burning out 16:24:37 yeah 16:24:47 huawei is totally happy to have me working on fixing scale and resiliency issues 16:25:01 but we have lower priority feature needs for operators too 16:25:31 huawei sounds like they're doing it right ;) 16:26:25 hey, 16:26:29 huawei is best way 16:26:38 * mriedem drops mic 16:26:45 please tell me that's the official slogan 16:26:53 hypothetically, if nova were to want to take a cycle merging no new features and just working on decomposition, tech debt, usability fixes, bugs... any feel for how many of the current nova core reviewers' management chains would reassign them to somewhere else? 16:28:03 It wouldn't just be cores that we would need to worry about 16:28:05 I've always assumed employers would continue having people do the same amount of feature work, with a note that it won't be merged until the following cycle 16:28:08 but actual on the ground contributors 16:28:29 and then double the feature requests the following cycle 16:28:39 TheJulia: if the actual on the ground contributors who get reassigned wer ethe ones pushing the unending flood of feature review requests, then that sounds like what they'd want? 16:29:26 fungi: except those same devs probably also contribute bug fixes, right? 16:29:34 and suddenly you've lost some percentage of bug fixers 16:29:34 or is the general makeup of a random (not necessarily core reviewer) contributor to nova doing a mix of implementing new features and fixing bugs? 16:29:43 jroll: yeah, that's what i'm wondering 16:29:49 good question 16:30:23 tragedy of the commons scenarios tend toward a lot of people pushing agenda patches and a handful working on fixing existing issues 16:30:23 dhellmann: That might be an interesting metric for your next set of reports. ^ 16:30:37 I think it would be like trying to cross the doldrums with a sailboat. 16:30:43 * jroll needs food, this was a good conversation. I'm bummed I came in so late 16:30:45 dhellmann: How many contributors are doing feature work, how many are doing bug fixes, and what's the venn diagram look like of the two. 16:31:06 yeah, it can be hard to tell the difference between a bug fix and a feature request 16:31:24 I would like to figure that out, though, so if anyone has ideas 16:31:29 I suppose it depends on the team's policies around using Closes-bug for actual bugs. 16:31:46 dhellmann: closes-bug: vs implements: would get you most of the way in nova 16:31:51 yeah, and teams moving to storyboard will just have "story" or "task" 16:31:58 Probably not complete, but it would get part way there. 16:32:50 it would be a very fuzzy metric, but might be striking enough to still hold some insights 16:33:19 okay, so we've basically wrapped up discussion i suppose? i'll close this log out 16:33:22 _many_ implements are tech debt fixes 16:33:30 not strictly bugs 16:33:56 I'm more interested in the difference between the things the team wants and the things the team has thrust upon it 16:34:13 a lot of teams also don't have strict policies about filing bugs if there's already a fix available... so generalizing across teams could be tough 16:35:28 for infra work, bug reports were generally placeholders for we know something is broken but don't know what the fix should be yet. if it was faster to write the patch than the bug, we didn't saddle contributors with having to do both 16:36:39 Yeah, when it takes 20 minutes to file the bug, and a minute to fix it, and maybe 2-3 minutes to type a verbose commit message summarizing why... it adds up quick. 16:37:49 (because the bug has to be written with the bus factor accounted for) 16:39:00 Speaking of negative perception... (sorry if topic is past already) 16:39:01 you have to balance that against how many minutes future readers spend trying to figure out why the heck that change was made 16:39:09 On a positive note, when I attempted to contribute to Nova in the past (like years ago), I used to perceive Nova in a very negative way. 16:39:09 I felt I was ignored and got no feedback whatsoever for months. The only feedback was an automated reply saying "sorry, deadline has past, please propose for next version". It didn't feel like I was interacting with humans at all. Only some kind of big administration where forms and numbers are the norm. 16:39:15 During that same period, I was also contributing to Cinder and had a totally different (positive) experience. So it played a lot in my negative perception of the Nova project as I was expected the same (positive) experience. 16:39:15 Nowadays I get a human reply and feedback. (thanks to mriedem, melwitt, dansmith and others I forgot) It drastically changed my perception of the Nova project. I agree it's not perfect but at least it feels more human. So there is that which I feel is a great improvement. ++ 16:39:53 zaneb: an excellent argument for very detailed commit messages 16:39:56 zaneb: indeed, or be able to pick it up with sufficent context. 16:40:24 i just don't see a ton of benefit, personally, to copying my detailed commit message into a bug and immediately closing it 16:40:39 mgagne++ 16:40:42 mgagne: that reminds me of something mrhillsman told me once, or maybe smcginnis said it from an ops meetup - that ops people feel like they shouldn't or can't ask questions or raise issues in the -nova channel 16:40:48 which is definitely a perception issue 16:40:48 cdent: you too ++ 16:41:09 mgagne: I was kind of thinking the same thing earlier this week that I actually went on a business trip, got back and had two nova patches merged and I was like "WOW!" 16:41:39 mriedem: you know what I said at an ops meetup? "you need a friend to get stuffed reviewed or merged" 16:42:09 fungi: benefits are that other people reporting/searching for the same bug may find it, and you can track the status in multiple branches. obviously this applies less to infra than some other things 16:42:17 well, honestly knowing a person that is running a certain system does help, like when i think 'caching scheduler' i know who i can talk to 16:42:20 TheJulia: I appreciate your feedback for the times I attempt to contribute to Ironic ++ 16:42:20 mgagne: thank you, and we appreciate the effort you make to work with us and get things fixed in nova 16:42:32 mgagne: yeah, I agree that there is definitely a FOAF aspect to openstack contribution that isn't always ideal, but isn't always a negative either 16:42:50 mriedem: so I'm the caching scheduler guy now, don't know how to feel about that one :P 16:42:59 mgagne: I really do try, sadly there is no way for me to reply to every single patch :( (and I actually feel bad about that reality....) 16:43:00 well, it was rax 16:43:04 but now you're the main person 16:43:12 mgagne: tarred! 16:43:27 mgagne: you can also be the "loves network_data.json guy" 16:43:31 if you want 16:43:35 zaneb: it may also just be that i'm used to looking at git history to find why changes were made (at least if they're minor enough changes to not require a release note or security advisory) 16:44:32 cdent: OpenStack is not alone in that.... In some other projects I've had to reach out through a friend because I couldn't illicit a response. :\ 16:45:01 TheJulia: yeah wasn't trying to say openstack was unique in that regard, just that it happens 16:45:02 the random dead cat questions also matter, those get ignored 16:45:07 I suspect it part of scale 16:45:17 "i deployed openstack and i can't create a server, please help" 16:45:20 cdent: Yeah 16:45:28 like, that's where we tell people to hit the forum or #openstack channel 16:45:45 but, an operator saying, "i'm upgrading and hitting x, i think it's due to y but need some help here" 16:45:53 mnaser: ^ is great about that 16:46:05 mriedem: Then there is also the "Uhh, that is weird, uhhhhhh Could you give us this laundry list of things we need to help you?" and then crickets are beamed in through time using a transporter 16:46:39 #chair TheJulia cdent zaneb dhellmann smcginnis dims ttx 16:46:40 Current chairs: TheJulia cdent dhellmann dims fungi smcginnis ttx zaneb 16:46:56 okay, i'm about to disappear and grab lunch/run errands. our office hour is nearly an office double-hour at this point but someone can #endmeeting when appropriate 16:47:49 Do we have anything else to discuss? Things to get off our chests? great beers or coffees to share? 16:48:27 fungi : nice reworking of the base services draft 16:48:46 I think it's safe to close the logs at this point 16:48:50 TheJulia : do you want to do the honors? 16:49:13 Sure! Since it seems like the time ship relativity beamed in crickets. 16:49:34 * cdent chirps 16:49:34 #endmeeting