15:00:49 <anteaya> #startmeeting third-party
15:00:50 <openstack> Meeting started Mon Mar  2 15:00:49 2015 UTC and is due to finish in 60 minutes.  The chair is anteaya. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:51 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:54 <openstack> The meeting name has been set to 'third_party'
15:01:10 <patrickeast> hello
15:01:14 <rhe00> hi
15:01:20 <wznoinsk> hi
15:01:34 <anteaya> hi folks
15:01:40 <anteaya> how is everyone today
15:01:59 <patrickeast> good, yourself?
15:02:00 <rhe00> it's Monday
15:02:20 <wznoinsk> all good, thanks
15:02:45 <anteaya> good glad to hear it
15:02:58 <anteaya> any thoughts on what folks would like to discuss today?
15:03:44 <anteaya> perhaps it being monday things haven't had a chance to break yet?
15:04:03 <anteaya> wznoinsk: you had some questions at the end of last week, did you get the answers you were looking for?
15:04:47 <wznoinsk> yes, I did, will talk to Nova guys once I'm ready to comment back, got added to Third Party Gerrit Group to avoid spam as well
15:05:05 <anteaya> good work
15:05:25 <anteaya> asking for permission to comment on patches is best done at the nova meeting
15:05:36 <anteaya> add an agenda item when you are ready
15:05:42 <anteaya> then the discussion can be logged
15:05:49 <anteaya> make sense
15:06:08 <anteaya> ?
15:06:23 <wznoinsk> yes, will do
15:06:25 <rhe00> so we have to ask the Nova group for permission to comment even though we comment on Cinder patches?
15:06:28 <anteaya> okay great
15:06:43 <anteaya> yes
15:06:47 <anteaya> every project is different
15:06:54 <anteaya> is taht news to you?
15:07:09 <anteaya> I'm interested in how you see things
15:07:35 <rhe00> that is news, but if that is the process then I'm fine with that
15:07:50 <anteaya> okay well let me take a minute and share some information
15:07:59 <patrickeast> i had assumed voting for sure was per project, but commenting was fair game as long as it was actually relelveent, makes ense though to ask first
15:08:02 <anteaya> are you aware that openstack is comprised of separate projects?
15:08:11 <patrickeast> relevent* even
15:08:14 <wznoinsk> I think the lack of that req in the 3rd party ci setup guide makes a bit of confusion - I know the rework of the docs is undergoing so we have to live with a few discrepancies for that short time
15:08:30 <anteaya> it won't be a requirement
15:08:45 <anteaya> since there are some situations like grandfathering where folks didn't ask
15:09:03 <rhe00> yes, but I thought we were only commenting and voting on Cinder if we are in the Cinder group
15:09:19 <anteaya> in the infra third party ci document, requriements are the things you have to do to avoid having your account disabled by infra
15:09:28 <anteaya> important things like providing logs
15:09:35 <anteaya> and having timestamps in the logs
15:10:13 <anteaya> I think part of the problem is taht many third party ci folks only see the world through their eyes, we have to do this thing
15:10:13 <rhe00> is there a max time limit for how long the test is allowed to take before producing a result on a patch?
15:10:22 <anteaya> and never see the world from the other side
15:10:27 <anteaya> the side of the developer
15:10:42 <anteaya> have any of you ever submitted patches to a project?
15:10:46 <wznoinsk> possibly if I'd see 'it's worth mentoning or asking for permission* on commenting' I'd go to the meeting and talk, not having it htere equals is not required so given the guide is a finished list people are surprised it's actually a good practice still
15:11:07 <anteaya> politeness and ettiquette
15:11:11 <anteaya> they matter
15:11:20 <wznoinsk> true, as much as documentation
15:11:29 <anteaya> and people are suprised because they think that once the tick off all the things that they are done
15:11:48 <anteaya> you will see in the requirements that folks are supposded to be available for conversation
15:11:53 <anteaya> including participating on irc
15:12:25 <wznoinsk> I am, I wouldn't realize I have to ask for permission if I wouldn't catch Mellanox emails on ML
15:12:32 <anteaya> Maintainers are encouraged to be in IRC regularly to make it faster to contact them.
15:12:41 <anteaya> well you asked on irc
15:12:43 <anteaya> right?
15:12:46 <anteaya> to clarify
15:12:57 <anteaya> so that fits the requirement
15:13:02 <anteaya> things change all the time
15:13:18 <anteaya> and part of the problem with having a list is that people check off the items and think they are done
15:13:21 <wznoinsk> Mellanox ci was suspended cause they didn't ask for permission to comment, that was a thread on mailing list
15:13:29 <anteaya> and then they get defensive if things change
15:13:37 <anteaya> since they figure that they shouldn't be expected to do that
15:13:51 <anteaya> Mellanox ci isn't suspended
15:14:00 <anteaya> never was as a result of that email
15:14:04 <wznoinsk> with someting as complex we have to follow a list, if asking for permission is only a curtosy suspending CIs shouldn't be depend on that
15:14:14 <anteaya> nowhere in that email did it say that account is disabled
15:14:26 <anteaya> right
15:14:34 <anteaya> which is why it isn't a requirment
15:14:54 <anteaya> since I only will disable accounts under special circumstances for that
15:15:11 <anteaya> comment on infra patches without jim's permisison and you will be suspended
15:15:21 <anteaya> or disabled is the word I use
15:15:28 <anteaya> all others have a conversation
15:15:36 <anteaya> which is what we are having
15:15:51 <anteaya> and you are paying attention and present on irc and asking
15:16:07 <anteaya> which are all the things I look for in Maintainers are encouraged to be in IRC regularly to make it faster to contact them.
15:16:09 <rhe00> so are we supposed to comment on glance, horizon, etc as well? just asking to make sure I am not missing one component
15:16:25 <anteaya> supposed to comment
15:16:29 <wznoinsk> so in general, makes some noise and confusion if it's not in the docs to represent it on irc/ask for permission, work for you guys especially
15:16:38 <anteaya> the optimum for OpenStack is if you don't comment on anything ever
15:16:59 <anteaya> since having accounts commenting is really a drain on resources for all of openstack developers
15:17:27 <anteaya> it is only because vendors insist on having drivers that I was forced to set up this line of communicaiton in the first place
15:17:50 <rhe00> by commenting I mean SUCCESS / FAILURE
15:17:52 <anteaya> wznoinsk: can you rephrase?
15:17:59 <anteaya> rhe00: right, yes I know
15:18:11 <anteaya> keep in mind I am a woman, not a guy
15:18:42 <anteaya> infra had the tools but noone knew how to set them up
15:18:54 <anteaya> we had thought folks in this space could serve themselves
15:18:59 <wznoinsk> the confusion makes work for you, rhe00 seems to be new to the 'ask for permission' thing, so a lot of people by the looks of things
15:19:04 <anteaya> and would understand the developer's perspective
15:19:30 <anteaya> wznoinsk: third party ci makes work for me
15:19:48 <anteaya> the optimum is if everyone was like dan prince
15:19:56 <anteaya> and was a developer as well
15:20:12 <anteaya> most third party ci operators have no idea what the developer experience is
15:20:14 <wznoinsk> so maybe that should be in 3rd party meetings agenda to have 'best practice' for newcomers
15:20:33 <anteaya> I'm fine with that
15:20:40 <anteaya> given your experience thus far
15:20:50 <anteaya> what would you suggest as best practices?
15:21:20 <anteaya> top of my list is submit a patch to a project and understand what the developer experience is
15:21:26 <anteaya> that never seems to happen
15:21:28 <wznoinsk> drop in for a 3rd party meeting and present your new ci at least once
15:21:52 <anteaya> what is the definiton of present your new ci?
15:22:09 <patrickeast> ^ that happens at the thirdparty working group meetings
15:22:21 <patrickeast> there has been a few “show and tell”s
15:22:25 <anteaya> that can happen at all meetings
15:22:38 <wznoinsk> patrickeast: yes, but that's for already working/stable ones I guess rather for newbies
15:22:39 <anteaya> all I need is people doing it
15:22:42 <asselin_> hi
15:22:52 <anteaya> which is why I started off by asking what folks wanted to talk about
15:22:55 <anteaya> hey asselin_
15:23:11 <anteaya> wznoinsk: do you want to present your ci?
15:23:27 <patrickeast> wznoinsk: gotcha, so you mean more like a ci review or something for new ones?
15:23:29 <anteaya> or what would you tell someone who did so?
15:23:34 <wznoinsk> anteaya: I do agree, it's all about people, but I take their side why they're confused if there's only a suggestion to be on irc and not even something 'if your project is nova, cinder please ask for permission or attend a metting to introduce yourself/your ci'
15:23:47 <anteaya> that is fine
15:23:55 <anteaya> and I can appreciate that perspective
15:24:11 <anteaya> what you don't have is any idea of how your work affects the developer
15:24:21 <anteaya> as they are the one that has to consume your output
15:24:27 <anteaya> which isnt' your fault
15:24:30 <anteaya> it is common
15:24:45 <anteaya> because most manger's don't see that as a necessity
15:24:51 <anteaya> to take the time to do that
15:24:56 <anteaya> and it isn't on the check list
15:25:01 <anteaya> and won't be
15:25:19 <anteaya> but the difference between a functioning ci and a great one
15:25:31 <anteaya> is the extent to which the operator understands the dev perspective
15:25:36 <wznoinsk> well I'm a software engineer aka developer (new to this world but still), I have the CI task to ramp and troubleshoot the neutron/nova/devstack in containers stuff by the way
15:25:52 <wznoinsk> anteaya: I do understand that, hence we have you between Ci ops and devs right? ;-)
15:26:01 <anteaya> wznoinsk: yeah I know
15:26:12 <rhe00> I'm a developer and CI operator
15:26:16 <anteaya> which is why you get the responses you do
15:26:45 <anteaya> rhe00: you have patches submitted to openstack projects?
15:26:58 <anteaya> rhe00: you have experience consuming other third party ci log output?
15:27:07 <rhe00> well, only a driver so far.
15:27:19 <anteaya> well try to expand yourself a bit
15:27:21 <anteaya> fix a bug
15:27:27 <anteaya> consume other third party ci output
15:27:38 <anteaya> then you will be able to increase your value in the communtiy
15:27:45 <rhe00> yes, I have asked a couple of vendors to help me look at their failure outputs
15:27:53 <anteaya> since you will be able to experience what life is like for other devs
15:27:59 <anteaya> rhe00: awesome
15:28:07 <patrickeast> i think maybe just consuming the thirdparty output and knowing the rules about adding a new ci are not quite the same…
15:28:09 <anteaya> care to discribe that experience?
15:28:22 <anteaya> patrickeast: go on
15:28:25 <wznoinsk> anteaya: I really appreciate the time and the work you have to go thru while dealing with newbies, like me for example, I'm a bit worried that given the expansion of openstack community any leaks/soft rules in documentation may have a knock-on effect and slow things a bit
15:28:26 <patrickeast> from asking about getting voting permissions in cinder meetings it seems like most folks there at least know about the same as us
15:28:34 <patrickeast> even cores don’t know the rules infra has
15:28:36 <patrickeast> vs nova
15:28:38 <patrickeast> vs neutron
15:28:39 <patrickeast> etc
15:29:08 <wznoinsk> anteaya: you've touched a very good point, some guidelines (aka devs wishlist) regarding shape and form of artifacts would be good to have (although not sure whether that's possible to achieve)
15:29:10 <anteaya> wznoinsk: me too
15:29:26 <anteaya> patrickeast: yes
15:29:33 <anteaya> which is a big part of the problem
15:29:49 <anteaya> since this space expanded far faster than we have been able to keep up
15:29:56 <anteaya> and I burnt out in novemeber
15:30:11 <patrickeast> so for setting up and debugging a ci, knowing it is working correctly, it makes sense that having some experience with what it is supposed to look like helps, i don’t think that being a developer would have fixed the primary debate this moring of “should we ask for permission, and how are we supposed to know that we should”
15:30:19 <anteaya> so now I am doing the best I can do
15:30:27 <anteaya> and there are gaps, and I know about them
15:30:30 <patrickeast> dont get me wrong, im not pointing fingers or anything
15:30:44 <anteaya> patrickeast: of course
15:30:49 <patrickeast> just saying that being an openstack dev and ci maintainer do have some non intersecting parts
15:31:01 <anteaya> yes
15:31:29 <anteaya> but knowing what things are like from devs would at least have given the understanding of two things
15:31:49 <anteaya> why is is important to communicate with projects (that is the point of ask permission to comment)
15:32:06 <anteaya> and two that projects are highly different in how they self organize
15:32:21 <anteaya> so that commenting on one project doesn't mean you have permission to comment on another
15:32:26 <anteaya> make sense?
15:33:23 <patrickeast> yea i can agree with the communication parts
15:33:37 <patrickeast> the commenting i’m still not sure thats a given
15:33:46 <anteaya> it isnt' a given
15:33:51 <anteaya> not saying it is
15:33:58 <anteaya> I'm saying it isnt' a requirement
15:34:18 <anteaya> now if someone wants to offer a patch to that document
15:34:29 <anteaya> with a heading ettiquette or something
15:34:38 <anteaya> then that might be a good point to list
15:34:57 <patrickeast> yea
15:35:00 <anteaya> the only project taht commenting without permisison _will_ get you disabled without a doubt is infra
15:35:00 <asselin_> +1
15:35:14 <anteaya> the other projects are project discrimination
15:35:20 <patrickeast> maybe we should list project specific rules then? would that be helpful?
15:35:47 <anteaya> if thingee comes to me and says I want to diable systemx for not asking permisison to comment, I will disable because that is ptl's choice
15:35:48 <patrickeast> or suggestions*
15:36:03 <asselin_> if we do project specific rules, they should be on a wiki.
15:36:07 <anteaya> project specific things are in the projects wikipages
15:36:18 <anteaya> what asselin_ said
15:36:18 <asselin_> anteaya, +1
15:36:27 <anteaya> this is where communication is the requirement
15:36:38 <wznoinsk> or instead of rules create gerrit groups having CIs depending on permission for commenting/voting/gateign (which I guess you already have)
15:36:41 <anteaya> if you don't understand something, come and talk to us
15:36:46 <wznoinsk> and apply for membership
15:36:55 <patrickeast> ok cool, so then maybe just a comment that says “hey there might be project specific requirements that differ for each one, go check the wiki *before* commenting on patches” would probably fix the issue
15:37:02 <anteaya> well communicaiotn is more important
15:37:05 <anteaya> for instance
15:37:09 <anteaya> you folks show up to meetings
15:37:10 <asselin_> patrickeast, +1
15:37:15 <anteaya> I know how to get a hold of you
15:37:29 <anteaya> if anyone came to me with a complaint and I know it was your system I would ask you on irc
15:37:41 <anteaya> and give you a chance to reply (If I could)
15:37:57 <anteaya> but I tried to find someone from mellanox on irc
15:38:07 <anteaya> you can read that in the logs of the infra channel for that date and time
15:38:10 <anteaya> noone was there
15:38:14 <anteaya> so to the mailing list
15:38:37 <anteaya> disabling a system has become our only mechinsism of getting people to talk to us
15:38:44 <anteaya> since we can't find them otherwise
15:38:53 <anteaya> we never had to do it in the aerly days
15:38:58 <anteaya> folks were just available
15:39:09 <anteaya> now we have operators that we don't know who they are
15:39:12 <anteaya> they never talk to us
15:39:20 <anteaya> communication trumps all
15:39:44 <anteaya> so rather than getting hung up on rules and making more
15:39:53 <anteaya> they rarely solve the issue anyway
15:39:56 <anteaya> ask
15:40:02 <anteaya> and encourage others to do the same
15:40:45 <anteaya> but if you want to offer a patch please do
15:41:00 <anteaya> at the very least you get to experience the dev workflow by doing so
15:41:55 <patrickeast> sounds good
15:42:00 <anteaya> thanks
15:42:14 <anteaya> talking to you is by far my preference
15:42:35 <wznoinsk> regarding the prefered way devs would prefer the artifacts to be shown in, would the new in tree 3rd party ci be done this way? asselin_ anteaya ?
15:42:42 <anteaya> since once I can find you, I know you are trying to do what you understand, sometimes it is just a matter of expanding the perspective
15:43:24 <asselin_> wznoinsk, new in tree will use the same format as infra's jenkins
15:43:28 <anteaya> any plugin testing run by the infra team would have logging set up consistent with current logging
15:43:36 <anteaya> we wouldn't change logging format
15:44:03 <asselin_> (and if there's a change, it will be applied consistently to 'everything')
15:44:09 <anteaya> and yes, having a consistent logging experience for devs decreases the amount of energy devs need to expend to grok your logs
15:44:19 <wznoinsk> again I guess that's different from project to project but if I'd follow way/tools to present artifacts that in tree solutions uses would I make devs life easier?
15:44:23 <anteaya> for instance, http://64.119.130.115/156126/4/
15:44:39 <anteaya> even having a different font tells me diving in will be work
15:44:41 <asselin_> wznoinsk, yes
15:44:48 <anteaya> so I put it off until after the meeting
15:45:19 <anteaya> wznoinsk: and you are asking good questions
15:45:21 <anteaya> thank you
15:47:10 <anteaya> so spending time talking to other operators has value as well
15:47:56 <anteaya> did folks have anything else they would like to say on this topic?
15:48:22 <patrickeast> thats all ive got for it
15:48:34 <anteaya> would anyone like to discuss anything else?
15:48:49 <patrickeast> i have something you guys might be interested in
15:48:59 <patrickeast> i was bored over the weekend and made a thing https://github.com/patrick-east/scoreboard
15:49:15 <patrickeast> so for my ci right now i can only see when jenkins fails
15:49:26 <patrickeast> and not if zuul decides to post not_registered or error
15:49:52 <patrickeast> so with that it keeps watch for gerrit events and can build a dash thing like http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/?project=openstack%2Fcinder&user=&timeframe=
15:49:58 <anteaya> patrickeast: nicely done
15:50:05 <patrickeast> it has some serious perf problems right now
15:50:21 <patrickeast> but after a few more hours of tinkering it might be somewhat usable
15:50:39 <patrickeast> i dont recommend pointing it at all data right now
15:50:44 <patrickeast> your browser tab might crash
15:50:45 <patrickeast> lol
15:50:51 <patrickeast> just per project or user
15:50:58 <anteaya> nice
15:51:06 <anteaya> so having this would be useful to a project
15:51:07 <wznoinsk> looks good patrickeast
15:51:08 <asselin_> patrickeast, very nice
15:51:17 <patrickeast> and thats a top notch free tier ec2 instance so be gentle
15:51:31 <anteaya> and then hopefully someone on the project would take some time and look at the results
15:51:45 <anteaya> for instance the accounts taht are failing everything
15:51:46 <patrickeast> at the very least it very clearly shows who the bad apples are
15:51:49 <patrickeast> yea
15:52:02 <anteaya> right
15:52:14 <anteaya> which would _really_ help in the quality conversation
15:52:29 <anteaya> so first step would be trying to comminicate to them
15:52:39 <anteaya> if no response in a reasonable amount of time
15:52:47 <anteaya> request their systems be offlined
15:52:56 <anteaya> if they don't come back to learn how to improve their systems
15:53:04 <anteaya> you have just done us all a favour
15:53:26 <anteaya> so internal to the project first
15:53:39 <anteaya> and if you have a list of systems you want disabled, then come and talk to me
15:53:45 <rhe00> don't you have to offline the system right away? otherwise it would hold up check ins?
15:53:53 <anteaya> it cleans up teh space for the folks doing good work
15:54:07 <anteaya> rhe00: can you rephrase please?
15:54:10 <wznoinsk> having a bit of automation to notify the operators would be nice on top of it maybe? ;-)
15:54:26 <anteaya> wznoinsk: what do you picture?
15:54:36 <rhe00> let's say vendor X's CI is failing and voting -1 on every patch
15:54:37 <patrickeast> wznoinsk: yea thats something more like what sweston was working on with that other dashboard
15:54:44 <patrickeast> this one is only for displaying data
15:54:51 <anteaya> if an op is not paying attention if their system is failing every tested patch, that is a concern for me
15:54:52 <rhe00> it would prevent patches to merge
15:55:03 <wznoinsk> if failures of a CI are >98% in certain amount of time (a day, a week) send a notification there's something bad going on
15:55:14 <anteaya> rhe00: yes, so in that case someone, anyone could report that and the system is disabled
15:55:15 <patrickeast> imo a ci operator should be watching that
15:55:38 <anteaya> patrickeast: +
15:55:43 <rhe00> but that would have to happen immediately or it would hold up numerous patches in the queue.
15:55:49 <wznoinsk> anteaya: that's very thru, I'm trying to cover the edge cases when someone couldn't check for some reasons (but yet it's normally taking care of the ci properly)
15:56:10 <anteaya> rhe00: right so if you ever see that, report that in the infra channel and we would disable that account right away
15:56:28 <asselin_> rhe00, there are 2 levels of ci. One is 'voting' the other is 'non-voting/just commenting'
15:56:30 <anteaya> wznoinsk: I'm not interested in edge cases
15:56:49 <anteaya> if folks can't be bothered to check their own results they risk getting disabled
15:56:57 <asselin_> rhe00, only the 'voting' ones would cause an immediate disruption
15:57:37 <anteaya> wznoinsk: I'm not here for third party ci folks
15:57:46 <anteaya> I'm here to protect the devs
15:57:46 <rhe00> asselin: I assume all CIs has to be voting or it makes less sense to have CIs, if there is a failure comment but the patch is merged anyway then what good did the CI do? once the patch is merged then the developer submitting the patch might go on to other work
15:58:07 <anteaya> wznoinsk: if third party operators want to figure out how to play nice, I'll help
15:58:14 <anteaya> wznoinsk: that is as far as I can go
15:58:17 <asselin_> rhe00, currently in cinder, no cis are allowed to vote
15:58:32 <rhe00> I know, but by march 19th all of them will
15:58:46 <asselin_> rhe00, and you're correct, last week we had a patch land that failed non-voting ci, which caused all future patches to fail.
15:58:47 <patrickeast> rhe00: even if the ci votes -1 the patches can (afaik) be pushed through
15:58:53 <anteaya> rhe00: look at the dev mailing list, walter boring had a good post, to which jgriffith had a good reply
15:59:11 <patrickeast> rhe00: cinder probably wont have voting ci’s until later this year, iirc they wanted to wait until they were more stable
15:59:17 <rhe00> anteaya: you mean the one about lefthand ci failing
15:59:24 <asselin_> rhe00, yes
15:59:24 <anteaya> rhe00: failing ci's only deter human reviewers
15:59:25 <rhe00> a patch and the patch got merged
15:59:27 <anteaya> rhe00: yes
15:59:42 <anteaya> third party ci's won't ever prevent merging
15:59:53 <rhe00> hmm
15:59:56 <anteaya> which is why operators need to pay attention to their own systems
15:59:56 <rhe00> ok
15:59:59 <anteaya> and we are at time
16:00:04 <anteaya> thanks everyone
16:00:07 <anteaya> see you next week
16:00:12 <anteaya> #endmeeting