18:00:49 #startmeeting third-party 18:00:50 Meeting started Mon Jun 16 18:00:49 2014 UTC and is due to finish in 60 minutes. The chair is anteaya. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:51 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:54 The meeting name has been set to 'third_party' 18:00:58 hi all 18:01:05 who is here for the third-party meeting? 18:01:10 I am 18:01:13 o/ 18:01:15 I am 18:01:18 o/ 18:01:24 o/ 18:01:25 o/ 18:01:35 great welcome 18:01:39 here is our agenda 18:01:43 #link https://wiki.openstack.org/wiki/Meetings/ThirdParty#Agenda_for_next_meeting 18:01:58 #topic Welcome & Reminder of OpenStack Mission 18:02:04 so Welcome 18:02:11 anyone here for the first time? 18:02:15 o/ 18:02:26 hi SergeyLukjanov 18:02:27 (not for the first time) 18:02:31 yeah :D 18:02:43 so reminding everyone what the openstack mission is 18:02:47 #info The OpenStack Open Source Cloud Mission: to produce the ubiquitous Open Source Cloud Computing platform that will meet the needs of public and private clouds regardless of size, by being simple to implement and massively scalable. 18:02:48 I joined the infra when we talked about CI. Not this one though 18:02:56 lyxus: welcome 18:03:10 #topic Review of previous week's open action items 18:03:25 there are no open action items from last week 18:03:29 that was quick 18:03:41 #topic Announcements 18:03:48 anyone have any announcements? 18:03:58 mestery: you will see I moved your item 18:04:14 anteaya: thx 18:04:19 no announcements? 18:04:23 let's move on 18:04:27 #topic OpenStack Program items 18:04:43 #info Review of Neutron 3rd party driver/plugin requirements wiki page (mestery): 18:04:47 mestery: the floor is yours 18:04:53 anteaya: Thanks 18:05:00 #link 18:05:03 #link https://wiki.openstack.org/wiki/NeutronPlugins 18:05:22 So, this somewhat ties into this page as well: 18:05:30 #link https://wiki.openstack.org/wiki/NeutronThirdPartyTesting 18:05:40 Mostly, I had hoped to solicit feedback from everyone at this meeting. 18:05:54 anteaya has worked with me on parts of this already, and her help was much appreciated! 18:05:59 np 18:06:03 Does anyone have comments? 18:06:10 so all of it or any part in particular? 18:06:30 anteaya: Well, perhaps the general direction, but specifics are fine as well. ;) 18:06:32 if folks are seeing this for the first time it might be a bit of a chunk to give feedback right now 18:06:41 okay 18:06:43 Mostly, I wanted to document hte process elements here so things are written down and expectations are known 18:06:50 right 18:06:58 and I goal I stand behind 18:07:03 thanks! 18:07:12 anyone willing to read this and give mestery some feedback on it? 18:07:24 doesn't have to be right now but maybe in the next two days? 18:07:25 trinaths: Your input would be great here if you have time! 18:07:27 I can volunteer for that 18:07:29 mestery, I had seen the second page before, but I'll chime in 18:07:30 sweston: thanks! 18:07:37 krtaylor: perfect, thank you too! 18:07:37 mestery: neutron 3rd party testing links really helps in what is expected for a 3rd party CI testing 18:07:39 sweston krtaylor awesome 18:07:48 Most of the points make sense from my point of view. ie: provide logs, last timestamp, etc... The only input that I would have is to have a clear set of what kind of tests are required to be run 18:08:05 lyxus: good input, thanks! 18:08:46 anyone have anything more at this time? 18:08:53 Maybe another one :) 18:08:57 mestery, when I was doing my survey of third party documentation before summit, itwas one of the best I found, but I'll look at it in more detail 18:09:00 go ahead 18:09:39 mestery: [doubt] is that all the test cases be passed for a given patchset and change? 18:09:49 also could be nice to have the link to the neutron 3rd party requirements on the general 3rd party page 18:09:54 The health of the CI is extremely important. There was an email this AM about the % success vs failure. could you clarifiy that ? 18:09:57 trinaths: Yes, that would be the case I believe. 18:10:08 luqas: we don't have a general third party page yet, I am working on it 18:10:19 luqas: and yes once we have it, the links can be there 18:10:25 ie: does failure means voting -1 or wrongly reporting -1/+1 18:10:51 mestery: okay. 18:11:10 lyxus: Understood, I'll clarfiy that, good input! 18:11:15 failure means build failure 18:11:20 anteaya: I thought this one was the general http://ci.openstack.org/third_party.html 18:11:28 wrongly reporting means disagreement with jenkins 18:11:39 luqas: ah, I see what you mean 18:11:57 luqas: hmmmm, do feel free to offer a patch with that change and we can review it 18:12:12 anteaya, Not exaclty it could be an issue in the Third party infra that made a -1 where it should'nt be 18:12:17 I'm not saying yes or no right now, I would need to think about it and hear from others 18:12:36 lyxus: sorry I thought you were asking for definitions 18:12:58 i wasn't clear sorry ! 18:13:10 lyxus: if a third party ci system reported a failed build when they shouldn't what would be the situation? 18:13:27 1) the third party ci is mailfuntioning 18:13:31 well, it would be nice to have links to all the specifics for the different 3rd party components 18:13:35 2) the third party ci is incorrect 18:13:43 what are the other posssiblities? 18:13:55 yes is would 18:14:09 part of the reason for these meetings is to deine all the things 18:14:15 hm, maybe a terms/definitions section would be good for those starting out? 18:14:27 once we have defined all the things we can write them down and link to them 18:14:37 a good suggestion 18:14:38 I don't think so. I think it should be part of the metrics. (in my case, i don't do automatic -1, as i review them by hand ) 18:14:44 sounds good to me 18:14:56 it would be good to have phrasing that we agree on defined 18:15:02 lyxus: failed, means [1] stack components did not start or [2] unable download the packages from internet - were there any thing you have noticed of build failure 18:15:21 lyxus: this is what I am asking you 18:15:33 lyxus: please expand on your perspective 18:15:41 I don't understand your point yet 18:16:42 I'm not saying you don't have a point, I'm saying I don't know what it is 18:16:47 we should uncouple voting -1/+1 and failure maybe? 18:16:58 I agree 18:17:09 since failure and success are build outcomes 18:17:19 one is potentially the result of the other 18:17:20 yes 18:17:26 as I know, a ci build fails when [1] unable to download the packages from internet , [2] a stack service did not start or [3] a test completely fails. 18:17:29 potentially yes 18:17:42 I think i went to far on my idea. I was just wondering how we are going to measure the health of the CI. As long as we have clear requirements , I am fine with that. I just don't think that voting -1 should be part of the metrics 18:17:48 trinaths: those are the options I also am aware of 18:17:58 if there are others, I would like to know what they are 18:18:13 trinaths, or, it could be a race condition, we see that often 18:18:16 lyxus: okay fair enough 18:18:44 lyxus: why do you feel voting -1 should not be part of the metrics? 18:20:33 anteaya: it's possible someone commits code the legitimately breaks a driver. That's not a failure or instability of the ci system 18:20:43 akerr: right 18:20:53 that is one of the possible outcomes of testing 18:21:03 -1 counting will give you a lot of red herrings if you use them to look for instability 18:21:09 and rather the point of the whole affair 18:21:33 anteaya, Voting -1 means that there is a problem with the dev code. As a dev, if i have a -1 it takes time to investigate it and see the issue. As the CI maintainer before doing a -1, i want to be sure that it is a real problem. This thing might take a bit 18:21:38 akerr: how do you propose we identify unstable systems? 18:21:52 anteaya, Don't get me wrong -1 are important. 18:21:55 anteaya: I don't have a good answer for that, but I don't think -1 counting is the right way 18:21:57 lyxus: fair enough 18:22:14 lyxus: and if you need time to communicate that is fair 18:22:23 keep in mind this meeting is to share ideas 18:22:32 we have no authority to make decisions 18:22:40 decisions like with the programs 18:22:54 infra, neutron, nova, cinder, etc 18:23:00 we are here to share ideas 18:23:06 agreed, but after discussing, we might be able to influence them a little bit 18:23:18 eventually 18:23:30 akerr: okay, I'm open to hearing what you feel the right way is after you have had some time to give it some thought 18:23:34 sure 18:23:41 we can present good ideas 18:24:03 and improve documentation and disemmination of information 18:24:18 so any more on this topic today? 18:24:45 Not for me 18:24:48 great 18:24:52 let's move on 18:24:56 #topic Deadlines & Deprecations 18:25:06 mestery: anything for this topic today? 18:25:24 anteaya: Nothing that isn't in the published wiki pages or the email I sent. 18:25:35 let's put it in the logs as well 18:25:36 People ahve been very proactive about sending me emails, I need to reply and indicate they shoudl also copy the list. 18:25:44 in case people only read meeting logs 18:25:46 OK 18:25:50 thanks 18:26:04 The plan is to ensure all Neutron 3rd party CI systems are running by Juno-2. 18:26:16 If they are not, we will take steps to remove code from the upstream tree. 18:26:27 awesome thank you 18:26:32 I hope CI admins use this meeting to get help and work with the broader community before the deadline! 18:26:45 and you have at least two drivers slated for deprecation, yes? 18:26:51 mestery: thanks, me too 18:26:57 Correct; Linuxbridge and OVS will be removed in Juno-2. 18:27:01 thanks 18:27:03 ML2 is the replacement for both. 18:27:09 * anteaya nods 18:27:24 any other openstack program reps with deadlines or deprecations today? 18:27:45 moving on 18:27:49 #topic Highlighting a Program or Gerrit Account 18:28:04 #info How DriverLog can help tracking 3rd party CIs status 18:28:12 ilyashakhat__: is that your topic? 18:28:25 yep, i'm here to help) 18:28:29 great 18:28:31 welcome 18:28:39 let's start off with some background shall we 18:28:51 what is driverlog, and what is the current status? 18:29:52 DriverLog does 2 things: 1) it is a list of drivers 18:30:05 how is the list of drivers compiled? 18:30:18 2) for drivers with enabled CIs it collects votes and shows their status 18:30:21 what files are scraped to arrive at the driverlog list? 18:30:36 let's start off with the list of drivers 18:30:42 how does that occur? 18:30:57 is DriverLog, active now. can we access it 18:31:02 the list was created manually and updated by community 18:31:18 trinaths: http://stackalytics.com/report/driverlog 18:31:35 the patches are accepted to project stackforge/driverlog 18:31:36 ilyashakhat__: where is the file containing the list so that people can file patches? 18:32:02 yes, https://github.com/stackforge/driverlog/blob/master/etc/default_data.json 18:32:41 for every driver it is possible to specify static list of versions, 18:32:59 okay so for drivers either present that shouldn't be or missing that should be there, folks can file patches to that file, correct? 18:33:00 or this list can be calculated by gerrit reviews 18:33:19 static list of versions 18:33:22 anteaya: correct 18:33:28 what do you mean, openstack release versions? 18:33:40 yes 18:33:42 https://wiki.openstack.org/wiki/DriverLog 18:33:44 okay great 18:34:03 anyone with any other questions about the list of drivers and how that is compiled? 18:34:04 ilyashakhat__, maintainers: is it the code maintainer or Ci maintainer. I assume also that you can have more than one maintainer. Seem to be a list 18:34:44 lyxus: maintainer = person who can respond to any questions regarding the driver 18:34:50 it is a list 18:34:59 ilyashakhat__, thank you 18:35:03 ilyashakhat__: how is the list of maintainers compiled? 18:35:08 ilyashakhat__, when can a unlisted driver be added, after CI is approved for voting ? 18:35:11 where did you get that information? 18:35:49 anteaya: just digging over wiki-pages :) 18:36:02 ilyashakhat__: hmmmm, not really a canonical list 18:36:19 for something as sdague rightly pointed out is consummed by the media 18:36:28 trinaths: you may add driver first and then update it with CI info 18:36:49 ilyashakhat__: so maintainers add their driver when they want to? 18:36:56 yes 18:36:59 ilyashakhat__, okay. it like git commit ? 18:37:00 no review of status for them to be added? 18:37:16 trinaths: it is a stackforge project 18:37:20 anteaya: what do you mean? 18:37:25 trinaths: it follows the openstack workflow 18:37:37 anteaya: okay thanks 18:37:39 ilyashakhat__: so the thing with anything with third party is status 18:37:45 readyness 18:37:49 stability 18:38:08 and third party vendors are motivated to say they are ready when they are not ready 18:38:20 having driver on the list doesn't mean it complies to 3rd-party requirements 18:38:30 like mestery's round up of third party ci system's for neutron has shown 18:38:40 in that case the will get red mark meaning they have no CI 18:38:43 ilyashakhat__: what does having a driver on the list mean? 18:39:02 it means that driver exists in the world 18:39:02 what is the meaning of having a driver on the list 18:39:06 does it 18:39:16 you just said it means they submitted a patch 18:39:22 aaa 18:39:29 ilyashakhat__,then it will be a big list, when driver is listed here, it may be within the CI requirements 18:39:40 ilyashakhat__, If i want to modify the default_data.json. Pull request is the way to go ? 18:40:04 lyxus: git review 18:40:12 ilyashakhat__, ok 18:40:19 it follows standard OpenStack process 18:40:35 ilyashakhat__, perfect ! 18:40:36 ilyashakhat__, i think, the drivers/ci need to added whe CI meets the requirements and is voting .. 18:40:37 so my point is that driverlog is estabilishing that it has no meaning 18:40:54 if it is just a place where vendors can shove code for marketers to consume 18:41:25 why? 'CI tested' column is the right place to look for drivers with CI enabled 18:41:32 ah yes 18:41:39 let's get to CI tested 18:41:57 which we already established last week on the ml doesn't actually mean tested 18:42:05 so let's look at that column 18:42:17 the entries for that column are two icons 18:42:17 yep, i think this needs to be fixed 18:42:24 am I correct so far? 18:42:37 a green check mark and an arrow of some kind 18:42:48 pointing from lower left to upper right 18:42:56 ilyashakhat__: is this accurate so far? 18:43:14 (arrow means that link will be opened in new tab) 18:43:17 one thing is CI tested and another is that the CI fullfills the requirements 18:43:17 anteaya: yes 18:43:25 luqas: right 18:43:31 so maybe another mark is needed 18:43:33 ilyashakhat__: okay what do the icons mean? 18:43:59 red means "no CI", green "CI is configured and has at least one vote" 18:44:06 red what? 18:44:07 to tell if the CI is "certified" by OS 18:44:16 no certified 18:44:25 nothing is certified by anyone for any reason 18:44:30 luqas: don't use "certified" :) 18:44:36 sorry 18:44:38 :( 18:44:45 ilyashakhat__: so the check mark can be green or red? 18:44:46 red mark means that the driver has no CI 18:44:51 is that what you are say? 18:44:53 yes 18:45:01 what about the arrow 18:45:10 what colour choices does the arrow have 18:45:20 arrow is just a part of design 18:45:28 why have it? 18:45:36 what information is it conveying? 18:45:46 it means that the link will be opened in new tab 18:45:52 link to what? 18:45:57 web-link 18:46:02 but what does CI tested really mean? just running tests? or tested to pass some level of requirements? 18:46:04 web-link to what? 18:46:18 krtaylor: so far it appears to mean they have a gerrit account 18:46:35 anteaya: to the latest review where the driver's CI ran 18:46:35 which has voted at least once 18:46:36 green check mark is "gerrit account exists" 18:46:57 krtaylor: CI tested = just running tests 18:46:58 luqas: it could have voted on teh sandbox repo 18:47:08 luqas: that is useless information 18:47:15 ilyashakhat__: but it isn't 18:47:21 then it shouldnt be called "tested" 18:47:30 right now all it is saying is the gerrit account exists 18:47:42 yes, I agree with krtaylor 18:47:45 anteaya: then the vote on the sandbox repo shouldn't compute 18:47:51 "CI running" would be more descriptive 18:47:57 ok, we can rename it 18:48:01 luqas: shouldn't and does are two different things 18:48:05 ilyashakhat__: please do 18:48:22 ok 18:48:23 ilyashakhat__: I'm not saying I disagree with the direction 18:48:35 I am saying taht you are currently conveying inaccurate information 18:48:51 in a field rife with inaccurate information 18:48:57 ilyashakhat__, I'll patch today for PowerKVM, it was missed in the initial population 18:49:00 but we are community, and we can improve it :) 18:49:06 anteaya, agreed 18:49:11 making it very very hard for people who want to do the right thing 18:49:34 ilyashakhat__: good, the first improvement is to cease conveying inaccurate information 18:49:39 as fast as you can 18:50:13 actually CI exists is the accurate discription of that column 18:50:36 since it isn't polling on current status of systems 18:50:55 and has green check marks for systems that haven't voted since februaruy 18:51:09 anteaya, then can't it self populate from the CI group? 18:51:20 it doesn't seem to be doing so 18:51:26 if take only drivers with CIs, what kind of dashboard would you like to see? 18:51:34 I don't know what it can or can't do, I jsut know what it is doing 18:51:46 we need to move onto the next agenda item 18:52:04 ilyashakhat__: will you agree to rename that column from ci tested to ci exists? 18:52:17 anteaya: yes, I agree 18:52:25 and we can work on how to communicate other information on status 18:52:28 ilyashakhat__: thank you 18:52:42 #action ilyashakhat__ to rename driverlog ci tested? to ci exists 18:52:53 we can talk more about driverlog next week 18:52:56 fair enough? 18:53:21 okay I need to give trinaths some time 18:53:24 #info Discussion on Improving CI performance and Multi Node setup (trinaths) 18:53:29 trinaths: you're up 18:53:33 yes 18:53:54 what is on your mind 18:54:26 with my CI, I see that it takes about an hour for a build to complete. any ideas on lowering the test run time 18:54:43 trinaths: builds on infra take about an hour 18:54:55 so the build time on your system is about the same as infra 18:55:06 that's a good metric 18:55:06 depends on what tests you are running 18:55:13 what tests are you running? 18:55:18 getting all the openstack components from git take much time 18:55:25 can that be posted somewhere .. a lot of people ask about that 18:55:28 how much time? 18:55:38 sweston: sure, where do you suggest? 18:55:45 as suggested in Neutron3rdParty testing linl 18:56:10 mestery: any objection to adding that metric to the neutron third party wiki page? 18:56:11 anteaya: possibly a wiki 18:56:40 anteaya: I think adding that metric would be fine. 18:56:46 great thanks 18:56:47 is that longer time for a build a fault with my CI, any suggestions on improvement 18:56:47 I have cut down a lot of the building time by having a local pypi and having part some of the ubuntu packgages installed. 18:56:58 trinaths: do you want to add that and sweston can you review his changes? 18:57:15 lyxus: great, have you documented your process anywhere? 18:57:17 anteaya: glad to 18:57:21 on a blog post perhaps? 18:57:29 sweston: thanks 18:57:33 anteaya, I can write something if you want to 18:57:40 what changes ? 18:57:42 lyxus: that would be great if you could 18:57:57 anteaya, I will do that. 18:58:03 lyxus: thanks 18:58:12 and you can add the url to next week's meeting 18:58:22 and we can applaud in gratitude 18:58:24 :D 18:58:40 anteaya: yw 18:58:55 We'll see :) I will write it this week. 18:58:56 trinaths: do you want to add the information that it takes about an hour to run a build to the neutron wikipage? 18:59:05 trinaths: sweston is willing to help you 18:59:13 lyxus: awesome thank you 18:59:19 we are out of time 18:59:38 sorry if I didn't get to your item, please add it to next week's agenda 18:59:39 anteaya, sure. sweston help may solve the problem 18:59:43 thanks for a great meeting 18:59:44 good meeting! 18:59:48 see you next week 18:59:51 thanks ! 18:59:51 by 18:59:52 #endmeeting