21:53:57 <krtaylor> #startmeeting third-party
21:53:57 <openstack> Meeting started Wed Mar  4 21:53:57 2015 UTC and is due to finish in 60 minutes.  The chair is krtaylor. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:53:58 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:54:01 <openstack> The meeting name has been set to 'third_party'
21:54:47 <krtaylor> NOTE: this is a replay from the meeting held earlier today at 1500UTC, in order for it to be logged
21:54:56 <krtaylor> <asselin_> hi
21:54:56 <krtaylor> <ja_> morning
21:54:57 <krtaylor> <krtaylor> hm, no meeting services?
21:54:57 <krtaylor> <wznoinsk> hi
21:54:57 <krtaylor> <mmedvede> o/
21:54:57 <krtaylor> * patrickeast (~patrick.e@50-205-1-130-static.hfc.comcastbusiness.net) has joined #openstack-meeting-4
21:55:00 <krtaylor> <krtaylor> we aren't logging for some reason
21:55:02 <krtaylor> <krtaylor> hm, well I guess I cut/paste the log after the meeting to somewhere
21:55:04 <krtaylor> <krtaylor> who's here for third-party-ci-wg meeting?  :)
21:55:06 <krtaylor> <patrickeast> o/
21:55:08 <krtaylor> <ja_> here for 3p
21:55:12 <krtaylor> <asselin_> hi
21:55:14 <krtaylor> <krtaylor> nope, that didnt work either
21:55:16 <krtaylor> <krtaylor> oh well
21:55:18 <krtaylor> <wznoinsk> o/
21:55:20 <krtaylor> <krtaylor> hey everybody
21:55:22 <krtaylor> <krtaylor> I guess we'll carry on and see about restarting the service later
21:55:24 <krtaylor> <krtaylor> so, in case it is working, but being silent
21:55:26 <krtaylor> <krtaylor> here is the link for the agenda
21:55:28 <krtaylor> <krtaylor> #link https://wiki.openstack.org/wiki/Meetings/ThirdParty#3.2F4.2F15_1500_UTC
21:55:30 <krtaylor> <krtaylor> pretty light agenda today, but we can talk about some of the work being done
21:55:32 <krtaylor> <krtaylor> no system being highlighted today
21:55:34 <krtaylor> <krtaylor> let me know if you'd like to share what you are testing and how, problems you fixed, tools you built, etc
21:55:37 <krtaylor> <krtaylor> I'll get your system scheduled for a future meeting
21:55:39 <krtaylor> * hareeshp (hareeshp@nat/cisco/x-qsvcmndsyksqnxtv) has joined #openstack-meeting-4
21:55:43 <krtaylor> <krtaylor> any volunteers?
21:55:45 <krtaylor> <wznoinsk> I'd like to present our Intel Networking CI
21:55:47 <krtaylor> <wznoinsk> next meeting (in 2weeks) works for me
21:55:49 <krtaylor> <krtaylor> wznoinsk, excellent
21:55:51 <krtaylor> <krtaylor> I'll put you on the agenda
21:55:53 <krtaylor> * yamahata (~yamahata@c-67-160-193-250.hsd1.ca.comcast.net) has joined #openstack-meeting-4
21:55:55 <krtaylor> <krtaylor> wznoinsk, have you seen some of the previous highlight discussions?
21:55:57 <krtaylor> * yamahata has quit (Read error: Connection reset by peer)
21:55:59 <krtaylor> * yamahata (~yamahata@c-67-160-193-250.hsd1.ca.comcast.net) has joined #openstack-meeting-4
21:56:01 <krtaylor> <wznoinsk> yes, two and one from patrickeast last time
21:56:03 <krtaylor> <krtaylor> wznoinsk, great, I'll contact you and answer any questions about format
21:56:05 <krtaylor> <wznoinsk> if there's any formal way of presenting I think we can talk about it after the main topics
21:56:08 <krtaylor> <wznoinsk> good, thanks
21:56:12 <krtaylor> <krtaylor> I need to create a template for ideas to share, nothing too format
21:56:14 <krtaylor> <krtaylor> formal
21:56:16 <krtaylor> * asselin_ will step away for a few minutes
21:56:18 <krtaylor> <krtaylor> but some ideas on what would be useful to share with others
21:56:20 <krtaylor> <wznoinsk> I'll review the previous presentation as well again
21:56:22 <krtaylor> <krtaylor> #action krtaylor to put together a template for highlighting TPWG systems, suggestions for information to share
21:56:25 <krtaylor> <wznoinsk> but basically I'll be focusing on dockerizing the CI as we did
21:56:27 <krtaylor> * pkoniszewski has quit (Ping timeout: 244 seconds)
21:56:29 <krtaylor> <krtaylor> wznoinsk, excellent, I'll put you down
21:56:31 <krtaylor> <krtaylor> lets move on to the agenda and we can come back to this in open discussion
21:56:33 <krtaylor> <krtaylor> #topic Third-party CI documentation
21:56:35 <krtaylor> <krtaylor> so this is moving along, although very slowly
21:56:37 <krtaylor> <krtaylor> there have been a few patches merge
21:56:39 <krtaylor> <krtaylor> but more is needed, however, I am not pushing on running-your-own as much anymore with the new openstack-ci changes coming
21:56:44 <krtaylor> * david-lyle_afk has quit (Remote host closed the connection)
21:56:46 <krtaylor> <krtaylor> but we will talk about that in a few minutes
21:56:48 <krtaylor> <krtaylor> only 3 left in the queue
21:56:50 <krtaylor> * asselin_ is back
21:56:52 <krtaylor> <krtaylor> #link https://review.openstack.org/#/q/topic:third-party-ci-documentation,n,z
21:56:54 <krtaylor> * ChuckC_ (~ccarlino@76-218-11-17.lightspeed.sntcca.sbcglobal.net) has joined #openstack-meeting-4
21:56:57 <krtaylor> * ChuckC has quit (Ping timeout: 240 seconds)
21:56:59 <krtaylor> <ja_> #link https://review.openstack.org/#/c/149657/  looks pretty ready for >30 days.  you said things have been slow.  do we have a sense of the bottleneck?
21:57:02 <krtaylor> <krtaylor> yeah, I've been trying not to annoy infra cores any more than normal and get them approved  :)
21:57:05 <krtaylor> <krtaylor> they will happen, lots of reviews from us will help refine them
21:57:07 <krtaylor> <krtaylor> smaller chunks instead of TLDR helps too
21:57:09 <krtaylor> <krtaylor> but, I'll see if I can beg some +2's on them
21:57:13 <krtaylor> <krtaylor> any questions or volunteers on docs?
21:57:15 <krtaylor> <krtaylor> this work is kinda in limbo with the new changes coming, IMO
21:57:17 <krtaylor> <krtaylor> and that is a nice transition...
21:57:19 <krtaylor> <krtaylor> #topic Spec for in-tree 3rd party ci solution merged
21:57:21 <krtaylor> <krtaylor> aka openstack-ci
21:57:23 <krtaylor> <asselin_> krtaylor, I agree we should ask them to review. These should be merged and changed again later if necessary
21:57:26 <krtaylor> <krtaylor> agreed
21:57:28 <krtaylor> <asselin_> krtaylor, we can send them the gerrit topic
21:57:30 <krtaylor> * pmesserli (~pmesserli@50.56.229.5) has joined #openstack-meeting-4
21:57:32 <krtaylor> <krtaylor> yeah, and I think that will wrap up the documentation effort for a while, prob post summit
21:57:35 <krtaylor> * ChuckC_ has quit (Ping timeout: 250 seconds)
21:57:37 <krtaylor> <asselin_> yes, openstack-ci "in tree" spec merged on monday.
21:57:39 <krtaylor> <krtaylor> asselin_, anything you want to say on the spec merge?
21:57:43 <krtaylor> * lennyb_ (c12fa5fb@gateway/web/freenode/ip.193.47.165.251) has joined #openstack-meeting-4
21:57:45 <krtaylor> <krtaylor> yes, great news
21:57:47 <krtaylor> <asselin_> gerrit topic is changing to puppet-downstream
21:57:49 <krtaylor> <lennyb_> Hello
21:57:51 <krtaylor> <krtaylor> hm, I thought is got changed back yesterday
21:57:53 <krtaylor> <asselin_> #link https://review.openstack.org/#/q/topic:+downstream-puppet,n,z
21:57:55 <krtaylor> <asselin_> I mean, downstream-puppet
21:57:57 <krtaylor> <asselin_> krtaylor, no, we'll have one topic for the 2 related specs.
21:57:59 <krtaylor> * krtaylor thought it was decided to be openstack-ci
21:58:01 <krtaylor> <asselin_> the puppet module name will be that
21:58:03 <krtaylor> <asselin_> #link http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-03-03-19.01.txt
21:58:05 <krtaylor> <krtaylor> whatever, I don't have a strong feeling either way (not a fan of downstream) as long as it stops changing  :)
21:58:08 <krtaylor> <asselin_> krtaylor, agree.
21:58:12 <krtaylor> * openstack (~openstack@eavesdrop.openstack.org) has joined #openstack-meeting-4
21:58:15 <krtaylor> * openstack has quit (Changing host)
21:58:16 <krtaylor> * openstack (~openstack@openstack/openstack) has joined #openstack-meeting-4
21:58:18 <krtaylor> * ChanServ gives channel operator status to openstack
21:58:20 <krtaylor> <asselin_> krtaylor, for reviewers, it's easier to use one topic for the 2 specs since they're related.
21:58:23 <krtaylor> * rhe00 (~IceChat9@207.250.72.10) has joined #openstack-meeting-4
21:58:25 <krtaylor> <krtaylor> asselin_, yes, I was just referring to "19:27:44 <jeblair> actually the spec said "openstackci" as a topic"
21:58:28 <krtaylor> <krtaylor> whatever, good to clarify here
21:58:30 <krtaylor> <asselin_> so, we can start working on it now. First we'll create the new repo, and I'll add some tasks to storyboard
21:58:33 <krtaylor> <krtaylor> anyway, what is the plan to move forward, how can we help
21:58:35 <krtaylor> <krtaylor> ok, good
21:58:37 <krtaylor> <asselin_> ppl can self-assign from there so we don't duplicate effort
21:58:40 <krtaylor> <krtaylor> agreed
21:58:43 <krtaylor> * b3rnard0 (~b3rnard0@2001:4802:7800:1:adbb:e94c:ff20:c30) has left #openstack-meeting-4 ("Textual IRC Client: www.textualapp.com")
21:58:46 <krtaylor> <ja_> do we have a sense of when it would be usable by someone starting from scratch?
21:58:48 <krtaylor> <krtaylor> similar to what we did with puppet-module split out
21:58:50 <krtaylor> <asselin_> #link https://storyboard.openstack.org/#!/story/2000101
21:58:52 <krtaylor> * ajmiller (~ajmiller@74.202.214.170) has joined #openstack-meeting-4
21:58:54 <krtaylor> <ja_> e.g. would we tell someone about to start building a new one to wait 'n' weeks, or proceed now
21:58:57 <krtaylor> <krtaylor> target liberty I would assume?
21:59:00 <krtaylor> <asselin_> not much yet. I'll populate some items today. Anyone can add stuff too FYI
21:59:02 <krtaylor> <asselin_> there's no target...it's a priority effort
21:59:04 <krtaylor> <asselin_> krtaylor, but honestly, it would be great to be done by the liberty summit...
21:59:06 <krtaylor> <krtaylor> well as with puppet-module, use us for help
21:59:08 <krtaylor> <krtaylor> thats aggressive...but maybe doable
21:59:10 <krtaylor> <krtaylor> asselin_, if not a hack day there would be good too
21:59:14 <krtaylor> <asselin_> krtaylor, yes, we'll probably use that. it was very effective.
21:59:16 <krtaylor> <krtaylor> agreed, the Friday mini sprint is very cool, nice to have everyone in one room
21:59:18 <krtaylor> <asselin_> after we get a few patches going, we can do a sprint to review and get pieces merged
21:59:21 <krtaylor> <krtaylor> oh, I was talking about at summit
21:59:23 <krtaylor> <krtaylor> but a virtual sprint would be effective too
21:59:25 <krtaylor> * evgenyf has quit (Ping timeout: 252 seconds)
21:59:27 <krtaylor> <asselin_> yes, maybe after kilo releases during the downtime. in case there are riskier changes
21:59:30 <krtaylor> <krtaylor> agreed, this is the start of really doing something to improve the consumables
21:59:32 <krtaylor> <krtaylor> I encourage everyone to get involved in this
21:59:34 <krtaylor> <krtaylor> ok, anything esle you want to mention asselin_ ?
21:59:36 <krtaylor> <asselin_> not on this topic
21:59:38 <krtaylor> <krtaylor> ok, onward then
21:59:40 <krtaylor> <krtaylor> #topic Repo for third party tools
21:59:44 <krtaylor> <krtaylor> I have not made any progress on this, this week
21:59:46 <krtaylor> <krtaylor> day job keeps getting in the way
21:59:48 <krtaylor> <krtaylor> but, I plan on returning to this today/tomorrow
21:59:50 <krtaylor> * galstrom_zzz is now known as galstrom
21:59:52 <krtaylor> <krtaylor> here is the etherpad
21:59:54 <krtaylor> <krtaylor> #link https://etherpad.openstack.org/p/third-party-ci-wg-repo
21:59:56 <krtaylor> <krtaylor> there is some good ideas there
21:59:58 <krtaylor> <krtaylor> I am leaning toward starting slower, after thinking about it for a while
22:00:00 <krtaylor> <krtaylor> I think having an index of tools would be a good start
22:00:02 <krtaylor> <krtaylor> then we can see what we have and maybe better organize
22:00:04 <krtaylor> <patrickeast> i like that idea
22:00:06 <krtaylor> <krtaylor> this can also come out of the system highlight discussions
22:00:08 <krtaylor> <krtaylor> also, it gives us a way to collect up some tools and see how many we are talking about
22:00:13 <krtaylor> <krtaylor> if we only have a few that companies are willing to share, then not worth the effort
22:00:16 <krtaylor> <krtaylor> an index into individual github accounts would be great and serve the intent of why we are doing this in the first place
22:00:19 <krtaylor> <krtaylor> please share any thoughts in the etherpad, I will too
22:00:21 <krtaylor> <patrickeast> will do
22:00:23 <krtaylor> <krtaylor> any questions on this?
22:00:25 <krtaylor> <krtaylor> next then
22:00:27 <krtaylor> <krtaylor> #topic What to do with monitoring dashboard
22:00:29 <krtaylor> <krtaylor> is sweston around?
22:00:31 <krtaylor> <krtaylor> there has been some discussion on this
22:00:33 <krtaylor> <krtaylor> I have been thinking about it too
22:00:35 <krtaylor> <krtaylor> it seems that what everyone wanted was to get the previous dashboard (now radar) working again
22:00:38 <krtaylor> <patrickeast> it kind of looks like the spec hasn�t really moved forward in a while
22:00:40 <krtaylor> <krtaylor> yeah
22:00:44 <krtaylor> <patrickeast> oh
22:00:46 <krtaylor> <patrickeast> i showed this at the monday meeting
22:00:48 <krtaylor> <patrickeast> for anyone who missed that one http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/?project=openstack%2Fcinder&user=&timeframe=24
22:00:51 <krtaylor> <patrickeast> i put together a lame little dash
22:00:52 <krtaylor> <krtaylor> not lame, excellent tool
22:00:54 <krtaylor> <patrickeast> the original intent was that i can�t see when zuul reports a failure (only jenkins) so i wanted to monitor the event stream
22:00:57 <krtaylor> <mmedvede> cool!
22:00:59 <krtaylor> <patrickeast> but i let it watch all the ci accounts
22:01:01 <krtaylor> <patrickeast> still has some serious perf issues
22:01:03 <krtaylor> <patrickeast> needs more than a weekend of abuse to get working
22:01:05 <krtaylor> <krtaylor> yeah, and it just needs some historical stats to be exactly what we need
22:01:07 <krtaylor> * asselin_ likes it
22:01:09 <krtaylor> <patrickeast> imo its a different solution than what sweston�s spec was aiming for
22:01:13 <krtaylor> <krtaylor> patrickeast, did you ever see the radar tool working?
22:01:15 <krtaylor> <patrickeast> no control, no stats, trends
22:01:17 <krtaylor> <patrickeast> nope, before my time
22:01:18 <sweston> krtaylor: yes, I am here
22:01:19 <krtaylor> <patrickeast> https://github.com/patrick-east/scoreboard
22:01:21 <krtaylor> <patrickeast> ^ thats the source for it if anyone wants to run it for themselves
22:01:23 <krtaylor> <patrickeast> the ec2 thing that test one is running on sucks
22:01:25 <krtaylor> <patrickeast> works better on a local machine with a little more network bandwidth
22:01:27 <krtaylor> <patrickeast> easily modified to only watch your ci account and jenkins
22:01:29 <krtaylor> * asselin_ notes his ci needs some work
22:01:31 <krtaylor> <patrickeast> asselin: hehe its a harsh judge, if 1/5 of your tests fail it marks the ci as a fail
22:01:34 <krtaylor> <krtaylor> radar - it was simple gauges that showed the percentages of passed, failed, skipped tests for each ci system and upstream jenkins
22:01:37 <krtaylor> <patrickeast> ahh gotcha
22:01:39 <krtaylor> <krtaylor> it was a set sample, I believe it was 30 days
22:01:43 <krtaylor> <patrickeast> did it query gerrit or just track events?
22:01:45 <krtaylor> <patrickeast> or both
22:01:47 <krtaylor> <krtaylor> gathered and presented, but the urls changed for CI systems and it broke
22:01:49 <krtaylor> <krtaylor> it was kinda hacky, but worked fine
22:01:51 <krtaylor> <krtaylor> problem is that we don't have a master list of CI systems anymore
22:01:53 <krtaylor> <patrickeast> well, with a little bit of work i can add those kind of stats to this dash if folks are interested
22:01:56 <krtaylor> <krtaylor> except the maillist one
22:01:58 <krtaylor> <patrickeast> i basically record the gerrit events in the db and can do whatever with em
22:02:00 <krtaylor> <lennyb_> I made a small script to monitor last N Jenkins JObs to see if they failed. can it help you ?
22:02:03 <krtaylor> <krtaylor> jhesketh and I discussed using his gathering backend at paris summit
22:02:05 <krtaylor> <patrickeast> lennyb_: do you mean jenkins like the infra jenkins account posting on gerrit or a jenkins server in general?
22:02:08 <krtaylor> * ChuckC_ (~ccarlino@15.219.162.26) has joined #openstack-meeting-4
22:02:10 <krtaylor> <asselin_> #link https://github.com/Triniplex/third-party-ci-dashboard
22:02:14 <krtaylor> <asselin_> patrickeast, FYI^^
22:02:15 <sweston> krtaylor: can you catch me up?  I have been busy with investors all day
22:02:16 <krtaylor> <krtaylor> I think that part is fairly easy, getting the list of systems to monitor, harder
22:02:18 <krtaylor> <lennyb_> patrickeast, Jenkins server in general
22:02:20 <krtaylor> <patrickeast> asselin_: oh nice, thanks!
22:02:22 <krtaylor> <asselin_> #link http://dashboard.triniplex.com/#!/
22:02:24 <krtaylor> <asselin_> sweston had it running here, but I don't see anything now ^^
22:02:26 <krtaylor> <krtaylor> actually  #link https://github.com/stackforge/radar
22:02:28 <krtaylor> <patrickeast> lennyb_: gotcha, i was looking to catch events a bit further down the chain when comments are added to gerrit, i noticed that for my ci it would sometimes report things like NOT_REGISTERED or just ERROR from zuul and jenkins would never even know it happend
22:02:32 <krtaylor> <krtaylor> the requirement was to have a place where a developer (core) could go see if a system was behaving
22:02:35 <krtaylor> <krtaylor> this just came up with a patch that failed on a system, but was ignored and merged anyway
22:02:38 <krtaylor> <krtaylor> mainly because ci systems are not seen as reliable
22:02:40 <krtaylor> <patrickeast> yea its a problem in cinder right now
22:02:44 <krtaylor> <wznoinsk> patrickeast: listening to stream event using ssh may be more reliable
22:02:46 <krtaylor> <krtaylor> if someone could check if it was reliable, then its report could be trusted if a patch comment showed a failure
22:02:49 <krtaylor> <patrickeast> wznoinsk: yep, the scoreboard thing does just that with paramiko
22:02:51 <krtaylor> <lennyb_> patrickeast, this issue we solved by searching ERROR/Traceback etc in /var/log/zuul and checking that Jenkins has done something in the last few hours ( statistics :) )
22:02:54 <krtaylor> * matrohon has quit (Ping timeout: 246 seconds)
22:02:56 <krtaylor> <patrickeast> lennyb_: ahh yea that would work too
22:02:58 <krtaylor> <krtaylor> also, radar was done without spec, not sure if this effort really requires one, but it would be good for us to unify on the work and come up with one really good solution
22:03:01 <krtaylor> * galstrom is now known as galstrom_zzz
22:03:03 <krtaylor> <krtaylor> so the question is - what to do with this spec?
22:03:05 <krtaylor> <krtaylor> it seems to be gating this effort
22:03:07 <krtaylor> <patrickeast> imo there are a few big features in the spec� it should probably be broken up
22:03:09 <krtaylor> <krtaylor> sweston has put some fine work into it
22:03:13 <krtaylor> <patrickeast> some of them like the ci control channel and notifications are blocking it
22:03:15 <krtaylor> <patrickeast> when all we really need right now is a dash that shows us what is broken
22:03:17 <krtaylor> <krtaylor> I suggested that we start with a cmd line
22:03:19 <krtaylor> * galstrom_zzz is now known as galstrom
22:03:21 <krtaylor> <krtaylor> oops, we are out of time
22:03:23 <krtaylor> <krtaylor> let's continue via email
22:03:24 <sweston> krtaylor, asselin:  I just completed a move to a new office, and will be bringing the dashboard back up soon
22:03:25 <krtaylor> <asselin_> agree, we should start with something simple. Keeps scope creep out of the spec. And put those in a new spec.
22:03:28 <krtaylor> <krtaylor> thanks everyone, great meeting!
22:03:30 <krtaylor> #endmeeting