19:08:45 <jeblair> #startmeeting ci
19:08:46 <openstack> Meeting started Tue Oct 23 19:08:45 2012 UTC.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:08:47 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:08:48 <openstack> The meeting name has been set to 'ci'
19:09:53 <jeblair> #topic actions from last meeting
19:10:31 <jeblair> fungi: we're still waiting on the foundation server for the CLA thing, yeah?
19:11:14 <fungi> yes, toddmorey e-mailed me last week to say that he has something mostly working but was too busy to meet up with the summit going on
19:11:37 <fungi> so i'm waiting to hear back from him now that things are hopefully cooling down again
19:11:42 <jeblair> #action toddmorey provide a test foundation server
19:12:39 <jeblair> #topic summit
19:12:52 <jeblair> So we were all pretty busy at the summit
19:12:57 <clarkb> #link https://etherpad.openstack.org/state-of-ci
19:13:16 <mordred> wait, we were all at the summit?
19:13:18 <pabelanger> clarkb: Thanks for the link
19:13:20 <jeblair> Monty is going to translate those items into bugs
19:13:23 <mordred> I am
19:13:26 <mordred> I'm so happy about it
19:13:29 <jeblair> #action mordred bugify summit actions
19:13:36 <mordred> if people could go through other todo list items they got from the summit
19:13:44 <mordred> and add them to that (or just make bugs from them)
19:13:47 <mordred> I'd appreciate it
19:14:03 <mordred> I'm also going through the current grizzly list and getting rid of or re-prioritizing cruft
19:14:18 <jeblair> #action everyone collect action items from other summit session etherpads and register as bugs
19:14:49 <pabelanger> look at that 1 commit behind all of redhat for CI commits
19:14:50 <pabelanger> woot
19:14:56 <pabelanger> russellb: ^
19:15:13 <jeblair> pabelanger: nice showing!
19:15:54 <jeblair> any particular summit-related topics we should discuss here?
19:16:31 <fungi> i've renewed work on the continuously updating tags for branches on gerrit
19:16:41 <fungi> based on (brief) discussions at the summit
19:16:57 <fungi> more or less people agreeing with what jeblair and i already talked through
19:17:05 <clarkb> jeblair: the idea for a hosted log server might be worth bringing up
19:17:23 <jeblair> fungi: +1, and i saw you updated the bug
19:17:23 <fungi> clarkb: YES PLEASE
19:17:41 <jeblair> clarkb: yeah, why don't you give an overview?
19:18:44 <clarkb> everyone is testing openstack, but other than smokestack and us no one is reporting test results. Apparently companies make it hard to host their own results, but it was suggested that if we could provide a place to put logs people could just upload them there
19:19:00 <pabelanger> I'd be interested in getting more information about salt and puppet from the summit.  If somebody has a URL / presentation handy
19:19:54 <clarkb> also, we really need to make logs.o.o more easily consumed. The potential is there to fix these two problems with one handy log server setup
19:20:26 <clarkb> currently I am looking at logstash because it seems fairly configurable and there aer puppet modules to make it go (also their logo is awesome)
19:20:27 <fungi> ahh, i remember the testlog server discussions now. i was confusing that with the proposal for a central logserver for our infrastructure
19:20:57 <jeblair> fungi: perhaps those needs overlap a bit.
19:21:04 <mordred> jeblair: ++
19:21:17 <fungi> jeblair: potentially
19:21:39 <clarkb> if anyone has had experience with logstash or similar I am definitely interested in chatting
19:22:01 <fungi> last time i set up logservers it was rsyslogd and tenshi
19:22:02 <jeblair> pabelanger: nothing's coming to mind.  i'm sure lots of people were talking about those things, but i wasn't there.
19:22:39 <jeblair> pabelanger: other than some nice conversations we had with Ryan_Lane, but i think you know most of that.
19:22:56 <fungi> pabelanger: my only recollection of salt/puppet getting mentioned from a ci perspective was in the "state of ci" session
19:23:16 <mordred> yeah - the only thing is what we've talked about in channel
19:23:29 <mordred> that being, we'd like to start using salt to orchestrate our puppet runs
19:23:50 <mordred> so that we can make zuul trigger salt to immediately make changes on merge, rather than waiting for a double cron
19:24:04 <pabelanger> jeblair: cool.
19:24:11 <pabelanger> fungi: /nods
19:25:07 <jeblair> objections to moving on to server status/upgrades?
19:25:21 <fungi> seconded
19:25:32 <olaph> pabelanger: the only slides I can find are from a demonstration Dan Bode gave: http://www.slideshare.net/bodepd/openstack-grizzley-puppettalk
19:25:32 <jeblair> #topic server upgrades
19:25:58 <jeblair> So, we upgraded servers!
19:26:11 <clarkb> we did! it was awesome and we got 99% of it all working
19:26:20 <pabelanger> olaph: Ooo, checking it out. Thanks
19:26:41 <clarkb> the gerrit and jenkins master servers (all four) are now on rackspace nova
19:26:45 <pabelanger> jeblair: clarkb congrats
19:26:49 <clarkb> and have been upgraded to precise
19:26:54 <mordred> wq00t!
19:27:19 <jeblair> as is etherpad.o.o, which was also migrated from a sekrit rackspace server to one publicly managed by the infra team
19:27:30 <fungi> at this point there are no known regressions, right? just some new features we can't implement quite yet
19:27:30 <clarkb> in theory this also gives us ipv6 connectivity but openssh server apparently tickles a bug in ovs so as a precaution we have not created AAAA records for the two production servers yet
19:28:03 <clarkb> fungi: yup. I haven't heard any screaming. jgriffith did lose a pad but the link had the wrong url
19:28:19 <fungi> saw that. didn't sound like a bug, just a typo
19:28:40 <clarkb> yup
19:29:02 <clarkb> and the new etherpad hasn't melted down yet which is good
19:29:19 <fungi> aside from the usual jenkins bugginess, since reverted
19:29:19 <jeblair> clarkb: and we have system-level monitoring of it with cacti!
19:29:41 * fungi is very excited about cacti
19:29:44 <jeblair> #link http://cacti.openstack.org/cacti/graph_view.php
19:30:38 <jeblair> btw, for ram, i just pulled a number out of a hat when i made the etherpad host. it said 4g.
19:30:51 <jeblair> it looks like that was generous.
19:30:59 <fungi> is cacti running on a dedicated vm?
19:31:04 <jeblair> fungi: yes
19:31:21 <fungi> so if we also wanted to throw nagios or something on it down the road, that would probably be doable
19:31:44 <jeblair> fungi: yep
19:31:49 <fungi> spiffy
19:32:25 <jeblair> #topic grenade / quantum
19:32:27 <pabelanger> fungi: shouldn't be too hard.  I'd be able to help with that.  Use it with our asterisk instances
19:32:50 <fungi> pabelanger: cool. used it a ton myself over the years (long before it changed its name)
19:33:06 <jeblair> two devstack variants have been requested: grenade upgrade testing, and quantum.
19:33:27 <jeblair> i'm very close to being able to run grenade (next step: move the grenade project into openstack-dev)
19:33:57 <jeblair> and i have set up a silent job for quantum so that nachi can finish getting that working
19:34:59 <jeblair> on a related note, we should find out if people are okay with us not running n-vol tests for master, since n-vol and cinder are in folsom, i think n-vol may be deprecated now?
19:35:09 <jeblair> jgriffith, vishy: ^ do you have an opinion on that?
19:35:45 <mordred> anotherjesse said that we could go ahead and till that
19:35:46 <mordred> kill
19:36:05 <fungi> but continue running them for stable/folsom and stable/essex updates right?
19:36:40 <jgriffith> jeblair: I'm ok with it (I think)
19:36:47 <jeblair> fungi: i reckon so.
19:36:57 <jgriffith> jeblair: we shouldn't be making changes to nova-vol any longer so...
19:37:31 <jeblair> #action jeblair deconfigure n-vol testing on master
19:38:09 <jeblair> #topic testr and friends
19:38:35 <clarkb> #link https://review.openstack.org/#/c/14611/
19:39:18 <clarkb> I figured we needed to do a small proof of concept for testr and friends on a project that would be easy enough that this whole thing won't lose traction
19:39:19 <mordred> testr and testtools and fixtures are super cool
19:39:41 <clarkb> so with lifeless' help I hacked the above change for python-novaclient to use testr instead of nose
19:39:58 <jeblair> clarkb: good plan
19:40:02 <mordred> supersexy
19:40:14 <lifeless> o/
19:40:23 <mordred> lifeless: we like your libraries
19:40:45 <fungi> lifeless is an awesome librarian
19:40:52 <clarkb> I had to switch from unittest2 to testtools, update the base test class to capture stdout and save it in the subunit stream, correct setUp and tearDown methods (need to call parent class methods)
19:41:08 <clarkb> and some boilerplate config stuff in tox.ini and testr.conf
19:41:36 <clarkb> The current things we are missing are coverage, realtime output of test results, and human readable test logs
19:41:39 <mordred> clarkb: the changes to do testtools and setUp/tearDown should be landable without testr yeah?
19:42:00 <lifeless> clarkb: technically you get realtime output
19:42:08 <lifeless> clarkb: its just less IN YOUR FACE
19:42:19 <clarkb> mordred: yes, I think I should pull those out into another change and rebase things
19:42:25 <mordred> lifeless: s/realtime/IN YOUR FACE ANSI COLOR realtime output/
19:42:40 <jeblair> lifeless: what realtime output is there currently?
19:42:48 <lifeless> jeblair: it outputs errors as they happen.
19:42:55 <clarkb> and failures
19:43:02 <mordred> jeblair: it does what standard nose/unittest runs do without the openstack nose plugin
19:43:03 <lifeless> all classes of things going wrong.
19:43:12 <mordred> jeblair: ....F...E... etc
19:43:18 <lifeless> mordred: nope
19:43:23 <lifeless> mordred: it doesn't
19:43:25 <mordred> ok
19:43:37 <mordred> I thought I saw .'s yesterday
19:43:55 <lifeless> jeblair: it outputs anything outcome that will make the run be considered not-ok, as and when it happens.
19:43:59 <lifeless> with all its attachments etc.
19:44:15 <jeblair> so what other requirements are there driving the desire to implement something else?
19:44:38 <jeblair> i'm not sure blindly re-implementing openstack-nose-plugin without targeting specific goals is the best idea.
19:44:51 <mordred> prior experience I had trying to get us to plain nose before
19:45:25 <mordred> the people who have approval access have keeping that output as a requirement before they'd merge the change
19:45:38 <mordred> it's possible that we can convince them that the new features outweight the loss
19:45:45 <jeblair> mordred: yeah, but i seem to recall one specific objection was, eg, outputting either a "." or a test description rather than the test name.
19:46:19 <jeblair> mordred: so since testr-default-output != plain-nose-default-output, we should re-evaluate.  :)
19:46:43 <mordred> indeed ... and I think super-quick test runs might please people ... but I also think that writing an ansi color filter for subunit output shouldn't be terrible
19:46:47 <mordred> HOWEVER, you make a good point
19:47:03 <mordred> so when we have the rest of everything working, we'll have a chat with people to determine whether or not the color thing is important to them
19:47:24 <jeblair> i think identification of slow tests may have been a use-case too.
19:47:29 <mordred> indeed
19:47:30 <lifeless> testr slowest
19:47:33 <clarkb> for the other two issues: we can potentially get away with nose for coverage and subunit logs as the log files in the short term
19:47:34 <jeblair> (and that one in particular driving the color thing)
19:47:35 <lifeless> just run that :)
19:48:27 <jeblair> yeah, so lets see what the actual issues are, and perhaps there might be a more intelligent way to solve them other than expecting nova devs to stare at a screen identifying colors for 11 minutes while 3800 tests run.
19:48:31 <mordred> yeah ... the more we have to teach them how to run something new ...
19:48:34 <mordred> vishy: ping
19:49:02 <jeblair> mordred: we can tack on things like "testr slowest" in the tox config, right?
19:49:19 <clarkb> jeblair: yes, however tox short circuits on failure
19:49:21 <mordred> vishy: if we could reduce test run time by 10x or more and still had sensible output for slow things and failures, but lost the current color output formatting, would that be a non-starter?
19:50:12 <clarkb> jeblair: so we may need a lightweight wrapper like 'bash -c 'testr run --parallel ; testr slowest'
19:50:24 <lifeless> clarkb: you probably want pipefail there
19:50:40 <lifeless> clarkb: or perhaps something a little more sophisticated - preserve $? from testr run --parallel
19:50:40 <koolhead17> danwent, around?
19:50:42 <jeblair> clarkb: _if_ people care about slow test runs on failure (they might?)
19:50:47 <clarkb> lifeless: good point
19:50:59 <clarkb> jeblair: I think they should care on each run
19:51:09 <mordred> there is also the possibility that if hte system is sane and all, we can just show people how to do stuff with testr directly ...
19:51:20 <mordred> I just don't want to get bogged down in perception of regression
19:51:28 <clarkb> the tl;dr is yes we can add it to tox. it will just be a little ugly
19:51:53 <clarkb> however you don't see that ugly when running tox -efoo
19:52:19 <danwent> koolhead17: what's up?
19:53:07 <koolhead17> danwent, i changed the doc repo to released ubuntu cloud repo, it was testing at time guide was written.
19:53:25 <jeblair> koolhead17: we're having a ci meeting in this channel
19:53:44 <koolhead17> jeblair, srry :(
19:54:14 <jeblair> anyone else have something to say about testr?
19:54:39 <mordred> nope
19:54:42 <jeblair> #topic open discussion
19:54:46 <jeblair> 5 mins left!
19:54:53 <fungi> any bacon shortage updates?
19:55:01 <mordred> I currently have no bacon
19:55:09 <fungi> that sounds like a bug
19:55:33 <jeblair> i am fresh out.  but i will be eating more iberian ham soon, which makes me happy.
19:55:54 * clarkb is going to try and take the afternoon off if mordred doesn't scream about it
19:56:01 <fungi> i hope to have the commit-merged hook module for gerrit to update tags on review-dev checked in for review later this afternoon
19:56:15 <jeblair> sweet.
19:56:20 <fungi> then you can all kick it around
19:56:36 <jeblair> i'm playing around with graphite for collecting and displaying stats for devstack jobs (and maybe later, all jenkins jobs).
19:57:07 <fungi> i'm changing commit-merged into a .erb though so we can pass vars into it. hopefully that doesn't meet with concern
19:57:41 <jeblair> fungi: that should be fine in principle.
19:57:44 <fungi> since it will technically affect both review and review-dev from that perspective
19:58:05 <jeblair> oh, and jenkins-job-builder is getting more contributors!
19:58:15 <fungi> saw that. very exciting
19:58:26 <fungi> loops!
19:58:44 <jeblair> fungi: and loops and loops and loops...
19:59:04 <fungi> heh
19:59:16 <jeblair> #endmeeting