19:03:00 #startmeeting 19:03:01 Meeting started Tue Aug 21 19:03:00 2012 UTC. The chair is mtaylor. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:03:02 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:03:02 #meetingname CI 19:03:03 The meeting name has been set to 'ci' 19:03:15 oh wel 19:03:16 mtaylor: in general, how about the idea of "#startmeeting ci" 19:03:20 ++ 19:03:25 mtaylor: and making sure the logs go to /meetings/ci/DATE 19:03:27 also - now that we have logs.o.o ... 19:03:41 logs.o.o/meetings/ci/DATE ? 19:03:47 mtaylor: logs.o.o is build logs... 19:03:55 mtaylor: perhaps logs isn't the right name for that 19:04:10 mtaylor: we could change the build logs to "builds.openstack.org" if we want to use "logs" for irc logs 19:04:21 mtaylor: but currently, the host for irc logs is http://eavesdrop.openstack.org/meetings/ 19:04:21 jeblair: just a thought 19:04:37 mtaylor: nope - didn't see that 19:04:38 now's a good time to get all this straight, before we have to add any redirects. :) 19:04:42 it is - I was just thinking that we don't necessarily have to keep the logs on the same server as the bot 19:04:51 but I may also be making things more confuseling 19:04:58 mtaylor: oh, er, then we'd have to copy them. 19:05:00 mtaylor: or use afs. 19:05:26 jeblair: we should totally be using AFS 19:05:54 mtaylor: i think i've just bootstrapped a meetbot test host, so i can work out the details without having to use #openstack-meeting. :) 19:06:15 neat 19:06:35 so - that being said ... 19:06:50 you wanna tell people about logs.o.o (since I know we have tons of onlookers) 19:06:57 sure 19:07:16 clarkb and I have been working on a project to move all of the build logs and reports onto a static html server 19:07:17 * annegentle looks on 19:07:22 http://logs.openstack.org/ 19:07:41 it turns out the more build history and reporting jenkins maintains... 19:07:58 the slower it gets. it's just not meant to scale to the level we're using it at... 19:08:25 so the idea is to put all the build information on the static server 19:08:34 and tell jenkins it doesn't have to keep as much of it around 19:08:53 this also means that developers won't have to log into launchpad and wait 5 minutes to find out why a build failed 19:09:00 ++ 19:09:06 * mtaylor thinks it will be popular 19:09:11 as we'll link directly from gerrit to logs.o.o when it's all done 19:09:23 so, when you do that, how will people get pretty reports of their test output? 19:09:35 good question monty! 19:09:54 clarkb is rolling out a nose html output plugin... 19:10:14 and we'll copy an html test report over to logs.o.o 19:10:24 similarly for coverage reports, etc. 19:10:29 that sounds great! 19:10:33 we've already got the devstack logs copying over 19:10:40 does it have nifty little javascript controls to expand sections? 19:10:51 clarkb: ? 19:11:08 here's an example set of check jobs for a keystone change: http://logs.openstack.org/11737/1/check/ 19:11:21 (that's the change and patchset numbers in there, so it's easy to find things) 19:11:22 it does 19:11:51 clarkb: excellent 19:12:22 now that we have the html output plugin, what's between us and having zuul spit these links into gerrit reviews? 19:12:38 mtaylor: i want to make sure all the jobs are putting all of their data their first 19:12:45 the output will look like the sample here http://tungwaiyip.info/software/sample_test_report.html 19:12:52 greetings. :) 19:12:55 mtaylor: basically nothing -- but we have to merge a change into all the projects first 19:13:04 i invited maoy to join us 19:13:14 because he wanted to chat about the pylint job he's working on 19:13:19 sweet 19:13:25 #topic pylint 19:13:30 that change is adding the plugin to the test-requires 19:13:54 ok - wait, before we do pylint... 19:13:58 do we really need that in test-requires? 19:14:14 it's not actually a plugin we're expecting devs to run locally, are we? 19:14:24 clarkb: are you talking about pylint or something else? 19:14:29 it isn't, but we need the plugin to be installed into the virtual env 19:14:29 mtaylor: no, but we need it to be installed in the venv 19:14:33 maoy: something else 19:14:35 AH 19:14:38 bother 19:14:40 ok 19:14:46 that's gonna suck 19:14:50 mtaylor: why? 19:14:56 just merging that everywhere... 19:14:57 mtaylor: isn't that what test-requires is for? 19:15:01 mtaylor: oh, yes. :) 19:15:13 master, s/e, s/d of all projects 19:15:15 mtaylor: and to stable/* branches too! :) 19:15:16 * mtaylor cries 19:15:38 we might need to get that in soon before ttx thinks it's a new feature and therefore subject to FF 19:16:29 can we specify the dependency in run-tox somehow? 19:17:16 heh "echo 'nosehtmloutput >> tools/test-requires'" 19:17:21 ^ bad idea, don't do it. 19:17:22 jeblair: Error: "bad" is not a valid command. 19:18:16 that's right uvirtbot ... "bad" is NOT a valid command 19:18:56 jeblair: "grep nosehtmloutput tools/test-requires 2>&1 || echo 'nosehtmloutput >> tools/test-requires'" 19:19:01 I think would be more correct 19:19:13 mtaylor: do you want to go down that road? 19:19:18 jeblair: not REALLY 19:19:34 but I wouldn't mind if tox could do dependency injection via arguments or env vars... 19:19:43 i think we can propose a bunch of one-liners today 19:19:47 mtaylor: yes, i would like that too. 19:20:00 we're digressing ... 19:20:04 pylint! 19:20:09 maoy: hey there 19:20:13 yo 19:20:23 maoy: so your question to me... 19:20:37 maoy: was whether we could have pylint report to gerrit but not count its vote 19:20:37 https://jenkins.openstack.org/job/gate-nova-pylint/32/console 19:20:57 this is what it looks like when there is a bug, pretty sweet 19:21:03 maoy: is that something you want to do in the long run, or do you still want it to eventually be a real gating job? 19:21:45 jeblair: hard to say.. might not be that a bad idea to do it for a few months.. 19:21:54 in openstack i guess that's "long"? :) 19:22:10 http://logs.openstack.org/11444/9/silent/gate-nova-pylint/32/console.html 19:22:19 there's the new static url for that. :) 19:22:27 maoy: (that was our previous meeting topic) 19:22:42 that's SO MUCH QUICKER to load 19:22:49 oh. i c.. 19:22:58 https://jenkins.openstack.org/job/gate-nova-pylint/32/violations/file/nova/utils.py/? 19:23:09 anything static for this type? 19:23:45 maoy: no... remeber in the code review i mentioned i would like either an html or formatted text report? 19:23:54 maoy: this is why 19:24:12 jeblair: yup. this is really nice.. 19:24:38 jeblair: i can stand it being this slow. :) 19:24:47 not very very slow. :) 19:24:57 maoy: so if you output something nice to the console, or to a separate html or txt file, we can copy it over. 19:25:17 pylint has an html output format 19:25:20 maoy: i'm not clear -- are you saying you prefer slow and the pretty jenkins output? 19:25:23 jeblair: so - a generalized pylint and pep8 output format to html converter... 19:25:26 or what clarkb said 19:25:28 clarkb: yes, but maoy's script hides it 19:25:44 * mtaylor shuts up - jeblair and clarkb have things well in hand 19:26:21 clarkb: jeblair is right. I had to parse pylint's output so no html from pylint. 19:26:38 jeblair: I think the current output format is good enough for me. :) 19:27:03 jeblair: but I wonder if it's possible to link the result of this to Jenkin's comment in Gerrit. 19:27:37 maoy: so why not turn it on as a gate job when it's ready? 19:28:17 jeblair: that would take a while for people to get comfortable with it as a gate job I guess. 19:28:55 maoy: do you think people are going to pay attention to it if it's not a gate job? 19:29:22 jeblair: I think so. For example, smokestack is not gate job per se 19:29:39 but it's a warning sign 19:30:36 but I guess it's not some existing function that's already available to use in Jenkins, right? 19:30:38 i just remember people not ever really caring if they increased the pylint count, unless we gated on it, and the one project we were gating on pylint eventually got tired of it. 19:30:57 as a quick hack to get parsable output and html can you simply run pylint twice? 19:31:20 clarkb: he needs to filter the already seen errors from the output 19:31:32 ah 19:31:38 maoy: correct, it's not an existing feature; it would be a (fairly small) change to zuul to do that. 19:31:40 ckarkb: mtaylor is right. but we do have parsable output now 19:32:58 maoy: if we think that's the way we want to go, i can add that to zuul 19:33:37 mtaylor: you were around in the old pylint days, what do you think? 19:34:16 the spectrum is silent pipeline -> warning in jenkins -> gate function. 19:34:17 * LinuxJedi suddenly imagines mtaylor dressed as Obiwan 19:34:36 LinuxJedi: what's oubiwann wearing then? 19:34:38 at the moment it seems I have enough headwind to do it as a gate function 19:35:30 jeblair: a dusty old cloak, and talks about the good old days when he turned boys evil or something 19:35:54 but silent pipeline seems a bit too silent.. 19:36:02 so the change to zuul would be that an individual job could be marked as non-voting, so it could be included in the check and gate pipelines, but its vote isn't counted when determining whether to run the success or failure actions. 19:36:24 jeblair: that's exactly what I was hoping for.. 19:36:47 zuul would report "gate-nova-pylint: FAILURE", but if nothing else failed, the overall verified status would still be +1 or +2. 19:37:07 is that much better than simply running it as a check test and not a gate test? 19:37:13 I would be worried about confusing people 19:37:38 clarkb: that's a good idea too, but that would mean a -1 verified vote on check, and the same change could get a +2 verified vote on gate... 19:37:58 clarkb: interesting. I haven't thought about that.. 19:38:20 clarkb: that's also a change in how the system currently works... anyone who ignores verified=-1 changes would be disrupted by that 19:38:26 clarkb: but otherwise, does make a lot of sense. 19:38:49 jeblair: good pt. might slightly confusing.. 19:39:05 ya I think either way some education has to happen. 19:39:14 probably would have made more sense in a 0-2 rating on verified 19:39:22 so picking the less painful route which appears to be not counting a test against the verified vote is the way to go 19:39:56 we can annotate the test in the message with "non-voting" or something if that's helpful. 19:40:06 ++ 19:40:16 or is it possible for jenkins to post another comment separately just to show pylint result as FYI? 19:40:38 the way gerrit works that would initially hide one or the other comment 19:40:45 maoy: and it generates email, so we try to keep the number of comments down 19:40:56 that too :) 19:41:01 got it.. 19:41:38 okay, so i think we're all leaning toward the non-voting test solution.... 19:41:47 +1 on that 19:41:56 maoy: i'll work that up and point you at the change when it's ready 19:42:27 jeblair: thanks much. i'll talk to other nova folks to see their opinion. 19:42:43 jeblair: one thing is that pylint job does take a while to run.. not sure if it matters 19:43:22 tempest currently takes about 15 minutes I think 19:43:24 maoy: it takes about as long as the unit tests, and we run all the jobs in parallel, so there's no net change 19:43:35 and yeah, tempest takes even longer. :) 19:43:45 cool.. 19:43:57 maoy: we even test multiple changes in parallel (that's what zuul is all about) 19:44:23 btw, there is some change in lintstack 19:44:43 previously there is a pylint_exception committed in the source tree to record what errors have been seen before 19:44:46 now we just need to work on a way of testing changes before people write them 19:45:12 in the new code, that file is auto generated from HEAD~1 every time so there is no need to maintain that 19:45:12 LinuxJedi: easy "tox && /bin/false". they all fail. 19:45:18 lol :) 19:45:34 maoy: that may be a problem. 19:46:00 https://review.openstack.org/#/c/11444/9/tools/lintstack.sh 19:46:19 that's how I deal with it now.. 19:47:21 maoy: that makes some assumptions about the state of the git tree that may not be correct. 19:47:53 * jeblair is thinking 19:49:10 jeblair: actually - interestingly enough - if you have a sequence of merges 19:49:28 jeblair: shouldn't the first merge in the sequence throw the pylint error, but then others in the sequence not? 19:50:07 are you guys thinking about merges with multiple parents? 19:50:09 so the thing i'm considering is... 19:50:27 maoy: no, we do a lot of things in support of parallel testing 19:50:54 well, i am also thinking of merge commits 19:50:59 what if HEAD is a merge commit? 19:51:03 me. that too 19:51:05 mmm. that too 19:51:15 (it usually is, currently) 19:51:16 oh. you mean the files could be in use by other tests, like the unit tests? 19:51:39 maoy: let's talk about that in a second -- first let's focus on "HEAD is a merge commit" 19:51:57 maoy: you are comparing HEAD to HEAD~1... 19:52:29 maoy: wouldn't that compare the commit "Merge 'my change'" with "my change" 19:52:42 * mtaylor believes jeblair is correct 19:53:15 not really.. see the git logs here: https://jenkins.openstack.org/job/gate-nova-pylint/32/console 19:53:37 somehow it's fast-forwarded so HEAD is not a merge commit 19:53:49 maoy: i mean, it might compare "Merge 'my change'" with "the previous head", but that depends on the order of the parent commits, which i'm not sure is guaranteed. 19:54:20 maoy: we fast-forward if possible, but that's only possible if you have rebased immediately before testing 19:54:33 maoy: otherwise, we merge, which is usually what happens when changes actually land 19:54:42 maoy: see the commit history here: https://github.com/openstack/nova/commits/master 19:54:53 jeblair: it seems that git review is doing that rebase automatically now 19:55:01 maoy: yes. and that's actually about to change 19:55:10 maoy: future versions won't do that anymore. 19:55:34 i think we're about to run out of time.. 19:55:39 jeblair: ok.. so what should I compare to against HEAD if not HEAD~1? 19:55:44 maybe we should continue this conversation in #openstack-infra 19:56:04 I think next step here is that we need to test the HEAD v. HEAD~1 logic in some different scenarios 19:56:20 and also continue the discussion in #openstack-infra 19:56:24 agree 19:56:52 k. I'm calling it 19:56:52 I have to leave for half an hour around 4pm EDT (i.e. in 5 minutes).. 19:57:01 #endmeeting