15:02:30 <anteaya> #startmeeting third-party
15:02:31 <openstack> Meeting started Mon Apr  6 15:02:30 2015 UTC and is due to finish in 60 minutes.  The chair is anteaya. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:33 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:36 <openstack> The meeting name has been set to 'third_party'
15:02:39 <anteaya> hello
15:02:51 <ctlaugh> hi
15:02:52 <akerr> o/
15:02:56 <asselin> hi
15:02:59 <anteaya> raise your hand if you are here for the third party meeting
15:03:00 <krtaylor> o/
15:03:02 <anteaya> hi
15:03:04 <eantyshev> hi
15:03:10 <anteaya> so eantyshev you had a question
15:03:16 <beecee> hi
15:03:43 <anteaya> eantyshev: would you like to share your question?
15:05:10 <anteaya> or maybe later
15:05:24 <anteaya> does anyone else have somethign they would like to discuss today?
15:05:30 <eantyshev> anteaya: SCP publisher plugin seems like not storing logs when one of build steps fails
15:05:39 <anteaya> okay here we go
15:06:01 <anteaya> eantyshev: alright well we are the most helpful when we have a stacktrace to look at
15:06:28 <anteaya> so if you could find a stacktrace and paste it somewhere, paste.openstack.org is an option
15:06:36 <anteaya> and bring us the url we can all look
15:06:45 <eantyshev> thought this is its normal behavior...
15:06:50 <anteaya> and then someone can offer a suggestion of what to do next if they have one
15:07:19 <eantyshev> no trace, it just doesn't do anything
15:07:24 <anteaya> well if a build fails we need to know why
15:07:31 <anteaya> so we need logs in that situation
15:07:33 <asselin> eantyshev, which "build step"
15:07:59 <eantyshev> devstack-gate
15:08:29 <asselin> can you paste the code for that step?
15:08:50 <asselin> http://paste.openstack.org/
15:10:13 <krtaylor> eantyshev, I'd say look at others setup for SCP, if the testing moves on before copy is complete it may not capture all the artifacts
15:10:34 <krtaylor> eantyshev, I can't help much here, we are using swift
15:10:38 <eantyshev> asselin: http://paste.openstack.org/show/198549/
15:11:25 <anteaya> eantyshev: how long is the build running before it fails?
15:11:29 <akerr> i've seen instances where the test will take longer than the job timeout and that will result in no logs being uploaded.  What is the type of build error?
15:12:06 <anteaya> akerr: yes, that is why having a build-timeout 5 minutes longer than the job timeout is important
15:12:26 <eantyshev> krtaylor: afraid not in my case, the only way I could get them exported was to abort the build
15:12:40 <anteaya> the build-timeout is 6 hours longer, which is a tad excessive but at least it is longer
15:13:39 <eantyshev> akerr: http://paste.openstack.org/show/198550/
15:15:42 <eantyshev> I noticed jenkins.openstack.org is of 1.565.3 , and puppet deployed me 1.580.2
15:16:24 <anteaya> eantyshev: how long does the job run before it fails?
15:16:39 <eantyshev> anteaya: 30min
15:16:46 <anteaya> okay
15:17:21 <anteaya> can you run devstack successfully by hand on your environment?
15:18:11 <eantyshev> anteaya: yes, it fails in tempest
15:18:16 <anteaya> and just so I get a sense, who else has items they would like to discuss today?
15:18:43 <anteaya> eantyshev: so when you run devstack by hand in your environment, you tempest tests fail, did I understand you correctly?
15:19:00 <eantyshev> anteaya: yes
15:20:07 <anteaya> okay well it dosn't have anything to do with your jenkins version then
15:20:53 <anteaya> until you can get devstack and your tempest tests passing in your environment when running them manually, devstack-gate is just going to continue to fail for you
15:21:12 <akerr> eantyshev: Do you have the "Copy after failure" option checked for the scp publisher plugin on the job?
15:21:24 <anteaya> so focus on running devstack by hand in your set up and learn the logs for the tempest tests
15:21:32 <anteaya> akerr: good thought
15:23:09 <anteaya> does anyone else have anything they would like to discuss today?
15:23:10 <eantyshev> akerr: I don't have such an option in job -> configure
15:23:39 <anteaya> eantyshev: do you have a gui for your plugin?
15:23:40 <beecee> anteaya: persistent disconnects using the jenkins ssh plugin in ec2 if anyone has a good fix for that
15:23:49 <anteaya> beecee: you are up next
15:24:18 <akerr> eantyshev: do you use jenkins job builder or do you manually define the tests?
15:25:22 <eantyshev> akerr: I'm using Hudson SCP publisher plugin https://wiki.jenkins-ci.org/display/JENKINS/SCP+plugin
15:25:48 <eantyshev> akerr: jobs are made by JJB
15:26:35 <anteaya> eantyshev: I'm going to have to cut this off now, to make some room for beecee
15:26:56 <anteaya> eantyshev: the best idea is to continue working on your issue and chatting on irc with others
15:27:17 <eantyshev> anteaya: Okay
15:27:24 <anteaya> is there someone in the meeting who can take some time this week and help eantyshev with some basic navigation of his/her setup?
15:27:39 <anteaya> :(
15:27:44 <anteaya> that is a disappointment
15:27:54 <anteaya> I didn't mean for them to leave
15:28:03 <anteaya> well moving on then
15:28:08 <anteaya> beecee: you were saying?
15:28:51 <beecee> anteaya: ok, jenkins 1.605, EC2 plugin, provisions a new instances each time gerrit says so.
15:29:18 <beecee> just sometimes the slave doesn't respond to jenkins at random points during the devstack-gate build
15:29:28 <anteaya> that isn't helpful
15:29:38 <anteaya> any artifacts you can share?
15:29:40 <beecee> even tried on a xlarge instance in case it was mem/cpu related
15:30:05 <beecee> not really sorry, was more hoping for a 'flip this bit you dumbkoff' response ;)
15:30:58 <anteaya> ah
15:31:10 <anteaya> well I don't use the phrase dumbkoff personally
15:31:21 <anteaya> so there is a disappointment for you right off the bat
15:31:29 <anteaya> anyone using the EC2 plugin?
15:31:46 <ameade> nope :(
15:31:57 <akerr> sorry to interrupt here, but in case eantyshev decides to read the logs of this meeting here is a link to the scp publisher definition for JJB which shows how to enable copy-after-failure: http://paste.openstack.org/show/198559/
15:32:20 <anteaya> akerr: thanks for that
15:32:34 <anteaya> #link http://paste.openstack.org/show/198559/
15:32:59 <anteaya> so if anyone sees eantyshev again please share that with him/her, and if you don't know how please ask
15:33:08 <asselin> on that note, here's the link to openstack's custom scp plugin: #link http://tarballs.openstack.org/ci/
15:33:09 <anteaya> beecee: we might not have much for you
15:33:14 <beecee> anyeaya: I'll go back to the google-fu saltmines on this one
15:33:27 <anteaya> beecee: do come back and share if you find a fix
15:33:39 <anteaya> so we can have something in the logs to share with others
15:33:46 <anteaya> asselin: thank you
15:34:17 <anteaya> beecee: sorry we don't have anything else for you today
15:34:27 <anteaya> does anyone have any other item they would like to discuss?
15:35:07 <asselin> I figured out my soft lockup issue last week
15:35:13 <anteaya> asselin: oh do tell
15:35:20 <asselin> #link https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1413540
15:35:21 <openstack> Launchpad bug 1413540 in linux (Ubuntu Trusty) "Trusty soft lockup issues with nested KVM" [High,In progress] - Assigned to Chris J Arges (arges)
15:35:30 <anteaya> well done
15:36:16 <asselin> just posted my workaround: use qemu on top of kvm. Default is kvm on top of kvm.
15:36:26 <anteaya> ah
15:36:47 <rhe00> asselin: did you get any further on your issue with test_minimum_basic_scenario?
15:37:01 <asselin> I played around with power setting on the baremetal bios. It seemed to help a bit, but not 100%.
15:37:33 <anteaya> asselin: glad it at least didn't take you backward
15:37:39 <asselin> rhe00, no, that was late last week. Will do that today. I was focusing on the soft lockup issue
15:38:06 <rhe00> asselin: ok, I didn't make any progress either over the weekend
15:38:31 <asselin> mtreinish said he'd help out today
15:38:37 <anteaya> who else was hitting that one?
15:38:47 <anteaya> I had thought there was one other person
15:38:59 <rhe00> anteaya: yes, trying to remember his nick
15:39:03 <anteaya> my memory is telling me 3 for some reason
15:39:12 <marcusvrn1> o/
15:39:21 <anteaya> it was in the infra channel right after this meeting last week
15:39:23 <rhe00> kaiser something?
15:39:25 <anteaya> marcusvrn1: was it you
15:39:31 <marcusvrn1> yes!
15:39:45 <anteaya> yay you are here
15:39:54 <anteaya> marcusvrn1: anything to share that might be helpful?
15:40:13 <anteaya> rhe00: kaisers1 was part of the discussion last week too, yes
15:40:18 <anteaya> good memory
15:40:30 <marcusvrn1> no...I didn't make any progress
15:40:37 <anteaya> okay that's fine
15:40:47 <anteaya> I hope you all are in #openstack-qa
15:40:49 <marcusvrn1> I'm continue on that
15:40:55 <asselin> Also, for cinder folks: tempest has a false positive! https://bugs.launchpad.net/tempest/+bug/1440227
15:40:57 <openstack> Launchpad bug 1440227 in tempest "encrypted volume tests don't check if a volume is actually encrypted" [Undecided,New]
15:41:09 <anteaya> so that when asselin and mtreinish talk about it, you can contribute your thoughts too
15:41:20 <asselin> so if you're passing that test, you might actually be failing....
15:41:26 <anteaya> asselin: wonderful, thanks for finding that
15:41:33 <marcusvrn1> I'll join the #openstack-qa
15:41:56 <anteaya> marcusvrn1: good idea
15:41:58 <rhe00> asselin: darn it! :)
15:42:33 <anteaya> are any of the operators willing to assign themselves that bug?
15:42:37 <rhe00> asselin: I was happy that mine was passing. thanks for the heads up. :)
15:43:00 <anteaya> when you agree to take on a bug, you don't have to write the code to fix it, you just have to find someone to help you with that
15:43:14 <anteaya> and update the bug report and make sure progress is being made
15:43:22 <rhe00> so it is a bug in the tempest test that doesn't check the encrypted flag?
15:43:23 <anteaya> you can get lots of help if you take a bug
15:43:43 <asselin> rhe00, no doubt. We had one driver passing, and one failing. After investigating, couldn't figure out why the one passing was working. It was a driver bug.
15:44:33 <marcusvrn1> asselin: my driver is passing too
15:44:34 <asselin> rhe00, right, if the driver incorrectly doesn't set the encrypted flag it will pass the encrypted volume test
15:45:04 <rhe00> ok, but if we fix the bug in tempest, alot of CI's will start failing. Is it up to the bug fixer to fix all the drivers as well or? just asking what the common prcatice is?
15:46:04 <asselin> rhe00, yes, if they aren't setting the encrypted flag.
15:46:25 <rhe00> asselin: ok
15:46:26 <asselin> not sure what the process is if there's a lot of driver's with that issue. Better to stay ahead
15:47:06 <anteaya> to fix the bug in tempest you fix the tempest bug
15:47:23 <anteaya> it is the driver maintainer's responsiblity to fix their driver
15:47:29 <anteaya> which is why it is good you are here
15:47:41 <asselin> anteaya, +1
15:47:44 <rhe00> anteaya: yes. :)
15:47:45 <anteaya> the point of the test is to test that something is working
15:47:59 <marcusvrn1> anteaya: +1
15:48:06 <anteaya> if it isn't working, we need to know (failing tests) and for folks to fix what is there's to fix
15:48:18 <anteaya> so if you take on the bug, all you have to do is fix the tempest test
15:48:32 <anteaya> then it will properly fail when it actually is failing now anyway
15:48:40 <anteaya> and folks will know about it and can fix their code
15:48:43 <anteaya> make sense?
15:48:50 <beecee> fail for the right reason instead of the wrong reason
15:48:55 <anteaya> exactly
15:49:07 <anteaya> so first we have to fail for the right reason
15:49:16 <rhe00> ok, is it still time to do it for Kilo?
15:49:26 <asselin> anteaya, or pass for the right reasons :)
15:49:30 <anteaya> right
15:49:46 <anteaya> rhe00: the right question is how do we ensure code quality for kilo
15:49:59 <anteaya> so someone take this to cinder core in cinder channel
15:50:07 <anteaya> make sure it gets on the cinder meeting agenda
15:50:27 <anteaya> then cinder will know and prioritize if as they need to for rc's
15:50:29 <anteaya> make sense?
15:50:34 <asselin> I can add it to cinder agenda
15:50:39 <anteaya> thank you
15:51:02 <anteaya> and please have a chat in cinder channel so that folks know and can figure out the priority
15:51:22 <anteaya> thanks for bringing this up
15:51:30 <asselin> anteaya, +1 probably better to do that + add a topic on mailing list
15:51:52 <anteaya> asselin: I agree thought what is posted to the ml is thingee's call I would say
15:52:04 <anteaya> as clarity is important, not confusion
15:52:09 <asselin> anteaya, ok
15:52:12 <anteaya> thanks
15:52:26 <anteaya> when you dno't know whether to say something or not?
15:52:32 <anteaya> say something in open source
15:52:49 <anteaya> we would rather know something is broken and address it with confidence
15:53:09 <anteaya> than ship something someone knew was broken but didn't tell anyone else
15:53:20 <anteaya> at the very least release notes are good
15:53:26 <anteaya> make sense?
15:53:43 <asselin> +1
15:53:48 <anteaya> great
15:53:53 <anteaya> so 6 minutes
15:54:01 <anteaya> anyone have anything else?
15:54:42 <anteaya> we can stare at our screens for another 5 minutes or move on
15:54:56 <anteaya> I'm for moving on
15:55:00 <anteaya> any objections?
15:55:15 <anteaya> okay thanks everyone
15:55:22 <anteaya> I appreciate your participation
15:55:26 <anteaya> see you next week
15:55:36 <anteaya> #endmeeting