15:06:45 <anteaya> #startmeeting third-party
15:06:46 <openstack> Meeting started Mon Sep  5 15:06:45 2016 UTC and is due to finish in 60 minutes.  The chair is anteaya. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:06:47 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:06:49 <openstack> The meeting name has been set to 'third_party'
15:06:56 <anteaya> sorry lennyb
15:07:01 <anteaya> and thanks for the ping
15:07:04 <anteaya> was reviewing a patch
15:07:07 <eantyshev> o/
15:07:23 <lennyb> hi
15:07:38 <anteaya> hello
15:07:42 <anteaya> how are you today?
15:07:51 <anteaya> sorry I'm late starting
15:07:59 <anteaya> do we have anything to talk about today?
15:08:05 <lennyb> fine, thanks.how are you? Yes.
15:08:08 <lennyb> In case of broken CI, what is the best/easiest way to run commits back or debug the issue? I mean I dont know what commit broke me, it can be neutron/cinder/devstack
15:08:08 <lennyb> for example #link http://13.69.151.247/Cinder-ISER-ISCSI/2574_cloudx-26//testr_results.html.gz
15:08:43 <anteaya> good questions
15:08:59 <anteaya> lennyb: was this the first test that failed
15:09:05 <anteaya> often it can be a timing issue
15:09:08 <lennyb> no
15:09:14 <anteaya> find the first failure
15:09:21 <eantyshev> I also wanted to make an announcement
15:09:32 <anteaya> and then check the timestamps off all the patches that merged before that failure
15:09:35 <lennyb> it started to fail a while ago, but I cant tell exactly when
15:09:39 <anteaya> but since the last success
15:09:47 <anteaya> eantyshev: ack, you are next, thank you
15:09:52 <anteaya> lennyb: oh okay
15:10:12 <anteaya> well you asked about the best way and in my opinion, the best way is time stamp comparison
15:10:36 <lennyb> I know I can export a branch in devstack (stable/mitaka) and it will apply to all
15:10:49 <anteaya> if you can't work backwards to timestamps then I think you are one at a time removing patches to find the last best state
15:10:55 <anteaya> or that
15:12:05 <lennyb> can I provide a local.conf with a special commit for neutron for example?
15:12:11 <anteaya> ah sure
15:12:25 <anteaya> I'm not sure how much help I personally can be if you do
15:12:32 <anteaya> but sure you can do so
15:12:46 <anteaya> perhaps eantyshev might be get something useful from that
15:13:30 <lennyb> I know it works with refs/.../patchset number, but I need to map gerrit patchset with git commit
15:13:47 <anteaya> eantyshev: we'll finish with lennyb and then you can share your announcement, if that is okay with you?
15:13:58 <anteaya> lennyb: hmmmm
15:14:09 <eantyshev> anteaya: I don't see the problem at first glance
15:14:20 <anteaya> eantyshev: thank you
15:14:22 <eantyshev> anteaya: perfect
15:14:26 <anteaya> thanks
15:14:51 <anteaya> lennyb: we might be into the territory where we would need fungi, clarkb or jeblair's help
15:15:10 <lennyb> ok, I will try to ping them later
15:15:23 <anteaya> but it is Labor Day today in the states so they might not be online today
15:15:34 <anteaya> lennyb: you can keep explaining and eantyshev and I can listen
15:15:43 <anteaya> if you have more explaining to share
15:15:56 <anteaya> I'm just not sure how much help I can be
15:16:45 <lennyb> Since each stack.sh run pulls the latest code I am interested to find a way to be persistent with the devstack
15:17:08 <anteaya> well that depends on what you want to test
15:17:20 <anteaya> devstack for testing is designed to test the most recent code
15:17:37 <eantyshev> lennyb: we solve this by working with local clone of devstack
15:17:51 <anteaya> if you don't want devstack to clone new code but want it to run with existing code you can set the config to offline I do believe
15:18:06 <anteaya> the exact config setting is in the devstack docs
15:18:15 <mmedvede> lennyb: if you want to go the path of bisecting which patch broke you, you can use commit sha as well. I once had to do it. It was tricky due to having to decide which repository out of about 20 to bisect
15:18:24 <anteaya> hey mmedvede, glad you are here
15:19:20 <mmedvede> lennyb: if you think it is devstack that broke you, makes it easier. Definitely many ways to pin devstack. One of them is to pin it in pre_test_hook (if you use devstack-gate)
15:19:58 <mmedvede> hi anteaya
15:19:59 <lennyb> the problem that I dont know were to start bisecting ( cinder/neutron/nova/devstack ) :).  I will start probably with devstack
15:20:36 <anteaya> lennyb: yeah if you don't have timestamps to help you, I think you just pick one at a time, unless you have some indication in the test where to begin
15:20:50 <lennyb> thanks.
15:20:56 <mmedvede> lennyb: as anteaya mentioned, if you can narrow down time when you started breaking, it would help. You can then check which repositories had patches merge during that time window
15:21:12 <anteaya> thank you for asking about it, I hope you make progress
15:21:32 <lennyb> thanks. I am done with the issue.
15:22:00 <anteaya> thanks lennyb
15:22:13 <anteaya> anymore before we move to eantyshev's announcement?
15:22:27 <anteaya> eantyshev: you're up
15:22:52 <eantyshev> last time I asked here which version of Jenkins I need to setup CI. Turned out, I need some old version like 1.651.1. It needs some amendment to puppet-openstackci, which depend on puppet-jenkins (https://review.openstack.org/#/c/354086/)
15:23:23 <eantyshev> but the above change doesn't work, due to https://issues.jenkins-ci.org/browse/INFRA-92
15:23:54 <anteaya> hmmmm, that's unfortunate
15:24:01 <eantyshev> as jenkins doesn't provide old versions in apt repositories. They only available as .deb files
15:24:31 <anteaya> this is an old issue
15:24:45 <eantyshev> So, i'm deciding: should I strive to get rid of Jenkins in my CI, too
15:24:51 <anteaya> the bug isn't getting any attention
15:25:03 <anteaya> can you install from .deb files at all?
15:25:10 <anteaya> can you try that route first?
15:25:38 <anteaya> I'm guessing asselin is observing labor day today
15:26:03 <eantyshev> I tried, but I want this to be automated in puppet, so I don't have to manually update everything
15:27:26 <anteaya> eantyshev: hmmm, I think we talked about taking this to the infra list last time but then it got resolved and we didn't have a ml discussion about it
15:27:55 <anteaya> eantyshev: given this upstream issue that isn't getting any attention, do you feel you would post to the ml about this?
15:28:07 <anteaya> this will affect other puppet jenkins users
15:28:26 <anteaya> I think we need to increase the general awareness of what you have uncovered here
15:28:31 <anteaya> eantyshev: what do you think?
15:28:47 <eantyshev> well, this is really a chance
15:29:01 <anteaya> chance or change?
15:29:11 <eantyshev> I'll try it, to post this to ml
15:29:17 <mmedvede> there was also the security workaround patch to make newer versions of jenkins work, at cost of disabling the security feature that breaks it. https://review.openstack.org/#/c/334400/
15:29:19 <anteaya> thank you I appreciate it
15:29:30 <mmedvede> it would not merge as is
15:30:27 <anteaya> yeah, I think a post kicking off a general chat about the puppet-jenkins module would be good
15:30:41 <anteaya> mention the upstream version issue and this security issue
15:30:54 <anteaya> we want the puppet-jenkins module to be usable
15:31:07 <eantyshev> mmedvede: well, that's not the same problem, I know this one
15:31:15 <anteaya> and if it has gotten into the state where it isn't we need to discuss this
15:31:35 <anteaya> eantyshev: right but mmedvede's point I believe is that both issues affect the same module
15:31:58 <anteaya> so having the discussion about the module as a whole might be the most effective way to introduce the topic
15:32:08 <anteaya> rather than considering the issues to be separate
15:33:37 <mmedvede> eantyshev: my point was that the reason you need older jenkins is that it does not have that security feature in it
15:34:19 <mmedvede> so these two related. 3 ways to get jenkins to work: explicitly allow certain parameters to be passed
15:34:20 <anteaya> mmedvede: ah sorry
15:34:27 <anteaya> mmedvede: thanks for clarifying
15:34:28 <mmedvede> 2) use older version of jenkins
15:34:40 <mmedvede> 3) disable security in new version of jenkins
15:37:39 <anteaya> eantyshev: what do you think?
15:39:13 <anteaya> are we still discussing this item?
15:39:23 <anteaya> are we going to see a mailing list post?
15:39:28 <anteaya> what would we like to do?
15:40:41 <anteaya> I can only assume that eantyshev has gone offline for some reason
15:40:59 <mmedvede> looks like it
15:41:05 <anteaya> does anyone have any other items they would like to discuss today?
15:41:19 <anteaya> mmedvede: yeah, so we can come back to this or follow up in -infra
15:41:37 <anteaya> anyone with any other items today?
15:42:32 <anteaya> okay so when eantyshev comes back we can discuss in the -infra channel or pick up at next weeks' meeting whichever he likes
15:42:44 <anteaya> does anyone have any objections to me closing today's meeting?
15:43:04 <anteaya> note I'm offline the rest of today to observe Canada's Labour Day
15:43:11 <lennyb> enjoy
15:43:11 <anteaya> thank you all for attending
15:43:15 <anteaya> lennyb: thank you
15:43:22 <anteaya> enjoy the rest of your day/evening
15:43:25 <mmedvede> thanks anteaya
15:43:28 <anteaya> see you all next week
15:43:33 <anteaya> #endmeeting