18:02:18 <jaypipes> #startmeeting third-party-ci 18:02:20 <openstack> Meeting started Mon Mar 17 18:02:18 2014 UTC and is due to finish in 60 minutes. The chair is jaypipes. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:02:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:02:24 <openstack> The meeting name has been set to 'third_party_ci' 18:02:32 <jaypipes> good afternoon folks. 18:02:32 <jgriffith> hey ya 18:02:39 <luqas> hi 18:02:43 <jaypipes> #link https://etherpad.openstack.org/p/third-party-ci-workshop 18:02:48 <krtaylor> hello all 18:03:02 <jaypipes> I spent some time cleaning up the above etherpad... let's continue to use it. 18:04:03 <jaypipes> For question #2, mtreinish, jeblair and clarkb have gotten a DEVSTACK_TEMPEST_REGEX env variable into devstack-gate now. 18:04:23 <jaypipes> so I should update that section accordingly. IIRC, lukego has experimented successfully with it. 18:05:06 <jaypipes> If you're on the etherpad, please do make sure you set your name in the right-hand pane please (click on the little person icon thingee and enter you name) 18:08:51 <jaypipes> OK, sorry about that. Done cleaning up question 2. 18:09:04 <jaypipes> Does anyone have questions that we can tackle today on third party CI? 18:09:16 <jaypipes> anyone have any updates about how their CI installation/setup is going? 18:09:38 <jaypipes> jgriffith: how is Solidfire's going? run into roadbumps we can assist with? 18:10:17 <jgriffith> jaypipes: yes please 18:10:22 <jgriffith> jaypipes: I'm reproducing the issue now 18:10:31 <jgriffith> jaypipes: ran into problems just setting up :( 18:10:32 <jaypipes> jgriffith: ok, sounds good. 18:10:44 <jaypipes> jgriffith: perfect, that's what this meeting is all about :) 18:10:53 <jgriffith> Coolness 18:11:27 <jgriffith> jaypipes: so I spin up a simple 12.04 instance, copy my keys over 18:11:39 <jgriffith> clone my repo and wget the script 18:11:51 <jgriffith> The problem I've been hitting is that I get a couple of errors 18:12:02 * jgriffith can't find the pastebin so running again 18:12:07 <jaypipes> no probs 18:13:21 <rfolco> jaypipes, very quick question to confirm... "recheck <3rd-party>" is triggering Jenkins too. recheck-<3rd-party> with dash will do the trick ? 18:14:25 <jaypipes> rfolco: hmm, I don't think so, but lemme check the regex... 18:14:55 <jaypipes> rfolco: https://github.com/jaypipes/os-ext-testing-data/blob/master/etc/zuul/layout.yaml#L15 18:15:10 <jaypipes> rfolco: I do not believe a dash will work, no. 18:15:33 <rfolco> jaypipes, we did this: comment_filter: (?i)^\s*recheck myvendor\s*$ 18:15:41 <jaypipes> rfolco: if you really wanted (not recommended because it diverges from upstream), you could change that regex if you wish 18:16:00 <jaypipes> rfolco: that's a different specific regex, here: https://github.com/jaypipes/os-ext-testing-data/blob/master/etc/zuul/layout.yaml#L20 18:16:11 <jaypipes> rfolco: you are welcome to change that to whatever you want. 18:16:42 <jaypipes> rfolco: the first regex, though, is what the upstream system typically uses for recheck bug notifiers. 18:17:02 <jaypipes> rfolco: the second regex is something you can change to have a custom "only recheck on my system" event. 18:17:14 <jaypipes> rfolco: make sense? 18:17:28 <rfolco> jaypipes, yes. We actually added the second one. 18:17:30 <jaypipes> kk 18:17:43 <rfolco> jaypipes, not sure why its triggering Jenkins too. 18:18:27 <jaypipes> rfolco: are you asking why recheck-$your_vendor is triggering a jenkins job run? 18:19:03 <rfolco> jaypipes, I was asking if a dash is necessary to run our CI ONLY. 18:19:09 <rfolco> example: https://review.openstack.org/#/c/79239/ 18:19:13 <jaypipes> no, it should not be at all. 18:20:06 <jaypipes> rfolco: no, no dash should be necessary if your regex is (?i)^\s*recheck pkvm\s*$ 18:20:14 <rfolco> jaypipes, it has triggered Jenkins too.... I'll double check what happened. Thank you for confirming. 18:20:18 <jgriffith> jaypipes: FWIW I managed to get passed the issue I was seeing before, see if I make it through everything else :) 18:20:33 <jaypipes> jgriffith: do you remember what the old issue was? 18:20:44 <jgriffith> jaypipes: I'll find it 18:20:49 <jaypipes> k 18:22:32 <jgriffith> jaypipes: http://paste.openstack.org/show/73667/ 18:23:09 <jgriffith> jaypipes: uncertain why the httpd stuff was having issues 18:23:33 <jaypipes> jgriffith: yeah, that last line meant Jenkins was still starting by the time jenkins-jobs was being executed. 18:23:41 <jgriffith> Oh interesting 18:23:47 <jaypipes> jgriffith: nothing to worry about. might just have been a slow startup initially. 18:24:10 <jaypipes> either a really fast puppet or a really slow Jenkins :) 18:24:29 <jgriffith> jaypipes: so timing is everything :) 18:24:55 <jaypipes> jgriffith: well, sometimes services say they are up (jenkins has a nasty habit of this), but they aren't fully up. 18:25:12 <jgriffith> ahh... that's rather annoying 18:25:36 <jaypipes> jgriffith: for example, you can shut down jenkins (service jenkins stop), then wait a second, then do service jenkins start, and then immediately run jenkins-jobs, and it will fail. 18:25:51 <jgriffith> jaypipes: oh, nice 18:25:55 <jaypipes> jgriffith: because jenkins service reported success to init.d, but it's not fully uip. 18:26:09 <jgriffith> jaypipes: I'll def want to keep that in mind I think 18:26:26 <jaypipes> yeah... especially if you do restarts of jenkins during debugging of the CI env. 18:26:29 <jaypipes> something to keep in mind. 18:28:13 <jgriffith> jaypipes: oh crap 18:28:23 <jgriffith> jaypipes: that was slave (part 2) 18:28:26 <jaypipes> always like to hear that ... 18:28:34 <jgriffith> jaypipes: the issues I was seeing were on master (part 1) 18:28:43 <jaypipes> jgriffith: slave shouldn't install jenkins-jobs, jenkins, or zuul... :( 18:28:56 <jaypipes> jgriffith: did you by accident run install_master.sh on the slave VM? :) 18:29:16 <jgriffith> jaypipes: so initially I was just doing the master 18:29:30 <jgriffith> jaypipes: I spun up a fresh instance to try again but accidentally ran slave setup 18:29:57 <jgriffith> jaypipes: I'll need to build another instance for the master which is where I saw that original error 18:30:02 <jgriffith> jaypipes: make sense? 18:30:15 <jaypipes> jgriffith: I see :) yeah, you'll need to redo the master I'm afraid. 18:30:37 <jaypipes> jgriffith: and if you changed your jenkins SSH keys, you will need to redo the slave as well. 18:30:58 <jgriffith> jaypipes: k 18:32:16 <jaypipes> luqas: how is your CI going? any issues we can assist with? 18:33:54 <luqas> jaypipes: hi 18:34:15 <luqas> well continous work on our CI system 18:34:23 <luqas> always something to do! 18:34:29 <jaypipes> :) indeed. 18:35:22 <luqas> I have a couple of things done before I get a concrete question, so I'll wait for next week I guess 18:35:55 <jaypipes> ok, no worries, luqas. you can also email the openstack-infra ML with questions and I will gather for next week's meeting, too. 18:36:20 <luqas> jaypipes: perfect 18:37:53 <krtaylor> Just fyi, as I have discussed a few times in -infra, I am organizing a 3rd party implementer self-help group to try and offload the infra core folks a little 18:38:26 <jaypipes> krtaylor: cool, do you want to that group to take over this meeting? 18:38:36 <krtaylor> I created an etherpad to get some ideas down #link https://etherpad.openstack.org/p/3rdPartyTesting 18:39:02 <krtaylor> jaypipes, well, I think initially this meeting could be a great input to that 18:39:09 <jaypipes> krtaylor: could you link to the etherpad with the Q&A? 18:39:18 <krtaylor> jaypipes, will do 18:39:43 <jaypipes> cheerio. yes, krtaylor would be great to align efforts and have a single place for 3rd party implementers to meet and help each other. 18:40:07 <krtaylor> one of the first things I'd like to do is standardize the ML tag, mainly because my mail filter is horrible 18:40:19 <jaypipes> hehe, no crap. agreed ;) 18:40:45 <jaypipes> krtaylor: how about 3PCI? 18:41:24 <krtaylor> hm, I'll add that, may be too short for beginners though 18:41:35 <jaypipes> possibly, yeah. 18:42:26 <jaypipes> krtaylor: so, my long term goal for all of this was to push pieces of my blog, the os-ext-testing and os-ext-testing-data repos, and all of the Q&A material into ci.openstack.org/third-party-testing 18:42:40 <jaypipes> krtaylor: do you concur with that long-term goal? 18:43:21 <jaypipes> krtaylor: I've actually been delaying doing any patches to the ci documentation upstream while the os-ext-testing stuff bakes a bit (and even will propose os-ext-testing into a stackforge repo when it's fully baked) 18:43:30 <krtaylor> yes, I see a lot of the same questions over and over internally, the Q&A distilled to a FAQ and links would be a huge help 18:43:37 <jaypipes> great 18:44:15 <jaypipes> well, how about we make it a goal to, by atlanta, have made good progress on that? 18:44:23 <krtaylor> agreed, this may wind up being a set of doc patches when all done 18:44:24 <krtaylor> yes 18:44:34 <jaypipes> getting all the docs up into ci.openstack.org, and having os-ext-testing turned into a stackforge repo? 18:45:53 <jgriffith> jaypipes: so not sure how to debug this... the master script finished 18:46:07 <jgriffith> jaypipes: but can't connect to the jenkins web interface (times out) 18:46:18 <jaypipes> jgriffith: k, can you see jenkins GUI at $HOST_IP:8080? 18:46:23 <krtaylor> jaypipes, I think that's a good goal 18:46:43 <jgriffith> jaypipes: no, that's what I mean when I say connection times out 18:46:51 <jgriffith> jaypipes: ps -ef shows jenkins running 18:47:00 <jgriffith> but it seems the web portion is still jacked up 18:47:31 <jaypipes> jgriffith: tail -n50 /var/log/jenkins/*.log? 18:47:37 <jgriffith> jaypipes: just did 18:47:41 <jaypipes> jgriffith: to a pastebin... 18:47:43 <jgriffith> says "fully up and running" 18:48:10 <jaypipes> hmm, odd... is there an iptables rule on your main host that is preventing traffic to the master's IP? 18:48:13 <jgriffith> jaypipes: http://paste.openstack.org/show/73685/ 18:48:24 <jgriffith> jaypipes: ahh 18:48:27 <jgriffith> jaypipes: bet that's it 18:48:38 <jgriffith> jaypipes: actually I'll be it's my openstack secgroups 18:49:02 <jaypipes> yup. 18:49:15 <jaypipes> I've not seen this before either: Dependency token-macro (1.5.1) doesn't exist 18:49:21 <jaypipes> not sure if that is important or not, though. 18:49:28 <jgriffith> me neither :) 18:49:38 <jaypipes> clarkb, fungi: ever seen "Dependency token-macro (1.5.1) doesn't exist" in a jenkins log before? 18:49:41 <jgriffith> yeah, so I've only got 22 18:49:47 <jgriffith> on TCP 18:49:51 <jaypipes> jgriffith: open that puppy up! 18:49:53 <jaypipes> :P 18:49:59 <jaypipes> 8080 18:50:28 <clarkb> jaypipes: I have not 18:50:39 <clarkb> but guessing its an interplugin dependency that needs resolving 18:50:52 <jgriffith> jaypipes: yay for me 18:50:58 <jaypipes> :) 18:50:59 <jgriffith> that was it 18:51:19 <jaypipes> clarkb: prolly, yeah... let's see if opening up the sec groups helped that. 18:51:25 <jgriffith> jaypipes: any other ports off the top of your head while I'm at it? 18:51:26 <jaypipes> jgriffith: re-run install_master.sh 18:51:40 <jgriffith> jaypipes: do I need to clean anything out manually first? 18:51:44 <jgriffith> jaypipes: or just rerun? 18:51:44 <jaypipes> jgriffith: no 18:51:45 <fungi> jaypipes: not familiar looking to me 18:51:48 <jgriffith> k 18:52:07 <jaypipes> jgriffith: just 80, 8080, 443 and 22 18:53:30 <fungi> also, late in responding, but i'm very much in favor of krtaylor's proposal to lighten the load on infra, as long as that group is not afraid to ask us questions (please don't disappear and start coming up with solutions in a vacuum!) 18:54:29 <krtaylor> fungi, agreed, we'll have to keep you all in the loop somehow 18:54:42 <jaypipes> fungi: ++ 18:54:47 <jgriffith> jaypipes: reran, don't see any errors in the new run 18:54:51 <jgriffith> jaypipes: think I'll proceed 18:55:00 <jaypipes> coo 18:55:58 <fungi> always happy to help when there are questions, but an faq would be a great start (something deeper than the current third-party ci doc we have--it desperately wants to be improved) 18:57:17 <jaypipes> gonna end this meeting since infra is up next for their weekly meeting... see you all next week. 18:57:20 <jaypipes> #endmeeting