14:01:20 <SergeyLukjanov> #startmeeting sahara
14:01:21 <openstack> Meeting started Thu Jan 15 14:01:20 2015 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:23 <SergeyLukjanov> hi folks
14:01:25 <openstack> The meeting name has been set to 'sahara'
14:01:25 <crobertsrh> hello/
14:01:36 <SergeyLukjanov> #link http://wiki.openstack.org/wiki/Meetings/SaharaAgenda
14:01:38 <elmiko> mornin
14:01:39 <SergeyLukjanov> crobertsrh, morning
14:01:44 <SergeyLukjanov> elmiko, morning
14:02:20 <SergeyLukjanov> #topic sahara@horizon status (croberts, NikitaKonovalov)
14:02:25 <SergeyLukjanov> crobertsrh, NikitaKonovalov, please
14:02:42 <crobertsrh> Usual update:  Horizon reviews linger.
14:02:56 <SergeyLukjanov> heh
14:02:56 <NikitaKonovalov> So we have a bunch of changes as always waiting
14:03:07 <crobertsrh> I'm part way finished with a "cluster creation guide" mode
14:03:08 <SergeyLukjanov> crobertsrh, NikitaKonovalov, could you please make a single etherpad
14:03:19 <SergeyLukjanov> with all changes on review in horizon related sahara
14:03:32 <SergeyLukjanov> with launchpad bug id, title and state
14:03:37 <crobertsrh> That's a fine idea.
14:03:53 <NikitaKonovalov> I guess we could use a single topic in the changes, and then filter by it in gerrit
14:04:00 <aignatov> o/
14:04:15 <sreshetnyak> hi
14:04:41 <NikitaKonovalov> but the etherpad will definitely work
14:05:22 <sreshetnyak> i'm working on migration to openjdk and new integration tests
14:06:45 <tmckay> I've just about got spark/swift support done for EDP.  There is one issue, though, which is the jackson-mapper-asl jar version.  Not sure what to do about that.  Upgrade of CDH version might fix it.
14:07:12 <tmckay> Or, we may have to carry a version of the jar ourselves, or we might have to fix up the hadoop plugin to avoid the issue
14:07:29 <tmckay> Currently I am adding the jackson jar to the nodes manually for development
14:07:53 <tmckay> Spark wrapper is based on oikawa 's Oozie Java wrapper, thanks oikawa!
14:08:11 <tmckay> maybe he's not here
14:08:43 <kazuki_oikawa> /
14:08:55 <tmckay> oh, hi!  wrong name :)
14:09:25 <tmckay> SergeyLukjanov, also, we should talk about location to store wrapper jars and sources
14:09:42 * tmckay hasn't checked comments on the review last couple day
14:10:54 <tmckay> no new comments on https://review.openstack.org/#/c/115884/
14:10:57 <huichun> Hi sreshetnyak are you working on adding new integration tests for the others services by writing the examples ?
14:11:00 <tmckay> #link https://review.openstack.org/#/c/115884/
14:11:49 <SergeyLukjanov> (/me forgot to switch topic)
14:12:26 <sreshetnyak> Hi huichun. I'm working on adding new integration tests for all services and all plugins
14:12:53 <SergeyLukjanov> #topic News / updates
14:12:53 <sreshetnyak> #link https://review.openstack.org/#/c/147219/
14:13:50 <crobertsrh> I'm still working on a "guided mode" for cluster creation.  It's almost done.  Then I'll do something for job creation/execution.
14:14:12 <elmiko> i've been continuing work on the sahara security guide, there is a lot of ground to cover but i'm making progress. also, i was able to run bandit against the sahara codebase, it went smoothly and reported some thing that we might investigate.
14:14:16 <elmiko> #link https://gist.github.com/elmiko/94f063583912a5d8c3cb
14:14:19 <SergeyLukjanov> tmckay, the best location is tarballs.o.o I'm still working on generic solution to store things there for sahara and hope to make it soon
14:14:20 <elmiko> ^^ bandit output
14:14:32 <crobertsrh> In general, it's a single page with plenty of descriptive text for each step and links to the required forms to complete each step.  Hopefully, it will help the newer users find their way.
14:14:43 <tmckay> SergeyLukjanov, okay.  Maybe we can store a jackson jar there if we need to, as well
14:14:53 <SergeyLukjanov> tmckay, yup I think
14:15:04 <kchen> hi sreshetnyak. Huichun is mainly focus on CDH plugin services.
14:15:23 <kchen> I think his work is a bit different from yours.
14:16:44 <kchen> crobertsrh. what do you mean by "guided mode"?
14:17:14 <crobertsrh> It's a "wizard-like" mode to help novice users through cluster creation.
14:17:37 <crobertsrh> It currently utilizes the existing forms, but ties them together in an understandable way
14:17:57 <kchen> it sounds good to me. I am also thinking of this question.
14:18:02 <tmckay> kchen, maybe huichun can help me figure out the CDH spark assembly jackson-mapper-asl version issue
14:18:27 <crobertsrh> kchen:  I hope to have a patch up sometime soon.
14:18:39 <tmckay> crobertsrh, it occurred to me yesterday that the UI might not have the support in it yet for buttons to set the data source substitution options
14:18:41 <crobertsrh> Or, I may just have screenshots for next week's meeting.
14:18:55 <kchen> crobertsrh:  great.
14:18:55 <crobertsrh> tmckay:  that is correct.
14:19:00 <tmckay> crobertsrh, we haven't talked about that since 2014 :)
14:19:32 <SergeyLukjanov> :)
14:19:32 <tmckay> patch should be merged soon.  I thought it was already but SergeyLukjanov dinged my commit message ;-)
14:19:44 <SergeyLukjanov> oops, sorry
14:19:48 <huichun_> hi sreshetnyak  sorry for lost connection, I am working on the integration tests for the new servcies too, but met some issue
14:19:53 <tmckay> haha, no problem
14:20:02 <kchen> crobertsrh, do we have a review link for that patch?
14:20:18 <crobertsrh> Not yet.
14:20:43 <crobertsrh> It's still rather ugly at the moment.  I'm still playing with some ideas and trying things that may or may not work.
14:21:14 <huichun_> for example. how can we add classpath in the example jar?   for example. we wrote a HBaseTest.jar, but it depends on hbase-client.jar, so how can we pass `hbase classpath` by using java action?
14:21:14 <tmckay> huichun_, do you know a lot about CDH?  I might need to talk with you about the jackson issue I've run into
14:21:31 <huichun_> hi tmckay
14:21:50 <tmckay> I'm talking about the Spark image specifically
14:22:11 <weiting> tmckay, which version you used in Spark, version 4, right?
14:22:21 <tmckay> weiting, yes
14:22:30 <kchen> I am working on CDH version management now, and I expected to have a patch soon. The first patch draft maybe by early next week.
14:22:35 <tmckay> 5 might fix the issue, haven't checked
14:22:54 <huichun_> tmckay, sorry for lost connection, miss your words just now
14:24:01 <tmckay> huichun_, that's okay.  I have an issue with the spark assembly jar in the Spark image for Sahara.  I wondered if you had some expertise there (it is a CDH image)
14:24:46 <SergeyLukjanov> kchen, that's cool, we're really missing version management for CDH
14:25:01 <SergeyLukjanov> oh, we have an action item from the last meeting for elmiko -- ACTION: elmiko to run bandit against codebase
14:25:07 <elmiko> yea
14:25:09 <elmiko> #link https://gist.github.com/elmiko/94f063583912a5d8c3cb
14:25:12 <elmiko> that's the result
14:25:29 * SergeyLukjanov looking
14:25:29 <elmiko> it went pretty smoothly, i just followed the bandit instructions and it ran without issue
14:25:42 <huichun_> hi tmckay, we could talk it offline by email?
14:25:53 <tmckay> huichun_, certainly, thanks
14:26:17 <kchen> oh one thing. currently cdh integration edp and hive test can pass. so need we turn it back to voting?
14:26:28 <elmiko> SergeyLukjanov: it mostly doesn't like us using /tmp, but probably doesn't realize that's on the nodes.
14:26:42 <elmiko> still, might be something to investigate for improving on-node security
14:26:48 <tmckay> kchen, +1
14:27:20 <huichun_> hi, SergeyLukjanov, should we add  shell action and sqoop action in edp job type?
14:27:44 <SergeyLukjanov> huichun_, good question, I think spec needed for it and could be discussed in spec review
14:28:18 <elmiko> SergeyLukjanov: i think next steps with regards to bandit will be to dig into these warnings and determine what the OSSG recommends
14:28:19 <tmckay> huichun_, also, one thing we've talked about but never had time to implement is raw Oozie workflows
14:28:24 <huichun_> ok. I have made a bp, and will update the specs
14:28:32 <tmckay> huichun_, user uploads own workflow.xml
14:29:04 <tmckay> that would allow users to use more Oozie features, like dags and so on
14:29:43 <huichun_> tmckay, you mean user can upload his own workflow.xml by horizion page?
14:29:44 <SergeyLukjanov> huichun_, thx
14:29:52 <tmckay> huichun_, yes
14:29:58 <SergeyLukjanov> elmiko, thanks for making security work
14:30:13 <elmiko> SergeyLukjanov: np, i'll keep pushing forward
14:30:57 <huichun_> i found this problem, currently we have no way to add like classpath for the Java action, right ? tcmky
14:31:04 <huichun_> tmckay
14:31:57 <tmckay> huichun_, hmmm.  There is a java options config that can be set, but I'm not sure how Oozie handles that
14:32:34 <tmckay> huichun_, but we have a general mechanism of "edp.xxx" configs where we can pass anything we need to the EDP engines.  So we should be able to add it if we need something else
14:33:18 <tmckay> huichun, edp.java.java_opts
14:33:52 <huichun> hi, tmckay, we have tried
14:34:09 <huichun> but it's not work, we pass `hbase classpath` in the Java_opts, but failed
14:34:34 <tmckay> huichun, I can help you investigate, I think
14:35:01 <huichun> hi, tmckay, we can talk this offline
14:35:04 <huichun> thanx tmckay
14:35:10 <tmckay> huichun, agreed
14:36:14 <SergeyLukjanov> okay, let's move on :)
14:36:25 <SergeyLukjanov> #topic Bug/doc/spec days
14:36:35 <SergeyLukjanov> big topic I'd like to chat about
14:36:52 <SergeyLukjanov> IMO we need bug triage and bug fix days
14:36:57 <elmiko> +1
14:37:01 <SergeyLukjanov> we already have many issues in lp
14:37:04 <SergeyLukjanov> too many imo
14:37:14 <crobertsrh> +1.  Been a long time since last bug day
14:37:14 <SergeyLukjanov> docs are not bad
14:37:31 <elmiko> we also need more reviews on bug fixes
14:37:59 <tmckay> +1
14:38:04 <tmckay> maybe even 2 days
14:38:08 <tellesnobrega_> +1
14:38:20 <tmckay> next week?
14:38:35 <SergeyLukjanov> https://review.openstack.org/#/q/sahara-specs+AND+status:open,n,z specs are in a good shape IMO
14:39:01 <SergeyLukjanov> yeah, we need two separated days
14:39:14 <SergeyLukjanov> one to triage and plan what should be fixed, find volunteers
14:39:26 <SergeyLukjanov> and one to only fix prioritized issues
14:39:45 <elmiko> i'd be ok with a 2 day hack-a-thon ;)
14:39:54 <elmiko> er, bug-fix-a-thon
14:40:14 <SergeyLukjanov> it could be done like a virtual sprint
14:40:54 <SergeyLukjanov> it means we'll agree on some time to start on
14:41:12 <elmiko> tmckay proposed sometime next week, is that too soon?
14:41:14 <SergeyLukjanov> nope, it works bad for this task, rollback :)
14:41:36 <SergeyLukjanov> what's about next two mondays?
14:41:44 <SergeyLukjanov> jan 19 and 26
14:41:53 <tmckay> +1, sounds good to me
14:42:00 <elmiko> 19th for planning/triage, 26th for fixing?
14:42:05 <SergeyLukjanov> it makes volunteers able to fix some bugs after triaging and before the bug fix day
14:42:09 <SergeyLukjanov> elmiko, yeah
14:42:12 <elmiko> +1
14:42:13 <crobertsrh> Those should work for me. +1
14:42:40 <SergeyLukjanov> at what time could we try to start between locations?
14:42:41 <tellesnobrega_> i can help out as well
14:42:47 <sreshetnyak> +1
14:43:21 <kchen> will we hold a meeting for that or not?
14:43:30 <elmiko> SergeyLukjanov: for planning we could the same time as this meeitng
14:43:43 <tmckay> SergeyLukjanov, when the first contributors wake up on Monday til the last ones go to bed :)
14:44:00 <tmckay> So  maybe it starts Sunday night in my time zone
14:44:27 <elmiko> well, if we plan for the 19th at UTC14:00 in #openstack-sahara, we could try to catch many people. (although it will be tough for west coast USA)
14:45:03 <tmckay> elmiko, there is always eavesdrop, too, if someone has to catchup
14:45:19 <SergeyLukjanov> elmiko, ++ for Jan 18 1400 UTC
14:45:21 <elmiko> kchen, weiting, kazuki_oikawa, how late/early is UTC14:00 for you guys?
14:45:31 <kchen> 10pm
14:45:48 <tmckay> SergeyLukjanov, 18th or 19th
14:45:50 <weiting> UTC+8 in our zone
14:46:06 <elmiko> k, thanks. kinda late to start triaging bugs though...
14:46:11 <kchen> the time is ok for us
14:46:42 <kazuki_oikawa> 11pm (UTC+9). but the time is ok.
14:46:50 <elmiko> ok, great then
14:46:55 <huichun> the same time as we are right now
14:47:23 <elmiko> yea, i just thought it might be late to start triaging. but if you guys are ok with it that's awesome =)
14:47:24 <SergeyLukjanov> tmckay, 19th
14:47:51 <SergeyLukjanov> yeah, that's awesome!
14:48:17 <SergeyLukjanov> #agreed bug triage day at Jan 19th starting at 1400 UTC
14:48:25 <SergeyLukjanov> what's about bug fixing day?
14:48:30 <SergeyLukjanov> the same, but Jan 26th?
14:48:38 <mattf> isn't that every day?
14:49:05 <elmiko> lol
14:49:14 <tmckay> mattf, some days are bug creation days
14:49:17 <kchen> :D
14:49:29 <mattf> tmckay, good call
14:50:02 <tmckay> SergeyLukjanov, +1 same on 26th
14:50:12 <elmiko> SergeyLukjanov: +1
14:51:14 <tosky> so is it also for specs, or are they handled in the usual way?
14:51:51 <elmiko> tosky: we are mostly up to date on specs
14:51:54 <SergeyLukjanov> specs are in a good shape
14:52:00 <elmiko> but too many bugs =(
14:52:03 <SergeyLukjanov> yeah
14:52:18 <SergeyLukjanov> #agreed bug fix day at Jan 26th starting at 1400 UTC
14:52:39 <SergeyLukjanov> 7 mins left, anything else to discuss?
14:52:46 <SergeyLukjanov> #topic Open discussions
14:53:20 <elmiko> always looking for more eyes on the ongoing security guide etherpad
14:53:25 <elmiko> #link https://etherpad.openstack.org/p/sahara-security-guide-notes
14:54:30 <tellesnobrega_> i have a question about storm edp, are we planing to have it in kilo?
14:55:52 <tosky> can we talk about the new yaml-based test scenario spec on the channel after the meeting?
14:55:59 <tosky> also: did you recently try to generate the hdp2 centos6._6_ image?
14:56:21 <tmckay> tellesnobrega_, is it done?
14:56:30 <SergeyLukjanov> tosky, sure (re new tests)
14:56:38 <tmckay> tellesnobrega_, apologies, I'm behind on reviews
14:57:02 <tmckay> tellesnobrega_, if it's finished, and we have integration tests, and it passes ... and it's in by the feature freeze
14:57:17 <tmckay> then I would say "sure!" ^^
14:57:42 <tmckay> otherwise L
14:58:06 <tellesnobrega_> not ready, i will send the spec tomorrow or the day after
14:58:20 <tellesnobrega_> still need to approve that in order to implement it all
14:58:24 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Kilo_Release_Schedule
14:58:25 <tellesnobrega_> when is feature freeze?
14:58:32 <SergeyLukjanov> FPF is March 5
14:58:33 <tellesnobrega_> SergeyLukjanov, thanks
14:58:56 <SergeyLukjanov> so, I think you have enough time to get approves :)
14:59:09 <tmckay> tellesnobrega_, that's proposal freeze, feature freeze is March 19
14:59:16 <SergeyLukjanov> oh, folks, please, review Indirect access done by alazarev
14:59:26 <tellesnobrega_> yeah
14:59:40 <SergeyLukjanov> #endmeeting