18:00:02 #startmeeting sahara 18:00:03 Meeting started Thu Aug 14 18:00:02 2014 UTC and is due to finish in 60 minutes. The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:05 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:06 sahara folks, ping 18:00:07 The meeting name has been set to 'sahara' 18:00:11 o/ 18:00:14 hi 18:00:21 tmckay, are here already? 18:00:37 alazarev, sreshetnyak, ping 18:00:39 dmitryme 18:00:43 o/ 18:00:49 o/ 18:00:49 elmiko, does croberts already on PTO? 18:01:23 o/ 18:01:26 SergeyLukjanov: yes, baby arrived two nights ago :) 18:01:31 SergeyLukjanov: you are fast today, 18:00:02 2014 UTC 18:01:36 hi 18:01:42 wow! awesome news! 18:01:46 elmiko, yay! my congrats to him if you'll see him :) 18:02:01 my congrats too ;) 18:02:03 i'll pass on the well wishes 18:02:07 grats! 18:02:18 okay, ley's start 18:02:25 #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda#Next_meetings 18:02:37 #topic sahara@horizon status (croberts, NikitaKonovalov) 18:02:44 NikitaKonovalov, could you please make an update? 18:02:52 #link https://etherpad.openstack.org/p/sahara-juno-post-merge-changes 18:03:01 NikitaKonovalov, does bootstrap 3 patch merged? 18:03:40 hm 18:03:47 it means "nope"? 18:03:48 ;) 18:03:55 not merget 18:04:21 there is on big change from Chad 18:04:35 and a few small fixed from me 18:04:45 NikitaKonovalov, could you please track them and make them landed? 18:05:03 ok, I'l keep a track of them 18:05:20 it has a lot of +1 from sahara guys, but no marks from horizon 18:05:21 https://review.openstack.org/#/c/110680 18:05:44 NikitaKonovalov, how about templates copy? 18:06:05 the change is also on review 18:06:18 but no horizon reviewvers yet 18:06:23 SergeyLukjanov: templates copy are dependant on bootstrap 3 patch 18:06:50 alazarev, oh, yeah 18:07:11 NikitaKonovalov, anything else on dashboard topic? 18:07:37 here I am 18:07:51 I think we should clean the old repo as soon as possible 18:08:01 NikitaKonovalov, +1 18:08:02 some people put change requests tehre 18:08:06 from time to time 18:08:24 any objections to remove old dashboard from repo and tweak README to go to the horizon? 18:08:31 SergeyLukjanov: I fixed templates list error handling and it was merged, this is the only sahara patch not in bootstrap3 chain 18:08:43 no objections 18:08:46 no objections 18:09:05 #agreed remove old dashboard from repo and tweak README to go to the horizon 18:09:10 devstack has two Saharas now :) 18:09:13 any voluteers to do it asap? 18:09:27 SergeyLukjanov: we will still have access to icehouse dashboard though? 18:09:42 elmiko, sure, just remove saharadashboard dir in master 18:09:46 SergeyLukjanov: I can take that 18:09:48 elmiko, and tweak README 18:09:58 NikitaKonovalov, okay, thx, please, do it tomorrow ;) 18:10:27 #action NikitaKonovalov to remove dashboard code from master of sahara-dasboard and tweak README with instructions 18:10:33 SergeyLukjanov: cool, thanks 18:10:36 remove the code? 18:10:40 I can remove dashboard from devstack 18:10:53 alazarev, it's done already by myself 18:10:59 alazarev, just waiting for review 18:11:10 SergeyLukjanov: oh, ok 18:11:12 I mean, are you going to keep the repository but move it somewhere else (not openstack namespace), or just put a big README? 18:11:28 tosky, code is already merged to Horizon 18:11:45 tosky, only selenium tests and README with instructions will be in sahara-dashboard repo 18:11:56 tosky: delete code from master, maintain branches only 18:11:56 tosky, the stable/icehouse branch will not be changed 18:12:13 SergeyLukjanov: forgot to mention, mattf won't be able to make it today 18:12:37 elmiko, yeah, I see :) 18:12:49 #topic News / updates 18:12:52 folks, please 18:13:05 SergeyLukjanov, just so you know it was on purpose, and he mentioned it ahead of time :) 18:13:17 SergeyLukjanov, alazarev: ack, thanks 18:13:19 tmckay, ack :) 18:13:31 I was working on devstack changes (small refactoring and sahara-dashboard removal) 18:13:40 i've been working on chasing down the details needed to update the swift authentication spec. it's complicated and i'd definitely like to talk about the changes we will ask of operators in the name of increased security 18:13:43 just looked at spark integration tests adding, looks nice, I think we can merge it :) 18:13:44 I'm working on various CDH fixes and review 18:14:18 I have a few things to say about Oozie in relation to elmiko swift auth changes in open discussion. Basically, possible ways to lockdown Oozie (because creds will still be in workflow) 18:14:30 aignatov, we should have job that runs it before merge 18:14:45 aignatov, ack, it doesn't do scaling but I want to help with swift auth first, seems like a priority for Juno 18:15:03 i’ve already merged it since it has 2 +2 18:15:05 bug fixing, pushing security group auto creation, ops error handling and antiaffinity for heat 18:15:36 SergeyLukjanov: exactly :) 18:15:38 I also have several topics for open discussion 18:15:43 also, on swift test -- I really want to use spark wordcount, but until we have spark with swift, it's too much trouble to make it use local hdfs (with an upload of a data file, etc etc) 18:15:53 so, I just used SparkPi with no IO 18:16:50 spark with swift may have to wait until K, not sure about Juno. I think it may ride on top of the hadoop swift changes (I think swift delegates to hadoop for file access) but I don't know 18:16:53 elmiko, tmckay, guys, do you think it's possible to migrate to the new security model in J? 18:17:15 it's getting close. 18:17:23 elmiko, tmckay, assuming that we should keep backward compat / upgradebility 18:17:43 SergeyLukjanov: code wise, i think we can make it. will be a push, i'm more concerned about the integration issues i outlined in the email. 18:17:45 One reason I want to look at Oozie lockdown. Even if we miss, Oozie lockdown would improve the situation a lot 18:18:25 SergeyLukjanov: if we switch to the new method(proxy users with trusts), i think we could keep backward compat as needed. 18:18:32 tmckay, elmiko, could you please ack that you're planning to switch in backward compat style/ 18:18:52 ack, I agree on backward compat. We just need a cluster config to say which method 18:19:06 SergeyLukjanov: i'd like to remove it, but i think we can write the hadoop-swift plugin to take both styles of auth into account 18:19:10 tmckay, elmiko, I just would like to be sure that we'll be able to upgrade Icehouse sahara to Juno one ;) 18:19:29 basically, if the hadoop-swift see a trust id/domain id with it's user creds, then it will attempt new style, otherwise old style. 18:19:41 I agree. That should be a goal. Better to wait a release than break stuff if it comes to that, imho. 18:19:55 tmckay, yup 18:20:00 elmiko, ++ 18:20:17 okey, so, we have 2.5 weeks to land it 18:20:17 and the job manager will generate the workflow based on cluster config (or data source object content, or something). Shouldn't be too hard. 18:20:43 SergeyLukjanov: i think the proposals we are making to the hadoop-swift component will actually make it more functional with keystone v3, we adding features from v3. 18:21:12 For spark, we will have to figure out how to get the auth info into the hadoop config (if indeed spark is delegating to hadoop for file access) 18:21:27 but I think spark swift is probably K 18:21:39 external hdfs will have to be good enough for Juno 18:21:47 ++ 18:21:50 elmiko: but we should still support keystone v2 18:22:53 alazarev: i think we can by having the hadoop-swift component look to see what parameters it receives. if it receives just username/pass, then v2. if it receives username/pass/trust/domain then v3. 18:23:17 elmiko: sounds good 18:23:34 my big concern though, is that we will need to distribute the new hadoop-swift based images and the packages. 18:23:34 elmiko, small correction 18:23:35 elmiko, ack. I'm guessing it gets a new client for each access 18:23:51 elmiko, v3 should be always ued with fallback to v2 with creds 18:23:59 SergeyLukjanov: ack 18:24:19 elmiko, I'm working on building images in gate 18:24:36 and I'd like to propose moving hadoop-swiftfs to spearated repo 18:24:49 SergeyLukjanov: ++ 18:24:54 elmiko: we already distribute hadoop-swift since upstrean doesn’t support deta locality 18:25:06 alazarev, yup 18:25:11 alazarev: ok, great 18:25:21 also, folks, please take a look at https://review.openstack.org/#/c/113591/ 18:25:33 #topic extract hadoop-swiftfs to the separated repo 18:25:35 i'd like to start making PRs for the change in spec 18:25:38 so, the topic is 18:25:45 I +2'd already 18:25:47 please, share your thoughts folks 18:25:49 :) 18:26:13 if we move it to a new repo, sahara-extras will be dead? 18:26:22 elmiko, I think so 18:26:34 SergeyLukjanov, so edp-examples will be gone, then only thing left in sahara-extra is the hadoop plugin. Why move it? 18:26:45 i don't mind moving it as part of the work we are doing on swift/auth. since we'll need to make changes anyway. 18:26:49 just a name change at that point, right? 18:27:00 tmckay, examples already gone 18:27:22 tmckay, good point for renaming 18:27:48 rename and some cleanup to the READMEs 18:27:52 SergeyLukjanov: will it be stackforge, openstack or hadoop-something? 18:27:58 Ideas for new name? 18:28:01 tmckay, but probably we'd like to keep the stable/icehouse in -extra 18:28:11 ah, good point 18:28:34 tmckay, so, it's better to move hadoop-swiftfs out of extra 18:28:35 are we going to be adding a component for spark-swift access? 18:28:48 tmckay, but keep -extra as is with deprecation mark 18:28:53 elmiko, we don't know yet 18:28:57 it might "just work" 18:29:08 * elmiko crosses fingers 18:29:12 elmiko, willb suggested to me a long time ago that swift just delegated to hadoop 18:29:13 imho it's bad idea because hadoop vendors will not support custom hadoop-swiftfs library 18:29:36 tmckay: hdfs is pretty the same in spark, so it should just work 18:29:45 I think so 18:29:51 sreshetnyak: which part is a bad idea? 18:29:53 haven't poked into that yet 18:30:18 now we have hadoop-fs in hadoop repo, but it is outdated 18:31:09 if we manage to make changes fast enough we can use it, this should be a goal 18:31:22 elmiko, extract hadoop-swiftfs library 18:31:27 on the other hand the one in -extra works ok, so, no real need to change it now 18:31:39 sreshetnyak, good idea 18:31:51 we should decide first what to do with code in hadoop 18:32:26 sreshetnyak: are you saying we should leave the hadoop-swiftfs in the extras repo? 18:33:11 I think the right approach is to decide where this code should live - in hadoop eco or in openstack 18:33:28 and if hadoop eco than in hadoop repo or separatedly as contrib fs 18:33:30 problem with hadoop repo is that changes need to be reviewed by hadoop guys, it is slow and non-trivial for openstack eco 18:33:34 SergeyLukjanov: would hadoop consider accepting it? 18:33:50 elmiko, some version of this code is already in hadoop 18:33:52 alazarev: that concerns me too 18:34:04 alazarev, yup, it's the main concern 18:34:20 it took 6 months to merge first version 18:34:24 SergeyLukjanov: ok, but the real issue is we need to iterate with changes to the openstack infrastructure. which is must faster than hadoop currently. 18:34:35 it could be kept separate until it's integrated, as it happened in the past, I think 18:34:41 that seems like we should keep it separate 18:34:48 so, -extra works ok for us now, let's not spend time for useless moving now 18:35:11 it 18:35:17 i'm good with less work =) 18:35:23 +1 18:35:29 it's easy enough to extract, but we'll still have -extra 18:35:35 because of stable branch 18:35:50 and so there is not so cool to just add one more repo in fact ;) 18:36:09 if we're going to change the hadoop-swiftfs for this auth stuff, maybe we should leave it in -extra until sometime in K 18:36:20 elmiko, yup 18:36:30 or we'll have some good ideas on summit 18:36:37 let's hope :) 18:36:48 I think the right approach is to decide where this code should live - in hadoop eco or in openstack 18:36:51 22:34 SergeyLukjanov@: and if hadoop eco than in hadoop repo or separatedly as contrib fs 18:36:53 22:34 alazarev : problem with hadoop repo is that changes need to be reviewed by hadoop guys, it is slow and non-trivial for openstack eco 18:36:57 22:34 elmiko : SergeyLukjanov: would hadoop consider accepting it? 18:37:12 sorry, wrong message 18:37:15 sreshetnyak: it seems like we need to keep it in openstack eco for now 18:37:34 yeah, for J at least 18:37:53 #agreed no changes needed, at least in J 18:37:57 okay, let's move on 18:38:05 elmiko, ok, agree 18:38:10 #topic Juno 3 (Sept 4) 18:38:16 #link https://wiki.openstack.org/wiki/Juno_Release_Schedule 18:38:29 so, just reminder - winter is coming! 18:38:34 lol 18:38:55 alazarev, what's your topic to discuss? 18:39:11 hdfs usage in edp? 18:39:39 SergeyLukjanov: sqlite support, broken pig in hadoop2, EDP dependancy on HDFS 18:39:51 #topic sqlite support 18:40:25 I’m Ok with removing sqlite support, but we need to anounce this somehow 18:40:37 so, AFAIK other project have same migrations with ALTER COLUMN that isn't supported by SQLit 18:41:33 aignatov, looks like neutron has such migrations 18:41:44 we need to clear docs from sqlite in this case 18:41:44 aignatov, are you using MySQL for dev? 18:42:01 alazarev: good point, we need to put that as an item in our update doc 18:42:03 SergeyLukjanov: I think this topic was rised haklfyear ago when we started adding migrations to sahara with alembic 18:42:06 and I don't like hacking python/sqla to support sqlite 18:42:08 I use sqlite. Been meaning to switch. 18:42:15 and decided to get rid of sqlite support 18:42:19 I remember that 18:42:30 aignatov, yes, I remember something like this 18:42:47 oh yeah, we were having the same migration the whole icehouse 18:42:59 until the pre-release squashing of migrations 18:43:06 aignatov: yep, me too. Then we found a way to make migrations for Icehouse without alter, by sqashing them 18:43:16 so sqlite worked for Icehouse 18:43:24 I just remove ALTERS :) works fine 18:43:26 yes, but after that changes were merged for icehouse and incompatibility disappear 18:43:58 tmckay: don’t do that in any environment you value :-) 18:43:59 just kidding -- not suggesting that as a real solution. But it does work when you're in a hurry. 18:44:14 :) 18:44:18 tmckay: now you can do it in one file, it is Ok with 2-3, nearly impossible with 10+ 18:44:30 #agreed to drop sqlite support, need to cleanup docs 18:44:44 * tmckay time to switch finally 18:44:47 I can clean up docs and samples 18:45:01 (we have sqlite in samples too) 18:45:02 #agreed to not hack migrations by removing ALTER from them in production 18:45:13 alazarev, yes, please 18:45:16 lol, did we really need to say that? 18:45:26 tmckay: me too, gotta set up that local mysql server... 18:45:27 #action alazarev to cleanup sqlite from docs and samples 18:45:30 to not hack _officially_ :) 18:45:36 the same with neutron to answer on SergeyLukjanov question, you can play with sqlite but it’s not for production and migrations :) 18:45:43 tmckay, I'm just kidding ;) 18:46:48 next topic? 18:46:51 #topic EDP dependency on HDFS 18:47:48 Jon Maron implemented “mkdir -p” manually because HDFS FsShell API changed in hadoop2 18:48:05 now we have now we have https://bugs.launchpad.net/sahara/+bug/1355114 and https://bugs.launchpad.net/sahara/+bug/1356582 18:48:08 Launchpad bug 1355114 in sahara "[HDP] Parallel running EDP jobs failed on HDP1 cluster" [Medium,Confirmed] 18:48:55 ok, is it impossible to fix that? 18:49:18 the main problem is that EDP calls HDFS directly without knowledge what HDFS it is 18:49:35 I see several ways to fix 18:49:50 alazarev, and w/o knowledge about existence of HDFS (storm for ex.) 18:50:29 1. put HDFS (or other storage) communication into plugins (hard to implement, will save from problems with storm) 18:51:01 2. expose hadoop version (1 or 2) from plugins and add “if” to EDP 18:51:13 * SergeyLukjanov don't like #2 18:51:50 me too 18:52:21 any objections against #1? 18:52:26 3. require clusters to provide unified way for HDFS operations (need to think more) 18:52:37 ca it be fixed that way -> plugin has new SPI method “say to EDP engine which storage you are using for temporary purposes” 18:52:47 *require plugins 18:53:13 i like #1, but sounds complicated 18:53:17 I like #1, especially if we have implementations that can be shared by plugins. If multiple plugins use hdfs with the same interface, then we should only implement once. 18:53:20 I actually like #2 18:53:38 #2 is not terrible. 18:53:47 but I am also ok with tmckay’s correction of #1 18:53:54 Allocate a class that wraps hdfs funtcions based on hadoop version 18:54:10 I don’t like how SPI currently implemented, may be we should switch to multiple interfaces 18:54:12 I think either one can work 18:54:21 tmckay, some sort of storage abstractor? 18:54:38 e.g. we can add “get_storage_manager” method to PSI 18:54:51 *SPI 18:54:59 alazarev: yes, I meant the same 18:55:10 yeah. Ask the plugin what the hadoop version is, pass it off to a factory, get "MyGreatHDFS" object back, "myobj.makedir(path, "dashp=True)" 18:55:20 * tmckay made that up real quick 18:55:34 I like it 18:56:41 tmckay: this is exactly solution with storage manager, plugins will just return shared impl 18:56:54 yes, get_storage_manager, basically the same idea 18:57:03 almost the same as the EDP engine stuff 18:57:17 doesn't have to be a formal plugin, just a factory and common interface 18:57:39 tmckay, ++ 18:57:43 I think we are all agreeing :) 18:57:47 so we got the decision, looks like nice bluprint to be implemented :) 18:57:52 tmckay: storage manager and EDP engine are hardly tight 18:58:10 do we want this in J? 18:58:30 yes, they are related -- could be the same object 18:58:46 or, at least in the same directory 18:58:55 alazarev, I think it'll be cool to fix this issues in J 18:58:55 EDP engine is currently just "status, run, kill" 18:59:06 1 min left 18:59:20 #topic Open discussion 18:59:26 last seconds :) 18:59:38 with broken pig in oozie I think the best solution is custom oozie build 18:59:41 okay, quick comment on Oozie. Either with Ooozie config settings, or firewall settings, I think we need to block oozie server access to Sahara and cluster nodes 18:59:47 i have a few concerns about what we will asking operators to change about their keystone configs for the swift/auth stuff. but i will put it an email. 18:59:57 so that people cannot spy on jobs, read configs, and thereby get creds and trust tokens 19:00:07 let's continue in our channel 19:00:09 thanks folks 19:00:10 I am looking in to both -- firewall setup, and oozie configs 19:00:11 #endmeeting