14:01:13 <tellesnobrega> #startmeeting sahara
14:01:13 <openstack> Meeting started Thu Jul 12 14:01:13 2018 UTC and is due to finish in 60 minutes.  The chair is tellesnobrega. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:16 <openstack> The meeting name has been set to 'sahara'
14:01:53 <tellesnobrega> show off hands o/
14:02:02 <jeremyfreudberg> o/
14:02:04 <tosky> o/
14:02:06 <zchkun> o/
14:02:41 <tellesnobrega> #topic News/Updates
14:03:28 <tellesnobrega> I'm currently testing the ssl problem in ambari
14:03:39 <tellesnobrega> HDP 2.3 with ambari 2.2.0.0 works
14:03:48 <jeremyfreudberg> i am hard at work on s3 and apiv2 ecosystem. rushing to finish everything
14:03:51 <zchkun> I'm working on MapR6 upgrade. The current phase is creating the image with sahara-image-pack.
14:04:06 <tellesnobrega> next in line is HDP 2.4 with ambari 2.2.1.0
14:04:15 <tosky> jeremyfreudberg: I have few questions about S3, but for later
14:04:24 <jeremyfreudberg> tosky: certainly
14:04:34 <tellesnobrega> we can have a topic on it
14:04:37 <tellesnobrega> and apiv2 for sure
14:04:39 <tosky> I helped testing some of the patches for new plugins and S3 and gently nagged people to merge my patches :)
14:04:56 <tellesnobrega> other than that I'm also working on spark on sahara-image-pack
14:06:50 <tellesnobrega> any more news?
14:07:15 <jeremyfreudberg> just a reminder that i am gone the next two weeks.... or rather, unable to write code ( i can still review )
14:07:43 <tellesnobrega> ok
14:08:22 <tellesnobrega> lets move on
14:08:28 <tellesnobrega> #topic S3
14:08:33 <tellesnobrega> tosky, you have the floor
14:11:14 <tosky> I merged the S3 patch, but I'm not 100% sure about the differences between using s3:// and no http/https vs s3a:// and the need for it
14:11:49 <tosky> the definition of S3 resources for job binaries and data sources if different right now
14:12:11 <jeremyfreudberg> yes, the definition is different, and intentionally
14:12:23 <jeremyfreudberg> one is designed to work with boto, and the other is designed to work with hadoop-aws.jar
14:13:33 <tosky> but we are abstractor (!)
14:14:13 <tosky> can't we provide a unified way for users?
14:15:13 <jeremyfreudberg> maybe
14:15:19 <tellesnobrega> that would be the best way
14:15:33 <jeremyfreudberg> they aren't exactly the same thing, though
14:15:43 <tellesnobrega> we aim to be a front, makes things easier on the client side
14:15:50 <jeremyfreudberg> a job binary is a single file, but a data souce may be a directory or a wildcard
14:16:17 <jeremyfreudberg> i do see your point, though
14:16:18 <tosky> but isn't it something that we can catch on validation, before using the data source or the job binary?
14:16:46 <jeremyfreudberg> i can definitely add validation for the http or not thing
14:16:51 <jeremyfreudberg> there is already validation for s3 / s3a
14:17:22 <tosky> that would be better, and also the s3 vs s3a
14:17:35 <tosky> if we can hide the difference, that would be good too
14:18:52 <jeremyfreudberg> i am not sure i want to create a new unified addressing format
14:18:56 <jeremyfreudberg> that may be confusing for users too
14:19:58 <tellesnobrega> how is it used outside of sahara?
14:19:58 <tosky> do you see a use case where users may end up using the address (for data sources, if we unify to s3:/) wrongly
14:20:00 <tosky> ?
14:20:41 <tosky> or the other way: if it's easier for people to use this on hadoop stuff, maybe we could go with s3a also for job binaries
14:20:49 <jeremyfreudberg> yes, because for example, spark jobs take everything in as a positional arguemnt, either datasource://<ds-uuid> or s3a://actual/path
14:21:15 <jeremyfreudberg> it would be wrong to do s3a for job binaries, because they aren't s3a
14:21:57 <tosky> but we know that, do users need to do that?
14:22:19 <tosky> I know that they are not s3a; I'm just saying to do the conversion on the fly before using them
14:22:32 <tosky> before passing them to the code that uses them
14:23:17 <jeremyfreudberg> hmm
14:23:27 <tosky> same for the other way: in case of using s3:/, that would need to be changed to s3a:/ when passed to the spark job or other users
14:24:02 <tosky> the idea is to hide the different protocol on the sahara side; the different code for job binaries and data sources would see the protocol required
14:25:30 <tellesnobrega> I like this approach, maintains our UX improvement that we aim every cycle
14:25:56 <jeremyfreudberg> i will have to think about it
14:27:00 <tellesnobrega> jeremyfreudberg, give some thought and let us know how feasible it is, if it is too complicated, with all the apiv2 stuff we need fixing, we can postpone it, but it would be nice to have it
14:27:34 <tosky> uhm, we can't postpone it too much, because then we would have to support both schema
14:27:51 <jeremyfreudberg> yep
14:27:54 <tosky> but that wouldn't be a problem I guess, we can just not document the old non-unified option
14:27:55 <tellesnobrega> tosky, true
14:27:55 <tosky> anyway
14:28:19 <jeremyfreudberg> s3 is a big topic - anything else on it?
14:28:34 <tellesnobrega> not from me
14:28:56 <jeremyfreudberg> tosky?
14:29:02 <tosky> nothing else right now
14:29:08 <jeremyfreudberg> oki
14:29:32 <tellesnobrega> lets move on
14:29:33 <tosky> I will have more questions when I do more work on the tests, probably, but for now it's fine
14:29:51 <tellesnobrega> #topic APIv2
14:30:25 <tellesnobrega> jeremyfreudberg, I'm a little outdated on status on apiv2, can you give a quick summary of is done and what is still needed and how far we are from finishing it
14:31:08 <jeremyfreudberg> yep:
14:32:44 <jeremyfreudberg> in the past month, i made changes in APIv2's create multiple clusters and force delete, fixed some other (final )inconsistencies, and am now working on microversion(ish) and dashboard
14:33:28 <jeremyfreudberg> still to do (maybe in Stein) will be OSC for APIv2, good api-ref, and testing
14:35:57 <tosky> can we publish api-ref for v3 right now? I think it should be easy
14:36:07 <tosky> s/v3/v2/ :D
14:36:17 <tellesnobrega> I got worried a bit about v3
14:36:19 <tosky> it would help also as reference for the tempest client
14:36:21 <jeremyfreudberg> tosky -- it's not as magic as we'd like, there is some manual stuff to do for api ref
14:36:43 <tellesnobrega> jeremyfreudberg, I can start working on OSC
14:36:45 <jeremyfreudberg> also, we need to decide if we want to go EXPERIMENTAL->CURRENT in rocky
14:36:49 <tellesnobrega> hopefully I can finish in time
14:37:01 <tellesnobrega> we need OSC to have it current right?
14:37:03 <tosky> CURRENT means enabled, but not default, right?
14:37:12 <jeremyfreudberg> tosky: well, the user requests his version
14:37:22 <jeremyfreudberg> osc and dashboard could still default to 1.1
14:37:37 <jeremyfreudberg> but if we make it CURRENT, then that means all future changes will have to be in microversions
14:37:43 <tosky> and with old unmodified clients, you would still get the old version?
14:38:36 <tellesnobrega> I believe so
14:38:50 <tellesnobrega> I vote we go for CURRENT, but I would like to have OSC done as well
14:39:00 <tellesnobrega> client freeze is next week?
14:39:06 <tosky> I would vote to go for CURRENT only if scenario tests are passing
14:39:17 <tosky> which means support in saharaclient
14:39:22 <tosky> which means probably not enough time
14:39:30 <tellesnobrega> that seems like a good line
14:39:43 <jeremyfreudberg> tellesnobrega: client library freeze is same as rocky-3
14:39:59 <jeremyfreudberg> tosky: support is in saharaclient
14:40:00 <tellesnobrega> july 26th
14:40:02 <tosky> otherwise I don't see it too sound to say "yes, it's stable, but if you want to have it working you need microversining 2.5"
14:40:21 <jeremyfreudberg> i'm happy to do EXPERIMENTAL for one more cycle
14:40:38 <tellesnobrega> I guess I will have time to finish by july 26th
14:41:32 <jeremyfreudberg> now that we will have dashboard (and hopefully OSC), that will serve as its own kind of testing (through use)
14:41:44 <jeremyfreudberg> so we'll have one cycle more to naturally discover problems
14:41:57 <tosky> if you don't mind, unless you think you are working on that or something that touches that, I can take a look at api-ref generation
14:42:01 <tellesnobrega> but I would say, keep experimental for now. but in a couple weeks, if we are more confident we can set the target to CURRENT
14:42:29 <jeremyfreudberg> tosky: please work on api-ref. server side "should" already be done by now
14:42:32 <tosky> oki
14:42:56 <tellesnobrega> awesome
14:42:58 <tellesnobrega> do we have a plan?
14:43:17 <jeremyfreudberg> tellesnobrega: i will propse a patch that brings it to CURRENT, and we probably won't merge it, but it will be there just in case
14:43:23 <tosky> on the other side, switching to CURRENT at the beginning of cycle would allow maybe some fixes without having to change microversioning?
14:43:48 <tosky> or marking it as CURRENT triggers the need for proper microversioned changes even if it's not released yet?
14:44:09 <jeremyfreudberg> tosky: i'm fairly certain a CURRENT api has to be a stable api... but i'm not sure exactly how that works relative to release cycle
14:44:27 <tosky> need to dig
14:44:29 <tosky> ok
14:44:34 <tellesnobrega> I think inside the cycle we can still make changes without microversioning
14:44:36 <tosky> anyway, we have some required blockers
14:44:37 <tellesnobrega> for the first one
14:45:11 <jeremyfreudberg> tosky - is anything blocking, or is all just "very nice to have"?
14:45:24 <jeremyfreudberg> and by very nice, i mean, very very very nice
14:45:48 <tosky> I would say that the first iteration of API v2 should work in the same way as the current v1 from the point of view of the gates
14:45:58 <tosky> hence scenario tests (and probably tempest tests too)
14:46:40 <jeremyfreudberg> tosky: my dashboard patch (at least when it is completed) will give you a good idea of how saharaclient interaction has changed from apiv1.1->apiv2, and you can incorporate that into sahara-tests
14:46:59 <tellesnobrega> by same way you mean 1 to 1 feature wise?
14:47:00 <tosky> oki
14:47:18 <tosky> tellesnobrega: well, we should not have visible regressions
14:47:39 <tosky> of course we have something less (did we cut internal job binaries already?) and some features more
14:47:41 <tellesnobrega> no regressions for sure
14:47:46 <tellesnobrega> can we have more features?
14:47:49 <jeremyfreudberg> tosky: apiv2 has no JBI
14:48:15 <jeremyfreudberg> tosky: APIv2 new features include force delete, and decommission specific node
14:48:15 <tellesnobrega> by that I mean, boot_from_volume
14:48:21 <jeremyfreudberg> tosky: and yes, boot from volume
14:48:53 <tellesnobrega> I will need to work decommission on the dashboard but we have it on apiv2
14:49:07 <tosky> we already have more features: my baseline is that, apart from the changed API and the know removed feature, *at least* the operations available also previously should work
14:49:28 <jeremyfreudberg> tosky: yep
14:49:33 <tellesnobrega> totally agree
14:49:34 <tosky> and by "work" I mean "in a reproducible way on zuul"
14:49:35 <tosky> oki :)
14:49:39 <jeremyfreudberg> tellesnobrega: i am trying to get decomission node in dashboard done this week
14:49:59 <jeremyfreudberg> if i don't finish it, i would love your help. you will have to do boot from volume yourself
14:50:03 <tellesnobrega> jeremyfreudberg, I appreciate that
14:50:19 <tellesnobrega> boot from volume is done
14:50:25 <tellesnobrega> on dashboard as well
14:50:33 <tellesnobrega> it is only a check_box
14:51:05 <jeremyfreudberg> tellesnobrega: oh, right, but you will have to adapt into the apiv2 way of dashobard things, after my patch
14:51:19 <tellesnobrega> jeremyfreudberg, sure
14:51:48 <jeremyfreudberg> tellesnobrega: i also suggest that you split out the saharaclient boot from volume into 2 separate patches
14:51:51 <jeremyfreudberg> basic client, and then osc
14:52:00 <tellesnobrega> ok
14:52:00 <jeremyfreudberg> the basic client must be merged this cycle for dashboard to work
14:52:02 <tellesnobrega> I can do that
14:52:08 <jeremyfreudberg> thanks!
14:52:44 <tellesnobrega> do we have anything else on apiv2?
14:53:08 <jeremyfreudberg> yeah, the fake microversion thing -- but just leave your comments (if any) on the review -- we are running out of metting time
14:53:27 <tellesnobrega> sure, I will take a look this afternoon
14:54:26 <tellesnobrega> #topic Open Discussion
14:54:33 <tosky> two quick notes:
14:54:37 <tellesnobrega> we have about 5 minutes ish
14:54:43 <tosky> - please review zchkun's patch about code sharing in vanilla - I would, but I don't have an easy to use vanilla environment around
14:55:06 <tellesnobrega> tosky, I can do that as well
14:55:20 <tosky> - you may see patches from me in the not-far future about switching the test runner for sahara-dashboard from nose to django runner, as horizon (and most dashboards) did
14:55:25 <tosky> that's it
14:55:48 <tellesnobrega> we will keep an eye on it
14:56:03 <tellesnobrega> zchkun, let me know if you need help with the image for mapr6
14:56:23 <tosky> oh, and if you wonder why https://review.openstack.org/#/c/579114/ didn't merge - it's because it depends on https://review.openstack.org/#/c/571480/
14:56:48 <jeremyfreudberg> another one: tosky, since you seem to have a cloudera environment -- have you tried s3 there? if not, i will probably try it sometime in August, and call it a Bugfix
14:56:51 <jeremyfreudberg> if needed
14:57:15 <tosky> jeremyfreudberg: I didn't try it yet; I can probably, so I can see how job works
14:57:21 <tosky> a job works
14:58:02 <jeremyfreudberg> yep
14:58:50 <zchkun> tellesnobrega,  I am  building the image  and need to take some more time
14:58:54 <tellesnobrega> jeremyfreudberg, tosky I plan to test s3 as well nad I might need help, watch out for pings on it
14:59:04 <tellesnobrega> zchkun, cool, let me know if you need me
14:59:19 <zchkun> ok, thanks
14:59:34 <tellesnobrega> and we are out of time
14:59:44 <tellesnobrega> great meeting everyone
14:59:52 <tosky> my 3rd meeting in a row :D
15:00:00 <tosky> but yeah, good meeting
15:00:04 <tellesnobrega> you should get some rest
15:00:12 <jeremyfreudberg> yes, lots to do
15:00:12 <tellesnobrega> thanks everyone
15:00:16 <jeremyfreudberg> keep up the good work
15:00:17 <tellesnobrega> see you all next week
15:00:21 <tellesnobrega> yeap
15:00:23 <tellesnobrega> :)
15:00:27 <tellesnobrega> #endmeeting