21:00:20 <devkulkarni> #startmeeting Solum Team Meeting
21:00:21 <openstack> Meeting started Tue Feb 17 21:00:20 2015 UTC and is due to finish in 60 minutes.  The chair is devkulkarni. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:25 <openstack> The meeting name has been set to 'solum_team_meeting'
21:00:38 <devkulkarni> #link https://wiki.openstack.org/wiki/Meetings/Solum#Agenda_for_2015-02-17_2100_UTC
21:00:48 <devkulkarni> #topic Roll Call
21:00:50 <datsun180b> ed cranford
21:00:56 <devkulkarni> devdatta kulkarni
21:00:58 <muralia> murali allada
21:01:00 <phiche1> philip cheong
21:01:08 <devkulkarni> hi ed, murali, philip
21:01:23 <devkulkarni> please take a moment to look through our agenda for today
21:01:30 <devkulkarni> #link https://wiki.openstack.org/wiki/Meetings/Solum#Agenda_for_2015-02-17_2100_UTC
21:02:07 <phiche1> if only it were so simple on hotel wifi :(
21:02:31 <muralia> oh cant you access it?
21:02:32 <devkulkarni> :( .. don't worry.. I will paste each item here
21:03:02 <devkulkarni> lets wait for another minute to see if anyone else joins us
21:03:31 <devkulkarni> ok, lets proceed
21:03:35 <phiche1> yay it loaded finally
21:03:37 <phiche1> just slow
21:03:41 <devkulkarni> #topic Announcements
21:04:11 <devkulkarni> so recently several new features got added/merged
21:04:23 <devkulkarni> I wanted to give an update on those today
21:04:31 <devkulkarni> (muralia): Creation and usage of custom language packs
21:04:41 <devkulkarni> muralia: floor is yours
21:04:53 <muralia> dev, should we wait until next week when we have more people here?
21:05:01 <muralia> or should we continue?
21:05:06 <devkulkarni> muralia: good point
21:05:11 <devkulkarni> we can do this again too
21:05:19 <devkulkarni> we do have few folks today
21:05:20 <muralia> sure. i'll give a brief intro
21:05:27 <datsun180b> it's going on the record either way
21:05:27 <devkulkarni> so not a bad idea to give brief updates
21:05:40 <devkulkarni> datsun180b: right
21:05:48 <devkulkarni> muralia: cool.
21:06:01 <muralia> so, one new feature i've worked on this month is the custom languagepack feature.
21:06:35 <muralia> basically, this deals with the definition of the run time environment for solums users apps
21:07:15 <muralia> as an example, if i want to create a python web app which talks to a mysql instance
21:07:45 <gpilz> a "stack" as heroku calls them
21:07:48 <muralia> what i require is the python stack and a lysql ibrary to talk to mysql
21:07:53 <muralia> *mysql
21:08:03 <muralia> this requires us to create a custom language pack.
21:08:16 <devkulkarni> gpilz: our notion of a stack is basically a docker image that has required dependencies
21:08:20 <muralia> all of the code for this feature has been merged as of today, so you can start using it
21:08:46 <devkulkarni> so muralia, in nutshell, how does it work?
21:08:57 <devkulkarni> do we create a Docker image and then use it in the plan file?
21:09:06 <muralia> so you need to use the cli to create a languagepack first.
21:09:07 <devkulkarni> to build the application DU?
21:09:16 <muralia> the command for this is
21:09:32 <muralia> solum languagepack build <git_repo> <metadata_file>
21:09:42 <james_li> don't think we can deploy a DU directly
21:10:06 <muralia> james_li: yes, we create the languagepack first
21:10:26 <muralia> then include it in the planfile, just a uuid ref, when we create the app
21:10:41 <muralia> solum languagepack build <git_repo> <metadata_file>
21:11:06 <devkulkarni> I see. so instead of "auto" that we currently specify, we will specify the uuid of the built language pack?
21:11:08 <muralia> so the gitrepo we provide needs to have a Dockerfile. which defines what we need in the run time
21:11:22 <james_li> devkulkarni: or name i thiknk
21:11:34 <muralia> let me paste a sample planfile here for us to look at. yes, james or name
21:11:53 <muralia> https://github.com/murali44/python-mysql/blob/master/planfile.yaml
21:12:47 <muralia> see the languagepack field in the planfile. this is the uuid of a LP i've created in my own environment. this is how we define a LP to be used, during app creation
21:13:02 <devkulkarni> cool
21:13:10 <muralia> now lets see what an LP really is
21:13:12 <muralia> https://github.com/murali44/solum-languagepack-python
21:13:35 <muralia> is a sample python languagepack that i created. it has a dockerfile
21:13:55 <muralia> if you look in the dockerfile, it just include the libraries i'll need for my app
21:14:20 <muralia> notice the existence of a build.sh file which is included in the docker file.
21:14:42 <muralia> that script will be called by Solum during the build phase of the CI workflow
21:15:07 <muralia> for python the build phase just installs pip libraries.
21:15:11 <devkulkarni> you mean, the build-app script will call this build.sh?
21:15:27 <muralia> yes
21:15:40 <devkulkarni> muralia: one question
21:15:45 <muralia> yes
21:15:53 <devkulkarni> in the plan file, you have unittest_cmd attribute
21:15:57 <devkulkarni> what is the purpose of that?
21:16:08 <devkulkarni> similary, I see there is run_cmd as well
21:16:27 <muralia> i was going to get to it after finishing this topic, otherwise we'll be all over the place.
21:16:28 <devkulkarni> could you talk about those? are they related to custom language packs?
21:16:33 <devkulkarni> oh okay
21:16:35 <muralia> let me finish this and come to that
21:16:38 <devkulkarni> sorry, continue :)
21:17:11 <muralia> so basically, i;ve introduced the build.sh file in the languagepack because different things need to happen for different types of apps
21:17:49 <muralia> got python it is just installing pip libraries, but we can imagine a java app requiring a 'maven build' command being called to create a war file
21:18:05 <muralia> of a c++ app needs to be built to create an exe
21:18:05 <devkulkarni> sure
21:18:26 <muralia> so language specific build steps will go into the build.sh file.
21:18:30 <gpilz> just to be clear - the users app will run in the docker container created from the custom language pack?
21:18:41 <muralia> exactly
21:19:12 <muralia> the solum CI workflow will look for the build.sh file and invoke it
21:19:32 <devkulkarni> so muralia, you are saying that build.sh needs to be provided by custom language pack author
21:19:39 <muralia> yes
21:19:43 <devkulkarni> i guess makes sense
21:20:07 <devkulkarni> do you have another example of a custom language pack?
21:20:26 <muralia> james_li might have a java sample.
21:20:49 <muralia> https://github.com/jamesyli/solum-lp-java
21:20:50 <mkam> https://github.com/jamesyli/solum-lp-java
21:20:52 <muralia> found it :)
21:21:19 <devkulkarni> james_li: ping
21:21:47 <devkulkarni> in build.sh for the java lang pack, you have a comment to copy the war file
21:21:59 <devkulkarni> is that step missing, or is it not needed?
21:22:25 <devkulkarni> thanks mkam, muralia for the link
21:22:52 <devkulkarni> muralia: please continue, while we wait to hear from james_l
21:23:21 <muralia> so the command solum languagepack build talks the repo url
21:23:29 <muralia> takes
21:23:36 <muralia> and gives us a uuid for the LP
21:23:56 <devkulkarni> where do you store the built LP?
21:24:08 <muralia> next thing to talk about it the test_cmd and the run_cmd in the planfile.
21:24:15 <devkulkarni> ok
21:24:53 <muralia> the test_cmd executes unit tests and this is specified by the user
21:25:15 <muralia> because we dont want to bind users to a particular tool, for example.
21:25:32 <datsun180b> tox, for example
21:25:45 <devkulkarni> so the exact command to run should be specified in unittest_cmd?
21:25:55 <muralia> in the CI flow, solum checks for a test-requirements.txt file in the app repo and install those for the user.
21:26:13 <muralia> yes, once the requirements are installed, the cmd is executed.
21:26:35 <devkulkarni> whose responsibility is it to ensure that the required tools (pep8, tox) etc. are available for the app?
21:26:56 <devkulkarni> is it user's responsibility to install them as part of custom language pack creation?
21:27:08 <muralia> like i mentioned, solum looks for a test-requirement.txt file in the app repo
21:27:39 <akshayc> hi
21:27:44 <devkulkarni> right, but there need to be available 'tox' on the runtime environment
21:27:47 <devkulkarni> hi akshayc
21:28:08 <devkulkarni> muralia is getting us up to speed on the custom language pack feature
21:28:17 <akshayc> ok…. cool….
21:28:17 <muralia> yes, all OS level dependencies are defined in the LP
21:28:26 <devkulkarni> muralia: got it
21:28:41 <muralia> so if you want to use TOX, it needs to be installed on the LP
21:28:49 <muralia> note that before this happens. solum first downloads the docker image for the LP defined in the planfile.
21:28:57 <devkulkarni> ok, makes sense
21:29:07 <muralia> builds a new on top of the LP
21:29:15 <gpilz> seems like we could build a common, base docker image
21:29:21 <muralia> and runs unit tests in it
21:29:27 <gpilz> that had stuff like tox, etc. already installed
21:29:35 <devkulkarni> gpilz: yes, that makes sense
21:29:59 <devkulkarni> we could have custom solum images that have common packages/libraries on them
21:30:08 <muralia> yes, gpliz, the next step is to figure out a stratergy to create vendor created lp for common usage and make it easy for users to reuse them
21:30:09 <devkulkarni> we could publish them to docker hub
21:31:01 <devkulkarni> muralia: quick question..
21:31:07 <muralia> yes
21:31:12 <devkulkarni> where are the custom LPs and DUs stored?
21:31:38 <muralia> once the unit tests pass, we start from the LP again
21:31:48 <muralia> this time we check for a requirements.txt file
21:32:12 <akshayc> ok
21:32:15 <muralia> install those libraries, inject the user code
21:32:31 <muralia> and include the run_cmd provided in the planfile as our entry point
21:32:39 <muralia> this now become our DU
21:32:44 <muralia> which can be deployed.
21:32:59 <devkulkarni> entry point, as in the entry point on the docker image (DU)?
21:33:04 <muralia> yes
21:33:07 <devkulkarni> ok
21:33:23 <muralia> so when we deploy the DU, we do "docker run image"
21:33:42 <muralia> the DU is now stored.
21:33:55 <muralia> Dev, to answer your question
21:34:11 <muralia> i've included multiple options for storing LP's and DU's
21:34:25 <muralia> Docker_registry, glance and swift
21:34:47 <devkulkarni> oh cool
21:34:48 <muralia> this is to address different requirements for different vendors
21:35:01 <muralia> the openstack way is to use glance to store images
21:35:06 <devkulkarni> right
21:35:21 <muralia> however, to make use of dockers layering feature
21:35:27 <muralia> we need to use the docker registry
21:35:35 <muralia> which is why i've include that too
21:36:03 <muralia> note that this requires a vendor to now run and operate a docker registry
21:36:14 <muralia> or use the public docker hub
21:36:22 <devkulkarni> so, LPs and DUs that get stored on docker_registry, are they tenant specific?
21:36:42 <devkulkarni> I am guessing that for glance and swift, you are storing them per tenant
21:36:45 <devkulkarni> is that correct?
21:36:53 <muralia> no they are not, if vendors need that feature, they need to use swift or glance as the backend
21:37:09 <devkulkarni> ok
21:37:12 <muralia> docker registry is not a multi tenant backend yet
21:37:44 <muralia> theres a new flag called image_storage in  solum.conf
21:37:57 <muralia> you can set that to one of the 3 options ive mentioned
21:38:12 <devkulkarni> you mean, docker_registry, glance, swift ?
21:38:16 <muralia> yes
21:38:20 <devkulkarni> cool
21:38:49 <muralia> if you choose glance or swift, solum will talk to keystone, impersonate the tenant and store LPs and DUs in the tenants account
21:39:21 <muralia> so the users code and DU is kept private at all times
21:39:42 <devkulkarni> muralia: impersonation will be only required for theusecase when there is a webhook trigger sent to solum?
21:39:43 <devkulkarni> right?
21:40:04 <muralia> thats right
21:40:14 <devkulkarni> for regular flow (where users are directly interacting as them) there is no impersonation
21:40:15 <devkulkarni> cool
21:40:34 <muralia> im going to update the wiki with instructions and links to sample LPs and sample apps soon
21:40:44 <devkulkarni> muralia: that would be really helpful
21:40:50 <muralia> cool
21:40:57 <devkulkarni> muralia: thanks a lot for the update!!
21:41:06 <devkulkarni> we will look forward to the updated instructions
21:41:15 <muralia> sure. if anyone has any questions, ping me on irc
21:41:15 <akshayc> nice
21:41:33 <devkulkarni> we are still in the Announcement topic today :)
21:41:46 <akshayc> is all the code for building is stored under contrib right?
21:41:51 <devkulkarni> we could continue with the rest of the announcements, or we could keep them for next week
21:42:01 <devkulkarni> akshayc: that is correct
21:42:09 <muralia> yes
21:42:10 <devkulkarni> contrib/lp-cedarish/build-*
21:42:30 <devkulkarni> what do you all prefer? we could go directly to Task review or to open discussion
21:42:32 <akshayc> ok…. and we currently support
21:42:49 <akshayc> no just wondering the correct location
21:43:01 <devkulkarni> akshayc: yes, that is the correct location
21:43:10 <datsun180b> i vote proceeed to tasks
21:43:19 <devkulkarni> datsun180b: ok, lets do that then
21:43:28 <akshayc> because i am trying to rework the building scripts
21:43:47 <devkulkarni> akshayc: yes, i believe you are continuing from where ravips left off
21:43:51 <devkulkarni> is that correct?
21:43:54 <akshayc> ya….
21:44:03 <devkulkarni> cool
21:44:12 <devkulkarni> lets circle back on that in the open discussion
21:44:23 <devkulkarni> lets quickly go through some of the tasks that we have for today
21:44:24 <akshayc> ok
21:44:34 <devkulkarni> #topic Task Review
21:44:43 <devkulkarni> (muralia): Remove Image creation API. Move image creation functionality as a resource under rest of the solum API tree
21:44:54 <devkulkarni> muralia: mind getting us up to speed on this one?
21:44:59 <muralia> yes
21:45:25 <muralia> while working on the LP feature, i realized we have two api endpoint in solum
21:45:42 <muralia> one is the solum-api which is the endpoint we talk to for creating and deploy apps
21:45:47 <muralia> the other is a builder-api
21:45:56 <muralia> which deals with image creation.
21:46:11 <devkulkarni> right
21:46:34 <muralia> when we started working on this project, we thought we would need a separate api for creating images and a different one for deploying these images
21:46:57 <muralia> we've changed our approach since then and are using the solum-api for creating and deploying apps.
21:47:35 <muralia> the LP creation however happens by talkign to the builder-api
21:48:11 <muralia> i chose to do that because of the way the LP resource was designed in the main solum-api
21:48:44 <muralia> the LP resource was just a metadata store, which doesnt even write to the solum db. it just stores metadata in glance
21:49:37 <devkulkarni> okay..
21:49:43 <muralia> this doesnt make sense for solum anymore. so I'm propose consolidating our two api's into just one. basically, i'll change the LP resource to support all the LP features i described earlier
21:50:11 <muralia> there is just no need to two separate apis, 1 dealing with apps and 1 dealing with LPs.
21:50:24 <devkulkarni> so these will be: CRUD on LPs
21:50:31 <muralia> yes
21:50:57 <mkam> +1 for a single api
21:50:58 <devkulkarni> where is the LP related metadata being stored currently?
21:51:01 <muralia> this change will not affect an end user.
21:51:24 <muralia> its just moving where the functionality lives. and consolidating 2 endpoints into 1
21:51:44 <devkulkarni> seems fine to me
21:52:27 <devkulkarni> datsun180b, gpilz, phiche1, akshayc: any thoughts?
21:52:28 <muralia> in my LP feature work, im storing the metadata in the solum db now. this will make it easy for me to implement features to discover and search for LPs
21:52:29 <gpilz> makes sense
21:52:41 <datsun180b> more simpler is more betterer
21:52:51 <devkulkarni> muralia: I see
21:53:00 <devkulkarni> thanks datsun180b, gpilz
21:53:08 <devkulkarni> should we take a agreed on this?
21:53:18 <gpilz> yes
21:53:20 <datsun180b> yes
21:53:21 <akshayc> yes
21:53:25 <devkulkarni> thanks!
21:53:57 <muralia> cool. thanks. i'll get started on that
21:54:16 <devkulkarni> #agreed We will consolidate our endpoints under a single API endpoint. Specifically, we will get rid of solum-builder-api endpoint and move language pack creation functionality under solum-api
21:54:36 <devkulkarni> we have only few minutes remaining today.
21:54:57 <devkulkarni> so I am going to skip the rest of the task reviews, and open up discussion
21:55:03 <devkulkarni> #topic open discussion
21:55:26 <devkulkarni> akshayc: floor is yours
21:55:31 <akshayc> ok
21:55:41 <akshayc> i have trying to move from bash to python
21:55:52 <akshayc> i was looking for some libraries
21:56:01 <akshayc> i know there is docker-py
21:56:06 <akshayc> for docker functions
21:56:18 <devkulkarni> is docker-py in global-requirements?
21:56:20 <akshayc> do we have any specific version to use
21:56:32 <devkulkarni> if not, our gates won't pass
21:56:37 <akshayc> not sure…. it is in the newer version
21:56:51 <akshayc> i will not shift the current working
21:57:00 <akshayc> i will work in parallel
21:57:05 <devkulkarni> got it.
21:57:16 <akshayc> when the work is stable, we can switch
21:57:31 <devkulkarni> but we should go docker-py route in your parallel stream only if eventually we would be able to use that in upstream solum
21:57:54 <devkulkarni> and that is why I was asking if we know whether docker-py has been included in global-requirements yet?
21:58:04 <devkulkarni> if we don't know, we should find that out
21:58:09 <datsun180b> nope
21:58:17 <datsun180b> https://github.com/openstack/requirements/blob/master/global-requirements.txt
21:58:49 <akshayc> ya…. its not there….
21:58:52 <datsun180b> keyworded bookmarks C:
21:59:00 <devkulkarni> datsun180b: but, wasn't there an issue in our vagrant around nova where it was using dockery-py? so it must have been available, no?
21:59:08 <akshayc> the other option is to write shell commands in python
21:59:19 <datsun180b> nova-docker
21:59:21 <devkulkarni> fabric
21:59:31 <datsun180b> https://github.com/stackforge/nova-docker/blob/master/requirements.txt
21:59:36 <devkulkarni> sorry, nova-docker
21:59:49 <akshayc> yep… nova-docker is using it….
22:00:15 <devkulkarni> okay, so if nova-docker is using it, we could use it as well
22:00:24 <datsun180b> but https://github.com/ed-/nova-docker/tree/solum-pin is not
22:00:39 <devkulkarni> hmm..
22:00:44 <devkulkarni> okay, we are out of time
22:00:47 <datsun180b> compatibility issues mainly
22:00:52 <akshayc> yep
22:00:56 <devkulkarni> akshayc: lets continue in solum
22:01:05 <devkulkarni> thanks muralia for the updates today
22:01:09 <akshayc> ok
22:01:10 <muralia> sure
22:01:15 <devkulkarni> we will continue with more updates in the next meeting
22:01:20 <devkulkarni> #endmeeting