20:00:06 <xgerman> #startmeeting octavia
20:00:07 <openstack> Meeting started Wed Mar 11 20:00:06 2015 UTC and is due to finish in 60 minutes.  The chair is xgerman. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:11 <openstack> The meeting name has been set to 'octavia'
20:00:16 <dougwig> o/
20:00:17 <xgerman> #chair blogan
20:00:18 <openstack> Current chairs: blogan xgerman
20:00:21 <Aish> 0/
20:00:26 <bharath> o/
20:00:38 <mwang2> o/
20:00:40 <johnsom> o/
20:00:43 <ajmiller> o/
20:00:43 <xgerman> Agenda: https://wiki.openstack.org/wiki/Octavia/Weekly_Meeting_Agenda#Meeting_2015-03-11
20:01:14 <xgerman> let's see if blogan is back from taco after daylight saving change
20:01:34 <jorgem> o/
20:01:36 <rm_work> o/
20:01:36 <fnaval> o.
20:01:39 <fnaval> o/
20:01:42 <ptoohill> o/
20:01:43 <TrevorV> o/
20:02:13 <rm_work> blogan is on PTO :P
20:02:20 <xgerman> towed?
20:02:23 <rm_work> but who knows, he might show up
20:02:25 <dougwig> rm_work: oh, you mean he's working from home?
20:02:32 <ptoohill> lol
20:02:35 <TrevorV> Yes dougwig that's the troof
20:02:40 <rm_work> yes, his last two days of PTO he did :)
20:02:57 <xgerman> #topic Announcements
20:03:24 <dougwig> if he's here, i can tell him that my car got towed this morning (bad starter)
20:03:38 <jorgem> that doesn't count
20:03:39 <ptoohill> lol
20:03:43 <xgerman> lol
20:03:51 <rm_work> dougwig: you didn't just fix it in your driveway? :P
20:03:54 <xgerman> jorgem +1
20:03:58 <rm_work> starter isn't too bed
20:04:01 <rm_work> *too bad
20:04:21 <rm_work> bet it's like $100 at Oreilly's
20:04:26 <xgerman> ok, so we had mark give us a demo of akanda rug yesterday and it looks pretty impressive
20:04:53 <xgerman> but he also promised us to give us a version to play around with... which hasn't happened...
20:05:08 <rm_work> yeah, if we decided to move to that, it would mean scrapping a huge portion of our current design for Octavia, right? basically all of the current VM interfaces?
20:05:42 <xgerman> yeah, I think it will be a big refactor
20:06:09 <xgerman> but from an operator perspective I liked the tooling for listing running vms, etc,
20:06:35 <xgerman> anyhow, I will reserver judegment to once I had a chance to play with it :-)
20:07:15 <xgerman> dougwig? any lb vendor perspective?
20:07:36 <johnsom> It sounded like there was still some work to be done before everything we would need is in akanda, but it didn't sound like a lot
20:07:39 <dougwig> i think we'd be nuts to rewrite all of that infrastructure ourselves.
20:08:08 <xgerman> +1
20:08:19 <dougwig> i mean, we can have simple drivers for nova, plumbing, and amphoras for demos, but when it comes to making them production ready, an akanda driver for those three functions could get us pretty far down the road.
20:08:26 <dougwig> that was my initial impression, anyway.
20:09:00 <ajmiller> +1
20:09:38 <dougwig> that is predicated on us getting bits and it all working, of course.
20:09:43 <sballe__> o/ sorry for being late
20:09:48 <ptoohill> So, we would probably need to invest time in Akanda to get it ready for our needs, unless we expect them to do that for us?
20:10:05 <jorgem> dougwig: correct, I want to get something working end to end asap
20:10:05 <xgerman> yeah, I think we will need to engage with them
20:10:26 <xgerman> jorgem +1 - I see akanda AFTER we have end to end
20:10:27 <sballe__> dougwig: +1
20:10:27 <jorgem> ptoohill: I hope it's the latter
20:10:34 <jorgem> xgerman: +2
20:10:37 <ptoohill> i dont think it will be jorgem
20:10:46 <sballe__> jorgem: +1
20:10:51 <ptoohill> and if it is, were at the mercy of their timeline
20:11:05 <ptoohill> which im not sure our needs are their top priority
20:11:37 <xgerman> +1
20:12:05 <sballe__> ptoohill: It is open source and I agree regarding the priority. We would need to see and assess its status before we deicde to go with it
20:12:45 <ptoohill> agreed, im just wondering if and how much of my time well be spending getting this new framework ready for things we already have planned out
20:13:02 <sballe__> based on the demo we only got to see what they wanted us to see. We need to dig deeper
20:13:04 <ptoohill> or already in flux
20:13:19 <xgerman> sballe__ +1 we need to get the code and assess
20:13:21 <ptoohill> agreed sballe__, just my initial thoughts
20:13:38 <sballe__> ptoohill: I agree
20:14:09 <xgerman> ptoohill we will be spending time getting our drivers production ready, too, so without the kanda code it's hard to assess
20:14:32 <xgerman> anyhow, I think we are on the same page... so moving on:
20:14:43 <xgerman> #topic Brief progress reports
20:14:49 <sballe__> xgerman: +1
20:15:25 <TrevorV> SSH Driver should be ready to review, I just still haven't tested it locally by running the code by hand
20:15:31 <TrevorV> That's what i'll be doing today/tomorrow
20:15:40 <johnsom> Continued progress on the controller worker.  I plan to post another patch set today that will add load balancer functionality
20:16:22 <xgerman> Agent API server works for most functions need to add SSL and some other minor stuff
20:16:24 <ptoohill> Im an Octavia slacker, all foxus has been in neutron-lbaas
20:16:26 <mwang2> continue working on  add config drive to nova compute driver and change code for health manager based on review comment
20:16:44 <johnsom> I was hoping blogan would be here.  I'm interested to know his timeline/plans for the network driver.
20:16:51 <sballe__> I have started work on Amphora REST driver
20:17:16 <sballe__> thanks for ptoohill TrevorV ajmiller for their help :-)]
20:17:18 <TrevorV> johnsom assume he's a lazy bumb until next Monday.
20:17:36 <TrevorV> ( sballe__ still not sure what I did, but you're welcome :D )
20:17:53 <sballe__> :-)
20:17:53 <ptoohill> Your presence TrevorV
20:18:31 <xgerman> #topic Discuss AmphoraDriver:
20:18:51 <xgerman> well, with ssh being ready I guess TrevorV won the race :-)
20:19:54 <xgerman> (my intention for this topic was to highlight we are close with the REST driver and could skip ssh but since we have it...)
20:20:50 <sballe__> I'll continue to work on the amphora driver we can always decide later
20:21:20 <TrevorV> Well xgerman I don't think having both would be a problem anyway, since we should potentially still prioritize the REST driver for reviews and such
20:21:27 <rm_work> I'm still liking the prospect of SSH driver for production deployments, after my last conversation with dougwig about it
20:21:28 <xgerman> our aim was always to go with REST but we felt ssh got us to the end-to-enfd quicker
20:21:33 <rm_work> so I wouldn't want to skip it regardless :P
20:22:03 <sballe__> rm_work: +1 I agree
20:22:04 <xgerman> rm_work any reason why ssh is better than REST?
20:22:09 <rm_work> a few
20:22:15 <xgerman> shoot:
20:22:18 <rm_work> dougwig: do you want to go through them, or should I? :P
20:22:48 <rm_work> 1) nothing additional running on the Amphora, so virtually zero overhead
20:22:57 <johnsom> Curious on this myself.
20:22:59 <rm_work> 2) SSH is as provably secure as anything can be
20:23:28 <rm_work> 3) No logic deployed to the Amphorae, so updates would be entirely service-side (better scaling for updates)
20:23:28 <ptoohill> lit the fire under rm_work :)
20:23:45 <dougwig> i like it because the amphora image becomes *download cloud image from canonical*.  it's there, adding keys is common in openstack, bash is a fine DSL for configuring haproxy.
20:24:01 <dougwig> no bugs in software we don't write.
20:24:21 <xgerman> well, we still need some agent to push stats/health
20:24:32 <sballe__> xgerman: +1
20:24:52 <rm_work> was stats a pull or a push?
20:24:54 <ptoohill> Other stats besides the ones maintianed by haproxy right
20:25:02 <ptoohill> theres a few types of stats i thought
20:25:27 <ptoohill> like instance stats, but think we can collect those else where?
20:25:50 <xgerman> even the ones maintained by haproxy need to get to the controller/ceilometer/...
20:26:07 <ptoohill> you can query for those
20:26:13 <dougwig> scp can copy over an agent or deb.  it becomes self-updating.
20:26:17 <ptoohill> its ones that are not set up to be tracked
20:26:28 <rm_work> dougwig speaks truth
20:26:41 <xgerman> yeah, he does :-)
20:26:49 <ptoohill> like cpu stuffs
20:27:17 <xgerman> well, I know people who sell aproduct which ssh's into each host and gathers those stats
20:27:20 <TrevorV> All those informations can be derived from the SSH connection
20:27:28 <TrevorV> through the SSH connection***
20:27:35 <rm_work> if the stats are "haproxy stats", then yeah those can be pulled through SSH
20:27:41 <xgerman> yep, but that doesn't scale very well
20:27:57 <sballe__> xgerman: +1
20:28:03 <ptoohill> thgeres others i thought we wanted to collect
20:28:06 <rm_work> I thought we'd decided to go with pull-stats too...
20:28:10 <ptoohill> not just haproxy
20:28:17 <rm_work> and actually, it should scale fine -- you split up the amphorae between different workers
20:28:29 <rm_work> scales pretty easily IMO?
20:28:38 <TrevorV> ptoohill like what?  Basic networking stats and hardware information?  That can still be retrieved through SSH connection, right?
20:28:50 <xgerman> yep, I am not questioning that
20:28:52 <ptoohill> sure static stats can
20:29:00 <rm_work> though there might still need to be SOME code that lives on the Amphora, like were we still planning to have a push-based heartbeat?
20:29:05 <ptoohill> but i thought we wanted to collect data of cpu usage/mem etc
20:29:19 <ptoohill> but im sure that could be done without an agent of sorts, maybe
20:29:20 <xgerman> yep, we want but you cna do that though SSH, too
20:29:21 <ptoohill> idk
20:29:33 <ptoohill> ah, very true
20:29:38 <ptoohill> just have it poll
20:29:38 <TrevorV> +1 xgerman that's what I was talking about
20:29:39 <rm_work> well, the point being if it's push based, it needs to have a daemon running ON the amphora
20:29:40 <johnsom> The code that is in the repo was for a push model.
20:29:53 <rm_work> so that can do whatever
20:30:01 <xgerman> yeah, even UDP because TCP was too heavy :-)
20:30:08 <rm_work> heh
20:30:13 <rm_work> yeah, barclaac's code
20:30:17 <rm_work> which is fine to remain as-is
20:30:31 <rm_work> we're talking about the controller->amphora direction
20:30:35 <rm_work> like config updates, etc
20:30:36 <bedis> (morning)
20:30:41 <rm_work> and, I thought, stats pulling
20:30:43 <johnsom> The current design puts a lot of weight on the agent monitoring the haproxy instances
20:31:20 <dougwig> i'd wager that the "health check" in 0.5 could be just keeping an ssh connection hot and noticing when it drops.
20:31:29 <xgerman> my main worry is that pushing out code with ssh, etc. makes us into some poor man's ansible/chef/etc.
20:31:51 <bedis> note: it is possible to exports stats through a TCP socket (ciphered)
20:32:00 <bedis> or CSV through the stats page
20:32:21 <xgerman> bedi, neat!
20:32:57 <bedis> http://demo.haproxy.org/
20:32:59 <bedis> and
20:33:00 <bedis> http://demo.haproxy.org/;csv
20:33:06 <bedis> just an example
20:33:31 <bedis> you can even export stats for a single frontend or backend:
20:33:32 <bedis> http://demo.haproxy.org/;csv?scope=www
20:33:34 <xgerman> I can see a short term (0.5) need for ssh but long term I would think REST is better (since we can use golden images; make sure nobody ssh i and turns it into a botnet)
20:33:34 <bedis> :)
20:34:15 <rm_work> well, I assume SSH would only be bound to the management interface
20:34:33 <rm_work> but I don't know if this conversation is precisely on-topic/necessary right now
20:34:48 <rm_work> where in the meeting were we? :P
20:35:19 <TrevorV> One thing to note, even when considering the REST api on the amphora, its still a pull model to gather info/details about the amphora, am I wrong?
20:35:57 <ptoohill> the api client will make the request to gather that data, so yes, its still a 'pull' model
20:36:04 <xgerman> TrevorV the plan is to write an gent which has a REST API and some way to push stars
20:36:13 <xgerman> stars=stats
20:36:18 <xgerman> but they can be two pices
20:36:21 <ptoohill> unless its planned to do something else :P
20:37:05 <xgerman> ptoohill the stats argument was to defeat the assertion we can use stock ubuntu/don't need to install anything
20:37:13 <rm_work> well, depends on what is push and what is pull
20:37:27 <rm_work> and yeah, we still need to deploy SOME agent on the amphora
20:37:31 <rm_work> but it can be much more minimal;
20:37:33 <bedis> actually it depends who does it :)
20:37:42 <rm_work> and it could be *deployed* via SSH, making updates much easier
20:38:26 <xgerman> and that ties into how operators run their world - if it's golden images; or configured at ruuntime
20:38:36 <rm_work> true
20:39:27 <xgerman> (and I know my security people running an ssh system will only fly with waivers)
20:39:37 <rm_work> well
20:39:47 <xgerman> and heavy app armor
20:39:50 <rm_work> ask your security people how much more comfortable they are with a custom REST-based solution you wrote
20:40:07 <TrevorV> xgerman I don't remember us every automatically pushing stats from the amphora.  I knew of the heartbeat, but that's the only thing I remember as a communication out from the amphora without a request being made
20:40:08 <rm_work> seems there is much more likleihood of that being insecure than SSH being insecure
20:40:17 <TrevorV> ever automatically***
20:40:32 <rm_work> hence: <rm_work>	2) SSH is as provably secure as anything can be
20:41:36 <ptoohill> Not sure what were argining anymore, but sounds like we should keep/use both ssh and rest to satisfy everyones needs
20:41:44 <xgerman> yep
20:41:56 <ptoohill> and do a push/pull model and make it configurable ;)
20:41:58 <TrevorV> That's a given, but now I'm confused about the priorities of the amphora.
20:42:08 <xgerman> TrevorV: http://octavia.io/review/master/design/version0.5/component-design.html scroll down to "Some notes on Controller <-> Amphorae communication"
20:42:38 <xgerman> yeah, priorities are confusing
20:43:33 <xgerman> I think that horse is glue
20:43:43 <xgerman> #topic Open Discussion
20:44:21 * dougwig used to eat paste as a kid.
20:44:38 <bedis> dougwig.rb !!!!
20:45:00 <rm_work> "And the following would happen over TCP: * haproxy / tls certificate configuration changes"
20:45:06 <rm_work> that is not aphora->controller
20:45:09 <rm_work> that's the other way around
20:45:25 <rm_work> so no, it wouldn't have anything to do with a REST API controller-side
20:45:59 <rm_work> the only things I see listed in that section that are relevant are things that would be handled by barclaac's heartbeat code, or by the controller->amphora SSH connection driver
20:47:01 <xgerman> yep, I just pointed that out since TrevorV didn't recall pushing stats amphora -> Controller
20:47:18 <xgerman> but both ssh and REST are covered by the spec
20:47:43 <xgerman> so as ptoohill said different operators have different needs
20:48:12 <xgerman> ...
20:48:15 <rm_work> OH yeah I see, the bullets got messed up
20:48:31 * ptoohill Likes goldfishes
20:49:01 <rm_work> this stuff is really not well worded
20:49:01 <rm_work> T_T
20:50:00 <xgerman> agreed
20:50:44 <rm_work> TrevorV pointed out what I had forgotten -- I think a lot of this stuff is supposed to be sent over *as part of* the heartbeats from barclaac's daemon
20:50:46 <rm_work> over UDP
20:51:09 <rm_work> at least, specifically, “edge” alert notifications (change in status) from the amphora to the controller
20:51:24 <rm_work> which i assume means "member node UP/DOWN notifications", since haproxy is tracking those
20:51:29 <rm_work> and that's how I assume we'd be notified
20:51:39 <TrevorV> Which makes his code much more intimately connected to the API driver or SSH driver or whatever to get those "agents" set up appropriately.
20:51:42 <xgerman> yeah, that's what we use the health manager for
20:52:01 <xgerman> (recieving those upodates)
20:52:20 <xgerman> TrevorV +1
20:52:54 <TrevorV> rm_work brought up that the base image will have the agent that sends those UDP connections defined.
20:53:04 <rm_work> the health-manager listens for UDP messages?
20:53:06 <TrevorV> UDP informations***
20:53:12 <rm_work> I guess that makes sense, but i hadn't seen that code yet
20:53:23 <xgerman> no, driver does; health manager has logic to persist it into DB
20:53:27 <rm_work> ah
20:53:30 <rm_work> which driver does?
20:53:36 <rm_work> the Amphora Driver?
20:53:37 <xgerman> amphora_driver
20:53:47 <rm_work> if so, that'd be shared code between both the REST driver and the SSH driver
20:53:59 <xgerman> yep
20:55:04 <xgerman> they will be pretty similar the REST server doesn't do muh but some with open to move files and subprocess to start/stop things
20:55:31 <rm_work> yeah I'd hope the listening on UDP part would be above the SSH/REST layer
20:55:51 <xgerman> rm_work, will be an extra thread...
20:55:54 <rm_work> ah right
20:56:05 <rm_work> so it's not really part of the Amphora Driver, it seems
20:56:08 <rm_work> at least, as we have it
20:56:10 <xgerman> it is
20:56:24 <rm_work> because the amphora_driver is a specification for an interface to talk TO the amphora
20:56:28 <rm_work> and is not a thread
20:56:40 <rm_work> it's just code that is run when there are updates to be sent
20:56:59 <xgerman> + it also fires up a thread to lisetn for the stats
20:57:00 <rm_work> there needs to be a different class like amphora_listener to actually have an open socket for that, i'd imagine
20:57:03 <rm_work> err
20:57:09 <rm_work> except it fires up a thread when?
20:57:12 <rm_work> not per-update
20:57:36 <rm_work> it doesn't really make sense for a class that is instantiated/used as a result of queue actions
20:57:43 <xgerman> no, I think we will amend the spec so you cna say start_thread()
20:57:47 <rm_work> to spin up a long-term listening thread
20:58:06 <xgerman> since that thread needs to run as part of the health_manager BUT NOT the deploy worker
20:58:24 <rm_work> I mean, technically that would work, but it seems like a bad design decision to just shove it in with the rest of the outbound amphora communication stuff
20:58:47 <rm_work> since that thread and the rest of the amp driver will never share any code or call each other
20:58:50 <xgerman> well, I would hate to need TWO drivers to talk with Amphoras
20:58:58 <rm_work> it'll only be calling health-manager persistence methods
20:59:01 <rm_work> err, no
20:59:15 <rm_work> you'd need one class that is responsible for opening a socket and acting as a long-term listener
20:59:28 <xgerman> ok, got it two classes; one driver
20:59:30 <rm_work> and one class that is part of a lib that is run for on-queue-action updates
20:59:55 <rm_work> well, it also kind of makes sense to split it out, since it's not relevant whether the driver is SSH or REST
20:59:59 <johnsom> Yeah, the point of having it in the driver was for non-amphora deployments.
21:00:01 <rm_work> unless it lives at a layer ABOVE that part
21:00:18 <rm_work> so if it lives in the Amphora Driver code, we need to have an extra layer
21:00:34 <rm_work> because we don't want to be duplicating that code between the SSH and REST implementations
21:00:52 <rm_work> that should be easy enough to figure out though, since hopefully that makes sense?
21:01:14 <xgerman> yep, we also can share the code which makes haproxy.cfg files
21:01:23 <rm_work> yeah probably
21:01:46 <xgerman> anyhow, time's out
21:01:50 <xgerman> #endmeeting