17:01:19 <Kiall> #startmeeting Designate
17:01:20 <openstack> Meeting started Wed Feb  4 17:01:19 2015 UTC and is due to finish in 60 minutes.  The chair is Kiall. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:01:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:01:23 <openstack> The meeting name has been set to 'designate'
17:01:42 <Kiall> Heya - Who's about?
17:01:44 <timsim> o/
17:01:44 <vinod> o/
17:01:48 <rjrjr_> o/
17:02:15 <mugsie> o/
17:03:08 <Kiall> No  actions from last meet, so first topic straight to the first topic:
17:03:11 <Kiall> #topic Kilo Release Status (kiall - recurring)
17:03:17 <ekarlso> ello
17:03:17 <Kiall> #link https://launchpad.net/designate/+milestone/kilo-2
17:03:55 <Kiall> We have lots of bugs marked as in progress, meaning code reviews are up.. k2 sha1 needs to be given to Thierry tomorrow - The rest of my day is code reviews, if others can do the same hopefully we can land a few of those.
17:04:19 <Kiall> Anything that looks risky to land, shouldn't be landed :)
17:04:57 <Kiall> Anyone have any reviews they think really need to land?
17:05:06 <Kiall> I think rjrjr_'s cache/pools one probably needs to
17:05:26 <timsim> https://review.openstack.org/#/c/148386/ imo
17:05:50 <Kiall> Cool - betsy or mugsie will need to review or approve ^ one
17:06:20 <Kiall> Any others?
17:06:58 <Kiall> I'll take that as a no then ;)
17:07:05 <timsim> Yeah just rjrjr_ 's
17:07:13 <vinod> betsy is off today so mugsie needs to step in :-)
17:07:33 <Kiall> Hah - mugsie is out too, but attending the meet ;)
17:08:43 <Kiall> Okay - We'll see what we can land before tomorrow, let's move on.. I'll skip the next topic (Pools - Where are we? (kiall - recurring)) since we decided to remove that from the recurring agenda..
17:08:43 <mugsie> if people send me lists of revirews - i will see if i can grt them done this evemning
17:08:50 <Kiall> mugsie: will do..
17:09:06 <Kiall> #topic Summit Talks Submission (Deadline 2/9) (timsim)
17:09:23 <timsim> So we need to get our talk submissions in by next monday for Vancouver.
17:09:25 <timsim> #link https://www.openstack.org/summit/vancouver-2015/call-for-speakers/
17:09:43 <timsim> But we haven't really talked about what we want to do all that much, so I figured we should :)
17:10:01 <Kiall> Hah - Yea, do we know who's likely to be able to attend?
17:10:32 <rjrjr_> if i am presenting, i will be able to attend. :)
17:10:38 <timsim> Not really (from our point of view). Having an accepted talk would increase those chances.
17:11:43 <rjrjr_> designate installation workshop, pool manager, designate overview, designate agent - all sound like worthy discussions
17:11:48 <Kiall> Okay - Well, Should we try organize a meet on Friday then to decide and submit something?
17:12:07 <timsim> Sure. Maybe throw up an etherpad with ideas and rough abstracts before then?
17:12:16 <mugsie> +1
17:12:20 <rjrjr_> +1
17:12:28 <Kiall> timsim: want to take charge of that? ;)
17:12:37 <timsim> Kiall: Sure.
17:12:56 <timsim> #link https://etherpad.openstack.org/p/designate-vancouver-talks
17:13:09 <Kiall> #action timsim to start etherpad and mail everyone with a meet time for Friday :)
17:13:16 <Kiall> and people to fill ideas into ^ etherpad
17:14:07 <timsim> Cool. We can move on :)
17:14:19 <Kiall> Trying to find the tab with the agenda to do so ;)
17:14:35 <Kiall> #topic Next Sprint
17:14:50 <Kiall> timsim: you added this one I think?
17:14:52 <mugsie> docs
17:14:59 <Kiall> Yep
17:15:00 <Kiall> #topic https://etherpad.openstack.org/p/designate-documentation-sprint
17:15:02 <Kiall> lol
17:15:11 <Kiall> #topic Next Sprint
17:15:16 <Kiall> #link https://etherpad.openstack.org/p/designate-documentation-sprint
17:15:20 <timsim> Actually just left over from last week, just wanted to remind folks to add their ideas there.
17:15:20 <Kiall> That's better.
17:15:45 <Kiall> Ah, I see ... I've put all mine down, has everyone else?
17:16:04 <timsim> I haven't taken all the time I should have to have thought about it, but I've added a few.
17:16:17 <Kiall> And - everyone's free (US Morning, EU Afternoon) on Feb 13th to do it?
17:16:38 <rjrjr_> i'll be there.
17:16:40 <timsim> Yep
17:16:45 <vinod> I'' be there
17:16:55 <mugsie> yup
17:16:58 <Kiall> Cool - At the next meet, we should divvy up any areas listed out so people can think about them a little in advance.
17:17:19 <Kiall> But - It looks like only myself and timsim have added suggestions, I've sure we can find more :)
17:17:44 <timsim> Agreed.
17:18:27 <Kiall> rjrjr_ / mugsie / vinod - anything you think of, please add to the list before next weeks meeting then..
17:18:35 <vinod> sure
17:18:54 <Kiall> Okay .. Lets move on so..
17:18:55 <Kiall> #topic Bug triage (timsim)
17:19:23 <timsim> Just wondered if we should triage some of the bugs out there, there seem to be quite a few. If not, move on :P
17:19:47 <Kiall> Yea - The "untriaged-bot" list of bugs has been growing, and the list of stuff triaged but not fixed/in-progress is growing too..
17:20:17 <Kiall> I also think we have a large number of invalid bugs..
17:20:57 <timsim> Maybe at some point we get them together and go through them. Or action someone.
17:21:13 <Kiall> Any suggestions on how we can make tame the list more easily? Ideally, not a once off triage meet, something ongoing so we can get the list down and keep it down :)
17:22:23 <Kiall> Should we add a "Bug Triage" item to the agenda, and work through new bugs each week trying to find owners etc?
17:22:33 <vinod> how about a recurring agenda item for the irc and do it - ideally the last item on the agenda
17:22:36 <timsim> I think, ideally there would be some designated (lol pun) team of people that looked at and made some determination on new bugs with 2 or 3 days. If they weren't handled during the week we could go through the ones from the week during the meet
17:22:55 <timsim> s/with 2 or 3/within 2 or 3
17:23:07 <Kiall> I think the people on the meet are probably those same people :)
17:23:23 <timsim> Granted.
17:23:32 <timsim> As the project grows it might be a good habit.
17:24:03 <timsim> I like vinod's idea though.
17:24:05 <Kiall> ++ - But I think for now doing it once a week as a group in the meet works..
17:24:19 <Kiall> Okay - Anyone disagree?
17:24:30 <rjrjr_> sounds good to me!
17:24:36 <Kiall> #action kiall to put Weekly Bug Triage on agenda as a recurring item..
17:24:40 <Kiall> sorted :)
17:24:48 <timsim> Nice
17:25:11 <Kiall> #topic Open Discussion
17:25:45 <rjrjr_> over 1/3rd of our outstanding bugs are in "Fix Committed", so that's good.
17:26:11 <Kiall> So - Apologies for being so absent the last little while, myself and mugsie have had 3 days in the office since returning from the christmas break.. We're (almost) done with travel at this point.. And the next one shouldn't be as disruptive as the others.
17:27:10 <Kiall> rjrjr_: yep, it's the In Progress ones I'm worried about :) All the Fix Comitted will be moved to Fix Released tomorrow
17:27:18 <Kiall> once Thierry cut's the k2 release for us
17:27:54 <rjrjr_> a dozen in "In progress".  nice!
17:27:54 <Kiall> Anyone have any off-agenda topics to bring up?
17:28:19 <timsim> dvorak?
17:28:38 <dvorak> sure
17:29:25 <dvorak> I'm investingating deploying designate for our prod environment, and right now we have a cross-region galera cluster, but rabbit clusters that are local to each region
17:30:06 <dvorak> I'm wondering if it'd be a crazy thing to have local designate instances in each region share the database in order to be able to have domains available in multiple regions
17:30:26 <dvorak> or if that's going to explode in interesting ways
17:30:38 <Kiall> So, I'm assuming you're already OK with the latency introduced with cross-region galera cluster?
17:30:51 <dvorak> yeah, we use it for keystone and for horizon caching
17:31:01 <dvorak> we own the network between the datacenters, so that's managable
17:31:14 <dvorak> horizon session state I mean
17:31:20 <timsim> What kind of scale are we talking about?
17:31:48 <dvorak> we have separate, local galera clusters for other services that it doesn't make sense to replicate
17:31:56 <Kiall> Okay - Well, In theory if you have all the designate components deployed in each region, sharing a DB, that should work just fine assuming your OK with the synchnous replication lag... EXCEPT for 1 quirk..
17:32:04 <dvorak> timsim: scale in what dimension?
17:32:29 <timsim> How many zones/records are you expecting, and how many changes are you expecting to make?
17:33:23 <dvorak> honestly, it's all greenfield from a DNS standpoint right now, so it's hard to say, but I'd say less than low hundreds of tenants, probably relatively static records, other than instance spinup/down, which would probably again be hundreds to low thousands per day
17:33:29 <dvorak> that's all really guessing though
17:33:41 <Kiall> The domain table has a serial # column, and galera does opportunistic locking inside transactions, so 2 changes to one zone at each side can conflict - one will get dupmed out as a SQL Deadlock, which is different behaviour to what would happen with a single non-clustered MySQL..
17:34:06 <dvorak> Kiall: and in that case will the transaction get retried, or an error returned to the requester?
17:34:32 <Kiall> dvorak: Today - No, I have a review up to make it retry.. but it's got out of date and needs an update: https://review.openstack.org/#/c/134524/
17:34:38 <Kiall> Once it's merged to master I plan on backporting to Juno
17:35:54 <timsim> I feel like that can probably work, excluding where changes might be made to a zone in two places at once. Changes might be slow to propogate though, as you'd have two wait for a syncing mechanism to come through and get changes, rather than pushing them out right away (they'll go out right away in one DC, but the other will take longer)
17:36:01 <Kiall> A temporary workaround for that not being merged would be to use a single DB server as "master" for all the regions, the latency is larger again that way, but if the regions are close, it should be OK..
17:36:30 <dvorak> that'd probably be workable.  We already do that inside a given region anyway
17:36:32 <Kiall> (We actually use Galera for HP Cloud DNS, and even within a region only every write to a single cluster member at once..)
17:37:16 <Kiall> timsim: galera is synchronous replication, unlike standard mysql replication..
17:37:21 <dvorak> I suspect our primary use case for most updates will be instance launch, and if galera takes longer to replicatte than the instance takes to boot, then I guess that'd be very good problem, or a very bad problem
17:37:36 <Kiall> A transaction won't commit until the change has been propagated to all the DB servers in the cluster
17:37:36 <dvorak> for keystone we typically see replication <1-2 seconds
17:37:40 <timsim> Wait, Kiall with Juno how would you push those changes out to two separate dns deployments?
17:38:07 <dvorak> and I suspect the tokens are a lot bigger transactions than the designate ones would be
17:38:35 <Kiall> dvorak: Well, designate's DB schema has more "hot" points than Keystones.. Specially the domain serial number
17:38:41 <dvorak> Kiall: it has to be certified commitable by the other galera cluster members, but it doesn't have to be commited or visible yet
17:39:02 <Kiall> issuing a new token in Keystone is a simple INSERT, while adding a record in Designate is a INSERT and UPDATE
17:39:23 <dvorak> ie, they have the data, they've validated it won't fail due to conflict, but it isn't necessarily commited on all members when the client returns
17:39:33 <dvorak> Kiall: nod, I'm speaking strictly about the replication lag time
17:39:35 <Kiall> Ah - I didn't know that..
17:40:13 <dvorak> we have a monitoring metric for cross-region keystone replication, that's how we got into this
17:40:14 <Kiall> Anyway - The other option BTW - Is to deploy all of designate designate in 1 region, and onlt designate-sink in the other region - pointed at the first regions RMQ
17:40:34 <dvorak> we'll get a token in one region, then measure how long it takes to be valid in the other
17:40:46 <dvorak> that's not a bad option either, for initial deployment
17:41:06 <dvorak> ok, so sounds like there are some good options to explore
17:41:07 <Kiall> Yea, I think once retry on deadlock lands and is backported to Juno, the problem mostly goes away
17:41:22 <rjrjr_> kiall, designate sink will read from one RMQ and write to another?
17:41:24 <dvorak> Kiall: I assume that'd be before kilo releases?
17:41:25 <Kiall> At that points is really all about if your happy to deal with the replication lag
17:41:33 <Kiall> rjrjr_: DOOH
17:41:47 <rjrjr_> or are you talking about shovelling the traffic between RMQ?
17:41:47 <Kiall> dvorak: that last suggestion won't work, as rjrjr_ found out the hard way a while back.
17:41:54 <dvorak> I don't see the replication lag as an issue
17:42:20 <dvorak> I'm fine with it taking a few seconds for new records to be valid
17:42:56 <Kiall> Yea, for DNS a little bit of lag really isn't the end of the world
17:43:21 <dvorak> fair enough, I'm also ok with *trying* to do cross region DB across if need be also.  This is really at the exploratory stages.  We may not have a hard requirement for cross-site domains, but I suspect it would be desireable
17:43:38 <Kiall> dvorak: let us know how your testing works out :)
17:43:45 <Kiall> Okay - Any other off topic items?
17:43:51 <rjrjr_> dvorak, we are in the same boat.  rolling out next week!
17:44:11 <rjrjr_> we are going with regional databases to start, then galera after some testing.
17:44:48 <timsim> Does Friday at the time of this meeting, or an hour before work for everyone to plan vancouver talks?
17:45:22 <Kiall> timsim: yea, that works for me
17:45:53 <vinod> +1
17:46:29 <rjrjr_> which one are we agreeing to. :)
17:46:36 <dvorak> No idea if it'll be accepted, but I optimistically proposed a talk about our deployment that I haven't started :)
17:46:41 <Kiall> rjrjr_: either for me ;)
17:46:47 <timsim> Cool. I'll schedule it for 16:00 UTC
17:46:48 <Kiall> dvorak: oh nice :)
17:46:56 <timsim> 10 AM texas time.
17:47:11 <rjrjr_> sounds good.
17:47:18 <timsim> For two hours, and if we want to go an hour later, we'll do that :P
17:47:51 <timsim> Alright, I'm all good :)
17:48:10 <Kiall> Okay - Works for me.. :)
17:48:20 <Kiall> Anything else before we call it a day?
17:48:48 <Kiall> Reminder - Add docs bits to the docs sprint etherpad, and code review anything tagged k2 that you can.. tomorrow's the day
17:50:04 <Kiall> Okay - See you guys later, time for me to get on those code reviews :)
17:50:05 <Kiall> #endmeeting