17:05:10 #startmeeting designate 17:05:11 Meeting started Wed May 28 17:05:10 2014 UTC and is due to finish in 60 minutes. The chair is Kiall. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:05:12 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:05:15 The meeting name has been set to 'designate' 17:05:23 Hey all! Who's about? 17:05:29 here 17:05:30 o\ 17:05:33 heya 17:05:34 o/ 17:05:38 Sorry for starting late, head deep in git hell ;) 17:05:40 hi 17:05:42 hi 17:05:45 Kiall: I'm here to talk about anything Joe added to the agenda 17:06:01 hey there 17:06:36 #topic Review action items from last week 17:06:58 First two are richm, he's not here at the moment, so we'll come back to them 17:07:14 eankutse: open blueprint for regionalized mdns servers / to file blueprint around region-aware handing of NOTIFY's and AXFR's 17:07:25 done 17:07:34 Oh - Cool, I must have missed them 17:07:37 Links? 17:07:39 will post link later 17:07:43 K 17:07:59 last one was for me, re-add multi-region mdns to agenda once ^ is done.. 17:08:15 It'll be on the agenda next week, once we have a chance to review the BP's 17:08:34 #topic mDNS Questions 17:08:54 Okay - So there's a bunch of questions re mDNS here: https://wiki.openstack.org/wiki/Meetings/Designate#Agenda 17:09:05 #Link https://blueprints.launchpad.net/designate/+spec/mdns-designate-mdns-region-awareness 17:09:12 eankutse, I'm guessing they were you? 17:09:15 those are probably mostly my fault 17:09:30 jbratton: oh, fair enough ;) 17:09:40 jbratton: good items :-) 17:09:45 Okay .. So, first up.. "How does a server start up from scratch look like?" 17:10:08 I made submission for first cut of service code 17:10:14 tho build is failing 17:10:36 yeah, does it do zone transfers for every zone from scratch? or if not, how does it know what was changed while it was down? 17:10:37 service starts up like all other designate services bt 17:10:41 but 17:10:54 also sits on tcp/udp port 17:11:06 from where it will talk to name servers 17:11:15 jbratton: so, I think there's probably 2 different things to handle in this.. Brand new/replacement server, and a server that was down for maintainance for short period 17:11:27 yeah, true, I see that was broken up in to two points 17:11:37 Oh - I missed that :) 17:11:50 so I guess the first thing is.. how does it get the list of zones for the initial run 17:11:59 building out the config file from them 17:12:15 The brand new/replacement server scenario? 17:12:19 Actually, these are issues that I am thinking through currently and would be updating the blueprint as I go. I welcome ideas on these as well. 17:12:21 yeah 17:12:26 Shouldn't that just be a lookup in the database for the respective tags. 17:12:46 I'm talking about from an operational standpoint.. I'm a server admin :) 17:13:50 jbratton: would you clarify a little bi? 17:14:02 so miniDNS is up 17:14:09 for the first time 17:14:15 sure, so I run a BIND nameserver, I start BIND, now I need to get a list of every single zone and add them to the config 17:14:30 knowing that it will be out of date almost immediately, but it is still a starting point 17:15:35 is this going to be part of a BIND plugin that still needs to be written? 17:15:44 jbratton: very good question. I need to think through that a bit. I'll update the blueprint and design docs as we go. 17:16:18 currently this is done with a "sync" command 17:16:23 (not sink) 17:16:30 My $0.02 would say that if you send designate a request with the IP/hostname of the BIND server you set up and it will send a Notify to that server 17:16:38 and can be done via the api by an admin 17:16:39 #action eankutse to get initial answers to jbrattons questions 17:16:47 mugsie: yea, it's pretty naive though 17:17:07 but that patter could be extended to target hosts / NSs 17:17:10 pattern* 17:17:38 Ideally, we need a way to monitor that all the expected zones are actually on the nameserver, and automatically re-create any missing zones. 17:17:40 which would cause our backend to try and recreate all domains on the host 17:17:44 cool, because once it gets the list, it can zone transfer all of them and get the current version 17:18:01 Kiall: that might be difficult on a new server install 17:18:04 Kiall: that would be great to catch anything added while it is starting up 17:18:24 exactlly - we need to do that in any case 17:18:30 (lost msgs etc) 17:18:33 yeah 17:19:03 but the inital "here is 1million domains" could be the sync 17:19:22 sure.. basically, I just need the zone names, because then it can zone transfer them all from mdns 17:19:51 Yea - It's more complicated that that though, unless we can freeze any zone create/deletes on the pool for the duration 17:20:31 yeah 17:20:35 e.g. the server get's a list of 1mil domains and starts chewing on them, midway, one of the pending zones is deleted 17:20:40 Kiall - theory would those messages not just buuld up in the queue? 17:20:59 and then you would get an eventual consistancy? 17:21:05 well, at least with BIND, it starts up almost instantly when no zones are present, then it starts working on the zone transfers 17:21:19 so it can take requests to add new zones while it is also going through 1 million zone transfers 17:21:51 jbratton: Yea, i think the actual AXFR bit should be fine - or maybe we're talking about different things? 17:21:55 and if something is deleting in the middle.. it just won't transfer it 17:22:13 If a server fails, and a new one replaces it, it's BIND config is empty, we need to populate it 17:22:46 right, so we create the BIND config, start it, and then it will start up and become responsive while simultaneously issuing tons of zone transfers 17:23:16 it will respond with SERVFAIL for zones it hasn't loaded yet and it will respond authoritatively for zones it has already transferred 17:23:46 and if a zone is deleted while it is going through the list doing transfers, it just removes it from the list of pending transfers 17:23:56 using rndc addzone/delzone 17:24:48 jbratton: Are you thinking that there is a designate-mdns command that sits on the client to help facilitate this? 17:24:57 Yea - As graham mentioned, I think we can take a similar approach for actually populating the BIND config.. Allow create/delete messages to build up while we action a snapshot of the zones.. Then start processing the messages, and catching up 17:25:04 isn't this a problem for server pools, not mini-dns? 17:25:19 right, it isn't so much a mdns question 17:25:25 rjrjr_: I think it applies to both 17:25:33 RaginBajin: that would work too 17:25:40 yeah, it will just be more pronounced in server pools 17:25:42 if pools don't exist, we still need to get BIND/PowerDNS/NSD etc synced up 17:25:51 I leave it to you all to implement this stuff, I just know how it all runs on the nameserver :) 17:26:02 ;) 17:26:12 jbratton: you have great input ffrom the ops side of things 17:26:14 Yea - I think we need to give this one some though 17:26:18 thought* 17:26:26 (from an implementation POV) 17:26:30 jbratton: this is why it i s good to have an ops person shouting "that wont work!!!" 17:26:46 I'll talk to you as I design/implement mdns 17:26:46 that's why I'm here! 17:26:53 :-) 17:27:21 How about if zones are created on the backend (outside of designate), do they/will they be added to mdns? 17:27:45 #action eankutse to pick Jbratton's brain on ops side of things for minidns/pools 17:27:47 vinod1: So - if the BIND server has an existing zone, "import" it? 17:28:38 yes - essentially will "sync" be bidirectional or unidirectional? 17:28:55 Ahh.. Okay, I'd have thought unidirectional 17:29:23 we have no way to associate zones with tenant's etc if they can go both way 17:30:07 Okay, Let's park this for today? 17:30:10 How about if a designate created zone is deleted at the backend - the scenario being designate is down and the zone needs to be deleted asap 17:30:22 #action kiall to write our various initial load scenarios 17:30:58 vinod1: I believe we should re-create that zone when Designate is back online / notices. 17:31:03 vinod1: at least currently, we would break synchronization.. remove it once it is back up, then resync 17:31:08 or that 17:31:40 jbratton: yea, same thing really.. break synchronization, delete from designate, then there's nothing for designate to notice is missing 17:31:56 yeah.. it's a problem to solve no matter what implementation you have 17:32:12 I agree on the unidirectional thing 17:32:24 But from Designate's POV, I think it should re-create zones that "mysteriously" go missing from the backend DNS server 17:32:59 urgent zone deletion is probably the least common reason for a zone to go "poof" on a DNS server ;) 17:33:20 so that means, if anyone deletes something from backend (manually) they have to remove it from storage/central too if they don't want if resurrected 17:33:24 yeah, and zone deletion is almost never an emergency thing 17:33:57 imho opinion, people should not be touching the backend servers that designate manages 17:34:23 yeah, I agree, we never do that on our current implementation 17:34:24 so, in the small egde cases that they need to we should not make too many compramises on design 17:34:26 Well, not touching the list of zones 17:34:44 yeah - thatswhat I meant 17:34:57 I assume someone will be looking to make sure that they are still running 17:35:00 ;) 17:35:08 Okay - I'll write up the various initial load scenarios I can think of on an etherpad tomorrow and email them out, hopefully we can get some clearer ideas that way :) 17:35:46 That kinda covered the first 3 of the items jbratton added, last one was 17:35:49 "I noticed during the demo that listing records in a zone doesn't include NS records. That seems weird, because how do you modify them?" 17:36:13 We don't currently expose the (system defined) NS or SOA records in the API 17:36:44 makes sense from a corporate standpoint, but from a customer service standpoint, customers really like to white label nameservers 17:36:55 they want ns.mydomain.com instead 17:37:11 There was some talk in the past about changing that, but I don't think we ever settled on something? 17:37:22 no afik 17:37:49 i am not sure how we could do it so that customers don;t bork their zones 17:38:06 same as you do currently.. don't let them delete the last NS record 17:38:10 We can likely display them to the V2 API pretty easily, but allowing them to reband them would be more work :) 17:38:31 On the plus side, if we expose them, it's not a breaking API change later to allow them be edited 17:38:41 yeah 17:38:49 yeah ++ on showing them in REST API then 17:38:52 we couldn't implement it until they could be changed.. but that's something I can bug our guys about :) 17:38:57 jbratton: Treat it as a regular record type? 17:39:18 what is the use case? 17:39:28 customers want ns.mydomain.com instead of ns.rackspace.com 17:39:37 or they want to delegate a subdomain 17:39:37 jbratton: yea, API changes are much harder to do though ;) So getting all the breaking changes done ASAP is important 17:39:52 delegate a subdomain works today BTW 17:39:57 oh, cool 17:40:06 then the main use case is white labeling 17:40:21 we just don't allow customers to change the NS records at the root of the zone - i.e. white label 17:41:17 so going back to server pools.. how would someone know which pool they were on? 17:41:19 #action kiall to file BP on exposing the NS (and SOA?) record in the V2 API 17:41:28 because they would want to know what NS records to use for their domain 17:41:44 or do you envision that being something the implementation should define 17:41:50 jbratton: today, you can call /v1/domains/$ID/servers and get the list back 17:42:01 okay, cool 17:42:40 (I would imagine that still get's kept even if the NS records are exposed in the normal records list, as it's easier to pick out for UI's etc) 17:43:01 yeah, definitely, in case you are making administrative changes to the pools 17:43:30 in v2 can we not get the same list as /v2/zones//nameservers? 17:43:44 vinod1: yea, we can 17:44:32 Okay - Recap... BP to be filed for showing the NS (and SOA?) records in the v2 API - that avoids a breaking API change in the future if we allow customers to white label them. *** Etherpad to be setup with initial load/post-maint scenarios for discussion 17:44:57 thanks, that addresses the remainder of my concerns 17:45:01 I'll email the etherpad out tomorrow, and we can figure out where to go from there... 17:45:39 anything else on those Q's before we move on? 17:46:03 #topic Open Discussion 17:46:18 Okay - Any off agenda items? 17:46:27 i have one 17:46:43 If we are sharing backend BIND server from the existing dns infrastructure, In the mDNS —-> BIND communication, is the BIND server going to act as a slave for tenant Zone? 17:47:14 bbaby: heya 17:47:15 yes 17:47:28 mDNS is the master here 17:47:34 So, yes, with mDNS BIND will always be setup as a slave 17:47:57 the "stealth" masters will become slaves 17:48:03 for mDNS to update them 17:48:48 vinod1: that random gate failure - I've finally got some logs to debug it :) 17:48:56 (this was https://bugs.launchpad.net/designate/+bug/1321873 ) 17:49:07 vinod1: but before I do, did you get anywhere debugging it? 17:49:27 no i did not make much headway on this bug 17:49:44 bbaby: so this brings up an issue where the BIND stealth master has to both a slave and master (to the other name servers) 17:49:47 Okay .. I'll dig into the logs I've got and see what's happening 17:50:27 eankutse, slaves can be masters for other slaves. 17:50:27 eankutse: Didn't we settle on avoiding that scenario where possible? 17:50:29 heh... I spent a few hours trying to recreate that gate failure. I didn't realize it was a known issue, and I didn't want to say "works on my machine" 17:50:33 eankutse: sure 17:50:47 rjrjr: yes. 17:51:01 timfreund: lol live and learn :) 17:51:01 but I think it takes quite a bit of work to do 17:51:10 maybe? 17:51:13 rjrjr_: not really.. it requires views and notifies don't go out right away 17:51:22 k 17:51:32 jbratton: ? 17:51:48 if you define a zone as a slave, it won't send a notify out as far as I know 17:52:19 jbratton: would you clarify a bit? 17:52:20 jbratton: you mean to say that all tenant zone need to go under a specific view under BIND (or underlying DNS infrastructure) ? 17:52:22 jbratton: yea, PowerDNS is certainly that way.. I'm guessing you're talking about BIND being difficult to do that with too 17:52:25 nope - I think we covered that - thats why we have multiple hosts associated with serves 17:52:52 bbaby: No, there's two conversations being had at once I think :) 17:53:05 eankutse: if you want a BIND nameserver to be both a master and a slave for a zone, you have to define two views, the view where it is a slave and the view where it is the master 17:53:16 but the master view won't know when the slave view updates 17:53:19 so you would need to write scripts 17:53:26 it's a hack is what I'm trying to say :) 17:53:28 * mugsie 's brain hurts 17:53:35 But I thought we had a way to make it work without that 17:53:43 betsy: we do 17:53:48 we aren't using any stealth masters anymore 17:53:50 with Designate 17:53:59 Ok. That’s what I thought 17:53:59 mdns will be our stealth masters 17:54:00 nope, but we can define multiple hosts as one sever 17:54:05 mugsie: multiple hosts is about having N physical servers behind a single NS record name 17:54:46 yeah, which gets rid of the requirments (along with mdns only nofitying in the right region) for the 2nd teir of masters 17:55:18 Okay - Any other off agenda items? 3 mins left before Trove kick us out 17:55:30 :-D 17:55:31 Kiall: :) 17:55:44 One quick admin question: is there a project mailing list that I should subscribe to? You mentioned mailing stuff out earlier. 17:55:44 hub_cap: miss taco mac? 17:55:53 timfreund: Hello :-) 17:55:56 too much taco mac... 17:55:57 timfreund: the openstack-dev list 17:55:59 mugsie: TACO MAC!!! 17:56:09 Hey eankutse! 17:56:14 timfreund: for the occasional stuff like this it tends to just be direct email.. But we really should be using the openstack-dev list 17:56:26 OK, I'll subscribe there. Thanks! 17:56:27 99% of the emailed out stuff is related to meetups ups ;) 17:56:29 Kiall: i assumed you were sending it to the dev list ;) 17:56:36 i'd like to revisit N tier next discussion. 17:56:38 mugsie: yep - I will 17:56:49 rjrjr_: cool - lash it on the agenda 17:56:53 rjrjr_: Sure 17:56:55 Kiall: may be you have answered before. Is the designate packaged now (.deb/.rpm)?. I want to try it on poc environment 17:57:16 bbaby: the previous release is packaged in https://launchpad.net/~designate-ppa/+archive/havana 17:57:23 they haven't been updated for icehouse yet 17:57:23 thnx 17:57:47 Okay - That's time I think? 17:57:50 yu[ 17:57:53 o/ 17:57:54 Any last works? 17:57:56 words* 17:58:05 Guess not ;) Thanks all 17:58:06 #endmeeting