09:04:29 #startmeeting ha 09:04:30 Meeting started Mon Oct 3 09:04:29 2016 UTC and is due to finish in 60 minutes. The chair is aspiers. Information about MeetBot at http://wiki.debian.org/MeetBot. 09:04:31 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 09:04:33 The meeting name has been set to 'ha' 09:05:01 #topic status updates 09:05:17 I'm not sure if you saw the discussions on the clusterlab lists 09:05:28 unfortunately no 09:05:43 * ddeja should subscribe to clusterlabs list 09:06:12 #info discussions on clusterlab lists about the future of OpenStack RAs 09:06:18 http://clusterlabs.org/pipermail/developers/2016-September/000339.html 09:07:13 although Red Hat are mostly bypassing this discussion, since they are ditching Pacemaker for (almost?) all active/active services 09:07:35 I've also heard they want to do so 09:07:38 apparently they will rely on HAproxy and the services themselves for correctness, and some other monitoring system for reporting failures 09:08:06 so every service should be able to function correctly even when any of its dependencies are dead or misbehaving 09:08:22 that sounds like a bit of a leap of faith to me, at least at the current time 09:08:25 but ICBW 09:09:16 Barcelona is rapidly approaching! did you get travel approved yet? 09:09:26 nope... 09:09:29 :-/ 09:09:50 yup, I think the same 09:10:05 mine is confirmed 09:10:14 that's cool 09:10:22 I really want to get 3-4 of the specs completed before we arrive 09:10:52 Me too, it is finally after the release week, so I have time for specs preparation 09:11:22 great 09:11:27 maybe we can form a quick plan now 09:11:31 ok 09:11:37 are there any HA meetings in Barcelona calendar? 09:11:58 radeks: sadly not, I pushed quite hard for it but no luck 09:12:30 yeahhh this explain why I couldn't find anything :) 09:12:43 radeks: http://lists.openstack.org/pipermail/openstack-dev/2016-August/100570.html 09:12:49 aspiers, radeks: but there would be some unofficial meetings, like in Austin 09:12:54 right 09:12:59 we would send email to OpenstackDev 09:13:08 great 09:13:21 I think we could maybe use cross-project space, but only if we have specs which clearly explain the scope 09:13:36 sorry im late 09:13:49 hey samP! welcome back :) 09:14:05 aspiers: thanks 09:14:16 #topic specs 09:14:24 so I merged the VM monitoring spec 09:14:44 \o/ 09:14:45 I think it's basically done, although a few more details on the event payload wouldn't hurt, and also something about SSL support 09:15:30 we could discuss it in Barcelona 09:15:39 for the benefit of lurkers, we are talking about the specs in https://etherpad.openstack.org/p/newton-instance-ha 09:15:46 lines 115 onwards 09:15:59 I will do host monitoring - that should be easy 09:16:09 obviously host recovery is a very important one 09:16:17 ddeja: you still OK to start that one? 09:16:27 aspiers: yes 09:16:27 I have to do remaining sepc..(TT) 09:16:40 aspiers: I'll send first draft this week 09:16:47 great :) 09:16:51 (you can put an action on me ;) 09:17:00 #action ddeja to draft host recovery spec 09:17:06 #action aspiers to draft host monitoring spec 09:18:40 I am wondering if we still need a libvirt OCF RA 09:19:00 aspiers: why not? 09:19:19 oh yeah, we do 09:19:19 isnt that basically coverd in VM monitoring? 09:19:36 let me find the clusterlabs list discussion on this 09:23:05 http://clusterlabs.org/pipermail/users/2016-May/003040.html was a long thread about informing RAs about recovery 09:23:30 aspiers: oh, thism thread 09:23:45 I remember reading it... 09:24:23 let me find a link to the consensus 09:24:34 but I don't remember how this thread is connected with monitoring or not the VMs in libvirt? 09:24:52 it was about services AFAIK 09:25:24 http://clusterlabs.org/pipermail/users/2016-June/003344.html 09:25:52 so the original discussion we had in Austin was: if nova-compute fails, we want to try to restart a few times before fencing - right? 09:26:03 there was such idea 09:26:10 indeed 09:26:21 but when we give up on the retries, we want to do nova service-disable 09:26:30 yup 09:26:36 and similarly for if libvirt fails 09:26:50 yup 09:26:50 aspiers: that discussion start with how to recover process failures 09:27:03 but I think the consensus of this long thread on the list was, *every* time nova-compute stops, do nova service-disable 09:27:11 not just after the final failed retry 09:27:20 and when it starts, do nova service-enable 09:27:32 so we don't handle the final retry as a special case 09:27:49 ok, agreee 09:27:55 if we take that approach, we don't even need an OCF RA for libvirt 09:28:11 since there is already an ordering (or group) constraint linking libvirtd with nova-compute 09:28:30 so if libvirtd fails, nova-compute is automatically stopped, and then nova service-disable is called 09:28:38 aspiers: OCF RA for libvirt was meant for monitoring the vms inside libvirt? 09:29:00 no, just for monitoring libvirtd 09:29:18 the spec I just merged is for monitoring the VMs 09:29:18 oh, OK 09:29:30 now we are on the same page 09:29:36 cool :) 09:29:49 the only reason I can think of for writing a libvirtd OCF RA, is to improve the monitoring 09:30:03 so yes, if we want to disable nova every time it stops, we can skip the RA for libvirt 09:30:04 e.g. if libvirtd is running but hung, then better monitoring could catch this 09:30:05 aspiers: what about other processes such as iscsid or mutipathd 09:30:50 it could do virsh sysinfo or similar every 10 seconds 09:31:13 samP: interesting point 09:31:16 aspiers: I think we start that discussion for how monitor the processes. 09:31:41 aspiers: then, dicussion was limited to nova-compute and libvirtd 09:32:01 samP: I think it's the same situation for iscsid / multipathd / libvirtd 09:32:10 if all are dependencies of nova-compute 09:32:30 then Pacemaker needs to monitor all of them, and have order/group constraints 09:32:41 but I think that's not OpenStack-specific 09:33:24 e.g. there is already https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/iscsi 09:34:15 samP: but if you want to write a spec about the storage side, I'm definitely in favour :) 09:34:39 however if backend storage fails, I guess the only recovery action is fencing 09:34:42 which makes it simpler 09:34:48 aspiers: yep, I will write some details. so we can take a close look onit 09:35:04 OTOH, if libvirtd dies, the operator might choose a policy which keeps some VMs still running 09:35:20 they might prefer to do manual recovery of libvirtd, instead of killing important pets 09:35:40 aspiers: ype, that is one way to recover 09:36:17 ah, we already have that in the etherpad 09:36:24 lines 136-138 09:36:35 cool :) 09:37:09 aspiers: yep. missed it. thanks 09:37:55 #action aspiers to draft specs for libvirt and NovaCompute OCF RAs 09:38:09 samP: maybe you could start on the VM recovery spec? 09:38:31 yes, I will. 09:39:10 #action samP to draft VM recovery spec 09:39:16 aspiers: thx 09:39:28 OK great, I think that's a good enough plan for this week - plenty for everyone to do :) 09:39:37 aspiers: yup 09:39:42 BTW samP how was NYC Ops meetup? 09:39:58 or did we already discuss that? :) 09:40:00 I can't remember 09:40:19 I think we did 09:40:21 aspiers: I think we already did 09:40:32 OK never mind ;-) 09:40:33 samP sent us links to etherpad 09:40:45 oh yeah, I remember now 09:41:00 #topic local area meetups 09:41:06 I attended the OpenStack London meetup last week 09:41:12 there were ~70 people 09:41:19 maybe 50, not sure 09:41:30 anyway, I asked everyone how many pets they are seeing in OpenStack 09:41:39 and apparently it's still very, very common 09:41:52 that's somehow good for us 09:41:55 and lack of HA seems to be a significant problem, especially in forklift migrations from VMware 09:42:01 so yes, we are not wasting our time here :) 09:42:01 it means our work is not useless ;) 09:42:04 exactly 09:42:26 I think that will be true for a long time yet 09:42:45 thought you might appreciate hearing some extra motivation for our work ;) 09:42:53 :) 09:43:22 #topic next generation HA architecture 09:43:22 I was on a Wroclaw OpenStack meetup last month 09:43:30 oh sorry, changed topic too soon 09:43:39 but no discusson about Pets vs Cows 09:43:50 aspiers: ddeja : you may find the NY ops details in following meeting, http://eavesdrop.openstack.org/meetings/ha/2016/ha.2016-08-29-09.33.log.txt 09:43:53 Poland OpenStack community is not very mature unfortunately 09:44:00 samP: thanks :) 09:44:07 OK 09:44:19 I think my colleague Michal Jura goes to those 09:44:46 aspiers: I may saw him on attendes list 09:45:22 but not sure. Nevertheless, it is 350km from my city, so I'm not attending it every time 09:45:23 I'm pretty sure he has presented there a few times 09:45:27 OK 09:45:32 I guess it's not local for him either 09:45:50 yes, SUSE Poland is in different city also :) 09:46:03 right :) 09:47:03 ok, we can move to the topic ;) 09:47:33 ok 09:47:36 http://docs.openstack.org/ha-guide/intro-ha-controller.html#common-deployment-architectures 09:47:56 https://review.openstack.org/#/c/372843/ 09:48:01 has been merged 09:48:07 to document Red Hat's new approach 09:48:40 which reduces use of Pacemaker to managing active/passive resources, like the VIPs 09:48:48 and compute HA, of course 09:49:26 I'm slowly coming around to this idea, but I think there are unresolved questions 09:49:40 e.g. it depends on all OpenStack services behaving correctly when their dependencies misbehave 09:49:58 like I said earlier, it sounds like a leap of faith - I'll believe it works when I see it :) 09:50:18 and also it requires some monitoring dashboard outside Pacemaker to see the health of all active/active services 09:50:18 aspiers: I belive Fuel works that way 09:51:05 well it may move away some of the pain which comes from Pacemaker + Rabbit 09:51:28 this is the biggest issue AFAIK and see in the field 09:51:36 My collegue wanted my help to configure something on a Fuel OpenStack cloud and I was suprised that there was only 5 or 6 services added to pacemaker 09:51:36 IIUC, here are the Fuel OCF RAs in use https://review.openstack.org/gitweb?p=openstack/fuel-library.git;a=tree;f=files/fuel-ha-utils/ocf;hb=HEAD 09:52:14 radeks: the biggest pain you see is Pacemaker+Rabbit? 09:52:20 radeks: you mean active/passive? 09:52:23 aspiers: yes, there was only network-related services 09:52:35 ddeja: were they using keepalived? 09:52:47 I don't know 09:53:37 aspiers: what I mean by this is a problems rabbit has with full HA mirroring of queues, there is a performance penalty and many other like nasty behavior when network gets partitioned 09:54:25 aaspires: and when you have to for whatever reason restart rabbitmq, all of them or one of them this is becoming a problem 09:54:50 radeks: ah, but with mirrored queues are you still using Pacemaker? 09:54:55 for Rabbit I mean 09:55:21 yes, this is the default for TripleO for example 09:56:11 aspires: if you are lucky with 3 controllers and you want to restart rabbitmq your OpenStack services goes down for at least 20 minutes 09:56:12 interesting, I didn't realise that active/active Rabbit with mirrored queues would still be controlled by Pacemaker 09:56:43 well anyway, we're running out of time 09:56:53 5 minutes left ;) 09:56:54 all the OpenStack services has dependency on rabbit, at least in TripleO 09:57:10 I don't expect any upstream actions on this right now 09:57:23 but I wanted to make sure everyone is aware of it, and the fact that it is now documented in the HA guide 09:57:25 it's probably good subject for discussion in Barcelona 09:57:42 radeks: for sure :) although I think Red Hat is already set on this path 100% :) 09:58:54 I started this thread http://clusterlabs.org/pipermail/developers/2016-September/000339.html to try and minimise the difference between the two approaches 09:59:23 if the "older" approach of still using Pacemaker to manage active/active services could reuse systemd units, then there would not be a big difference 09:59:39 but currently all the OCF RAs reinvent the systemd wheel (or vice-versa) 09:59:52 but yes, let's discuss in Barcelona 10:00:06 radeks: if you are coming, please make sure to say hello :) 10:00:17 alright, we're out of time 10:00:25 if there is anything else, let's continue on #openstack-ha 10:00:30 thanks all, and bye for now! 10:00:35 #endmeeting