15:04:40 <johnthetubaguy> #startmeeting XenAPI
15:04:41 <openstack> Meeting started Wed Aug 28 15:04:40 2013 UTC and is due to finish in 60 minutes.  The chair is johnthetubaguy. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:04:43 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:04:45 <openstack> The meeting name has been set to 'xenapi'
15:04:52 <johnthetubaguy> so, hello all
15:04:53 <BobBall> Weird.
15:05:10 <johnthetubaguy> usual agenda OK for everyone?
15:05:20 <johnthetubaguy> nothing big from me, other people got things to discuss?
15:05:20 <BobBall> yup
15:05:28 <matel> uh
15:06:27 <johnthetubaguy> #topic actions from last meeting
15:06:29 <matel> I just updated my patches, and I am planning to look at other xenapi patches, maybe that helps for reviewers.
15:06:37 <johnthetubaguy> BobBall: did you look into that VDI.copy issue?
15:06:51 <BobBall> do you mean whether we need safe_copy_vdi
15:07:00 <johnthetubaguy> yep
15:07:54 <BobBall> no - but one sec
15:08:00 <BobBall> carry on with otehrs then I might have one!
15:08:10 <johnthetubaguy> OK… will put down action for next time
15:08:31 <johnthetubaguy> #action BobBall look into VDI.copy workaround and if it is still needed or not
15:08:53 <johnthetubaguy> seems like its probably because of parallel VDI.copy or something like that
15:08:57 <johnthetubaguy> anyways, moving on
15:09:05 <johnthetubaguy> #topic Blueprints
15:09:08 <johnthetubaguy> any news here
15:09:13 <johnthetubaguy> matel: you said about patches
15:09:26 <matel> yes.
15:10:43 <matel> I updated all patches related to https://blueprints.launchpad.net/nova/+spec/xenapi-supported-image-import-export
15:11:26 <matel> What's the question?
15:11:54 <johnthetubaguy> ah, thats all
15:11:56 <johnthetubaguy> cool
15:12:12 <johnthetubaguy> no more for blueprints I guess
15:12:20 <johnthetubaguy> #topic Docs
15:12:21 <matel> Don't think so
15:12:24 <johnthetubaguy> any docs updates
15:12:26 <johnthetubaguy> ?
15:12:47 <matel> I plan to finish the doc scanning this week.
15:12:58 <matel> Need to look at where we are with the re-structure.
15:13:24 <johnthetubaguy> cool
15:13:37 <johnthetubaguy> good to get that looked over again
15:14:06 <matel> I think the main re-structure change has been merged, so that should not block.
15:14:19 <johnthetubaguy> cool
15:14:27 <johnthetubaguy> #topic Bugs and QA
15:14:40 <johnthetubaguy> any bug questions or QA updates?
15:14:45 <matel> I kill a bug whenever I find one.
15:15:01 <BobBall> I guess we should have a QA update
15:15:11 <BobBall> Smokestack has been struggling occasionally under the weight of reviews
15:15:16 <BobBall> particularly the 4 XS hosts
15:15:24 <johnthetubaguy> don't forget the xenserver tag, an I will triage the bug ASAPI
15:15:25 <BobBall> In fact, for a short period, the XS jobs were disabled
15:15:42 <matel> Are we adding more resources?
15:15:53 <BobBall> I've prototyped a cloud-based XS installation
15:15:54 <BobBall> which mostly works
15:16:07 <matel> How beefy is the VM?
15:16:11 <BobBall> although yesterday Dan raised a concern with one of the tests and I'm not sure if itw as a code issue or an infrastructure issue
15:16:16 <BobBall> beefyish
15:16:19 <BobBall> but the problem is CPU
15:16:29 <BobBall> single CPU speed
15:16:51 <johnthetubaguy> how much are you doing in a run? building the DomU?
15:17:02 <BobBall> That can't be increased unfortunately so it does run slower (about 10 mins slower) in a VM in RAX cloud rather than hardware
15:17:07 <BobBall> that's 30 minutes vs 20
15:17:18 <BobBall> No - this is just smokestack
15:17:21 <BobBall> so not devstack at all
15:17:25 <johnthetubaguy> ah, I guess thats to be expected for nested virt?
15:17:36 <BobBall> perhaps yes
15:17:46 <BobBall> but the real problem is that the virt is HVM and can't use PV drivers
15:18:06 <johnthetubaguy> right, but that shouldn't hit CPU usage?
15:18:13 <BobBall> because we can only use one xenstore (which must be for the nested VMs) therefore XS can't use PV disk/network
15:18:20 <BobBall> so it's all emulated
15:18:30 <matel> 150% - I wasn't expecting this much overhead.
15:18:35 <BobBall> hence using CPU :)
15:19:24 <johnthetubaguy> ah, OK, still a bit surprising high
15:19:26 <matel> Should we try putting it to KVM?
15:19:38 <BobBall> I've considered it, but I want to get it working right on Xen first
15:19:41 <BobBall> since that's what we know
15:19:52 <johnthetubaguy> lol, same issue on KVM I would guess?
15:20:02 <BobBall> if we can push it to the HP cloud then that's based on KVM and we could use virtio to boost performance
15:20:05 <matel> sure, I was just curious what would be the overhead there.
15:20:12 <johnthetubaguy> ah, I see
15:20:14 <BobBall> (I think)
15:20:30 <BobBall> although that's more work than I've got time to do ATM
15:20:35 <BobBall> and I don't have an HP account ;)
15:21:10 <johnthetubaguy> anyways, I guess running in Dom0 might be faster in that setup, because the inner VM is less hampered by performance?
15:22:03 <BobBall> ish
15:22:11 <BobBall> but I think the real problem is Xen being HVM with no PV drivers
15:22:21 <antonym> how large is the instance?
15:22:37 <matel> Bob, 30 min is the test run, or it's the whole stuff with the setup?
15:22:45 <johnthetubaguy> is it not because you are running rabbit and mysql inside a VM thats inside a VM?
15:23:28 <matel> would be nice to do some profiling to see what slows us down.
15:23:40 <johnthetubaguy> matel: +1
15:23:42 <BobBall> whole stuff
15:24:07 <matel> Bob: how much is the setup part, and how much time does the test run take?
15:24:10 <BobBall> I agree that'd be nice
15:24:12 <BobBall> dunno :)
15:24:21 <BobBall> only have the initial results from running it once yesterday
15:24:23 <matel> How larg is the instance?
15:24:39 <BobBall> 8GB
15:24:51 <BobBall> but you can get any instance size you like
15:24:51 <antonym> so you have 4 vcpus at least
15:25:01 <BobBall> sure - but most of the work is single threaded
15:25:06 <antonym> yeah
15:25:26 <BobBall> I think
15:25:32 <BobBall> it's a shame dprince isn't online atm
15:25:42 <BobBall> oh hang on
15:25:44 <BobBall> maybe he is
15:25:51 <dprince> BobBall: I'm here... just on a call.
15:25:53 <BobBall> oh
15:26:00 <BobBall> ok
15:26:14 <johnthetubaguy> yeah, I suspect the work going on inside the DomU is really slowed down because its in a nested VM, which has super sucky performance
15:26:29 <BobBall> but it shouldn't be
15:26:33 <BobBall> don't forget it's running PV
15:26:41 <BobBall> so it's only PV in one HVM container
15:26:53 <BobBall> so CPU performance should be no worse than one HVM
15:27:01 <BobBall> it doesn't need to trap the hypercalls etc (I think)
15:27:27 <antonym> would it run better on certain versions of xs?
15:28:03 <BobBall> Not sure - I'd need to ask about the version of Xen and whether it'd affect the pefromance nested
15:28:20 <BobBall> but there may be things we can do - e.g. modifying smokestack to make less use of disk
15:28:27 <antonym> if it does, we should be able to help resolve that
15:28:56 <BobBall> yeah
15:29:12 <BobBall> it's a bit rubbish if the real restriction is that we can't have Xen using PV drivers though...
15:29:21 <antonym> at least it works :) should be able to optimize from there hopefully
15:29:42 <BobBall> and it's in the cloud - so we can just scale out, right? ;)
15:29:52 <antonym> of course!
15:30:03 <johnthetubaguy> http://www.slideshare.net/xen_com_mgr/nested-virtualization-update-from-intel
15:30:07 <BobBall> Unfortunately one problem is that the gate is now running jobs in parallel
15:30:34 <BobBall> so should be a lot faster
15:31:36 <johnthetubaguy> from those slides, looks like some recent work really helped
15:31:47 <johnthetubaguy> not sure what version of Xen that maps to
15:32:09 <matel> Slides are a year old.
15:32:36 <BobBall> I'll look into that
15:32:50 <matel> Read the comment :-)
15:33:04 <BobBall> nice comment :)
15:33:08 <antonym> heh
15:33:24 <BobBall> Not sure if it'll help though - might be intel specific - we'll see.
15:33:27 <BobBall> I'll investigate a bit
15:33:30 <BobBall> gimme an action!
15:33:35 <BobBall> brb.
15:33:35 <johnthetubaguy> yes, good points
15:33:47 <johnthetubaguy> it could be nested virt on AMD performance issues
15:33:56 <johnthetubaguy> anyways, lets move onwards
15:34:14 <matel> sure
15:34:17 <johnthetubaguy> #topic Open Discussion
15:34:23 <johnthetubaguy> any more things people want to cover?
15:34:42 <johnthetubaguy> will wait for BobBall to return an confirm, I think he had something else
15:34:56 <matel> Let's review each other's changes, that's all.
15:35:49 <johnthetubaguy> indeed, much reviewing to be done all around
15:36:03 <BobBall> nothing else
15:36:13 <BobBall> oh yes
15:36:14 <BobBall> that's it
15:36:22 <BobBall> reviews
15:36:25 <BobBall> we need another core :)
15:36:32 <BobBall> changes with +2 should just be able to go in I hope
15:36:37 <BobBall> I've got a few of them
15:36:58 <BobBall> should I just add ppl to my review? or is there a better way to encourage them?
15:37:22 <matel> I guess everyone is busy these days.
15:37:23 <johnthetubaguy> catch people on IRC I guess, but mostly I think people are working on the priority of the blueprints
15:37:58 <johnthetubaguy> I have lots of changes pending myself (14)
15:38:37 <matel> yes, I will look at them
15:39:08 <BobBall> ok
15:39:12 <BobBall> are we done then
15:39:18 <matel> Cool, thanks.
15:39:40 <johnthetubaguy> thanks all
15:39:44 <johnthetubaguy> #endmeeting