17:03:17 #startmeeting XenAPI 17:03:18 Meeting started Wed Jan 16 17:03:17 2013 UTC. The chair is johngarbutt. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:03:19 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:03:21 The meeting name has been set to 'xenapi' 17:03:29 Hi all 17:03:38 #topic actions from last meeting 17:03:38 hi 17:03:39 hello 17:03:41 howdy 17:03:56 Morning! 17:04:10 one action was for bobball to check about any limits on the xenstore stuff 17:04:23 I think it was 2k for any xenstore value? 17:04:51 That's right - it's a page of 4k but with a bunch of headers, then we use a round value of 2048 bytes 17:05:06 right 17:05:18 Thanks again, BobBall. 17:05:30 #info 2k limit on size of xenstore value 17:05:45 hth! 17:06:04 the other think was to discuss resetting passwords and looking towards cloud-init rather than xenapi specific agent 17:06:15 I think we scheduled that for 17:30 17:06:27 #topic blueprints 17:06:37 anyone got any blueprint worries? 17:07:08 mikal: is config drive done now? blueprint says in progress? 17:07:31 There are the same review requests from last week I think 17:08:02 sounds like nothing more on this one? 17:08:19 #topic docs 17:08:42 I updated the hypervisor wiki page to note that config drive hit trunk 17:09:09 #topic QA 17:09:41 any more updates on gating trunk? 17:10:36 #topic open discussion 17:10:45 anything people want to raise? 17:11:39 #info I got XCP iso running on virtual box, then openstack running using devstack, makes for a neat dev environment 17:12:36 is this meeting time still good with people? is it still useful? 17:12:37 It's been a quiet week I think 17:12:56 I am keen to keep doing this weekly, so we have a good time to raise things when they come up 17:12:59 I can bring up force detach if you like 17:12:59 :) 17:13:10 any news on san stufF ? :p 17:13:16 yes, it is extremely helpful to me, i am still learning the space though 17:13:30 guitarzan: fire away 17:14:09 guitarzan: is it related to volumes? 17:14:11 it's the same request... we (rackspace) would really like to be able to do it 17:14:16 matelakat: yeah 17:14:46 Oh, I see, today you might wait until a VM reboots before it happens, no way to force it? 17:15:02 yes 17:15:16 and I'm not even sure the support guys can always get it to happen during a reboot 17:15:32 but I'm not entirely clear on that case 17:15:35 Guests with PV tools installed? Doesn't the detach happen from XS immediately? or is it OpenStack that's postponing it becuase it's not sure it can happen? 17:15:49 you get disk in use from XenServer 17:15:49 BobBall: xen won't detach because it's in use by the pv guests 17:16:03 I guess it's mounted, right? 17:16:14 oh - of course. The detach is negotiated with the guest, yes. 17:16:18 mounted, part of a raid, lvm, someone used the wrong side of the pillow 17:16:59 guitarzan, Just for confirmation, if you run a vbd-unplug with force=true, does that succeed? 17:17:07 it is one of those things that might be kernel specific I guess, due to pvops code 17:17:07 BobBall: someone was supposed to get me your email address and I was going to send you a couple of emails 17:17:26 BobBall: I'm not sure... I didn't know about force=true! 17:17:29 guitarzan, I'm bob.ball@citrix.com - feel free to send me as many emails as you like 17:17:31 do we have the bug for this? 17:18:00 BobBall: thanks! 17:18:18 one other interesting xen issue is we've seen pbd-unplug not actually remove the iscsi connection 17:18:24 does nova have a force detach? 17:18:26 BobBall: that's the other one I needed to ask you 17:18:32 johngarbutt: I'm not sure 17:18:56 I guess the current code goes into a retry loop checking to see if the volume gets detached 17:18:56 I think clay added it, but I think that was just the cinder side 17:19:11 I guess the nice thing is to add a force remove (admin only operation?) 17:19:23 or is it users that get worried? 17:19:28 guitarzan, I think that the iSCSI connection is managed by the SR - so since the PBD could in theory be re-plugged as long as the SR is active then that connection is expected to stay? johngarbutt - is that right? 17:19:45 well, they ran an sr forget as well 17:19:50 this has only happened a few times 17:19:52 oh - drat! 17:20:10 it certainly should get tidied up 17:20:13 it's not exactly an unconfirmed bug, but I don't know how to reproduce it 17:20:15 #link https://bugs.launchpad.net/nova/+bug/1030108 17:20:18 Launchpad bug 1030108 in nova "Detaching a volume from a XenAPI instance fails if XenAPI thinks it is in use" [Medium,Confirmed] 17:20:41 So you've seen some cases where the SR doesn't exist in XS any more, yet the low level iscsi connection is still active? 17:20:42 I think that is when I became aware of the issues here 17:21:10 BobBall: yes, very infrequently 17:21:21 johngarbutt: got it 17:21:37 that bug specifies the "remove from queue" half, but I think a force is more wanted 17:22:00 yep, we have a force, but bad things might happen to the data in the volume 17:22:05 guitarzan, That's not ideal! We'll have to look into possible causes for that 17:22:10 johngarbutt: san? 17:22:38 BobBall: it would be great if we could explain it at least. it's possible that it happened during some manual sr/pbd cleanup after failed resizes and such 17:23:49 zykes: no real news there I am afraid 17:23:51 guitarzan, Can we get a copy of the SMlog from a system that this has occured on? There should be some logging of when we try and log out of the iscsi session 17:24:00 johngarbutt: doh 17:24:39 BobBall: I will look, but it's probably logrotated away by now 17:24:46 #action BobBall and guitarzan to get some logs about occasional iSCSI issue 17:24:47 if I see another one, I will definitely grab logs 17:25:22 Great, thanks. Ideally grab a bugtool so that other possible logs are also contained (e.g. messages, xensource.log) and the xapi DB to check state etc 17:25:58 Is this something that you fall over quickly when it happens? (i.e. because you can't reconnect the iscsi target to another host or similar?) 17:26:07 cool, so that covers most things 17:26:30 BobBall: no, we just found this doing some backend volume clean/audit 17:26:36 any more for any more (before we move on to passwords without an agent 17:27:22 Hmmm okay. Well let's take this offline - I'll have a bit of a think and a dig to refresh myself on how we handle the iscsi sessions and hopefully we'll get an occurence soon we can debug 17:28:12 great, thanks 17:29:09 #topic password reset in a post xenapi agent world 17:29:23 hi, so there were some extra folks going to join us 17:29:59 who is in the channel and interested? 17:30:28 alexpilotti: hows things? 17:30:32 pvo: hows tings? 17:30:55 johngarbutt: hey 17:30:58 #link https://blueprints.launchpad.net/nova/+spec/get-password 17:31:14 so looks like there is a patch from vish to make the old and new work together 17:31:26 #link https://review.openstack.org/#/c/19746/ 17:31:48 basically, if you set a password with the agent, you can get the encrypted password using the new nova call 17:32:11 alexpilotti: what is the plan with hyper-v and cloud-init? 17:32:28 or rather cloud-init on windows 17:32:43 johngarbutt: 1) Implementing the patch that Vish proposed 17:32:56 using the metadata service 17:33:01 post to metadata service? 17:33:05 yep 17:33:14 #link https://gist.github.com/4008762 17:33:17 2) doing it via guest / host communication 17:33:42 letting the user choosing between 1 and 2 17:33:47 so how do you kick off the getting of the password, is it just on VM create? 17:34:15 that's the safest way 17:34:30 this way the password don't travel unencrypted on any channel 17:34:43 *doesn't 17:34:53 I think the xenapi side requirement is that we can reset the password without a reboot, at the users request 17:35:08 that's what I want to do as well 17:35:12 I know the user currently specifies the password, but I think we are OK with it being generated 17:35:19 as you say, much safer 17:35:28 here's BTW the API from nova.api.metadata import password; password.set_password(context, ecrypted_pass) 17:35:45 to be used on the host side to set the password 17:35:51 right, makes sense, thanks 17:36:10 so I guess you want the hypervisor system so you don't need the metadata service 17:36:26 I'm thinking on transforming cloud-init from a simple "inizialize and exit" service to a service that listens all the time for requests 17:36:42 which is similar to the way in which it works on Azure, for example 17:36:58 interesting. where would the requests come from in your case? 17:36:59 setting the password is the most obvious scenario 17:37:16 I guess puppet kick might be another, eventually 17:37:18 with a Nova extension, like the one you are already supporting 17:37:46 I mean, how does cloud-init get communicated to? 17:38:01 does it poll metadata service? does config drive get updated? 17:38:11 I'd like to have an sbstract API on top of each hypervisor's communication channel 17:38:33 let's say a simple value get / set semantyc 17:38:37 So it's more like an agent :-) 17:38:39 OK, I think that makes sense 17:38:53 have you seen the agent api in xenapi, one sec 17:39:08 with also some event listening channel 17:39:18 y, the whole stuff remonds me to the xenapi agent. 17:39:27 johngarbutt: can you sned me a link? 17:39:39 #link https://github.com/openstack/nova/blob/master/nova/virt/xenapi/agent.py 17:39:42 *reminds 17:39:45 here now… sorry had something run over. 17:39:52 if we can avoid to reivent the well is even better :-) 17:39:57 * wheel, lol 17:40:08 and on the server side: #link https://github.com/openstack/nova/blob/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/agent 17:40:22 if __name__ == "__main__":    XenAPIPlugin.dispatch(        {"version": version,        "key_init": key_init,        "password": password,        "resetnetwork": resetnetwork,        "inject_file": inject_file,        "agentupdate": agent_update}) 17:40:34 hmm, that is a bit specific I guess 17:41:41 so agent.py runs on the guest? 17:41:53 yes 17:41:59 no 17:42:00 sorry 17:42:10 it runs on the hypervisor as an API extension 17:42:16 along with this code #link https://github.com/openstack/nova/blob/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/xenstore.py 17:42:23 it is called from nova-compute 17:42:31 Johngarbutt: do you have a link to the agent source code? 17:42:35 basically there is a shared key value store 17:43:00 is it still here? https://launchpad.net/openstack-guest-agents 17:43:10 or moved to github? 17:43:33 http://wiki.openstack.org/GuestAgent 17:43:39 https://github.com/rackspace/openstack-guest-agents-unix 17:43:51 thats the one, cheers 17:44:39 alexpilotti: does it help? 17:44:56 bit information overload, lets set back a little 17:45:00 looking! 17:45:37 password generate on guest, sent to use encrypted via metadata service: done by nova, cloud-init still needs to support 17:46:09 password generate is triggered by VM start 17:46:15 got it 17:46:17 at the same time as key injection 17:46:31 johngarbutt: thanks, that helps, I almost lost the context. 17:46:34 what we need is: 17:46:42 1) method to kick off password reset 17:46:58 (yes, github code is current for agent) 17:47:07 2) method to send encrypted generated password when not using metadata service 17:47:45 you could do 1 by making cloud init poll the metadata service 17:47:51 1) I'd go with an extension like "reset-password" or similar 17:48:10 I think vish already almost added that 17:48:13 and use vishy's get-password to retrieve it 17:48:15 clear-password or something? 17:48:25 i think cloud-init is not a daemon today, correct? 17:48:30 ie, just runs once at boot 17:48:34 * comstud is not sure 17:48:35 comstud: correct 17:48:38 right, good point 17:48:55 comstud: and smoser doesn't like the idea of transforming it into an agent 17:49:00 I like the idea of just re-triggering cloud-init in certain cases, and an agent would do that I guess 17:49:13 right, that is worth knowing, thanks 17:49:24 my take is that cloud-init should become a full agent 17:49:34 switching out the agent for cloud-init affects more than just password reset 17:49:40 it also affects any networking changes that are made 17:49:54 good point 17:49:56 right now we're able to tell the agent to re-configure networking 17:50:03 I'd also fifferentiate the command plugins 17:50:20 nice word! 17:50:32 lool 17:50:44 forget ny typos :-D 17:50:49 :-) 17:51:10 I'm notoriously a disaster in writing in English quickly :-) 17:51:29 I guess the other point is to get images that can work in more places at once 17:51:31 I should deploy more than 4 fingers probably ;-) 17:51:38 back to the idea 17:51:46 a command can have a property 17:52:01 that indicates if it can be executed only at boot 17:52:14 so for example, password reset could be triggered any time 17:52:24 network injection only at boot, etc 17:52:36 alexpilotti: this is assuming its a daemon, right? 17:52:47 erm, I think we want network reset at other times too, but I see what you mean 17:52:50 that's an option. 17:53:00 pvo ^ 17:53:06 alexpilotti: how would it get triggered at other times? 17:53:08 pvo: on WIndows it's a service 17:53:16 ok 17:53:22 he's proposing it become a daemon 17:53:32 with certian commands marked as boot-time-only 17:53:50 a Windows service is a daemon, no? 17:53:55 y 17:53:59 johngarbutt: and yeah, network reconfiguration should be allowed at any time, IMO 17:54:00 (admittedly, its been a while for me and windows) 17:54:08 I'd go with password-reset Nova API extension -> hypervisor sends message via comm channel to guest -> cloud-init agent triggers event -> command plugin execution 17:54:17 this isn't windows 95 where we should have to reboot to get changes. 17:54:18 :) 17:54:35 or windows ME 17:54:38 yew 17:54:40 anyways 17:54:41 ;) 17:54:48 comstud: reemind me to put a flag to set the host on fire if somebody tries to deploy Win95 :-D 17:55:00 heheh 17:55:04 I miss windows 95... 17:55:10 But I digress 17:55:31 I think we might need to separate the cloud-init and deamon thingy 17:55:32 I have a windows 3.11 VM.. w/ IE 3. 17:55:32 BobBall: "de gustibus non disputandum est" :-) 17:55:46 could we say there is an agent that kicks cloud-init, for example 17:55:46 johngarbutt: +1 17:55:51 johngarbutt: thats kinda what we did on our windows agent. 17:55:54 its two services 17:55:59 johngarbutt: smoser anyway doesnt like the idea 17:56:08 alexpilotti: is that the only reason to kill it? 17:56:12 johngarbutt: on WIndows, I'd prefer to deploy a single WIndows service 17:56:36 I think we do just have an extra agent that can support extra abilities that cloud-init cannot provide 17:56:40 in XenAPI terms it means agent and configdrive living together? 17:56:42 It's optional... 17:56:52 since cloudbase-init (aka cloud-init for Windows) is already running as a service 17:56:57 if you don't have it in your VM.. you can't password-reset or reconfig networking without rebooting 17:57:06 etc 17:57:10 comstud: correct 17:57:18 I am thinking, what if we start again, what would we build... 17:57:24 good point, it is optional 17:57:28 comstud: IMO "reset-password" shoud throw an Exception if the opration fails 17:57:42 right 17:57:43 due to missing agent code 17:57:49 I guess cloud-init takes configdrive or metadata stuff and does things 17:57:57 johngarbutt: correct 17:58:09 ideally we would like to add another alternative: hypervisor transport 17:58:19 (such as xenstore) 17:58:27 the other thing is when do start cloud-init 17:58:39 that is either at boot, or maybe via an agent 17:58:42 My 2c are: configdrive/metadata RO, hypervisor transport RW 17:58:51 what the agent does today could move to a cloud-init plugin? 17:59:06 alexpilotti: good point 17:59:25 (do ping if there is another meeting now) 17:59:34 does that sound crazy or good? 17:59:39 johngarbutt: form a first check, your agent's design looks very similar to cloud-init and cloudbase-init 17:59:45 the agent cloud become part of cloud-init, maybe 17:59:55 smoser: ping 18:00:11 yeah, that's what's been thrown around before 18:00:25 for sure we can at least write a common code base 18:00:26 things that are RO and aren't supported by cloud-init directly can be cloud-init plugins 18:00:42 and if the argument cloud-init <> agent wins 18:00:51 RW is separate agent still, comm via XenStore or whatever secure mechanism if you have to talk between hyp and guest 18:00:57 at least cloud-init and the agent become wrappers on the common code 18:01:18 well password stuff now means metadata can accept data 18:01:30 johngarbutt: I hate it 18:01:35 :-) 18:01:35 but the trigger from a two way channel is the agent 18:01:37 alexpilotti, my ears were ringing. 18:01:47 smoser: hi! 18:02:19 smoser: we are discussing about the old cloud-init vs agent 18:02:29 i just personally do not see much value in an agent that is tied to a hypervisor/cloud-platform. 18:02:38 we are over time here, we might want to nip onto #openstack-nova? 18:02:50 np 18:02:54 sure 18:03:08 thanks all 18:03:17 see some of in the other channel 18:03:19 I suggest -dev, as it's a discussion that might attract more people there 18:03:25 good point 18:03:31 #endmeeting