19:01:14 #startmeeting infra 19:01:15 Meeting started Tue Jun 11 19:01:14 2013 UTC. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:16 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:18 The meeting name has been set to 'infra' 19:01:30 helo 19:01:36 can we comment when we use #link ? 19:01:46 zul: ehlo will provide a lot more options 19:01:54 like #link agenda http:// ...? 19:02:09 i thought that didn't work 19:02:09 o/ 19:02:13 #help 19:02:14 fungi: if sendmail wasnt broken then it might 19:02:31 o/ 19:02:44 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting 19:03:03 #topic actions from last meeting 19:03:07 #link http://eavesdrop.openstack.org/meetings/infra/2013/infra.2013-06-04-19.01.html 19:03:17 mordred made a signup thingy for bootcamp! 19:03:27 Indeed! 19:03:29 mordred: but i think it's still missing some people, right? 19:03:34 Quick Q - do you prefer us to raise items for AOB now, or wait until the open discussion to raise them? 19:03:38 I'm assuming that to be the case 19:03:47 mordred: aren't there supposed to be some people from mirantis? 19:04:15 jeblair: yes. and also possibly dell and ibm (not sean) and piston 19:04:22 BobBall: typically at the end (unless you have time constraints) 19:04:40 no, that's fine :) Just some people prefer the other way - was just asking :) 19:04:42 BobBall: there's an open discussion time at the end (sometimes), but the agenda is open, so you can add items to https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting before the meeting too 19:04:42 jeblair: so, I tihnk it's going to be a crowd! 19:04:55 excellent 19:05:07 oh drat - wish I had edited the agenda to put what I want to talk about at the start :D 19:05:13 * BobBall will wait patiently :) 19:05:31 jeblair: you don't like wearing hats to meetings? 19:05:33 reed did summarize the trystack thing, but i haven't seen any responses 19:05:51 yeah, no response 19:05:52 signup thingy for bootcamp? 19:06:22 https://docs.google.com/forms/d/1ZV1Ct6GlaTioWajEVoBcahRNBHRDfBRxiLRYgmQj_lg/viewform 19:06:25 meetings need hats 19:06:29 #link https://docs.google.com/forms/d/1ZV1Ct6GlaTioWajEVoBcahRNBHRDfBRxiLRYgmQj_lg/viewform 19:06:39 hats need signup thingies 19:06:49 lloydde: a bootcamp for people who would like to become serious regular contributors to openstack-infra 19:06:58 rgr 19:07:28 defined as people who fix at least as much stuff as they break ;) 19:07:36 (hopefully) 19:07:45 zul is working on a py3k ppa; though i think at the moment we're looking at f18 for testing py3k... 19:08:11 fungi: careful, we don't want to recklessly change the criteria and lose mordred. :) 19:08:15 i think zul posted the link for that in #-infra on friday? 19:08:17 ha 19:08:29 sometimes you eat the bug, and sometimes the bug eats you 19:08:36 fungi: yeah its ready to be tetsed 19:08:52 zul: that was quantal-sane but not for precise, right? 19:08:57 zul: can you #link it here? 19:09:02 reed: ping 19:09:03 fungi: precise sane 19:09:11 clarkb: just a sec 19:09:38 #link https://launchpad.net/~zulcss/+archive/py3k 19:09:42 aha. so that gives us the opportunity to do tox -epy33 as long as we stop pip-installing things system-wide 19:09:44 sorry folks, being dragged in multiple conversations 19:09:55 its suacy python3 rebuilt for precise 19:10:13 (because im running saucy) 19:10:22 like a good boy 19:10:40 #topic Mailing list 19:10:48 reed: anything going on with the general mailing list move? 19:11:34 no relevant progress to report yet :( 19:11:42 bummer 19:11:53 #topic gearman (for zuul) 19:12:02 hey i have to qualify that now. that's cool. :) 19:12:19 anyway, i think zuul-gearman and gearman-plugin are ready for use in production... 19:12:29 jeblair: ready today or ready after a couple more changes? 19:12:51 (it wasn't clear to me after you saw the null pointer exception and if the fix was sufficient) 19:13:52 also, I am super excited about getting gearman zuul going. So many shiny things can happen after that 19:14:19 ++ 19:14:23 i'd like to make the change during a slightly less busy time... perhaps some time friday 19:14:33 that sounds great 19:14:40 and be prepared to roll the change back if it doesn't work out 19:14:43 do it right now after I just pushed changes to every single core project! 19:15:11 clarkb: you should blog about that gearman zuul thing 19:15:20 clarkb: i believe my fix for the npe is sufficient -- at least it should keep things going if something weird happens 19:15:27 jeblair: ok 19:15:33 ttx: no I think that honor goes to jeblair and zaro 19:15:54 jeblair: do we want to conflate the zuul change window with the pending project renames/cleanup, or am i jumping ahead in the meeting agenda? 19:15:58 clarkb: as long as they are as excited as you are, that works 19:15:59 clarkb: (even if it doesn't address the actual cause) 19:16:35 ttx: i am very excited. 19:16:40 this is my excited face. 19:16:48 I want to feel the excitement too 19:17:01 * mordred is weirded out by jeblair's excited face 19:17:02 jeblair: where will the other master be?: 19:17:02 ^W oops too late 19:17:03 fungi: i was thinking it would be a good idea to combine them 19:17:03 ttx: Error: "W" is not a valid command. 19:17:18 zaro: what other master? 19:17:39 jeblair: won't you setup with 2 masters or are you starting with just 1 ? 19:17:43 okay, cool. i was going to draft up an etherpad with cut-n-pasty steps for the renames, so we can integrate the zuul upgrade/rollback steps 19:17:46 zaro: oh, just one. :) 19:18:16 fungi: yeah, let's see if we can schedule all those at the same time. if we can't, oh well. 19:19:24 should we aim for friday morning, or friday afternoon? (vaguely us-time)? 19:20:02 i'm good with either... afternoons tend to be pretty light on gate activity 19:20:09 ++ morning 19:20:26 mornings less so, but if we announce in advance then it seems fine with me 19:20:43 poll? 19:20:52 clarkb, mordred: thoughts? 19:20:55 yeah I can do either, but afternoons are quieter and I don't mind spending time friday afternoon assisting 19:21:02 jeblair: I could go either 19:21:11 it might not be terrible to do morning though 19:21:20 so that we can see it in action for a while while we're all still around 19:21:30 since this is, well, a fairly fundamental change to the world order 19:21:45 Hah - Yea, Friday afternoon deploys *always* result in a whole weekend off ;) 19:21:59 Esp the major ones :) 19:22:20 i'm leaning toward afternoon.... 19:22:28 kk 19:22:35 because if we find that we have to do an immediate rollback, there's less mess to clean up. 19:22:43 i'll be around well into the night (my time) if things break horribly and rollback isn't smooth 19:23:13 #topic project renames 19:23:18 hub_cap isn't around... 19:23:25 but kiall is. 19:23:44 kiall: did you decide you wanted to do the moniker->designate rename now? 19:24:06 So - If Gerrit is going down anyway, lets rename.. That way if the incubation proposal is rejected, no need to schedule another downtime. 19:24:37 kiall: if it is accepted, we can always rename you to the openstack org whenever we rename quantum to mutnauq 19:24:44 kiall, ttx: if designate is incubated, when would it be accepted? 19:24:54 jeblair: it would be at least 2 weeks out 19:25:06 25th or so would be the TC meet 19:25:17 yeah minimum in +2weeks 19:25:17 i just pinged hub_cap in #-trove 19:25:19 okay, if we do the rename now, there is the potential to have two renames in fairly rapid succession... :/ 19:25:19 andit might take us more than one meeting to decide 19:25:41 kiall: but that mostly affects you... 19:25:50 kiall: so i'm okay with it if you are. :) 19:26:12 Ah, we'll survive 2 renames.. The less we have to take gerrit down, the better IMO. 19:26:40 So - whatever causes the least disruption to overall set of projects. 19:27:00 kiall: is friday afternoon us-time okay with you? that'll be a hard cutover for developers where they'll have to update remotes, etc, afterwords. 19:27:19 Yea, that's fine. 19:27:39 the trove crowd says they think hub_cap is afk all day 19:28:08 i asked him to hit us up in #-infra whenever he can to talk about the possibility of getting renamed friday afternoon 19:28:14 fungi: cool, thx 19:28:20 ttx: any chance mutnuaq will be accepted by friday? 19:28:35 mutnuaq? 19:28:47 zul: it's what jeblair and I decided quantum should be renamed to 19:28:47 19:28:51 mordred: the new name might be in by Friday 19:29:01 mordred: ah 19:29:02 mordred: I thought we would get it today tbh 19:29:03 * markmcclain is still waiting on legal feedback from the latest list 19:29:09 cool 19:29:16 jeblair: did Mark say anything at the staff call ? 19:29:34 ttx: mark and jbryce were not on it; no one else had heard anything. :( 19:29:47 it was a quick call 19:30:01 so anyway, mutnuaq soon come 19:30:01 heh 19:30:19 #topic Hosting gitweb on another machine 19:30:45 so gerrit's been being affected by web spiders a lot recently... 19:30:56 well, one spider (bing) 19:31:01 thanks microsoft! 19:31:19 so perhaps we should run gitweb (or gitblit or something) on another machine 19:31:23 how important is gitweb? do we need to run it at all? (I only use it to look at acl files) 19:31:27 does anyone use that gitweb? 19:31:28 * ttx 's train running, may disappear unexpectantly 19:31:34 pleia2: o/ 19:31:35 I have used it before 19:31:36 clarkb: +1 19:31:49 but I usually use github 19:31:56 pleia2: it gets lots of hits. :) 19:32:04 jeblair: from not bing? 19:32:04 pleia2: yea, more often than I would have thought I would.. 19:32:14 i think it dovetails nicely into wanting to have somewhere to point people to get copies of our git repos which aren't a commercial closed-source proprietary service 19:32:30 fair enough 19:32:33 so a lot of projects you know, maintain a git.project.org where they run git and gitweb or cgit or gitblit... 19:32:36 sticking a web interface on top of that becomes gravy 19:32:56 yeah ok. I am not against it just wondering what the use case was. sounds like there is a good one 19:33:20 it's also good for automated testing... 19:33:34 and since gerrit has gitweb links baked in (and a way to massage the urls for them?) gitweb is probably an easy sell to keep that integration intact 19:33:35 Is a better solution to rate limit and/or block spiders from hitting the UI? 19:33:57 kiall: the first robot.txts file was installed yesterday :) 19:33:58 kiall: already done, belt and braces 19:34:03 i'd rather have peoples test rigs hammering at git.openstack.org than gerrit, even with our local caching mirror 19:34:18 ++ 19:34:37 fungi: yeah, the gitweb links are pretty flexible; mediawiki links them to their gitblit server 19:34:53 in addition to the robots.txt which tells bing to cool its heels, we're still actively dropping connections from their bots with iptables 19:35:05 fungi: maybe about time to remove that... 19:35:14 fungi: lol, how are they going to read the robots.txt then? ;) 19:35:37 the robots.txt is for when the iptables rules get undone 19:36:14 and yeah, i can carefully undo those after the meeting 19:36:20 jeblair: so, external seems good- pending long bikeshed conversation about which git server thign to run 19:36:33 mordred: yes. any quick thoughts on that? 19:37:08 i guess we should look at gitweb, and probably gitblit because wikimedia use it; though that means running another java app. however, it's fast and pretty. 19:37:14 jeblair: not really. gitlab is pretty, but also heavy weight. gitweb is what we're using now. people like cgit. gitblit is java 19:38:15 kernel.org uses cgit 19:38:53 okay, so i'll file a bug and suggest that whoever takes it look at those 4 options. 19:39:04 kk 19:39:11 I'd be fine with any of them I think 19:39:30 I do like that git.wikimedia.org and git.kernel.org give indexes 19:39:39 the git.kernel.org index looks friendly to our org structure 19:39:42 the wikimedia folks apparently chose gitblit at least partially because it efficiently produces tarballs 19:40:12 ah. well, I do not care about that feature 19:40:14 i don't think that's really a requirement for us, since we currently produce branch-tip archives 19:40:24 in fact, I want to avoid people thinking that's a feature 19:40:44 I can probably look at those 4 and draft up some pros/cons for our next meeting 19:40:47 that's good to know. that could be an anti-requirement. 19:40:52 yeah, people already want to download tarballs of our stuff from github, and that's a recipe for disaster 19:41:00 yeah 19:41:03 (either don't support that or be able to turn it off) 19:41:07 ++ 19:41:34 pleia2: cool, you want to go ahead and file the bug and assign it to yourself, at least for the first part of this? 19:41:41 jeblair: will do 19:42:10 pleia2: any toci things to discuss? 19:42:24 just quick progress update 19:42:26 #action pleia2 file bug and look into gitweb-style options 19:42:31 #topic TripleO Testing (TOCI) (pleia2) 19:42:46 not a ton of news, but I can now boot the dib-created image in LXC 19:42:56 whee! 19:42:59 pleia2: does that perform much better? 19:42:59 working out some networking issues now for proper testing to make sure all the pieces actually work 19:42:59 * anteaya applauds 19:43:07 (I have dnsmasq chaos happening on my desktop right now) 19:43:20 mmm. desktop dnsmasq makes everything better 19:43:22 clarkb: yeah, lxc is zippy on hpcloud :) 19:44:00 that's it for now 19:44:11 #topic open discussion 19:44:24 BobBall: ? 19:44:25 * BobBall peers round the corner 19:44:28 hi :) 19:44:42 So - I sent an email to the list a few days ago about XenServer gating trunk 19:44:55 I just wanted to pop in to have a chat with a few of you to see what you thought the best way forward was 19:45:03 and what I can do to make some progress :) 19:45:18 what platform requirements would the tests have? 19:45:39 as compared with the current qemu-based testing i mean 19:45:39 We can run on the RS cloud - but I've not tested on the HP cloud (XenServer on top of KVM would be ... interesting!) 19:46:41 BobBall: actually, it would be interesting if that worked (kvm on kvm doesn't work), I can test if you have some instructions for how I'd go about it 19:46:49 ^^ in hpcloud 19:46:50 perhaps I don't understand enough about the current system - what do you mean by qemu based? 19:46:50 pleia2: Error: "^" is not a valid command. 19:46:57 so, dprince has automation around doing this we can look at, right? 19:47:13 dprince's automation is running on physical 19:47:15 also, we need a xenserver instance to exist outside of the context of the openstack install, correct? 19:47:32 BobBall: we're currently only running the devstack-gate tests on hpcloud because rs got very slow a while back... i think we saw something like 150% of hpcloud's runtime... 19:47:46 jeblair: > 200% in many cases 19:48:08 I wonder what xenserver on top of xenserver performance in rax cloud would look like? 19:48:16 pleia2: Sure thing - got an email addr I can send some details to? It's basically "install and set tgt=off" for XS but I don't know what'd work for KVM. It's running under Qemu in both cases, with no PV, so it _should_ work... 19:48:29 BobBall: lyz@princessleia.com - thanks! 19:49:00 Performance wouldn't be great - disk access and network would be emulated rather than PV but we might be able to mitigate with a small ram disk for the SR 19:49:23 BobBall: I have no idea what PV or SR mean 19:49:31 does xenserver require licensing? 19:49:31 and yes, mordred, nova runs in a VM for a XenServer install - might make it easy(?) to re-use VMs rather than install XS each time 19:50:06 no clarkb - no licensing - Free edition does everything that's needed and as of the next version of XenServer there will be no paid-for licenses (within the month) 19:50:15 cool 19:50:16 BobBall: I think we're going to need to learn more about XenServer 19:50:22 got it. so this wouldn't necessarily be an all-in-one devstack vm with xenserver installed in the same base vm i guess 19:50:34 BobBall: part of the problem I'm having right now is that I'm asking questoins based on words I've heard people say in passing 19:50:53 No problem at all :) 19:51:00 There are other options too that we might consider 19:51:03 that's an excellent question from fungi - is this something accomplishable via devstack? 19:51:29 in an all-in-one? or is this something that our other work on multi-machine deployments might be needed for? 19:51:31 I'm mainly interested in XenAPI - XenServer is the ideal, but we can also install XenAPI-like packages on many other bases (e.g. CentOS, Ubuntu etc) 19:51:47 Yes, devstack installs on XS - but it takes a while because it sets up a Ubuntu VM 19:52:00 i was more wondering whether xenserver can be integrated within devstack itself 19:52:13 in other words, can xenserver run within a devstack machine 19:52:24 well, I know lifeless has a todo list item to talk to rackspace about getting their xenserver install done via tripleo 19:52:38 I don't see a reason why not fungi 19:52:44 not tested that though 19:52:48 so, it's entirely possible that we might have a decent story for how this works once we have a story for how testing anything via tripleo works 19:52:56 but after all it's just a VM running in qemu! 19:53:04 in which case we already have a way to roll devstack onto a vm and run things and gate on that 19:53:05 mordred: yeah, I'm thinking this is all closely related 19:53:21 pleia2: yes. I think I'd like to avoid solving this as a special case 19:53:30 pleia2: and instead deal with it as part of the general case 19:53:34 * pleia2 nods 19:53:35 Yeah - I read the comments about tripleo but I didn't really understand the bigger picture of how it fitted together :) 19:53:49 pleia2: cool news 19:54:02 lifeless: any thoughts on the above re: xenserver? 19:54:09 mordred: quick synopsis? 19:54:21 lifeless: BobBall wants to test xenserver in the gate 19:54:41 lifeless: there are logistical issues (we need to install xenserver somewhere) 19:55:00 lifeless: and it seems like somethign that we might want to solve as an iteration of pleia2's work on tripleo testing 19:55:03 install either as VM or physical 19:55:08 cool; both devstack and bootstack will require a VM with a XenServer image in it to do that. 19:55:17 or a physical machine with ditto 19:55:23 yah 19:55:34 tripleo is probably a better harness for doing it on physical 19:55:35 well, let's leave physical machien out of the loop for now 19:55:42 as we're targeting that end2end 19:55:46 mordred: so devstack-gate supports multiple image types... 19:55:58 mordred: perhaps it's a matter of doing this: https://wiki.openstack.org/wiki/XenServer/InstallDebianUbuntu#Installing_XCP_on_Ubuntu_12.04 19:56:01 devstack knows how to setup such a VM already I think? If so thats the lowest cost path to get going. 19:56:41 hmmm - just thinking out loud... it should be quite easy to generate an image that has an installed XS with a prepared devstack VM (minus the repos of course) already installed... Then that can just be restored, started, and it's all ready 19:57:06 BobBall: well, we have machinery to do image saving and all of that 19:57:25 BobBall: the devstack-gate tests run devstack on prepapred images... 19:57:29 the thing we need to sort is what content are we putting in to the images we're going to boot in the cloud and whatnot 19:57:38 I see 19:57:41 BobBall: the test itself runs devstack because we want that to be part of the test 19:57:44 mordred: so my thoughts are that it doesn't seem all that coupled to tripleo||devstack, other than like tripleo it wants a different content on the root image 19:57:59 mordred: and tripleo thanks to lxc doesn't need that anymore. 19:58:07 agree 19:58:13 I thought it was more complex originally 19:58:20 BobBall: but we prepare images for it by caching things devstack needs onto it 19:58:37 understood 19:58:43 BobBall: I believe we could also prepare a variant with xen already installed 19:59:13 BobBall: so then the test would be "spin up vm from previously prepared image with xen installed. run devstack. run tempest" 19:59:27 BobBall: would that general approach work? 19:59:34 yup. the main question will be around performance 19:59:48 of running that image that has xen installed on a vm in the cloud 19:59:48 * olaph pretends to be a timer. ding! 20:00:02 BobBall: https://github.com/openstack-infra/devstack-gate/blob/master/README.rst 20:00:05 but I think it's a worthwhile path forward to investigate 20:00:07 yes - with one minor comment - as I said, with devstack running in a VM under Xen we'd need to set that VM up too and include it in the base image 20:00:26 BobBall: you may want to read that to familiarize yourself with devstack-gate's process 20:00:43 i think we can go over if we want, since i believe ttx cancelled the tc meeting (that's what would be following this, right?) 20:00:45 BobBall: okay, i think i'm missing something because it sounds like you're talking about an extra level of virtualization than i'm considering 20:01:14 ok - so what actions do I need to take? 1) email pleia2 with how to test XS on HP's cloud, 2) read up on devstack-gate's process, 3) test performance of tempest in RS cloud 20:01:41 BobBall: well we already know the performance and we won't be able to gate on it. I think we need to see what we can do with hp cloud 20:01:46 jeblair: XenAPI is installed on the base OS (which could be virtual), but nova and the rest of devstack run in a VM on that host 20:01:54 unless xen makes everything magically faster 20:02:04 clarkb: yeah 20:02:06 BobBall: why do they run in a vm? 20:02:19 clarkb: I meant the relative degredation of having xen virtualised for the tempest tests 20:02:34 can xenapi and nova and the rest of devstack not both run on the same host? 20:02:44 BobBall: why do they run in a vm on that host? the current system just runs devstack+nova on the cloud host. 20:03:00 mordred: its' not the xen way 20:03:10 There are a number of reasons we run in a VM - in theory we could run in dom0, yes, but that'd need quite a few changes. Things like mounting the VMs disks is easier in a VM rather than dom0 - although doable 20:03:28 mordred: you have a privileged vm instead AIUI 20:03:34 the biggest reason is historical I'm afraid 20:03:44 ok. so that sounds lke it's more work 20:03:53 *nod* 20:04:01 because all of a sudden we're back to running devstack in a vm on the host that we're running on 20:04:05 which our automation does not do right now 20:04:16 ahhh - understood 20:04:32 mordred: that's what lead you to think tripleo was a fit? 20:04:35 yes 20:04:46 because it's a similar problem space of layers 20:04:54 not that we need tripleo for xen 20:05:20 So - in summary of that - it would be significantly easier if devstack+nova were running in dom0 20:05:24 mordred: yeah, in that case, i think pleia2's work would help, but if it could run in dom0, then it would be a pretty trivial devstack-gate change 20:05:49 yes to both of you 20:06:02 We would definitely need xen-api changes to handle that, but it's eminently doable 20:06:03 mordred: (in fact, i think devananda added the hook for that already) 20:06:05 so, let's have the takeaways be that BobBall and pleia2 talk about xenserver some more 20:06:11 great 20:06:20 that BobBall learn a little bit about devstack-gate 20:06:25 *nod* 20:06:25 and that all of us brainstorm a little bit 20:06:34 sounds good 20:06:38 ++ 20:06:43 agreed 20:06:50 Works for me 20:06:53 whee! 20:06:54 is BobBall attending the infra bootcamp? 20:06:57 Sounds like I've got a fun week! 20:07:03 BobBall: are you attending the infra bootcamp? 20:07:24 I get the feeling that I should have been paying more attention when you were talking about the bootcamp earlier 20:07:53 I'd like to say yes - this is an important thing for us 20:08:11 so I'll look into it with a very probable yes 20:09:02 cool. tl;dr - June 27/28 NYC - the aim is to get people bootstrapped with the knowledge they need to start being core contributors to openstack infra 20:09:34 Wow - I'd get a trip to NYC out of it too! Now I'll definitely push for it. 20:09:46 ha 20:10:16 thanks everyone! 20:10:18 #endmeeting