19:02:29 <jeblair> #startmeeting infra
19:02:30 <openstack> Meeting started Tue Mar  5 19:02:29 2013 UTC.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:31 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:33 <openstack> The meeting name has been set to 'infra'
19:02:39 <jeblair> before fungi falls asleep on the desk
19:02:47 <fungi> heh. that was pushups
19:02:50 <fungi> i think
19:03:01 <fungi> could be headdesk
19:03:07 <jeblair> fungi: it's all about context
19:03:11 <clarkb> new tie fighter
19:03:13 <jeblair> #link http://eavesdrop.openstack.org/meetings/infra/2013/infra.2013-02-26-19.02.html
19:03:19 <jeblair> #topic last meeting
19:03:36 <jeblair> fungi: did you disable the old wiki some time after march 1?
19:03:53 <fungi> um, it's still sometime after march 1st right?
19:03:59 <jeblair> fungi: and will be for some time!
19:04:00 * fungi does that right after the meeting ;)
19:04:14 <jeblair> fungi: so you'll just shut down services, yeah?
19:04:42 <jeblair> oh, it's disabling an apache vhost, iirc
19:04:47 <fungi> jeblair: i was just planning on commenting back out the scriptalias and aliases i'd added back to the vhost config on it to make it work before
19:04:55 <jeblair> since we still have tarballs there.
19:04:58 <jeblair> fungi: cool.
19:05:16 <fungi> i think ttx may have still had other stuff being served from that vhost or something
19:05:29 <jeblair> mordred: did you yell at hpcloud to fix the openstackjenkins account (az3 doesn't work)?
19:05:38 <mordred> I did
19:05:44 <mordred> so far, nothing
19:05:55 <jeblair> fungi: yeah, i believe ttx is working on making the release status stuff nicely installable
19:06:05 <jeblair> mordred: awesome.
19:06:15 <jeblair> #action mordred yell at hpcloud some more about az3
19:06:22 <pleia2> hehe
19:06:49 <jeblair> #topic CLA
19:07:07 <fungi> it happened
19:07:10 <jeblair> yay!
19:07:14 <fungi> fallout continues
19:07:33 <fungi> generally successful, but as always communication is never perfect
19:07:54 <pleia2> I got a few DMs on twitter I was able to help with, so I'm glad I announced it there (@openstack retweeted)
19:07:58 <fungi> so we have lots of people and some percentage are bound to get confused
19:08:15 <pleia2> no actual problems though, all just email mismatch
19:08:23 <clarkb> >200 people seem to have successfully done it though
19:08:35 * fungi checks the number again real fast
19:08:36 <clarkb> which I think is a success is for just a few days
19:08:49 <clarkb> pretend that second "is" never existed
19:09:06 <anteaya> clarkb, I have a filter for such things
19:09:08 <fungi> 294 now
19:09:11 <pleia2> woo
19:09:13 <anteaya> nice
19:09:21 <fungi> we'll break 300 today utc easy
19:09:29 <jeblair> that's 58% of ATCs for an entire release cycle.
19:09:46 <fungi> i guess some people do actually work on this project. good news!
19:09:51 <pleia2> :)
19:10:35 <mordred> yeah. that's actually a nice stat to keep in our back pocket, when people ask "how many of the ATCs are _real_"
19:10:36 <fungi> todd said he was going to try to make a change to help some of the e-mail mismatch issue this week
19:10:51 <pleia2> great
19:11:21 <fungi> if someone wanted to do a time-trended cla signing chart, i could provide raw data
19:12:01 <fungi> you can infer some level of activity of different percentages of the devs from how long it took them to sign
19:12:11 <pleia2> cool
19:12:16 <zaro> sorry, my pc clock is stuck on 10:58.
19:12:25 <anteaya> fungi, would graphviz work for that?
19:12:44 <fungi> anteaya: it might, though it would probably be overkill. gnuplot would be my go-to there
19:12:47 <pleia2> I'm thinking something more like gnuplot
19:13:12 <anteaya> fungi, pleia2 I haven't worked with gnuplot but it if takes less than 2 hours to do, I will do it
19:13:30 <clarkb> iirc you give it a csv and it draws graph
19:13:46 <fungi> anyway, i think that's generally it for cla stuff. oh and we trimmed the icla preamble and made the system and usg cla descriptions more scary yesterday
19:13:46 <pleia2> I seem to recall it being pretty easy to do simple stuff
19:14:06 <clarkb> fungi: maybe we can add gerrit specific instructions to the page?
19:14:18 <anteaya> if I can get something out of spending 2 hours max on it, then we will have a chart
19:14:21 <clarkb> eg email must match foundation email and you need to enter contact info
19:14:48 <fungi> clarkb: perhaps... though i still doubt the people who had issues would have succeeded had that been in there
19:15:01 <fungi> but at least we can point to it and say we did all we could
19:15:08 <clarkb> no, but if we have it then we can at least ask people to read the docs
19:15:19 <fungi> my thoughts as well
19:15:34 <jeblair> gnuplot is not Free Software, which is actually why I'm more familiar with R.
19:15:52 <clarkb> really? but it says gnu
19:15:53 <fungi> gnuplot is nonfree? i should pay more attention to my vrms
19:16:03 <anteaya> jeblair, what is R.?
19:16:05 <jeblair> fungi: it's dfsg and osi free.
19:16:10 <fungi> ahh
19:16:27 <jeblair> fungi: but not rms free.  you have to distribute patches, you can't distribute modified source.
19:16:41 <fungi> ugh. how qmail of them
19:16:47 <clarkb> interesting
19:16:47 <jeblair> fungi: it's not gnu software, they just thought the name was cool.
19:16:53 <fungi> well, sure
19:16:55 <pleia2> huh, I didn't realize
19:17:02 <jeblair> anteaya: statistics package.  like "S".
19:17:13 <anteaya> kinda like vitamin water, that isn't
19:17:23 <anteaya> jeblair, hmmm
19:17:39 <fungi> yeah, r is also a good choice. tons of the cran extensions are packaged in debian for a long time
19:17:42 <clarkb> I do have one more CLA related question
19:18:07 <clarkb> the corporate cla and government cla process has not changed and no resigning of them is necessary, correct?
19:18:14 <anteaya> well, I will focus on creating a graph with fungi's data for 2 hours, and if I am successful, we will have a graph and if not we won't
19:18:21 <anteaya> how's that?
19:18:25 <mordred> anteaya: ++
19:18:30 <clarkb> fungi: there has been some confusion of that. you may want to check the mailling list and respond to those questions if you haven't already.
19:18:32 <anteaya> thank you
19:18:41 <fungi> clarkb: the process has not changed, but the contents of those agreements may have. that's left between corporate or usg departments and the foundation to suss out
19:18:48 <clarkb> (I am still trying to catch up on mail this morning)
19:18:56 <fungi> i will take a look at the ml. got behind on that today
19:19:29 <fungi> it's mid-afternoon here and i'm still trying to catch up on e-mail (failing too)
19:19:55 <jeblair> yeah, i'd rather we not wade too much into that other than to direct people to contact the foundation
19:20:03 <fungi> that's the plan
19:20:07 <clarkb> ++
19:20:16 <fungi> hopefully one of them will have already responded by the time i read it
19:20:29 <jeblair> i mean, actually, i'd love for us to be the legal department, because things would get a whole lot simpler.  :)
19:20:38 <jeblair> but at the moment, i think we aren't.
19:20:53 <fungi> i was getting a bunch of similar questions in #-dev yesterday and people wouldn't take "talk to the foundation" as an answer
19:21:08 <fungi> because, you know, that's effort
19:21:46 <jeblair> right, the place to go for legal advice regarding a contract signed between two corporations is a guy on irc.
19:21:53 <pleia2> hah
19:22:15 <fungi> apparently. i try not to understand the thought process there too much
19:22:23 <pleia2> I'd love to see that conversation with the lawyers "fungi said it was ok" "...fungi?"
19:22:32 * fungi hides
19:22:36 <pleia2> :)
19:22:37 <fungi> okay, other cla questions?
19:23:01 <fungi> put a fork in it. next topic?
19:23:03 <jeblair> fungi: do you think the email mismatch thing is widespread enough we should do more communication?
19:23:11 <fungi> ahh, possibly
19:23:24 <fungi> we had a thread which covered it on the ml, but a separate announcement could help there
19:23:32 <clarkb> ++
19:23:36 <jeblair> fungi: or do you think we can just handle it one at a time until the profile system is updated?
19:23:48 <fungi> most of the people who were taken aback by it admitted up front they don't read the ml though
19:23:55 <pleia2> yeah
19:23:55 <clarkb> a dedicated thread makes it googleable too
19:24:18 <jeblair> couldn't hurt i guess.  is the wiki updated with that tidbit?
19:24:32 <fungi> also, todd's changes probably won't help the larger issue which is people don't know what e-mail address they used with the foundation when they signed up a year ago
19:24:49 <fungi> i think the wiki states it explicitly, but i'll make sure and fix it if not
19:24:57 <fungi> and send an announcement in summary
19:25:09 <jeblair> cool
19:25:15 <jeblair> #action fungi send ml announcement highlighting email matching for cla
19:25:30 <mordred> also, tweet a link to the message
19:25:37 <mordred> because apparently some people do read twitter
19:25:45 * fungi tweets mordred a new buzzword
19:25:55 * mordred fungis a new tweet
19:26:02 <clarkb> it should rain wherever mordred is today
19:26:09 <pleia2> ah yeah, I'll tweet the msg
19:26:13 <fungi> someone in here will likely be more than happy to tweet links to e-mail messages
19:26:19 <fungi> thanks pleia2!
19:26:24 <jeblair> #topic jenkins slave operating systems
19:26:50 <jeblair> quantal: i think we probably don't care about jclouds wrt to quantal anymore, yeah?
19:26:59 <clarkb> I don't think we do
19:27:00 <fungi> sounds not
19:27:04 <jeblair> since we decided to stop using jclouds because it keeps giving us bad slaves
19:27:15 <fungi> and rackspace has been a bit of a black hole on that quantal ticket anyway
19:27:17 <clarkb> does devstack-gate use the same metadata?
19:27:25 <clarkb> I don't think it does (it uses explicit image names iirc)
19:27:30 <jeblair> clarkb: it's basically the same as the launch script
19:27:39 <fungi> i updated it again yesterday and still no word back since mid last week
19:27:43 <jeblair> which i think fungi has sucessfully used to launch quantal, yeah?
19:27:47 <clarkb> jeblair: ya
19:27:59 <fungi> clarkb: jeblair: yeah novaclient's fine with it. jclouds is not
19:28:13 <fungi> so no worries there
19:28:21 <fungi> rhel is very very close now! i need to see if we got more eyes on dprince's last few changes
19:28:27 <pleia2> \o/
19:28:35 <clarkb> fungi: you did, but I didn't want to approve right beforethe meeting
19:28:40 <jeblair> fungi: so with that in mind, does that mean we're ready to spin up a small pool of quantal slaves and move some projects over?
19:28:57 <jeblair> fungi: i guess they should be 8g slaves too, so we get the 4vcpus we need.
19:28:58 <fungi> jeblair: sure. we have 4 but i can add more
19:29:11 <fungi> oh, well quantal1-4 are 2g
19:29:18 <fungi> i can trash them and re-spin
19:29:29 <jeblair> fungi: yeah, let's do that, and maybe spin up 8.
19:29:47 <fungi> okay so 8x 8g quantal jenkins slaves coming up right after the meeting
19:30:22 <fungi> and if dprince's last changes go through then maybe i'll have a rhel6 slave on jenkins for us too
19:30:30 <jeblair> groovy!
19:30:39 <fungi> assuming nothing else unexpected crops up tehre
19:30:42 <fungi> er, there
19:30:53 <jeblair> #action fungi spin up quantal slaves
19:31:25 * fungi is getting dizzy already
19:31:31 <jeblair> #topic grenade
19:31:54 <jeblair> dtroyer_zz: ping
19:32:12 <jeblair> so there's a review proposed to add grenade to gerrit
19:32:22 <fungi> his nick makes me think a pong is not to be expected for a while
19:32:40 <jeblair> though dtroyer_zz hasn't left feedback on it yet
19:33:10 <jeblair> with that in place, we should be able to start testing it (non-voting).
19:33:26 <jeblair> #action jeblair wake up dtroyer and ask him about grenade
19:33:35 <markwash> dtroyer_zz should be back soonish (20 min?)
19:33:57 <fungi> just in time for the end of the meeting ;)
19:34:03 <markwash> like a boss
19:34:09 <clarkb> office wifi has me on my phone now
19:34:10 <jeblair> heh
19:34:31 <fungi> clarkb: you should find a job with a technology company
19:34:36 <jeblair> #topic gearman
19:34:36 <pleia2> zing
19:34:58 <jeblair> zaro: progress is being made, i see!
19:35:14 <zaro> yes.
19:35:25 <jeblair> zaro: i think we should start trying to get a handle on the gearman queue cancel problem
19:35:40 <jeblair> since that's something that needs changing in gearman
19:35:53 <jeblair> mordred: did you mention gearman plugins last week?
19:35:56 <clarkb> zaro is experiencing the same network trouble
19:36:13 <fungi> the aforementioned wireless network i guess
19:36:27 <fungi> ahh, you said same, not some
19:37:17 <jeblair> 19:40:18 <mordred> in any case, I think he said that someone made a gearman extension to do it at some point...
19:37:33 <mordred> yeah. something like that
19:37:46 <zaro> ok. i think wifi is better now.
19:37:55 <jeblair> mordred or zaro: have either of you been able to confirm that's a real thing, or was mordred making that up?
19:37:56 <zaro> yes. will talk to brian again.
19:38:00 <mordred> but we haven't gotten all that far figuring out what that extension is
19:38:02 <mordred> cool
19:38:24 <mordred> jeblair and I also discussed that we should go through the mental exercise of ensuring that there isn't a way to achieve our needs without the extension
19:38:25 <jeblair> okay, let me know if i can help.
19:38:27 <mordred> just to be sure
19:38:51 <mordred> and also, so that when we bug people about it, we have all of our bases covered for crazy questions they'll ask "why don't you just ... "
19:39:07 <jeblair> mordred: i believe zaro has some emails from me on the subject.  if you need me to, i can try to dig them up.
19:39:27 <jeblair> zaro: one thing i noticed is that there's some stub methods to have the jenkins plugin cancel queue items..
19:39:41 <jeblair> zaro: but i don't think that's strictly necessary; i think we'd want zuul to do it directly
19:40:31 <zaro> i thought zuul just tells the plugin to cancel but the plugin actually does the work?
19:40:57 <jeblair> zaro: we need to cancel items on the gearman queue before they are picked up by jenkins
19:41:03 <jeblair> zaro: so it's extra work to involve jenkins in that
19:41:14 <zaro> cool! i'll remove.
19:41:27 <jeblair> zaro: (this is separate from jenkins being able to abort running jobs, or even cancel items on the _jenkins_ queue)
19:42:13 <jeblair> zaro: we can talk after the meeting if we need to go into more detail
19:42:29 <jeblair> #topic reviewday
19:42:32 <zaro> sure.
19:42:44 <jeblair> pleia2: how's it going?
19:42:52 <pleia2> just wrangling some final variable issues to make the reviewday module full indepentent of openstack specifics
19:43:20 <pleia2> should be done soon
19:44:17 <jeblair> pleia2: cool, you have everything you need?
19:44:24 <pleia2> yes, thanks
19:44:29 <jeblair> #topic Pypi mirror / requirements
19:44:46 <fungi> mirroring has been reworked per our brainstorming on friday
19:45:08 <jeblair> fungi: thanks!  it looks like we now have a mirror with everything!
19:45:10 <fungi> seems good so far, though we're still ironing out minor issues in projects and a few oversights in the script
19:45:12 <mordred> woot
19:45:16 <clarkb> now up to 729 packages. one problem with a project requiring babel twice
19:45:29 <mordred> really? how did that work at all ever?
19:45:32 <fungi> yeah, i agree we should fix that in the project in question
19:45:51 <clarkb> mordred we think tox pip installs differently
19:45:53 <fungi> mordred: tox might not care because it could invoke pip install differently
19:46:12 <jeblair> i believe this change just ran with _only_ our mirror configured for the devstack jobs: https://review.openstack.org/#/c/23579/
19:46:17 <jeblair> and succeeded.
19:46:33 <fungi> clarkb: similarly did a few unit test runs for projects against it yesterday, i think
19:46:46 <anteaya> yay
19:46:52 <clarkb> yes nova and zuul master were happy
19:46:53 <jeblair> it turns out, setting that up is easy.  but using only the mirror for unit tests, etc, is harder.
19:47:00 <jeblair> #link https://review.openstack.org/#/c/23513/
19:47:03 <zaro1> me on phone too
19:47:17 <jeblair> that's one way we could handle it.  it's not nearly as elegant as i'd like
19:47:30 <jeblair> particularly since it involved tracking down everywhere we invoked tox.
19:47:38 <fungi> i meant to ask, are there global configuration files we could be using for those instead of multiple per-user ones?
19:47:53 <jeblair> fungi: good question, but if there are, it's even harder.
19:47:57 <fungi> like an /etc/pip.conf et cetera
19:48:09 <jeblair> fungi: because we use the same slaves to run openstack, openstack-infra, and stackforge.
19:48:28 <jeblair> and if we start using the mirror to enforce openstack/requirements, those are 3 different mirror configurations
19:48:37 <jeblair> and we don't know what should be in place until the job starts.
19:49:04 <jeblair> (unless we start having dedicated infra and stackforge slaves)
19:49:05 <fungi> right. i think someone mentioned multiple parallel mirrors (could be linkfarms even) on the pypi server
19:49:18 <fungi> which would need per-job changes
19:49:20 <fungi> that's tough
19:49:32 <jeblair> so my complicated change above handles that
19:49:44 <jeblair> by running a script that does the user-level mirror config before any job that runs tox.
19:50:21 <fungi> yeah, makes sense. i need to finish reviewing that
19:50:37 <clarkb> jeblair: you could push the script into tox.ini
19:50:47 <clarkb> and use the tox jenkins/hudson stuff to only do it on jenkins slaves
19:51:06 <clarkb> (not necessarily sold on that idea just throwing it out there)
19:51:33 <fungi> anyway, my initial question about global configs was more in the context of your devstack-gate change not that one
19:52:08 <jeblair> clarkb: that does mean a lot of changes to tox.ini, and that's if that works.  most ideas that we have about tox.ini don't work because exactly the wrong set of things are available to the jenkins env.
19:53:05 <jeblair> clarkb: do you mean as an extra command= step?
19:53:14 <clarkb> jeblair: yes that runs before the other commands
19:54:23 <jeblair> clarkb: that require a special jenkins section, and we wanted to avoid that
19:54:34 <jeblair> clarkb: (jenkins should run what devs run)
19:54:41 <clarkb> ++
19:55:21 <jeblair> at any rate, i'd love more ideas.  that change still feels way too complicated to me.
19:55:30 <fungi> though it's worth noting that there are unit test sets which won't work if you don't use our pypi mirror at all
19:55:42 <jeblair> mordred: you should take a look at it too.
19:55:42 <fungi> as i discovered troubleshooting test breakage for nova essex
19:55:48 <mordred> jeblair: was just reading it
19:56:13 <mordred> btw - pypi upstream is moving away from external links
19:56:17 <jeblair> fungi: yeah, eventually we should address purging things from the mirror.
19:56:28 <jeblair> mordred: that's brilliant!
19:56:30 <mordred> and hpcloud may be giving them a swift/cdn account into which to put package files
19:56:42 * fungi is very happy about the upstream pypi changes discussed on catalog-sig
19:57:19 <fungi> mordred: i thought they had piped up saying they had no shortage of storage and had already settled on their cdn of choice and shut up already?
19:57:32 <jeblair> at any rate, i think next steps are to merge some version of the changes i proposed or equivalent
19:57:51 <jeblair> and then to change the mirror script to only look at openstack/requirements
19:58:27 <mordred> fungi: oh yeah? great.
19:58:29 <jeblair> and then we work on how to use openstack/requirements.
19:58:45 <mordred> I had a private message from a person about getting an account, but they may have gotten it sorted
19:58:56 <jeblair> mordred: is there an openstack/requirements summit session?
19:59:15 <clarkb> that remidns me I need to add a logs session /me is really going to do that today
19:59:21 <fungi> mordred: the discussions around storage and cdn needs were being had by people who didn't actually manage pypi, and the one who does eventually chimed in and said he has it sorted. maybe it's hp but the names thrown around didn't seem familiar
19:59:38 <jeblair> #action clarkb register logs summit session
19:59:47 <mordred> k. great
20:00:10 <jeblair> yes, there is a requirements session.
20:00:36 <jeblair> i think that's it?
20:00:42 * olaph waves bye
20:00:52 <fungi> bye olaph
20:00:57 <jeblair> thanks!
20:01:02 <pleia2> I should be good to talk about baremetal stuff soon
20:01:02 <jeblair> #endmeeting