19:02:51 #startmeeting infra 19:02:51 Meeting started Tue May 14 19:02:51 2013 UTC. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:52 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:02:55 The meeting name has been set to 'infra' 19:03:07 pabelanger, mordred: ping 19:03:44 public service reminder: when mass-deleting servers from the cli, make quadruply sure to use the right pattern 19:03:58 triple-checking is clearly insufficient (for me at least) 19:04:08 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting 19:04:35 #action actions from last meeting 19:04:39 #topic actions from last meeting 19:04:42 even 19:05:06 I have put up a bug day schedule on our wiki page 19:05:16 #help 19:05:18 o/ 19:05:26 meetbot? 19:05:33 #link https://wiki.openstack.org/wiki/InfraTeam#Bugs 19:05:40 metbot is in here... 19:06:09 o/ 19:06:40 however, meetbot is missing his cloak 19:06:41 each bugday is the tuesday after a milestone and starts at 1600UTC so that we can go through them before this meeting 19:06:51 yay 19:07:31 i think he came back in on a netsplit while services was still absent or something. want me to bounce the process? 19:07:32 ah, so it may just not be able to set the topic 19:07:42 fungi: nah, we'll just live without the topic changes for now 19:07:49 and assume the logging commands are working 19:07:50 k 19:07:56 * fungi checks that 19:08:09 my hope is that these bug days will become less needed and shorter as we get better at using the tracking day to day 19:08:21 I have also got hpcloud az3 in the devstack-gate rotation using the new account. I will do az2 and az1 next over the course of today assuming things rotate out quick enough 19:08:28 pleia2: ++ 19:08:32 ++ 19:08:38 meetbot seems to be logging to eavesdrop okay 19:09:01 fungi: is our foundation meeting at 1600 utc? 19:09:26 jeblair: possibly. looking 19:10:11 oh, right normally yes 19:10:20 we can shift the time as necessary. I threw something out there so that we can have this discussion :) 19:10:26 we meet tuesdays at 1600 utc (foundation staff) 19:10:39 fungi: is 1700 to early then? 19:10:43 maybe we could do it at 1700? that still gives us 2 hours 19:10:58 that works for me. I will update the wiki 19:11:01 clarkb: 1700 is fine by me 19:11:21 we can continue post this meeting if we really need to but as pleia2 says hopefully we get good at it and that isn't necessary :) 19:11:38 clarkb: testr? 19:12:24 "clarkb to ping markmc and sdague about move to testr" 19:12:37 I filed https://bugs.launchpad.net/python-keystoneclient/+bug/1177924 to track testr adoption across the projects that haven't switched 19:12:39 Launchpad bug 1177924 in python-keystoneclient "Use testr instead of nose as the unittest runner." [Undecided,In progress] 19:13:16 had a quick chat with sdague and realized he probably isn't the best person to drive this. ttx mentioned getting it targeted to a milestone so I have been trying to ping people involved with those projects to drum up support 19:13:50 which has started the ball rolling 19:14:02 clarkb: cool; it looks like there are reviews listed too 19:14:28 yup. I have resurrected mordred's python-keystoneclient change and asalkeld_ is doing ceilometer 19:14:48 oh and sbaker is doing heatclient 19:16:03 clarkb: anything else re testr? 19:16:23 I don't think so. basically going to continue arguing for it so that we can get interest and effort 19:16:31 "fungi open a bug about replacing mirror26 and assign it to himself" 19:16:32 and support as necessary 19:16:42 bug opened, worked and closed 19:16:58 woow 19:17:09 mirror26 now runs centos and the old oneiric one is gone 19:17:14 i deleted lists and eavesdrop 19:17:27 i also moved paste and planet to new servers 19:17:40 mordred did inform the tc of current testing plans 19:17:42 i saw him do it 19:17:46 mordred: any yelling? 19:17:48 righteous 19:18:06 I don't remember any 19:18:13 fungi open bug about spinning up new precise slaves, then do it 19:18:22 that happened. 19:18:25 he did it twice ;) 19:18:27 twice, even, for good measure 19:18:30 yeah 19:18:37 collateral damage aside 19:18:39 all done 19:18:48 and the right quantal slaves are deleted now 19:18:52 "clarkb to get hpcloud az3 sorted out" 19:19:05 (not the quantal slaves which were named precise[0-9]+) 19:19:24 that seems to be mostly sorted. trying to pay attention to test results to make sure az3 isn't failing devstack gate tests for weird reasons 19:19:44 clarkb: cool. probably good to double check the runtime too 19:19:48 yup 19:19:52 "fungi add change to disable nova py26 tests for folsom" 19:20:15 that was done when the remaining jobs were changed over to centos6 node labels 19:20:26 annegentle: ping 19:20:32 and that bug is wrapped up and closed out now as well 19:21:21 cool. i think there was some overlap with agenda items and items from last meeting 19:21:31 so i think we're nominally ready to skip to the docs topics 19:21:45 though i think clarkb wanted annegentle here for those... 19:21:58 yeah particularly the for the one about publishing old docs 19:22:23 basically do we want to publish the docs based on the eol tags? or just not bother? 19:22:24 clarkb: do we have a quorum for the maven topic? 19:22:38 jeblair: I think so 19:22:54 clarkb: yeah, good question for annegentle. :) 19:23:00 #topic clouddocs-maven-plugin internalization 19:23:08 #link https://bugs.launchpad.net/openstack-ci/+bug/1082812 19:23:09 Launchpad bug 1082812 in openstack-ci "Move rackspace/clouddocs-maven-plugin to openstack-dev/clouddocs-maven-plugin" [High,Triaged] 19:24:02 so I am a horrible person and forget names, but do we want to make clouddocs-maven-plugin a stackforge project and have them drive most of the things or put it under the openstack-infra umbrella and make things go? 19:24:44 clarkb: openstack-infra i think. 19:24:53 clarkb: is it a problem that https://github.com/rackspace/clouddocs-maven-plugin does not exist? 19:25:13 jeblair: yes, I think that means it is currently a private repo 19:25:14 jeblair: sorry - got distracted - no yelling, they were fine with it 19:25:24 i think it moved to https://github.com/rackerlabs/clouddocs-maven-plugin 19:25:27 ah 19:25:38 David Cramer authored a commit a day ago 19:25:46 clarkb: i think it was david cramer we talked to. :) 19:26:04 ok, in that case I can whip up a change to slurp it in and have dwcramer give us the go ahead 19:26:20 dcramer_ seems to be in here... the same? 19:26:21 the other piece to that is publishing it to maven nexus, zaro made that sound easy so I am not too worried about it 19:26:21 clarkb: my notes said that the release process would be: 19:26:28 generate two commits (remove snapshot, inc+add snapshot) to pom.xml 19:26:33 then tag the correct one after merging 19:26:44 (inc == increment version number) 19:26:53 clarkb: theoretically 19:27:00 yup then the tagging will kick off a maven nexus publish build 19:27:36 (it is worth noting that this release dance is one of the reasons Gerrit is trying Buck) 19:27:48 and all of this is probably very similar to what we should need to do for the gearman plugin 19:28:45 clarkb: eot? 19:28:47 yes. teaching jenkins/gerrit about mvn release process - or teaching mvn about gerrit process is the game 19:28:48 yup 19:29:05 #topic What can infra do to enable remote participation at the next Design Summit? 19:29:10 reed: ping 19:29:14 ping 19:29:23 I can't play that game 19:29:45 we have basically two issues for the next summit: 19:30:07 - we have never found a good technical solution for remote participation at the design sessions 19:30:37 - we only practiced IRC for meetings as a community 19:31:24 I think we should have a way for the community to practice how to hold discussions across different places 19:31:57 before I start the conversation online where I know people will suggest we use google hangout, i'd like to explore what the infra team can do 19:32:35 the main alternatives I see to hangout/skype are XMPP+jingle if we want video and mumble/murmur 19:32:39 is the problem we are trying to solve general realtime collaboration or realtime collaboration specific to the summit because I think they are different 19:32:43 *different problems 19:32:56 thinking about the summit design sessions, what goes on is mostly a three-way combination of voice discussion, slideshow presentations and etherpad note taking 19:33:03 clarkb, the issue is specific to summit 19:33:15 in the general case you can assume everyone is using the same set of tools, but in design sessions half of your attendance will be in one place and the other half will use the tools 19:33:20 and to user group meetings, if you want 19:33:55 etherpad is already remotely joinable, so we basically need something to bridge the gap for voice and slideshows i think 19:34:14 fungi: i don't think slideshows are that important. they should be discouraged for design summit sessions anyway. 19:34:18 we could demand no slide decks to solve half that problem :) 19:34:21 jeblair: jinx 19:34:24 jeblair: ++ 19:34:26 jeblair: granted, i agree with you on that 19:34:33 fungi, faces would be a nice addition 19:34:33 or post slides somewhere you can download them ... 19:34:55 slides can also be done in html :) 19:34:58 sure. 19:35:09 so it's the voice and video part. 19:35:16 asterisk can do it 19:35:29 russellb: you have a highlight on... XMPP ? 19:35:36 russellb, SIP? 19:35:50 reed: SIP and XMPP at the same time, if you'd like 19:36:01 does that place unreasonable requirements on the network at the conference center? they never seem to handle things well 19:36:06 consider we'll be in Hong Kong, so networking can be challenging, too :) 19:36:10 how about focusing on just audio -- i'm a little concerned about bandwidth and network reliability... 19:36:19 jeblair: ++ 19:36:25 any voice+video (*especially* video) solution is going to be bandwidth hungry 19:36:35 clarkb, god only knows what will happen there... but we can make them pay super-serious attention to that 19:36:38 just audio is even easier 19:36:39 without faces, i think we still need something which will pop up a subtitle indicating who's talking 19:36:55 one-way audio with IRC feedback is what Ubuntu used to use at UDS 19:36:59 I am not familiar with asterisk but mumble does rooms like irc channels 19:37:25 with a projection of an IRC channel all the time, difficult to ignore 19:37:30 ttx: that seemed very reliable 19:37:48 critical participants can be hooked in in two-way voice, if necessary 19:37:59 having been on my share of two-way conference calls, it gets unmanageable once you have 10 or more people trying to keep from talking over top of one another, without a moderator to voice and devoice them 19:38:02 i like that. 19:38:13 the main drawback of that is that I can't see how we can have that system tested/deployed before the Summit 19:38:20 I explain: 19:38:24 I mean, the session moderator knows if he has a ccritical participant and can hook him in 19:38:38 everyone else can have one-way audio + IRC 19:38:49 my idea is to ask user groups to test the remote system we put in place so we get to HK with practice and knowledge 19:38:49 ttx: +1 19:38:50 just need decent audio equipment so that it's actually usable ... 19:38:51 that sounds reasonable to me 19:39:32 we could even abandon the idea of lurkers on one-way audio + IRC and just support critical participants in a ad-hoc manner 19:39:36 I don't see how IRC+audio streaming can be made enough for user group Atlanta to chime in the meeting in Chicago, for example 19:39:50 russellb: we had "decent" audio equipment in san diego (not awesome, but usable). the techs who wired the rooms did a less than perfect job though 19:39:53 reed: i think we could still test that in the manner you are discussing 19:40:10 frankly, you know when you've a critical person missing from the discussion. 19:40:12 jeblair, I think we need video, too 19:40:13 fungi: was that when we had the webex things? 19:40:29 russellb: yes, webex itself is crap though, in my opinion 19:40:44 needing video, but being concerned about conference internet, is going to be a bit concerning 19:40:51 reed: the thing we can't test is how the network in hong kong is going to perform, which is why i think focusing on resilient low-bandwith protocols is our best way to prepare for that. 19:40:54 so I think having decent phones in each room that a small set of people, invited by the PTL, can call in... 19:41:07 is all we need from a technical perspective 19:41:22 russellb, we will take care of the infrastructure later, once we have the specs ready... it's a matter of writing down the contracts 19:41:23 one-way audio + IRC is a plus 19:41:53 just a phone is a bit of a pain, not everyone will be able to hear it, and the people on the phone wont' hear the whole room 19:42:02 reed: wasn't there already a spec for the network? i seem to remember some bandwidth number is the proposal responses from the venue 19:42:05 not much better than someone's laptop 19:42:06 Personally I don't mind if we lose the US lurkers. We'll have the APAC ones in the rooms to replace them. 19:42:19 jeblair, the bandwidth was never an issue afaik 19:42:26 problem is the lurkers complain when they can't hear something 19:42:51 AFAIK we never had issues at the summit with streaming videos 19:42:57 russellb: that's why, personally, I would get rid of the intermediary setup 19:43:24 russellb: have a solution for critical participants if we need them 19:43:25 the problems users saw at this summit were for few failed hardware and some closed ports 19:43:52 russellb: but not try and fail at something that pretends to enable remote participation 19:44:41 in san diego, i had to investigate and restart webex on average twice a day, and i was only responsible for one room 19:44:58 is there a system that we can put in place that supports chat, audio and video and, if networking fails, drop video? 19:45:16 ttx: so that's a big first question we need to answer: do we want to enable general streaming for everyone, or focus just on the critical participants (invite only)? 19:45:33 I think chat is IRC and should remain IRC as we already have a ton of tooling around it and it is familiar 19:45:36 fungi, what was the cause of that? networking or webex failures? 19:45:43 jeblair: yes 19:45:45 reed: networking and hardware failures 19:46:07 reed: my room had the laptop that kept locking up too 19:46:25 so we need something that can basically run as a daemon 19:46:28 jeblair: I think we can do a good job at handling critical participation... not convinced we can build a good story for general remote participation... and doing a bad one is worse than not doing it 19:46:31 headless preferrably 19:46:48 ttx: indeed, things will go wrong, so we should keep it as simple as we can 19:46:53 ++ 19:47:10 ttx: for critical participation we're investigating the possibility to pay for their travel 19:47:12 so the critical-participants-only solution seems to be in line with the atc-only physical access to the rooms. open remote participation is likely to draw non-participant lurkers 19:47:47 reed: it's a bit difficult though. some people don't know they are critical until the session is up 19:47:51 much in the same way that discouraging non-atc participants helped a bunch in portland 19:47:57 flying them on short notice is kinda expensive 19:48:07 ttx, got it, make sense 19:48:22 in all cases you'll have critical people that won't show up 19:48:30 even when it's in the US we have that scenario 19:48:50 yeah 19:49:03 so what you're suggesting is to ... 19:49:27 so having a conference phone and bridges set up in the rooms might be all we need 19:49:30 proceed like we did in Portland? nothing globally, and ad-hoc per room? 19:49:36 dedicated conference bridge number per room, with a conference room phone 19:49:43 that ^ 19:50:05 and dedicated irc channels and etherpads per session 19:50:15 reed: which isn't ad-hoc at all. i definitely think we should spend some time setting this up and making sure it works. 19:50:19 or per room. eg somewhere people can find the info if mordred doesn't do it ahead of time 19:50:55 jeblair, not very sexy, but we can throw it out there 19:51:26 reed: i love things that work, regardless of how sexy they are. 19:51:31 i'd rather have functional than sexy 19:51:36 or what jeblair said 19:51:41 clarkb: re: dedicated irc channels -- that's a bit useless as we ignore them 19:51:41 you guys 19:52:01 ttx: yeah I am just thinking that a lot of the sessions don't have a remote meet point until they start 19:52:04 ttx: if we projected them like UDS 19:52:16 ttx: so we need someway of making sure that the important participants know what things are ahead of time 19:52:20 clarkb: it's actually harder to ignore the etherpad, and the chat there 19:52:21 alternatively, we could encourage the use of the etherpad chat 19:52:25 yeah, projecting them would make it work, i think 19:52:25 irc or etherpad etc 19:52:33 we'll need a webpage to summarize it all 19:52:48 sure. We can have a box to project the etherpad in each room 19:52:50 the agenda will have to contain the IRC channel and the phone number for the room 19:53:17 reed: hmm. If too many people join the bridge it will be useless for the critical people 19:53:20 etherpad chat is ok too though, really ... everyone is looking at etherpad already anyway, and it's archived with the notes 19:53:28 I'd rather have the moderator invite critical people in 19:53:30 we'd still need a little more cultural pressure to not ignore them. i saw a tendency in some design sessions to not pay attention to the etherpad chat window even though it was on the projector 19:53:35 I still think that we won't find a way to test this system with the user groups, it's not useful for that use case 19:53:35 you can set up a bridge to be listen-only by default unless you have a code to be a talker 19:53:55 russellb: should we attempt to use ip telephony or pots? 19:54:05 russellb: ah, interesting 19:54:28 either ... ip should be fine, but depends on reliability of your network 19:54:33 russellb: it's just that... I suspect we would set up the phone only in sessions where we know someone is missing 19:54:34 (of course) 19:54:58 ttx: if we're doing this with asterisk, we might be able to scale out listening to a large number of listeners, and allow them to listen in-browser, etc. 19:55:08 also ip would be way cheaper. 19:55:19 (so lurkers aren't using up telephone resources) 19:55:19 ok, sounds good 19:55:23 jeblair: that means setting up the phone call at every session 19:55:41 ..or you would just keep it running ? 19:56:02 how do you suggest we test this? 19:56:17 * ttx remembers reed running between rooms to check that the Webex stuff was still up 19:56:22 we could try running this meeting with the system one day 19:56:34 i think it would just be a conference phone dialed into the bridge number for that room as the call moderator, with written instructions on how to redial if it's disconnected 19:56:40 not exactly the way I intended... 19:57:25 so to summarise 19:57:45 you're suggestion Asterisk server/SIP and phones in each room 19:58:00 russellb: would you be willing to help? 19:58:04 yes 19:58:14 i intend to ask pabelanger too when he's around. :) 19:58:22 now that's half a dozen international calls from hong kong to wherever our bridge number is, connected 8 hours a day for 5 days, more or less. not sure what the expense winds up looking like for that 19:58:24 I'm still not convinced this is the best way to proceed 19:58:43 fungi: via IP though, free 19:58:48 hopefully 19:58:49 fungi: that does make trying to use SIP compelling. 19:58:49 but I do note the convenience of good old telephones 19:58:52 unless we do voip over the conference network yes 19:59:17 and ... need to check regulatory issues 19:59:20 some countries block voip :( 19:59:25 i note the stability of good old telephones vs voip across the world from a conference 20:00:12 we should probably found out how much pots calls would cost from the facility 20:00:15 russellb: but this is Hong-Kong... oh wait. 20:00:18 voip participants make sense, but for the conference room an actual phone line per room would likely be less finicky 20:00:31 okay, we're at time... 20:00:43 I think we can keep discussing this next week, I'll start a thread on the list, probably 20:00:43 ttx: is there a meeting in here now? 20:00:49 jeblair: no. 20:00:52 the TC... 20:01:05 you can overrun... a bit. 20:01:05 canceled today 20:01:08 No TC meeting this week 20:01:21 want me to bounce the bot real quick between meetings? i can do it after we wrap this one up i guess 20:01:22 reed: yeah, let's do that; we'll resume next week 20:01:45 yeah, I think we wont' go too far anyway and we have time :) 20:01:47 reed: and try to get people who are willing to help here 20:01:56 yep, will do 20:02:01 thanks everybody 20:02:11 i'm certainly willing to arrive at the summit early and help set up 20:02:18 as am i 20:02:44 ok, gotta go 20:02:47 ok, thanks 20:02:56 thanks everyone, see you next week 20:03:02 #endmeeting