19:01:46 <jeblair> #startmeeting infra 19:01:47 <openstack> Meeting started Tue Jun 17 19:01:46 2014 UTC and is due to finish in 60 minutes. The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:48 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:49 <reed> o/ halfway here, the other half helping martin 19:01:50 <ttx> o/ 19:01:51 <openstack> The meeting name has been set to 'infra' 19:01:59 <jeblair> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting 19:02:00 <jeblair> agenda ^ 19:02:06 <jeblair> #link http://eavesdrop.openstack.org/meetings/infra/2014/infra.2014-06-10-19.02.html 19:02:09 <jeblair> last meeting ^ 19:02:10 <nibalizer> o/ 19:02:28 <jeblair> #topic Upgrade jenkins build-timeout plugin (zaro) 19:02:39 <jeblair> #link https://review.openstack.org/#/c/97651 19:02:51 <zaro> just wanted to know what the process is for this to get done 19:03:12 <jeblair> cool, as i recall, we almost upgraded to this until we saw that the released a version that fixed a perf bug 19:03:17 <zaro> do we just wait for next opportunity or plan something? 19:03:37 <jeblair> or rather, we almost upgraded to 1.13, and saw that they released 1.14 which fixed the perf bug 19:03:44 * fungi lurks from a wireless modem in the midst of an empty townhouse 19:03:46 <jeblair> but it looks like from your commit msg, 1.14 should be safe for us 19:03:59 <anteaya> hey fungi 19:03:59 <zaro> yep. 19:04:15 <jeblair> so i think what we should do is a rolling upgrade of the numbered jenkins masters 19:04:27 * morganfainberg is lurking here as well. 19:04:36 <jeblair> an infra-core should put one in shutdown mode, upgrade it when all jobs are finished 19:04:57 <jeblair> keep an eye on that one jenkins for a bit, and if it works, maybe do 2 or more at a time after that 19:05:09 <clarkb> ++ 19:05:15 <jesusaurus> ++ 19:05:21 <fungi> sounds good 19:05:22 <clarkb> I am happy to volunteer for that once I am feeling well 19:05:25 <anteaya> do we need a slow time for this or anytime? 19:05:33 <jeblair> anteaya: anytime will be okay 19:05:37 <anteaya> k 19:05:48 <jeblair> i can pitch in after today 19:05:54 <anteaya> clarkb: think you might be better by the end of the week? 19:06:05 <jesusaurus> how many jenkins masters are there again? 19:06:08 <clarkb> anteaya: yes and if I am not I will be cranky 19:06:10 <clarkb> jesusaurus: 8 19:06:11 <anteaya> 8 I think 19:06:18 <anteaya> clarkb: no cranky 19:06:25 <jeblair> jesusaurus: 8 -- seven numbered, plus jenkins.o.o itself, which we'll probably do last 19:06:57 <jesusaurus> and how long does it usually take for a master to finish its currently running jobs? 19:06:59 <jeblair> i'll try to start on it tomorrow 19:07:02 <jeblair> jesusaurus: about an hour 19:07:22 <jeblair> this might end up taking a few days, which i think is okay, it's not urgent 19:07:31 <jesusaurus> so a rolling upgrade is at quickest a full day 19:07:41 <jeblair> yup 19:08:24 <jeblair> #action jeblair and clarkb if feeling better to upgrade jenkins timeout plugin starting june 18 19:08:41 <jeblair> zaro: good? 19:08:54 <zaro> sounds good 19:09:22 <jeblair> #topic Create release of python-jenkins? (zaro) 19:09:41 <jeblair> this project has progressed quickly. :) 19:09:42 <zaro> any plan for this at all? i think it would help jjb 19:09:54 <jeblair> zaro: who is active on python-jenkins? 19:10:12 <zaro> me, hashar and msbramo 19:11:52 <jeblair> zaro: do you think everyone is in agreement about making a release? 19:12:26 <zaro> i haven't checked that. just wanted to know if there was any plan to release anytime soon? 19:12:33 <jeblair> zaro: is python-jenkins fully pbr now? 19:12:38 <zaro> i can double check with the group and let you know. 19:13:07 <zaro> jeblair: don't remember off the top of my head, but i think there might still be an oustanding change for pbr. 19:13:33 <jeblair> zaro: i'd propose the following: 19:13:40 <jeblair> 1) merge pbr support 19:13:40 <zaro> i think this is it https://review.openstack.org/#/c/90455 19:14:14 <jeblair> 2) get agreement from hashar and msbramo about making a release 19:14:44 <zaro> can you help with 1? review i mean? 19:15:19 <jeblair> zaro: i think you and hashar both know how to push tags correctly for a pbr release yeah? if so, i think we should add you, and you can teach msbramo if he wants to join too 19:15:25 <jeblair> 3) one of you pushes a tag 19:15:49 <zaro> sounds good 19:16:18 <clarkb> ++ 19:16:24 <jeblair> yeah, i can review that. hopefully mordred too 19:16:34 * mordred can argue with mgagne ... 19:16:52 <mordred> a new version of python-jenkins is not going to get added to precise in any case 19:16:59 <mgagne> mordred: ok 19:17:06 <jeblair> i note a comment about supporting precise; if we find we need to do a security release, we could always branch from the initial import commit and make such a release manually 19:17:07 <mordred> so there's _always_ going to be a PPA involved for people who want new python-jenkins on old ubuntu 19:17:17 <mgagne> mordred: the essence of my comment is: "It's not a strong -1. I'm however warning about the work this could generate for downstream packaging python-jenkins which is a detail we often forget." 19:17:23 <mordred> mgagne: ++ 19:18:29 <jeblair> mgagne, zaro: we should probably do a version bump significant enough to note the major change introduced by pbr 19:19:03 <jeblair> (and leave room for a security point release from the current version number if we need to) 19:19:07 <mgagne> mordred: openstack is "cool" because it has special attention from downstream, not python-jenkins. So we shouldn't lightly add tools known to be exclusively used by openstack projects to non-openstack projects and hope downstream will keep up with it. 19:20:06 <jeblair> aw man, i thought we were cooler than openstack ;) 19:20:30 <mgagne> jeblair: uberstack 19:20:39 <anteaya> with umlaut 19:20:45 <jeblair> okay, we can probably continue that on the review 19:20:53 <nibalizer> überstack 19:20:58 <anteaya> there we go 19:21:00 <jeblair> if pbr doesn't land, erm, we'll do something else :) 19:21:17 <jeblair> #topic roadmap for gerrit to consume the openid provider (reed) 19:21:21 <mordred> I'm going to continue to push on that topic 19:21:36 <mordred> but I grok mgagne's concerns - so I'l see if I can write something up re: solutions 19:21:54 <reed> so... 19:21:59 <jeblair> #link http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2014-06-13.log 19:22:08 <jeblair> timestamp 2014-06-13T18:20:11 19:22:33 <reed> the current status of the openid project is that the code is working on openstackid-dev 19:23:22 <reed> the connection to the real user database is not live yet, it's pending the move of such database from cloudsites to a Rackspace virtual server 19:23:43 <anteaya> who is in charge of that move? 19:23:48 <anteaya> and is it scheduled? 19:23:53 <reed> the move has been approved and is technically without problems, it just needs to happen 19:24:06 <jeblair> reed: why not trove? 19:24:19 <reed> Todd is in charge of that move, it has not been scheduled yet (I have no ETA) 19:24:21 <mordred> ++ 19:24:27 <mordred> that was ++ to "why not trove" 19:24:35 <reed> jeblair, because trove requires a lot more changes and time we don't have 19:24:51 <jeblair> reed: it's just a mysql database 19:25:05 <reed> jeblair, I think it has to do with the deployment system that they use 19:25:29 <anteaya> can we get Todd to come to Germany? 19:25:32 <reed> jeblair, honestly, I don't know the details, trove was not considered a 'simple move' and I ditched it 19:25:54 <reed> anteaya, you can ask, not sure it would help 19:26:08 <anteaya> kk, I'll see what I can do 19:26:49 <reed> to go back to the timeline, once we move Silverstripe to RAX Managed Servers we can think about deploying to the final 'home' 19:26:59 <mordred> WOW. WHAT???? 19:27:01 <anteaya> what is Silverstripe? 19:27:28 <mordred> ok. I'm sorry. I'm not going to keep my mouth shut about that. that's the most assinine thing I've ever heard from the openstack foundation 19:27:31 <reed> that will be something like openstackid.org or openstackcommunity.org or anything 19:27:51 <reed> mordred: no, it's not, we can take that on the list if you prefer 19:27:52 <mordred> WE MAKE CLOUD SOFTWARE. the foundation should USE IT 19:28:19 <reed> mordred: yep, rackspace does cloud software too and the move is to openstack based cloud 19:28:31 <mordred> you just said "RAX Managed Servers" 19:28:48 <mordred> that's a non-cloud product that rackspace sells to people where you get a server that they help you admin 19:29:11 * mordred walks away before he screams more 19:29:14 <reed> mordred: wrong name then.. it's RAX product where they run the virtual machines for you 19:29:25 <ttx> managed cloud instances maybe 19:29:34 <reed> that's it,ttx, thanks 19:29:41 <ttx> though my mouth hurts just sayign it 19:30:06 <jeblair> reed: does the foundation have any intention of having the db and the rest of the website (silverstripe) managed as part of the infrastructure program? 19:30:12 <reed> the thing is that we need to have openid code in the same cloud as Silverstripe 19:30:13 <fungi> concerns were raised during previous discussions that the foundation wanted a vendor with 24x7 on-call support rather than something vulunteer-run 19:30:27 <mordred> reed: ^^ jeblair's question? 19:30:27 <ttx> "Managed Cloud" 19:30:28 <reed> jeblair, I can't answer for that 19:31:01 <jeblair> reed: i ask because we developed a roadmap about two years ago that involved publishing the existing site to cloud sites as an open source app as part of our workflow 19:31:17 <ttx> (http://www.rackspace.com/cloud/service-levels/) 19:31:21 <clarkb> it was san diego then again portland 19:31:23 <jeblair> reed: and the current plan you describe seems to involve the app continuing to be proprietary and hosted outside of project-acessible resources 19:31:37 <reed> jeblair, right, I think it's a matter of resources 19:31:56 <reed> we go back to the underlying issue: deploying with you costs a lot of time 19:32:14 <jeblair> reed: i have been volunteering for two years to do all of the work 19:32:15 <reed> *a lot* is a crucial word 19:32:16 <fungi> there still seem to be some interested parties in foundation management who insist on service-level agreements and a list of escalation phone numbers, et cetera 19:32:19 <mordred> jeblair: ++ 19:32:19 <ttx> jeblair: I think that's the crux of the issue. Does the Foundation want this system to be run by the project infra team, or separate 19:32:32 <reed> hi fungi 19:32:45 <fungi> howdy reed 19:32:56 <mordred> ttx: well, no. the crux of _this_ issue is if we can let a portion of the infra production system be outside of our ability to fix it 19:32:59 <reed> ttx: the crux of the issue is that these things need to move fast and infra is the opposite of fast, based on current experience 19:33:09 <reed> and that's not a judgement, to be clear 19:33:11 <mordred> it is 19:33:20 <mordred> and I'm quite literally shaking with rage at the moment 19:33:27 <mrmartin> in middle-term we could host the Foundation web related project in a paas hosted by the infra team... 19:33:30 <ttx> reed: I think in that case, slowness is not really on infra side 19:34:13 <fungi> measured, deliberate, careful, maintainable, sustainable... those things *might* be seen by some as the opposite of "fast" 19:34:27 <jeblair> reed: i will repeat one more time for the record what i've said for the past two years -- point me at the git repo with the site source code and it will be hosted in infra within 2 days. 19:34:32 <reed> mordred: we can discuss all you want, but openstack.org launched the marketplace in 3 months and openid is still months away 19:35:13 <reed> jeblair, for the records, we can keep playing this blame game or move on to the topic at hand 19:35:27 <jeblair> reed: if todd and jonathan had put 2 hours of work into the site, then i would have been able to help it move. 19:35:37 <reed> jeblair, I believe you 19:35:44 <jeblair> reed: as is, we've been told that todd has no time available because he's working on things like the marketplace 19:35:52 <fungi> mordred puppeted stackalytics in an afternoon, fwiw. i don't know that you can really say the systems automation part is entirely to blame for those sort of development timelines 19:35:57 <jeblair> it's the opposite of open collaboration and open source 19:36:02 <mordred> fungi: ++ 19:36:09 <reed> jeblair, but besides that, Todd and Jonathan and tipit would have had to put in a lot of extra hours to do things a lot differently 19:36:09 <mordred> fungi: took 1 hour 19:36:19 <fungi> mordred: so i exaggerated ;) 19:36:41 <reed> I have no doubt about your knowledge, it's the rest of the world that I'm concerned about 19:36:55 <mordred> reed: the only people who seem to have problems are Todd and tipit 19:37:10 <reed> mordred: martin has similar issues 19:37:12 <mordred> the rest of the world thinks we're pretty amazing and go out of their way to copy things we do 19:37:19 <mordred> in any case 19:37:32 <mordred> if openstackid depends on a database that's run by tipit and todd 19:37:33 <reed> mordred: i know that 19:37:42 <mordred> I'm not sure we can inject it into our production workflow 19:37:46 <jeblair> reed: okay, so to the task at hand, i don't personally have an interest in making the project infrastructure depend on _more_ proprietary non-community-run systems. i am willing to help work on making them depend on fewer ones. 19:37:47 <mordred> because I do not have operational trust in them 19:37:54 <jeblair> or what mordred said 19:37:56 <mordred> based on their trackrecord of breaking things 19:38:12 <mordred> the do not know how to run operational systems to the level we require 19:38:16 <mordred> and have proven this over and over again 19:38:25 <jeblair> i'm also losing trust because the current road map does not match what we have discussed. not by a long shot 19:38:27 <mordred> so it's a non-starter until this is sorted 19:38:35 <reed> got it 19:38:41 <reed> and it's good for me to, for the time being 19:38:53 <jeblair> and i do not think we should allow ourselves to end up in an awkward situation based on that 19:39:09 <reed> all I care is that I manage to get openid on our infrastructure first, even if it calls a database hosted on openstack, run by rackspace 19:39:10 <ttx> mordred, reed: i'll take the action of clarifying the ownership/control issue 19:39:28 <ttx> because this uncertainty has lasted for too long 19:39:36 <mordred> ttx: thank you 19:40:04 <reed> so first we have that db run on openstack and then we can insist on deploying it with scripts managed in the #infra 19:40:58 <reed> I can live with gerrit not adopting openid provider until the SS database is managed via infra 19:41:12 <reed> only warning: everybody breaks things, not only tipit 19:41:16 <mordred> reed: totally 19:41:23 <mordred> reed: we can just fix them when they break in our world 19:41:30 <mordred> reed: and I think the above seems reasonable 19:41:55 <reed> cool 19:42:50 <jeblair> reed: anything else on this? 19:43:02 <fungi> and this is not really an abnormal expectation. at a former employer, no new systems went into production until they were in deployment automation and documented such that anyone on the requisite support teams could step in and solve them when there was an issue 19:43:03 <reed> ... I don't think so 19:43:53 <jeblair> fungi: that sounds so professional! 19:43:54 <reed> jeblair, next topic for me 19:43:58 <jeblair> #topic Review proposal towards a better integration of gerrit and openstack people db (reed) 19:44:16 <reed> didn't we talk about this last week already? 19:44:20 <anteaya> yes 19:44:34 <jeblair> sorry, wasn't here; next topic again? 19:44:34 <anteaya> did we cover everything you needed last week? 19:44:54 <reed> anteaya, I think so, I'm good so far 19:44:58 <jeblair> #topic Request for comments on the horizon split plan (rdopiera) 19:45:00 <anteaya> great, thanks 19:45:05 <reed> there is a message to the mlist too 19:45:10 <anteaya> this was a left over from last week as well 19:45:18 <anteaya> din't know if we were finished here or not 19:45:27 <anteaya> he isn't in channel 19:46:17 <anteaya> looks like a fresher topic is in order 19:46:55 <anteaya> did we lose jim? 19:47:10 <jeblair> #topic Review Third Party wiki templates - seed pages (anteaya) 19:47:18 <anteaya> there we go 19:47:21 <jeblair> this meeting is speeding up! 19:47:23 <anteaya> #link https://wiki.openstack.org/wiki/ThirdPartySystems/IBMPowerKVMCI 19:47:33 <anteaya> #link https://wiki.openstack.org/wiki/ThirdPartySystems/OpenStack-Neutron 19:47:40 <anteaya> we looked at these last week 19:47:55 <anteaya> reed wanted a more prominent title, I don't know how to do that 19:48:07 <anteaya> are we ready to go live with these? 19:48:11 <anteaya> folks are keen to use them 19:48:42 <reed> I support this sort of standardization wholeheartedly 19:48:45 <anteaya> any reason to not start using them? 19:48:46 <krtaylor> anteaya, I'd like to change the wording a bit on ours, but generally, yes 19:48:48 <anteaya> reed: thanks 19:48:51 <clarkb> I think we can go ahead with what we have and work on the templates as we go 19:48:57 <clarkb> as far as format for larger title goes 19:48:58 <anteaya> krtaylor: right but the template 19:49:02 <krtaylor> yes 19:49:07 <anteaya> great thanks 19:49:13 <anteaya> I'm good to move on 19:49:37 <reed> i only wish mediawiki had a better UI... one day, it'll get there :) 19:50:01 <fungi> yeah, it looks like an encyclopedia or something 19:50:19 <SergeyLukjanov> I like how template looks like // https://wiki.openstack.org/wiki/ThirdPartySystems/IBMPowerKVMCI 19:50:38 <anteaya> SergeyLukjanov: thanks 19:51:01 <jeblair> anteaya: do we have docs prepared telling people how to fill these out, etc? 19:51:05 <jeblair> (and that they should fill them out)? 19:51:09 <anteaya> no 19:51:22 <anteaya> I was going to start small and get some good examples 19:51:28 <anteaya> and then work on a docs page 19:51:39 <anteaya> where should the doc page be, anyplace special? 19:51:50 <fungi> next order of business is probably to submit a change for review to update third-party.rst 19:51:50 <anteaya> or just somewhere on the wiki? 19:51:56 <reed> on the wiki ? 19:52:04 <anteaya> fungi: that sounds good 19:52:17 <jeblair> yeah, what fungi said 19:52:20 <anteaya> kk 19:52:30 <anteaya> my next topic is a quick one 19:52:34 <jeblair> then once that's there, we can write an email letting people know about the changes and link to it 19:52:39 <fungi> whether the bulk of the documentation goes onto the wiki with just a stub paragraph and link in that file or whether it's all documented in that file, i don't really have a preference 19:52:41 <anteaya> kk 19:52:54 <anteaya> I'll offer something and get some feedback 19:53:01 <jeblair> i think the bulk should be in third-party.rst 19:53:03 <anteaya> and expand the example pages 19:53:07 <anteaya> I can do that 19:54:02 <jeblair> #topic Formatting Automated Gerrit Account Names (anteaya) 19:54:11 <anteaya> #link https://wiki.openstack.org/wiki/ThirdPartySystems/OpenStack-Neutron 19:54:20 <anteaya> so we agreed to some formatting last week 19:54:29 <anteaya> my question is where should I publish it? 19:54:35 <anteaya> thirdparty.rst? 19:54:44 <jeblair> anteaya: do you mean https://etherpad.openstack.org/p/automated-gerrit-account-naming-format ? 19:54:58 <anteaya> yes 19:55:02 <anteaya> that was the link 19:55:02 <clarkb> anteaya: or even on the wiki. I think because so much data goes into the wiki it may be good to make thirdparty.rst more of a stub into the wiki 19:55:05 <anteaya> sorry about that 19:55:19 <clarkb> but liek fungi said I think either way will work 19:55:24 <anteaya> lines 11-16 is what I would like to document 19:55:34 <anteaya> actually I need to get infra-manual going 19:55:49 <jeblair> clarkb: i'm strongly in favor of rst; our requirements are not subject to anyone editing them, they are subject to review 19:55:49 <anteaya> and would like to have a third party section in infra-manual 19:55:55 <clarkb> jeblair: good point 19:56:03 <anteaya> I can get behind that 19:56:12 <jeblair> anteaya: yeah, infra-manual could work too 19:56:18 <clarkb> ++ 19:56:23 <jeblair> anteaya: i think lines 11-16 are good for rst docs 19:56:25 <mordred> infra-manual++ 19:56:34 <anteaya> thirdparty.rst for now and infra-manual for expansion 19:56:37 <anteaya> jeblair: thanks 19:56:40 <anteaya> I'm done 19:56:42 <fungi> i agree that something going through gerrit gets votes and an approval/audit trail, so good thing for policy document 19:56:44 <fungi> s 19:56:51 <anteaya> agreed 19:58:05 <clarkb> I have two things I wanted to put on peoples radars that don't really need a topic. 1) we need to restart zuul to pick up a bugfix around the swift stuff and 2) elasticsearch seems to be hving trouble with its rax volumes 19:58:15 <jeblair> #topic Open Discussion 19:58:23 <anteaya> ianw had an item 19:58:25 <clarkb> if we bump our volume quota we can shift to ssd volumes and hopefully make ES happy 19:58:26 <anteaya> f20 19:58:37 <jeblair> anteaya: there was no name and we're out of time for that i think 19:58:46 <anteaya> oh I did add him 19:58:47 <clarkb> but we can't do that without downtime with our current volume quota 19:58:48 <anteaya> sorry ianw 19:59:04 <anteaya> but I added him after the meeting started 19:59:21 <fungi> reminder: i'm not really around at the moment... but this place gets furniture and an internet connection tomorrow, so i should be catching up again late in the week 19:59:30 <jeblair> clarkb: if we can't get in touch with pvo, we should ask phschwartz 19:59:36 <anteaya> fungi: glad to have you back once you are back 19:59:40 <jeblair> also, i've started a new job at HP 19:59:46 <anteaya> fungi: hope things are going smoothly so far 19:59:47 <clarkb> jeblair: ya he pointed at pvo but we can keep asking around :) 19:59:52 <anteaya> jeblair: yay! 19:59:54 <wenlock_> jeblair grats! welcome 20:00:17 <jeblair> it's an architectural role relating to HP's use of our development tools/systems/methodologies 20:00:30 <anteaya> awesome 20:00:51 <jeblair> my interaction with openstack will be substantially similar, so no huge changes there 20:01:04 <jeblair> and i think we're out of time 20:01:10 <zaro> good to hear! 20:01:12 <jeblair> #endmeeting