19:01:03 <jeblair> #startmeeting infra
19:01:03 <jeblair> #link agenda https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:01:03 <jeblair> #link previous meeting http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-06-23-19.02.html
19:01:03 <openstack> Meeting started Tue Jun 30 19:01:03 2015 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:07 <openstack> The meeting name has been set to 'infra'
19:01:08 <pabelanger> o/
19:01:09 <mrmartin> o/
19:01:21 <jeblair> #topic Announcements
19:01:30 <jeblair> #info adding asselin to puppet-openstackci-core
19:01:44 <jhesketh> o/
19:01:45 <jeblair> yay!
19:01:51 <pleia2> woo :)
19:01:52 <pabelanger> /clap
19:01:57 <ianw> o/
19:02:10 <fungi> asselin: thanks for volunteering for this burden
19:02:22 <jeblair> i'll do that real soon now, and hopefully we'll be able to add some more folks soon too
19:02:27 <jeblair> i know there's interest
19:02:44 <jeblair> #topic Specs approval
19:02:51 <jeblair> #topic Specs approval: RefStack Site Hosting (hogepodge, davidlenwell)
19:02:56 <jeblair> #link refstack site hosting spec http://specs.openstack.org/openstack-infra/infra-specs/specs/refstack_dot_org.html
19:02:59 <jeblair> #info refstack site hosting spec was approved
19:03:03 <hogepodge> o/
19:03:08 <SpamapS> o/
19:03:26 <fungi> i've been more terrible than usual about finding time to review these, so thanks to everyone else who did and picked up my slack
19:04:03 <hogepodge> thanks everyone for approving refstack
19:04:25 <hogepodge> Happy to help with the deployment.
19:04:34 <fungi> hogepodge: i'll hold you to that ;)
19:04:50 <hogepodge> \o/
19:05:00 <jeblair> anything else we should cover on this?
19:05:39 <pabelanger> I have 1, but can wait until open discussion
19:06:08 <jeblair> pabelanger: we don't have a huge agenda today, go for it
19:06:10 <mordred> o/
19:06:50 <pabelanger> So, I created a spec to host the trystack.org since under -infra.  The static content of the web site, not the openstack bits: https://review.openstack.org/#/c/195098/
19:07:14 <jeblair> pabelanger: oh, i thought you had a refstack thing
19:07:15 <pabelanger> So far, a team member at RedHat is open to the idea of moving it into trystack.o.o
19:07:22 <pabelanger> jeblair, no, something new
19:07:31 <pabelanger> sorry, if I was unclear
19:07:34 <fungi> sounds like a separate meeting topic
19:07:41 <jeblair> yeah, let's move on
19:07:43 <pabelanger> roger
19:07:46 <jeblair> #topic Priority Efforts (Migration to Zanata)
19:07:58 <jeblair> we had some action items from last time
19:08:02 <jeblair> jeblair set up translate@o.o infra-root alias
19:08:04 <jeblair> i did that
19:08:07 <jeblair> pleia2 sign up translate@ openstackid account
19:08:23 <pleia2> the account is created on openstackid
19:08:29 <jeblair> er, except i think i called it 'zanata'
19:08:34 <jeblair> which i think is the better choice
19:08:37 <pleia2> yeah
19:08:49 <jeblair> and then:
19:08:49 <jeblair> fungi mrmartin pleia2 investigate "admin interface" for openstack id, get service account group assigned somehow
19:08:57 <jeblair> anything come of that?
19:09:05 <pleia2> zanata was pointed at openstackid-dev though, so a patch merged this morning to have zanata use that, so soon I can set up the zanata account itself
19:09:07 <fungi> yeah, that definitely didn't happen on my part
19:09:13 <pleia2> not yet
19:09:14 <fungi> i still need to ask around about that
19:09:19 <mrmartin> not yet, we need to check it
19:09:26 <jeblair> #action fungi mrmartin pleia2 investigate "admin interface" for openstack id, get service account group assigned somehow
19:09:26 <mrmartin> not a big deal, need to allocate time
19:09:30 <pleia2> thanks mrmartin, let me know what you need
19:09:35 <mrmartin> time
19:09:37 <pleia2> :)
19:09:43 <mrmartin> :)
19:09:43 <pleia2> *from me
19:09:46 * fungi knows that feeling
19:10:32 <jeblair> See discussion on https://review.openstack.org/#/c/187867/ need infra input on translation-projects.txt vs. changes to projects.yaml
19:10:44 <jeblair> we also have that in the meeting agenda ^
19:11:09 <pleia2> yep, this is a result of a chat between AJaeger and StevenK over (my) night
19:11:17 <pleia2> I summarized it in the review, need input from others
19:11:47 <pleia2> do we want to create a new translation-projects.txt file in project-config, or can we ad a translate: true stanza to projects.yaml to add projects to zanata
19:11:54 <pleia2> s/ad/add
19:12:10 <jeblair> ah, yeah, i think despite appearances to the contrary, we actually don't want to have tons of files named "projects.yaml" so i like (b)
19:12:13 * AJaeger would love not to have an admin go into Zanata and manually set up a new project
19:12:23 <pleia2> either way, good news is we can now add projects via code review, not through the translations web API like we do with transifex now :)
19:12:29 <pleia2> AJaeger ++
19:12:30 <jeblair> AJaeger: yeah, that's definitely the goal
19:12:30 <fungi> we've added fields to gerrit/projects.yaml for non-gerrit-specific aspects which are still repo-relative
19:12:43 <pleia2> s/API/UI
19:12:45 <pleia2> phew
19:13:27 <fungi> it's mostly for stuff driven from jeepyb, but i don't think it has to be _just_ that
19:13:34 * pleia2 nods
19:13:40 <AJaeger> pleia2: Zanata operation could also be done via puppet
19:13:50 <fungi> for example, storyboard was consuming that directly for creating project-groups
19:13:54 <pleia2> AJaeger: yeah, exciting times :)
19:14:04 <jeblair> i'd say we have consensus on b, unless there are objections?
19:14:11 <pleia2> wfm
19:14:28 <tchaypo> Sgtm
19:14:38 <pleia2> I'll let StevenK know and we'll work on the required changes elsewhere to support it
19:14:39 <AJaeger> whatever works ;)
19:14:47 <AJaeger> thanks, pleia2
19:14:56 <jeblair> any other zanata items?
19:15:12 <pleia2> we're handing it off to Daisy this week, and opening testing to the broader community next week
19:15:18 <pleia2> so, yay :)
19:15:19 <pleia2> that's all
19:15:20 <jeblair> awesome!
19:15:52 <jeblair> #topic trystack web site hosting (pabelanger)
19:15:57 <jeblair> okay, late addition! :)
19:16:16 <pabelanger> ha, ya sorry. Was going to bring it up in open discussion
19:16:46 <pabelanger> either way, I think there is a good chance to move trystack.org under -infra if people are open to it.  Not the openstack bits, just the static contents of the website
19:16:49 <pabelanger> spec is up
19:16:54 <pabelanger> and puppet bits working too
19:17:26 <pabelanger> the spec: https://review.openstack.org/#/c/195098/
19:17:47 <pabelanger> I assume we'd get the people managing the site now, and openstack foundation board to chime in on the move
19:17:52 <fungi> that seems like relatively low-hanging fruit we should knock out if all parties are in agreement. thanks!
19:18:05 <SpamapS> is it a truly static website?
19:18:12 <pabelanger> SpamapS, yup
19:18:18 <SpamapS> is there any reason thats not just in swift then?
19:18:28 <fungi> the openstack service bits of that (and the embarrassing facebook groups auth model it's reliant on for the moment) will be a tougher nut to crack
19:18:32 <pabelanger> SpamapS, https://github.com/trystack/trystack_org_webcontents
19:18:39 <pabelanger> the plan would be to move that in to -infra too
19:19:03 <jeblair> yeah, i think getting the web site hosted is a good first step to make it more of a community effort
19:19:30 <jeblair> i don't want us to get involved in running the site yet, but after we have infra-cloud up and running for a while, i think we will be in a position to start talking abotu that
19:19:57 <jeblair> sorry, by that last, i mean running the actual openstack service
19:20:18 <mrmartin> we should handle this the same way, as we are doing for openstackid, group, and maybe o.o, typical web projects
19:20:26 <mrmartin> and ask.o.o
19:20:31 <jeblair> mrmartin: handle what?
19:20:39 <fungi> there's still a large unknown around that piece, which is self-service sign-up/single-sign-on for keystone
19:20:41 <mrmartin> trystack.o.o website
19:21:04 <jeblair> mrmartin: as a static site it's actualy a bit simpler and does not need its own server, but yes, in general
19:21:10 <fungi> mrmartin: well, the trystack website is just static content (docs)
19:21:12 <mrmartin> yeah, but self-service signup is a solvable part, if we have some spec / doc for that
19:21:25 * SpamapS feels like we got off track?
19:21:33 <fungi> yep
19:21:49 <fungi> two different problems to solve. the easy one is the trystack.org static content website. that's our topic
19:22:44 <pabelanger> Yes, right now the site is mostly unmanaged, and it would be fairly painless to move it into the -infra domain to start monitoring / managing it
19:22:56 <pabelanger> any other changes to login or servers, would need more detailed specs
19:23:00 <jeblair> SpamapS: i suppose we could put it in swift; we don't really have a generalized 'host a static site in swift' model.  even the proposed docs publishing change has problems with that model that are hard to overcome.
19:23:42 <jeblair> whereas, apache vhost is fairly common and easy
19:23:42 <fungi> i'm not opposed to someone solving that challenge, but we do already have virtual machines which can use apache to serve local files
19:24:24 <jeblair> if anyone is interested in the problems, see http://specs.openstack.org/openstack-infra/infra-specs/specs/doc-publishing.html
19:24:51 <jeblair> at least, i hope we captured the issues in there :)
19:24:53 <pabelanger> mrmartin, this is what it would look like if we imported today: https://review.openstack.org/#/c/194816/
19:24:54 <fungi> our uploading/serving job logs in swift and the os-loganalyze project is a good example of how that gets pretty complicated quickly
19:25:05 <SpamapS> jeblair: I think thats fine in each narrow case. From a broader view, one has to wonder why it isn't easier to use swift for its exact intended purpose though.
19:25:24 <pabelanger> sorry SpamapS ^
19:25:35 <jeblair> SpamapS: yeah, we provide feedback to swift folks for sure
19:25:48 <fungi> SpamapS: part of it comes down to how different service providers choose to set up their swift services too
19:25:59 <SpamapS> fungi: bingo
19:26:04 <jeblair> SpamapS: our use case is arguably not its exact intended purpose; it gets into splitting hairs too.
19:26:47 <jeblair> anyway, hope that provides some background :)
19:27:00 <jeblair> pabelanger: thanks for kicking that off
19:27:13 <jeblair> #topic Open discussion
19:27:15 <pabelanger> jeblair, np
19:27:21 <SpamapS> jeblair: totally, I don't want to discuss any details, however I do want to register my strong belief that the general direction I think we should take is to push things toward static hosting being done in swift in general. I understand that is easy to say, and not easy to do. :)
19:27:43 <SpamapS> Same reason we use Trove right? :)
19:28:00 <mrmartin> SpamapS: so you suggest to move all static content into swift?
19:28:03 <jeblair> SpamapS: which has also not been a resounding success
19:28:24 <asselin> o/
19:28:29 <SpamapS> Anyway, regarding infra-cloud, just a general status update: On the test node I have ironic,keystone,nuetron, and glance working. Nova is installed, but nova-api refuses to respond with anything other than 404's for all requests.
19:28:37 <SpamapS> mrmartin: absolutely.
19:28:42 <greghaynes> SpamapS: w00t
19:28:58 <SpamapS> mrmartin: the fact that there is any friction to that is a sign that investment is needed at any of those friction points.
19:29:02 <pabelanger> So, I managed to find the issue with our puppet-apply-centos6 gate job.  It looks to be an issue within the git client (1.7). Moving to 1.8 fixes the performance issue (30+mins down to 4mins).
19:29:13 <pabelanger> For the most part, i did a JJB that upgrades git inline: https://review.openstack.org/#/c/196761/
19:29:18 <pabelanger> would be curious to get some feedback
19:29:24 <pleia2> thanks pabelanger
19:29:38 <SpamapS> mrmartin: long term it should result in having less resources to manage, and more scalability.
19:29:42 <jeblair> SpamapS: please understand that this thought has occurred to us
19:29:57 <SpamapS> jeblair: I'm certain it has. I want to make sure we keep looping back to it.
19:29:59 <fungi> pabelanger: were you able to figure out whether a glibc change or similar earlier this year on centos 6 is what caused that to suddenly start happening for us?
19:30:00 <mrmartin> SpamapS:  the shortest path is going with static.o.o for trystack but in middleterm I can imagine some efforts to move static content to swift
19:30:14 <SpamapS> jeblair: perhaps those with more historical context are more weary than I? ;)
19:30:27 <pabelanger> fungi, no, I didn't get that far.  I wanted to first try a newer client to see if it was affected too.
19:30:34 <jeblair> SpamapS: i hope you will find some of that in the spec i linked
19:30:54 <pabelanger> fungi, I was going to look into the 1.8 branch and see if I could find a commit that addressed the problem
19:31:02 <SpamapS> jeblair: I did, thank you. :) (I've also been trying to withdraw from this point for about 10 minutes now... ;)
19:31:34 <jeblair> k, let's talk about centos6 git
19:31:47 <jeblair> istr we tried to use git:// on centos6 jobs for a similar problem in the past...
19:31:55 <jeblair> does that still help things?  did that regress?
19:31:56 <fungi> SpamapS: put down the swift and back away slowly ;)
19:33:09 <pabelanger> jeblair, I tried git:// and http:// for URI's there was an improvement but still some delay
19:33:31 <ttx> Open discussion, I spun up a phabricator instance to play a bit with CustomFields
19:33:32 <pabelanger> https was the biggest hit to performance
19:33:45 <ttx> will play tomorrow
19:34:33 <pabelanger> jeblair, at the end of the day, upgrading git yielded the same performance as ubuntu hosts
19:34:45 <pabelanger> I never could get centos6 down to 5min mark using different URIs
19:35:06 <jeblair> pabelanger: so we can fix this with newer git on centos, or potentially finding the underlying thing on centos that changed (eg, glibc?)
19:35:15 <pabelanger> right
19:35:23 <jeblair> or we can stop using centos6 and move to centos7
19:35:26 <jeblair> perhaps
19:35:31 <pabelanger> yes, that was another thought
19:35:41 <pabelanger> I was looking this morning what centos6 we had for -infra and puppet
19:35:45 <pabelanger> i didn't see much
19:35:58 <jeblair> i suspect we can upgrade infra fairly easily
19:36:01 <pabelanger> I was talking to fungi about bare-centos7 image and such
19:36:11 <jeblair> the question is what test jobs need centos6
19:36:28 <pabelanger> I could get a list together for that
19:37:11 <fungi> worrisome is that if we upgrade our git servers to centos 7 we may stop supporting shallow clones
19:37:24 <fungi> is git there as new as on ubuntu trusty?
19:38:22 <pabelanger> unsure for me
19:39:08 <jeblair> it's not clear that the git authors want to support that, so...
19:40:05 <fungi> i'd be okay with declaring that we don't support shallow clones from our git server farm, but not sure who might rebel over that
19:40:23 <mordred> they can rebel all they want - we can't run centos6 for forever
19:40:30 <jeblair> we can point them at the git bug
19:40:33 <mordred> ++
19:40:39 <mordred> maybe a friend somewhere will fix it
19:41:11 <fungi> great point
19:41:38 <jeblair> so i think the ideal solutions would be: 1) underlying centos6 bug is fixed; 2) centos6 upgrades git; 3) we remove our use of centos6; 4) we self-upgrade git on our centos6 images
19:41:48 <jeblair> in order of preference ^
19:41:49 <jeblair> yeah?
19:42:33 <pabelanger> That would be best case IMO
19:43:46 <jeblair> pabelanger: thanks for digging into it
19:43:52 <pabelanger> np
19:44:06 <jeblair> ttx: cool, good luck!
19:44:30 <ttx> jeblair: I think I'll killfile that OOO from mpstor.com dude
19:44:49 <jeblair> that hasn't happened yet?
19:45:02 <jeblair> ttx: thank you :)
19:45:05 <ttx> jeblair: I assumed it was, but they keep on coming
19:45:37 <ttx> btw if someone wants to volunteer to help on -dev I'll take it as a co-admin
19:46:41 <ttx> hmm, he isn't subscribed (or not subscribed anymore)
19:47:13 <jeblair> ttx: let us know if we need to dig into logs or something
19:47:37 <asselin> Hi, I'd like to remind people of the common-ci virtual sprint next week (July 8-9): Sign up here: https://etherpad.openstack.org/p/common-ci-sprint
19:48:13 <greghaynes> ttx: I think its on -anounce
19:48:16 <jeblair> hey look new signups!
19:48:20 <asselin> #link common-ci virtual sprint https://etherpad.openstack.org/p/common-ci-sprint
19:48:37 <greghaynes> (remember we set -anounce to reply to -dev)
19:49:34 <ttx> jeblair: ok, I got it. He is subscribed to openstack@lists.o.o and there is a whitelist on -dev for senders from that list
19:49:48 <greghaynes> ah
19:49:52 <ttx> two options. Disable the whitelist or remove him from openstack list
19:50:28 <jeblair> i think removing from openstack makes the most sense
19:50:43 <jeblair> (i think adding him to a blacklist is a 3rd option, but also probably not the best)
19:50:47 <ttx> or I can add him to reject_these_nonmembers but no idea what takes precedence
19:51:03 <ttx> easier that he resubscribes when back from vacation.
19:51:20 <ttx> If I could find the guy who invented out of office notices
19:51:28 <ttx> I'd make him SUFFER
19:52:01 <jeblair> ttx: redirect all ooo notices to them
19:52:14 <tchaypo> you could only make him suffer after he gets back in the office
19:52:38 <ttx> ok, for the record he was unsubbed.
19:52:55 <fungi> http://lists-archives.com/git/843412-setup_temporary_shallow-use-tempfile-module.html looks promising
19:53:46 <jeblair> oh good there' hope
19:54:05 <fungi> yeah, i think they're still deciding how to solve it and less whether to solve it
19:55:06 <jeblair> anything else or should we wrap up?
19:56:11 <jeblair> thanks all!
19:56:14 <jeblair> #endmeeting